2024-09-17T15:52:10.440 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T15:52:10.461 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T15:52:10.550 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024 branch: wip-bharath10-testing-2024-09-17-1348 description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} email: skanta@redhat.com first_in_suite: false job_id: '7909024' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath10-testing-2024-09-17-1348 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-bharath10-testing-2024-09-17-1348 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 3421 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath10-testing-2024-09-17-1348 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 targets: smithi017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwaXItMCXNuvyIe7imWEwRtmPYXEOKZZq34qGkVr9ruXQSg8jh3Ygfs9DHVakjlvqrxup21S2iV5iofA3HwZIc= smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGkpd3kP7sXPLbsfEaZcEMqbzaOgpmXhInjF1QN2RCvKddfk3VKJgZf8Bz3luJOvyXga0Ai2T8WtE22LsHBnXPY= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWz5u5t1H27KTwEJ2Ubj5Q/F03DZjydl3R7arixAMkKk3DXUztk0/UrnlJ8ZA+NsLLdx6DPlRtMwF2TQbBngOg= tasks: - install: branch: squid exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_15:01:33 tube: smithi user: skanta verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T15:52:10.550 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa; will attempt to use it 2024-09-17T15:52:10.551 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks 2024-09-17T15:52:10.551 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T15:52:10.553 INFO:teuthology.task.internal:Saving configuration 2024-09-17T15:52:10.568 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T15:52:10.575 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T15:52:10.595 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:47:06.282405', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwaXItMCXNuvyIe7imWEwRtmPYXEOKZZq34qGkVr9ruXQSg8jh3Ygfs9DHVakjlvqrxup21S2iV5iofA3HwZIc='} 2024-09-17T15:52:10.609 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:47:06.279254', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGkpd3kP7sXPLbsfEaZcEMqbzaOgpmXhInjF1QN2RCvKddfk3VKJgZf8Bz3luJOvyXga0Ai2T8WtE22LsHBnXPY='} 2024-09-17T15:52:10.623 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:47:06.280998', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWz5u5t1H27KTwEJ2Ubj5Q/F03DZjydl3R7arixAMkKk3DXUztk0/UrnlJ8ZA+NsLLdx6DPlRtMwF2TQbBngOg='} 2024-09-17T15:52:10.623 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T15:52:10.629 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-17T15:52:10.629 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-09-17T15:52:10.629 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-09-17T15:52:10.629 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T15:52:10.704 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7521a13be0>, signals=[15]) 2024-09-17T15:52:10.704 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T15:52:10.711 INFO:teuthology.task.internal:Opening connections... 2024-09-17T15:52:10.711 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2024-09-17T15:52:10.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:10.789 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-09-17T15:52:10.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:10.866 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T15:52:10.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:10.942 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T15:52:10.949 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2024-09-17T15:52:10.967 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2024-09-17T15:52:10.967 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2024-09-17T15:52:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="CentOS Stream" 2024-09-17T15:52:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="9" 2024-09-17T15:52:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:ID="centos" 2024-09-17T15:52:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="rhel fedora" 2024-09-17T15:52:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="9" 2024-09-17T15:52:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el9" 2024-09-17T15:52:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T15:52:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2024-09-17T15:52:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:LOGO="fedora-logo-icon" 2024-09-17T15:52:11.025 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T15:52:11.026 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://centos.org/" 2024-09-17T15:52:11.026 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T15:52:11.026 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T15:52:11.026 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T15:52:11.027 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2024-09-17T15:52:11.047 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-09-17T15:52:11.066 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-09-17T15:52:11.066 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-09-17T15:52:11.124 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-09-17T15:52:11.124 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2024-09-17T15:52:11.124 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-09-17T15:52:11.124 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-09-17T15:52:11.124 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2024-09-17T15:52:11.124 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2024-09-17T15:52:11.125 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T15:52:11.125 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-09-17T15:52:11.125 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2024-09-17T15:52:11.125 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T15:52:11.125 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-09-17T15:52:11.125 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T15:52:11.125 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T15:52:11.125 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T15:52:11.125 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-09-17T15:52:11.147 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-09-17T15:52:11.166 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-09-17T15:52:11.166 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-09-17T15:52:11.224 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-09-17T15:52:11.224 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-09-17T15:52:11.224 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-09-17T15:52:11.224 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-09-17T15:52:11.224 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-09-17T15:52:11.224 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-09-17T15:52:11.224 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T15:52:11.224 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-09-17T15:52:11.225 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-09-17T15:52:11.225 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T15:52:11.225 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-09-17T15:52:11.225 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T15:52:11.225 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T15:52:11.225 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T15:52:11.225 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-09-17T15:52:11.247 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T15:52:11.257 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T15:52:11.263 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T15:52:11.263 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:52:11.266 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:52:11.268 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:52:11.284 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T15:52:11.290 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T15:52:11.290 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:52:11.323 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:52:11.325 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:52:11.435 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T15:52:11.512 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T15:52:11.526 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T15:52:11.526 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T15:52:11.526 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:52:11.527 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:52:11.527 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:52:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:52:11.544 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-09-17T15:52:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:52:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:52:11.546 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-09-17T15:52:11.546 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-09-17T15:52:11.601 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-503.el9.x86_64 2024-09-17T15:52:11.601 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-503.el9.x86_64 2024-09-17T15:52:11.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-09-17T15:52:11.604 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-503.el9.x86_64 2024-09-17T15:52:11.604 INFO:teuthology.task.kernel:Running kernel on smithi017: 5.14.0-503.el9.x86_64 2024-09-17T15:52:11.604 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum install -y kernel 2024-09-17T15:52:11.605 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-503.el9.x86_64 2024-09-17T15:52:11.605 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.14.0-503.el9.x86_64 2024-09-17T15:52:11.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2024-09-17T15:52:14.217 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.3 MB 00:00 2024-09-17T15:52:14.299 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 8.3 MB 00:00 2024-09-17T15:52:16.034 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 33 MB/s | 20 MB 00:00 2024-09-17T15:52:16.129 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.3 MB 00:02 2024-09-17T15:52:17.454 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 20 MB 00:02 2024-09-17T15:52:19.272 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 20 MB 00:02 2024-09-17T15:52:20.484 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-17T15:52:22.452 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 19 kB 00:00 2024-09-17T15:52:22.654 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.5 MB 00:01 2024-09-17T15:52:23.348 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-17T15:52:24.213 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 19 kB 00:00 2024-09-17T15:52:24.906 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-17T15:52:25.150 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-17T15:52:26.357 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-17T15:52:27.285 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-17T15:52:28.945 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T15:52:30.614 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T15:52:30.893 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T15:52:30.893 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T15:52:31.004 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T15:52:31.014 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T15:52:31.014 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-09-17T15:52:31.015 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T15:52:31.016 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 79 M 2024-09-17T15:52:31.016 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 126 M 2024-09-17T15:52:31.016 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T15:52:31.421 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 279 kB/s | 36 kB 00:00 2024-09-17T15:52:32.479 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T15:52:32.479 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T15:52:32.584 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T15:52:32.598 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T15:52:32.598 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-17T15:52:32.598 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T15:52:32.598 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T15:52:32.598 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T15:52:32.598 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-17T15:52:32.598 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T15:52:32.598 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T15:52:32.598 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T15:52:32.598 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T15:52:32.599 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T15:52:32.599 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T15:52:32.599 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-09-17T15:52:32.599 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T15:52:32.599 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 79 M 2024-09-17T15:52:32.599 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 126 M 2024-09-17T15:52:32.599 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T15:52:32.658 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-17T15:52:32.824 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 320 kB/s | 36 kB 00:00 2024-09-17T15:52:34.564 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T15:52:34.564 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T15:52:34.657 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T15:52:34.667 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T15:52:34.667 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T15:52:34.667 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T15:52:34.667 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T15:52:34.667 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T15:52:34.667 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T15:52:34.667 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T15:52:34.667 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T15:52:34.667 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T15:52:34.668 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T15:52:34.668 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T15:52:34.668 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T15:52:34.668 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-09-17T15:52:34.668 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T15:52:34.669 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 79 M 2024-09-17T15:52:34.669 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 126 M 2024-09-17T15:52:34.669 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T15:52:35.276 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 64 kB/s | 36 kB 00:00 2024-09-17T15:52:36.118 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-17T15:52:36.315 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 4.3 MB/s | 16 MB 00:03 2024-09-17T15:52:36.755 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): kernel-modules-core-5.14.0-508.el9.x86_6 5.4 MB/s | 29 MB 00:05 2024-09-17T15:52:36.801 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 17 MB/s | 35 MB 00:02 2024-09-17T15:52:37.626 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-17T15:52:37.626 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:37.627 INFO:teuthology.orchestra.run.smithi173.stdout:Total 27 MB/s | 79 MB 00:02 2024-09-17T15:52:38.265 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 6.2 MB/s | 35 MB 00:05 2024-09-17T15:52:38.311 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T15:52:38.556 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 16 MB 00:07 2024-09-17T15:52:38.659 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T15:52:38.659 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T15:52:38.999 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 4.6 MB/s | 29 MB 00:06 2024-09-17T15:52:38.999 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:38.999 INFO:teuthology.orchestra.run.smithi071.stdout:Total 12 MB/s | 79 MB 00:06 2024-09-17T15:52:39.463 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 4.2 MB/s | 35 MB 00:08 2024-09-17T15:52:39.463 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:52:39.464 INFO:teuthology.orchestra.run.smithi017.stdout:Total 9.3 MB/s | 79 MB 00:08 2024-09-17T15:52:39.633 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T15:52:39.633 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T15:52:39.679 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T15:52:40.024 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T15:52:40.025 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T15:52:40.177 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T15:52:40.555 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T15:52:40.555 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T15:52:40.991 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T15:52:40.992 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T15:52:41.537 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T15:52:41.538 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T15:52:42.041 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T15:52:42.688 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:52:42.768 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:52:43.415 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T15:52:44.103 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T15:52:44.175 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:52:44.267 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:52:44.751 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:52:44.859 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:52:45.518 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:52:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:52:47.023 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:52:47.172 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:52:47.614 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:52:47.741 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:52:50.467 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:52:50.700 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:52:51.990 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:52:52.030 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:52:52.864 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:52:52.908 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:52:55.545 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:52:56.858 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:52:58.051 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:33.874 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:33.915 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:34.279 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:34.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:53:34.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:53:34.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:53:35.147 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:35.147 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T15:53:35.147 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T15:53:35.147 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T15:53:35.147 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T15:53:35.148 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T15:53:35.148 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T15:53:35.148 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T15:53:35.148 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T15:53:35.412 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:35.452 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:35.504 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:53:35.592 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:53:35.592 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:53:35.592 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T15:53:35.592 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:53:35.593 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:53:35.593 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-17T15:53:35.593 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:53:35.593 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:53:35.652 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:53:35.652 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:53:35.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-09-17T15:53:35.695 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-09-17T15:53:35.695 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-09-17T15:53:35.695 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-09-17T15:53:35.695 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 82.el9 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8274670 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T15:53:35.696 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-09-17T15:53:35.697 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-09-17T15:53:35.697 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T15:53:35.697 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T15:53:35.697 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T15:53:35.697 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T15:53:35.697 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-09-17T15:53:35.697 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T15:53:35.697 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T15:53:35.698 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T15:53:35.698 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-508.el9.x86_64 2024-09-17T15:53:35.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T15:53:35.795 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:35.795 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:53:35.795 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:53:35.795 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:53:36.342 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-09-17T15:53:36.629 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:36.629 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T15:53:36.630 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T15:53:36.630 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T15:53:36.630 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T15:53:36.630 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T15:53:36.630 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T15:53:36.630 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T15:53:36.630 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T15:53:36.971 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:53:37.057 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:53:37.057 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:53:37.057 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T15:53:37.057 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:53:37.057 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:53:37.057 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T15:53:37.058 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:53:37.058 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:53:37.076 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:37.116 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:53:37.116 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:53:37.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2024-09-17T15:53:37.117 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:37.148 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2024-09-17T15:53:37.148 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2024-09-17T15:53:37.148 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.06 2024-09-17T15:53:37.149 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 82.el9 2024-09-17T15:53:37.149 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2024-09-17T15:53:37.149 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T15:53:37.149 INFO:teuthology.orchestra.run.smithi071.stdout:Group : Unspecified 2024-09-17T15:53:37.149 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 8274670 2024-09-17T15:53:37.149 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2024-09-17T15:53:37.149 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T15:53:37.149 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T15:53:37.149 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : builder@centos.org 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T15:53:37.150 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T15:53:37.151 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T15:53:37.151 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 5.14.0-508.el9.x86_64 2024-09-17T15:53:37.152 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T15:53:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:53:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:53:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:53:37.623 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T15:53:37.653 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-09-17T15:53:37.656 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-09-17T15:53:37.672 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.sNw9vxk68U 2024-09-17T15:53:37.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sNw9vxk68U 2024-09-17T15:53:37.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.sNw9vxk68U 2024-09-17T15:53:37.848 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-09-17T15:53:37.874 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.sNw9vxk68U is 6KB 2024-09-17T15:53:37.923 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.sNw9vxk68U 2024-09-17T15:53:37.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T15:53:38.001 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T15:53:38.001 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T15:53:38.001 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T15:53:38.001 INFO:teuthology.orchestra.run.smithi173.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T15:53:38.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T15:53:38.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-09-17T15:53:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:53:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T15:53:38.317 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T15:53:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T15:53:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T15:53:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T15:53:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T15:53:38.317 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T15:53:38.317 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T15:53:38.685 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:53:38.770 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:53:38.771 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:53:38.771 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T15:53:38.771 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:53:38.771 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:53:38.771 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T15:53:38.771 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:53:38.771 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:53:38.830 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:53:38.830 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:53:38.830 DEBUG:teuthology.orchestra.run.smithi017:> sudo rpm -qi grub2-tools 2024-09-17T15:53:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:Name : grub2-tools 2024-09-17T15:53:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:Epoch : 1 2024-09-17T15:53:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:Version : 2.06 2024-09-17T15:53:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:Release : 82.el9 2024-09-17T15:53:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: x86_64 2024-09-17T15:53:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T15:53:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:Group : Unspecified 2024-09-17T15:53:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:Size : 8274670 2024-09-17T15:53:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:License : GPLv3+ 2024-09-17T15:53:38.862 INFO:teuthology.orchestra.run.smithi017.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:Packager : builder@centos.org 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:Vendor : CentOS 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:Summary : Support tools for GRUB. 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:Description : 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout:hardware devices. 2024-09-17T15:53:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T15:53:38.864 INFO:teuthology.orchestra.run.smithi017.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T15:53:38.864 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T15:53:38.864 INFO:teuthology.task.kernel:Updating grub on smithi017 to boot 5.14.0-508.el9.x86_64 2024-09-17T15:53:38.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T15:53:39.057 INFO:teuthology.orchestra.run.smithi071.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T15:53:39.088 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-09-17T15:53:39.094 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-09-17T15:53:39.109 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.goPkXXcfen 2024-09-17T15:53:39.110 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.goPkXXcfen 2024-09-17T15:53:39.183 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.goPkXXcfen 2024-09-17T15:53:39.344 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.goPkXXcfen is 6KB 2024-09-17T15:53:39.379 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.goPkXXcfen 2024-09-17T15:53:39.398 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T15:53:39.472 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T15:53:39.473 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T15:53:39.473 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T15:53:39.473 INFO:teuthology.orchestra.run.smithi071.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T15:53:39.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T15:53:39.522 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2024-09-17T15:53:39.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-09-17T15:53:40.881 INFO:teuthology.orchestra.run.smithi017.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T15:53:40.921 INFO:teuthology.orchestra.run.smithi017.stderr:done 2024-09-17T15:53:40.922 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-09-17T15:53:40.938 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.iqUMhC5vXE 2024-09-17T15:53:40.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iqUMhC5vXE 2024-09-17T15:53:41.004 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.iqUMhC5vXE 2024-09-17T15:53:41.155 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.iqUMhC5vXE is 6KB 2024-09-17T15:53:41.203 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.iqUMhC5vXE 2024-09-17T15:53:41.218 DEBUG:teuthology.orchestra.run.smithi017:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T15:53:41.280 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T15:53:41.280 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T15:53:41.280 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T15:53:41.280 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T15:53:41.281 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T15:53:41.432 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2024-09-17T15:54:08.173 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-17T15:54:08.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T15:54:08.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:54:09.633 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T15:54:09.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-17T15:54:09.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:54:11.437 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T15:54:11.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-17T15:54:11.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:54:26.493 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-09-17T15:54:28.029 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-09-17T15:54:29.821 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2024-09-17T15:54:35.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T15:54:35.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:54:37.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-17T15:54:37.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:54:38.561 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-09-17T15:54:38.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-17T15:54:38.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:54:40.093 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-09-17T15:54:50.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T15:54:50.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:54:52.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-17T15:54:52.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:55:22.467 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-09-17T15:55:22.857 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T15:55:22.857 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-508.el9.x86_64"... 2024-09-17T15:55:22.857 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-09-17T15:55:22.871 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-508.el9.x86_64 2024-09-17T15:55:22.871 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T15:55:22.872 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:55:22.872 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T15:55:23.844 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-09-17T15:55:23.872 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-17T15:55:23.872 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:55:23.932 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-09-17T15:55:23.950 DEBUG:teuthology.parallel:result is None 2024-09-17T15:55:24.422 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-17T15:55:24.423 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-17T15:55:24.423 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-09-17T15:55:24.466 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-508.el9.x86_64 2024-09-17T15:55:24.467 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T15:55:24.467 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:55:24.467 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T15:55:25.468 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T15:55:25.468 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:55:25.668 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-09-17T15:55:25.688 DEBUG:teuthology.parallel:result is None 2024-09-17T15:55:38.826 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T15:55:50.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-17T15:55:50.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:55:51.112 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-09-17T15:55:51.454 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-17T15:55:51.454 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T15:55:51.455 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-09-17T15:55:51.472 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-508.el9.x86_64 2024-09-17T15:55:51.473 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T15:55:51.473 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:55:51.473 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T15:55:52.474 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T15:55:52.474 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:55:52.541 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2024-09-17T15:55:52.570 DEBUG:teuthology.parallel:result is None 2024-09-17T15:55:52.570 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T15:55:52.577 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T15:55:52.577 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:55:52.580 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:55:52.583 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:55:52.603 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T15:55:52.610 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T15:55:52.620 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T15:55:52.620 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:55:52.639 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:55:52.642 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:55:52.739 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T15:55:52.746 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T15:55:52.747 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:55:52.749 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:55:52.751 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:55:52.783 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:55:52.798 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:55:52.798 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:55:52.798 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:55:52.799 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:55:52.801 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:55:52.803 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T15:55:52.812 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T15:55:52.812 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:55:52.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:55:52.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:55:52.879 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T15:55:52.888 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T15:55:52.889 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:55:52.917 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:55:52.920 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:55:52.936 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:55:53.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:55:53.069 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:55:53.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:55:53.197 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T15:55:53.197 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:55:53.264 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:55:53.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:55:53.358 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:55:53.444 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:55:53.488 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T15:55:53.488 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:55:53.555 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:55:53.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:55:53.651 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:55:53.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:55:53.778 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T15:55:53.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:55:53.844 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2024-09-17T15:55:53.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-09-17T15:55:53.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-09-17T15:55:53.891 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:55:53.895 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:55:53.924 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:55:54.431 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T15:55:54.440 INFO:teuthology.task.internal:Starting timer... 2024-09-17T15:55:54.440 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T15:55:54.496 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T15:55:54.578 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-17T15:55:54.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo service auditd rotate 2024-09-17T15:55:54.632 INFO:teuthology.orchestra.run.smithi017.stdout:Rotating logs: 2024-09-17T15:55:54.634 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-09-17T15:55:54.694 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-09-17T15:55:54.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-09-17T15:55:54.760 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-09-17T15:55:54.763 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T15:55:54.763 DEBUG:teuthology.orchestra.run.smithi017:> /usr/sbin/getenforce 2024-09-17T15:55:54.785 INFO:teuthology.orchestra.run.smithi017.stdout:Permissive 2024-09-17T15:55:54.785 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-09-17T15:55:54.807 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-09-17T15:55:54.807 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-09-17T15:55:54.829 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-09-17T15:55:54.829 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi017.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-09-17T15:55:54.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-17T15:55:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:55:54.860 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-17T15:55:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:55:54.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-17T15:55:54.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:55:54.919 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T15:55:54.919 DEBUG:teuthology.orchestra.run.smithi017:> sudo /usr/sbin/setenforce permissive 2024-09-17T15:55:54.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-09-17T15:55:54.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-09-17T15:55:55.034 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T15:55:55.044 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-17T15:55:55.047 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-17T15:55:55.161 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T15:55:55.172 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T15:55:55.187 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi071.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T15:59:42.646 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-09-17T15:59:42.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-17T15:59:42.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:59:42.730 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-09-17T15:59:42.822 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-17T15:59:42.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-17T15:59:42.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:59:42.914 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-09-17T15:59:42.988 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-09-17T15:59:42.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T15:59:42.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:59:43.068 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-09-17T15:59:43.151 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-09-17T15:59:43.151 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T15:59:43.162 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T15:59:43.162 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:59:43.163 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:59:43.166 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:59:43.166 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:59:43.168 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:59:43.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:59:43.195 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T15:59:43.198 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T15:59:43.210 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T15:59:43.215 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T15:59:43.223 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T15:59:43.234 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-09-17T15:59:43.238 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T15:59:43.243 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: ntpd: command not found 2024-09-17T15:59:43.245 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-09-17T15:59:43.255 INFO:teuthology.orchestra.run.smithi017.stdout:506 Cannot talk to daemon 2024-09-17T15:59:43.259 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T15:59:43.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-09-17T15:59:43.271 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T15:59:43.273 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T15:59:43.275 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-09-17T15:59:43.286 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T15:59:43.289 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T15:59:43.304 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T15:59:43.327 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:59:43.330 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:59:43.330 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-09-17T15:59:43.330 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.330 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.330 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.331 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.345 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:59:43.350 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:59:43.350 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-09-17T15:59:43.350 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.350 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.350 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.350 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.367 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:59:43.370 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:59:43.370 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-09-17T15:59:43.370 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.370 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.370 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.370 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:59:43.371 INFO:teuthology.run_tasks:Running task install... 2024-09-17T15:59:43.379 DEBUG:teuthology.task.install:project ceph 2024-09-17T15:59:43.379 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T15:59:43.379 DEBUG:teuthology.task.install:config {'branch': 'squid', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'} 2024-09-17T15:59:43.379 INFO:teuthology.task.install:Using flavor: default 2024-09-17T15:59:43.385 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T15:59:43.385 INFO:teuthology.task.install:extra packages: [] 2024-09-17T15:59:43.385 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T15:59:43.385 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T15:59:43.386 INFO:teuthology.packaging:ref: None 2024-09-17T15:59:43.386 INFO:teuthology.packaging:tag: None 2024-09-17T15:59:43.386 INFO:teuthology.packaging:branch: squid 2024-09-17T15:59:43.386 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:59:43.386 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-17T15:59:43.388 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T15:59:43.388 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T15:59:43.388 INFO:teuthology.packaging:ref: None 2024-09-17T15:59:43.388 INFO:teuthology.packaging:tag: None 2024-09-17T15:59:43.388 INFO:teuthology.packaging:branch: squid 2024-09-17T15:59:43.388 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:59:43.389 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-17T15:59:43.389 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T15:59:43.389 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T15:59:43.390 INFO:teuthology.packaging:ref: None 2024-09-17T15:59:43.390 INFO:teuthology.packaging:tag: None 2024-09-17T15:59:43.390 INFO:teuthology.packaging:branch: squid 2024-09-17T15:59:43.390 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:59:43.390 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-17T15:59:43.602 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/ 2024-09-17T15:59:43.603 INFO:teuthology.task.install.rpm:Package version is 19.2.0-0 2024-09-17T15:59:43.635 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T15:59:43.635 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T15:59:43.636 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T15:59:43.641 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/ 2024-09-17T15:59:43.641 INFO:teuthology.task.install.rpm:Package version is 19.2.0-0 2024-09-17T15:59:43.646 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/ 2024-09-17T15:59:43.647 INFO:teuthology.task.install.rpm:Package version is 19.2.0-0 2024-09-17T15:59:43.665 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T15:59:43.665 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T15:59:43.665 INFO:teuthology.packaging:ref: None 2024-09-17T15:59:43.665 INFO:teuthology.packaging:tag: None 2024-09-17T15:59:43.665 INFO:teuthology.packaging:branch: squid 2024-09-17T15:59:43.665 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:59:43.666 DEBUG:teuthology.orchestra.run.smithi017:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T15:59:43.694 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T15:59:43.695 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T15:59:43.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T15:59:43.700 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/squid/8e98666cb045318799c9328f1ae906cc2d724fd2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T15:59:43.700 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T15:59:43.700 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T15:59:43.722 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T15:59:43.722 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T15:59:43.722 INFO:teuthology.packaging:ref: None 2024-09-17T15:59:43.722 INFO:teuthology.packaging:tag: None 2024-09-17T15:59:43.722 INFO:teuthology.packaging:branch: squid 2024-09-17T15:59:43.722 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:59:43.723 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T15:59:43.725 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T15:59:43.725 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T15:59:43.725 INFO:teuthology.packaging:ref: None 2024-09-17T15:59:43.725 INFO:teuthology.packaging:tag: None 2024-09-17T15:59:43.726 INFO:teuthology.packaging:branch: squid 2024-09-17T15:59:43.726 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:59:43.726 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T15:59:43.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T15:59:43.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T15:59:43.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T15:59:43.810 DEBUG:teuthology.orchestra.run.smithi017:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T15:59:43.864 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T15:59:43.867 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T15:59:43.875 INFO:teuthology.orchestra.run.smithi017.stdout:check_obsoletes = 1 2024-09-17T15:59:43.877 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum clean all 2024-09-17T15:59:43.929 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-09-17T15:59:43.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-09-17T15:59:43.931 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2024-09-17T15:59:43.933 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2024-09-17T15:59:44.113 INFO:teuthology.orchestra.run.smithi017.stdout:45 files removed 2024-09-17T15:59:44.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-radosgw 2024-09-17T15:59:44.158 INFO:teuthology.orchestra.run.smithi071.stdout:45 files removed 2024-09-17T15:59:44.167 INFO:teuthology.orchestra.run.smithi173.stdout:45 files removed 2024-09-17T15:59:44.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2024-09-17T15:59:44.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-09-17T15:59:44.530 INFO:teuthology.orchestra.run.smithi017.stdout:ceph packages for x86_64 943 kB/s | 83 kB 00:00 2024-09-17T15:59:44.558 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 1.0 MB/s | 83 kB 00:00 2024-09-17T15:59:44.560 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 916 kB/s | 83 kB 00:00 2024-09-17T15:59:44.647 INFO:teuthology.orchestra.run.smithi017.stdout:ceph noarch packages 166 kB/s | 11 kB 00:00 2024-09-17T15:59:44.669 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 186 kB/s | 11 kB 00:00 2024-09-17T15:59:44.685 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 144 kB/s | 11 kB 00:00 2024-09-17T15:59:44.765 INFO:teuthology.orchestra.run.smithi017.stdout:ceph source packages 27 kB/s | 2.2 kB 00:00 2024-09-17T15:59:44.769 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 34 kB/s | 2.2 kB 00:00 2024-09-17T15:59:44.802 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 27 kB/s | 2.2 kB 00:00 2024-09-17T15:59:46.602 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.3 MB 00:01 2024-09-17T15:59:46.677 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.3 MB 00:01 2024-09-17T15:59:48.503 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-17T15:59:48.701 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.3 MB 00:03 2024-09-17T15:59:49.854 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 20 MB 00:02 2024-09-17T15:59:52.796 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-17T15:59:54.157 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 20 MB 00:04 2024-09-17T15:59:54.355 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 86 kB/s | 19 kB 00:00 2024-09-17T15:59:55.317 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-17T15:59:55.495 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-17T15:59:57.371 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 19 kB 00:00 2024-09-17T15:59:58.266 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-17T15:59:59.875 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-17T16:00:00.748 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-17T16:00:01.609 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-17T16:00:02.530 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-17T16:00:02.965 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:00:02.981 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:02.981 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.2.0-0.el9 ceph 11 M 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.2.0-0.el9 ceph 3.4 M 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.2.0-0.el9 ceph 3.2 M 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.2.0-0.el9 ceph 5.4 M 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.2.0-0.el9 ceph 21 M 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.2.0-0.el9 ceph 25 k 2024-09-17T16:00:02.982 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.2.0-0.el9 ceph 751 k 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.2.0-0.el9 ceph 506 k 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.2.0-0.el9 ceph 5.4 M 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T16:00:02.983 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.2.0-0.el9 ceph 45 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.2.0-0.el9 ceph 135 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.2.0-0.el9 ceph 161 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.2.0-0.el9 ceph 320 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.2.0-0.el9 ceph 301 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.2.0-0.el9 ceph 99 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-09-17T16:00:02.984 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T16:00:02.985 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:02.985 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:00:02.985 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:02.985 INFO:teuthology.orchestra.run.smithi173.stdout:Install 30 Packages 2024-09-17T16:00:02.985 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-09-17T16:00:02.985 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:02.985 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 60 M 2024-09-17T16:00:02.986 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:00:03.639 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-17T16:00:05.282 INFO:teuthology.orchestra.run.smithi173.stdout:(1/32): ceph-base-19.2.0-0.el9.x86_64.rpm 16 MB/s | 5.4 MB 00:00 2024-09-17T16:00:05.307 INFO:teuthology.orchestra.run.smithi173.stdout:(2/32): ceph-selinux-19.2.0-0.el9.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-09-17T16:00:05.341 INFO:teuthology.orchestra.run.smithi173.stdout:(3/32): libcephfs2-19.2.0-0.el9.x86_64.rpm 22 MB/s | 751 kB 00:00 2024-09-17T16:00:05.383 INFO:teuthology.orchestra.run.smithi173.stdout:(4/32): libradosstriper1-19.2.0-0.el9.x86_64.rp 12 MB/s | 506 kB 00:00 2024-09-17T16:00:05.550 INFO:teuthology.orchestra.run.smithi173.stdout:(5/32): ceph-radosgw-19.2.0-0.el9.x86_64.rpm 17 MB/s | 11 MB 00:00 2024-09-17T16:00:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:(6/32): python3-ceph-argparse-19.2.0-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-09-17T16:00:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:(7/32): librgw2-19.2.0-0.el9.x86_64.rpm 19 MB/s | 5.4 MB 00:00 2024-09-17T16:00:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:(8/32): python3-ceph-common-19.2.0-0.el9.x86_64 1.2 MB/s | 135 kB 00:00 2024-09-17T16:00:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:(9/32): python3-cephfs-19.2.0-0.el9.x86_64.rpm 3.2 MB/s | 161 kB 00:00 2024-09-17T16:00:05.734 INFO:teuthology.orchestra.run.smithi173.stdout:(10/32): python3-rados-19.2.0-0.el9.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2024-09-17T16:00:05.760 INFO:teuthology.orchestra.run.smithi173.stdout:(11/32): python3-rbd-19.2.0-0.el9.x86_64.rpm 5.9 MB/s | 301 kB 00:00 2024-09-17T16:00:05.785 INFO:teuthology.orchestra.run.smithi173.stdout:(12/32): python3-rgw-19.2.0-0.el9.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-09-17T16:00:05.865 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:19.2.0-0.el9 ceph 11 M 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:19.2.0-0.el9 ceph 3.4 M 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:19.2.0-0.el9 ceph 3.2 M 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T16:00:05.882 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:19.2.0-0.el9 ceph 5.4 M 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:19.2.0-0.el9 ceph 21 M 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:19.2.0-0.el9 ceph 25 k 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:19.2.0-0.el9 ceph 751 k 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:19.2.0-0.el9 ceph 506 k 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:19.2.0-0.el9 ceph 5.4 M 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T16:00:05.883 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:19.2.0-0.el9 ceph 45 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:19.2.0-0.el9 ceph 135 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:19.2.0-0.el9 ceph 161 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:19.2.0-0.el9 ceph 320 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:19.2.0-0.el9 ceph 301 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:19.2.0-0.el9 ceph 99 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T16:00:05.884 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-09-17T16:00:05.885 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T16:00:05.885 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:05.885 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:00:05.885 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:05.885 INFO:teuthology.orchestra.run.smithi071.stdout:Install 30 Packages 2024-09-17T16:00:05.885 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2024-09-17T16:00:05.885 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:05.885 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 60 M 2024-09-17T16:00:05.886 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:00:06.052 INFO:teuthology.orchestra.run.smithi173.stdout:(13/32): ceph-common-19.2.0-0.el9.x86_64.rpm 19 MB/s | 21 MB 00:01 2024-09-17T16:00:06.077 INFO:teuthology.orchestra.run.smithi173.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 337 kB/s | 107 kB 00:00 2024-09-17T16:00:06.136 INFO:teuthology.orchestra.run.smithi173.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 457 kB/s | 160 kB 00:00 2024-09-17T16:00:06.161 INFO:teuthology.orchestra.run.smithi173.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-09-17T16:00:06.245 INFO:teuthology.orchestra.run.smithi173.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 507 kB/s | 42 kB 00:00 2024-09-17T16:00:06.270 INFO:teuthology.orchestra.run.smithi173.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-17T16:00:06.328 INFO:teuthology.orchestra.run.smithi173.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.6 MB/s | 662 kB 00:00 2024-09-17T16:00:06.354 INFO:teuthology.orchestra.run.smithi173.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-17T16:00:06.387 INFO:teuthology.orchestra.run.smithi173.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-09-17T16:00:06.413 INFO:teuthology.orchestra.run.smithi173.stdout:(22/32): qatlib-service-24.02.0-1.el9.x86_64.rp 252 kB/s | 36 kB 00:00 2024-09-17T16:00:06.454 INFO:teuthology.orchestra.run.smithi173.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-17T16:00:06.480 INFO:teuthology.orchestra.run.smithi173.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 438 kB/s | 26 kB 00:00 2024-09-17T16:00:06.505 INFO:teuthology.orchestra.run.smithi173.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.0 MB/s | 49 kB 00:00 2024-09-17T16:00:06.530 INFO:teuthology.orchestra.run.smithi173.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T16:00:06.572 INFO:teuthology.orchestra.run.smithi173.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-17T16:00:06.623 INFO:teuthology.orchestra.run.smithi071.stdout:(1/32): ceph-base-19.2.0-0.el9.x86_64.rpm 40 MB/s | 5.4 MB 00:00 2024-09-17T16:00:06.639 INFO:teuthology.orchestra.run.smithi173.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-09-17T16:00:06.648 INFO:teuthology.orchestra.run.smithi071.stdout:(2/32): ceph-selinux-19.2.0-0.el9.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-09-17T16:00:06.690 INFO:teuthology.orchestra.run.smithi071.stdout:(3/32): libcephfs2-19.2.0-0.el9.x86_64.rpm 18 MB/s | 751 kB 00:00 2024-09-17T16:00:06.706 INFO:teuthology.orchestra.run.smithi173.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-17T16:00:06.732 INFO:teuthology.orchestra.run.smithi071.stdout:(4/32): libradosstriper1-19.2.0-0.el9.x86_64.rp 12 MB/s | 506 kB 00:00 2024-09-17T16:00:06.773 INFO:teuthology.orchestra.run.smithi173.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-17T16:00:06.840 INFO:teuthology.orchestra.run.smithi173.stdout:(31/32): librados2-19.2.0-0.el9.x86_64.rpm 17 MB/s | 3.4 MB 00:00 2024-09-17T16:00:06.882 INFO:teuthology.orchestra.run.smithi071.stdout:(5/32): ceph-radosgw-19.2.0-0.el9.x86_64.rpm 27 MB/s | 11 MB 00:00 2024-09-17T16:00:06.898 INFO:teuthology.orchestra.run.smithi173.stdout:(32/32): librbd1-19.2.0-0.el9.x86_64.rpm 16 MB/s | 3.2 MB 00:00 2024-09-17T16:00:06.904 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:06.904 INFO:teuthology.orchestra.run.smithi173.stdout:Total 15 MB/s | 60 MB 00:03 2024-09-17T16:00:06.909 INFO:teuthology.orchestra.run.smithi071.stdout:(6/32): python3-ceph-argparse-19.2.0-0.el9.x86_ 1.6 MB/s | 45 kB 00:00 2024-09-17T16:00:06.935 INFO:teuthology.orchestra.run.smithi071.stdout:(7/32): python3-ceph-common-19.2.0-0.el9.x86_64 5.3 MB/s | 135 kB 00:00 2024-09-17T16:00:06.960 INFO:teuthology.orchestra.run.smithi071.stdout:(8/32): python3-cephfs-19.2.0-0.el9.x86_64.rpm 6.4 MB/s | 161 kB 00:00 2024-09-17T16:00:06.985 INFO:teuthology.orchestra.run.smithi071.stdout:(9/32): python3-rados-19.2.0-0.el9.x86_64.rpm 13 MB/s | 320 kB 00:00 2024-09-17T16:00:07.007 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:00:07.043 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:00:07.043 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:00:07.060 INFO:teuthology.orchestra.run.smithi071.stdout:(10/32): librgw2-19.2.0-0.el9.x86_64.rpm 16 MB/s | 5.4 MB 00:00 2024-09-17T16:00:07.086 INFO:teuthology.orchestra.run.smithi071.stdout:(11/32): python3-rbd-19.2.0-0.el9.x86_64.rpm 2.9 MB/s | 301 kB 00:00 2024-09-17T16:00:07.111 INFO:teuthology.orchestra.run.smithi071.stdout:(12/32): python3-rgw-19.2.0-0.el9.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-09-17T16:00:07.345 INFO:teuthology.orchestra.run.smithi071.stdout:(13/32): ceph-common-19.2.0-0.el9.x86_64.rpm 25 MB/s | 21 MB 00:00 2024-09-17T16:00:07.395 INFO:teuthology.orchestra.run.smithi071.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 346 kB/s | 107 kB 00:00 2024-09-17T16:00:07.456 INFO:teuthology.orchestra.run.smithi071.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 468 kB/s | 160 kB 00:00 2024-09-17T16:00:07.479 INFO:teuthology.orchestra.run.smithi071.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 339 kB/s | 45 kB 00:00 2024-09-17T16:00:07.554 INFO:teuthology.orchestra.run.smithi071.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 562 kB/s | 42 kB 00:00 2024-09-17T16:00:07.588 INFO:teuthology.orchestra.run.smithi071.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-17T16:00:07.638 INFO:teuthology.orchestra.run.smithi071.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 718 kB/s | 36 kB 00:00 2024-09-17T16:00:07.663 INFO:teuthology.orchestra.run.smithi071.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-17T16:00:07.688 INFO:teuthology.orchestra.run.smithi071.stdout:(21/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-09-17T16:00:07.713 INFO:teuthology.orchestra.run.smithi071.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-17T16:00:07.719 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:00:07.719 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:00:07.781 INFO:teuthology.orchestra.run.smithi071.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-17T16:00:07.806 INFO:teuthology.orchestra.run.smithi071.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 340 kB/s | 26 kB 00:00 2024-09-17T16:00:07.831 INFO:teuthology.orchestra.run.smithi071.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-09-17T16:00:07.856 INFO:teuthology.orchestra.run.smithi071.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T16:00:07.898 INFO:teuthology.orchestra.run.smithi071.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-17T16:00:07.923 INFO:teuthology.orchestra.run.smithi071.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-17T16:00:08.007 INFO:teuthology.orchestra.run.smithi071.stdout:(29/32): librados2-19.2.0-0.el9.x86_64.rpm 41 MB/s | 3.4 MB 00:00 2024-09-17T16:00:08.049 INFO:teuthology.orchestra.run.smithi071.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-17T16:00:08.124 INFO:teuthology.orchestra.run.smithi071.stdout:(31/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-17T16:00:08.174 INFO:teuthology.orchestra.run.smithi071.stdout:(32/32): librbd1-19.2.0-0.el9.x86_64.rpm 19 MB/s | 3.2 MB 00:00 2024-09-17T16:00:08.178 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:08.178 INFO:teuthology.orchestra.run.smithi071.stdout:Total 26 MB/s | 60 MB 00:02 2024-09-17T16:00:08.219 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-17T16:00:08.279 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:00:08.316 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:00:08.316 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:00:08.999 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:00:08.999 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:00:09.184 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:00:09.269 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-17T16:00:09.303 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-17T16:00:09.328 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-17T16:00:09.372 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-17T16:00:09.397 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-17T16:00:09.621 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-17T16:00:09.674 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-17T16:00:10.031 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-17T16:00:10.155 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.2.0-0.el9.x86_64 7/34 2024-09-17T16:00:10.231 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.2.0-0.el9.x86_64 7/34 2024-09-17T16:00:10.320 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.2.0-0.el9.x86_64 8/34 2024-09-17T16:00:10.348 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.2.0-0.el9.x86_64 9/34 2024-09-17T16:00:10.412 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.2.0-0.el9.x86_64 9/34 2024-09-17T16:00:10.448 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-17T16:00:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.2.0-0.el9.x86_64 11/34 2024-09-17T16:00:10.545 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:00:10.562 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.2.0-0.el9.x86_64 12/34 2024-09-17T16:00:10.592 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.2.0-0.el9.x86_64 13/34 2024-09-17T16:00:10.629 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-17T16:00:10.656 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.2.0-0.el9.x86_64 13/34 2024-09-17T16:00:10.662 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-17T16:00:10.688 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-17T16:00:10.691 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-17T16:00:10.692 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:00:10.707 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:00:10.707 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-17T16:00:10.707 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:00:10.707 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:00:10.707 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw x86_64 2:19.2.0-0.el9 ceph 11 M 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrading: 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: librados2 x86_64 2:19.2.0-0.el9 ceph 3.4 M 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1 x86_64 2:19.2.0-0.el9 ceph 3.2 M 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base x86_64 2:19.2.0-0.el9 ceph 5.4 M 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common x86_64 2:19.2.0-0.el9 ceph 21 M 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux x86_64 2:19.2.0-0.el9 ceph 25 k 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T16:00:10.708 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2 x86_64 2:19.2.0-0.el9 ceph 751 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1 x86_64 2:19.2.0-0.el9 ceph 506 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2 x86_64 2:19.2.0-0.el9 ceph 5.4 M 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse x86_64 2:19.2.0-0.el9 ceph 45 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common x86_64 2:19.2.0-0.el9 ceph 135 k 2024-09-17T16:00:10.709 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs x86_64 2:19.2.0-0.el9 ceph 161 k 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados x86_64 2:19.2.0-0.el9 ceph 320 k 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd x86_64 2:19.2.0-0.el9 ceph 301 k 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw x86_64 2:19.2.0-0.el9 ceph 99 k 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout:Installing weak dependencies: 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:10.710 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:00:10.711 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:00:10.711 INFO:teuthology.orchestra.run.smithi017.stdout:Install 30 Packages 2024-09-17T16:00:10.711 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrade 2 Packages 2024-09-17T16:00:10.711 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:10.711 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-17T16:00:10.712 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 60 M 2024-09-17T16:00:10.712 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:00:10.746 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-17T16:00:10.797 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-17T16:00:10.834 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-17T16:00:10.875 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-17T16:00:10.913 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-17T16:00:10.919 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-17T16:00:10.952 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-17T16:00:10.974 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-17T16:00:10.990 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-17T16:00:11.298 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-17T16:00:11.340 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.2.0-0.el9.x86_64 22/34 2024-09-17T16:00:11.354 INFO:teuthology.orchestra.run.smithi017.stdout:(1/32): ceph-base-19.2.0-0.el9.x86_64.rpm 36 MB/s | 5.4 MB 00:00 2024-09-17T16:00:11.368 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-17T16:00:11.379 INFO:teuthology.orchestra.run.smithi017.stdout:(2/32): ceph-selinux-19.2.0-0.el9.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-09-17T16:00:11.413 INFO:teuthology.orchestra.run.smithi017.stdout:(3/32): libcephfs2-19.2.0-0.el9.x86_64.rpm 22 MB/s | 751 kB 00:00 2024-09-17T16:00:11.414 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:19.2.0-0.el9.x86_64 7/34 2024-09-17T16:00:11.430 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.2.0-0.el9.x86_64 22/34 2024-09-17T16:00:11.455 INFO:teuthology.orchestra.run.smithi017.stdout:(4/32): libradosstriper1-19.2.0-0.el9.x86_64.rp 12 MB/s | 506 kB 00:00 2024-09-17T16:00:11.469 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.2.0-0.el9.x86_64 23/34 2024-09-17T16:00:11.506 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-17T16:00:11.506 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:19.2.0-0.el9.x86_64 7/34 2024-09-17T16:00:11.580 INFO:teuthology.orchestra.run.smithi017.stdout:(5/32): librgw2-19.2.0-0.el9.x86_64.rpm 43 MB/s | 5.4 MB 00:00 2024-09-17T16:00:11.605 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:19.2.0-0.el9.x86_64 8/34 2024-09-17T16:00:11.605 INFO:teuthology.orchestra.run.smithi017.stdout:(6/32): python3-ceph-argparse-19.2.0-0.el9.x86_ 1.8 MB/s | 45 kB 00:00 2024-09-17T16:00:11.630 INFO:teuthology.orchestra.run.smithi017.stdout:(7/32): python3-ceph-common-19.2.0-0.el9.x86_64 5.3 MB/s | 135 kB 00:00 2024-09-17T16:00:11.640 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:19.2.0-0.el9.x86_64 9/34 2024-09-17T16:00:11.705 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:19.2.0-0.el9.x86_64 9/34 2024-09-17T16:00:11.732 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-17T16:00:11.748 INFO:teuthology.orchestra.run.smithi017.stdout:(8/32): ceph-radosgw-19.2.0-0.el9.x86_64.rpm 19 MB/s | 11 MB 00:00 2024-09-17T16:00:11.773 INFO:teuthology.orchestra.run.smithi017.stdout:(9/32): python3-cephfs-19.2.0-0.el9.x86_64.rpm 1.1 MB/s | 161 kB 00:00 2024-09-17T16:00:11.776 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:19.2.0-0.el9.x86_64 11/34 2024-09-17T16:00:11.807 INFO:teuthology.orchestra.run.smithi017.stdout:(10/32): python3-rados-19.2.0-0.el9.x86_64.rpm 5.3 MB/s | 320 kB 00:00 2024-09-17T16:00:11.832 INFO:teuthology.orchestra.run.smithi017.stdout:(11/32): python3-rgw-19.2.0-0.el9.x86_64.rpm 3.9 MB/s | 99 kB 00:00 2024-09-17T16:00:11.846 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:19.2.0-0.el9.x86_64 12/34 2024-09-17T16:00:11.857 INFO:teuthology.orchestra.run.smithi017.stdout:(12/32): python3-rbd-19.2.0-0.el9.x86_64.rpm 3.5 MB/s | 301 kB 00:00 2024-09-17T16:00:11.876 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:19.2.0-0.el9.x86_64 13/34 2024-09-17T16:00:11.940 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:19.2.0-0.el9.x86_64 13/34 2024-09-17T16:00:11.974 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-17T16:00:11.983 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-17T16:00:12.018 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.2.0-0.el9.x86_64 26/34 2024-09-17T16:00:12.089 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.2.0-0.el9.x86_64 26/34 2024-09-17T16:00:12.093 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-17T16:00:12.099 INFO:teuthology.orchestra.run.smithi017.stdout:(13/32): ceph-common-19.2.0-0.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-09-17T16:00:12.125 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-17T16:00:12.141 INFO:teuthology.orchestra.run.smithi017.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 346 kB/s | 107 kB 00:00 2024-09-17T16:00:12.166 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.2.0-0.el9.x86_64 27/34 2024-09-17T16:00:12.166 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-17T16:00:12.174 INFO:teuthology.orchestra.run.smithi017.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 505 kB/s | 160 kB 00:00 2024-09-17T16:00:12.195 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.2.0-0.el9.x86_64 28/34 2024-09-17T16:00:12.207 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-17T16:00:12.208 INFO:teuthology.orchestra.run.smithi017.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-09-17T16:00:12.236 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-17T16:00:12.242 INFO:teuthology.orchestra.run.smithi017.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 292 kB 00:00 2024-09-17T16:00:12.267 INFO:teuthology.orchestra.run.smithi017.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 717 kB/s | 42 kB 00:00 2024-09-17T16:00:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-17T16:00:12.301 INFO:teuthology.orchestra.run.smithi017.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-17T16:00:12.326 INFO:teuthology.orchestra.run.smithi017.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 612 kB/s | 36 kB 00:00 2024-09-17T16:00:12.352 INFO:teuthology.orchestra.run.smithi017.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-17T16:00:12.377 INFO:teuthology.orchestra.run.smithi017.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 626 kB/s | 47 kB 00:00 2024-09-17T16:00:12.444 INFO:teuthology.orchestra.run.smithi017.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-17T16:00:12.469 INFO:teuthology.orchestra.run.smithi017.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-17T16:00:12.495 INFO:teuthology.orchestra.run.smithi017.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-09-17T16:00:12.520 INFO:teuthology.orchestra.run.smithi017.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T16:00:12.553 INFO:teuthology.orchestra.run.smithi017.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T16:00:12.595 INFO:teuthology.orchestra.run.smithi017.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-17T16:00:12.597 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-17T16:00:12.632 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:19.2.0-0.el9.x86_64 22/34 2024-09-17T16:00:12.654 INFO:teuthology.orchestra.run.smithi017.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-17T16:00:12.721 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:19.2.0-0.el9.x86_64 22/34 2024-09-17T16:00:12.730 INFO:teuthology.orchestra.run.smithi017.stdout:(30/32): librados2-19.2.0-0.el9.x86_64.rpm 26 MB/s | 3.4 MB 00:00 2024-09-17T16:00:12.753 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:19.2.0-0.el9.x86_64 23/34 2024-09-17T16:00:12.788 INFO:teuthology.orchestra.run.smithi017.stdout:(31/32): librbd1-19.2.0-0.el9.x86_64.rpm 24 MB/s | 3.2 MB 00:00 2024-09-17T16:00:12.793 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-17T16:00:12.863 INFO:teuthology.orchestra.run.smithi017.stdout:(32/32): libarrow-9.0.0-11.el9.x86_64.rpm 8.6 MB/s | 4.4 MB 00:00 2024-09-17T16:00:12.868 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:12.868 INFO:teuthology.orchestra.run.smithi017.stdout:Total 28 MB/s | 60 MB 00:02 2024-09-17T16:00:12.974 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:00:13.015 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:00:13.015 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:00:13.275 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-17T16:00:13.318 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:19.2.0-0.el9.x86_64 26/34 2024-09-17T16:00:13.389 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:19.2.0-0.el9.x86_64 26/34 2024-09-17T16:00:13.466 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:19.2.0-0.el9.x86_64 27/34 2024-09-17T16:00:13.495 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:19.2.0-0.el9.x86_64 28/34 2024-09-17T16:00:13.712 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:00:13.713 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:00:14.493 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.2.0-0.el9.x86_64 29/34 2024-09-17T16:00:14.533 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.2.0-0.el9.x86_64 29/34 2024-09-17T16:00:15.112 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.2.0-0.el9.x86_64 29/34 2024-09-17T16:00:15.153 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.2.0-0.el9.x86_64 30/34 2024-09-17T16:00:15.247 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:00:15.252 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.2.0-0.el9.x86_64 30/34 2024-09-17T16:00:15.253 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T16:00:15.253 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T16:00:15.253 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:15.296 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.2.0-0.el9.x86_64 31/34 2024-09-17T16:00:15.339 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-17T16:00:15.377 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-17T16:00:15.402 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-17T16:00:15.424 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-17T16:00:15.462 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-17T16:00:15.628 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-17T16:00:15.692 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-17T16:00:15.758 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.2.0-0.el9.x86_64 29/34 2024-09-17T16:00:15.799 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:19.2.0-0.el9.x86_64 29/34 2024-09-17T16:00:16.106 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-17T16:00:16.170 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librados2-2:19.2.0-0.el9.x86_64 7/34 2024-09-17T16:00:16.264 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-2:19.2.0-0.el9.x86_64 7/34 2024-09-17T16:00:16.366 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rados-2:19.2.0-0.el9.x86_64 8/34 2024-09-17T16:00:16.379 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.2.0-0.el9.x86_64 29/34 2024-09-17T16:00:16.397 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs2-2:19.2.0-0.el9.x86_64 9/34 2024-09-17T16:00:16.419 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:19.2.0-0.el9.x86_64 30/34 2024-09-17T16:00:16.460 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs2-2:19.2.0-0.el9.x86_64 9/34 2024-09-17T16:00:16.497 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-17T16:00:16.524 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:19.2.0-0.el9.x86_64 30/34 2024-09-17T16:00:16.525 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T16:00:16.525 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T16:00:16.525 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:16.541 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-argparse-2:19.2.0-0.el9.x86_64 11/34 2024-09-17T16:00:16.570 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:19.2.0-0.el9.x86_64 31/34 2024-09-17T16:00:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cephfs-2:19.2.0-0.el9.x86_64 12/34 2024-09-17T16:00:16.649 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libradosstriper1-2:19.2.0-0.el9.x86_64 13/34 2024-09-17T16:00:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libradosstriper1-2:19.2.0-0.el9.x86_64 13/34 2024-09-17T16:00:16.749 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-17T16:00:16.859 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-17T16:00:16.891 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-17T16:00:16.932 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-17T16:00:16.972 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-17T16:00:17.002 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-17T16:00:17.051 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-17T16:00:17.384 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-17T16:00:17.422 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librbd1-2:19.2.0-0.el9.x86_64 22/34 2024-09-17T16:00:17.514 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-2:19.2.0-0.el9.x86_64 22/34 2024-09-17T16:00:17.547 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rbd-2:19.2.0-0.el9.x86_64 23/34 2024-09-17T16:00:17.588 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-17T16:00:18.089 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-17T16:00:18.117 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librgw2-2:19.2.0-0.el9.x86_64 26/34 2024-09-17T16:00:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librgw2-2:19.2.0-0.el9.x86_64 26/34 2024-09-17T16:00:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rgw-2:19.2.0-0.el9.x86_64 27/34 2024-09-17T16:00:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-common-2:19.2.0-0.el9.x86_64 28/34 2024-09-17T16:00:20.689 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:19.2.0-0.el9.x86_64 29/34 2024-09-17T16:00:20.724 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-common-2:19.2.0-0.el9.x86_64 29/34 2024-09-17T16:00:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:19.2.0-0.el9.x86_64 29/34 2024-09-17T16:00:21.361 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-base-2:19.2.0-0.el9.x86_64 30/34 2024-09-17T16:00:21.479 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-base-2:19.2.0-0.el9.x86_64 30/34 2024-09-17T16:00:21.479 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T16:00:21.479 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T16:00:21.479 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:21.537 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-selinux-2:19.2.0-0.el9.x86_64 31/34 2024-09-17T16:00:25.442 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.2.0-0.el9.x86_64 31/34 2024-09-17T16:00:25.442 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-09-17T16:00:25.442 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-09-17T16:00:25.442 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-09-17T16:00:25.443 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-09-17T16:00:25.443 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-09-17T16:00:25.443 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-09-17T16:00:25.443 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-09-17T16:00:25.443 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:26.429 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.2.0-0.el9.x86_64 32/34 2024-09-17T16:00:26.455 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.2.0-0.el9.x86_64 32/34 2024-09-17T16:00:26.455 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:26.455 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T16:00:26.455 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T16:00:26.455 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T16:00:26.455 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:26.457 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-17T16:00:26.466 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:19.2.0-0.el9.x86_64 31/34 2024-09-17T16:00:26.466 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2024-09-17T16:00:26.466 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2024-09-17T16:00:26.466 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2024-09-17T16:00:26.466 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2024-09-17T16:00:26.466 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2024-09-17T16:00:26.466 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2024-09-17T16:00:26.467 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2024-09-17T16:00:26.467 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:26.532 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-17T16:00:26.532 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T16:00:27.443 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:19.2.0-0.el9.x86_64 32/34 2024-09-17T16:00:27.469 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:19.2.0-0.el9.x86_64 32/34 2024-09-17T16:00:27.469 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:27.469 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T16:00:27.469 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T16:00:27.469 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T16:00:27.469 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:27.471 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-17T16:00:27.554 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-17T16:00:27.554 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T16:00:27.751 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T16:00:27.751 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.2.0-0.el9.x86_64 1/34 2024-09-17T16:00:27.751 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.2.0-0.el9.x86_64 2/34 2024-09-17T16:00:27.751 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.2.0-0.el9.x86_64 3/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.2.0-0.el9.x86_64 4/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.2.0-0.el9.x86_64 5/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.2.0-0.el9.x86_64 6/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.2.0-0.el9.x86_64 7/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.2.0-0.el9.x86_64 8/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.2.0-0.el9.x86_64 9/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.2.0-0.el9.x86_64 10/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.2.0-0.el9.x86_64 11/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.2.0-0.el9.x86_64 12/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.2.0-0.el9.x86_64 13/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-17T16:00:27.752 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-17T16:00:27.753 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-17T16:00:27.754 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-17T16:00:27.754 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-17T16:00:27.754 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.2.0-0.el9.x86_64 31/34 2024-09-17T16:00:27.755 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-17T16:00:27.755 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.2.0-0.el9.x86_64 33/34 2024-09-17T16:00:28.741 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T16:00:28.741 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:19.2.0-0.el9.x86_64 1/34 2024-09-17T16:00:28.741 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:19.2.0-0.el9.x86_64 2/34 2024-09-17T16:00:28.741 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:19.2.0-0.el9.x86_64 3/34 2024-09-17T16:00:28.741 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:19.2.0-0.el9.x86_64 4/34 2024-09-17T16:00:28.741 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:19.2.0-0.el9.x86_64 5/34 2024-09-17T16:00:28.741 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:19.2.0-0.el9.x86_64 6/34 2024-09-17T16:00:28.741 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:19.2.0-0.el9.x86_64 7/34 2024-09-17T16:00:28.741 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:19.2.0-0.el9.x86_64 8/34 2024-09-17T16:00:28.741 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:19.2.0-0.el9.x86_64 9/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:19.2.0-0.el9.x86_64 10/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:19.2.0-0.el9.x86_64 11/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:19.2.0-0.el9.x86_64 12/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:19.2.0-0.el9.x86_64 13/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-17T16:00:28.742 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-17T16:00:28.743 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-17T16:00:28.743 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-17T16:00:28.743 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-17T16:00:28.743 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-17T16:00:28.743 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-17T16:00:28.743 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-17T16:00:28.743 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-17T16:00:28.743 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-17T16:00:28.743 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:19.2.0-0.el9.x86_64 31/34 2024-09-17T16:00:28.744 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-17T16:00:28.744 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:19.2.0-0.el9.x86_64 33/34 2024-09-17T16:00:28.814 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T16:00:28.814 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:28.814 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-09-17T16:00:28.814 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.2.0-0.el9.x86_64 librbd1-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.814 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:00:28.814 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T16:00:28.814 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.814 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.814 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.815 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.815 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T16:00:28.815 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T16:00:28.815 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T16:00:28.815 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.815 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T16:00:28.815 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T16:00:28.815 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T16:00:28.815 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:28.816 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T16:00:28.817 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T16:00:28.817 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T16:00:28.817 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T16:00:28.817 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T16:00:28.817 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:28.817 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:00:29.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-09-17T16:00:29.703 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T16:00:29.703 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:29.703 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-09-17T16:00:29.703 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:19.2.0-0.el9.x86_64 librbd1-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.703 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:00:29.703 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T16:00:29.703 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T16:00:29.704 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T16:00:29.705 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T16:00:29.706 INFO:teuthology.orchestra.run.smithi071.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T16:00:29.706 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T16:00:29.706 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:29.706 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:00:29.737 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:29 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:00:29.869 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:00:29.870 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:29.870 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T16:00:29.870 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:29.870 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:00:29.870 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.2.0-0.el9 ceph 49 M 2024-09-17T16:00:29.870 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T16:00:29.871 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.2.0-0.el9 ceph 163 k 2024-09-17T16:00:29.871 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T16:00:29.871 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T16:00:29.871 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:29.871 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:00:29.871 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:29.871 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-09-17T16:00:29.871 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:29.871 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 49 M 2024-09-17T16:00:29.872 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 208 M 2024-09-17T16:00:29.872 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:00:30.092 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2024-09-17T16:00:30.212 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): libcephsqlite-19.2.0-0.el9.x86_64.rpm 2.6 MB/s | 163 kB 00:00 2024-09-17T16:00:30.683 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:27 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:00:30.821 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:00:30.822 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:30.822 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-17T16:00:30.822 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:30.822 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:00:30.822 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:19.2.0-0.el9 ceph 49 M 2024-09-17T16:00:30.822 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-17T16:00:30.823 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:19.2.0-0.el9 ceph 163 k 2024-09-17T16:00:30.823 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T16:00:30.823 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T16:00:30.823 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:30.823 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:00:30.823 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:30.823 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-09-17T16:00:30.823 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:30.824 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 49 M 2024-09-17T16:00:30.824 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 208 M 2024-09-17T16:00:30.824 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:00:30.829 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): ceph-test-19.2.0-0.el9.x86_64.rpm 72 MB/s | 49 MB 00:00 2024-09-17T16:00:31.008 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): libcephsqlite-19.2.0-0.el9.x86_64.rpm 2.9 MB/s | 163 kB 00:00 2024-09-17T16:00:31.279 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 60 kB/s | 64 kB 00:01 2024-09-17T16:00:31.455 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): socat-1.7.4.1-6.el9.x86_64.rpm 233 kB/s | 304 kB 00:01 2024-09-17T16:00:31.457 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:31.457 INFO:teuthology.orchestra.run.smithi173.stdout:Total 31 MB/s | 49 MB 00:01 2024-09-17T16:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:00:31.625 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): ceph-test-19.2.0-0.el9.x86_64.rpm 72 MB/s | 49 MB 00:00 2024-09-17T16:00:31.650 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 99 kB/s | 64 kB 00:00 2024-09-17T16:00:31.878 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:00:31.878 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:00:32.134 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): socat-1.7.4.1-6.el9.x86_64.rpm 258 kB/s | 304 kB 00:01 2024-09-17T16:00:32.136 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:32.136 INFO:teuthology.orchestra.run.smithi071.stdout:Total 37 MB/s | 49 MB 00:01 2024-09-17T16:00:32.164 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:00:32.185 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:00:32.185 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:00:32.317 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-selinux-2:19.2.0-0.el9.x86_64 31/34 2024-09-17T16:00:32.317 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /sys 2024-09-17T16:00:32.317 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /proc 2024-09-17T16:00:32.318 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /mnt 2024-09-17T16:00:32.318 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /var/tmp 2024-09-17T16:00:32.318 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /home 2024-09-17T16:00:32.318 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /root 2024-09-17T16:00:32.318 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /tmp 2024-09-17T16:00:32.318 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:32.429 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:00:32.509 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T16:00:32.554 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T16:00:32.555 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:00:32.556 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:00:32.581 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.2.0-0.el9.x86_64 3/4 2024-09-17T16:00:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:00:33.169 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T16:00:33.207 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T16:00:33.235 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:19.2.0-0.el9.x86_64 3/4 2024-09-17T16:00:33.315 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-radosgw-2:19.2.0-0.el9.x86_64 32/34 2024-09-17T16:00:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-radosgw-2:19.2.0-0.el9.x86_64 32/34 2024-09-17T16:00:33.342 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:33.342 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T16:00:33.342 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T16:00:33.342 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T16:00:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-17T16:00:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-17T16:00:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T16:00:34.647 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T16:00:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-base-2:19.2.0-0.el9.x86_64 1/34 2024-09-17T16:00:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-common-2:19.2.0-0.el9.x86_64 2/34 2024-09-17T16:00:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-radosgw-2:19.2.0-0.el9.x86_64 3/34 2024-09-17T16:00:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-selinux-2:19.2.0-0.el9.x86_64 4/34 2024-09-17T16:00:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs2-2:19.2.0-0.el9.x86_64 5/34 2024-09-17T16:00:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libradosstriper1-2:19.2.0-0.el9.x86_64 6/34 2024-09-17T16:00:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librgw2-2:19.2.0-0.el9.x86_64 7/34 2024-09-17T16:00:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-argparse-2:19.2.0-0.el9.x86_64 8/34 2024-09-17T16:00:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-common-2:19.2.0-0.el9.x86_64 9/34 2024-09-17T16:00:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cephfs-2:19.2.0-0.el9.x86_64 10/34 2024-09-17T16:00:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rados-2:19.2.0-0.el9.x86_64 11/34 2024-09-17T16:00:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rbd-2:19.2.0-0.el9.x86_64 12/34 2024-09-17T16:00:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rgw-2:19.2.0-0.el9.x86_64 13/34 2024-09-17T16:00:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-17T16:00:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-17T16:00:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-17T16:00:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-17T16:00:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-17T16:00:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-17T16:00:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-17T16:00:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-2:19.2.0-0.el9.x86_64 31/34 2024-09-17T16:00:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-17T16:00:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-2:19.2.0-0.el9.x86_64 33/34 2024-09-17T16:00:35.858 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout:Upgraded: 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout: librados2-2:19.2.0-0.el9.x86_64 librbd1-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T16:00:35.859 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:35.861 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:00:36.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-test 2024-09-17T16:00:36.821 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:28 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:00:36.970 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:00:36.971 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:00:36.971 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-17T16:00:36.971 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:00:36.971 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:00:36.971 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test x86_64 2:19.2.0-0.el9 ceph 49 M 2024-09-17T16:00:36.971 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-17T16:00:36.971 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite x86_64 2:19.2.0-0.el9 ceph 163 k 2024-09-17T16:00:36.971 INFO:teuthology.orchestra.run.smithi017.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T16:00:36.971 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T16:00:36.971 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:36.972 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:00:36.972 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:00:36.972 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-09-17T16:00:36.972 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:36.972 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 49 M 2024-09-17T16:00:36.972 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 208 M 2024-09-17T16:00:36.972 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:00:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.2.0-0.el9.x86_64 3/4 2024-09-17T16:00:37.174 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): libcephsqlite-19.2.0-0.el9.x86_64.rpm 2.5 MB/s | 163 kB 00:00 2024-09-17T16:00:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.2.0-0.el9.x86_64 4/4 2024-09-17T16:00:37.316 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 450 kB/s | 64 kB 00:00 2024-09-17T16:00:37.375 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-09-17T16:00:37.820 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:19.2.0-0.el9.x86_64 3/4 2024-09-17T16:00:37.833 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): ceph-test-19.2.0-0.el9.x86_64.rpm 67 MB/s | 49 MB 00:00 2024-09-17T16:00:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:Total 57 MB/s | 49 MB 00:00 2024-09-17T16:00:37.864 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:00:37.887 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:00:37.887 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:00:37.908 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:19.2.0-0.el9.x86_64 4/4 2024-09-17T16:00:38.277 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:00:38.278 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:00:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:00:39.047 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T16:00:39.091 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T16:00:39.127 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephsqlite-2:19.2.0-0.el9.x86_64 3/4 2024-09-17T16:00:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:19.2.0-0.el9.x86_64 4/4 2024-09-17T16:00:39.703 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.2.0-0.el9.x86_64 1/4 2024-09-17T16:00:39.704 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.2.0-0.el9.x86_64 2/4 2024-09-17T16:00:39.704 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T16:00:40.036 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T16:00:40.036 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:40.036 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:00:40.036 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.2.0-0.el9.x86_64 libcephsqlite-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:40.036 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-6.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T16:00:40.036 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:40.036 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:00:40.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-09-17T16:00:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:19.2.0-0.el9.x86_64 4/4 2024-09-17T16:00:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:19.2.0-0.el9.x86_64 1/4 2024-09-17T16:00:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:19.2.0-0.el9.x86_64 2/4 2024-09-17T16:00:40.398 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T16:00:40.730 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T16:00:40.731 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:40.731 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:00:40.731 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:19.2.0-0.el9.x86_64 libcephsqlite-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:40.731 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-6.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T16:00:40.731 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:40.731 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:00:40.955 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:40 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:00:41.041 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2024-09-17T16:00:41.094 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:00:41.096 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:41.096 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-09-17T16:00:41.096 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:41.096 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.2.0-0.el9 ceph 6.5 k 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.2.0-0.el9 ceph 2.4 M 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.2.0-0.el9 ceph 1.5 M 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.2.0-0.el9 ceph-noarch 249 k 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.2.0-0.el9 ceph 4.7 M 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.2.0-0.el9 ceph 17 M 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T16:00:41.097 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T16:00:41.098 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T16:00:41.099 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T16:00:41.100 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:41.100 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:00:41.100 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:41.100 INFO:teuthology.orchestra.run.smithi173.stdout:Install 37 Packages 2024-09-17T16:00:41.100 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:41.100 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 30 M 2024-09-17T16:00:41.101 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 108 M 2024-09-17T16:00:41.101 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:00:41.489 INFO:teuthology.orchestra.run.smithi173.stdout:(1/37): ceph-19.2.0-0.el9.x86_64.rpm 140 kB/s | 6.5 kB 00:00 2024-09-17T16:00:41.540 INFO:teuthology.orchestra.run.smithi173.stdout:(2/37): ceph-mgr-19.2.0-0.el9.x86_64.rpm 16 MB/s | 1.5 MB 00:00 2024-09-17T16:00:41.598 INFO:teuthology.orchestra.run.smithi173.stdout:(3/37): ceph-mds-19.2.0-0.el9.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-09-17T16:00:41.624 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:38 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:00:41.632 INFO:teuthology.orchestra.run.smithi173.stdout:(4/37): ceph-mgr-modules-core-19.2.0-0.el9.noar 7.3 MB/s | 249 kB 00:00 2024-09-17T16:00:41.716 INFO:teuthology.orchestra.run.smithi173.stdout:(5/37): ceph-mon-19.2.0-0.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-09-17T16:00:41.758 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:00:41.760 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:41.760 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-09-17T16:00:41.760 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:19.2.0-0.el9 ceph 6.5 k 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:19.2.0-0.el9 ceph 2.4 M 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:19.2.0-0.el9 ceph 1.5 M 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:19.2.0-0.el9 ceph-noarch 249 k 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:19.2.0-0.el9 ceph 4.7 M 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:19.2.0-0.el9 ceph 17 M 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T16:00:41.761 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T16:00:41.762 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T16:00:41.763 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T16:00:41.764 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T16:00:41.764 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T16:00:41.764 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T16:00:41.764 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T16:00:41.764 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:41.764 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:00:41.764 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:41.764 INFO:teuthology.orchestra.run.smithi071.stdout:Install 37 Packages 2024-09-17T16:00:41.764 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:41.765 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 30 M 2024-09-17T16:00:41.765 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 108 M 2024-09-17T16:00:41.765 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:00:41.932 INFO:teuthology.orchestra.run.smithi173.stdout:(6/37): ceph-osd-19.2.0-0.el9.x86_64.rpm 43 MB/s | 17 MB 00:00 2024-09-17T16:00:41.958 INFO:teuthology.orchestra.run.smithi173.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 778 kB/s | 253 kB 00:00 2024-09-17T16:00:42.000 INFO:teuthology.orchestra.run.smithi173.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-17T16:00:42.041 INFO:teuthology.orchestra.run.smithi173.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-17T16:00:42.068 INFO:teuthology.orchestra.run.smithi173.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 795 kB/s | 106 kB 00:00 2024-09-17T16:00:42.092 INFO:teuthology.orchestra.run.smithi173.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-17T16:00:42.159 INFO:teuthology.orchestra.run.smithi173.stdout:(12/37): python3-cryptography-36.0.1-4.el9.x86_ 2.8 MB/s | 1.2 MB 00:00 2024-09-17T16:00:42.451 INFO:teuthology.orchestra.run.smithi173.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 119 kB/s | 35 kB 00:00 2024-09-17T16:00:42.500 INFO:teuthology.orchestra.run.smithi071.stdout:(1/37): ceph-19.2.0-0.el9.x86_64.rpm 107 kB/s | 6.5 kB 00:00 2024-09-17T16:00:42.550 INFO:teuthology.orchestra.run.smithi071.stdout:(2/37): ceph-mgr-19.2.0-0.el9.x86_64.rpm 14 MB/s | 1.5 MB 00:00 2024-09-17T16:00:42.600 INFO:teuthology.orchestra.run.smithi071.stdout:(3/37): ceph-mds-19.2.0-0.el9.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-09-17T16:00:42.618 INFO:teuthology.orchestra.run.smithi173.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 342 kB/s | 188 kB 00:00 2024-09-17T16:00:42.625 INFO:teuthology.orchestra.run.smithi071.stdout:(4/37): ceph-mgr-modules-core-19.2.0-0.el9.noar 9.7 MB/s | 249 kB 00:00 2024-09-17T16:00:42.709 INFO:teuthology.orchestra.run.smithi071.stdout:(5/37): ceph-mon-19.2.0-0.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-09-17T16:00:42.727 INFO:teuthology.orchestra.run.smithi173.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 271 kB/s | 172 kB 00:00 2024-09-17T16:00:42.752 INFO:teuthology.orchestra.run.smithi173.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 257 kB/s | 77 kB 00:00 2024-09-17T16:00:42.772 INFO:teuthology.orchestra.run.smithi173.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 332 kB/s | 51 kB 00:00 2024-09-17T16:00:42.814 INFO:teuthology.orchestra.run.smithi173.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-09-17T16:00:42.839 INFO:teuthology.orchestra.run.smithi173.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 370 kB/s | 42 kB 00:00 2024-09-17T16:00:42.865 INFO:teuthology.orchestra.run.smithi173.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 862 kB/s | 43 kB 00:00 2024-09-17T16:00:42.899 INFO:teuthology.orchestra.run.smithi173.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 152 kB/s | 22 kB 00:00 2024-09-17T16:00:42.940 INFO:teuthology.orchestra.run.smithi173.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-17T16:00:42.959 INFO:teuthology.orchestra.run.smithi071.stdout:(6/37): ceph-osd-19.2.0-0.el9.x86_64.rpm 41 MB/s | 17 MB 00:00 2024-09-17T16:00:42.966 INFO:teuthology.orchestra.run.smithi173.stdout:(23/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-09-17T16:00:42.991 INFO:teuthology.orchestra.run.smithi173.stdout:(24/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 118 kB/s | 11 kB 00:00 2024-09-17T16:00:43.016 INFO:teuthology.orchestra.run.smithi173.stdout:(25/37): python3-jaraco-classes-3.2.1-5.el9.noa 240 kB/s | 18 kB 00:00 2024-09-17T16:00:43.041 INFO:teuthology.orchestra.run.smithi173.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 937 kB/s | 23 kB 00:00 2024-09-17T16:00:43.068 INFO:teuthology.orchestra.run.smithi071.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 573 kB/s | 253 kB 00:00 2024-09-17T16:00:43.075 INFO:teuthology.orchestra.run.smithi173.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 333 kB/s | 19 kB 00:00 2024-09-17T16:00:43.100 INFO:teuthology.orchestra.run.smithi173.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 906 kB/s | 46 kB 00:00 2024-09-17T16:00:43.118 INFO:teuthology.orchestra.run.smithi071.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-17T16:00:43.125 INFO:teuthology.orchestra.run.smithi173.stdout:(29/37): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-09-17T16:00:43.143 INFO:teuthology.orchestra.run.smithi071.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 580 kB/s | 106 kB 00:00 2024-09-17T16:00:43.151 INFO:teuthology.orchestra.run.smithi173.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-17T16:00:43.176 INFO:teuthology.orchestra.run.smithi173.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-17T16:00:43.185 INFO:teuthology.orchestra.run.smithi071.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.6 MB/s | 1.2 MB 00:00 2024-09-17T16:00:43.201 INFO:teuthology.orchestra.run.smithi173.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-17T16:00:43.210 INFO:teuthology.orchestra.run.smithi071.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-09-17T16:00:43.226 INFO:teuthology.orchestra.run.smithi173.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-17T16:00:43.236 INFO:teuthology.orchestra.run.smithi071.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-09-17T16:00:43.252 INFO:teuthology.orchestra.run.smithi173.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-17T16:00:43.294 INFO:teuthology.orchestra.run.smithi173.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-17T16:00:43.319 INFO:teuthology.orchestra.run.smithi173.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 300 kB/s | 20 kB 00:00 2024-09-17T16:00:43.345 INFO:teuthology.orchestra.run.smithi173.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.5 MB/s | 427 kB 00:00 2024-09-17T16:00:43.348 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:43.348 INFO:teuthology.orchestra.run.smithi173.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-17T16:00:43.504 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:00:43.519 INFO:teuthology.orchestra.run.smithi071.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 123 kB/s | 35 kB 00:00 2024-09-17T16:00:43.544 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:00:43.544 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:00:43.753 INFO:teuthology.orchestra.run.smithi071.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 317 kB/s | 172 kB 00:00 2024-09-17T16:00:43.778 INFO:teuthology.orchestra.run.smithi071.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 318 kB/s | 188 kB 00:00 2024-09-17T16:00:43.803 INFO:teuthology.orchestra.run.smithi071.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 272 kB/s | 77 kB 00:00 2024-09-17T16:00:43.862 INFO:teuthology.orchestra.run.smithi071.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 469 kB/s | 51 kB 00:00 2024-09-17T16:00:43.887 INFO:teuthology.orchestra.run.smithi071.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 266 kB/s | 22 kB 00:00 2024-09-17T16:00:43.913 INFO:teuthology.orchestra.run.smithi071.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 310 kB/s | 42 kB 00:00 2024-09-17T16:00:43.963 INFO:teuthology.orchestra.run.smithi071.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 578 kB/s | 43 kB 00:00 2024-09-17T16:00:43.988 INFO:teuthology.orchestra.run.smithi071.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-09-17T16:00:43.995 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephsqlite-2:19.2.0-0.el9.x86_64 3/4 2024-09-17T16:00:44.013 INFO:teuthology.orchestra.run.smithi071.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 430 kB/s | 11 kB 00:00 2024-09-17T16:00:44.024 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:00:44.025 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:00:44.038 INFO:teuthology.orchestra.run.smithi071.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 710 kB/s | 18 kB 00:00 2024-09-17T16:00:44.064 INFO:teuthology.orchestra.run.smithi071.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-17T16:00:44.078 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-test-2:19.2.0-0.el9.x86_64 4/4 2024-09-17T16:00:44.089 INFO:teuthology.orchestra.run.smithi071.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-17T16:00:44.114 INFO:teuthology.orchestra.run.smithi071.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-09-17T16:00:44.140 INFO:teuthology.orchestra.run.smithi071.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-17T16:00:44.165 INFO:teuthology.orchestra.run.smithi071.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-17T16:00:44.190 INFO:teuthology.orchestra.run.smithi071.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-17T16:00:44.207 INFO:teuthology.orchestra.run.smithi071.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-17T16:00:44.232 INFO:teuthology.orchestra.run.smithi071.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-09-17T16:00:44.258 INFO:teuthology.orchestra.run.smithi071.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-17T16:00:44.283 INFO:teuthology.orchestra.run.smithi071.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-17T16:00:44.308 INFO:teuthology.orchestra.run.smithi071.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-17T16:00:44.333 INFO:teuthology.orchestra.run.smithi071.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 806 kB/s | 20 kB 00:00 2024-09-17T16:00:44.359 INFO:teuthology.orchestra.run.smithi071.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-17T16:00:44.392 INFO:teuthology.orchestra.run.smithi071.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-17T16:00:44.396 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:44.396 INFO:teuthology.orchestra.run.smithi071.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-17T16:00:44.549 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:00:44.590 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:00:44.590 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:00:45.060 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:00:45.060 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:00:45.409 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:00:45.472 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T16:00:45.512 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T16:00:45.541 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T16:00:45.613 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T16:00:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T16:00:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T16:00:45.791 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T16:00:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T16:00:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T16:00:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T16:00:45.974 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T16:00:46.008 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T16:00:46.041 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T16:00:46.082 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T16:00:46.117 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T16:00:46.151 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T16:00:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T16:00:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T16:00:46.323 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T16:00:46.417 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:00:46.470 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T16:00:46.490 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T16:00:46.537 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T16:00:46.557 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T16:00:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-test-2:19.2.0-0.el9.x86_64 4/4 2024-09-17T16:00:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-test-2:19.2.0-0.el9.x86_64 1/4 2024-09-17T16:00:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephsqlite-2:19.2.0-0.el9.x86_64 2/4 2024-09-17T16:00:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T16:00:46.567 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T16:00:46.610 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T16:00:46.639 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T16:00:46.648 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T16:00:46.683 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T16:00:46.697 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T16:00:46.716 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T16:00:46.757 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T16:00:46.808 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T16:00:46.886 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T16:00:46.926 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T16:00:46.926 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:46.926 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:00:46.926 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test-2:19.2.0-0.el9.x86_64 libcephsqlite-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:46.926 INFO:teuthology.orchestra.run.smithi017.stdout: socat-1.7.4.1-6.el9.x86_64 xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T16:00:46.927 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:46.927 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:00:46.935 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T16:00:46.949 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T16:00:46.969 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T16:00:47.000 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T16:00:47.002 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T16:00:47.034 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T16:00:47.067 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T16:00:47.081 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T16:00:47.108 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T16:00:47.151 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T16:00:47.170 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T16:00:47.185 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T16:00:47.197 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph 2024-09-17T16:00:47.210 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T16:00:47.226 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T16:00:47.258 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T16:00:47.327 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T16:00:47.331 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T16:00:47.471 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T16:00:47.499 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.2.0-0.el9.noarch 32/37 2024-09-17T16:00:47.529 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.2.0-0.el9.x86_64 33/37 2024-09-17T16:00:47.550 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T16:00:47.555 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.2.0-0.el9.x86_64 33/37 2024-09-17T16:00:47.555 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:47.556 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T16:00:47.556 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T16:00:47.556 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T16:00:47.556 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:47.602 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T16:00:47.641 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T16:00:47.698 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T16:00:47.753 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T16:00:47.810 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:39 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:00:47.949 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T16:00:47.954 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:00:47.957 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:00:47.957 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout: ceph x86_64 2:19.2.0-0.el9 ceph 6.5 k 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds x86_64 2:19.2.0-0.el9 ceph 2.4 M 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr x86_64 2:19.2.0-0.el9 ceph 1.5 M 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core noarch 2:19.2.0-0.el9 ceph-noarch 249 k 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon x86_64 2:19.2.0-0.el9 ceph 4.7 M 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd x86_64 2:19.2.0-0.el9 ceph 17 M 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T16:00:47.958 INFO:teuthology.orchestra.run.smithi017.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T16:00:47.959 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T16:00:47.960 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T16:00:47.961 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T16:00:47.961 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T16:00:47.961 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:47.961 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:00:47.961 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:00:47.961 INFO:teuthology.orchestra.run.smithi017.stdout:Install 37 Packages 2024-09-17T16:00:47.961 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:47.962 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 30 M 2024-09-17T16:00:47.962 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 108 M 2024-09-17T16:00:47.962 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:00:48.010 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T16:00:48.098 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T16:00:48.187 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T16:00:48.235 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T16:00:48.352 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T16:00:48.508 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:19.2.0-0.el9.noarch 32/37 2024-09-17T16:00:48.536 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:19.2.0-0.el9.x86_64 33/37 2024-09-17T16:00:48.562 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:19.2.0-0.el9.x86_64 33/37 2024-09-17T16:00:48.562 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:48.562 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T16:00:48.562 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T16:00:48.563 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T16:00:48.563 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:48.839 INFO:teuthology.orchestra.run.smithi017.stdout:(1/37): ceph-19.2.0-0.el9.x86_64.rpm 92 kB/s | 6.5 kB 00:00 2024-09-17T16:00:48.889 INFO:teuthology.orchestra.run.smithi017.stdout:(2/37): ceph-mgr-19.2.0-0.el9.x86_64.rpm 13 MB/s | 1.5 MB 00:00 2024-09-17T16:00:48.948 INFO:teuthology.orchestra.run.smithi017.stdout:(3/37): ceph-mds-19.2.0-0.el9.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-09-17T16:00:48.973 INFO:teuthology.orchestra.run.smithi017.stdout:(4/37): ceph-mgr-modules-core-19.2.0-0.el9.noar 9.8 MB/s | 249 kB 00:00 2024-09-17T16:00:48.996 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.2.0-0.el9.x86_64 34/37 2024-09-17T16:00:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.2.0-0.el9.x86_64 34/37 2024-09-17T16:00:49.024 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:49.024 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T16:00:49.024 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T16:00:49.024 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T16:00:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:49.057 INFO:teuthology.orchestra.run.smithi017.stdout:(5/37): ceph-mon-19.2.0-0.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-09-17T16:00:49.274 INFO:teuthology.orchestra.run.smithi017.stdout:(6/37): ceph-osd-19.2.0-0.el9.x86_64.rpm 44 MB/s | 17 MB 00:00 2024-09-17T16:00:49.299 INFO:teuthology.orchestra.run.smithi017.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 778 kB/s | 253 kB 00:00 2024-09-17T16:00:49.366 INFO:teuthology.orchestra.run.smithi017.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-17T16:00:49.416 INFO:teuthology.orchestra.run.smithi017.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 3.4 MB/s | 1.2 MB 00:00 2024-09-17T16:00:49.441 INFO:teuthology.orchestra.run.smithi017.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 636 kB/s | 106 kB 00:00 2024-09-17T16:00:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.2.0-0.el9.x86_64 35/37 2024-09-17T16:00:49.467 INFO:teuthology.orchestra.run.smithi017.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 1.2 MB/s | 125 kB 00:00 2024-09-17T16:00:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.2.0-0.el9.x86_64 35/37 2024-09-17T16:00:49.479 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:49.479 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T16:00:49.479 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T16:00:49.479 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T16:00:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:49.501 INFO:teuthology.orchestra.run.smithi017.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-17T16:00:49.643 INFO:teuthology.orchestra.run.smithi017.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 245 kB/s | 35 kB 00:00 2024-09-17T16:00:49.701 INFO:teuthology.orchestra.run.smithi017.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 737 kB/s | 172 kB 00:00 2024-09-17T16:00:49.705 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.2.0-0.el9.x86_64 36/37 2024-09-17T16:00:49.726 INFO:teuthology.orchestra.run.smithi017.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 662 kB/s | 188 kB 00:00 2024-09-17T16:00:49.729 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.2.0-0.el9.x86_64 36/37 2024-09-17T16:00:49.729 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:49.729 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T16:00:49.729 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T16:00:49.729 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T16:00:49.729 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:49.769 INFO:teuthology.orchestra.run.smithi017.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 614 kB/s | 77 kB 00:00 2024-09-17T16:00:49.785 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.2.0-0.el9.x86_64 37/37 2024-09-17T16:00:49.785 INFO:teuthology.orchestra.run.smithi017.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 710 kB/s | 42 kB 00:00 2024-09-17T16:00:49.802 INFO:teuthology.orchestra.run.smithi017.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 506 kB/s | 51 kB 00:00 2024-09-17T16:00:49.845 INFO:teuthology.orchestra.run.smithi017.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-17T16:00:49.878 INFO:teuthology.orchestra.run.smithi017.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.4 MB/s | 172 kB 00:00 2024-09-17T16:00:49.895 INFO:teuthology.orchestra.run.smithi017.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-17T16:00:49.912 INFO:teuthology.orchestra.run.smithi017.stdout:(22/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 399 kB/s | 43 kB 00:00 2024-09-17T16:00:49.937 INFO:teuthology.orchestra.run.smithi017.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 430 kB/s | 11 kB 00:00 2024-09-17T16:00:49.954 INFO:teuthology.orchestra.run.smithi017.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-09-17T16:00:49.971 INFO:teuthology.orchestra.run.smithi017.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 741 kB/s | 23 kB 00:00 2024-09-17T16:00:49.988 INFO:teuthology.orchestra.run.smithi017.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 1.2 MB/s | 20 kB 00:00 2024-09-17T16:00:50.003 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:19.2.0-0.el9.x86_64 34/37 2024-09-17T16:00:50.005 INFO:teuthology.orchestra.run.smithi017.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 581 kB/s | 19 kB 00:00 2024-09-17T16:00:50.031 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:19.2.0-0.el9.x86_64 34/37 2024-09-17T16:00:50.031 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:50.031 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T16:00:50.031 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T16:00:50.031 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T16:00:50.032 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:50.038 INFO:teuthology.orchestra.run.smithi017.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-09-17T16:00:50.055 INFO:teuthology.orchestra.run.smithi017.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 689 kB/s | 46 kB 00:00 2024-09-17T16:00:50.081 INFO:teuthology.orchestra.run.smithi017.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-17T16:00:50.097 INFO:teuthology.orchestra.run.smithi017.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-09-17T16:00:50.114 INFO:teuthology.orchestra.run.smithi017.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-17T16:00:50.140 INFO:teuthology.orchestra.run.smithi017.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-17T16:00:50.165 INFO:teuthology.orchestra.run.smithi017.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-17T16:00:50.190 INFO:teuthology.orchestra.run.smithi017.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-09-17T16:00:50.240 INFO:teuthology.orchestra.run.smithi017.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-09-17T16:00:50.266 INFO:teuthology.orchestra.run.smithi017.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-09-17T16:00:50.269 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:50.270 INFO:teuthology.orchestra.run.smithi017.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-17T16:00:50.424 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:00:50.469 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:00:50.469 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:00:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:19.2.0-0.el9.x86_64 35/37 2024-09-17T16:00:50.495 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:19.2.0-0.el9.x86_64 35/37 2024-09-17T16:00:50.495 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:50.495 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T16:00:50.495 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T16:00:50.495 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T16:00:50.495 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:50.722 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:19.2.0-0.el9.x86_64 36/37 2024-09-17T16:00:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:19.2.0-0.el9.x86_64 36/37 2024-09-17T16:00:50.746 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:50.746 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T16:00:50.746 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T16:00:50.746 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T16:00:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:19.2.0-0.el9.x86_64 37/37 2024-09-17T16:00:50.978 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:00:50.978 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:00:51.632 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:19.2.0-0.el9.x86_64 37/37 2024-09-17T16:00:51.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.2.0-0.el9.x86_64 1/37 2024-09-17T16:00:51.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.2.0-0.el9.x86_64 2/37 2024-09-17T16:00:51.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.2.0-0.el9.x86_64 3/37 2024-09-17T16:00:51.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.2.0-0.el9.x86_64 4/37 2024-09-17T16:00:51.632 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.2.0-0.el9.x86_64 5/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.2.0-0.el9.noarch 6/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T16:00:51.633 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T16:00:51.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T16:00:51.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T16:00:51.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T16:00:51.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T16:00:51.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T16:00:51.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T16:00:51.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T16:00:51.634 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T16:00:51.635 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T16:00:51.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T16:00:51.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T16:00:51.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T16:00:51.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T16:00:51.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T16:00:51.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T16:00:51.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T16:00:51.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T16:00:51.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T16:00:51.637 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T16:00:52.516 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:00:52.594 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T16:00:52.644 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T16:00:52.671 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T16:00:52.678 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:19.2.0-0.el9.x86_64 37/37 2024-09-17T16:00:52.678 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:19.2.0-0.el9.x86_64 1/37 2024-09-17T16:00:52.678 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:19.2.0-0.el9.x86_64 2/37 2024-09-17T16:00:52.678 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:19.2.0-0.el9.x86_64 3/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:19.2.0-0.el9.x86_64 4/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:19.2.0-0.el9.x86_64 5/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:19.2.0-0.el9.noarch 6/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T16:00:52.679 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T16:00:52.680 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T16:00:52.681 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T16:00:52.681 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T16:00:52.682 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T16:00:52.682 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T16:00:52.682 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T16:00:52.682 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T16:00:52.682 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T16:00:52.682 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T16:00:52.682 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T16:00:52.683 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T16:00:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T16:00:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:52.719 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.2.0-0.el9.noarch 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T16:00:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T16:00:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T16:00:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T16:00:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T16:00:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T16:00:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T16:00:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T16:00:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T16:00:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T16:00:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T16:00:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:52.722 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:00:52.747 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T16:00:52.804 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T16:00:52.837 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T16:00:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T16:00:53.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-09-17T16:00:53.136 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T16:00:53.190 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T16:00:53.235 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T16:00:53.273 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T16:00:53.308 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T16:00:53.341 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T16:00:53.391 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T16:00:53.434 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T16:00:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T16:00:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T16:00:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T16:00:53.634 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T16:00:53.679 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:00:53.725 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T16:00:53.725 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:53.725 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:19.2.0-0.el9.noarch 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T16:00:53.726 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T16:00:53.727 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T16:00:53.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T16:00:53.729 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:53.729 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:00:53.767 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:00:53.788 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T16:00:53.810 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:00:53.811 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T16:00:53.811 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:00:53.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-09-17T16:00:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T16:00:53.946 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T16:00:53.983 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T16:00:54.041 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T16:00:54.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2024-09-17T16:00:54.105 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T16:00:54.305 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T16:00:54.362 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T16:00:54.443 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T16:00:54.446 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:54 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:00:54.541 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T16:00:54.577 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:00:54.578 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T16:00:54.578 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:54.578 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T16:00:54.578 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:54.578 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:00:54.578 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.2.0-0.el9 ceph-noarch 765 k 2024-09-17T16:00:54.578 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:54.578 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:00:54.579 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:54.579 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T16:00:54.579 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:54.579 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 765 k 2024-09-17T16:00:54.579 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 771 k 2024-09-17T16:00:54.579 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:00:54.639 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-19.2.0-0.el9.noarch.rpm 13 MB/s | 765 kB 00:00 2024-09-17T16:00:54.639 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:54.639 INFO:teuthology.orchestra.run.smithi173.stdout:Total 12 MB/s | 765 kB 00:00 2024-09-17T16:00:54.640 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:00:54.645 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:00:54.645 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:00:54.651 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:00:54.651 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:00:54.680 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:51 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:00:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T16:00:54.753 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:00:54.768 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:00:54.809 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:00:54.810 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-17T16:00:54.811 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:00:54.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2024-09-17T16:00:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-modules-core-2:19.2.0-0.el9.noarch 32/37 2024-09-17T16:00:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-2:19.2.0-0.el9.x86_64 33/37 2024-09-17T16:00:54.941 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-2:19.2.0-0.el9.x86_64 33/37 2024-09-17T16:00:54.941 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:54.941 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T16:00:54.941 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T16:00:54.941 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T16:00:54.941 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:55.242 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:00:55.334 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:00:55.443 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:52 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:00:55.572 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:00:55.573 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:55.573 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-17T16:00:55.573 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:55.573 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:00:55.573 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:19.2.0-0.el9 ceph-noarch 765 k 2024-09-17T16:00:55.573 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:55.573 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:00:55.574 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:55.574 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-17T16:00:55.574 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:55.574 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 765 k 2024-09-17T16:00:55.574 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 771 k 2024-09-17T16:00:55.574 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:00:55.637 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-19.2.0-0.el9.noarch.rpm 12 MB/s | 765 kB 00:00 2024-09-17T16:00:55.637 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:55.637 INFO:teuthology.orchestra.run.smithi071.stdout:Total 12 MB/s | 765 kB 00:00 2024-09-17T16:00:55.637 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:00:55.642 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:00:55.643 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:00:55.648 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:00:55.649 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:00:55.713 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:00:55.718 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:00:55.967 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:00:55.968 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:55.968 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:00:55.968 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.2.0-0.el9.noarch 2024-09-17T16:00:55.968 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:55.968 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:00:56.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T16:00:56.180 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:00:56.265 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:00:56.473 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-osd-2:19.2.0-0.el9.x86_64 34/37 2024-09-17T16:00:56.502 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-osd-2:19.2.0-0.el9.x86_64 34/37 2024-09-17T16:00:56.502 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:56.502 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T16:00:56.502 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T16:00:56.502 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T16:00:56.502 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:56.633 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:00:56.764 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:00:56.894 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:00:56.894 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:56.894 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:00:56.895 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:19.2.0-0.el9.noarch 2024-09-17T16:00:56.895 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:56.895 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:00:56.900 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:00:56.900 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:56.900 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-09-17T16:00:56.900 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:56.900 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:00:56.900 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.2.0-0.el9 ceph 145 k 2024-09-17T16:00:56.901 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:56.901 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:00:56.901 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:56.901 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T16:00:56.901 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:56.901 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 145 k 2024-09-17T16:00:56.901 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 435 k 2024-09-17T16:00:56.902 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:00:56.945 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-19.2.0-0.el9.x86_64 3.3 MB/s | 145 kB 00:00 2024-09-17T16:00:56.945 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:56.946 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.2 MB/s | 145 kB 00:00 2024-09-17T16:00:56.946 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:00:56.953 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:00:56.953 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:00:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mon-2:19.2.0-0.el9.x86_64 35/37 2024-09-17T16:00:56.991 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mon-2:19.2.0-0.el9.x86_64 35/37 2024-09-17T16:00:56.991 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:56.991 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T16:00:56.991 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T16:00:56.991 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T16:00:56.991 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:57.011 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:00:57.012 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:00:57.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T16:00:57.181 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:00:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mds-2:19.2.0-0.el9.x86_64 36/37 2024-09-17T16:00:57.258 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mds-2:19.2.0-0.el9.x86_64 36/37 2024-09-17T16:00:57.258 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:57.259 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T16:00:57.259 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T16:00:57.259 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T16:00:57.259 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:00:57.262 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:00:57.287 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:00:57.287 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:57.287 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T16:00:57.287 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:57.324 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-2:19.2.0-0.el9.x86_64 37/37 2024-09-17T16:00:57.683 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:54 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:00:57.818 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:00:57.818 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:57.818 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-09-17T16:00:57.819 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:57.819 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:00:57.819 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:19.2.0-0.el9 ceph 145 k 2024-09-17T16:00:57.819 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:57.819 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:00:57.819 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:00:57.819 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-17T16:00:57.819 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:57.820 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 145 k 2024-09-17T16:00:57.820 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 435 k 2024-09-17T16:00:57.820 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:00:57.858 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-19.2.0-0.el9.x86_64 3.7 MB/s | 145 kB 00:00 2024-09-17T16:00:57.859 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:00:57.859 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.5 MB/s | 145 kB 00:00 2024-09-17T16:00:57.859 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:00:57.866 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:00:57.866 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:00:57.923 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:00:57.923 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:00:58.071 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:00:58.118 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:00:58.142 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:00:58.142 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:00:58.143 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T16:00:58.143 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:58.233 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:00:58.233 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:58.233 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:00:58.233 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:58.233 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:58.233 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:00:58.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-09-17T16:00:59.076 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:00:59.078 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:00:59.078 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:59.078 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:00:59.078 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 2024-09-17T16:00:59.079 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:00:59.079 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:00:59.168 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:00:59.210 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:00:59.211 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T16:00:59.212 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:00:59.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-09-17T16:00:59.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2024-09-17T16:00:59.340 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-2:19.2.0-0.el9.x86_64 37/37 2024-09-17T16:00:59.340 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-2:19.2.0-0.el9.x86_64 1/37 2024-09-17T16:00:59.340 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mds-2:19.2.0-0.el9.x86_64 2/37 2024-09-17T16:00:59.340 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-2:19.2.0-0.el9.x86_64 3/37 2024-09-17T16:00:59.340 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mon-2:19.2.0-0.el9.x86_64 4/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-osd-2:19.2.0-0.el9.x86_64 5/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-modules-core-2:19.2.0-0.el9.noarch 6/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T16:00:59.341 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T16:00:59.342 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T16:00:59.343 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T16:00:59.343 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T16:00:59.343 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T16:00:59.344 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T16:00:59.344 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T16:00:59.344 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T16:00:59.344 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T16:00:59.344 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T16:00:59.850 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:00:59.917 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:00:59.982 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:00:59.982 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:59.982 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T16:00:59.982 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:59.983 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:00:59.983 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.2.0-0.el9 ceph 896 k 2024-09-17T16:00:59.983 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-09-17T16:00:59.983 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T16:00:59.983 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:59.983 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:00:59.983 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:00:59.983 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-09-17T16:00:59.983 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:00:59.984 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 976 k 2024-09-17T16:00:59.984 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 2.9 M 2024-09-17T16:00:59.984 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:01:00.006 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:00.049 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:00.049 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-17T16:01:00.050 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:00.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2024-09-17T16:01:00.157 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): ceph-fuse-19.2.0-0.el9.x86_64.rpm 12 MB/s | 896 kB 00:00 2024-09-17T16:01:00.190 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 739 kB/s | 80 kB 00:00 2024-09-17T16:01:00.191 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:00.191 INFO:teuthology.orchestra.run.smithi173.stdout:Total 4.6 MB/s | 976 kB 00:00 2024-09-17T16:01:00.203 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core-2:19.2.0-0.el9.noarch 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T16:01:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T16:01:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T16:01:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T16:01:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T16:01:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T16:01:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T16:01:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T16:01:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T16:01:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T16:01:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:00.208 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:00.225 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:01:00.225 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:01:00.308 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:01:00.308 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:01:00.490 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:01:00.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-base 2024-09-17T16:01:00.626 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T16:01:00.693 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:57 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:00.697 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.2.0-0.el9.x86_64 2/2 2024-09-17T16:01:00.824 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:00.824 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:00.824 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:00.824 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:00.824 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:01:00.825 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:19.2.0-0.el9 ceph 896 k 2024-09-17T16:01:00.825 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-09-17T16:01:00.825 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T16:01:00.825 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:00.825 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:01:00.825 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:00.825 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-09-17T16:01:00.825 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:00.826 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 976 k 2024-09-17T16:01:00.826 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.9 M 2024-09-17T16:01:00.826 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:01:00.988 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): ceph-fuse-19.2.0-0.el9.x86_64.rpm 14 MB/s | 896 kB 00:00 2024-09-17T16:01:01.096 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 462 kB/s | 80 kB 00:00 2024-09-17T16:01:01.097 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:01.097 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.5 MB/s | 976 kB 00:00 2024-09-17T16:01:01.108 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:01:01.130 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:01:01.130 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:01:01.179 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:01.203 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:01:01.203 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:01:01.270 INFO:teuthology.orchestra.run.smithi017.stdout:Package ceph-base-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:01.316 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:01.317 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-17T16:01:01.317 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:01.387 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install cephadm 2024-09-17T16:01:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:01:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:19.2.0-0.el9.x86_64 2/2 2024-09-17T16:01:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.2.0-0.el9.x86_64 1/2 2024-09-17T16:01:01.533 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T16:01:01.603 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:19.2.0-0.el9.x86_64 2/2 2024-09-17T16:01:01.720 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T16:01:01.720 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:01.720 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:01:01.720 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.2.0-0.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T16:01:01.720 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:01.721 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:01.990 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:01.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-09-17T16:01:02.129 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:02.129 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:02.129 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:02.129 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:02.130 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:01:02.130 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm noarch 2:19.2.0-0.el9 ceph-noarch 765 k 2024-09-17T16:01:02.130 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:02.130 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:01:02.130 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:02.130 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-17T16:01:02.130 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:02.130 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 765 k 2024-09-17T16:01:02.130 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 771 k 2024-09-17T16:01:02.130 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:01:02.191 INFO:teuthology.orchestra.run.smithi017.stdout:cephadm-19.2.0-0.el9.noarch.rpm 12 MB/s | 765 kB 00:00 2024-09-17T16:01:02.192 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:02.192 INFO:teuthology.orchestra.run.smithi017.stdout:Total 12 MB/s | 765 kB 00:00 2024-09-17T16:01:02.192 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:01:02.198 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:01:02.198 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:01:02.204 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:01:02.204 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:01:02.279 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:01:02.306 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:19.2.0-0.el9.x86_64 2/2 2024-09-17T16:01:02.306 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:19.2.0-0.el9.x86_64 1/2 2024-09-17T16:01:02.588 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T16:01:02.588 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:02.588 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:01:02.588 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:19.2.0-0.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T16:01:02.588 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:02.589 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:02.589 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:02 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:02.722 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:02.723 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:02.723 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:02.723 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:02.723 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:01:02.723 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.2.0-0.el9 ceph 126 k 2024-09-17T16:01:02.723 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:02.723 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:01:02.723 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:02.723 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T16:01:02.723 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:02.724 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 126 k 2024-09-17T16:01:02.724 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 456 k 2024-09-17T16:01:02.724 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:01:02.744 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:01:02.770 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-19.2.0-0.el9.x86_64.rpm 2.7 MB/s | 126 kB 00:00 2024-09-17T16:01:02.771 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:02.771 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.6 MB/s | 126 kB 00:00 2024-09-17T16:01:02.771 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:01:02.776 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:01:02.776 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:01:02.834 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:01:02.835 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:01:02.839 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:01:02.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2024-09-17T16:01:02.980 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:01:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:01:03.426 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:00 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:03.482 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : cephadm-2:19.2.0-0.el9.noarch 1/1 2024-09-17T16:01:03.482 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:03.482 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:01:03.482 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm-2:19.2.0-0.el9.noarch 2024-09-17T16:01:03.482 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:03.482 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:03.559 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:19.2.0-0.el9 ceph 126 k 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:03.560 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 126 k 2024-09-17T16:01:03.561 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 456 k 2024-09-17T16:01:03.561 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:01:03.600 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-19.2.0-0.el9.x86_64.rpm 3.1 MB/s | 126 kB 00:00 2024-09-17T16:01:03.601 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:03.602 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.0 MB/s | 126 kB 00:00 2024-09-17T16:01:03.602 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:01:03.607 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:01:03.607 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:01:03.664 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:01:03.664 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:01:03.698 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T16:01:03.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:03.724 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:03.724 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:01:03.724 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:03.724 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:03.724 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:03.803 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:01:03.904 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:03.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-09-17T16:01:04.311 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:04.318 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:04.453 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:04.453 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-09-17T16:01:04.453 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:04.453 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:01:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache x86_64 2:19.2.0-0.el9 ceph 145 k 2024-09-17T16:01:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:04.454 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:01:04.454 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:04.454 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-17T16:01:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:04.454 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 145 k 2024-09-17T16:01:04.454 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 435 k 2024-09-17T16:01:04.455 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:01:04.507 INFO:teuthology.orchestra.run.smithi017.stdout:ceph-immutable-object-cache-19.2.0-0.el9.x86_64 2.7 MB/s | 145 kB 00:00 2024-09-17T16:01:04.507 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:04.508 INFO:teuthology.orchestra.run.smithi017.stdout:Total 2.6 MB/s | 145 kB 00:00 2024-09-17T16:01:04.508 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:01:04.516 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:01:04.516 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:01:04.531 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:04 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:04.577 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:01:04.577 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:01:04.594 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:04.594 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:04.594 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:01:04.594 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:04.594 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:04.594 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:04.621 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:04.666 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:04.667 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T16:01:04.667 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:04.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-09-17T16:01:04.733 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:01:04.799 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:04.826 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2024-09-17T16:01:04.826 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:04.826 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:01:04.826 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T16:01:04.826 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:05.315 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:05 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:05.410 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:02 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:05.447 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:05.448 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:05.448 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:05.448 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:05.449 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:01:05.449 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.2.0-0.el9 ceph 32 k 2024-09-17T16:01:05.449 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:05.449 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:01:05.449 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:05.449 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T16:01:05.449 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:05.449 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 32 k 2024-09-17T16:01:05.449 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 142 k 2024-09-17T16:01:05.449 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:01:05.502 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:05.506 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-19.2.0-0.el9.x86_64.rpm 551 kB/s | 32 kB 00:00 2024-09-17T16:01:05.507 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:05.507 INFO:teuthology.orchestra.run.smithi173.stdout:Total 539 kB/s | 32 kB 00:00 2024-09-17T16:01:05.508 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:01:05.511 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:01:05.511 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:01:05.525 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:01:05.525 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:01:05.545 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:05.546 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-17T16:01:05.546 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:05.618 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:01:05.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2024-09-17T16:01:05.724 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:05.725 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:05.725 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:01:05.725 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:05.725 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:05.725 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:05.735 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:05.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr 2024-09-17T16:01:06.099 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:06.221 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:03 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:06.323 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:01:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:06.323 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:06.351 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:06.352 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:06.352 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:06.352 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:06.352 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:01:06.352 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:19.2.0-0.el9 ceph 32 k 2024-09-17T16:01:06.352 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:06.352 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:01:06.352 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:06.353 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-17T16:01:06.353 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:06.353 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 32 k 2024-09-17T16:01:06.353 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 142 k 2024-09-17T16:01:06.353 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:01:06.411 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-19.2.0-0.el9.x86_64.rpm 543 kB/s | 32 kB 00:00 2024-09-17T16:01:06.412 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:06.412 INFO:teuthology.orchestra.run.smithi071.stdout:Total 531 kB/s | 32 kB 00:00 2024-09-17T16:01:06.412 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:01:06.416 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:01:06.416 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:01:06.432 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:01:06.432 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:01:06.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-09-17T16:01:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:01:06.596 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:58 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:06.648 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:06.688 INFO:teuthology.orchestra.run.smithi017.stdout:Package ceph-mgr-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:06.734 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:06.735 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-17T16:01:06.735 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:06.800 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-fuse 2024-09-17T16:01:07.007 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:07.057 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:07 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:07.147 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:07.188 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:07.189 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T16:01:07.189 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:07.233 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:07.233 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:07.233 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:01:07.233 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:07.233 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:07.233 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:07.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-09-17T16:01:07.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2024-09-17T16:01:07.401 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:07.538 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:07.538 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse x86_64 2:19.2.0-0.el9 ceph 896 k 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout:Install 2 Packages 2024-09-17T16:01:07.539 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:07.540 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 976 k 2024-09-17T16:01:07.540 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 2.9 M 2024-09-17T16:01:07.540 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:01:07.825 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:07 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:07.895 INFO:teuthology.orchestra.run.smithi017.stdout:(1/2): ceph-fuse-19.2.0-0.el9.x86_64.rpm 13 MB/s | 896 kB 00:00 2024-09-17T16:01:07.914 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:07.951 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:04 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:07.955 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:07.956 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T16:01:07.956 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:08.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-09-17T16:01:08.040 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:08.082 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:08.083 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-17T16:01:08.083 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:08.145 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2024-09-17T16:01:08.295 INFO:teuthology.orchestra.run.smithi017.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 171 kB/s | 80 kB 00:00 2024-09-17T16:01:08.295 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:08.296 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.3 MB/s | 976 kB 00:00 2024-09-17T16:01:08.309 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:01:08.333 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:01:08.333 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:01:08.415 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:01:08.415 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:01:08.593 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:08 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:08.619 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:01:08.683 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:08.719 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:05 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:08.724 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:08.725 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T16:01:08.725 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:08.776 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T16:01:08.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-09-17T16:01:08.808 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:08.845 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-fuse-2:19.2.0-0.el9.x86_64 2/2 2024-09-17T16:01:08.850 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:08.851 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-17T16:01:08.851 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:08.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2024-09-17T16:01:09.363 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:09 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:09.454 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:09.483 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:06 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:09.495 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:09.496 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T16:01:09.497 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:09.511 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-fuse-2:19.2.0-0.el9.x86_64 2/2 2024-09-17T16:01:09.511 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-fuse-2:19.2.0-0.el9.x86_64 1/2 2024-09-17T16:01:09.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-09-17T16:01:09.568 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:09.610 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:09.611 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-17T16:01:09.611 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:09.672 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2024-09-17T16:01:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T16:01:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:09.807 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:01:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse-2:19.2.0-0.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T16:01:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:09.807 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:10.066 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librados-devel 2024-09-17T16:01:10.134 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:10 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:10.223 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:10.242 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:07 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:10.266 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:10.267 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T16:01:10.267 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:10.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-09-17T16:01:10.332 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:10.373 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:10.374 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-17T16:01:10.375 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:10.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2024-09-17T16:01:10.674 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:02 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:10.812 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel x86_64 2:19.2.0-0.el9 ceph 126 k 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 126 k 2024-09-17T16:01:10.813 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 456 k 2024-09-17T16:01:10.814 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:01:10.858 INFO:teuthology.orchestra.run.smithi017.stdout:librados-devel-19.2.0-0.el9.x86_64.rpm 2.8 MB/s | 126 kB 00:00 2024-09-17T16:01:10.858 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:10.859 INFO:teuthology.orchestra.run.smithi017.stdout:Total 2.7 MB/s | 126 kB 00:00 2024-09-17T16:01:10.859 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:01:10.864 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:01:10.865 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:01:10.920 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:10 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:10.925 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:01:10.925 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:01:11.014 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:11.014 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:08 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:11.059 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:11.060 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-09-17T16:01:11.060 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:01:11.104 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:11.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-09-17T16:01:11.146 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:11.147 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-17T16:01:11.148 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librados-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:11.212 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2024-09-17T16:01:11.647 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:11.727 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:11 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:11.801 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:08 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:11.862 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.2.0-0.el9 ceph 85 k 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 85 k 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 231 k 2024-09-17T16:01:11.863 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:01:11.894 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:11.913 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:11.913 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:11.913 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:01:11.913 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:11.913 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:11.913 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:11.919 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-19.2.0-0.el9.x86_64.rpm 1.5 MB/s | 85 kB 00:00 2024-09-17T16:01:11.919 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:11.919 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.5 MB/s | 85 kB 00:00 2024-09-17T16:01:11.920 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:01:11.927 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:01:11.927 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:01:11.939 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:11.940 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-09-17T16:01:11.940 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:11.983 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:01:11.983 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:01:12.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2024-09-17T16:01:12.122 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:01:12.161 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install libcephfs2 2024-09-17T16:01:12.213 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:12.597 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:09 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:12.730 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:12.730 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:12.730 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:12.730 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:12.731 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:01:12.731 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:19.2.0-0.el9 ceph 85 k 2024-09-17T16:01:12.731 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:12.731 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:01:12.731 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:12.731 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-17T16:01:12.731 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:12.731 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 85 k 2024-09-17T16:01:12.731 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 231 k 2024-09-17T16:01:12.731 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:01:12.764 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:04 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:12.781 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-19.2.0-0.el9.x86_64.rpm 1.7 MB/s | 85 kB 00:00 2024-09-17T16:01:12.782 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:12.782 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.6 MB/s | 85 kB 00:00 2024-09-17T16:01:12.782 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:01:12.789 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:01:12.789 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:01:12.846 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:01:12.846 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:01:12.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:12.848 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:12.848 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:01:12.848 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:12.848 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:12.848 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:12.858 INFO:teuthology.orchestra.run.smithi017.stdout:Package libcephfs2-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:12.904 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:12.906 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-17T16:01:12.906 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:12.981 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install libcephfs-devel 2024-09-17T16:01:12.985 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:01:13.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-09-17T16:01:13.077 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:13.515 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:13.583 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:05 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:13.656 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:13 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:13.721 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:13.722 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:13.722 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:13.722 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:01:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel x86_64 2:19.2.0-0.el9 ceph 32 k 2024-09-17T16:01:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:13.722 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:01:13.722 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:13.722 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-17T16:01:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:13.723 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 32 k 2024-09-17T16:01:13.723 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 142 k 2024-09-17T16:01:13.723 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:01:13.767 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:13.767 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:13.768 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:01:13.768 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:13.768 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:13.768 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:13.779 INFO:teuthology.orchestra.run.smithi017.stdout:libcephfs-devel-19.2.0-0.el9.x86_64.rpm 559 kB/s | 32 kB 00:00 2024-09-17T16:01:13.779 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:13.780 INFO:teuthology.orchestra.run.smithi017.stdout:Total 547 kB/s | 32 kB 00:00 2024-09-17T16:01:13.780 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:01:13.783 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:01:13.783 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:01:13.788 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:13.789 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:13.789 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:13.789 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:01:13.790 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.2.0-0.el9 ceph 3.1 M 2024-09-17T16:01:13.790 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:01:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T16:01:13.790 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.1 M 2024-09-17T16:01:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 13 M 2024-09-17T16:01:13.790 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:01:13.811 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:01:13.812 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:01:13.879 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-19.2.0-0.el9.x86_64.rpm 35 MB/s | 3.1 MB 00:00 2024-09-17T16:01:13.880 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:13.881 INFO:teuthology.orchestra.run.smithi173.stdout:Total 34 MB/s | 3.1 MB 00:00 2024-09-17T16:01:13.881 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:01:13.889 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:01:13.890 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:01:13.893 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:01:13.966 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:01:13.967 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:01:13.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2024-09-17T16:01:14.018 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:14.387 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:01:14.413 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:11 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:14.609 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:01:14.609 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T16:01:14.609 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T16:01:14.609 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T16:01:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:14.639 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs-devel-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:14.639 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:14.639 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:01:14.639 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:14.639 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:14.639 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:14.717 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:01:14.717 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:19.2.0-0.el9 ceph 3.1 M 2024-09-17T16:01:14.717 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:01:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-17T16:01:14.718 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.1 M 2024-09-17T16:01:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 13 M 2024-09-17T16:01:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:01:14.825 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-19.2.0-0.el9.x86_64.rpm 29 MB/s | 3.1 MB 00:00 2024-09-17T16:01:14.826 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:14.826 INFO:teuthology.orchestra.run.smithi071.stdout:Total 29 MB/s | 3.1 MB 00:00 2024-09-17T16:01:14.826 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:01:14.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librados2 2024-09-17T16:01:14.836 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:01:14.836 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:01:14.913 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:01:14.913 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:01:15.337 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:01:15.427 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:07 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:15.493 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:15.519 INFO:teuthology.orchestra.run.smithi017.stdout:Package librados2-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:15.521 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:15.521 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:01:15.521 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T16:01:15.521 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T16:01:15.521 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T16:01:15.521 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:15.563 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:15.564 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:01:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:15.564 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:15.565 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-17T16:01:15.565 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:15.635 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librbd1 2024-09-17T16:01:15.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-09-17T16:01:16.234 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:08 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:16.326 INFO:teuthology.orchestra.run.smithi017.stdout:Package librbd1-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:16.371 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:16.373 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-17T16:01:16.373 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:16.391 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:16 ago on Tue 17 Sep 2024 04:00:00 PM UTC. 2024-09-17T16:01:16.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rados 2024-09-17T16:01:16.492 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:16.493 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:16.493 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:01:16.493 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:16.493 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:16.493 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:16.529 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-09-17T16:01:16.530 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:16.530 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:16.530 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:16.530 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-09-17T16:01:16.530 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.2.0-0.el9 ceph 170 k 2024-09-17T16:01:16.530 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:16.530 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-09-17T16:01:16.530 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-09-17T16:01:16.530 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-09-17T16:01:16.530 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:16.531 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 170 k 2024-09-17T16:01:16.531 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 494 k 2024-09-17T16:01:16.531 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-09-17T16:01:16.575 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-19.2.0-0.el9.x86_64.rpm 3.8 MB/s | 170 kB 00:00 2024-09-17T16:01:16.576 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:16.576 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.6 MB/s | 170 kB 00:00 2024-09-17T16:01:16.576 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-09-17T16:01:16.584 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-09-17T16:01:16.584 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-09-17T16:01:16.640 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-09-17T16:01:16.641 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-09-17T16:01:16.729 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2024-09-17T16:01:16.790 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-09-17T16:01:16.888 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:17.041 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:09 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:17.134 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rados-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:17.179 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:17.180 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-17T16:01:17.180 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:17.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rgw 2024-09-17T16:01:17.293 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:17.310 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:14 ago on Tue 17 Sep 2024 04:00:03 PM UTC. 2024-09-17T16:01:17.441 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-09-17T16:01:17.442 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:17.442 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:17.442 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:17.442 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-09-17T16:01:17.442 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:19.2.0-0.el9 ceph 170 k 2024-09-17T16:01:17.442 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:17.442 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-09-17T16:01:17.442 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-09-17T16:01:17.442 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-09-17T16:01:17.443 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:17.443 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 170 k 2024-09-17T16:01:17.443 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 494 k 2024-09-17T16:01:17.443 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-09-17T16:01:17.487 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-19.2.0-0.el9.x86_64.rpm 3.8 MB/s | 170 kB 00:00 2024-09-17T16:01:17.487 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:17.488 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.7 MB/s | 170 kB 00:00 2024-09-17T16:01:17.488 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-09-17T16:01:17.495 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-09-17T16:01:17.495 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-09-17T16:01:17.542 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:17.542 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:17.542 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-09-17T16:01:17.542 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:17.542 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:01:17.542 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-09-17T16:01:17.552 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-09-17T16:01:17.552 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-09-17T16:01:17.705 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-09-17T16:01:17.787 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:17.859 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:09 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:17.954 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rgw-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:17.999 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:18.000 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-17T16:01:18.000 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:18.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-cephfs 2024-09-17T16:01:18.220 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:18.448 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:18.448 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:18.448 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-09-17T16:01:18.448 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:18.448 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:01:18.448 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-09-17T16:01:18.689 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:18.690 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:10 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:18.781 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-cephfs-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:18.825 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:18.826 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-17T16:01:18.826 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:18.889 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rbd 2024-09-17T16:01:19.486 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:11 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:19.579 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rbd-2:19.2.0-0.el9.x86_64 is already installed. 2024-09-17T16:01:19.624 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:19.625 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-17T16:01:19.625 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:19.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-fuse 2024-09-17T16:01:20.299 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:12 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:20.436 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse x86_64 2:19.2.0-0.el9 ceph 85 k 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:20.437 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 85 k 2024-09-17T16:01:20.438 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 231 k 2024-09-17T16:01:20.438 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:01:20.483 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-fuse-19.2.0-0.el9.x86_64.rpm 1.8 MB/s | 85 kB 00:00 2024-09-17T16:01:20.484 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:20.484 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.8 MB/s | 85 kB 00:00 2024-09-17T16:01:20.484 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:01:20.492 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:01:20.492 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:01:20.551 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:01:20.551 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:01:20.714 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:01:20.823 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-fuse-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:21.215 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-fuse-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:21.478 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-fuse-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:21.478 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:21.478 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:01:21.478 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:21.478 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:21.478 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:21.711 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-mirror 2024-09-17T16:01:22.312 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:14 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:22.451 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:22.451 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror x86_64 2:19.2.0-0.el9 ceph 3.1 M 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 3.1 M 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 13 M 2024-09-17T16:01:22.452 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:01:22.545 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-mirror-19.2.0-0.el9.x86_64.rpm 33 MB/s | 3.1 MB 00:00 2024-09-17T16:01:22.546 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:22.546 INFO:teuthology.orchestra.run.smithi017.stdout:Total 33 MB/s | 3.1 MB 00:00 2024-09-17T16:01:22.546 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:01:22.556 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:01:22.556 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:01:22.635 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:01:22.635 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:01:23.079 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:01:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-mirror-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-mirror-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:23.266 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T16:01:23.266 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T16:01:23.266 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T16:01:23.266 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T16:01:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:24.323 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-mirror-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:24.323 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:24.323 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:01:24.323 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:24.323 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:24.323 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:24.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-nbd 2024-09-17T16:01:25.147 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:17 ago on Tue 17 Sep 2024 04:00:08 PM UTC. 2024-09-17T16:01:25.283 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-17T16:01:25.283 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:25.284 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-17T16:01:25.284 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:25.284 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-17T16:01:25.284 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd x86_64 2:19.2.0-0.el9 ceph 170 k 2024-09-17T16:01:25.284 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:25.284 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-17T16:01:25.284 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-17T16:01:25.284 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-17T16:01:25.284 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:25.285 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 170 k 2024-09-17T16:01:25.285 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 494 k 2024-09-17T16:01:25.285 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-17T16:01:25.334 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-nbd-19.2.0-0.el9.x86_64.rpm 3.4 MB/s | 170 kB 00:00 2024-09-17T16:01:25.334 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:01:25.335 INFO:teuthology.orchestra.run.smithi017.stdout:Total 3.3 MB/s | 170 kB 00:00 2024-09-17T16:01:25.335 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-17T16:01:25.342 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-17T16:01:25.342 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-17T16:01:25.403 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-17T16:01:25.403 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-17T16:01:25.569 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-17T16:01:25.673 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-nbd-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-nbd-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:26.363 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-nbd-2:19.2.0-0.el9.x86_64 1/1 2024-09-17T16:01:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:26.364 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-17T16:01:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd-2:19.2.0-0.el9.x86_64 2024-09-17T16:01:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:01:26.364 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-17T16:01:26.601 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:26.602 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T16:01:26.602 INFO:teuthology.packaging:ref: None 2024-09-17T16:01:26.602 INFO:teuthology.packaging:tag: None 2024-09-17T16:01:26.602 INFO:teuthology.packaging:branch: squid 2024-09-17T16:01:26.602 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:01:26.602 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-17T16:01:26.833 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T16:01:26.855 INFO:teuthology.orchestra.run.smithi017.stdout:19.2.0-0.el9 2024-09-17T16:01:26.856 INFO:teuthology.packaging:The installed version of ceph is 19.2.0-0.el9 2024-09-17T16:01:26.856 INFO:teuthology.task.install:The correct ceph version 19.2.0-0 is installed. 2024-09-17T16:01:26.858 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T16:01:26.858 INFO:teuthology.packaging:ref: None 2024-09-17T16:01:26.858 INFO:teuthology.packaging:tag: None 2024-09-17T16:01:26.858 INFO:teuthology.packaging:branch: squid 2024-09-17T16:01:26.858 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:01:26.858 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-17T16:01:27.088 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T16:01:27.113 INFO:teuthology.orchestra.run.smithi071.stdout:19.2.0-0.el9 2024-09-17T16:01:27.113 INFO:teuthology.packaging:The installed version of ceph is 19.2.0-0.el9 2024-09-17T16:01:27.113 INFO:teuthology.task.install:The correct ceph version 19.2.0-0 is installed. 2024-09-17T16:01:27.115 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T16:01:27.115 INFO:teuthology.packaging:ref: None 2024-09-17T16:01:27.115 INFO:teuthology.packaging:tag: None 2024-09-17T16:01:27.115 INFO:teuthology.packaging:branch: squid 2024-09-17T16:01:27.115 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:01:27.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-17T16:01:27.350 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi173.stdout:19.2.0-0.el9 2024-09-17T16:01:27.376 INFO:teuthology.packaging:The installed version of ceph is 19.2.0-0.el9 2024-09-17T16:01:27.376 INFO:teuthology.task.install:The correct ceph version 19.2.0-0 is installed. 2024-09-17T16:01:27.378 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T16:01:27.378 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:01:27.378 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T16:01:27.410 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T16:01:27.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T16:01:27.441 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T16:01:27.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T16:01:27.471 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T16:01:27.471 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:01:27.471 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T16:01:27.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T16:01:27.572 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T16:01:27.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T16:01:27.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T16:01:27.669 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T16:01:27.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T16:01:27.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T16:01:27.768 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T16:01:27.768 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:01:27.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T16:01:27.799 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T16:01:27.869 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T16:01:27.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T16:01:27.899 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T16:01:27.968 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T16:01:27.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T16:01:27.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T16:01:28.067 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T16:01:28.067 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:01:28.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T16:01:28.098 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T16:01:28.167 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T16:01:28.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T16:01:28.197 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T16:01:28.265 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T16:01:28.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T16:01:28.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T16:01:28.362 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-17T16:01:28.460 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'} 2024-09-17T16:01:28.460 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:01:28.461 INFO:tasks.cephadm:Cluster fsid is 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:01:28.461 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-17T16:01:28.461 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.17', 'mon.b': '172.21.15.71', 'mon.c': '172.21.15.173'} 2024-09-17T16:01:28.461 INFO:tasks.cephadm:First mon is mon.a on smithi017 2024-09-17T16:01:28.461 INFO:tasks.cephadm:First mgr is y 2024-09-17T16:01:28.461 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-17T16:01:28.462 DEBUG:teuthology.orchestra.run.smithi017:> sudo hostname $(hostname -s) 2024-09-17T16:01:28.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo hostname $(hostname -s) 2024-09-17T16:01:28.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-09-17T16:01:28.550 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-17T16:01:28.551 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:01:28.697 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83109/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4975-g1889a165', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4975.g1889a165'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-17 12:26:12.491371', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-bharath10-testing-2024-09-17-1348', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83109/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4975-g1889a165', 'node_name': '172.21.5.40+adami10', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4975.g1889a165'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-17 10:41:40.067980', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-bharath10-testing-2024-09-17-1348', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-17T16:01:28.840 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-bharath10-testing-2024-09-17-1348, sha1 1889a165b5128787ec693450a3d1c5df5d6dc203 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:01:28.842 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/x86_64/flavors/default/cephadm 2024-09-17T16:01:28.842 INFO:tasks.cephadm:Downloading cephadm from url: https://3.chacra.ceph.com/binaries/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/x86_64/flavors/default/cephadm 2024-09-17T16:01:28.842 DEBUG:teuthology.orchestra.run.smithi017:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T16:01:29.199 INFO:teuthology.orchestra.run.smithi017.stdout:-rw-r--r--. 1 ubuntu ubuntu 804195 Sep 17 16:01 /home/ubuntu/cephtest/cephadm 2024-09-17T16:01:29.200 DEBUG:teuthology.orchestra.run.smithi071:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T16:01:29.564 INFO:teuthology.orchestra.run.smithi071.stdout:-rw-r--r--. 1 ubuntu ubuntu 804195 Sep 17 16:01 /home/ubuntu/cephtest/cephadm 2024-09-17T16:01:29.565 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T16:01:29.916 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 804195 Sep 17 16:01 /home/ubuntu/cephtest/cephadm 2024-09-17T16:01:29.916 DEBUG:teuthology.orchestra.run.smithi017:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T16:01:29.938 DEBUG:teuthology.orchestra.run.smithi071:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T16:01:29.957 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T16:01:29.993 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 on all hosts... 2024-09-17T16:01:29.994 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 pull 2024-09-17T16:01:29.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 pull 2024-09-17T16:01:30.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 pull 2024-09-17T16:01:31.538 INFO:teuthology.orchestra.run.smithi071.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203... 2024-09-17T16:01:31.605 INFO:teuthology.orchestra.run.smithi017.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203... 2024-09-17T16:01:31.637 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203... 2024-09-17T16:02:19.841 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:02:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T16:02:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f", 2024-09-17T16:02:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-09-17T16:02:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0" 2024-09-17T16:02:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:02:19.842 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:02:24.574 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:02:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T16:02:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "image_id": "a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f", 2024-09-17T16:02:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "repo_digests": [ 2024-09-17T16:02:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0" 2024-09-17T16:02:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:02:24.575 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:02:37.931 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:02:37.931 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T16:02:37.931 INFO:teuthology.orchestra.run.smithi071.stdout: "image_id": "a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f", 2024-09-17T16:02:37.931 INFO:teuthology.orchestra.run.smithi071.stdout: "repo_digests": [ 2024-09-17T16:02:37.931 INFO:teuthology.orchestra.run.smithi071.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0" 2024-09-17T16:02:37.932 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:02:37.932 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:02:37.960 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph 2024-09-17T16:02:37.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph 2024-09-17T16:02:38.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-09-17T16:02:38.059 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /etc/ceph 2024-09-17T16:02:38.090 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /etc/ceph 2024-09-17T16:02:38.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-09-17T16:02:38.143 INFO:tasks.cephadm:Writing seed config... 2024-09-17T16:02:38.144 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-09-17T16:02:38.144 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-09-17T16:02:38.145 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [global] ms inject delay max = 1 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [global] ms inject delay probability = 0.005 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [global] ms inject delay type = osd 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [global] ms inject internal delays = 0.002 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [osd] osd backoff on degraded = True 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-09-17T16:02:38.146 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-17T16:02:38.147 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-17T16:02:38.148 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-17T16:02:38.148 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-17T16:02:38.148 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:02:38.148 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-17T16:02:38.167 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 19341aec-750e-11ef-bceb-c7b262605968 mon client directed command retry = 5 mon election default strategy = 3 ms inject delay max = 1 ms inject delay probability = 0.005 ms inject delay type = osd ms inject internal delays = 0.002 ms inject socket failures = 2500 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd backoff on degraded = True osd backoff on peering = True osd blocked scrub grace period = 3600 osd debug reject backfill probability = 0.3 osd max backfills = 3 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-17T16:02:38.167 DEBUG:teuthology.orchestra.run.smithi017:mon.a> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.a.service 2024-09-17T16:02:38.210 DEBUG:teuthology.orchestra.run.smithi017:mgr.y> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@mgr.y.service 2024-09-17T16:02:38.252 INFO:tasks.cephadm:Bootstrapping... 2024-09-17T16:02:38.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 -v bootstrap --fsid 19341aec-750e-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.17 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-17T16:02:38.468 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-17T16:02:38.468 INFO:teuthology.orchestra.run.smithi017.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203', '-v', 'bootstrap', '--fsid', '19341aec-750e-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.17', '--skip-admin-label'] 2024-09-17T16:02:38.469 INFO:teuthology.orchestra.run.smithi017.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-17T16:02:38.469 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying podman|docker is present... 2024-09-17T16:02:38.496 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stdout 5.2.2 2024-09-17T16:02:38.496 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying lvm2 is present... 2024-09-17T16:02:38.496 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying time synchronization is in place... 2024-09-17T16:02:38.504 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-17T16:02:38.504 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T16:02:38.510 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-17T16:02:38.511 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2024-09-17T16:02:38.517 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout enabled 2024-09-17T16:02:38.524 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout active 2024-09-17T16:02:38.524 INFO:teuthology.orchestra.run.smithi017.stdout:Unit chronyd.service is enabled and running 2024-09-17T16:02:38.524 INFO:teuthology.orchestra.run.smithi017.stdout:Repeating the final host check... 2024-09-17T16:02:38.547 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stdout 5.2.2 2024-09-17T16:02:38.548 INFO:teuthology.orchestra.run.smithi017.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-17T16:02:38.548 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl is present 2024-09-17T16:02:38.548 INFO:teuthology.orchestra.run.smithi017.stdout:lvcreate is present 2024-09-17T16:02:38.554 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-17T16:02:38.555 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T16:02:38.561 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-17T16:02:38.561 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2024-09-17T16:02:38.568 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout enabled 2024-09-17T16:02:38.574 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout active 2024-09-17T16:02:38.574 INFO:teuthology.orchestra.run.smithi017.stdout:Unit chronyd.service is enabled and running 2024-09-17T16:02:38.574 INFO:teuthology.orchestra.run.smithi017.stdout:Host looks OK 2024-09-17T16:02:38.575 INFO:teuthology.orchestra.run.smithi017.stdout:Cluster fsid: 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:02:38.575 INFO:teuthology.orchestra.run.smithi017.stdout:Acquiring lock 140540914779472 on /run/cephadm/19341aec-750e-11ef-bceb-c7b262605968.lock 2024-09-17T16:02:38.575 INFO:teuthology.orchestra.run.smithi017.stdout:Lock 140540914779472 acquired on /run/cephadm/19341aec-750e-11ef-bceb-c7b262605968.lock 2024-09-17T16:02:38.575 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying IP 172.21.15.17 port 3300 ... 2024-09-17T16:02:38.575 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying IP 172.21.15.17 port 6789 ... 2024-09-17T16:02:38.575 INFO:teuthology.orchestra.run.smithi017.stdout:Base mon IP(s) is [172.21.15.17:3300, 172.21.15.17:6789], mon addrv is [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2024-09-17T16:02:38.579 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.17 metric 100 2024-09-17T16:02:38.579 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.17 metric 100 2024-09-17T16:02:38.581 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-17T16:02:38.582 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-17T16:02:38.582 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2024-09-17T16:02:38.584 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-17T16:02:38.584 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-17T16:02:38.584 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-17T16:02:38.584 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout 2: ens1f0: mtu 1500 state UP qlen 1000 2024-09-17T16:02:38.584 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:d992/64 scope link 2024-09-17T16:02:38.584 INFO:teuthology.orchestra.run.smithi017.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-17T16:02:38.585 INFO:teuthology.orchestra.run.smithi017.stdout:Mon IP `172.21.15.17` is in CIDR network `172.21.0.0/20` 2024-09-17T16:02:38.585 INFO:teuthology.orchestra.run.smithi017.stdout:Mon IP `172.21.15.17` is in CIDR network `172.21.0.0/20` 2024-09-17T16:02:38.585 INFO:teuthology.orchestra.run.smithi017.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-17T16:02:38.586 INFO:teuthology.orchestra.run.smithi017.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-17T16:02:38.586 INFO:teuthology.orchestra.run.smithi017.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203... 2024-09-17T16:02:38.999 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stdout a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f 2024-09-17T16:02:39.000 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203... 2024-09-17T16:02:39.000 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Getting image source signatures 2024-09-17T16:02:39.000 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Copying blob sha256:e11a7257e47fe71e77db5b8b96dd43de1731e859ead23756d44c5e906da20a42 2024-09-17T16:02:39.000 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-17T16:02:39.000 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Copying config sha256:a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f 2024-09-17T16:02:39.000 INFO:teuthology.orchestra.run.smithi017.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-17T16:02:39.913 INFO:teuthology.orchestra.run.smithi017.stdout:ceph: stdout ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev) 2024-09-17T16:02:39.913 INFO:teuthology.orchestra.run.smithi017.stdout:Ceph version: ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev) 2024-09-17T16:02:39.913 INFO:teuthology.orchestra.run.smithi017.stdout:Extracting ceph user uid/gid from container image... 2024-09-17T16:02:40.809 INFO:teuthology.orchestra.run.smithi017.stdout:stat: stdout 167 167 2024-09-17T16:02:40.810 INFO:teuthology.orchestra.run.smithi017.stdout:Creating initial keys... 2024-09-17T16:02:41.815 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph-authtool: stdout AQAhqOlmri8TBxAAs9exkXIjulEqCYLsYnbEMA== 2024-09-17T16:02:42.720 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph-authtool: stdout AQAiqOlmGqzQBxAArNQegpmMlvwt+3R5xwZkKA== 2024-09-17T16:02:43.734 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph-authtool: stdout AQAjqOlm1cWcAxAAUUtGmnHdDEDiNqQPm+zxig== 2024-09-17T16:02:43.735 INFO:teuthology.orchestra.run.smithi017.stdout:Creating initial monmap... 2024-09-17T16:02:44.631 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T16:02:44.631 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-17T16:02:44.631 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:02:44.631 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T16:02:44.631 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool for a [v2:172.21.15.17:3300,v1:172.21.15.17:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T16:02:44.632 INFO:teuthology.orchestra.run.smithi017.stdout:setting min_mon_release = quincy 2024-09-17T16:02:44.632 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: set fsid to 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:02:44.632 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T16:02:44.632 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:02:44.632 INFO:teuthology.orchestra.run.smithi017.stdout:Creating mon... 2024-09-17T16:02:45.536 INFO:teuthology.orchestra.run.smithi017.stdout:create mon.a on 2024-09-17T16:02:45.770 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-17T16:02:45.919 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-17T16:02:46.098 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-19341aec-750e-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-19341aec-750e-11ef-bceb-c7b262605968.target. 2024-09-17T16:02:46.099 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-19341aec-750e-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-19341aec-750e-11ef-bceb-c7b262605968.target. 2024-09-17T16:02:46.342 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.a 2024-09-17T16:02:46.342 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to reset failed state of unit ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-17T16:02:46.529 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-19341aec-750e-11ef-bceb-c7b262605968.target.wants/ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-19341aec-750e-11ef-bceb-c7b262605968@.service. 2024-09-17T16:02:47.027 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T16:02:47.027 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout disabled 2024-09-17T16:02:47.035 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T16:02:47.035 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2024-09-17T16:02:47.035 INFO:teuthology.orchestra.run.smithi017.stdout:firewalld.service is not enabled 2024-09-17T16:02:47.035 INFO:teuthology.orchestra.run.smithi017.stdout:Not possible to enable service . firewalld.service is not available 2024-09-17T16:02:47.036 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mon to start... 2024-09-17T16:02:47.036 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mon... 2024-09-17T16:02:48.031 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout cluster: 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout id: 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout services: 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.539397s) 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout data: 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-17T16:02:48.032 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-17T16:02:48.033 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-17T16:02:48.033 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout pgs: 2024-09-17T16:02:48.033 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:02:48.033 INFO:teuthology.orchestra.run.smithi017.stdout:mon is available 2024-09-17T16:02:48.033 INFO:teuthology.orchestra.run.smithi017.stdout:Assimilating anything we can from ceph.conf... 2024-09-17T16:02:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:02:48 smithi017 ceph-mon[27195]: from='client.? 172.21.15.17:0/2782212162' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-17T16:02:51.597 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:02:51.597 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [global] 2024-09-17T16:02:51.597 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout fsid = 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:02:51.597 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-17T16:02:51.597 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2024-09-17T16:02:51.598 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-17T16:02:51.598 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-17T16:02:51.598 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-17T16:02:51.598 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-17T16:02:51.598 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:02:51.598 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-17T16:02:51.598 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-17T16:02:51.598 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:02:51.599 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [osd] 2024-09-17T16:02:51.599 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-17T16:02:51.599 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-17T16:02:51.599 INFO:teuthology.orchestra.run.smithi017.stdout:Generating new minimal ceph.conf... 2024-09-17T16:02:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:02:55 smithi017 ceph-mon[27195]: from='client.? 172.21.15.17:0/1080959390' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:03:01.703 INFO:teuthology.orchestra.run.smithi017.stdout:Restarting the monitor... 2024-09-17T16:03:01.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:01 smithi017 systemd[1]: Stopping Ceph mon.a for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:03:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:02 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27191]: 2024-09-17T16:03:02.681+0000 7fd3403e9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:03:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:02 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27191]: 2024-09-17T16:03:02.681+0000 7fd3403e9640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-17T16:03:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:02 smithi017 podman[27447]: 2024-09-17 16:03:02.906909458 +0000 UTC m=+0.954808854 container died 820bf537988e12b8c0ce2f157e27dfe2bdf7faac7e4f36555db59586e8150b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16) 2024-09-17T16:03:03.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:03 smithi017 podman[27447]: 2024-09-17 16:03:03.246211286 +0000 UTC m=+1.294110681 container cleanup 820bf537988e12b8c0ce2f157e27dfe2bdf7faac7e4f36555db59586e8150b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T16:03:03.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:03 smithi017 bash[27447]: ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a 2024-09-17T16:03:03.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:03 smithi017 podman[27459]: 2024-09-17 16:03:03.413899364 +0000 UTC m=+0.504721190 container remove 820bf537988e12b8c0ce2f157e27dfe2bdf7faac7e4f36555db59586e8150b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-17T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:03 smithi017 systemd[1]: ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:03 smithi017 systemd[1]: Stopped Ceph mon.a for 19341aec-750e-11ef-bceb-c7b262605968. 2024-09-17T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:03 smithi017 systemd[1]: Starting Ceph mon.a for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:03 smithi017 podman[27553]: 2024-09-17 16:03:03.889634252 +0000 UTC m=+0.113204608 container create efaebd570b8a107a6fa3d4b5460d96b81d9c8016b5481270e67557b7c83f2702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-17T16:03:03.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:03 smithi017 podman[27553]: 2024-09-17 16:03:03.793253282 +0000 UTC m=+0.016823653 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T16:03:04.181 INFO:teuthology.orchestra.run.smithi017.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-17T16:03:04.303 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 podman[27553]: 2024-09-17 16:03:04.100382496 +0000 UTC m=+0.323952866 container init efaebd570b8a107a6fa3d4b5460d96b81d9c8016b5481270e67557b7c83f2702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-17T16:03:04.303 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 podman[27553]: 2024-09-17 16:03:04.103698548 +0000 UTC m=+0.327268906 container start efaebd570b8a107a6fa3d4b5460d96b81d9c8016b5481270e67557b7c83f2702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev), process ceph-mon, pid 2 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: pidfile_write: ignore empty --pid-file 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: load: jerasure load: lrc 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: RocksDB version: 7.9.2 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Git sha 0 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Compile date 2024-09-17 09:38:39 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: DB SUMMARY 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: DB Session ID: 9PH2I56DCI257993CN5X 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: CURRENT file: CURRENT 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T16:03:04.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 102547 ; 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.error_if_exists: 0 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.create_if_missing: 0 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.paranoid_checks: 1 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.env: 0x55bdab5480e0 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.info_log: 0x55bdacb5b700 2024-09-17T16:03:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T16:03:04.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.statistics: (nil) 2024-09-17T16:03:04.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.use_fsync: 0 2024-09-17T16:03:04.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_log_file_size: 0 2024-09-17T16:03:04.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T16:03:04.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T16:03:04.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T16:03:04.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T16:03:04.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.allow_fallocate: 1 2024-09-17T16:03:04.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T16:03:04.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.use_direct_reads: 0 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.db_log_dir: 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.wal_dir: 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T16:03:04.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.write_buffer_manager: 0x55bdacb5fb80 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.unordered_write: 0 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T16:03:04.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.row_cache: None 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.wal_filter: None 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.two_write_queues: 0 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.wal_compression: 0 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.atomic_flush: 0 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T16:03:04.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.log_readahead_size: 0 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_background_jobs: 2 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_background_compactions: -1 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_subcompactions: 1 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T16:03:04.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T16:03:04.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T16:03:04.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T16:03:04.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T16:03:04.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T16:03:04.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_open_files: -1 2024-09-17T16:03:04.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T16:03:04.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T16:03:04.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T16:03:04.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_background_flushes: -1 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Compression algorithms supported: 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: kZSTD supported: 0 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: kXpressCompression supported: 0 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: kBZip2Compression supported: 0 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: kLZ4Compression supported: 1 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: kZlibCompression supported: 1 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: kSnappyCompression supported: 1 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-17T16:03:04.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.merge_operator: 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_filter: None 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bdacb5b320) 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_type: 0 2024-09-17T16:03:04.314 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_index_type: 0 2024-09-17T16:03:04.315 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_shortening: 1 2024-09-17T16:03:04.315 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T16:03:04.315 INFO:journalctl@ceph.mon.a.smithi017.stdout: checksum: 4 2024-09-17T16:03:04.315 INFO:journalctl@ceph.mon.a.smithi017.stdout: no_block_cache: 0 2024-09-17T16:03:04.315 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache: 0x55bdacb4f610 2024-09-17T16:03:04.315 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-09-17T16:03:04.315 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_options: 2024-09-17T16:03:04.315 INFO:journalctl@ceph.mon.a.smithi017.stdout: capacity : 536870912 2024-09-17T16:03:04.315 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_shard_bits : 4 2024-09-17T16:03:04.315 INFO:journalctl@ceph.mon.a.smithi017.stdout: strict_capacity_limit : 0 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_compressed: (nil) 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: persistent_cache: (nil) 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size: 4096 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size_deviation: 10 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_restart_interval: 16 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_block_restart_interval: 1 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: metadata_block_size: 4096 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: partition_filters: 0 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: use_delta_encoding: 1 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: filter_policy: bloomfilter 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: whole_key_filtering: 1 2024-09-17T16:03:04.316 INFO:journalctl@ceph.mon.a.smithi017.stdout: verify_compression: 0 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout: format_version: 5 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_index_compression: 1 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_align: 0 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout: max_auto_readahead_size: 262144 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout: prepopulate_block_cache: 0 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout: initial_auto_readahead_size: 8192 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compression: NoCompression 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T16:03:04.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.num_levels: 7 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T16:03:04.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T16:03:04.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T16:03:04.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T16:03:04.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.inplace_update_support: 0 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.bloom_locality: 0 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.max_successive_merges: 0 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T16:03:04.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.ttl: 2592000 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.enable_blob_files: false 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.min_blob_size: 0 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T16:03:04.323 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 59d64c2b-9198-4d82-86b9-fcb02d082e16 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726588984129920, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726588984131037, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 97800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 313, "table_properties": {"data_size": 95783, "index_size": 273, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13618, "raw_average_key_size": 50, "raw_value_size": 88369, "raw_average_value_size": 324, "num_data_blocks": 12, "num_entries": 272, "num_filter_entries": 272, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726588984, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "59d64c2b-9198-4d82-86b9-fcb02d082e16", "db_session_id": "9PH2I56DCI257993CN5X", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726588984131134, "job": 1, "event": "recovery_finished"} 2024-09-17T16:03:04.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bdacb7ee00 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: DB pointer 0x55bdacc80000 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** DB Stats ** 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T16:03:04.325 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 2/0 97.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 97.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 130.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T16:03:04.326 INFO:journalctl@ceph.mon.a.smithi017.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 130.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T16:03:04.327 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 28.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 28.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout: Block cache BinnedLRUCache@0x55bdacb4f610#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: starting mon.a rank 0 at public addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] at bind addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: mon.a@-1(???) e1 preinit fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: mon.a@-1(???).mds e1 new map 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: mon.a@-1(???).mds e1 print_map 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout: e1 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout: btime 2024-09-17T16:02:46:978559+0000 2024-09-17T16:03:04.328 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout: legacy client fscid: -1 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout: No filesystems configured 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 bash[27553]: efaebd570b8a107a6fa3d4b5460d96b81d9c8016b5481270e67557b7c83f2702 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 systemd[1]: Started Ceph mon.a for 19341aec-750e-11ef-bceb-c7b262605968. 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T16:03:04.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: monmap epoch 1 2024-09-17T16:03:04.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:03:04.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: last_changed 2024-09-17T16:02:44.038562+0000 2024-09-17T16:03:04.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: created 2024-09-17T16:02:44.038562+0000 2024-09-17T16:03:04.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: min_mon_release 19 (squid) 2024-09-17T16:03:04.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: election_strategy: 1 2024-09-17T16:03:04.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-17T16:03:04.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: fsmap 2024-09-17T16:03:04.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: osdmap e1: 0 total, 0 up, 0 in 2024-09-17T16:03:04.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:04 smithi017 ceph-mon[27571]: mgrmap e1: no daemons active 2024-09-17T16:03:05.320 INFO:teuthology.orchestra.run.smithi017.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-17T16:03:05.341 INFO:teuthology.orchestra.run.smithi017.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-17T16:03:05.341 INFO:teuthology.orchestra.run.smithi017.stdout:Creating mgr... 2024-09-17T16:03:05.342 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-17T16:03:05.342 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-17T16:03:05.606 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-19341aec-750e-11ef-bceb-c7b262605968@mgr.y 2024-09-17T16:03:05.606 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Failed to reset failed state of unit ceph-19341aec-750e-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-19341aec-750e-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-17T16:03:05.765 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-19341aec-750e-11ef-bceb-c7b262605968.target.wants/ceph-19341aec-750e-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-19341aec-750e-11ef-bceb-c7b262605968@.service. 2024-09-17T16:03:06.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3728428306' entity='client.admin' 2024-09-17T16:03:06.277 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T16:03:06.277 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout disabled 2024-09-17T16:03:06.284 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T16:03:06.284 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2024-09-17T16:03:06.285 INFO:teuthology.orchestra.run.smithi017.stdout:firewalld.service is not enabled 2024-09-17T16:03:06.285 INFO:teuthology.orchestra.run.smithi017.stdout:Not possible to enable service . firewalld.service is not available 2024-09-17T16:03:06.293 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T16:03:06.294 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout disabled 2024-09-17T16:03:06.301 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T16:03:06.301 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2024-09-17T16:03:06.301 INFO:teuthology.orchestra.run.smithi017.stdout:firewalld.service is not enabled 2024-09-17T16:03:06.301 INFO:teuthology.orchestra.run.smithi017.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-17T16:03:06.302 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mgr to start... 2024-09-17T16:03:06.302 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mgr... 2024-09-17T16:03:06.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/746365473' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-17T16:03:07.432 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:03:07.432 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2024-09-17T16:03:07.432 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:03:07.432 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T16:03:07.432 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T16:03:07.432 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T16:03:07.432 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T16:03:07.432 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:07.432 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 0 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T16:03:07.433 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:07.434 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T16:03:07.434 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:07.434 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T16:03:07.434 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T16:03:07.434 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T16:03:07.434 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T16:03:07.434 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T16:03:07.434 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T16:03:07.437 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:07.437 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T16:03:07.437 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T16:03:07.437 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T16:03:07.438 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T16:03:07.438 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T16:03:07.438 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T16:03:07.438 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T16:03:07.438 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T16:03:07.438 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T16:03:07.438 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:07.438 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T16:03:07.438 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:07.438 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "btime": "2024-09-17T16:02:46:978559+0000", 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:07.439 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T16:03:07.440 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:07.440 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T16:02:46.979324+0000", 2024-09-17T16:03:07.440 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T16:03:07.440 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:07.440 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T16:03:07.440 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2024-09-17T16:03:07.440 INFO:teuthology.orchestra.run.smithi017.stdout:mgr not available, waiting (1/15)... 2024-09-17T16:03:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2373206602' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-17T16:03:10.487 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 0 2024-09-17T16:03:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T16:03:10.489 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T16:03:10.490 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T16:03:10.490 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T16:03:10.490 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T16:03:10.490 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T16:03:10.493 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:10.493 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T16:03:10.493 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T16:03:10.493 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T16:03:10.493 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T16:03:10.493 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T16:03:10.493 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T16:03:10.493 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T16:03:10.493 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T16:03:10.493 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "btime": "2024-09-17T16:02:46:978559+0000", 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T16:03:10.494 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T16:02:46.979324+0000", 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2024-09-17T16:03:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:mgr not available, waiting (2/15)... 2024-09-17T16:03:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: Activating manager daemon y 2024-09-17T16:03:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: mgrmap e2: y(active, starting, since 0.00310166s) 2024-09-17T16:03:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: from='mgr.14100 172.21.15.17:0/2109481554' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-17T16:03:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: from='mgr.14100 172.21.15.17:0/2109481554' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-17T16:03:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: from='mgr.14100 172.21.15.17:0/2109481554' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-17T16:03:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: from='mgr.14100 172.21.15.17:0/2109481554' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:03:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: from='mgr.14100 172.21.15.17:0/2109481554' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-17T16:03:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: Manager daemon y is now available 2024-09-17T16:03:11.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: from='mgr.14100 172.21.15.17:0/2109481554' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-17T16:03:11.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: from='mgr.14100 172.21.15.17:0/2109481554' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-17T16:03:11.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: from='mgr.14100 172.21.15.17:0/2109481554' entity='mgr.y' 2024-09-17T16:03:11.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: from='mgr.14100 172.21.15.17:0/2109481554' entity='mgr.y' 2024-09-17T16:03:11.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:11 smithi017 ceph-mon[27571]: from='mgr.14100 172.21.15.17:0/2109481554' entity='mgr.y' 2024-09-17T16:03:13.671 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:13 smithi017 ceph-mon[27571]: mgrmap e3: y(active, since 1.00742s) 2024-09-17T16:03:13.671 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1352613237' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-17T16:03:14.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:03:14.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2024-09-17T16:03:14.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:03:14.001 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "health": { 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 0 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "a" 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:14.002 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-17T16:03:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-17T16:03:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-17T16:03:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-17T16:03:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-17T16:03:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-17T16:03:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-17T16:03:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-17T16:03:14.003 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-17T16:03:14.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:14.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-17T16:03:14.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-17T16:03:14.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-17T16:03:14.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-17T16:03:14.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-17T16:03:14.006 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "btime": "2024-09-17T16:02:46:978559+0000", 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-17T16:03:14.007 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "restful" 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ], 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "modified": "2024-09-17T16:02:46.979324+0000", 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout }, 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2024-09-17T16:03:14.008 INFO:teuthology.orchestra.run.smithi017.stdout:mgr is available 2024-09-17T16:03:14.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:14 smithi017 ceph-mon[27571]: mgrmap e4: y(active, since 2s) 2024-09-17T16:03:15.223 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:03:15.223 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [global] 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout fsid = 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout [osd] 2024-09-17T16:03:15.224 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-17T16:03:15.225 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-17T16:03:15.225 INFO:teuthology.orchestra.run.smithi017.stdout:Enabling cephadm module... 2024-09-17T16:03:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:15 smithi017 ceph-mon[27571]: from='client.14114 172.21.15.17:0/3492424412' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-17T16:03:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/132152021' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-17T16:03:16.739 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:16 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: ignoring --setgroup ceph since I am not root 2024-09-17T16:03:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/132152021' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-17T16:03:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:17 smithi017 ceph-mon[27571]: mgrmap e5: y(active, since 5s) 2024-09-17T16:03:18.268 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2024-09-17T16:03:18.268 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-17T16:03:18.268 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T16:03:18.268 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-17T16:03:18.268 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-17T16:03:18.268 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2024-09-17T16:03:18.268 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for the mgr to restart... 2024-09-17T16:03:18.268 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mgr epoch 5... 2024-09-17T16:03:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2513695936' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-17T16:03:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:21 smithi017 ceph-mon[27571]: Active manager daemon y restarted 2024-09-17T16:03:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:21 smithi017 ceph-mon[27571]: Activating manager daemon y 2024-09-17T16:03:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:21 smithi017 ceph-mon[27571]: osdmap e2: 0 total, 0 up, 0 in 2024-09-17T16:03:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:21 smithi017 ceph-mon[27571]: mgrmap e6: y(active, starting, since 0.00332646s) 2024-09-17T16:03:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:21 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:03:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:21 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-17T16:03:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:21 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-17T16:03:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:21 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-17T16:03:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:21 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-17T16:03:22.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:21 smithi017 ceph-mon[27571]: Manager daemon y is now available 2024-09-17T16:03:23.281 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2024-09-17T16:03:23.282 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-17T16:03:23.282 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-17T16:03:23.282 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2024-09-17T16:03:23.282 INFO:teuthology.orchestra.run.smithi017.stdout:mgr epoch 5 is available 2024-09-17T16:03:23.282 INFO:teuthology.orchestra.run.smithi017.stdout:Verifying orchestrator module is enabled... 2024-09-17T16:03:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:23 smithi017 ceph-mon[27571]: mgrmap e7: y(active, since 1.00504s) 2024-09-17T16:03:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:23 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:23 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:23 smithi017 ceph-mon[27571]: Found migration_current of "None". Setting to last migration. 2024-09-17T16:03:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:23 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:23 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:23 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:03:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:23 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:03:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:23 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-17T16:03:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:23 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-17T16:03:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1490928390' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-17T16:03:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1490928390' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-17T16:03:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-mon[27571]: mgrmap e8: y(active, since 2s) 2024-09-17T16:03:24.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: 2024-09-17T16:03:24.753+0000 7fe590223640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:16:03:24] ENGINE Error in HTTPServer.serve 2024-09-17T16:03:24.989 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: Traceback (most recent call last): 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: self._connections.run(self.expiration_interval) 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: self._run(expiration_interval) 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: s = self.context.wrap_socket( 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: return self.sslsocket_class._create( 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T16:03:24.990 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: self.do_handshake() 2024-09-17T16:03:24.991 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T16:03:24.991 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: self._sslobj.do_handshake() 2024-09-17T16:03:24.991 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T16:03:24.991 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:24 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: 2024-09-17T16:03:25.038 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-17T16:03:25.038 INFO:teuthology.orchestra.run.smithi017.stdout:Setting orchestrator backend to cephadm... 2024-09-17T16:03:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: [17/Sep/2024:16:03:24] ENGINE Bus STARTING 2024-09-17T16:03:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: [17/Sep/2024:16:03:24] ENGINE Serving on https://172.21.15.17:7150 2024-09-17T16:03:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: [17/Sep/2024:16:03:24] ENGINE Error in HTTPServer.serve 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: self._connections.run(self.expiration_interval) 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: self._run(expiration_interval) 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: s = self.context.wrap_socket( 2024-09-17T16:03:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.sslsocket_class._create( 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.do_handshake() 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout: self._sslobj.do_handshake() 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: [17/Sep/2024:16:03:24] ENGINE Serving on http://172.21.15.17:8765 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: [17/Sep/2024:16:03:24] ENGINE Bus STARTED 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:03:25.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:03:25.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:25 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:03:27.499 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-17T16:03:27.500 INFO:teuthology.orchestra.run.smithi017.stdout:Generating ssh key... 2024-09-17T16:03:27.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:27 smithi017 ceph-mon[27571]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:28.735 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: Generating public/private rsa key pair. 2024-09-17T16:03:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: Your identification has been saved in /tmp/tmpbeu2m4cr/key 2024-09-17T16:03:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: Your public key has been saved in /tmp/tmpbeu2m4cr/key.pub 2024-09-17T16:03:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: The key fingerprint is: 2024-09-17T16:03:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: SHA256:giepu87pcQYBCB0M0SGEQ61E7LCeXY+HmnYmgiznz2w ceph-19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:03:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: The key's randomart image is: 2024-09-17T16:03:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: +---[RSA 3072]----+ 2024-09-17T16:03:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: |#X+o | 2024-09-17T16:03:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: |==+. | 2024-09-17T16:03:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: |+oo | 2024-09-17T16:03:28.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: |.o . + | 2024-09-17T16:03:28.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: |. + = * S | 2024-09-17T16:03:28.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: | o + = + | 2024-09-17T16:03:28.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: |o o = . | 2024-09-17T16:03:28.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: |o+.&Eo | 2024-09-17T16:03:28.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: |.=@+B | 2024-09-17T16:03:28.737 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: +----[SHA256]-----+ 2024-09-17T16:03:29.630 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:29 smithi017 ceph-mon[27571]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:29.630 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:29 smithi017 ceph-mon[27571]: Generating ssh key... 2024-09-17T16:03:29.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:29 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:29.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:29 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:30.202 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqgMJ9RshpiqhwrOlYkzH43FFqYDext3sAix+xKjiemxkm21uPPL2mEe19Gw+1uiObcEdYRNLcrQCezCLWpA4KqIaYFdOb6xNpjOQYAcShdqBJh6xhhZ0n5jHiyIWSuB5j9ILJQ45MeA4lCGckRw/+/4SAjfidXcUEFfojIWNv4mUQZHvISap2lHHVHanWJ2/0SBvn2qXpYdMpILzC89skcyBXAaMb0IR36ftrlcjvHy96u+DAB5SMWZyxqfZ48KlpsBjabcoW/x9O4eXNOUBRBYRqXyON3EUGybqY+8np4nApB4IYePkPoSqjAtL9gy04bmPzeEmP/o3RhbkQi4HFBBkDat8r81sSM8Asnll1F7bENWvSrTwO4QUoBsN4Yfszsv81cJQNUpKnquHCJjFHgp4QVtKFs5KohLn321voeMHp7vydJG/6r9tioIVfK+5BWEVEGJpQkeD9RjMMGuxnUYzYtNeXTUMkz96gQjzGR1U/1xGRrucOwEk+XqkXs3c= ceph-19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:03:30.203 INFO:teuthology.orchestra.run.smithi017.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-17T16:03:30.203 INFO:teuthology.orchestra.run.smithi017.stdout:Adding key to root@localhost authorized_keys... 2024-09-17T16:03:30.207 INFO:teuthology.orchestra.run.smithi017.stdout:Adding host smithi017... 2024-09-17T16:03:30.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:30 smithi017 ceph-mon[27571]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:31.811 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:31 smithi017 ceph-mon[27571]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi017", "addr": "172.21.15.17", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:32.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:32 smithi017 ceph-mon[27571]: Deploying cephadm binary to smithi017 2024-09-17T16:03:35.423 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout Added host 'smithi017' with addr '172.21.15.17' 2024-09-17T16:03:35.423 INFO:teuthology.orchestra.run.smithi017.stdout:Deploying unmanaged mon service... 2024-09-17T16:03:35.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:35 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:35.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:35 smithi017 ceph-mon[27571]: Added host smithi017 2024-09-17T16:03:35.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:35 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:03:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:37 smithi017 ceph-mon[27571]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:37 smithi017 ceph-mon[27571]: Saving service mon spec with placement count:5 2024-09-17T16:03:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:37 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:37.574 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-17T16:03:37.574 INFO:teuthology.orchestra.run.smithi017.stdout:Deploying unmanaged mgr service... 2024-09-17T16:03:38.477 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:38 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:38.477 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:38 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:39.167 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-17T16:03:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:39 smithi017 ceph-mon[27571]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:39 smithi017 ceph-mon[27571]: Saving service mgr spec with placement count:2 2024-09-17T16:03:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:39 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:40.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:40 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:40.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2744711545' entity='client.admin' 2024-09-17T16:03:40.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:40 smithi017 ceph-mon[27571]: from='mgr.14118 172.21.15.17:0/1876077496' entity='mgr.y' 2024-09-17T16:03:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1807353188' entity='client.admin' 2024-09-17T16:03:42.587 INFO:teuthology.orchestra.run.smithi017.stdout:Enabling the dashboard module... 2024-09-17T16:03:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:43 smithi017 ceph-mon[27571]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:03:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3858631053' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-17T16:03:44.489 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:44 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: ignoring --setuser ceph since I am not root 2024-09-17T16:03:44.489 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:44 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: ignoring --setgroup ceph since I am not root 2024-09-17T16:03:45.427 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:45 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3858631053' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-17T16:03:45.427 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:45 smithi017 ceph-mon[27571]: mgrmap e9: y(active, since 22s) 2024-09-17T16:03:46.040 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2024-09-17T16:03:46.041 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-17T16:03:46.041 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-17T16:03:46.041 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-17T16:03:46.041 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-17T16:03:46.041 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2024-09-17T16:03:46.041 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for the mgr to restart... 2024-09-17T16:03:46.041 INFO:teuthology.orchestra.run.smithi017.stdout:Waiting for mgr epoch 9... 2024-09-17T16:03:46.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2816364004' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-17T16:03:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:49 smithi017 ceph-mon[27571]: Active manager daemon y restarted 2024-09-17T16:03:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:49 smithi017 ceph-mon[27571]: Activating manager daemon y 2024-09-17T16:03:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:49 smithi017 ceph-mon[27571]: osdmap e3: 0 total, 0 up, 0 in 2024-09-17T16:03:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:49 smithi017 ceph-mon[27571]: mgrmap e10: y(active, starting, since 0.00351249s) 2024-09-17T16:03:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:03:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-17T16:03:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-17T16:03:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-17T16:03:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-17T16:03:50.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:50 smithi017 ceph-mon[27571]: Manager daemon y is now available 2024-09-17T16:03:50.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:03:50.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-17T16:03:50.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-17T16:03:51.104 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout { 2024-09-17T16:03:51.104 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-17T16:03:51.104 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-17T16:03:51.104 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout } 2024-09-17T16:03:51.105 INFO:teuthology.orchestra.run.smithi017.stdout:mgr epoch 9 is available 2024-09-17T16:03:51.105 INFO:teuthology.orchestra.run.smithi017.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-17T16:03:51.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-mon[27571]: mgrmap e11: y(active, since 1.06066s) 2024-09-17T16:03:51.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-mon[27571]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T16:03:51.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-mon[27571]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T16:03:51.892 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: 2024-09-17T16:03:51.623+0000 7f3812f0c640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:16:03:51] ENGINE Error in HTTPServer.serve 2024-09-17T16:03:51.892 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: Traceback (most recent call last): 2024-09-17T16:03:51.892 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T16:03:51.892 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: self._connections.run(self.expiration_interval) 2024-09-17T16:03:51.892 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T16:03:51.892 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: self._run(expiration_interval) 2024-09-17T16:03:51.892 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T16:03:51.892 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T16:03:51.892 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T16:03:51.892 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T16:03:51.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T16:03:51.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: s = self.context.wrap_socket( 2024-09-17T16:03:51.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T16:03:51.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: return self.sslsocket_class._create( 2024-09-17T16:03:51.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T16:03:51.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: self.do_handshake() 2024-09-17T16:03:51.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T16:03:51.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: self._sslobj.do_handshake() 2024-09-17T16:03:51.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T16:03:51.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:03:51 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: 2024-09-17T16:03:52.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:52 smithi017 ceph-mon[27571]: [17/Sep/2024:16:03:51] ENGINE Bus STARTING 2024-09-17T16:03:52.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:52 smithi017 ceph-mon[27571]: mgrmap e12: y(active, since 2s) 2024-09-17T16:03:52.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:52 smithi017 ceph-mon[27571]: [17/Sep/2024:16:03:51] ENGINE Serving on http://172.21.15.17:8765 2024-09-17T16:03:52.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:52 smithi017 ceph-mon[27571]: [17/Sep/2024:16:03:51] ENGINE Serving on https://172.21.15.17:7150 2024-09-17T16:03:52.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:52 smithi017 ceph-mon[27571]: [17/Sep/2024:16:03:51] ENGINE Bus STARTED 2024-09-17T16:03:52.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:52 smithi017 ceph-mon[27571]: [17/Sep/2024:16:03:51] ENGINE Error in HTTPServer.serve 2024-09-17T16:03:52.738 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-09-17T16:03:52.738 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T16:03:52.738 INFO:journalctl@ceph.mon.a.smithi017.stdout: self._connections.run(self.expiration_interval) 2024-09-17T16:03:52.738 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T16:03:52.738 INFO:journalctl@ceph.mon.a.smithi017.stdout: self._run(expiration_interval) 2024-09-17T16:03:52.738 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T16:03:52.738 INFO:journalctl@ceph.mon.a.smithi017.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T16:03:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T16:03:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T16:03:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T16:03:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout: s = self.context.wrap_socket( 2024-09-17T16:03:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T16:03:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.sslsocket_class._create( 2024-09-17T16:03:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T16:03:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.do_handshake() 2024-09-17T16:03:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T16:03:52.740 INFO:journalctl@ceph.mon.a.smithi017.stdout: self._sslobj.do_handshake() 2024-09-17T16:03:52.740 INFO:journalctl@ceph.mon.a.smithi017.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T16:03:52.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:52 smithi017 ceph-mon[27571]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:52.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:03:52.997 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+zCCAuOgAwIBAgIUHhiqnkKVqnAXB2nm2VyR+zwOTNUwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkxNzE2MDM1MloXDTM0\nMDkxODE2MDM1MlowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjE3MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEApHDatMK+NBA2hKXOsqnW6kOwgNGAF3QMhW5Q\n6qmCwGPsrRBvG3tdm4KHtcx71lerOAanm5FrezKT0j6Fd9A1clWmrFQyDKU2lkSA\nvXsfja6iRyYrTWkkIRBXUPTACgQslN7BoogYhl0O+uVmih3G9X8AkpwXw//RBtoP\no4ynT3udtipU7mSAyc16f2G+VPYd66CXGIOjSXluOqIuiQY5/9WB7Zd7An3K2lve\nQXf/M0UgCN5jT65nyM0yScI8eeKGLJzjI5rtTKVg0Uo0SDDAsevct20Y4K6M9idn\nh5iWYp90wsToEDxxXvPNd5FVUiOE3UW23wf67cjoe5APQmBL3qk53jWcUzHvJO4F\n4jB41l6NEIIDwZOWihNrK3ghY2c9mxwOEz3bKak8eKnoTAtvw5txrjJLAoiryM9o\n+AuwgSFArK9HtDbqjc99zW/agOd/XNrZxVHRP3OegeGBROWqWoScZL+/l+h0CRrC\n1seN2seTZrP3nCj9Mhm3Aa8pD8AIistVV32jqp7a3OD/bZul41GdGDmEoSfcuZYq\n+QunJ24XEZUQ8OmwhsUIARv+oxQhTdfG+DtTdlqEWVpjh+fdZnrAYnp6o+Gb7ac7\n5IA3AGEnyFP0nIZZ+Ea4/IgHQ+e8mfLulaefNVSMYatjyJMHXo/fFR4weFVcnPkf\njVEItiMCAwEAAaM/MD0wLQYDVR0RBCYwJIIJc21pdGhpMDE3ghFkYXNoYm9hcmRf\nc2VydmVyc4cErBUPETAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQAX\nxK2daTgsR1ciQ75oKZxGmwD5RO/nUqU5r1v1OzARmqO4bP0lelVtlVw0Vr50bBak\nT1PrgRUk8EuwO0ZFzzadnllz6N0rS0X6HOoCDLPMNcfE1fsR3SxNKJtcCsI3NMd4\nLWad/26eJxOlArz7ywInVW9aKn6iOL2fiUEkGtPVnVNuWzCOj4tsBMIsn1RgsniT\ngv5MoGXl/BChjhCkcfUVGyhnM+hKAafFd2xv74NoiMIc6eQbbumXUYH6m+eicl1i\n0T1LqNrgG7b6uiQ4StLPCYABwRkasvPSlx5z8/x+oXlfwBJfDcJV8aGvvE3UkvNG\nip6tUS0/DUY8UynQk9TqjWGNzwIFFuvormBlJNc65KQ7rNnwpiAfVrvec0q1fNV5\ny5EFmaedbgsEGfUKjRSaVcC8lPImom2dCpjsHPU6pxm/qUn6pBsQeE3WHAfFNzEI\nM9UYQo9VNz1wt5BaC2CfHTDWEshIcVHVHw6pKCE4/YdbnCV5KSZvOsrtLIVqn5bB\n+waWw9BAwZh6SNyKb8f+JnYTY8MMQa9/hcpDGsuNgmuTkH9ps9cISAAVwvVEoJmL\nkGE2B2WK8VMovg71pEXxgrx51E+UOfrOodguebmnHX/a+I/yaQCzjdJDNnRZmWMf\ns9ERfmvN+RaHAOb/Xx1EFmL8Ju4TAgR028wTKVB3dQ==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEApHDatMK+NBA2hKXOsqnW6kOwgNGAF3QMhW5Q6qmCwGPsrRBv\nG3tdm4KHtcx71lerOAanm5FrezKT0j6Fd9A1clWmrFQyDKU2lkSAvXsfja6iRyYr\nTWkkIRBXUPTACgQslN7BoogYhl0O+uVmih3G9X8AkpwXw//RBtoPo4ynT3udtipU\n7mSAyc16f2G+VPYd66CXGIOjSXluOqIuiQY5/9WB7Zd7An3K2lveQXf/M0UgCN5j\nT65nyM0yScI8eeKGLJzjI5rtTKVg0Uo0SDDAsevct20Y4K6M9idnh5iWYp90wsTo\nEDxxXvPNd5FVUiOE3UW23wf67cjoe5APQmBL3qk53jWcUzHvJO4F4jB41l6NEIID\nwZOWihNrK3ghY2c9mxwOEz3bKak8eKnoTAtvw5txrjJLAoiryM9o+AuwgSFArK9H\ntDbqjc99zW/agOd/XNrZxVHRP3OegeGBROWqWoScZL+/l+h0CRrC1seN2seTZrP3\nnCj9Mhm3Aa8pD8AIistVV32jqp7a3OD/bZul41GdGDmEoSfcuZYq+QunJ24XEZUQ\n8OmwhsUIARv+oxQhTdfG+DtTdlqEWVpjh+fdZnrAYnp6o+Gb7ac75IA3AGEnyFP0\nnIZZ+Ea4/IgHQ+e8mfLulaefNVSMYatjyJMHXo/fFR4weFVcnPkfjVEItiMCAwEA\nAQKCAgAG6fXQHtYHc10Pc7c+nNR78g+kQuq7UbZLD+nYC/uC4lCrsCKzyZqUdMcG\n9kuIphP9XY5DhLg7rFM5xkYvVPJCVeaVnU6jaHjklYnCLcCUYPDSxACTFyYdO8Yo\nEbpXUXicY8dJcaZyU8xS8T0/9mOMo+PexjBbndBolNeOfVX1ihGF/L6obVJa6chK\nPc5WXSjB5PUlYzeE2dWwImyJd1kN8Civ2CwkXSyfxLgSEZ0Jx1DC/rK3QXJ8p2AD\naObJCySwEIKeVx++Fr/83L1ZAa/Aa1XS4oPmwoALw3lfYqOEDwdGPygwspHjt6V+\nbv5i3zpxYUKAmyHKSKW5uENj+4+FAlgMq3uo3uk2QMhg8uY++v0YX/ufn7VsTffz\nGje6hImlQFe32ODx0iZDsJcOT41ROi0l4aSXXsA8/KPZFv4KrDOb15s3HJ0DyBQM\nX+bYXeC769gOF1e7AXLFIVPiQgsRzKfhEW0bZ7dIBpeKznl7JRByTbJJ3sExS9l2\nOQSM1vB4Al8uzvNG7Ba/t9tMM9eK6WTC7nuwdWI08gsjN3P8ecT+M4lGUuYYtrnC\ntZ2bgoCvXHqoLmj4z9Rh6FSzQKNt5RyvL4ixztnfH8WPeKQgoEmxvnsJXzM6UKe0\n0rk6wy9Cie8Kg2iNOVNSB88jG8/zJ9I8Vh4iJCoPkUWnLBlo0QKCAQEA2ln6jr5C\n6keLamf7pQ/DHnM0/2wKHHU6k0pvyy9QWydU/bLwZ/TJT/2t/ONqD1d/OeG7/G0h\nkf7Q5GYfFab+yv9UYY0vL8s6XreWQVlF7aPxMyqogalwH5w/7XRhaU54iwwPfuk0\nMS65J2DLTDbZMQc+oEv3vdnBYrO1FqD0z6Z0j1rsb3rjA6Dlztt1DNnoXu6xhgij\nePtVYsXaNG9SLaV+Fk0tyF0fDbUbxgip6ATXobEzZkDvNO0N5Ukq3cDROQJj1qzv\nfOFDO0J95+M944Cy1/2vUZnl/FZpre6VzIJEX1ISq54CAQ/E+ezKmO7xfvaGX/UW\nyJFhwa0Y4yuHMQKCAQEAwMtDV09f/GTVBTUJIDnv4NHeYgeQgds2YTabDOLFB1Vs\nKhCbyfsfQoWlkuPRWD6oTRabUS7v75rUcIf1i1LLwXzwgjbyHUS64bda6IbfZbXn\n0MAS4bsr5DfnLQvb0/AgnlB8hhT6DYQMGGY8KZpl1Bawuhn1/DQ0dkfPXJUgDLys\nkc3+KH1tmAFbfdDohMlm3PjJtxiuSfha1y3RWXPDb8getfPahrFSXU0YZgzow6rx\nwoJoD3LxWxYWkxnwB8WqzpQ0yIMd1iZMb1kd7K0ERoe5uWUsFLBSl36KQ8VkWQH3\nchyWFMIDQld2ovS/LYNQ+8hZ8K6m+Ai2hGYCvIAlkwKCAQBMcXPmxXVnz+RfPEg9\nYWdOIcdMJOblT6VSK5035TicsQhwSVUhD/Hqr0VaY8Od22QPP5gwp4Q5OJypWZdI\nXU6Wgw1nmaYLdXrZ7iwMdNSmfd7TqvuTaV53spwu5a+0zmQlSfOi2sIK9fCceKaz\n5//hykeClxYEwVeS1pdvdKpAiGYGSi68WKlhe9L4uJwiA6f2lFpSMnQIxvT9v4bF\nFLQQhv+9jweYPgHBKsaubhoNwKQ9V2xHQDR+sv9KZnM7b2XQL6YEYkb10WdYN88c\nf9IK8+Oo+oeGafq6fQZCfmSzVGfbd0BSFN9ev8aWOPA02nJxoJ0quwNe9Ibk51Tv\nMy4BAoIBAGfSDeWKnQnx0rUHJm7wRDkxSyxHHo6PWe1n1n9N7wRP//t3Di3qyZ6t\nEcjUXfvDYiaBMgq52AOTwSsITDZG0+rSOUImDKOuS2UZtB7b4pmHJ9rGVGSywqQX\nZqpfZHXxLd+TQDuEOFPm9Ax/oY5V0tgvj+Pn7wttoYLAkO4ucB/rypKJ94Aea1Ic\nCQTr5wr62iXM4/NxmwnCcEjIJ6CgMnKTtdYOB3PZMZ2E0srB0Obtu1sflqIbJ9YB\nQw7cPOIylBTCwtnnob2cyjR8LR0y3saMeHtZ/C739MIYJqTwO2WjJXmR41AyWkbz\nirOeUQVVXdr4vVZa9xdCueb6eCbCa3kCggEBANeTXeLNEjAjtj85tCJ2+MSlSQ64\nUswcOjV29qsUuWEdgvJ5iPH+UN5em4MTdv/cbFyZRrS1Vm1cA2gfk+AfPdHOtYty\n+cb3rktqnaAVm6gl08/Enw369aztWmarPdvg9Y3mQhmmxE+QETv9TY9ehYSYtsBX\n4UzbOob0TC0HGTsRWzEbBt2C3SqHOrwhGDG6BnFwQdJYrarIoxDN1ZARA1U3MxqP\nPoNzyovbE7xOfWrbX/WmgyQyG4XQx5htYlSZGKjHGt7WAPB1uhllA6t6fR7OyKYp\n1twIoc6lkQ0ATzKKeiDM6W9PCTS4XPZS3lrGr6BIZnwpmbZCnZBZI30dYIU=\n-----END RSA PRIVATE KEY-----\n"} 2024-09-17T16:03:54.255 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-17T16:03:54.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:03:54.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:03:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:55 smithi017 ceph-mon[27571]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:03:55.597 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-17T16:03:55.598 INFO:teuthology.orchestra.run.smithi017.stdout:Creating initial admin user... 2024-09-17T16:03:56.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:56 smithi017 ceph-mon[27571]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:57.156 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$6X0g6M0W5moFwlJm0bCbpuwRuTLy9k1ttAfY5MrQNwweXKpm4mKX6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726589036, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-17T16:03:57.156 INFO:teuthology.orchestra.run.smithi017.stdout:Fetching dashboard port number... 2024-09-17T16:03:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:57 smithi017 ceph-mon[27571]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:03:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:03:58.379 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stdout 8443 2024-09-17T16:03:58.392 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-17T16:03:58.392 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout disabled 2024-09-17T16:03:58.401 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-17T16:03:58.401 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stdout inactive 2024-09-17T16:03:58.401 INFO:teuthology.orchestra.run.smithi017.stdout:firewalld.service is not enabled 2024-09-17T16:03:58.401 INFO:teuthology.orchestra.run.smithi017.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-17T16:03:58.403 INFO:teuthology.orchestra.run.smithi017.stdout:Ceph Dashboard is now available at: 2024-09-17T16:03:58.403 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:03:58.403 INFO:teuthology.orchestra.run.smithi017.stdout: URL: https://smithi017.front.sepia.ceph.com:8443/ 2024-09-17T16:03:58.404 INFO:teuthology.orchestra.run.smithi017.stdout: User: admin 2024-09-17T16:03:58.404 INFO:teuthology.orchestra.run.smithi017.stdout: Password: mnmcpq6k88 2024-09-17T16:03:58.404 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:03:58.404 INFO:teuthology.orchestra.run.smithi017.stdout:Saving cluster configuration to /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config directory 2024-09-17T16:03:58.685 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:58 smithi017 ceph-mon[27571]: mgrmap e13: y(active, since 8s) 2024-09-17T16:03:58.685 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:03:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1905306169' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-17T16:03:59.835 INFO:teuthology.orchestra.run.smithi017.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-17T16:03:59.835 INFO:teuthology.orchestra.run.smithi017.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-17T16:03:59.835 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:03:59.835 INFO:teuthology.orchestra.run.smithi017.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-17T16:03:59.835 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:03:59.835 INFO:teuthology.orchestra.run.smithi017.stdout:Or, if you are only running a single cluster on this host: 2024-09-17T16:03:59.835 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout: ceph telemetry on 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout:For more information see: 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:03:59.836 INFO:teuthology.orchestra.run.smithi017.stdout:Bootstrap complete. 2024-09-17T16:03:59.879 INFO:tasks.cephadm:Fetching config... 2024-09-17T16:03:59.879 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:03:59.879 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-17T16:03:59.896 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-17T16:03:59.897 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:03:59.897 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-17T16:03:59.952 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-17T16:03:59.953 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:03:59.953 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-17T16:04:00.019 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-17T16:04:00.019 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:04:00.019 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-17T16:04:00.075 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-17T16:04:00.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqgMJ9RshpiqhwrOlYkzH43FFqYDext3sAix+xKjiemxkm21uPPL2mEe19Gw+1uiObcEdYRNLcrQCezCLWpA4KqIaYFdOb6xNpjOQYAcShdqBJh6xhhZ0n5jHiyIWSuB5j9ILJQ45MeA4lCGckRw/+/4SAjfidXcUEFfojIWNv4mUQZHvISap2lHHVHanWJ2/0SBvn2qXpYdMpILzC89skcyBXAaMb0IR36ftrlcjvHy96u+DAB5SMWZyxqfZ48KlpsBjabcoW/x9O4eXNOUBRBYRqXyON3EUGybqY+8np4nApB4IYePkPoSqjAtL9gy04bmPzeEmP/o3RhbkQi4HFBBkDat8r81sSM8Asnll1F7bENWvSrTwO4QUoBsN4Yfszsv81cJQNUpKnquHCJjFHgp4QVtKFs5KohLn321voeMHp7vydJG/6r9tioIVfK+5BWEVEGJpQkeD9RjMMGuxnUYzYtNeXTUMkz96gQjzGR1U/1xGRrucOwEk+XqkXs3c= ceph-19341aec-750e-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T16:04:00.165 INFO:teuthology.orchestra.run.smithi017.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqgMJ9RshpiqhwrOlYkzH43FFqYDext3sAix+xKjiemxkm21uPPL2mEe19Gw+1uiObcEdYRNLcrQCezCLWpA4KqIaYFdOb6xNpjOQYAcShdqBJh6xhhZ0n5jHiyIWSuB5j9ILJQ45MeA4lCGckRw/+/4SAjfidXcUEFfojIWNv4mUQZHvISap2lHHVHanWJ2/0SBvn2qXpYdMpILzC89skcyBXAaMb0IR36ftrlcjvHy96u+DAB5SMWZyxqfZ48KlpsBjabcoW/x9O4eXNOUBRBYRqXyON3EUGybqY+8np4nApB4IYePkPoSqjAtL9gy04bmPzeEmP/o3RhbkQi4HFBBkDat8r81sSM8Asnll1F7bENWvSrTwO4QUoBsN4Yfszsv81cJQNUpKnquHCJjFHgp4QVtKFs5KohLn321voeMHp7vydJG/6r9tioIVfK+5BWEVEGJpQkeD9RjMMGuxnUYzYtNeXTUMkz96gQjzGR1U/1xGRrucOwEk+XqkXs3c= ceph-19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:00.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqgMJ9RshpiqhwrOlYkzH43FFqYDext3sAix+xKjiemxkm21uPPL2mEe19Gw+1uiObcEdYRNLcrQCezCLWpA4KqIaYFdOb6xNpjOQYAcShdqBJh6xhhZ0n5jHiyIWSuB5j9ILJQ45MeA4lCGckRw/+/4SAjfidXcUEFfojIWNv4mUQZHvISap2lHHVHanWJ2/0SBvn2qXpYdMpILzC89skcyBXAaMb0IR36ftrlcjvHy96u+DAB5SMWZyxqfZ48KlpsBjabcoW/x9O4eXNOUBRBYRqXyON3EUGybqY+8np4nApB4IYePkPoSqjAtL9gy04bmPzeEmP/o3RhbkQi4HFBBkDat8r81sSM8Asnll1F7bENWvSrTwO4QUoBsN4Yfszsv81cJQNUpKnquHCJjFHgp4QVtKFs5KohLn321voeMHp7vydJG/6r9tioIVfK+5BWEVEGJpQkeD9RjMMGuxnUYzYtNeXTUMkz96gQjzGR1U/1xGRrucOwEk+XqkXs3c= ceph-19341aec-750e-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T16:04:00.225 INFO:teuthology.orchestra.run.smithi071.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqgMJ9RshpiqhwrOlYkzH43FFqYDext3sAix+xKjiemxkm21uPPL2mEe19Gw+1uiObcEdYRNLcrQCezCLWpA4KqIaYFdOb6xNpjOQYAcShdqBJh6xhhZ0n5jHiyIWSuB5j9ILJQ45MeA4lCGckRw/+/4SAjfidXcUEFfojIWNv4mUQZHvISap2lHHVHanWJ2/0SBvn2qXpYdMpILzC89skcyBXAaMb0IR36ftrlcjvHy96u+DAB5SMWZyxqfZ48KlpsBjabcoW/x9O4eXNOUBRBYRqXyON3EUGybqY+8np4nApB4IYePkPoSqjAtL9gy04bmPzeEmP/o3RhbkQi4HFBBkDat8r81sSM8Asnll1F7bENWvSrTwO4QUoBsN4Yfszsv81cJQNUpKnquHCJjFHgp4QVtKFs5KohLn321voeMHp7vydJG/6r9tioIVfK+5BWEVEGJpQkeD9RjMMGuxnUYzYtNeXTUMkz96gQjzGR1U/1xGRrucOwEk+XqkXs3c= ceph-19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:00.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqgMJ9RshpiqhwrOlYkzH43FFqYDext3sAix+xKjiemxkm21uPPL2mEe19Gw+1uiObcEdYRNLcrQCezCLWpA4KqIaYFdOb6xNpjOQYAcShdqBJh6xhhZ0n5jHiyIWSuB5j9ILJQ45MeA4lCGckRw/+/4SAjfidXcUEFfojIWNv4mUQZHvISap2lHHVHanWJ2/0SBvn2qXpYdMpILzC89skcyBXAaMb0IR36ftrlcjvHy96u+DAB5SMWZyxqfZ48KlpsBjabcoW/x9O4eXNOUBRBYRqXyON3EUGybqY+8np4nApB4IYePkPoSqjAtL9gy04bmPzeEmP/o3RhbkQi4HFBBkDat8r81sSM8Asnll1F7bENWvSrTwO4QUoBsN4Yfszsv81cJQNUpKnquHCJjFHgp4QVtKFs5KohLn321voeMHp7vydJG/6r9tioIVfK+5BWEVEGJpQkeD9RjMMGuxnUYzYtNeXTUMkz96gQjzGR1U/1xGRrucOwEk+XqkXs3c= ceph-19341aec-750e-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T16:04:00.335 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqgMJ9RshpiqhwrOlYkzH43FFqYDext3sAix+xKjiemxkm21uPPL2mEe19Gw+1uiObcEdYRNLcrQCezCLWpA4KqIaYFdOb6xNpjOQYAcShdqBJh6xhhZ0n5jHiyIWSuB5j9ILJQ45MeA4lCGckRw/+/4SAjfidXcUEFfojIWNv4mUQZHvISap2lHHVHanWJ2/0SBvn2qXpYdMpILzC89skcyBXAaMb0IR36ftrlcjvHy96u+DAB5SMWZyxqfZ48KlpsBjabcoW/x9O4eXNOUBRBYRqXyON3EUGybqY+8np4nApB4IYePkPoSqjAtL9gy04bmPzeEmP/o3RhbkQi4HFBBkDat8r81sSM8Asnll1F7bENWvSrTwO4QUoBsN4Yfszsv81cJQNUpKnquHCJjFHgp4QVtKFs5KohLn321voeMHp7vydJG/6r9tioIVfK+5BWEVEGJpQkeD9RjMMGuxnUYzYtNeXTUMkz96gQjzGR1U/1xGRrucOwEk+XqkXs3c= ceph-19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:00.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-17T16:04:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1263104217' entity='client.admin' 2024-09-17T16:04:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:04:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi017", "name": "osd_memory_target"} : dispatch 2024-09-17T16:04:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:02.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:02.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2348406875' entity='client.admin' 2024-09-17T16:04:02.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:02.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:02.779 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-17T16:04:02.779 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-17T16:04:03.477 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:04:04.925 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi071 2024-09-17T16:04:04.925 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T16:04:04.925 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/etc/ceph/ceph.conf 2024-09-17T16:04:04.944 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T16:04:04.945 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T16:04:05.003 INFO:tasks.cephadm:Adding host smithi071 to orchestrator... 2024-09-17T16:04:05.003 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch host add smithi071 2024-09-17T16:04:05.255 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:04:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:05 smithi017 ceph-mon[27571]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:05 smithi017 ceph-mon[27571]: Updating smithi017:/etc/ceph/ceph.conf 2024-09-17T16:04:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:06.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:06 smithi017 ceph-mon[27571]: Updating smithi017:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:06.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:06 smithi017 ceph-mon[27571]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-09-17T16:04:06.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:06 smithi017 ceph-mon[27571]: Updating smithi017:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T16:04:07.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:07 smithi017 ceph-mon[27571]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi071", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:08.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:08 smithi017 ceph-mon[27571]: Deploying cephadm binary to smithi071 2024-09-17T16:04:09.717 INFO:teuthology.orchestra.run.smithi017.stdout:Added host 'smithi071' with addr '172.21.15.71' 2024-09-17T16:04:10.479 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-17T16:04:10.736 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:04:10.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:10 smithi017 ceph-mon[27571]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:10.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:10.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:10 smithi017 ceph-mon[27571]: Added host smithi071 2024-09-17T16:04:10.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:10.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:10.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:11.393 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:04:11.396 INFO:teuthology.orchestra.run.smithi017.stdout:[{"addr": "172.21.15.17", "hostname": "smithi017", "labels": [], "status": ""}, {"addr": "172.21.15.71", "hostname": "smithi071", "labels": [], "status": ""}] 2024-09-17T16:04:12.023 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-09-17T16:04:12.023 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T16:04:12.023 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-09-17T16:04:12.044 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T16:04:12.044 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T16:04:12.101 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-09-17T16:04:12.101 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch host add smithi173 2024-09-17T16:04:12.357 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:04:12.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:12 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:13 smithi017 ceph-mon[27571]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:04:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:13 smithi017 ceph-mon[27571]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:14 smithi017 ceph-mon[27571]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"} : dispatch 2024-09-17T16:04:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:15 smithi017 ceph-mon[27571]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:15 smithi017 ceph-mon[27571]: Updating smithi071:/etc/ceph/ceph.conf 2024-09-17T16:04:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:15 smithi017 ceph-mon[27571]: Updating smithi071:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:15 smithi017 ceph-mon[27571]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-09-17T16:04:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:15 smithi017 ceph-mon[27571]: Updating smithi071:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T16:04:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:15 smithi017 ceph-mon[27571]: Deploying cephadm binary to smithi173 2024-09-17T16:04:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:15 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:15 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:15 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:16.789 INFO:teuthology.orchestra.run.smithi017.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-09-17T16:04:17.398 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:17 smithi017 ceph-mon[27571]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:17.398 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:17.398 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:17.398 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:17.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-17T16:04:17.684 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:04:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:04:18.345 INFO:teuthology.orchestra.run.smithi017.stdout:[{"addr": "172.21.15.17", "hostname": "smithi017", "labels": [], "status": ""}, {"addr": "172.21.15.71", "hostname": "smithi071", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-09-17T16:04:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:18 smithi017 ceph-mon[27571]: Added host smithi173 2024-09-17T16:04:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:18.958 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-17T16:04:18.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-17T16:04:19.206 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:04:19.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:19 smithi017 ceph-mon[27571]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:20.230 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2024-09-17T16:04:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:20 smithi017 ceph-mon[27571]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:04:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1866537771' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-17T16:04:20.824 INFO:tasks.cephadm:Adding mon.a on smithi017 2024-09-17T16:04:20.825 INFO:tasks.cephadm:Adding mon.b on smithi071 2024-09-17T16:04:20.825 INFO:tasks.cephadm:Adding mon.c on smithi173 2024-09-17T16:04:20.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi017:172.21.15.17=a;smithi071:172.21.15.71=b;smithi173:172.21.15.173=c' 2024-09-17T16:04:21.073 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:21 smithi017 ceph-mon[27571]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1866537771' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-17T16:04:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:21 smithi017 ceph-mon[27571]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"} : dispatch 2024-09-17T16:04:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:21.761 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-09-17T16:04:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T16:04:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: Updating smithi173:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T16:04:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: Updating smithi173:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T16:04:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:22.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:22.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:22.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:22.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:22.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:22.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:22.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:22.521 DEBUG:teuthology.orchestra.run.smithi071:mon.b> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.b.service 2024-09-17T16:04:22.523 DEBUG:teuthology.orchestra.run.smithi173:mon.c> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.c.service 2024-09-17T16:04:22.526 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T16:04:22.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T16:04:23.157 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:23 smithi017 ceph-mon[27571]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:23 smithi017 ceph-mon[27571]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi017:172.21.15.17=a;smithi071:172.21.15.71=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:23 smithi017 ceph-mon[27571]: Saving service mon spec with placement smithi017:172.21.15.17=a;smithi071:172.21.15.71=b;smithi173:172.21.15.173=c;count:3 2024-09-17T16:04:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:23 smithi017 ceph-mon[27571]: Deploying daemon mon.c on smithi173 2024-09-17T16:04:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:04:25.330 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"19341aec-750e-11ef-bceb-c7b262605968","modified":"2024-09-17T16:02:44.038562Z","created":"2024-09-17T16:02:44.038562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-17T16:04:25.330 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-09-17T16:04:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:25 smithi017 ceph-mon[27571]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/382290490' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:04:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:26 smithi173 systemd[1]: Started Ceph mon.c for 19341aec-750e-11ef-bceb-c7b262605968. 2024-09-17T16:04:26.970 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T16:04:26.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T16:04:27.221 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:04:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: mon.a calling monitor election 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: mon.c calling monitor election 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: monmap epoch 2 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: last_changed 2024-09-17T16:04:26.359932+0000 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: created 2024-09-17T16:02:44.038562+0000 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: min_mon_release 19 (squid) 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: election_strategy: 1 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: fsmap 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: mgrmap e13: y(active, since 41s) 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: overall HEALTH_OK 2024-09-17T16:04:31.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:31.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:31.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:31.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:31.742 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: mon.a calling monitor election 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: mon.c calling monitor election 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: monmap epoch 2 2024-09-17T16:04:31.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: last_changed 2024-09-17T16:04:26.359932+0000 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: created 2024-09-17T16:02:44.038562+0000 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: min_mon_release 19 (squid) 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: election_strategy: 1 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: fsmap 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: mgrmap e13: y(active, since 41s) 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: overall HEALTH_OK 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:31.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:31.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:32.369 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:04:32.369 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"19341aec-750e-11ef-bceb-c7b262605968","modified":"2024-09-17T16:04:26.359932Z","created":"2024-09-17T16:02:44.038562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T16:04:32.369 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-09-17T16:04:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:32 smithi173 ceph-mon[30530]: Deploying daemon mon.b on smithi071 2024-09-17T16:04:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:32 smithi173 ceph-mon[30530]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/841986828' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:04:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:32 smithi017 ceph-mon[27571]: Deploying daemon mon.b on smithi071 2024-09-17T16:04:32.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:32 smithi017 ceph-mon[27571]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:32.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:32.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/841986828' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:04:33.739 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:04:33 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: 2024-09-17T16:04:33.363+0000 7f384c14c640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T16:04:33.966 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T16:04:33.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T16:04:34.219 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:04:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:04:34.883 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"19341aec-750e-11ef-bceb-c7b262605968","modified":"2024-09-17T16:04:26.359932Z","created":"2024-09-17T16:02:44.038562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T16:04:34.884 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-09-17T16:04:34.884 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:34 smithi173 ceph-mon[30530]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:34.965 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: load: jerasure load: lrc 2024-09-17T16:04:34.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:34 smithi017 ceph-mon[27571]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: RocksDB version: 7.9.2 2024-09-17T16:04:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Git sha 0 2024-09-17T16:04:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Compile date 2024-09-17 09:38:39 2024-09-17T16:04:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: DB SUMMARY 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: DB Session ID: F8IDXQQECQWYPJ9K5F36 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: CURRENT file: CURRENT 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.error_if_exists: 0 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.create_if_missing: 0 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.paranoid_checks: 1 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T16:04:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.env: 0x56505d51e0e0 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.info_log: 0x56505fc33700 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.statistics: (nil) 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.use_fsync: 0 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_log_file_size: 0 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.allow_fallocate: 1 2024-09-17T16:04:35.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T16:04:35.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T16:04:35.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.use_direct_reads: 0 2024-09-17T16:04:35.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T16:04:35.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T16:04:35.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.db_log_dir: 2024-09-17T16:04:35.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.wal_dir: 2024-09-17T16:04:35.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T16:04:35.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T16:04:35.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T16:04:35.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T16:04:35.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T16:04:35.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T16:04:35.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T16:04:35.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T16:04:35.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.write_buffer_manager: 0x56505fc37b80 2024-09-17T16:04:35.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T16:04:35.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T16:04:35.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T16:04:35.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T16:04:35.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T16:04:35.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T16:04:35.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T16:04:35.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T16:04:35.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.unordered_write: 0 2024-09-17T16:04:35.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T16:04:35.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T16:04:35.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T16:04:35.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T16:04:35.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.row_cache: None 2024-09-17T16:04:35.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.wal_filter: None 2024-09-17T16:04:35.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T16:04:35.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T16:04:35.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.two_write_queues: 0 2024-09-17T16:04:35.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T16:04:35.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.wal_compression: 0 2024-09-17T16:04:35.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.atomic_flush: 0 2024-09-17T16:04:35.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T16:04:35.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T16:04:35.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T16:04:35.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.log_readahead_size: 0 2024-09-17T16:04:35.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T16:04:35.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T16:04:35.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T16:04:35.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T16:04:35.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T16:04:35.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T16:04:35.324 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T16:04:35.324 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_background_jobs: 2 2024-09-17T16:04:35.324 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_background_compactions: -1 2024-09-17T16:04:35.324 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_subcompactions: 1 2024-09-17T16:04:35.324 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T16:04:35.324 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T16:04:35.324 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T16:04:35.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T16:04:35.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T16:04:35.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T16:04:35.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T16:04:35.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T16:04:35.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_open_files: -1 2024-09-17T16:04:35.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T16:04:35.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T16:04:35.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T16:04:35.326 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T16:04:35.326 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_background_flushes: -1 2024-09-17T16:04:35.326 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Compression algorithms supported: 2024-09-17T16:04:35.326 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: kZSTD supported: 0 2024-09-17T16:04:35.326 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: kXpressCompression supported: 0 2024-09-17T16:04:35.326 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: kBZip2Compression supported: 0 2024-09-17T16:04:35.326 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T16:04:35.326 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: kLZ4Compression supported: 1 2024-09-17T16:04:35.326 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: kZlibCompression supported: 1 2024-09-17T16:04:35.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T16:04:35.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: kSnappyCompression supported: 1 2024-09-17T16:04:35.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T16:04:35.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T16:04:35.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-17T16:04:35.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T16:04:35.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T16:04:35.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.merge_operator: 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_filter: None 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56505fc33320) 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout: cache_index_and_filter_blocks: 1 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout: pin_top_level_index_and_filter: 1 2024-09-17T16:04:35.328 INFO:journalctl@ceph.mon.b.smithi071.stdout: index_type: 0 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: data_block_index_type: 0 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: index_shortening: 1 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: checksum: 4 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: no_block_cache: 0 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache: 0x56505fc27610 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache_name: BinnedLRUCache 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache_options: 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: capacity : 536870912 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: num_shard_bits : 4 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: strict_capacity_limit : 0 2024-09-17T16:04:35.329 INFO:journalctl@ceph.mon.b.smithi071.stdout: high_pri_pool_ratio: 0.000 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache_compressed: (nil) 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: persistent_cache: (nil) 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_size: 4096 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_size_deviation: 10 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_restart_interval: 16 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: index_block_restart_interval: 1 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: metadata_block_size: 4096 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: partition_filters: 0 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: use_delta_encoding: 1 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: filter_policy: bloomfilter 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: whole_key_filtering: 1 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: verify_compression: 0 2024-09-17T16:04:35.330 INFO:journalctl@ceph.mon.b.smithi071.stdout: read_amp_bytes_per_bit: 0 2024-09-17T16:04:35.331 INFO:journalctl@ceph.mon.b.smithi071.stdout: format_version: 5 2024-09-17T16:04:35.331 INFO:journalctl@ceph.mon.b.smithi071.stdout: enable_index_compression: 1 2024-09-17T16:04:35.331 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_align: 0 2024-09-17T16:04:35.331 INFO:journalctl@ceph.mon.b.smithi071.stdout: max_auto_readahead_size: 262144 2024-09-17T16:04:35.331 INFO:journalctl@ceph.mon.b.smithi071.stdout: prepopulate_block_cache: 0 2024-09-17T16:04:35.331 INFO:journalctl@ceph.mon.b.smithi071.stdout: initial_auto_readahead_size: 8192 2024-09-17T16:04:35.331 INFO:journalctl@ceph.mon.b.smithi071.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T16:04:35.331 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T16:04:35.331 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T16:04:35.332 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compression: NoCompression 2024-09-17T16:04:35.332 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T16:04:35.332 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T16:04:35.332 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T16:04:35.332 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.num_levels: 7 2024-09-17T16:04:35.332 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T16:04:35.332 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T16:04:35.333 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T16:04:35.333 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T16:04:35.333 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T16:04:35.333 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T16:04:35.333 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T16:04:35.333 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T16:04:35.333 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T16:04:35.333 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T16:04:35.333 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T16:04:35.334 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T16:04:35.334 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T16:04:35.334 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T16:04:35.334 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T16:04:35.334 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T16:04:35.334 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T16:04:35.334 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T16:04:35.334 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T16:04:35.335 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T16:04:35.335 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T16:04:35.335 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T16:04:35.335 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T16:04:35.335 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T16:04:35.335 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T16:04:35.335 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T16:04:35.335 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T16:04:35.335 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T16:04:35.336 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T16:04:35.336 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T16:04:35.336 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T16:04:35.336 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T16:04:35.336 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T16:04:35.336 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T16:04:35.336 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T16:04:35.336 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T16:04:35.337 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T16:04:35.337 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T16:04:35.337 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T16:04:35.337 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T16:04:35.337 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T16:04:35.337 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T16:04:35.337 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T16:04:35.338 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T16:04:35.338 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T16:04:35.338 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T16:04:35.338 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T16:04:35.338 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T16:04:35.338 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T16:04:35.338 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T16:04:35.339 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T16:04:35.339 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T16:04:35.339 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T16:04:35.339 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T16:04:35.339 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.inplace_update_support: 0 2024-09-17T16:04:35.339 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T16:04:35.339 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T16:04:35.339 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T16:04:35.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T16:04:35.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.bloom_locality: 0 2024-09-17T16:04:35.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.max_successive_merges: 0 2024-09-17T16:04:35.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T16:04:35.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T16:04:35.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T16:04:35.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T16:04:35.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.ttl: 2592000 2024-09-17T16:04:35.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T16:04:35.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T16:04:35.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T16:04:35.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.enable_blob_files: false 2024-09-17T16:04:35.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.min_blob_size: 0 2024-09-17T16:04:35.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T16:04:35.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T16:04:35.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T16:04:35.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T16:04:35.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T16:04:35.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T16:04:35.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T16:04:35.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T16:04:35.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T16:04:35.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-17T16:04:35.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 541e783c-7ba0-4ece-bb26-a9bba7efa674 2024-09-17T16:04:35.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726589074965911, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-17T16:04:35.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-17T16:04:35.343 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726589074966309, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726589074, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "541e783c-7ba0-4ece-bb26-a9bba7efa674", "db_session_id": "F8IDXQQECQWYPJ9K5F36", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-17T16:04:35.343 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726589074966401, "job": 1, "event": "recovery_finished"} 2024-09-17T16:04:35.343 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-17T16:04:35.343 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T16:04:35.343 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56505fc56e00 2024-09-17T16:04:35.343 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: DB pointer 0x56505fd58000 2024-09-17T16:04:35.343 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T16:04:35.343 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T16:04:35.344 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** DB Stats ** 2024-09-17T16:04:35.344 INFO:journalctl@ceph.mon.b.smithi071.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T16:04:35.344 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T16:04:35.344 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T16:04:35.344 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T16:04:35.344 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T16:04:35.344 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T16:04:35.344 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T16:04:35.345 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-09-17T16:04:35.345 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** Compaction Stats [default] ** 2024-09-17T16:04:35.345 INFO:journalctl@ceph.mon.b.smithi071.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T16:04:35.345 INFO:journalctl@ceph.mon.b.smithi071.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T16:04:35.345 INFO:journalctl@ceph.mon.b.smithi071.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T16:04:35.345 INFO:journalctl@ceph.mon.b.smithi071.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T16:04:35.345 INFO:journalctl@ceph.mon.b.smithi071.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T16:04:35.345 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-09-17T16:04:35.345 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** Compaction Stats [default] ** 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T16:04:35.346 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout: Block cache BinnedLRUCache@0x56505fc27610#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: using public_addr v2:172.21.15.71:0/0 -> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: starting mon.b rank -1 at public addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] at bind addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:35.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(???) e0 preinit fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:35.348 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-09-17T16:04:35.348 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: (Original Log Time 2024/09/17-16:04:34.989982) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-09-17T16:04:35.348 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-09-17T16:04:35.348 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726589074990091, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1149, "num_deletes": 11, "total_data_size": 2502485, "memory_usage": 2528160, "flush_reason": "Manual Compaction"} 2024-09-17T16:04:35.348 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-09-17T16:04:35.348 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).mds e1 new map 2024-09-17T16:04:35.348 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-17T16:04:35.348 INFO:journalctl@ceph.mon.b.smithi071.stdout: e1 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout: btime 2024-09-17T16:02:46:978559+0000 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout: legacy client fscid: -1 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout: No filesystems configured 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-17T16:04:35.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"} : dispatch 2024-09-17T16:04:35.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Updating smithi071:/etc/ceph/ceph.conf 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Updating smithi071:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Updating smithi071:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Deploying cephadm binary to smithi173 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Added host smithi173 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1866537771' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1866537771' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"} : dispatch 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Updating smithi173:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Updating smithi173:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.353 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi017:172.21.15.17=a;smithi071:172.21.15.71=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:35.354 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Saving service mon spec with placement smithi017:172.21.15.17=a;smithi071:172.21.15.71=b;smithi173:172.21.15.173=c;count:3 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Deploying daemon mon.c on smithi173 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/382290490' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.a calling monitor election 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.c calling monitor election 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.355 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:35.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:35.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T16:04:35.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: monmap epoch 2 2024-09-17T16:04:35.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:35.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: last_changed 2024-09-17T16:04:26.359932+0000 2024-09-17T16:04:35.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: created 2024-09-17T16:02:44.038562+0000 2024-09-17T16:04:35.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: min_mon_release 19 (squid) 2024-09-17T16:04:35.356 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: election_strategy: 1 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: fsmap 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mgrmap e13: y(active, since 41s) 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: overall HEALTH_OK 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:35.357 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:35.358 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: Deploying daemon mon.b on smithi071 2024-09-17T16:04:35.358 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.358 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:35.358 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/841986828' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:04:35.358 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:35.358 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:34 smithi071 ceph-mon[30002]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-17T16:04:35.358 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:35 smithi071 bash[29982]: e490eba52e0029a9da5a238aa2dbfb5a29a925d47ca6aa5193017f561bf9aa8d 2024-09-17T16:04:35.358 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:35 smithi071 systemd[1]: Started Ceph mon.b for 19341aec-750e-11ef-bceb-c7b262605968. 2024-09-17T16:04:36.485 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T16:04:36.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T16:04:36.738 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:04:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: mon.a calling monitor election 2024-09-17T16:04:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: mon.c calling monitor election 2024-09-17T16:04:40.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:04:40.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:40.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:40.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:40.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T16:04:40.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: monmap epoch 3 2024-09-17T16:04:40.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:40.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: last_changed 2024-09-17T16:04:34.995432+0000 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: created 2024-09-17T16:02:44.038562+0000 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: min_mon_release 19 (squid) 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: election_strategy: 1 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: fsmap 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: mgrmap e13: y(active, since 50s) 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-17T16:04:40.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T16:04:40.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: mon.b (rank 2) addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] is down (out of quorum) 2024-09-17T16:04:40.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:40.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:40.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:40.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:40.432 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:04:40.433 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"19341aec-750e-11ef-bceb-c7b262605968","modified":"2024-09-17T16:04:34.995432Z","created":"2024-09-17T16:02:44.038562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T16:04:40.433 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-09-17T16:04:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: mon.a calling monitor election 2024-09-17T16:04:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: mon.c calling monitor election 2024-09-17T16:04:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:04:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: monmap epoch 3 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: last_changed 2024-09-17T16:04:34.995432+0000 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: created 2024-09-17T16:02:44.038562+0000 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: min_mon_release 19 (squid) 2024-09-17T16:04:40.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: election_strategy: 1 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: fsmap 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: mgrmap e13: y(active, since 50s) 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: mon.b (rank 2) addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] is down (out of quorum) 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:40.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:41.150 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-17T16:04:41.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-17T16:04:41.374 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:41 smithi173 ceph-mon[30530]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:41.374 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:41.374 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:41.374 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/1797413217' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:04:41.374 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:41.417 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:04:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:41 smithi017 ceph-mon[27571]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/1797413217' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:04:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:42.133 INFO:teuthology.orchestra.run.smithi017.stdout:# minimal ceph.conf for 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:42.133 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2024-09-17T16:04:42.133 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:42.134 INFO:teuthology.orchestra.run.smithi017.stdout: mon_host = [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-09-17T16:04:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mon.a calling monitor election 2024-09-17T16:04:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mon.c calling monitor election 2024-09-17T16:04:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:04:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:04:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: monmap epoch 3 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: last_changed 2024-09-17T16:04:34.995432+0000 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: created 2024-09-17T16:02:44.038562+0000 2024-09-17T16:04:42.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: min_mon_release 19 (squid) 2024-09-17T16:04:42.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: election_strategy: 1 2024-09-17T16:04:42.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-17T16:04:42.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-09-17T16:04:42.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-17T16:04:42.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: fsmap 2024-09-17T16:04:42.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:42.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mgrmap e13: y(active, since 50s) 2024-09-17T16:04:42.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-17T16:04:42.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mon.b (rank 2) addr [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] is down (out of quorum) 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/1797413217' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:04:42.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:42.788 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-17T16:04:42.788 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:04:42.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T16:04:42.816 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:04:42.816 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T16:04:42.884 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T16:04:42.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T16:04:42.914 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T16:04:42.914 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T16:04:42.981 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T16:04:42.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T16:04:43.014 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T16:04:43.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T16:04:43.070 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: mon.b calling monitor election 2024-09-17T16:04:43.070 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: Updating smithi017:/etc/ceph/ceph.conf 2024-09-17T16:04:43.070 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: Updating smithi071:/etc/ceph/ceph.conf 2024-09-17T16:04:43.070 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T16:04:43.070 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:43.070 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: Updating smithi173:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:43.070 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: Updating smithi071:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:43.070 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: Updating smithi017:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:43.070 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: Reconfiguring daemon mon.a on smithi017 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: mon.b calling monitor election 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: mon.c calling monitor election 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: mon.a calling monitor election 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: monmap epoch 3 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: last_changed 2024-09-17T16:04:34.995432+0000 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: created 2024-09-17T16:02:44.038562+0000 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: min_mon_release 19 (squid) 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: election_strategy: 1 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-17T16:04:43.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-09-17T16:04:43.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-17T16:04:43.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: fsmap 2024-09-17T16:04:43.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:43.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: mgrmap e13: y(active, since 52s) 2024-09-17T16:04:43.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T16:04:43.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:04:43.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: overall HEALTH_OK 2024-09-17T16:04:43.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1897676073' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:43.084 INFO:tasks.cephadm:Adding mgr.y on smithi017 2024-09-17T16:04:43.084 INFO:tasks.cephadm:Adding mgr.x on smithi071 2024-09-17T16:04:43.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi017=y;smithi071=x' 2024-09-17T16:04:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mon.b calling monitor election 2024-09-17T16:04:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Updating smithi017:/etc/ceph/ceph.conf 2024-09-17T16:04:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Updating smithi071:/etc/ceph/ceph.conf 2024-09-17T16:04:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T16:04:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Updating smithi173:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Updating smithi071:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Updating smithi017:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Reconfiguring daemon mon.a on smithi017 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mon.b calling monitor election 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mon.c calling monitor election 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mon.a calling monitor election 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: monmap epoch 3 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: last_changed 2024-09-17T16:04:34.995432+0000 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: created 2024-09-17T16:02:44.038562+0000 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: min_mon_release 19 (squid) 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: election_strategy: 1 2024-09-17T16:04:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-17T16:04:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-09-17T16:04:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-17T16:04:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: fsmap 2024-09-17T16:04:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: mgrmap e13: y(active, since 52s) 2024-09-17T16:04:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T16:04:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:04:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: overall HEALTH_OK 2024-09-17T16:04:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1897676073' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:43.368 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:04:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: mon.b calling monitor election 2024-09-17T16:04:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: Updating smithi017:/etc/ceph/ceph.conf 2024-09-17T16:04:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: Updating smithi071:/etc/ceph/ceph.conf 2024-09-17T16:04:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: Updating smithi173:/etc/ceph/ceph.conf 2024-09-17T16:04:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: Updating smithi173:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: Updating smithi071:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: Updating smithi017:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T16:04:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T16:04:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: Reconfiguring daemon mon.a on smithi017 2024-09-17T16:04:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: mon.b calling monitor election 2024-09-17T16:04:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: mon.c calling monitor election 2024-09-17T16:04:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: mon.a calling monitor election 2024-09-17T16:04:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T16:04:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: monmap epoch 3 2024-09-17T16:04:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:04:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: last_changed 2024-09-17T16:04:34.995432+0000 2024-09-17T16:04:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: created 2024-09-17T16:02:44.038562+0000 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: min_mon_release 19 (squid) 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: election_strategy: 1 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: 0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: fsmap 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: mgrmap e13: y(active, since 52s) 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:04:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: overall HEALTH_OK 2024-09-17T16:04:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1897676073' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:43.998 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-09-17T16:04:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:43 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:43 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:44.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:43 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:44.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:43 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:44.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:43 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:04:44.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:43 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:44.526 DEBUG:teuthology.orchestra.run.smithi071:mgr.x> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@mgr.x.service 2024-09-17T16:04:44.529 DEBUG:tasks.cephadm:set 0 configs 2024-09-17T16:04:44.529 INFO:tasks.cephadm:Deploying OSDs... 2024-09-17T16:04:44.529 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:04:44.529 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T16:04:44.550 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T16:04:44.550 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2024-09-17T16:04:44.610 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T16:04:44.610 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:44.610 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-17T16:04:44.610 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:44.610 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:44.610 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-17 16:04:02.677327002 +0000 2024-09-17T16:04:44.610 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-17 15:59:23.879140204 +0000 2024-09-17T16:04:44.610 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-17 15:59:23.879140204 +0000 2024-09-17T16:04:44.610 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-17 15:59:23.879140204 +0000 2024-09-17T16:04:44.611 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T16:04:44.680 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-17T16:04:44.680 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-17T16:04:44.681 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000130766 s, 3.9 MB/s 2024-09-17T16:04:44.682 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T16:04:44.742 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2024-09-17T16:04:44.801 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T16:04:44.801 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:44.801 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-17T16:04:44.801 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:44.801 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:44.801 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-17 16:04:02.710326613 +0000 2024-09-17T16:04:44.801 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-17 15:59:24.201136210 +0000 2024-09-17T16:04:44.801 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-17 15:59:24.201136210 +0000 2024-09-17T16:04:44.802 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-17 15:59:24.201136210 +0000 2024-09-17T16:04:44.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T16:04:44.870 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-17T16:04:44.870 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-17T16:04:44.870 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000130081 s, 3.9 MB/s 2024-09-17T16:04:44.871 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T16:04:44.931 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2024-09-17T16:04:44.990 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T16:04:44.990 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:44.990 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-17T16:04:44.990 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:44.990 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:44.991 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-17 16:04:02.743326226 +0000 2024-09-17T16:04:44.991 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-17 15:59:24.532132104 +0000 2024-09-17T16:04:44.991 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-17 15:59:24.532132104 +0000 2024-09-17T16:04:44.991 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-17 15:59:24.532132104 +0000 2024-09-17T16:04:44.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T16:04:45.058 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-17T16:04:45.059 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-17T16:04:45.059 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000122075 s, 4.2 MB/s 2024-09-17T16:04:45.060 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T16:04:45.119 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2024-09-17T16:04:45.177 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T16:04:45.177 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:45.177 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-17T16:04:45.177 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:45.177 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:45.177 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-17 16:04:02.777325826 +0000 2024-09-17T16:04:45.177 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-17 15:59:24.856128085 +0000 2024-09-17T16:04:45.177 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-17 15:59:24.856128085 +0000 2024-09-17T16:04:45.177 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-17 15:59:24.856128085 +0000 2024-09-17T16:04:45.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T16:04:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:44 smithi017 ceph-mon[27571]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:44 smithi017 ceph-mon[27571]: Reconfiguring mon.b (monmap changed)... 2024-09-17T16:04:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T16:04:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:44 smithi017 ceph-mon[27571]: Reconfiguring daemon mon.b on smithi071 2024-09-17T16:04:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:44 smithi017 ceph-mon[27571]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi071=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:45.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:44 smithi017 ceph-mon[27571]: Saving service mgr spec with placement smithi017=y;smithi071=x;count:2 2024-09-17T16:04:45.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:45.245 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-17T16:04:45.245 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-17T16:04:45.245 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000111485 s, 4.6 MB/s 2024-09-17T16:04:45.246 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T16:04:45.304 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T16:04:45.304 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T16:04:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:44 smithi071 ceph-mon[30002]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:44 smithi071 ceph-mon[30002]: Reconfiguring mon.b (monmap changed)... 2024-09-17T16:04:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T16:04:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:44 smithi071 ceph-mon[30002]: Reconfiguring daemon mon.b on smithi071 2024-09-17T16:04:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:44 smithi071 ceph-mon[30002]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi071=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:44 smithi071 ceph-mon[30002]: Saving service mgr spec with placement smithi017=y;smithi071=x;count:2 2024-09-17T16:04:45.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:45.321 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T16:04:45.321 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-09-17T16:04:45.377 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T16:04:45.377 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:45.377 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-17T16:04:45.377 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:45.377 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:45.377 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-17 16:04:34.422783214 +0000 2024-09-17T16:04:45.377 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-17 15:59:20.266685877 +0000 2024-09-17T16:04:45.377 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-17 15:59:20.266685877 +0000 2024-09-17T16:04:45.378 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-17 15:59:20.266685877 +0000 2024-09-17T16:04:45.378 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T16:04:45.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:44 smithi173 ceph-mon[30530]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:44 smithi173 ceph-mon[30530]: Reconfiguring mon.b (monmap changed)... 2024-09-17T16:04:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T16:04:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:44 smithi173 ceph-mon[30530]: Reconfiguring daemon mon.b on smithi071 2024-09-17T16:04:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:44 smithi173 ceph-mon[30530]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi071=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:44 smithi173 ceph-mon[30530]: Saving service mgr spec with placement smithi017=y;smithi071=x;count:2 2024-09-17T16:04:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:45.440 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-17T16:04:45.441 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-17T16:04:45.441 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000216748 s, 2.4 MB/s 2024-09-17T16:04:45.442 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T16:04:45.498 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-09-17T16:04:45.553 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T16:04:45.554 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:45.554 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-17T16:04:45.554 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:45.554 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:45.554 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-17 16:04:34.423783197 +0000 2024-09-17T16:04:45.554 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-17 15:59:20.598680105 +0000 2024-09-17T16:04:45.554 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-17 15:59:20.598680105 +0000 2024-09-17T16:04:45.554 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-17 15:59:20.598680105 +0000 2024-09-17T16:04:45.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T16:04:45.617 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-17T16:04:45.617 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-17T16:04:45.617 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000222514 s, 2.3 MB/s 2024-09-17T16:04:45.618 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T16:04:45.675 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-09-17T16:04:45.730 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T16:04:45.730 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:45.730 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-17T16:04:45.730 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:45.730 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:45.730 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-17 16:04:34.423783197 +0000 2024-09-17T16:04:45.730 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-17 15:59:20.930674333 +0000 2024-09-17T16:04:45.730 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-17 15:59:20.930674333 +0000 2024-09-17T16:04:45.730 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-17 15:59:20.930674333 +0000 2024-09-17T16:04:45.731 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T16:04:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-17T16:04:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-17T16:04:45.793 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000234188 s, 2.2 MB/s 2024-09-17T16:04:45.795 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T16:04:45.851 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-09-17T16:04:45.906 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T16:04:45.906 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:45.906 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-17T16:04:45.906 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:45.906 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:45.906 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-09-17 16:04:34.423783197 +0000 2024-09-17T16:04:45.906 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-09-17 15:59:21.260668596 +0000 2024-09-17T16:04:45.907 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-09-17 15:59:21.260668596 +0000 2024-09-17T16:04:45.907 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-09-17 15:59:21.260668596 +0000 2024-09-17T16:04:45.907 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T16:04:45.969 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-09-17T16:04:45.969 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-09-17T16:04:45.969 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000212209 s, 2.4 MB/s 2024-09-17T16:04:45.970 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T16:04:46.029 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T16:04:46.029 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T16:04:46.045 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T16:04:46.045 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-09-17T16:04:46.102 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T16:04:46.102 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:46.102 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-17T16:04:46.102 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:46.102 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:46.102 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 16:04:25.428077690 +0000 2024-09-17T16:04:46.102 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 15:59:20.469645365 +0000 2024-09-17T16:04:46.102 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 15:59:20.469645365 +0000 2024-09-17T16:04:46.102 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 15:59:20.469645365 +0000 2024-09-17T16:04:46.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T16:04:46.167 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T16:04:46.168 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T16:04:46.168 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000264373 s, 1.9 MB/s 2024-09-17T16:04:46.169 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T16:04:46.225 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-09-17T16:04:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T16:04:46.281 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:46.281 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-17T16:04:46.281 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:46.282 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:46.282 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 16:04:25.428077690 +0000 2024-09-17T16:04:46.282 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 15:59:20.802637731 +0000 2024-09-17T16:04:46.282 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 15:59:20.802637731 +0000 2024-09-17T16:04:46.282 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 15:59:20.802637731 +0000 2024-09-17T16:04:46.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T16:04:46.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:46.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:46.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:46 smithi071 ceph-mon[30002]: Reconfiguring mon.c (monmap changed)... 2024-09-17T16:04:46.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:46.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T16:04:46.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:46.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:46 smithi071 ceph-mon[30002]: Reconfiguring daemon mon.c on smithi173 2024-09-17T16:04:46.345 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T16:04:46.345 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T16:04:46.345 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000219231 s, 2.3 MB/s 2024-09-17T16:04:46.347 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T16:04:46.403 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-09-17T16:04:46.458 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T16:04:46.458 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:46.458 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-17T16:04:46.458 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:46.459 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:46.459 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 16:04:25.429077667 +0000 2024-09-17T16:04:46.459 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 15:59:21.125630326 +0000 2024-09-17T16:04:46.459 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 15:59:21.125630326 +0000 2024-09-17T16:04:46.459 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 15:59:21.125630326 +0000 2024-09-17T16:04:46.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T16:04:46.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:46 smithi017 ceph-mon[27571]: Reconfiguring mon.c (monmap changed)... 2024-09-17T16:04:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T16:04:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:46 smithi017 ceph-mon[27571]: Reconfiguring daemon mon.c on smithi173 2024-09-17T16:04:46.520 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:46.520 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:46.520 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:46 smithi173 ceph-mon[30530]: Reconfiguring mon.c (monmap changed)... 2024-09-17T16:04:46.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-17T16:04:46.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-17T16:04:46.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:46.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:46 smithi173 ceph-mon[30530]: Reconfiguring daemon mon.c on smithi173 2024-09-17T16:04:46.522 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T16:04:46.523 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T16:04:46.523 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000173246 s, 3.0 MB/s 2024-09-17T16:04:46.524 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T16:04:46.581 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-09-17T16:04:46.636 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T16:04:46.637 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T16:04:46.640 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-17T16:04:46.641 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T16:04:46.641 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T16:04:46.641 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-09-17 16:04:25.429077667 +0000 2024-09-17T16:04:46.641 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-09-17 15:59:21.478622233 +0000 2024-09-17T16:04:46.641 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-09-17 15:59:21.478622233 +0000 2024-09-17T16:04:46.642 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-09-17 15:59:21.478622233 +0000 2024-09-17T16:04:46.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T16:04:46.700 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-09-17T16:04:46.700 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-09-17T16:04:46.700 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000192488 s, 2.7 MB/s 2024-09-17T16:04:46.701 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T16:04:46.757 INFO:tasks.cephadm:Deploying osd.0 on smithi017 with /dev/vg_nvme/lv_4... 2024-09-17T16:04:46.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-17T16:04:47.019 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:04:47.303 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:47.303 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:47.303 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:47.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:47.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:47.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:47.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:47.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T16:04:47.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T16:04:47.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T16:04:47.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:47.477 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:47.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:47.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:47.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:47.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:47.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:47.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:47.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T16:04:47.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T16:04:47.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T16:04:47.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T16:04:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:48.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:48 smithi017 ceph-mon[27571]: Deploying daemon mgr.x on smithi071 2024-09-17T16:04:48.546 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:48 smithi071 ceph-mon[30002]: Deploying daemon mgr.x on smithi071 2024-09-17T16:04:48.546 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:04:48 smithi071 systemd[1]: Starting Ceph mgr.x for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:04:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:48 smithi173 ceph-mon[30530]: Deploying daemon mgr.x on smithi071 2024-09-17T16:04:48.814 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:04:48 smithi071 podman[31029]: 2024-09-17 16:04:48.544093568 +0000 UTC m=+0.106270541 container create dc4de72e6389fa10fde978cb3016af0c574850d3fcefbf06a5411e99a412817c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-17T16:04:48.814 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:04:48 smithi071 podman[31029]: 2024-09-17 16:04:48.453504295 +0000 UTC m=+0.015681213 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:04:48.814 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:04:48 smithi071 podman[31029]: 2024-09-17 16:04:48.754416657 +0000 UTC m=+0.316593566 container init dc4de72e6389fa10fde978cb3016af0c574850d3fcefbf06a5411e99a412817c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-x, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:04:48.814 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:04:48 smithi071 podman[31029]: 2024-09-17 16:04:48.757126544 +0000 UTC m=+0.319303459 container start dc4de72e6389fa10fde978cb3016af0c574850d3fcefbf06a5411e99a412817c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-17T16:04:48.814 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:04:48 smithi071 ceph-mgr[31048]: -- 172.21.15.71:0/666209565 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5576c4e610e0 con 0x5576c4e3f400 2024-09-17T16:04:49.223 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:04:48 smithi071 bash[31029]: dc4de72e6389fa10fde978cb3016af0c574850d3fcefbf06a5411e99a412817c 2024-09-17T16:04:49.224 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:04:48 smithi071 systemd[1]: Started Ceph mgr.x for 19341aec-750e-11ef-bceb-c7b262605968. 2024-09-17T16:04:49.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:49 smithi017 ceph-mon[27571]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:49.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:49.483 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:04:49.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi017:vg_nvme/lv_4 2024-09-17T16:04:49.519 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:49 smithi071 ceph-mon[30002]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:49.519 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.519 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.519 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.519 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.520 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:49 smithi173 ceph-mon[30530]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:49.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:49.768 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:04:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T16:04:51.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T16:04:51.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T16:04:51.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:51.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: Reconfiguring daemon mgr.y on smithi017 2024-09-17T16:04:51.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:04:51.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:04:51.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T16:04:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T16:04:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T16:04:51.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:51.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: Reconfiguring daemon mgr.y on smithi017 2024-09-17T16:04:51.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:04:51.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:04:51.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:51.435 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:51.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:51.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:51.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:51.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T16:04:51.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-17T16:04:51.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-17T16:04:51.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:51.437 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: Reconfiguring daemon mgr.y on smithi017 2024-09-17T16:04:51.437 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:04:51.437 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:04:51.437 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:52.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:52 smithi017 ceph-mon[27571]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:52.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:52.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:52.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:52.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:52.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:52.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:52.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:52 smithi071 ceph-mon[30002]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:52 smithi173 ceph-mon[30530]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:04:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:04:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:04:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:04:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:04:53.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:53 smithi017 ceph-mon[27571]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:53.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:53 smithi071 ceph-mon[30002]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:53.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:53 smithi173 ceph-mon[30530]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1199996440' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb"} : dispatch 2024-09-17T16:04:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1199996440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb"}]': finished 2024-09-17T16:04:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:54 smithi017 ceph-mon[27571]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T16:04:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:04:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:54 smithi017 ceph-mon[27571]: Standby manager daemon x started 2024-09-17T16:04:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:54 smithi017 ceph-mon[27571]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-17T16:04:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:54 smithi017 ceph-mon[27571]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-17T16:04:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:54 smithi017 ceph-mon[27571]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-17T16:04:54.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:54 smithi017 ceph-mon[27571]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-17T16:04:54.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/727076856' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:04:54.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1199996440' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb"} : dispatch 2024-09-17T16:04:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1199996440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb"}]': finished 2024-09-17T16:04:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:54 smithi071 ceph-mon[30002]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T16:04:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:54 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:04:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:54 smithi071 ceph-mon[30002]: Standby manager daemon x started 2024-09-17T16:04:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:54 smithi071 ceph-mon[30002]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-17T16:04:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:54 smithi071 ceph-mon[30002]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-17T16:04:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:54 smithi071 ceph-mon[30002]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-17T16:04:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:54 smithi071 ceph-mon[30002]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-17T16:04:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/727076856' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:04:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1199996440' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb"} : dispatch 2024-09-17T16:04:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1199996440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb"}]': finished 2024-09-17T16:04:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:54 smithi173 ceph-mon[30530]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T16:04:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:54 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:04:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:54 smithi173 ceph-mon[30530]: Standby manager daemon x started 2024-09-17T16:04:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:54 smithi173 ceph-mon[30530]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-17T16:04:54.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:54 smithi173 ceph-mon[30530]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-17T16:04:54.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:54 smithi173 ceph-mon[30530]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-17T16:04:54.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:54 smithi173 ceph-mon[30530]: from='mgr.? 172.21.15.71:0/4187261865' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-17T16:04:54.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/727076856' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:04:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:55 smithi017 ceph-mon[27571]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:55 smithi017 ceph-mon[27571]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-17T16:04:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-17T16:04:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:55 smithi071 ceph-mon[30002]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:55 smithi071 ceph-mon[30002]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-17T16:04:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-17T16:04:55.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:55 smithi173 ceph-mon[30530]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:55 smithi173 ceph-mon[30530]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-17T16:04:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-17T16:04:57.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:57 smithi017 ceph-mon[27571]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:57 smithi071 ceph-mon[30002]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:57.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:57 smithi173 ceph-mon[30530]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:59.475 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:04:59 smithi017 ceph-mon[27571]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:04:59 smithi071 ceph-mon[30002]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:04:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:04:59 smithi173 ceph-mon[30530]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:01.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:01 smithi173 ceph-mon[30530]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:01.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-17T16:05:01.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:01.466 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:01 smithi017 ceph-mon[27571]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:01.466 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-17T16:05:01.466 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:01.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:01 smithi071 ceph-mon[30002]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:01.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-17T16:05:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:02.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:02 smithi017 ceph-mon[27571]: Deploying daemon osd.0 on smithi017 2024-09-17T16:05:02.548 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:02 smithi071 ceph-mon[30002]: Deploying daemon osd.0 on smithi017 2024-09-17T16:05:02.638 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:02 smithi173 ceph-mon[30530]: Deploying daemon osd.0 on smithi017 2024-09-17T16:05:03.469 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:03 smithi017 ceph-mon[27571]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:03.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:03 smithi071 ceph-mon[30002]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:03.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:03 smithi173 ceph-mon[30530]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:05 smithi017 ceph-mon[27571]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:05.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:05 smithi071 ceph-mon[30002]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:05.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:05 smithi173 ceph-mon[30530]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:07 smithi017 ceph-mon[27571]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:07 smithi071 ceph-mon[30002]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:07.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:07 smithi173 ceph-mon[30530]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:08 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:08 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:08 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:08.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:08 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:08.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:08 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:08 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:08.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:08 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:08 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:08 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:09.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:09 smithi017 ceph-mon[27571]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:09.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:09 smithi071 ceph-mon[30002]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:09 smithi173 ceph-mon[30530]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:10.919 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 0 on host 'smithi017' 2024-09-17T16:05:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:11 smithi017 ceph-mon[27571]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:11 smithi071 ceph-mon[30002]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:11 smithi173 ceph-mon[30530]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:11.526 DEBUG:teuthology.orchestra.run.smithi017:osd.0> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.0.service 2024-09-17T16:05:11.529 INFO:tasks.cephadm:Deploying osd.1 on smithi017 with /dev/vg_nvme/lv_3... 2024-09-17T16:05:11.529 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-17T16:05:12.824 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:05:12 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0[39749]: 2024-09-17T16:05:12.793+0000 7f8078bd9640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:05:12.857 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:05:13.077 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:13 smithi017 ceph-mon[27571]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:13.077 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:13 smithi017 ceph-mon[27571]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-17T16:05:13.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:13 smithi071 ceph-mon[30002]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:13.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:13 smithi071 ceph-mon[30002]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-17T16:05:13.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:13 smithi173 ceph-mon[30530]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:13 smithi173 ceph-mon[30530]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-17T16:05:14.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:14 smithi071 ceph-mon[30002]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T16:05:14.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:14 smithi071 ceph-mon[30002]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T16:05:14.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:14 smithi071 ceph-mon[30002]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:05:14.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:14 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:14.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:14 smithi071 ceph-mon[30002]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:05:14.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:14 smithi071 ceph-mon[30002]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T16:05:14.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:14 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:14.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:14 smithi017 ceph-mon[27571]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T16:05:14.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:14 smithi017 ceph-mon[27571]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T16:05:14.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:14 smithi017 ceph-mon[27571]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:05:14.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:14.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:14 smithi017 ceph-mon[27571]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:05:14.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:14 smithi017 ceph-mon[27571]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T16:05:14.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:14.350 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:05:14 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0[39749]: 2024-09-17T16:05:14.030+0000 7f8074b5c640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T16:05:14.351 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:05:14 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0[39749]: 2024-09-17T16:05:14.032+0000 7f8070983640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:05:14.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:14 smithi173 ceph-mon[30530]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T16:05:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:14 smithi173 ceph-mon[30530]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T16:05:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:14 smithi173 ceph-mon[30530]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:05:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:14 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:14 smithi173 ceph-mon[30530]: from='osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:05:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:14 smithi173 ceph-mon[30530]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T16:05:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:14 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:15.128 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:15 smithi017 ceph-mon[27571]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:15.128 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:15 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:15.129 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:15 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:15.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:15 smithi071 ceph-mon[30002]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:15 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:15 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:15 smithi173 ceph-mon[30530]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:15 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:15 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:16.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730] boot 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi017 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi017 to 17092M 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:16.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:05:16.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730] boot 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi017 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi017 to 17092M 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:16.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: osd.0 [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730] boot 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi017 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi017 to 17092M 2024-09-17T16:05:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:16.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:16.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:05:16.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi017:vg_nvme/lv_3 2024-09-17T16:05:16.910 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:05:17.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:17 smithi017 ceph-mon[27571]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:17.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:17 smithi017 ceph-mon[27571]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T16:05:17.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:17 smithi071 ceph-mon[30002]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:17 smithi071 ceph-mon[30002]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T16:05:17.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:17 smithi173 ceph-mon[30530]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T16:05:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:17 smithi173 ceph-mon[30530]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T16:05:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:05:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:05:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:18.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:05:18.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:05:18.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:18.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:05:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:05:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:19.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:19 smithi017 ceph-mon[27571]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:05:19.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:19 smithi017 ceph-mon[27571]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:19.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:19 smithi071 ceph-mon[30002]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:05:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:19 smithi071 ceph-mon[30002]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:19 smithi173 ceph-mon[30530]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:05:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:19 smithi173 ceph-mon[30530]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:20.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/248590606' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1"} : dispatch 2024-09-17T16:05:20.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/248590606' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1"}]': finished 2024-09-17T16:05:20.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:20 smithi017 ceph-mon[27571]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T16:05:20.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:20.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/248590606' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1"} : dispatch 2024-09-17T16:05:20.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/248590606' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1"}]': finished 2024-09-17T16:05:20.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:20 smithi071 ceph-mon[30002]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T16:05:20.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:20.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/248590606' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1"} : dispatch 2024-09-17T16:05:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/248590606' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1"}]': finished 2024-09-17T16:05:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:20 smithi173 ceph-mon[30530]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T16:05:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:21 smithi017 ceph-mon[27571]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3827071601' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:05:21.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:21 smithi071 ceph-mon[30002]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3827071601' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:05:21.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:21 smithi173 ceph-mon[30530]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:21.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3827071601' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:05:23.484 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:23 smithi017 ceph-mon[27571]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:23 smithi071 ceph-mon[30002]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:23.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:23 smithi173 ceph-mon[30530]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:25 smithi017 ceph-mon[27571]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:25.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:25 smithi071 ceph-mon[30002]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:25.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:25 smithi173 ceph-mon[30530]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:27 smithi017 ceph-mon[27571]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:27 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-17T16:05:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:27 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:27 smithi071 ceph-mon[30002]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:27 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-17T16:05:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:27 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:27.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:27 smithi173 ceph-mon[30530]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:27 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-17T16:05:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:27 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:28.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:28 smithi017 ceph-mon[27571]: Deploying daemon osd.1 on smithi017 2024-09-17T16:05:28.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:28 smithi071 ceph-mon[30002]: Deploying daemon osd.1 on smithi017 2024-09-17T16:05:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:28 smithi173 ceph-mon[30530]: Deploying daemon osd.1 on smithi017 2024-09-17T16:05:29.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:29 smithi017 ceph-mon[27571]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:29 smithi071 ceph-mon[30002]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:29 smithi173 ceph-mon[30530]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:31.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:31 smithi173 ceph-mon[30530]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:31 smithi017 ceph-mon[27571]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:31 smithi071 ceph-mon[30002]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:33.360 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:33 smithi017 ceph-mon[27571]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:33.360 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:33.360 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:33.360 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:33 smithi071 ceph-mon[30002]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:33.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:33 smithi173 ceph-mon[30530]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:35 smithi017 ceph-mon[27571]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:35 smithi071 ceph-mon[30002]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:35 smithi173 ceph-mon[30530]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:36.811 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:36 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:36.811 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:36 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:36.811 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:36 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:36.812 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:36 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:36.812 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:36 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:36.812 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:36 smithi017 ceph-mon[27571]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:36.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:36 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:36 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:36 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:36 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:36 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:36 smithi071 ceph-mon[30002]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:36.852 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 1 on host 'smithi017' 2024-09-17T16:05:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:36 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:36 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:36 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:36 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:36 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:36 smithi173 ceph-mon[30530]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:37.533 DEBUG:teuthology.orchestra.run.smithi017:osd.1> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.1.service 2024-09-17T16:05:37.536 INFO:tasks.cephadm:Deploying osd.2 on smithi017 with /dev/vg_nvme/lv_2... 2024-09-17T16:05:37.536 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-17T16:05:37.789 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:37 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:37.789 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:37 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:37.789 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:37 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:37.789 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:05:37 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1[45329]: 2024-09-17T16:05:37.660+0000 7fb457384640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:05:37.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:37 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:37.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:37 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:37.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:37 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:37.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:37 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:37 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:37 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:38.700 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:05:38.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:38 smithi017 ceph-mon[27571]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:38.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:38 smithi017 ceph-mon[27571]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-17T16:05:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:38 smithi071 ceph-mon[30002]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:38 smithi071 ceph-mon[30002]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-17T16:05:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:38 smithi173 ceph-mon[30530]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:38 smithi173 ceph-mon[30530]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-17T16:05:39.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:39 smithi017 ceph-mon[27571]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T16:05:39.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:39 smithi017 ceph-mon[27571]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T16:05:39.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:39 smithi017 ceph-mon[27571]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:05:39.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:39 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:39.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:39 smithi173 ceph-mon[30530]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T16:05:39.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:39 smithi173 ceph-mon[30530]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T16:05:39.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:39 smithi173 ceph-mon[30530]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:05:39.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:39 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:39.989 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:05:39 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1[45329]: 2024-09-17T16:05:39.809+0000 7fb453307640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T16:05:39.989 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:05:39 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1[45329]: 2024-09-17T16:05:39.812+0000 7fb44b126640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:05:40.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:39 smithi071 ceph-mon[30002]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T16:05:40.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:39 smithi071 ceph-mon[30002]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T16:05:40.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:39 smithi071 ceph-mon[30002]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:05:40.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:39 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:40 smithi173 ceph-mon[30530]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:05:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:40 smithi173 ceph-mon[30530]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T16:05:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:40 smithi173 ceph-mon[30530]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:40 smithi173 ceph-mon[30530]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' 2024-09-17T16:05:40.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:40 smithi017 ceph-mon[27571]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:05:40.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:40 smithi017 ceph-mon[27571]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T16:05:40.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:40.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:40.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:40 smithi017 ceph-mon[27571]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:40.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:40 smithi017 ceph-mon[27571]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' 2024-09-17T16:05:41.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:40 smithi071 ceph-mon[30002]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:05:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:40 smithi071 ceph-mon[30002]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T16:05:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:40 smithi071 ceph-mon[30002]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-17T16:05:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:40 smithi071 ceph-mon[30002]: from='osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826]' entity='osd.1' 2024-09-17T16:05:41.588 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:05:41.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:05:41.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:41.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi017 2024-09-17T16:05:41.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:41.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:41.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:05:41.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-17T16:05:41.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi017 to 8546M 2024-09-17T16:05:41.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:41.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:41.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:41.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:41.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826] boot 2024-09-17T16:05:41.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T16:05:41.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:05:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:05:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi017 2024-09-17T16:05:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:05:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-17T16:05:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi017 to 8546M 2024-09-17T16:05:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:41.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:41.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:41.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:41.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826] boot 2024-09-17T16:05:41.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T16:05:41.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:42.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi017 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi017 to 8546M 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:42.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:42.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: osd.1 [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826] boot 2024-09-17T16:05:42.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T16:05:42.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-17T16:05:42.193 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:05:42.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi017:vg_nvme/lv_2 2024-09-17T16:05:42.482 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:05:42.876 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:42 smithi017 ceph-mon[27571]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:42.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:42 smithi173 ceph-mon[30530]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:43.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:42 smithi071 ceph-mon[30002]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:43.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:43 smithi017 ceph-mon[27571]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T16:05:43.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:43 smithi017 ceph-mon[27571]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:05:43.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:43 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:05:43.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:43 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:05:43.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:43 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:43 smithi173 ceph-mon[30530]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T16:05:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:43 smithi173 ceph-mon[30530]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:05:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:43 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:05:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:43 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:05:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:43 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:44.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:43 smithi071 ceph-mon[30002]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T16:05:44.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:43 smithi071 ceph-mon[30002]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:05:44.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:43 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:05:44.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:43 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:05:44.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:43 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:45 smithi017 ceph-mon[27571]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:45.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:45 smithi071 ceph-mon[30002]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:45 smithi173 ceph-mon[30530]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:46.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1553491190' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2"} : dispatch 2024-09-17T16:05:46.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1553491190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2"}]': finished 2024-09-17T16:05:46.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:46 smithi017 ceph-mon[27571]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T16:05:46.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:05:46.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2478304747' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:05:46.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1553491190' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2"} : dispatch 2024-09-17T16:05:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1553491190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2"}]': finished 2024-09-17T16:05:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:46 smithi071 ceph-mon[30002]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T16:05:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:05:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2478304747' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:05:46.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1553491190' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2"} : dispatch 2024-09-17T16:05:46.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1553491190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2"}]': finished 2024-09-17T16:05:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:46 smithi173 ceph-mon[30530]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T16:05:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:05:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2478304747' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:05:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:47 smithi017 ceph-mon[27571]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:47 smithi071 ceph-mon[30002]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:47 smithi173 ceph-mon[30530]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:49 smithi017 ceph-mon[27571]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:49 smithi071 ceph-mon[30002]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:49 smithi173 ceph-mon[30530]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:51.463 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:51 smithi017 ceph-mon[27571]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:51 smithi071 ceph-mon[30002]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:51.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:51 smithi173 ceph-mon[30530]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:53 smithi017 ceph-mon[27571]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:53 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-17T16:05:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:53 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:53 smithi071 ceph-mon[30002]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:53 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-17T16:05:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:53 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:53 smithi173 ceph-mon[30530]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:53 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-17T16:05:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:53 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:54.424 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:54 smithi017 ceph-mon[27571]: Deploying daemon osd.2 on smithi017 2024-09-17T16:05:54.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:54 smithi071 ceph-mon[30002]: Deploying daemon osd.2 on smithi017 2024-09-17T16:05:54.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:54 smithi173 ceph-mon[30530]: Deploying daemon osd.2 on smithi017 2024-09-17T16:05:55.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:55 smithi017 ceph-mon[27571]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:55 smithi071 ceph-mon[30002]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:55.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:55 smithi173 ceph-mon[30530]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:57 smithi017 ceph-mon[27571]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:57 smithi071 ceph-mon[30002]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:57 smithi173 ceph-mon[30530]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:05:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:05:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:05:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:58.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:05:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:05:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:05:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:05:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:05:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:05:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:05:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:05:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:59.428 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:59 smithi017 ceph-mon[27571]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:59.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:59 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:59.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:59 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:59.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:05:59 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:59 smithi071 ceph-mon[30002]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:59 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:59 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:05:59 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:59.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:59 smithi173 ceph-mon[30530]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:05:59.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:59 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:05:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:59 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:05:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:05:59 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:01.278 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:01 smithi017 ceph-mon[27571]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:01.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:01 smithi173 ceph-mon[30530]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:01.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:01 smithi071 ceph-mon[30002]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:02.516 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 2 on host 'smithi017' 2024-09-17T16:06:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:02 smithi017 ceph-mon[27571]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:02 smithi071 ceph-mon[30002]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:02 smithi173 ceph-mon[30530]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:03.149 DEBUG:teuthology.orchestra.run.smithi017:osd.2> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.2.service 2024-09-17T16:06:03.152 INFO:tasks.cephadm:Deploying osd.3 on smithi017 with /dev/vg_nvme/lv_1... 2024-09-17T16:06:03.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-17T16:06:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:03 smithi017 ceph-mon[27571]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-17T16:06:03.739 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:03 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2[50593]: 2024-09-17T16:06:03.413+0000 7f25ae921640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:06:03.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:03 smithi173 ceph-mon[30530]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-17T16:06:04.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:03 smithi071 ceph-mon[30002]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-17T16:06:04.700 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:06:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:04 smithi017 ceph-mon[27571]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T16:06:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:04 smithi017 ceph-mon[27571]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T16:06:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:04 smithi017 ceph-mon[27571]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:06:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:04 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:04 smithi017 ceph-mon[27571]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:04 smithi173 ceph-mon[30530]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T16:06:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:04 smithi173 ceph-mon[30530]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T16:06:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:04 smithi173 ceph-mon[30530]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:06:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:04 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:04 smithi173 ceph-mon[30530]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:05.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:04 smithi071 ceph-mon[30002]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T16:06:05.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:04 smithi071 ceph-mon[30002]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T16:06:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:04 smithi071 ceph-mon[30002]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:06:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:04 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:04 smithi071 ceph-mon[30002]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:05.139 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:04 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2[50593]: 2024-09-17T16:06:04.842+0000 7f25aa8a4640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T16:06:05.139 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:04 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2[50593]: 2024-09-17T16:06:04.845+0000 7f25a26c3640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:06:05.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:05 smithi173 ceph-mon[30530]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:06:05.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:05 smithi173 ceph-mon[30530]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T16:06:05.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:05 smithi173 ceph-mon[30530]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' 2024-09-17T16:06:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:05 smithi017 ceph-mon[27571]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:06:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:05 smithi017 ceph-mon[27571]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T16:06:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:05 smithi017 ceph-mon[27571]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' 2024-09-17T16:06:06.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:05 smithi071 ceph-mon[30002]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:06:06.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:05 smithi071 ceph-mon[30002]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T16:06:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:05 smithi071 ceph-mon[30002]: from='osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780]' entity='osd.2' 2024-09-17T16:06:06.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:06.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:06.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780] boot 2024-09-17T16:06:06.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T16:06:06.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:06.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi017 2024-09-17T16:06:06.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:06.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:06.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:06.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:06.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:06.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi017 to 5697M 2024-09-17T16:06:06.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:06.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:06.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:06.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:06.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:06 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:06:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780] boot 2024-09-17T16:06:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T16:06:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi017 2024-09-17T16:06:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:06.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:06.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:06.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:06.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi017 to 5697M 2024-09-17T16:06:06.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:06.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:06.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:06.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:06.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:06 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:06:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-17T16:06:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: osd.2 [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780] boot 2024-09-17T16:06:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T16:06:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-17T16:06:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi017 2024-09-17T16:06:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:07.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:07.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:07.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:07.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi017 to 5697M 2024-09-17T16:06:07.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:07.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:07.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:07.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:07.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:06 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:06:07.700 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:06:07.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi017:vg_nvme/lv_1 2024-09-17T16:06:07.986 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:06:08.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:08 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:06:08.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:08 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:06:08.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:08 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T16:06:08.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:08 smithi017 ceph-mon[27571]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T16:06:08.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:08 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:06:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:08 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:06:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:08 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:06:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:08 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T16:06:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:08 smithi173 ceph-mon[30530]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T16:06:08.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:08 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:06:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:08 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:06:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:08 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:06:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:08 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-17T16:06:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:08 smithi071 ceph-mon[30002]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T16:06:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:08 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:06:09.353 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:06:08 smithi017 sudo[54649]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T16:06:09.353 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:06:08 smithi017 sudo[54649]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.353 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54649]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.354 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54649]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.354 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54652]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T16:06:09.354 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54652]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.354 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54652]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.354 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54652]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.354 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T16:06:09.354 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.354 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54655]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.354 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54655]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.354 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T16:06:09.355 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.355 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54658]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.355 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54658]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.355 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54661]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T16:06:09.355 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54661]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.355 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54661]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.355 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54661]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.355 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54664]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T16:06:09.355 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54664]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.355 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54664]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.356 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54664]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 ceph-mon[27571]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:09.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 ceph-mon[27571]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:06:09.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:06:09.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 ceph-mon[27571]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T16:06:09.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:06:09.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:06:09.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:09.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:09.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54667]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T16:06:09.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54667]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54667]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54667]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54670]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T16:06:09.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54670]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54670]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:09 smithi017 sudo[54670]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 ceph-mon[30530]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 ceph-mon[30530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:06:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:06:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 ceph-mon[30530]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T16:06:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:06:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:06:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 sudo[32006]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T16:06:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 sudo[32006]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 sudo[32006]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 sudo[32006]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 sudo[32009]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T16:06:09.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 sudo[32009]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 sudo[32009]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:09 smithi173 sudo[32009]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 ceph-mon[30002]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 ceph-mon[30002]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 ceph-mon[30002]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 sudo[31492]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 sudo[31492]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 sudo[31492]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 sudo[31492]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:09.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 sudo[31495]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T16:06:09.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 sudo[31495]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-17T16:06:09.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 sudo[31495]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-17T16:06:09.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:09 smithi071 sudo[31495]: pam_unix(sudo:session): session closed for user root 2024-09-17T16:06:10.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T16:06:10.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:10.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T16:06:10.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:10.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:10.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:10.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T16:06:10.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:06:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T16:06:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T16:06:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T16:06:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T16:06:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:06:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T16:06:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T16:06:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T16:06:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T16:06:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T16:06:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T16:06:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T16:06:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:11.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:11 smithi017 ceph-mon[27571]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:11.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:11 smithi017 ceph-mon[27571]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:06:11.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:11 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:06:11.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:11 smithi017 ceph-mon[27571]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T16:06:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:11 smithi173 ceph-mon[30530]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:11 smithi173 ceph-mon[30530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:06:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:11 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:06:11.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:11 smithi173 ceph-mon[30530]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T16:06:11.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:11 smithi071 ceph-mon[30002]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:11 smithi071 ceph-mon[30002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:06:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:11 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:06:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:11 smithi071 ceph-mon[30002]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T16:06:12.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2568224800' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1"} : dispatch 2024-09-17T16:06:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2568224800' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1"}]': finished 2024-09-17T16:06:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:12 smithi173 ceph-mon[30530]: osdmap e22: 4 total, 3 up, 4 in 2024-09-17T16:06:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:12 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/337912355' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:06:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2568224800' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1"} : dispatch 2024-09-17T16:06:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2568224800' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1"}]': finished 2024-09-17T16:06:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:12 smithi017 ceph-mon[27571]: osdmap e22: 4 total, 3 up, 4 in 2024-09-17T16:06:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:12 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/337912355' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:06:12.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2568224800' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1"} : dispatch 2024-09-17T16:06:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2568224800' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1"}]': finished 2024-09-17T16:06:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:12 smithi071 ceph-mon[30002]: osdmap e22: 4 total, 3 up, 4 in 2024-09-17T16:06:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:12 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/337912355' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:06:13.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:13 smithi173 ceph-mon[30530]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:13 smithi017 ceph-mon[27571]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:13 smithi071 ceph-mon[30002]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:15.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:15 smithi017 ceph-mon[27571]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:15.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:15 smithi173 ceph-mon[30530]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:15.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:15 smithi071 ceph-mon[30002]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:17.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:17 smithi017 ceph-mon[27571]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:17.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:17 smithi173 ceph-mon[30530]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:17.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:17 smithi071 ceph-mon[30002]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:19 smithi017 ceph-mon[27571]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:19 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:06:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:19 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:06:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:19 smithi017 ceph-mon[27571]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T16:06:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:19 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:19 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-17T16:06:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:19 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:19.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:19 smithi173 ceph-mon[30530]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:19 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:06:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:19 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:06:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:19 smithi173 ceph-mon[30530]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T16:06:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:19 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:19 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-17T16:06:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:19 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:19.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:19 smithi071 ceph-mon[30002]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:19 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:06:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:19 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:06:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:19 smithi071 ceph-mon[30002]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T16:06:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:19 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:19 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-17T16:06:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:19 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:20.183 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:20 smithi017 ceph-mon[27571]: Deploying daemon osd.3 on smithi017 2024-09-17T16:06:20.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:20 smithi173 ceph-mon[30530]: Deploying daemon osd.3 on smithi017 2024-09-17T16:06:20.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:20 smithi071 ceph-mon[30002]: Deploying daemon osd.3 on smithi017 2024-09-17T16:06:21.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:21 smithi173 ceph-mon[30530]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:21.432 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:21 smithi071 ceph-mon[30002]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:21.474 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:21 smithi017 ceph-mon[27571]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:23.230 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:23 smithi017 ceph-mon[27571]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:23 smithi173 ceph-mon[30530]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:23 smithi071 ceph-mon[30002]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:24.367 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:24 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:24.367 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:24 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:24.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:24 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:24.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:24 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:24 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:24 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:24.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:24 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:24.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:24 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:24.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:24 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:25 smithi017 ceph-mon[27571]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:25.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:25 smithi173 ceph-mon[30530]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:25 smithi071 ceph-mon[30002]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:26.810 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 3 on host 'smithi017' 2024-09-17T16:06:27.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:27 smithi173 ceph-mon[30530]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:27 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:27 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:27.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:27 smithi017 ceph-mon[27571]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:27.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:27 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:27.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:27 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:27 smithi071 ceph-mon[30002]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:27 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:27 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:27.690 DEBUG:teuthology.orchestra.run.smithi017:osd.3> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.3.service 2024-09-17T16:06:27.696 INFO:tasks.cephadm:Deploying osd.4 on smithi071 with /dev/vg_nvme/lv_4... 2024-09-17T16:06:27.696 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-17T16:06:27.949 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:06:28.856 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:06:28 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3[56373]: 2024-09-17T16:06:28.574+0000 7fdb9c11a640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:06:29.131 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:28 smithi071 ceph-mon[30002]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:29.132 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:29.132 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:29.132 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:28 smithi071 ceph-mon[30002]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-17T16:06:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:28 smithi173 ceph-mon[30530]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:28 smithi173 ceph-mon[30530]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-17T16:06:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:28 smithi017 ceph-mon[27571]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:28 smithi017 ceph-mon[27571]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-17T16:06:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:30 smithi173 ceph-mon[30530]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T16:06:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:30 smithi173 ceph-mon[30530]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T16:06:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:30 smithi173 ceph-mon[30530]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:06:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:30.407 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:30 smithi071 ceph-mon[30002]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T16:06:30.408 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:30 smithi071 ceph-mon[30002]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T16:06:30.408 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:30 smithi071 ceph-mon[30002]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:06:30.408 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:30.410 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:06:30.437 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi071:vg_nvme/lv_4 2024-09-17T16:06:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:30 smithi017 ceph-mon[27571]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T16:06:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:30 smithi017 ceph-mon[27571]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T16:06:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:30 smithi017 ceph-mon[27571]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]} : dispatch 2024-09-17T16:06:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:30.489 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:06:30 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3[56373]: 2024-09-17T16:06:30.122+0000 7fdb9809d640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T16:06:30.489 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:06:30 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3[56373]: 2024-09-17T16:06:30.126+0000 7fdb93ec4640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:06:30.686 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:06:31.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:31 smithi071 ceph-mon[30002]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:31.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:31 smithi071 ceph-mon[30002]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:06:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:31 smithi071 ceph-mon[30002]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T16:06:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:31.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:31 smithi173 ceph-mon[30530]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:31.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:31 smithi173 ceph-mon[30530]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:06:31.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:31 smithi173 ceph-mon[30530]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T16:06:31.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:31.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:31 smithi017 ceph-mon[27571]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-17T16:06:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:31 smithi017 ceph-mon[27571]: from='osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-09-17T16:06:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:31 smithi017 ceph-mon[27571]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T16:06:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:06:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:06:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275] boot 2024-09-17T16:06:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi017 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi017 to 4273M 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:32.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:32.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:32.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:32.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:06:32.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:06:32.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:32.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:06:32.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:06:32.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275] boot 2024-09-17T16:06:32.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T16:06:32.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:32.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi017 2024-09-17T16:06:32.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi017 to 4273M 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:32.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:32.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:06:32.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:06:32.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:06:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:06:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: osd.3 [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275] boot 2024-09-17T16:06:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T16:06:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-17T16:06:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi017 2024-09-17T16:06:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi017 to 4273M 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:32.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:32.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:32.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:06:32.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:06:32.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:33.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:33 smithi071 ceph-mon[30002]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:06:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:33 smithi071 ceph-mon[30002]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:33 smithi071 ceph-mon[30002]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T16:06:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:33 smithi173 ceph-mon[30530]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:06:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:33 smithi173 ceph-mon[30530]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:33 smithi173 ceph-mon[30530]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T16:06:33.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:33 smithi017 ceph-mon[27571]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:06:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:33 smithi017 ceph-mon[27571]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:33 smithi017 ceph-mon[27571]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T16:06:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.71:0/1239382969' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5"} : dispatch 2024-09-17T16:06:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:34 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5"} : dispatch 2024-09-17T16:06:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:34 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5"}]': finished 2024-09-17T16:06:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:34 smithi071 ceph-mon[30002]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T16:06:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:34.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.71:0/1655537247' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:06:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.71:0/1239382969' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5"} : dispatch 2024-09-17T16:06:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:34 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5"} : dispatch 2024-09-17T16:06:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:34 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5"}]': finished 2024-09-17T16:06:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:34 smithi173 ceph-mon[30530]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T16:06:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.71:0/1655537247' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:06:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.71:0/1239382969' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5"} : dispatch 2024-09-17T16:06:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:34 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5"} : dispatch 2024-09-17T16:06:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:34 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5"}]': finished 2024-09-17T16:06:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:34 smithi017 ceph-mon[27571]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T16:06:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.71:0/1655537247' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:06:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:35 smithi071 ceph-mon[30002]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:35.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:35 smithi173 ceph-mon[30530]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:35 smithi017 ceph-mon[27571]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:37.447 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:37 smithi071 ceph-mon[30002]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:37 smithi173 ceph-mon[30530]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:37.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:37 smithi017 ceph-mon[27571]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:39.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:39 smithi071 ceph-mon[30002]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:39 smithi173 ceph-mon[30530]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:39 smithi017 ceph-mon[27571]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:41.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:41 smithi071 ceph-mon[30002]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:41.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-17T16:06:41.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:41.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:41 smithi173 ceph-mon[30530]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-17T16:06:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:41 smithi017 ceph-mon[27571]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-17T16:06:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:42 smithi071 ceph-mon[30002]: Deploying daemon osd.4 on smithi071 2024-09-17T16:06:42.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:42 smithi173 ceph-mon[30530]: Deploying daemon osd.4 on smithi071 2024-09-17T16:06:42.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:42 smithi017 ceph-mon[27571]: Deploying daemon osd.4 on smithi071 2024-09-17T16:06:43.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:43 smithi071 ceph-mon[30002]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:43.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:43 smithi173 ceph-mon[30530]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:43.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:43 smithi017 ceph-mon[27571]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:45 smithi071 ceph-mon[30002]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:45.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:45 smithi173 ceph-mon[30530]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:45 smithi017 ceph-mon[27571]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:47 smithi071 ceph-mon[30002]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:47 smithi173 ceph-mon[30530]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:47 smithi017 ceph-mon[27571]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:49 smithi071 ceph-mon[30002]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:49.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:49 smithi173 ceph-mon[30530]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:49.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:49 smithi017 ceph-mon[27571]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:50.700 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:50.700 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:50.700 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:50.700 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:50.701 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:50.701 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:50 smithi071 ceph-mon[30002]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:50 smithi017 ceph-mon[27571]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:50.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:50.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:50 smithi173 ceph-mon[30530]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:50.988 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 4 on host 'smithi071' 2024-09-17T16:06:51.628 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:51.629 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:51.629 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:51.655 DEBUG:teuthology.orchestra.run.smithi071:osd.4> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.4.service 2024-09-17T16:06:51.657 INFO:tasks.cephadm:Deploying osd.5 on smithi071 with /dev/vg_nvme/lv_3... 2024-09-17T16:06:51.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-17T16:06:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:51.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:52.198 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:06:52 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:06:52.083+0000 7f9419a38640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:06:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:52 smithi017 ceph-mon[27571]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:52 smithi017 ceph-mon[27571]: from='osd.4 [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T16:06:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:52 smithi017 ceph-mon[27571]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T16:06:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:52 smithi071 ceph-mon[30002]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:52 smithi071 ceph-mon[30002]: from='osd.4 [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T16:06:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:52 smithi071 ceph-mon[30002]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T16:06:52.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:52 smithi173 ceph-mon[30530]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:52 smithi173 ceph-mon[30530]: from='osd.4 [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T16:06:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:52 smithi173 ceph-mon[30530]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-17T16:06:53.317 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:06:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:53 smithi017 ceph-mon[27571]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T16:06:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:53 smithi017 ceph-mon[27571]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T16:06:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:53 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:53 smithi017 ceph-mon[27571]: from='osd.4 [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:06:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:53 smithi017 ceph-mon[27571]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:06:53.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:53 smithi071 ceph-mon[30002]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T16:06:53.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:53 smithi071 ceph-mon[30002]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T16:06:53.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:53 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:53.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:53 smithi071 ceph-mon[30002]: from='osd.4 [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:06:53.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:53 smithi071 ceph-mon[30002]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:06:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:53 smithi173 ceph-mon[30530]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T16:06:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:53 smithi173 ceph-mon[30530]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T16:06:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:53 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:53 smithi173 ceph-mon[30530]: from='osd.4 [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:06:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:53 smithi173 ceph-mon[30530]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:06:54.023 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:06:53 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:06:53.923+0000 7f94159bb640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T16:06:54.023 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:06:53 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:06:53.929+0000 7f940d7da640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:06:54.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:54 smithi071 ceph-mon[30002]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:06:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:54 smithi071 ceph-mon[30002]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T16:06:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:54 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:54 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:54 smithi071 ceph-mon[30002]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:54 smithi071 ceph-mon[30002]: from='osd.4 ' entity='osd.4' 2024-09-17T16:06:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:54 smithi071 ceph-mon[30002]: osd.4 [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020] boot 2024-09-17T16:06:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:54 smithi071 ceph-mon[30002]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T16:06:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:54 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:54 smithi017 ceph-mon[27571]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:06:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:54 smithi017 ceph-mon[27571]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T16:06:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:54 smithi017 ceph-mon[27571]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:54 smithi017 ceph-mon[27571]: from='osd.4 ' entity='osd.4' 2024-09-17T16:06:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:54 smithi017 ceph-mon[27571]: osd.4 [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020] boot 2024-09-17T16:06:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:54 smithi017 ceph-mon[27571]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T16:06:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:54 smithi173 ceph-mon[30530]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:06:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:54 smithi173 ceph-mon[30530]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T16:06:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:54 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:54 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:54 smithi173 ceph-mon[30530]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-17T16:06:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:54 smithi173 ceph-mon[30530]: from='osd.4 ' entity='osd.4' 2024-09-17T16:06:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:54 smithi173 ceph-mon[30530]: osd.4 [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020] boot 2024-09-17T16:06:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:54 smithi173 ceph-mon[30530]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T16:06:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:54 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-17T16:06:55.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:06:55.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:06:55.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi071 2024-09-17T16:06:55.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:55.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi071 to 17088M 2024-09-17T16:06:55.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:55.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:55.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:55 smithi071 ceph-mon[30002]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T16:06:55.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi071 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi071 to 17088M 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:55 smithi173 ceph-mon[30530]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T16:06:55.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:06:55.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:06:55.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi071 2024-09-17T16:06:55.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:06:55.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi071 to 17088M 2024-09-17T16:06:55.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:55.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:55.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:55.991 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:55 smithi017 ceph-mon[27571]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T16:06:56.405 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:06:56.432 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi071:vg_nvme/lv_3 2024-09-17T16:06:56.675 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:06:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:57 smithi017 ceph-mon[27571]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:06:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:57 smithi017 ceph-mon[27571]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T16:06:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:57 smithi071 ceph-mon[30002]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:06:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:57 smithi071 ceph-mon[30002]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T16:06:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:57 smithi173 ceph-mon[30530]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:06:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:57 smithi173 ceph-mon[30530]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T16:06:58.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:58 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:06:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:06:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:06:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:06:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:58.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:58.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:58.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:58.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:58 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:06:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:06:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:06:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:06:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:06:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:06:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:06:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:06:58.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:06:58.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:06:58.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:58 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:06:59.236 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:59 smithi071 ceph-mon[30002]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-09-17T16:06:59.236 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:06:59 smithi071 ceph-mon[30002]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T16:06:59.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:59 smithi017 ceph-mon[27571]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-09-17T16:06:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:06:59 smithi017 ceph-mon[27571]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T16:06:59.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:59 smithi173 ceph-mon[30530]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-09-17T16:06:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:06:59 smithi173 ceph-mon[30530]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T16:07:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.71:0/1947158303' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f08464f8-610f-4775-87d0-803b569eb211"} : dispatch 2024-09-17T16:07:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:00 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f08464f8-610f-4775-87d0-803b569eb211"} : dispatch 2024-09-17T16:07:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:00 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f08464f8-610f-4775-87d0-803b569eb211"}]': finished 2024-09-17T16:07:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:00 smithi017 ceph-mon[27571]: osdmap e35: 6 total, 5 up, 6 in 2024-09-17T16:07:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.71:0/3066422360' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:07:00.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.71:0/1947158303' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f08464f8-610f-4775-87d0-803b569eb211"} : dispatch 2024-09-17T16:07:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:00 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f08464f8-610f-4775-87d0-803b569eb211"} : dispatch 2024-09-17T16:07:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:00 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f08464f8-610f-4775-87d0-803b569eb211"}]': finished 2024-09-17T16:07:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:00 smithi071 ceph-mon[30002]: osdmap e35: 6 total, 5 up, 6 in 2024-09-17T16:07:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.71:0/3066422360' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:07:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.71:0/1947158303' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f08464f8-610f-4775-87d0-803b569eb211"} : dispatch 2024-09-17T16:07:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:00 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f08464f8-610f-4775-87d0-803b569eb211"} : dispatch 2024-09-17T16:07:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:00 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f08464f8-610f-4775-87d0-803b569eb211"}]': finished 2024-09-17T16:07:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:00 smithi173 ceph-mon[30530]: osdmap e35: 6 total, 5 up, 6 in 2024-09-17T16:07:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.71:0/3066422360' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:07:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:01 smithi017 ceph-mon[27571]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-09-17T16:07:01.490 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:01 smithi071 ceph-mon[30002]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-09-17T16:07:01.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:01 smithi173 ceph-mon[30530]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-09-17T16:07:03.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:03 smithi017 ceph-mon[27571]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:07:03.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:03 smithi071 ceph-mon[30002]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:07:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:03 smithi173 ceph-mon[30530]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:07:05.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:05 smithi017 ceph-mon[27571]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-17T16:07:05.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:05 smithi071 ceph-mon[30002]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-17T16:07:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:05 smithi173 ceph-mon[30530]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-17T16:07:07.478 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:07 smithi071 ceph-mon[30002]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:07.478 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-17T16:07:07.478 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:07.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:07 smithi017 ceph-mon[27571]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-17T16:07:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:07 smithi173 ceph-mon[30530]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-17T16:07:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:08.382 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:08 smithi071 ceph-mon[30002]: Deploying daemon osd.5 on smithi071 2024-09-17T16:07:08.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:08 smithi017 ceph-mon[27571]: Deploying daemon osd.5 on smithi071 2024-09-17T16:07:08.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:08 smithi173 ceph-mon[30530]: Deploying daemon osd.5 on smithi071 2024-09-17T16:07:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:09 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:07:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:09 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:07:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:09 smithi017 ceph-mon[27571]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:09 smithi017 ceph-mon[27571]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T16:07:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:09.535 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:09 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:07:09.535 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:09 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:07:09.535 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:09 smithi071 ceph-mon[30002]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:09.535 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:09 smithi071 ceph-mon[30002]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T16:07:09.535 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:09 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:07:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:09 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:07:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:09 smithi173 ceph-mon[30530]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:09 smithi173 ceph-mon[30530]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T16:07:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:11.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:11 smithi017 ceph-mon[27571]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:11 smithi071 ceph-mon[30002]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:11.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:11 smithi173 ceph-mon[30530]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:13.409 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:13 smithi071 ceph-mon[30002]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:13.409 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:13 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:13.409 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:13 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:13.409 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:13 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:13 smithi017 ceph-mon[27571]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:13 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:13 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:13 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:13.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:13 smithi173 ceph-mon[30530]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:13 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:13 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:13 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:15.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:15 smithi071 ceph-mon[30002]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:15.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:15 smithi173 ceph-mon[30530]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:15 smithi017 ceph-mon[27571]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:16.546 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:16.546 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:16.546 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:16.547 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:16.547 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:16 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:16.547 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:16 smithi071 ceph-mon[30002]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:16.585 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 5 on host 'smithi071' 2024-09-17T16:07:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:16 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:16 smithi173 ceph-mon[30530]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:16 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:16 smithi017 ceph-mon[27571]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:17.244 DEBUG:teuthology.orchestra.run.smithi071:osd.5> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.5.service 2024-09-17T16:07:17.246 INFO:tasks.cephadm:Deploying osd.6 on smithi071 with /dev/vg_nvme/lv_2... 2024-09-17T16:07:17.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-17T16:07:17.509 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:17 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:17.509 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:17 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:17.509 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:17 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:17.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:17 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:17 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:17 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:17.814 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:07:17 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[39093]: 2024-09-17T16:07:17.505+0000 7f1ab6ac7640 -1 osd.5 0 log_to_monitors true 2024-09-17T16:07:17.996 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:07:18.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:18 smithi173 ceph-mon[30530]: from='osd.5 [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:07:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:18 smithi173 ceph-mon[30530]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:07:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:18 smithi173 ceph-mon[30530]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:18.652 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:18 smithi071 ceph-mon[30002]: from='osd.5 [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:07:18.652 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:18 smithi071 ceph-mon[30002]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:07:18.652 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:18 smithi071 ceph-mon[30002]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:18 smithi017 ceph-mon[27571]: from='osd.5 [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:07:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:18 smithi017 ceph-mon[27571]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:07:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:18 smithi017 ceph-mon[27571]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:19.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:07:19 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[39093]: 2024-09-17T16:07:19.379+0000 7f1ab2a4a640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T16:07:19.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:07:19 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[39093]: 2024-09-17T16:07:19.385+0000 7f1aae871640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:07:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:19 smithi071 ceph-mon[30002]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T16:07:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:19 smithi071 ceph-mon[30002]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T16:07:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:19 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:19 smithi071 ceph-mon[30002]: from='osd.5 [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:19 smithi071 ceph-mon[30002]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:19 smithi173 ceph-mon[30530]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T16:07:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:19 smithi173 ceph-mon[30530]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T16:07:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:19 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:19 smithi173 ceph-mon[30530]: from='osd.5 [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:19 smithi173 ceph-mon[30530]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:19 smithi017 ceph-mon[27571]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T16:07:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:19 smithi017 ceph-mon[27571]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T16:07:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:19 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:19 smithi017 ceph-mon[27571]: from='osd.5 [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:19 smithi017 ceph-mon[27571]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:20 smithi173 ceph-mon[30530]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:07:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:20 smithi173 ceph-mon[30530]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T16:07:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:20 smithi173 ceph-mon[30530]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:20.676 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:20 smithi071 ceph-mon[30002]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:07:20.677 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:20 smithi071 ceph-mon[30002]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T16:07:20.677 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:20.677 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:20.677 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:20 smithi071 ceph-mon[30002]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:20 smithi017 ceph-mon[27571]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:07:20.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:20 smithi017 ceph-mon[27571]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T16:07:20.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:20.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:20.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:20 smithi017 ceph-mon[27571]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-17T16:07:21.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:07:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:07:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: osd.5 [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307] boot 2024-09-17T16:07:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T16:07:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:07:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:07:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:07:21.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:21.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi071 to 8544M 2024-09-17T16:07:21.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:21.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:21.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:21.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:21.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:21.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:07:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:07:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: osd.5 [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307] boot 2024-09-17T16:07:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T16:07:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi071 to 8544M 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:21.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:07:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:07:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: osd.5 [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307] boot 2024-09-17T16:07:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T16:07:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:07:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi071 to 8544M 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:21.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:21.875 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:07:21.901 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi071:vg_nvme/lv_2 2024-09-17T16:07:22.143 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:07:22.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:22 smithi071 ceph-mon[30002]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T16:07:22.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:22 smithi071 ceph-mon[30002]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:22 smithi071 ceph-mon[30002]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T16:07:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:22 smithi173 ceph-mon[30530]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T16:07:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:22 smithi173 ceph-mon[30530]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:22 smithi173 ceph-mon[30530]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T16:07:22.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:22 smithi017 ceph-mon[27571]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T16:07:22.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:22 smithi017 ceph-mon[27571]: pgmap v136: 1 pgs: 1 remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:22.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:22 smithi017 ceph-mon[27571]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T16:07:23.446 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:23 smithi071 ceph-mon[30002]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:07:23.446 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:07:23.446 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:07:23.446 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:23 smithi173 ceph-mon[30530]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:07:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:07:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:07:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:23 smithi017 ceph-mon[27571]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:07:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:07:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:07:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:25.484 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:25 smithi071 ceph-mon[30002]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:25.484 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.71:0/1704781814' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31"} : dispatch 2024-09-17T16:07:25.484 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:25 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31"} : dispatch 2024-09-17T16:07:25.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:25 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31"}]': finished 2024-09-17T16:07:25.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:25 smithi071 ceph-mon[30002]: osdmap e42: 7 total, 6 up, 7 in 2024-09-17T16:07:25.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:25.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:25 smithi017 ceph-mon[27571]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.71:0/1704781814' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31"} : dispatch 2024-09-17T16:07:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:25 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31"} : dispatch 2024-09-17T16:07:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:25 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31"}]': finished 2024-09-17T16:07:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:25 smithi017 ceph-mon[27571]: osdmap e42: 7 total, 6 up, 7 in 2024-09-17T16:07:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:25 smithi173 ceph-mon[30530]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.71:0/1704781814' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31"} : dispatch 2024-09-17T16:07:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:25 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31"} : dispatch 2024-09-17T16:07:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:25 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31"}]': finished 2024-09-17T16:07:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:25 smithi173 ceph-mon[30530]: osdmap e42: 7 total, 6 up, 7 in 2024-09-17T16:07:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.71:0/1456606961' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:07:26.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.71:0/1456606961' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:07:26.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.71:0/1456606961' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:07:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:27 smithi017 ceph-mon[27571]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:27 smithi071 ceph-mon[30002]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:27.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:27 smithi173 ceph-mon[30530]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:29.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:29 smithi071 ceph-mon[30002]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-17T16:07:29.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:29 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:07:29.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:29 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:07:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:29 smithi017 ceph-mon[27571]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-17T16:07:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:29 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:07:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:29 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:07:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:29 smithi173 ceph-mon[30530]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-17T16:07:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:29 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:07:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:29 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:07:30.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:30 smithi071 ceph-mon[30002]: osdmap e43: 7 total, 6 up, 7 in 2024-09-17T16:07:30.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:30 smithi017 ceph-mon[27571]: osdmap e43: 7 total, 6 up, 7 in 2024-09-17T16:07:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:30.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:30 smithi173 ceph-mon[30530]: osdmap e43: 7 total, 6 up, 7 in 2024-09-17T16:07:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:31.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:31 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:07:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:31 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:07:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:31 smithi071 ceph-mon[30002]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-17T16:07:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:31 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:07:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:31 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:07:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:31 smithi017 ceph-mon[27571]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-17T16:07:31.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:31 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:07:31.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:31 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:07:31.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:31 smithi173 ceph-mon[30530]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-17T16:07:33.488 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:33 smithi071 ceph-mon[30002]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T16:07:33.488 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-17T16:07:33.488 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:33.488 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:33 smithi071 ceph-mon[30002]: Deploying daemon osd.6 on smithi071 2024-09-17T16:07:33.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:33 smithi017 ceph-mon[27571]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T16:07:33.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-17T16:07:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:33 smithi017 ceph-mon[27571]: Deploying daemon osd.6 on smithi071 2024-09-17T16:07:33.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:33 smithi173 ceph-mon[30530]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T16:07:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-17T16:07:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:33 smithi173 ceph-mon[30530]: Deploying daemon osd.6 on smithi071 2024-09-17T16:07:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:35 smithi017 ceph-mon[27571]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-17T16:07:35.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:35 smithi071 ceph-mon[30002]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-17T16:07:35.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:35 smithi173 ceph-mon[30530]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-17T16:07:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:37 smithi071 ceph-mon[30002]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:37.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:37 smithi173 ceph-mon[30530]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:37 smithi017 ceph-mon[27571]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:38.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:38 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:38 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:38 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:38.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:38 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:38.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:38 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:38.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:38 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:38 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:38 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:38 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:39.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:39 smithi071 ceph-mon[30002]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:39 smithi173 ceph-mon[30530]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:39 smithi017 ceph-mon[27571]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:41.541 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:41 smithi071 ceph-mon[30002]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:41.541 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:41.542 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:41.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:41 smithi173 ceph-mon[30530]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:41 smithi017 ceph-mon[27571]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:41.864 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 6 on host 'smithi071' 2024-09-17T16:07:42.445 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:42.446 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:42.446 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:42.446 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:42.446 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:42.446 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:42.471 DEBUG:teuthology.orchestra.run.smithi071:osd.6> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.6.service 2024-09-17T16:07:42.474 INFO:tasks.cephadm:Deploying osd.7 on smithi071 with /dev/vg_nvme/lv_1... 2024-09-17T16:07:42.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-17T16:07:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:42 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:42 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:42 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:42 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:42 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:42 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:42 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:42 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:42 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:42 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:07:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:42 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:42 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:43.324 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:07:43.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:43 smithi071 ceph-mon[30002]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:43 smithi071 ceph-mon[30002]: from='osd.6 [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T16:07:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:43 smithi071 ceph-mon[30002]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T16:07:43.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:07:43 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:07:43.215+0000 7f89607fd640 -1 osd.6 0 log_to_monitors true 2024-09-17T16:07:43.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:43 smithi173 ceph-mon[30530]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:43 smithi173 ceph-mon[30530]: from='osd.6 [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T16:07:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:43 smithi173 ceph-mon[30530]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T16:07:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:43 smithi017 ceph-mon[27571]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:43 smithi017 ceph-mon[27571]: from='osd.6 [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T16:07:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:43 smithi017 ceph-mon[27571]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-17T16:07:44.526 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:07:44 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:07:44.280+0000 7f895c780640 -1 osd.6 0 waiting for initial osdmap 2024-09-17T16:07:44.526 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:07:44 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:07:44.287+0000 7f89585a7640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:07:44.526 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:44 smithi071 ceph-mon[30002]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T16:07:44.526 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:44 smithi071 ceph-mon[30002]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T16:07:44.527 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:44.527 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:44 smithi071 ceph-mon[30002]: from='osd.6 [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:44.527 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:44 smithi071 ceph-mon[30002]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:44 smithi173 ceph-mon[30530]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T16:07:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:44 smithi173 ceph-mon[30530]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T16:07:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:44 smithi173 ceph-mon[30530]: from='osd.6 [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:44 smithi173 ceph-mon[30530]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:44 smithi017 ceph-mon[27571]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T16:07:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:44 smithi017 ceph-mon[27571]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T16:07:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:44 smithi017 ceph-mon[27571]: from='osd.6 [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:44 smithi017 ceph-mon[27571]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:07:45.534 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:45 smithi071 ceph-mon[30002]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:45.534 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:45 smithi071 ceph-mon[30002]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:07:45.534 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:45 smithi071 ceph-mon[30002]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T16:07:45.534 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:45 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:45.534 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:45 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:45 smithi173 ceph-mon[30530]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:45 smithi173 ceph-mon[30530]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:07:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:45 smithi173 ceph-mon[30530]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T16:07:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:45 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:45 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:45.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:45 smithi017 ceph-mon[27571]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-17T16:07:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:45 smithi017 ceph-mon[27571]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:07:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:45 smithi017 ceph-mon[27571]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T16:07:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:45 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:45 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:46.432 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:07:46.432 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:07:46.432 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: osd.6 [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628] boot 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: osdmap e46: 7 total, 7 up, 7 in 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:46 smithi071 ceph-mon[30002]: osdmap e47: 7 total, 7 up, 7 in 2024-09-17T16:07:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:07:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:07:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: osd.6 [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628] boot 2024-09-17T16:07:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: osdmap e46: 7 total, 7 up, 7 in 2024-09-17T16:07:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:46.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:46.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:46.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:46.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:46.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:46 smithi173 ceph-mon[30530]: osdmap e47: 7 total, 7 up, 7 in 2024-09-17T16:07:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:07:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:07:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: osd.6 [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628] boot 2024-09-17T16:07:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: osdmap e46: 7 total, 7 up, 7 in 2024-09-17T16:07:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-17T16:07:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-17T16:07:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:07:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:07:46.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:46 smithi017 ceph-mon[27571]: osdmap e47: 7 total, 7 up, 7 in 2024-09-17T16:07:47.154 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-09-17T16:07:47.179 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi071:vg_nvme/lv_1 2024-09-17T16:07:47.428 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:07:47.543 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:47 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi071 2024-09-17T16:07:47.543 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:47 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi071 to 5696M 2024-09-17T16:07:47.543 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:47 smithi071 ceph-mon[30002]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:47 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi071 2024-09-17T16:07:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:47 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi071 to 5696M 2024-09-17T16:07:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:47 smithi173 ceph-mon[30530]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:47 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi071 2024-09-17T16:07:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:47 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi071 to 5696M 2024-09-17T16:07:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:47 smithi017 ceph-mon[27571]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:48.483 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:48 smithi071 ceph-mon[30002]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T16:07:48.483 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:48 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:07:48.483 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:48 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:07:48.483 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:48 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:48 smithi173 ceph-mon[30530]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T16:07:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:48 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:07:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:48 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:07:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:48 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:48 smithi017 ceph-mon[27571]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T16:07:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:48 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:07:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:48 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:07:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:48 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:49 smithi071 ceph-mon[30002]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:49 smithi071 ceph-mon[30002]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:07:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:49 smithi071 ceph-mon[30002]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:07:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:49 smithi173 ceph-mon[30530]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:49 smithi173 ceph-mon[30530]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:07:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:49 smithi173 ceph-mon[30530]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:07:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:49 smithi017 ceph-mon[27571]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:49 smithi017 ceph-mon[27571]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:07:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:49 smithi017 ceph-mon[27571]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:07:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:51 smithi071 ceph-mon[30002]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.71:0/1794579060' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2"} : dispatch 2024-09-17T16:07:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:51 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2"} : dispatch 2024-09-17T16:07:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:51 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2"}]': finished 2024-09-17T16:07:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:51 smithi071 ceph-mon[30002]: osdmap e49: 8 total, 7 up, 8 in 2024-09-17T16:07:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:07:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.71:0/1721582169' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:07:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:51 smithi173 ceph-mon[30530]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.71:0/1794579060' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2"} : dispatch 2024-09-17T16:07:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:51 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2"} : dispatch 2024-09-17T16:07:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:51 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2"}]': finished 2024-09-17T16:07:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:51 smithi173 ceph-mon[30530]: osdmap e49: 8 total, 7 up, 8 in 2024-09-17T16:07:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:07:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.71:0/1721582169' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:07:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:51 smithi017 ceph-mon[27571]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.71:0/1794579060' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2"} : dispatch 2024-09-17T16:07:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:51 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2"} : dispatch 2024-09-17T16:07:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:51 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2"}]': finished 2024-09-17T16:07:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:51 smithi017 ceph-mon[27571]: osdmap e49: 8 total, 7 up, 8 in 2024-09-17T16:07:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:07:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.71:0/1721582169' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:07:52.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:52 smithi173 ceph-mon[30530]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:52 smithi173 ceph-mon[30530]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T16:07:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:52 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:07:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:52 smithi017 ceph-mon[27571]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:52 smithi017 ceph-mon[27571]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T16:07:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:52 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:07:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:52 smithi071 ceph-mon[30002]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:07:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:52 smithi071 ceph-mon[30002]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T16:07:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:52 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:07:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:54 smithi017 ceph-mon[27571]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-17T16:07:55.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:54 smithi071 ceph-mon[30002]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-17T16:07:55.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:54 smithi173 ceph-mon[30530]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-17T16:07:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:56 smithi017 ceph-mon[27571]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-17T16:07:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:56 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:07:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:56 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:07:57.033 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:56 smithi071 ceph-mon[30002]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-17T16:07:57.033 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:56 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:07:57.033 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:56 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:07:57.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:56 smithi173 ceph-mon[30530]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-17T16:07:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:56 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:07:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:56 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:07:58.044 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:57 smithi071 ceph-mon[30002]: osdmap e50: 8 total, 7 up, 8 in 2024-09-17T16:07:58.044 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:57 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:07:58.044 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:57 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-17T16:07:58.044 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:57 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:57 smithi173 ceph-mon[30530]: osdmap e50: 8 total, 7 up, 8 in 2024-09-17T16:07:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:57 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:07:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:57 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-17T16:07:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:57 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:58.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:57 smithi017 ceph-mon[27571]: osdmap e50: 8 total, 7 up, 8 in 2024-09-17T16:07:58.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:07:58.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-17T16:07:58.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:07:59.046 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:58 smithi071 ceph-mon[30002]: Deploying daemon osd.7 on smithi071 2024-09-17T16:07:59.046 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:07:58 smithi071 ceph-mon[30002]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T16:07:59.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:58 smithi173 ceph-mon[30530]: Deploying daemon osd.7 on smithi071 2024-09-17T16:07:59.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:07:58 smithi173 ceph-mon[30530]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T16:07:59.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:58 smithi017 ceph-mon[27571]: Deploying daemon osd.7 on smithi071 2024-09-17T16:07:59.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:07:58 smithi017 ceph-mon[27571]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T16:08:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:00 smithi071 ceph-mon[30002]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:01.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:00 smithi173 ceph-mon[30530]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:01.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:00 smithi017 ceph-mon[27571]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:02.999 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:02 smithi071 ceph-mon[30002]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:03.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:02 smithi173 ceph-mon[30530]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:03.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:02 smithi017 ceph-mon[27571]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:04.015 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:04.015 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:04.015 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:04.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:04.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:04.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:05.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:05 smithi071 ceph-mon[30002]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:05 smithi173 ceph-mon[30530]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:05 smithi017 ceph-mon[27571]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:06.346 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:06 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:08:06.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:06 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:08:06.384 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 7 on host 'smithi071' 2024-09-17T16:08:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:06 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:08:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:06 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:08:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:06 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:08:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:06 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:08:07.036 DEBUG:teuthology.orchestra.run.smithi071:osd.7> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.7.service 2024-09-17T16:08:07.038 INFO:tasks.cephadm:Deploying osd.8 on smithi173 with /dev/vg_nvme/lv_4... 2024-09-17T16:08:07.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-17T16:08:07.289 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:08:07.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:07 smithi173 ceph-mon[30530]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:07 smithi173 ceph-mon[30530]: osdmap e51: 8 total, 7 up, 8 in 2024-09-17T16:08:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:07 smithi071 ceph-mon[30002]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:07 smithi071 ceph-mon[30002]: osdmap e51: 8 total, 7 up, 8 in 2024-09-17T16:08:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:07 smithi017 ceph-mon[27571]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:07 smithi017 ceph-mon[27571]: osdmap e51: 8 total, 7 up, 8 in 2024-09-17T16:08:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:08.536 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:08 smithi071 ceph-mon[30002]: from='osd.7 [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:08:08.536 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:08 smithi071 ceph-mon[30002]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:08:08.537 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:08:08 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[50487]: 2024-09-17T16:08:08.268+0000 7fbe43864640 -1 osd.7 0 log_to_monitors true 2024-09-17T16:08:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:08 smithi173 ceph-mon[30530]: from='osd.7 [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:08:08.555 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:08 smithi173 ceph-mon[30530]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:08:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:08 smithi017 ceph-mon[27571]: from='osd.7 [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:08:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:08 smithi017 ceph-mon[27571]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:08:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:09 smithi173 ceph-mon[30530]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:09 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:08:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:09 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:08:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:09 smithi173 ceph-mon[30530]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T16:08:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:09 smithi173 ceph-mon[30530]: from='osd.7 [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:08:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:09 smithi173 ceph-mon[30530]: osdmap e52: 8 total, 7 up, 8 in 2024-09-17T16:08:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:09 smithi173 ceph-mon[30530]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:08:09.441 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:08:09 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[50487]: 2024-09-17T16:08:09.303+0000 7fbe3f7e7640 -1 osd.7 0 waiting for initial osdmap 2024-09-17T16:08:09.441 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:08:09 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[50487]: 2024-09-17T16:08:09.312+0000 7fbe3b60e640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:08:09.441 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:09 smithi071 ceph-mon[30002]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:09.442 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:09 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:08:09.442 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:09 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:08:09.442 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:09 smithi071 ceph-mon[30002]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T16:08:09.442 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:09 smithi071 ceph-mon[30002]: from='osd.7 [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:08:09.442 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:09 smithi071 ceph-mon[30002]: osdmap e52: 8 total, 7 up, 8 in 2024-09-17T16:08:09.442 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:09.442 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:09 smithi071 ceph-mon[30002]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:08:09.723 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:08:09.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:09 smithi017 ceph-mon[27571]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:09 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:08:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:09 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:08:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:09 smithi017 ceph-mon[27571]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T16:08:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:09 smithi017 ceph-mon[27571]: from='osd.7 [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:08:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:09 smithi017 ceph-mon[27571]: osdmap e52: 8 total, 7 up, 8 in 2024-09-17T16:08:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:09 smithi017 ceph-mon[27571]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:08:09.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-09-17T16:08:10.003 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:08:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:08:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T16:08:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi071 2024-09-17T16:08:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi071 to 4272M 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:10.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:10.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:10 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:08:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T16:08:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi071 2024-09-17T16:08:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi071 to 4272M 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:10.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:10 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.753 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-09-17T16:08:10.753 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T16:08:10.753 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:10.753 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:10.754 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-17T16:08:10.754 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi071 2024-09-17T16:08:10.754 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.754 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.754 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.754 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.754 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.755 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:10.755 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi071 to 4272M 2024-09-17T16:08:10.755 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.755 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:10.755 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:10.755 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:10.755 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:10.756 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:10.756 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:10.756 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:10 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:11.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:11 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:08:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:11 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:08:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:11 smithi071 ceph-mon[30002]: osd.7 [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174] boot 2024-09-17T16:08:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:11 smithi071 ceph-mon[30002]: osdmap e54: 8 total, 8 up, 8 in 2024-09-17T16:08:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:11 smithi071 ceph-mon[30002]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:08:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:08:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:08:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:11.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:11 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:08:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:11 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:08:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:11 smithi173 ceph-mon[30530]: osd.7 [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174] boot 2024-09-17T16:08:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:11 smithi173 ceph-mon[30530]: osdmap e54: 8 total, 8 up, 8 in 2024-09-17T16:08:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:11 smithi173 ceph-mon[30530]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:08:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:08:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:08:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:11 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:08:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:11 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:08:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:11 smithi017 ceph-mon[27571]: osd.7 [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174] boot 2024-09-17T16:08:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:11 smithi017 ceph-mon[27571]: osdmap e54: 8 total, 8 up, 8 in 2024-09-17T16:08:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:08:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:11 smithi017 ceph-mon[27571]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:08:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:08:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:08:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:12 smithi173 ceph-mon[30530]: osdmap e55: 8 total, 8 up, 8 in 2024-09-17T16:08:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:12 smithi173 ceph-mon[30530]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:12 smithi017 ceph-mon[27571]: osdmap e55: 8 total, 8 up, 8 in 2024-09-17T16:08:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:12 smithi017 ceph-mon[27571]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:12 smithi071 ceph-mon[30002]: osdmap e55: 8 total, 8 up, 8 in 2024-09-17T16:08:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:12 smithi071 ceph-mon[30002]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:13 smithi173 ceph-mon[30530]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:08:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:13 smithi173 ceph-mon[30530]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T16:08:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/1503949430' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035"} : dispatch 2024-09-17T16:08:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:13 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035"} : dispatch 2024-09-17T16:08:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:13 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035"}]': finished 2024-09-17T16:08:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:13 smithi173 ceph-mon[30530]: osdmap e57: 9 total, 8 up, 9 in 2024-09-17T16:08:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:13 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:13 smithi017 ceph-mon[27571]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:08:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:13 smithi017 ceph-mon[27571]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T16:08:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/1503949430' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035"} : dispatch 2024-09-17T16:08:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:13 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035"} : dispatch 2024-09-17T16:08:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:13 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035"}]': finished 2024-09-17T16:08:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:13 smithi017 ceph-mon[27571]: osdmap e57: 9 total, 8 up, 9 in 2024-09-17T16:08:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:13 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:13 smithi071 ceph-mon[30002]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:08:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:13 smithi071 ceph-mon[30002]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T16:08:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/1503949430' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035"} : dispatch 2024-09-17T16:08:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:13 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035"} : dispatch 2024-09-17T16:08:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:13 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035"}]': finished 2024-09-17T16:08:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:13 smithi071 ceph-mon[30002]: osdmap e57: 9 total, 8 up, 9 in 2024-09-17T16:08:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:13 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:14.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/247880236' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:08:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:14 smithi173 ceph-mon[30530]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:14.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/247880236' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:08:14.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:14 smithi017 ceph-mon[27571]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:14.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/247880236' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:08:14.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:14 smithi071 ceph-mon[30002]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:16.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:16 smithi173 ceph-mon[30530]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:16 smithi017 ceph-mon[27571]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:17.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:16 smithi071 ceph-mon[30002]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:17.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:17 smithi173 ceph-mon[30530]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T16:08:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:17 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:08:18.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:17 smithi071 ceph-mon[30002]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T16:08:18.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:17 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:08:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:17 smithi017 ceph-mon[27571]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T16:08:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:17 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:08:19.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:18 smithi071 ceph-mon[30002]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-17T16:08:19.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:18 smithi173 ceph-mon[30530]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-17T16:08:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:18 smithi017 ceph-mon[27571]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-17T16:08:21.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:20 smithi071 ceph-mon[30002]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T16:08:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-17T16:08:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:20 smithi071 ceph-mon[30002]: Deploying daemon osd.8 on smithi173 2024-09-17T16:08:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:20 smithi173 ceph-mon[30530]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T16:08:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-17T16:08:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:20 smithi173 ceph-mon[30530]: Deploying daemon osd.8 on smithi173 2024-09-17T16:08:21.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:20 smithi017 ceph-mon[27571]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T16:08:21.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-17T16:08:21.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:21.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:20 smithi017 ceph-mon[27571]: Deploying daemon osd.8 on smithi173 2024-09-17T16:08:23.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:22 smithi071 ceph-mon[30002]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-17T16:08:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:22 smithi173 ceph-mon[30530]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-17T16:08:23.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:22 smithi017 ceph-mon[27571]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-17T16:08:25.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:24 smithi071 ceph-mon[30002]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-17T16:08:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:24 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:08:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:24 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:08:25.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:24 smithi173 ceph-mon[30530]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-17T16:08:25.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:24 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:08:25.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:24 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:08:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:24 smithi017 ceph-mon[27571]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-17T16:08:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:24 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:08:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:24 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:08:26.007 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:25 smithi173 ceph-mon[30530]: osdmap e58: 9 total, 8 up, 9 in 2024-09-17T16:08:26.007 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:26.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:25 smithi071 ceph-mon[30002]: osdmap e58: 9 total, 8 up, 9 in 2024-09-17T16:08:26.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:25 smithi017 ceph-mon[27571]: osdmap e58: 9 total, 8 up, 9 in 2024-09-17T16:08:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:26.922 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:26 smithi173 ceph-mon[30530]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T16:08:26.922 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:26 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:26.922 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:26 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:26.922 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:26 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:27.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:26 smithi071 ceph-mon[30002]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T16:08:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:26 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:26 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:26 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:27.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:26 smithi017 ceph-mon[27571]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T16:08:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:26 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:26 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:26 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:29 smithi071 ceph-mon[30002]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:29 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:29 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:29 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:29 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:29 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:29 smithi173 ceph-mon[30530]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:29 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:29 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:29 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:29 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:29 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:29 smithi017 ceph-mon[27571]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:29 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:29 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:29 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:29.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:29 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:29.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:29 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:30.339 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 8 on host 'smithi173' 2024-09-17T16:08:30.919 DEBUG:teuthology.orchestra.run.smithi173:osd.8> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.8.service 2024-09-17T16:08:30.921 INFO:tasks.cephadm:Deploying osd.9 on smithi173 with /dev/vg_nvme/lv_3... 2024-09-17T16:08:30.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-17T16:08:31.495 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:31 smithi173 ceph-mon[30530]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:31.495 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:31.495 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:31.495 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:31 smithi071 ceph-mon[30002]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:31 smithi017 ceph-mon[27571]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:31.894 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:08:31 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:08:31.546+0000 7fa2313ff640 -1 osd.8 0 log_to_monitors true 2024-09-17T16:08:32.342 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:32 smithi173 ceph-mon[30530]: from='osd.8 [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-17T16:08:32.342 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:32 smithi173 ceph-mon[30530]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-17T16:08:32.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:32 smithi071 ceph-mon[30002]: from='osd.8 [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-17T16:08:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:32 smithi071 ceph-mon[30002]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-17T16:08:32.573 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:08:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:32 smithi017 ceph-mon[27571]: from='osd.8 [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-17T16:08:32.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:32 smithi017 ceph-mon[27571]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-17T16:08:33.569 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:33 smithi173 ceph-mon[30530]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:33.569 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:33 smithi173 ceph-mon[30530]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-17T16:08:33.569 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:33 smithi173 ceph-mon[30530]: osdmap e59: 9 total, 8 up, 9 in 2024-09-17T16:08:33.569 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:33.569 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:33 smithi173 ceph-mon[30530]: from='osd.8 [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:33.569 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:33 smithi173 ceph-mon[30530]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:33 smithi017 ceph-mon[27571]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:33 smithi017 ceph-mon[27571]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-17T16:08:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:33 smithi017 ceph-mon[27571]: osdmap e59: 9 total, 8 up, 9 in 2024-09-17T16:08:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:33 smithi017 ceph-mon[27571]: from='osd.8 [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:33 smithi017 ceph-mon[27571]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:33.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:33 smithi071 ceph-mon[30002]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:33 smithi071 ceph-mon[30002]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-17T16:08:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:33 smithi071 ceph-mon[30002]: osdmap e59: 9 total, 8 up, 9 in 2024-09-17T16:08:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:33 smithi071 ceph-mon[30002]: from='osd.8 [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:33 smithi071 ceph-mon[30002]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:33.827 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:08:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:08:33.733+0000 7fa22d382640 -1 osd.8 0 waiting for initial osdmap 2024-09-17T16:08:33.827 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:08:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:08:33.740+0000 7fa2251a1640 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:08:34.626 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:08:34.626 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: osdmap e60: 9 total, 8 up, 9 in 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='osd.8 ' entity='osd.8' 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi173 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi173 to 21184M 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.627 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: osd.8 [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865] boot 2024-09-17T16:08:34.628 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: osdmap e61: 9 total, 9 up, 9 in 2024-09-17T16:08:34.628 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:08:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: osdmap e60: 9 total, 8 up, 9 in 2024-09-17T16:08:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='osd.8 ' entity='osd.8' 2024-09-17T16:08:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi173 2024-09-17T16:08:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi173 to 21184M 2024-09-17T16:08:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: osd.8 [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865] boot 2024-09-17T16:08:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: osdmap e61: 9 total, 9 up, 9 in 2024-09-17T16:08:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: osdmap e60: 9 total, 8 up, 9 in 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='osd.8 ' entity='osd.8' 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi173 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:08:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi173 to 21184M 2024-09-17T16:08:34.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:34.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:34.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:34.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: osd.8 [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865] boot 2024-09-17T16:08:34.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: osdmap e61: 9 total, 9 up, 9 in 2024-09-17T16:08:34.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-17T16:08:35.525 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:35 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:08:35.525 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:35 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:08:35.525 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:35 smithi173 ceph-mon[30530]: osdmap e62: 9 total, 9 up, 9 in 2024-09-17T16:08:35.732 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:08:35.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:35 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:08:35.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:35 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:08:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:35 smithi017 ceph-mon[27571]: osdmap e62: 9 total, 9 up, 9 in 2024-09-17T16:08:35.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-09-17T16:08:35.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:35 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:08:35.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:35 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:08:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:35 smithi071 ceph-mon[30002]: osdmap e62: 9 total, 9 up, 9 in 2024-09-17T16:08:36.012 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:08:37.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:37 smithi173 ceph-mon[30530]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 564 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:37 smithi173 ceph-mon[30530]: osdmap e63: 9 total, 9 up, 9 in 2024-09-17T16:08:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:37 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:08:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:37 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:08:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:37 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:37 smithi017 ceph-mon[27571]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 564 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:37 smithi017 ceph-mon[27571]: osdmap e63: 9 total, 9 up, 9 in 2024-09-17T16:08:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:37 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:08:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:37 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:08:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:37 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:37 smithi071 ceph-mon[30002]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 564 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:37 smithi071 ceph-mon[30002]: osdmap e63: 9 total, 9 up, 9 in 2024-09-17T16:08:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:37 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:08:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:37 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:08:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:37 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:38.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:38 smithi173 ceph-mon[30530]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:08:38.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:38 smithi017 ceph-mon[27571]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:08:38.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:38 smithi071 ceph-mon[30002]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:08:39.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:39 smithi071 ceph-mon[30002]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 564 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-17T16:08:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:39 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:08:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:39 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:08:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/3470976190' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe"} : dispatch 2024-09-17T16:08:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:39 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe"} : dispatch 2024-09-17T16:08:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:39 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe"}]': finished 2024-09-17T16:08:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:39 smithi071 ceph-mon[30002]: osdmap e64: 10 total, 9 up, 10 in 2024-09-17T16:08:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:39 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:39 smithi173 ceph-mon[30530]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 564 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-17T16:08:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:39 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:08:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:39 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:08:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/3470976190' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe"} : dispatch 2024-09-17T16:08:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:39 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe"} : dispatch 2024-09-17T16:08:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:39 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe"}]': finished 2024-09-17T16:08:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:39 smithi173 ceph-mon[30530]: osdmap e64: 10 total, 9 up, 10 in 2024-09-17T16:08:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:39 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:39 smithi017 ceph-mon[27571]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 564 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-17T16:08:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:39 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:08:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:39 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:08:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/3470976190' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe"} : dispatch 2024-09-17T16:08:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:39 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe"} : dispatch 2024-09-17T16:08:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:39 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe"}]': finished 2024-09-17T16:08:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:39 smithi017 ceph-mon[27571]: osdmap e64: 10 total, 9 up, 10 in 2024-09-17T16:08:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:39 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:40.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:40 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:08:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:40 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:08:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/1769426135' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:08:40.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:40 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:08:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:40 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:08:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/1769426135' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:08:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:40 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:08:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:40 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:08:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/1769426135' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:08:41.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:41 smithi071 ceph-mon[30002]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-17T16:08:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:41 smithi071 ceph-mon[30002]: osdmap e65: 10 total, 9 up, 10 in 2024-09-17T16:08:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:41.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:41 smithi173 ceph-mon[30530]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-17T16:08:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:41 smithi173 ceph-mon[30530]: osdmap e65: 10 total, 9 up, 10 in 2024-09-17T16:08:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:41 smithi017 ceph-mon[27571]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-09-17T16:08:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:41 smithi017 ceph-mon[27571]: osdmap e65: 10 total, 9 up, 10 in 2024-09-17T16:08:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:43.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:43 smithi071 ceph-mon[30002]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:08:43.632 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:43 smithi173 ceph-mon[30530]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:08:43.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:43 smithi017 ceph-mon[27571]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:08:45.525 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:45 smithi173 ceph-mon[30530]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-17T16:08:45.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:45 smithi071 ceph-mon[30002]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-17T16:08:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:45 smithi017 ceph-mon[27571]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-17T16:08:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:47 smithi071 ceph-mon[30002]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-17T16:08:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:47 smithi173 ceph-mon[30530]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-17T16:08:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:47 smithi017 ceph-mon[27571]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-17T16:08:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:48.378 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:48 smithi173 ceph-mon[30530]: Deploying daemon osd.9 on smithi173 2024-09-17T16:08:48.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:48 smithi071 ceph-mon[30002]: Deploying daemon osd.9 on smithi173 2024-09-17T16:08:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:48 smithi017 ceph-mon[27571]: Deploying daemon osd.9 on smithi173 2024-09-17T16:08:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:49 smithi071 ceph-mon[30002]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:49 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:08:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:49 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:08:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:49 smithi071 ceph-mon[30002]: osdmap e66: 10 total, 9 up, 10 in 2024-09-17T16:08:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:49 smithi173 ceph-mon[30530]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:49 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:08:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:49 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:08:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:49 smithi173 ceph-mon[30530]: osdmap e66: 10 total, 9 up, 10 in 2024-09-17T16:08:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:49 smithi017 ceph-mon[27571]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:49 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:08:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:49 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:08:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:49 smithi017 ceph-mon[27571]: osdmap e66: 10 total, 9 up, 10 in 2024-09-17T16:08:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:51.345 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:51 smithi173 ceph-mon[30530]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:51 smithi071 ceph-mon[30002]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:51 smithi017 ceph-mon[27571]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:53.424 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:53 smithi173 ceph-mon[30530]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:53.424 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:53 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:53.424 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:53 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:53.424 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:53 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:53 smithi071 ceph-mon[30002]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:53 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:53 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:53 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:53 smithi017 ceph-mon[27571]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:53 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:53 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:53 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:55.526 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:55 smithi173 ceph-mon[30530]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:55.527 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:55.527 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:55.527 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:55.527 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:55.527 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:55.541 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:55 smithi017 ceph-mon[27571]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:55.541 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:55.541 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:55.541 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:55.541 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:55.541 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:55 smithi071 ceph-mon[30002]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:08:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:08:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:56.158 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 9 on host 'smithi173' 2024-09-17T16:08:56.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:56.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:56.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:56.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:56.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:56.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:08:56.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:56.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:08:56.840 DEBUG:teuthology.orchestra.run.smithi173:osd.9> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.9.service 2024-09-17T16:08:56.842 INFO:tasks.cephadm:Deploying osd.10 on smithi173 with /dev/vg_nvme/lv_2... 2024-09-17T16:08:56.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-17T16:08:57.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:57 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:08:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:57 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:08:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:57 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:08:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:57 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:08:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:57 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:08:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:57 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:08:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:08:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:08:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:08:57.985 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:08:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:08:57.706+0000 7f36331e9640 -1 osd.9 0 log_to_monitors true 2024-09-17T16:08:58.080 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:08:58.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:58 smithi071 ceph-mon[30002]: from='osd.9 [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-17T16:08:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:58 smithi071 ceph-mon[30002]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-17T16:08:58.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:58 smithi173 ceph-mon[30530]: from='osd.9 [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-17T16:08:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:58 smithi173 ceph-mon[30530]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-17T16:08:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:58 smithi017 ceph-mon[27571]: from='osd.9 [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-17T16:08:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:58 smithi017 ceph-mon[27571]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-17T16:08:59.450 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:59 smithi173 ceph-mon[30530]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:59.451 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:59 smithi173 ceph-mon[30530]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-17T16:08:59.451 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:59 smithi173 ceph-mon[30530]: osdmap e67: 10 total, 9 up, 10 in 2024-09-17T16:08:59.451 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:59 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:59.451 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:59 smithi173 ceph-mon[30530]: from='osd.9 [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:59.451 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:08:59 smithi173 ceph-mon[30530]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:59 smithi071 ceph-mon[30002]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:59 smithi071 ceph-mon[30002]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-17T16:08:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:59 smithi071 ceph-mon[30002]: osdmap e67: 10 total, 9 up, 10 in 2024-09-17T16:08:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:59 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:59 smithi071 ceph-mon[30002]: from='osd.9 [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:08:59 smithi071 ceph-mon[30002]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:59 smithi017 ceph-mon[27571]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-09-17T16:08:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:59 smithi017 ceph-mon[27571]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-17T16:08:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:59 smithi017 ceph-mon[27571]: osdmap e67: 10 total, 9 up, 10 in 2024-09-17T16:08:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:59 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:08:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:59 smithi017 ceph-mon[27571]: from='osd.9 [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:08:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:08:59 smithi017 ceph-mon[27571]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:00.105 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:08:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:08:59.739+0000 7f362f980640 -1 osd.9 0 waiting for initial osdmap 2024-09-17T16:09:00.106 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:08:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:08:59.746+0000 7f3626f8b640 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:09:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:09:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: osdmap e68: 10 total, 9 up, 10 in 2024-09-17T16:09:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='osd.9 ' entity='osd.9' 2024-09-17T16:09:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi173 2024-09-17T16:09:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi173 to 10592M 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:09:00.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:09:00.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:00.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:00.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:09:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: osdmap e68: 10 total, 9 up, 10 in 2024-09-17T16:09:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='osd.9 ' entity='osd.9' 2024-09-17T16:09:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi173 2024-09-17T16:09:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi173 to 10592M 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:09:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: osdmap e68: 10 total, 9 up, 10 in 2024-09-17T16:09:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='osd.9 ' entity='osd.9' 2024-09-17T16:09:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi173 2024-09-17T16:09:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi173 to 10592M 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:00.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:09:01.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-09-17T16:09:01.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:01 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:01 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:01 smithi071 ceph-mon[30002]: osd.9 [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352] boot 2024-09-17T16:09:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:01 smithi071 ceph-mon[30002]: osdmap e69: 10 total, 10 up, 10 in 2024-09-17T16:09:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:01.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:01 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:01 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:01 smithi173 ceph-mon[30530]: osd.9 [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352] boot 2024-09-17T16:09:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:01 smithi173 ceph-mon[30530]: osdmap e69: 10 total, 10 up, 10 in 2024-09-17T16:09:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:01.736 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:09:01.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:01 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:01.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:01 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:01 smithi017 ceph-mon[27571]: osd.9 [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352] boot 2024-09-17T16:09:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:01 smithi017 ceph-mon[27571]: osdmap e69: 10 total, 10 up, 10 in 2024-09-17T16:09:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-17T16:09:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:03 smithi071 ceph-mon[30002]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:03 smithi071 ceph-mon[30002]: osdmap e70: 10 total, 10 up, 10 in 2024-09-17T16:09:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:09:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:09:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:03.584 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:03 smithi173 ceph-mon[30530]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:03.585 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:03 smithi173 ceph-mon[30530]: osdmap e70: 10 total, 10 up, 10 in 2024-09-17T16:09:03.585 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:09:03.585 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:09:03.585 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:03 smithi017 ceph-mon[27571]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:03 smithi017 ceph-mon[27571]: osdmap e70: 10 total, 10 up, 10 in 2024-09-17T16:09:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:09:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:09:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:04.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:04 smithi071 ceph-mon[30002]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:09:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:04 smithi173 ceph-mon[30530]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:09:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:04 smithi017 ceph-mon[27571]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:09:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:05 smithi071 ceph-mon[30002]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:05 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:05 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/1163777652' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61"} : dispatch 2024-09-17T16:09:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:05 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61"} : dispatch 2024-09-17T16:09:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:05 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61"}]': finished 2024-09-17T16:09:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:05 smithi071 ceph-mon[30002]: osdmap e71: 11 total, 10 up, 11 in 2024-09-17T16:09:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/2390709626' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:09:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:05 smithi173 ceph-mon[30530]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:05 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:05 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/1163777652' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61"} : dispatch 2024-09-17T16:09:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:05 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61"} : dispatch 2024-09-17T16:09:05.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:05 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61"}]': finished 2024-09-17T16:09:05.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:05 smithi173 ceph-mon[30530]: osdmap e71: 11 total, 10 up, 11 in 2024-09-17T16:09:05.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:05.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/2390709626' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:09:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:05 smithi017 ceph-mon[27571]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:05 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:05 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/1163777652' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61"} : dispatch 2024-09-17T16:09:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:05 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61"} : dispatch 2024-09-17T16:09:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:05 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61"}]': finished 2024-09-17T16:09:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:05 smithi017 ceph-mon[27571]: osdmap e71: 11 total, 10 up, 11 in 2024-09-17T16:09:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:05.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/2390709626' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:09:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:07 smithi071 ceph-mon[30002]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:07.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:07 smithi173 ceph-mon[30530]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:07.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:07 smithi017 ceph-mon[27571]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:09.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:09 smithi173 ceph-mon[30530]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:09.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:09 smithi071 ceph-mon[30002]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:09 smithi017 ceph-mon[27571]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:10.614 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:10 smithi173 ceph-mon[30530]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:10 smithi017 ceph-mon[27571]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:10 smithi071 ceph-mon[30002]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:11.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-17T16:09:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:11.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-17T16:09:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:11.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-17T16:09:11.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:12.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:12 smithi173 ceph-mon[30530]: Deploying daemon osd.10 on smithi173 2024-09-17T16:09:12.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:12 smithi173 ceph-mon[30530]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:12 smithi017 ceph-mon[27571]: Deploying daemon osd.10 on smithi173 2024-09-17T16:09:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:12 smithi017 ceph-mon[27571]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:12 smithi071 ceph-mon[30002]: Deploying daemon osd.10 on smithi173 2024-09-17T16:09:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:12 smithi071 ceph-mon[30002]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:15.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:14 smithi071 ceph-mon[30002]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:14 smithi173 ceph-mon[30530]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:14 smithi017 ceph-mon[27571]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:16.869 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:16 smithi173 ceph-mon[30530]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:17.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:16 smithi071 ceph-mon[30002]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:16 smithi017 ceph-mon[27571]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:17 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:17 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:17 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:17 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:17 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:18.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:17 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:18.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:17 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:18.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:17 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:18.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:17 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:18.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:17 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:17 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:17 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:19.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:18 smithi173 ceph-mon[30530]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:19.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:18 smithi173 ceph-mon[30530]: osdmap e72: 11 total, 10 up, 11 in 2024-09-17T16:09:19.027 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:19.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:18 smithi071 ceph-mon[30002]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:19.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:18 smithi071 ceph-mon[30002]: osdmap e72: 11 total, 10 up, 11 in 2024-09-17T16:09:19.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:18 smithi017 ceph-mon[27571]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:18 smithi017 ceph-mon[27571]: osdmap e72: 11 total, 10 up, 11 in 2024-09-17T16:09:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:20.832 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:20.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:20.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:20 smithi173 ceph-mon[30530]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:20.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:20.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:20.833 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:20.873 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 10 on host 'smithi173' 2024-09-17T16:09:20.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:20 smithi017 ceph-mon[27571]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:20 smithi071 ceph-mon[30002]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:21.558 DEBUG:teuthology.orchestra.run.smithi173:osd.10> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.10.service 2024-09-17T16:09:21.560 INFO:tasks.cephadm:Deploying osd.11 on smithi173 with /dev/vg_nvme/lv_1... 2024-09-17T16:09:21.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-17T16:09:21.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:21.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:21.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:21.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:21.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:21.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:22.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:22.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:22.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:22.312 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:09:22.650 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:09:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[45258]: 2024-09-17T16:09:22.396+0000 7f16cd062640 -1 osd.10 0 log_to_monitors true 2024-09-17T16:09:22.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:22 smithi173 ceph-mon[30530]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:22.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:22 smithi173 ceph-mon[30530]: from='osd.10 [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:09:22.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:22 smithi173 ceph-mon[30530]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:09:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:22 smithi017 ceph-mon[27571]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:22 smithi017 ceph-mon[27571]: from='osd.10 [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:09:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:22 smithi017 ceph-mon[27571]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:09:23.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:22 smithi071 ceph-mon[30002]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:22 smithi071 ceph-mon[30002]: from='osd.10 [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:09:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:22 smithi071 ceph-mon[30002]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:09:23.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:23 smithi173 ceph-mon[30530]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-17T16:09:23.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:23 smithi173 ceph-mon[30530]: osdmap e73: 11 total, 10 up, 11 in 2024-09-17T16:09:23.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:23.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:23 smithi173 ceph-mon[30530]: from='osd.10 [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:23.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:23 smithi173 ceph-mon[30530]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:23.826 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:09:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[45258]: 2024-09-17T16:09:23.660+0000 7f16c8fe5640 -1 osd.10 0 waiting for initial osdmap 2024-09-17T16:09:23.826 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:09:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[45258]: 2024-09-17T16:09:23.667+0000 7f16c4e0c640 -1 osd.10 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:09:23.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:23 smithi017 ceph-mon[27571]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-17T16:09:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:23 smithi017 ceph-mon[27571]: osdmap e73: 11 total, 10 up, 11 in 2024-09-17T16:09:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:23 smithi017 ceph-mon[27571]: from='osd.10 [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:23 smithi017 ceph-mon[27571]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:24.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:23 smithi071 ceph-mon[30002]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-17T16:09:24.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:23 smithi071 ceph-mon[30002]: osdmap e73: 11 total, 10 up, 11 in 2024-09-17T16:09:24.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:24.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:23 smithi071 ceph-mon[30002]: from='osd.10 [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:24.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:23 smithi071 ceph-mon[30002]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:24 smithi017 ceph-mon[27571]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:09:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:24 smithi017 ceph-mon[27571]: osdmap e74: 11 total, 10 up, 11 in 2024-09-17T16:09:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:24 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:24 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:24 smithi017 ceph-mon[27571]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:24.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:24 smithi173 ceph-mon[30530]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:09:24.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:24 smithi173 ceph-mon[30530]: osdmap e74: 11 total, 10 up, 11 in 2024-09-17T16:09:24.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:24 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:24.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:24 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:24.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:24 smithi173 ceph-mon[30530]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:25.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:24 smithi071 ceph-mon[30002]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:09:25.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:24 smithi071 ceph-mon[30002]: osdmap e74: 11 total, 10 up, 11 in 2024-09-17T16:09:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:24 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:24 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:24 smithi071 ceph-mon[30002]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-09-17T16:09:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: osd.10 [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719] boot 2024-09-17T16:09:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: osdmap e75: 11 total, 11 up, 11 in 2024-09-17T16:09:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi173 2024-09-17T16:09:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi173 to 7061M 2024-09-17T16:09:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: osd.10 [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719] boot 2024-09-17T16:09:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: osdmap e75: 11 total, 11 up, 11 in 2024-09-17T16:09:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi173 2024-09-17T16:09:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi173 to 7061M 2024-09-17T16:09:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: osd.10 [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719] boot 2024-09-17T16:09:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: osdmap e75: 11 total, 11 up, 11 in 2024-09-17T16:09:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:09:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi173 2024-09-17T16:09:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi173 to 7061M 2024-09-17T16:09:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:26.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:26.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:26.518 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-09-17T16:09:26.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-09-17T16:09:26.792 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:09:27.167 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:27 smithi173 ceph-mon[30530]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:27.167 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:27 smithi173 ceph-mon[30530]: osdmap e76: 11 total, 11 up, 11 in 2024-09-17T16:09:27.167 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:27 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:27.168 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:27 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:27.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:27 smithi071 ceph-mon[30002]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:27 smithi071 ceph-mon[30002]: osdmap e76: 11 total, 11 up, 11 in 2024-09-17T16:09:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:27 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:27 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:27.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:27 smithi017 ceph-mon[27571]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:27 smithi017 ceph-mon[27571]: osdmap e76: 11 total, 11 up, 11 in 2024-09-17T16:09:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:27 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:27 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:28.137 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:28 smithi173 ceph-mon[30530]: osdmap e77: 11 total, 11 up, 11 in 2024-09-17T16:09:28.137 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:09:28.137 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:09:28.137 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:28.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:28 smithi071 ceph-mon[30002]: osdmap e77: 11 total, 11 up, 11 in 2024-09-17T16:09:28.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:09:28.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:09:28.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:28 smithi017 ceph-mon[27571]: osdmap e77: 11 total, 11 up, 11 in 2024-09-17T16:09:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-17T16:09:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-17T16:09:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:29.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:29 smithi071 ceph-mon[30002]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:09:29.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:29 smithi071 ceph-mon[30002]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:29 smithi173 ceph-mon[30530]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:09:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:29 smithi173 ceph-mon[30530]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:29.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:29 smithi017 ceph-mon[27571]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:09:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:29 smithi017 ceph-mon[27571]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:30.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:30 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:30 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/1482671911' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce178d21-377e-40ac-90fa-46009901114d"} : dispatch 2024-09-17T16:09:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:30 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce178d21-377e-40ac-90fa-46009901114d"} : dispatch 2024-09-17T16:09:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:30 smithi071 ceph-mon[30002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce178d21-377e-40ac-90fa-46009901114d"}]': finished 2024-09-17T16:09:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:30 smithi071 ceph-mon[30002]: osdmap e78: 12 total, 11 up, 12 in 2024-09-17T16:09:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:30 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:30 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/1482671911' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce178d21-377e-40ac-90fa-46009901114d"} : dispatch 2024-09-17T16:09:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:30 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce178d21-377e-40ac-90fa-46009901114d"} : dispatch 2024-09-17T16:09:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:30 smithi173 ceph-mon[30530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce178d21-377e-40ac-90fa-46009901114d"}]': finished 2024-09-17T16:09:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:30 smithi173 ceph-mon[30530]: osdmap e78: 12 total, 11 up, 12 in 2024-09-17T16:09:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:30 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:30 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/1482671911' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce178d21-377e-40ac-90fa-46009901114d"} : dispatch 2024-09-17T16:09:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:30 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ce178d21-377e-40ac-90fa-46009901114d"} : dispatch 2024-09-17T16:09:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:30 smithi017 ceph-mon[27571]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce178d21-377e-40ac-90fa-46009901114d"}]': finished 2024-09-17T16:09:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:30 smithi017 ceph-mon[27571]: osdmap e78: 12 total, 11 up, 12 in 2024-09-17T16:09:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:31 smithi071 ceph-mon[30002]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/809431930' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:09:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:31 smithi173 ceph-mon[30530]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/809431930' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:09:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:31 smithi017 ceph-mon[27571]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/809431930' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-17T16:09:33.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:33 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:33 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:33 smithi071 ceph-mon[30002]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:33 smithi071 ceph-mon[30002]: osdmap e79: 12 total, 11 up, 12 in 2024-09-17T16:09:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:33.626 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:33 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:33.626 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:33 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:33.626 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:33 smithi173 ceph-mon[30530]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:33.626 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:33 smithi173 ceph-mon[30530]: osdmap e79: 12 total, 11 up, 12 in 2024-09-17T16:09:33.626 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:33 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:33 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:33 smithi017 ceph-mon[27571]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:33 smithi017 ceph-mon[27571]: osdmap e79: 12 total, 11 up, 12 in 2024-09-17T16:09:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:35.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:35 smithi173 ceph-mon[30530]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:35 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:35 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:35 smithi071 ceph-mon[30002]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:35 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:35 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:35.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:35 smithi017 ceph-mon[27571]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:35 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:35 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:36.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:36 smithi173 ceph-mon[30530]: osdmap e80: 12 total, 11 up, 12 in 2024-09-17T16:09:36.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:36 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:36.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:36 smithi071 ceph-mon[30002]: osdmap e80: 12 total, 11 up, 12 in 2024-09-17T16:09:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:36 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:36.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:36 smithi017 ceph-mon[27571]: osdmap e80: 12 total, 11 up, 12 in 2024-09-17T16:09:36.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:36 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:37.530 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:37 smithi173 ceph-mon[30530]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:37.530 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:37 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-17T16:09:37.530 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:37 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:37 smithi071 ceph-mon[30002]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:37 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-17T16:09:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:37 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:37.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:37 smithi017 ceph-mon[27571]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:37 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-17T16:09:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:37 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:38.393 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:38 smithi173 ceph-mon[30530]: Deploying daemon osd.11 on smithi173 2024-09-17T16:09:38.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:38 smithi071 ceph-mon[30002]: Deploying daemon osd.11 on smithi173 2024-09-17T16:09:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:38 smithi017 ceph-mon[27571]: Deploying daemon osd.11 on smithi173 2024-09-17T16:09:39.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:39 smithi173 ceph-mon[30530]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:39.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:39 smithi071 ceph-mon[30002]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:39.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:39 smithi017 ceph-mon[27571]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:41.548 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:41 smithi173 ceph-mon[30530]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:41.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:41 smithi071 ceph-mon[30002]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:41 smithi017 ceph-mon[27571]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=53,logm=11} crc {auth=3668825664,config=359415709,health=3897446751,kv=134973589,logm=1612166554}) 2024-09-17T16:09:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1621750644}) 2024-09-17T16:09:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380825083}) 2024-09-17T16:09:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4087941921}) 2024-09-17T16:09:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161898033}) 2024-09-17T16:09:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1626615121}) 2024-09-17T16:09:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3234684501}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1216098766}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=679262936}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906372339}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3404157611}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=510489816}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3914503426}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3320915090}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2537598185}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3302929773}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=21} crc {logm=3277742400,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=3129585627,mon_config_key=3589290983}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2418552508}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4107783753}) 2024-09-17T16:09:42.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,nvmeofgw=3,osd_metadata=11,osd_pg_creating=1,osdmap=22} crc {mon_config_key=3212546927,monmap=2143941632,nvmeofgw=598337686,osd_metadata=921320339,osd_pg_creating=4193607613,osdmap=138479772}) 2024-09-17T16:09:42.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=95} crc {osdmap=617599641}) 2024-09-17T16:09:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=53,logm=11} crc {auth=3668825664,config=359415709,health=3897446751,kv=134973589,logm=1612166554}) 2024-09-17T16:09:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1621750644}) 2024-09-17T16:09:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380825083}) 2024-09-17T16:09:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4087941921}) 2024-09-17T16:09:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161898033}) 2024-09-17T16:09:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1626615121}) 2024-09-17T16:09:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3234684501}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1216098766}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=679262936}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906372339}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3404157611}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=510489816}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3914503426}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3320915090}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2537598185}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3302929773}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=21} crc {logm=3277742400,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=3129585627,mon_config_key=3589290983}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2418552508}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4107783753}) 2024-09-17T16:09:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,nvmeofgw=3,osd_metadata=11,osd_pg_creating=1,osdmap=22} crc {mon_config_key=3212546927,monmap=2143941632,nvmeofgw=598337686,osd_metadata=921320339,osd_pg_creating=4193607613,osdmap=138479772}) 2024-09-17T16:09:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=95} crc {osdmap=617599641}) 2024-09-17T16:09:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=53,logm=11} crc {auth=3668825664,config=359415709,health=3897446751,kv=134973589,logm=1612166554}) 2024-09-17T16:09:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1621750644}) 2024-09-17T16:09:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380825083}) 2024-09-17T16:09:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4087941921}) 2024-09-17T16:09:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2161898033}) 2024-09-17T16:09:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1626615121}) 2024-09-17T16:09:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3234684501}) 2024-09-17T16:09:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1216098766}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=679262936}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906372339}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3404157611}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=510489816}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3914503426}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3320915090}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2537598185}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3302929773}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=21} crc {logm=3277742400,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=3129585627,mon_config_key=3589290983}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2418552508}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4107783753}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,nvmeofgw=3,osd_metadata=11,osd_pg_creating=1,osdmap=22} crc {mon_config_key=3212546927,monmap=2143941632,nvmeofgw=598337686,osd_metadata=921320339,osd_pg_creating=4193607613,osdmap=138479772}) 2024-09-17T16:09:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=95} crc {osdmap=617599641}) 2024-09-17T16:09:43.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:43 smithi071 ceph-mon[30002]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:43.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:43 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:43 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:43 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:43.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:43 smithi173 ceph-mon[30530]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:43.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:43 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:43.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:43 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:43.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:43 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:43.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:43 smithi017 ceph-mon[27571]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:43 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:43 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:43 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:45.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:45 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:09:45.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:45 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:09:45.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:45 smithi173 ceph-mon[30530]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:45.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:45 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:09:45.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:45 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:09:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:45 smithi071 ceph-mon[30002]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:45 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:09:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:45 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:09:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:45 smithi017 ceph-mon[27571]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:46.110 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 11 on host 'smithi173' 2024-09-17T16:09:46.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:09:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:46.809 DEBUG:teuthology.orchestra.run.smithi173:osd.11> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.11.service 2024-09-17T16:09:46.811 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-17T16:09:46.812 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T16:09:47.080 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:09:47.310 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:09:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:09:47.292+0000 7fa6fcad6640 -1 osd.11 0 log_to_monitors true 2024-09-17T16:09:47.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:47 smithi017 ceph-mon[27571]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:47.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:47 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:47.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:47 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:47.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:47 smithi017 ceph-mon[27571]: from='osd.11 [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-17T16:09:47.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:47 smithi017 ceph-mon[27571]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-17T16:09:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:47 smithi071 ceph-mon[30002]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:47 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:47 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:47 smithi071 ceph-mon[30002]: from='osd.11 [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-17T16:09:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:47 smithi071 ceph-mon[30002]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-17T16:09:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:47 smithi173 ceph-mon[30530]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:47 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:47 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:47 smithi173 ceph-mon[30530]: from='osd.11 [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-17T16:09:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:47 smithi173 ceph-mon[30530]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-17T16:09:47.830 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:09:48.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:09:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:09:48.333+0000 7fa6f8a59640 -1 osd.11 0 waiting for initial osdmap 2024-09-17T16:09:48.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:09:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:09:48.340+0000 7fa6f4880640 -1 osd.11 82 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:09:48.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:48 smithi173 ceph-mon[30530]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-17T16:09:48.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:48 smithi173 ceph-mon[30530]: osdmap e81: 12 total, 11 up, 12 in 2024-09-17T16:09:48.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:48 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:48.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:48 smithi173 ceph-mon[30530]: from='osd.11 [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:48.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:48 smithi173 ceph-mon[30530]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:48.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:48 smithi173 ceph-mon[30530]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:48.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2559765120' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-17T16:09:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:48 smithi017 ceph-mon[27571]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-17T16:09:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:48 smithi017 ceph-mon[27571]: osdmap e81: 12 total, 11 up, 12 in 2024-09-17T16:09:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:48 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:48 smithi017 ceph-mon[27571]: from='osd.11 [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:48 smithi017 ceph-mon[27571]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:48 smithi017 ceph-mon[27571]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2559765120' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-17T16:09:48.710 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":81,"num_osds":12,"num_up_osds":11,"osd_up_since":1726589364,"num_in_osds":12,"osd_in_since":1726589369,"num_remapped_pgs":0} 2024-09-17T16:09:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:48 smithi071 ceph-mon[30002]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-17T16:09:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:48 smithi071 ceph-mon[30002]: osdmap e81: 12 total, 11 up, 12 in 2024-09-17T16:09:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:48 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:48 smithi071 ceph-mon[30002]: from='osd.11 [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:48 smithi071 ceph-mon[30002]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:09:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:48 smithi071 ceph-mon[30002]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-17T16:09:48.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2559765120' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-17T16:09:49.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:49 smithi173 ceph-mon[30530]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:09:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:49 smithi173 ceph-mon[30530]: osdmap e82: 12 total, 11 up, 12 in 2024-09-17T16:09:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:49 smithi173 ceph-mon[30530]: osd.11 [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377] boot 2024-09-17T16:09:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:49 smithi173 ceph-mon[30530]: osdmap e83: 12 total, 12 up, 12 in 2024-09-17T16:09:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:49.711 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T16:09:49.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:49 smithi017 ceph-mon[27571]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:09:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:49 smithi017 ceph-mon[27571]: osdmap e82: 12 total, 11 up, 12 in 2024-09-17T16:09:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:49 smithi017 ceph-mon[27571]: osd.11 [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377] boot 2024-09-17T16:09:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:49 smithi017 ceph-mon[27571]: osdmap e83: 12 total, 12 up, 12 in 2024-09-17T16:09:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:49 smithi071 ceph-mon[30002]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-09-17T16:09:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:49 smithi071 ceph-mon[30002]: osdmap e82: 12 total, 11 up, 12 in 2024-09-17T16:09:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:49 smithi071 ceph-mon[30002]: osd.11 [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377] boot 2024-09-17T16:09:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:49 smithi071 ceph-mon[30002]: osdmap e83: 12 total, 12 up, 12 in 2024-09-17T16:09:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-17T16:09:49.978 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:09:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:09:50.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:50.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: Detected new or changed devices on smithi173 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: Adjusting osd_memory_target on smithi173 to 5296M 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:50.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.663 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:09:50.663 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:50 smithi017 ceph-mon[27571]: osdmap e84: 12 total, 12 up, 12 in 2024-09-17T16:09:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: Detected new or changed devices on smithi173 2024-09-17T16:09:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: Adjusting osd_memory_target on smithi173 to 5296M 2024-09-17T16:09:50.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:50.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:50.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:09:50.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:50 smithi071 ceph-mon[30002]: osdmap e84: 12 total, 12 up, 12 in 2024-09-17T16:09:50.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:50.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:50.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: Detected new or changed devices on smithi173 2024-09-17T16:09:50.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-17T16:09:50.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: Adjusting osd_memory_target on smithi173 to 5296M 2024-09-17T16:09:50.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:09:50.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:09:50.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:09:50.901 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:09:50.901 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:50 smithi173 ceph-mon[30530]: osdmap e84: 12 total, 12 up, 12 in 2024-09-17T16:09:51.466 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":84,"num_osds":12,"num_up_osds":12,"osd_up_since":1726589389,"num_in_osds":12,"osd_in_since":1726589369,"num_remapped_pgs":0} 2024-09-17T16:09:51.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T16:09:51.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1624750810' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-17T16:09:51.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:51 smithi173 ceph-mon[30530]: osdmap e85: 12 total, 12 up, 12 in 2024-09-17T16:09:51.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1624750810' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-17T16:09:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:51 smithi017 ceph-mon[27571]: osdmap e85: 12 total, 12 up, 12 in 2024-09-17T16:09:51.766 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:09:51.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1624750810' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-17T16:09:51.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:51 smithi071 ceph-mon[30002]: osdmap e85: 12 total, 12 up, 12 in 2024-09-17T16:09:52.423 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:09:52.423 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":85,"fsid":"19341aec-750e-11ef-bceb-c7b262605968","created":"2024-09-17T16:02:46.978716+0000","modified":"2024-09-17T16:09:51.244403+0000","last_up_change":"2024-09-17T16:09:49.234769+0000","last_in_change":"2024-09-17T16:09:29.753083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:06:06.088375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"139c41b7-e38b-42d2-ae0b-70b94041dbfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6803","nonce":3081446730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6805","nonce":3081446730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6809","nonce":3081446730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6807","nonce":3081446730}]},"public_addr":"172.21.15.17:6803/3081446730","cluster_addr":"172.21.15.17:6805/3081446730","heartbeat_back_addr":"172.21.15.17:6809/3081446730","heartbeat_front_addr":"172.21.15.17:6807/3081446730","state":["exists","up"]},{"osd":1,"uuid":"ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6811","nonce":603840826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6813","nonce":603840826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6817","nonce":603840826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6815","nonce":603840826}]},"public_addr":"172.21.15.17:6811/603840826","cluster_addr":"172.21.15.17:6813/603840826","heartbeat_back_addr":"172.21.15.17:6817/603840826","heartbeat_front_addr":"172.21.15.17:6815/603840826","state":["exists","up"]},{"osd":2,"uuid":"efb6c3f1-6157-4581-a39d-bda978aea6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6819","nonce":2706983780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6821","nonce":2706983780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6825","nonce":2706983780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6823","nonce":2706983780}]},"public_addr":"172.21.15.17:6819/2706983780","cluster_addr":"172.21.15.17:6821/2706983780","heartbeat_back_addr":"172.21.15.17:6825/2706983780","heartbeat_front_addr":"172.21.15.17:6823/2706983780","state":["exists","up"]},{"osd":3,"uuid":"37609e4c-d0cb-464f-a3c6-2d6e530149e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6827","nonce":3512825275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6829","nonce":3512825275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6833","nonce":3512825275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6831","nonce":3512825275}]},"public_addr":"172.21.15.17:6827/3512825275","cluster_addr":"172.21.15.17:6829/3512825275","heartbeat_back_addr":"172.21.15.17:6833/3512825275","heartbeat_front_addr":"172.21.15.17:6831/3512825275","state":["exists","up"]},{"osd":4,"uuid":"bd9e473c-bf74-4621-92d0-7031d7db10e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6801","nonce":146827020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6803","nonce":146827020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6807","nonce":146827020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6805","nonce":146827020}]},"public_addr":"172.21.15.71:6801/146827020","cluster_addr":"172.21.15.71:6803/146827020","heartbeat_back_addr":"172.21.15.71:6807/146827020","heartbeat_front_addr":"172.21.15.71:6805/146827020","state":["exists","up"]},{"osd":5,"uuid":"f08464f8-610f-4775-87d0-803b569eb211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6809","nonce":1837749307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6811","nonce":1837749307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6815","nonce":1837749307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6813","nonce":1837749307}]},"public_addr":"172.21.15.71:6809/1837749307","cluster_addr":"172.21.15.71:6811/1837749307","heartbeat_back_addr":"172.21.15.71:6815/1837749307","heartbeat_front_addr":"172.21.15.71:6813/1837749307","state":["exists","up"]},{"osd":6,"uuid":"d05170c6-0258-43da-b81b-cc9cc64b5d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6817","nonce":2384780628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6819","nonce":2384780628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6823","nonce":2384780628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6821","nonce":2384780628}]},"public_addr":"172.21.15.71:6817/2384780628","cluster_addr":"172.21.15.71:6819/2384780628","heartbeat_back_addr":"172.21.15.71:6823/2384780628","heartbeat_front_addr":"172.21.15.71:6821/2384780628","state":["exists","up"]},{"osd":7,"uuid":"0d34db33-5d4d-4761-887d-cf1a63dbf3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6825","nonce":1816911174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6827","nonce":1816911174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6831","nonce":1816911174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6829","nonce":1816911174}]},"public_addr":"172.21.15.71:6825/1816911174","cluster_addr":"172.21.15.71:6827/1816911174","heartbeat_back_addr":"172.21.15.71:6831/1816911174","heartbeat_front_addr":"172.21.15.71:6829/1816911174","state":["exists","up"]},{"osd":8,"uuid":"f419836d-0416-4789-80ad-a45ae2ccf035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6801","nonce":527945865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6803","nonce":527945865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6807","nonce":527945865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6805","nonce":527945865}]},"public_addr":"172.21.15.173:6801/527945865","cluster_addr":"172.21.15.173:6803/527945865","heartbeat_back_addr":"172.21.15.173:6807/527945865","heartbeat_front_addr":"172.21.15.173:6805/527945865","state":["exists","up"]},{"osd":9,"uuid":"9dcaa1d8-e921-427c-8a2c-a368658a5bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6809","nonce":3589162352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6811","nonce":3589162352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6815","nonce":3589162352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6813","nonce":3589162352}]},"public_addr":"172.21.15.173:6809/3589162352","cluster_addr":"172.21.15.173:6811/3589162352","heartbeat_back_addr":"172.21.15.173:6815/3589162352","heartbeat_front_addr":"172.21.15.173:6813/3589162352","state":["exists","up"]},{"osd":10,"uuid":"ee08b73f-f02b-4795-87bc-ab559b7faf61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6817","nonce":2428879719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6819","nonce":2428879719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6823","nonce":2428879719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6821","nonce":2428879719}]},"public_addr":"172.21.15.173:6817/2428879719","cluster_addr":"172.21.15.173:6819/2428879719","heartbeat_back_addr":"172.21.15.173:6823/2428879719","heartbeat_front_addr":"172.21.15.173:6821/2428879719","state":["exists","up"]},{"osd":11,"uuid":"ce178d21-377e-40ac-90fa-46009901114d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6825","nonce":1731820377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6827","nonce":1731820377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6831","nonce":1731820377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6829","nonce":1731820377}]},"public_addr":"172.21.15.173:6825/1731820377","cluster_addr":"172.21.15.173:6827/1731820377","heartbeat_back_addr":"172.21.15.173:6831/1731820377","heartbeat_front_addr":"172.21.15.173:6829/1731820377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:08:39.750823+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:16.759797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:34.291557+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:26.152381+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:29.640349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:08:47.973112+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:04.350666+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:31.517857+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:46.481251+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:08:58.708821+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:23.410794+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:48.266458+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/3205448177":"2024-09-18T16:03:49.423969+0000","172.21.15.17:6801/250768364":"2024-09-18T16:03:49.423969+0000","172.21.15.17:6800/250768364":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/394168293":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/4225461221":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/3672343901":"2024-09-18T16:03:21.713930+0000","172.21.15.17:0/419827434":"2024-09-18T16:03:21.713930+0000","172.21.15.17:6801/365103448":"2024-09-18T16:03:21.713930+0000","172.21.15.17:6800/365103448":"2024-09-18T16:03:21.713930+0000","172.21.15.17:0/2688765754":"2024-09-18T16:03:21.713930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:09:52.721 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:52 smithi017 ceph-mon[27571]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-09-17T16:09:52.722 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3744165351' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:09:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:52 smithi071 ceph-mon[30002]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-09-17T16:09:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3744165351' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:09:52.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:52 smithi173 ceph-mon[30530]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-09-17T16:09:52.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3744165351' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:09:53.012 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T16:09:53.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-17T16:09:53.270 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:09:53.915 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 1 2024-09-17T16:09:54.560 INFO:tasks.cephadm:Setting up client nodes... 2024-09-17T16:09:54.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T16:09:54.816 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:09:54.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:54 smithi017 ceph-mon[27571]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T16:09:54.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/511219617' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-17T16:09:55.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:54 smithi071 ceph-mon[30002]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T16:09:55.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/511219617' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-17T16:09:55.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:54 smithi173 ceph-mon[30530]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-17T16:09:55.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/511219617' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-17T16:09:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2024-09-17T16:09:55.554 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQDTqelmD+DdIBAATEQOBzSJKW5QwkCxJf5MqQ== 2024-09-17T16:09:55.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:55 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:55.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:55 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:55.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:55 smithi017 ceph-mon[27571]: osdmap e86: 12 total, 12 up, 12 in 2024-09-17T16:09:55.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1900433308' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:09:55.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1900433308' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T16:09:56.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:55 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:56.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:55 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:56.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:55 smithi071 ceph-mon[30002]: osdmap e86: 12 total, 12 up, 12 in 2024-09-17T16:09:56.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1900433308' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:09:56.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1900433308' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T16:09:56.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:55 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:56.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:55 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:56.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:55 smithi173 ceph-mon[30530]: osdmap e86: 12 total, 12 up, 12 in 2024-09-17T16:09:56.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1900433308' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:09:56.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1900433308' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T16:09:56.294 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-17T16:09:56.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-17T16:09:56.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T16:09:56.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T16:09:56.591 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:09:56.966 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:56 smithi071 ceph-mon[30002]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:09:56.966 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:56 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:09:56.966 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:56 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:09:57.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:56 smithi173 ceph-mon[30530]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:09:57.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:56 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:09:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:56 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:09:57.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:56 smithi017 ceph-mon[27571]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:09:57.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:56 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:09:57.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:56 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:09:57.324 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2024-09-17T16:09:57.325 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDVqelmOAf6EhAAYInq3VeaSpdP/uCUBJq7YQ== 2024-09-17T16:09:58.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:57 smithi071 ceph-mon[30002]: osdmap e87: 12 total, 12 up, 12 in 2024-09-17T16:09:58.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.71:0/3753119486' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:09:58.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:57 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:09:58.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:57 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T16:09:58.033 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-09-17T16:09:58.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-17T16:09:58.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-17T16:09:58.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T16:09:58.099 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:57 smithi173 ceph-mon[30530]: osdmap e87: 12 total, 12 up, 12 in 2024-09-17T16:09:58.100 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.71:0/3753119486' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:09:58.100 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:57 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:09:58.100 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:57 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T16:09:58.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:57 smithi017 ceph-mon[27571]: osdmap e87: 12 total, 12 up, 12 in 2024-09-17T16:09:58.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.71:0/3753119486' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:09:58.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:57 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:09:58.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:57 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T16:09:58.329 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:09:59.005 INFO:teuthology.orchestra.run.smithi173.stdout:[client.2] 2024-09-17T16:09:59.006 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDWqelmcayXOxAAxsQrpUuRDoAlrz/MHdc7vw== 2024-09-17T16:09:59.006 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:58 smithi173 ceph-mon[30530]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-17T16:09:59.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:58 smithi071 ceph-mon[30002]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-17T16:09:59.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:58 smithi017 ceph-mon[27571]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-17T16:09:59.692 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-09-17T16:09:59.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-09-17T16:09:59.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-17T16:09:59.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-17T16:10:00.039 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:00.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.173:0/3468932327' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:10:00.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:59 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:10:00.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:09:59 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T16:10:00.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.173:0/3468932327' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:10:00.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:59 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:10:00.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:09:59 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T16:10:00.160 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.173:0/3468932327' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:10:00.160 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:59 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-17T16:10:00.160 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:09:59 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T16:10:00.738 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:10:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:00 smithi017 ceph-mon[27571]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:00 smithi017 ceph-mon[27571]: overall HEALTH_OK 2024-09-17T16:10:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3226063539' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-17T16:10:01.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:00 smithi071 ceph-mon[30002]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:01.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:00 smithi071 ceph-mon[30002]: overall HEALTH_OK 2024-09-17T16:10:01.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3226063539' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-17T16:10:01.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:00 smithi173 ceph-mon[30530]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:00 smithi173 ceph-mon[30530]: overall HEALTH_OK 2024-09-17T16:10:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3226063539' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-17T16:10:01.451 INFO:teuthology.orchestra.run.smithi017.stdout:[{"version":26,"timestamp":"2024-09-17T16:09:49.479847+0000","name":"","changes":[{"name":"osd/host:smithi173/osd_memory_target","previous_value":"7404391628","new_value":"5553293721"}]}] 2024-09-17T16:10:01.451 INFO:tasks.ceph_manager:config epoch is 26 2024-09-17T16:10:01.452 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T16:10:01.452 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-17T16:10:01.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-17T16:10:01.713 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:02.398 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:10:03.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:02 smithi017 ceph-mon[27571]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:03.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/770414125' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-17T16:10:03.037 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":15,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4138576009},{"type":"v1","addr":"172.21.15.17:6801","nonce":4138576009}]},"active_addr":"172.21.15.17:6801/4138576009","active_change":"2024-09-17T16:03:49.424113+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24104,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.17:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":3458652294}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":2484390953}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":2003326067}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":2249401310}]}]} 2024-09-17T16:10:03.043 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-17T16:10:03.043 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-17T16:10:03.043 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T16:10:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:02 smithi071 ceph-mon[30002]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/770414125' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-17T16:10:03.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:02 smithi173 ceph-mon[30530]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/770414125' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-17T16:10:03.297 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:03.913 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:10:03.913 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":87,"fsid":"19341aec-750e-11ef-bceb-c7b262605968","created":"2024-09-17T16:02:46.978716+0000","modified":"2024-09-17T16:09:56.759300+0000","last_up_change":"2024-09-17T16:09:49.234769+0000","last_in_change":"2024-09-17T16:09:29.753083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:06:06.088375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"139c41b7-e38b-42d2-ae0b-70b94041dbfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6803","nonce":3081446730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6805","nonce":3081446730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6809","nonce":3081446730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6807","nonce":3081446730}]},"public_addr":"172.21.15.17:6803/3081446730","cluster_addr":"172.21.15.17:6805/3081446730","heartbeat_back_addr":"172.21.15.17:6809/3081446730","heartbeat_front_addr":"172.21.15.17:6807/3081446730","state":["exists","up"]},{"osd":1,"uuid":"ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6811","nonce":603840826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6813","nonce":603840826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6817","nonce":603840826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6815","nonce":603840826}]},"public_addr":"172.21.15.17:6811/603840826","cluster_addr":"172.21.15.17:6813/603840826","heartbeat_back_addr":"172.21.15.17:6817/603840826","heartbeat_front_addr":"172.21.15.17:6815/603840826","state":["exists","up"]},{"osd":2,"uuid":"efb6c3f1-6157-4581-a39d-bda978aea6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6819","nonce":2706983780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6821","nonce":2706983780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6825","nonce":2706983780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6823","nonce":2706983780}]},"public_addr":"172.21.15.17:6819/2706983780","cluster_addr":"172.21.15.17:6821/2706983780","heartbeat_back_addr":"172.21.15.17:6825/2706983780","heartbeat_front_addr":"172.21.15.17:6823/2706983780","state":["exists","up"]},{"osd":3,"uuid":"37609e4c-d0cb-464f-a3c6-2d6e530149e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6827","nonce":3512825275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6829","nonce":3512825275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6833","nonce":3512825275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6831","nonce":3512825275}]},"public_addr":"172.21.15.17:6827/3512825275","cluster_addr":"172.21.15.17:6829/3512825275","heartbeat_back_addr":"172.21.15.17:6833/3512825275","heartbeat_front_addr":"172.21.15.17:6831/3512825275","state":["exists","up"]},{"osd":4,"uuid":"bd9e473c-bf74-4621-92d0-7031d7db10e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6801","nonce":146827020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6803","nonce":146827020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6807","nonce":146827020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6805","nonce":146827020}]},"public_addr":"172.21.15.71:6801/146827020","cluster_addr":"172.21.15.71:6803/146827020","heartbeat_back_addr":"172.21.15.71:6807/146827020","heartbeat_front_addr":"172.21.15.71:6805/146827020","state":["exists","up"]},{"osd":5,"uuid":"f08464f8-610f-4775-87d0-803b569eb211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6809","nonce":1837749307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6811","nonce":1837749307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6815","nonce":1837749307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6813","nonce":1837749307}]},"public_addr":"172.21.15.71:6809/1837749307","cluster_addr":"172.21.15.71:6811/1837749307","heartbeat_back_addr":"172.21.15.71:6815/1837749307","heartbeat_front_addr":"172.21.15.71:6813/1837749307","state":["exists","up"]},{"osd":6,"uuid":"d05170c6-0258-43da-b81b-cc9cc64b5d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6817","nonce":2384780628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6819","nonce":2384780628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6823","nonce":2384780628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6821","nonce":2384780628}]},"public_addr":"172.21.15.71:6817/2384780628","cluster_addr":"172.21.15.71:6819/2384780628","heartbeat_back_addr":"172.21.15.71:6823/2384780628","heartbeat_front_addr":"172.21.15.71:6821/2384780628","state":["exists","up"]},{"osd":7,"uuid":"0d34db33-5d4d-4761-887d-cf1a63dbf3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6825","nonce":1816911174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6827","nonce":1816911174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6831","nonce":1816911174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6829","nonce":1816911174}]},"public_addr":"172.21.15.71:6825/1816911174","cluster_addr":"172.21.15.71:6827/1816911174","heartbeat_back_addr":"172.21.15.71:6831/1816911174","heartbeat_front_addr":"172.21.15.71:6829/1816911174","state":["exists","up"]},{"osd":8,"uuid":"f419836d-0416-4789-80ad-a45ae2ccf035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6801","nonce":527945865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6803","nonce":527945865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6807","nonce":527945865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6805","nonce":527945865}]},"public_addr":"172.21.15.173:6801/527945865","cluster_addr":"172.21.15.173:6803/527945865","heartbeat_back_addr":"172.21.15.173:6807/527945865","heartbeat_front_addr":"172.21.15.173:6805/527945865","state":["exists","up"]},{"osd":9,"uuid":"9dcaa1d8-e921-427c-8a2c-a368658a5bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6809","nonce":3589162352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6811","nonce":3589162352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6815","nonce":3589162352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6813","nonce":3589162352}]},"public_addr":"172.21.15.173:6809/3589162352","cluster_addr":"172.21.15.173:6811/3589162352","heartbeat_back_addr":"172.21.15.173:6815/3589162352","heartbeat_front_addr":"172.21.15.173:6813/3589162352","state":["exists","up"]},{"osd":10,"uuid":"ee08b73f-f02b-4795-87bc-ab559b7faf61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6817","nonce":2428879719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6819","nonce":2428879719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6823","nonce":2428879719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6821","nonce":2428879719}]},"public_addr":"172.21.15.173:6817/2428879719","cluster_addr":"172.21.15.173:6819/2428879719","heartbeat_back_addr":"172.21.15.173:6823/2428879719","heartbeat_front_addr":"172.21.15.173:6821/2428879719","state":["exists","up"]},{"osd":11,"uuid":"ce178d21-377e-40ac-90fa-46009901114d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6825","nonce":1731820377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6827","nonce":1731820377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6831","nonce":1731820377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6829","nonce":1731820377}]},"public_addr":"172.21.15.173:6825/1731820377","cluster_addr":"172.21.15.173:6827/1731820377","heartbeat_back_addr":"172.21.15.173:6831/1731820377","heartbeat_front_addr":"172.21.15.173:6829/1731820377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:53.858721+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:16.759797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:34.291557+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:26.152381+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:29.640349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:55.916063+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:04.350666+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:31.517857+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:46.481251+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:08:58.708821+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:23.410794+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:48.266458+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/3205448177":"2024-09-18T16:03:49.423969+0000","172.21.15.17:6801/250768364":"2024-09-18T16:03:49.423969+0000","172.21.15.17:6800/250768364":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/394168293":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/4225461221":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/3672343901":"2024-09-18T16:03:21.713930+0000","172.21.15.17:0/419827434":"2024-09-18T16:03:21.713930+0000","172.21.15.17:6801/365103448":"2024-09-18T16:03:21.713930+0000","172.21.15.17:6800/365103448":"2024-09-18T16:03:21.713930+0000","172.21.15.17:0/2688765754":"2024-09-18T16:03:21.713930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:10:04.569 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-17T16:10:04.569 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T16:10:04.824 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:04.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:04 smithi017 ceph-mon[27571]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:04.933 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1957485236' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:10:05.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:04 smithi071 ceph-mon[30002]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:05.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1957485236' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:10:05.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:04 smithi173 ceph-mon[30530]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1957485236' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:10:05.458 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:10:05.458 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":87,"fsid":"19341aec-750e-11ef-bceb-c7b262605968","created":"2024-09-17T16:02:46.978716+0000","modified":"2024-09-17T16:09:56.759300+0000","last_up_change":"2024-09-17T16:09:49.234769+0000","last_in_change":"2024-09-17T16:09:29.753083+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:06:06.088375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"139c41b7-e38b-42d2-ae0b-70b94041dbfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6803","nonce":3081446730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6805","nonce":3081446730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6809","nonce":3081446730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6807","nonce":3081446730}]},"public_addr":"172.21.15.17:6803/3081446730","cluster_addr":"172.21.15.17:6805/3081446730","heartbeat_back_addr":"172.21.15.17:6809/3081446730","heartbeat_front_addr":"172.21.15.17:6807/3081446730","state":["exists","up"]},{"osd":1,"uuid":"ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6811","nonce":603840826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6813","nonce":603840826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6817","nonce":603840826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6815","nonce":603840826}]},"public_addr":"172.21.15.17:6811/603840826","cluster_addr":"172.21.15.17:6813/603840826","heartbeat_back_addr":"172.21.15.17:6817/603840826","heartbeat_front_addr":"172.21.15.17:6815/603840826","state":["exists","up"]},{"osd":2,"uuid":"efb6c3f1-6157-4581-a39d-bda978aea6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6819","nonce":2706983780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6821","nonce":2706983780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6825","nonce":2706983780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6823","nonce":2706983780}]},"public_addr":"172.21.15.17:6819/2706983780","cluster_addr":"172.21.15.17:6821/2706983780","heartbeat_back_addr":"172.21.15.17:6825/2706983780","heartbeat_front_addr":"172.21.15.17:6823/2706983780","state":["exists","up"]},{"osd":3,"uuid":"37609e4c-d0cb-464f-a3c6-2d6e530149e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6827","nonce":3512825275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6829","nonce":3512825275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6833","nonce":3512825275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6831","nonce":3512825275}]},"public_addr":"172.21.15.17:6827/3512825275","cluster_addr":"172.21.15.17:6829/3512825275","heartbeat_back_addr":"172.21.15.17:6833/3512825275","heartbeat_front_addr":"172.21.15.17:6831/3512825275","state":["exists","up"]},{"osd":4,"uuid":"bd9e473c-bf74-4621-92d0-7031d7db10e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6801","nonce":146827020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6803","nonce":146827020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6807","nonce":146827020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6805","nonce":146827020}]},"public_addr":"172.21.15.71:6801/146827020","cluster_addr":"172.21.15.71:6803/146827020","heartbeat_back_addr":"172.21.15.71:6807/146827020","heartbeat_front_addr":"172.21.15.71:6805/146827020","state":["exists","up"]},{"osd":5,"uuid":"f08464f8-610f-4775-87d0-803b569eb211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6809","nonce":1837749307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6811","nonce":1837749307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6815","nonce":1837749307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":1837749307},{"type":"v1","addr":"172.21.15.71:6813","nonce":1837749307}]},"public_addr":"172.21.15.71:6809/1837749307","cluster_addr":"172.21.15.71:6811/1837749307","heartbeat_back_addr":"172.21.15.71:6815/1837749307","heartbeat_front_addr":"172.21.15.71:6813/1837749307","state":["exists","up"]},{"osd":6,"uuid":"d05170c6-0258-43da-b81b-cc9cc64b5d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6817","nonce":2384780628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6819","nonce":2384780628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6823","nonce":2384780628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6821","nonce":2384780628}]},"public_addr":"172.21.15.71:6817/2384780628","cluster_addr":"172.21.15.71:6819/2384780628","heartbeat_back_addr":"172.21.15.71:6823/2384780628","heartbeat_front_addr":"172.21.15.71:6821/2384780628","state":["exists","up"]},{"osd":7,"uuid":"0d34db33-5d4d-4761-887d-cf1a63dbf3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6825","nonce":1816911174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6827","nonce":1816911174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6831","nonce":1816911174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1816911174},{"type":"v1","addr":"172.21.15.71:6829","nonce":1816911174}]},"public_addr":"172.21.15.71:6825/1816911174","cluster_addr":"172.21.15.71:6827/1816911174","heartbeat_back_addr":"172.21.15.71:6831/1816911174","heartbeat_front_addr":"172.21.15.71:6829/1816911174","state":["exists","up"]},{"osd":8,"uuid":"f419836d-0416-4789-80ad-a45ae2ccf035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6801","nonce":527945865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6803","nonce":527945865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6807","nonce":527945865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6805","nonce":527945865}]},"public_addr":"172.21.15.173:6801/527945865","cluster_addr":"172.21.15.173:6803/527945865","heartbeat_back_addr":"172.21.15.173:6807/527945865","heartbeat_front_addr":"172.21.15.173:6805/527945865","state":["exists","up"]},{"osd":9,"uuid":"9dcaa1d8-e921-427c-8a2c-a368658a5bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6809","nonce":3589162352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6811","nonce":3589162352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6815","nonce":3589162352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6813","nonce":3589162352}]},"public_addr":"172.21.15.173:6809/3589162352","cluster_addr":"172.21.15.173:6811/3589162352","heartbeat_back_addr":"172.21.15.173:6815/3589162352","heartbeat_front_addr":"172.21.15.173:6813/3589162352","state":["exists","up"]},{"osd":10,"uuid":"ee08b73f-f02b-4795-87bc-ab559b7faf61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6817","nonce":2428879719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6819","nonce":2428879719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6823","nonce":2428879719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2428879719},{"type":"v1","addr":"172.21.15.173:6821","nonce":2428879719}]},"public_addr":"172.21.15.173:6817/2428879719","cluster_addr":"172.21.15.173:6819/2428879719","heartbeat_back_addr":"172.21.15.173:6823/2428879719","heartbeat_front_addr":"172.21.15.173:6821/2428879719","state":["exists","up"]},{"osd":11,"uuid":"ce178d21-377e-40ac-90fa-46009901114d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6825","nonce":1731820377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6827","nonce":1731820377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6831","nonce":1731820377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6829","nonce":1731820377}]},"public_addr":"172.21.15.173:6825/1731820377","cluster_addr":"172.21.15.173:6827/1731820377","heartbeat_back_addr":"172.21.15.173:6831/1731820377","heartbeat_front_addr":"172.21.15.173:6829/1731820377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:53.858721+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:16.759797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:34.291557+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:26.152381+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:29.640349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:55.916063+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:04.350666+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:31.517857+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:46.481251+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:08:58.708821+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:23.410794+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:09:48.266458+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/3205448177":"2024-09-18T16:03:49.423969+0000","172.21.15.17:6801/250768364":"2024-09-18T16:03:49.423969+0000","172.21.15.17:6800/250768364":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/394168293":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/4225461221":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/3672343901":"2024-09-18T16:03:21.713930+0000","172.21.15.17:0/419827434":"2024-09-18T16:03:21.713930+0000","172.21.15.17:6801/365103448":"2024-09-18T16:03:21.713930+0000","172.21.15.17:6800/365103448":"2024-09-18T16:03:21.713930+0000","172.21.15.17:0/2688765754":"2024-09-18T16:03:21.713930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:10:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:10:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:10:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:10:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:10:06.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:10:06.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:10:06.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:06.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:06.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:06.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:06.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:06.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2250640552' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:10:06.065 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:10:06.065 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:06.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:06.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2250640552' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:10:06.094 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-17T16:10:06.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-17T16:10:06.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-17T16:10:06.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-17T16:10:06.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-17T16:10:06.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-17T16:10:06.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-17T16:10:06.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-17T16:10:06.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-17T16:10:06.098 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-17T16:10:06.098 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-17T16:10:06.099 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-17T16:10:06.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:06.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:06.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:06.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2250640552' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:10:06.891 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.898 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.900 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.903 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.904 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.904 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.908 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.908 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.911 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.911 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.919 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:06.980 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:07.420 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:07 smithi017 ceph-mon[27571]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:07 smithi071 ceph-mon[30002]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:07.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:07 smithi173 ceph-mon[30530]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:09.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:10:09.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:10:09.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:10:09.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:09.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:09.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:09.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:09.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:10:09.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:10:09.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:10:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:09.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:10:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:10:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:10:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:11.401 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:11 smithi017 ceph-mon[27571]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:11.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:11 smithi071 ceph-mon[30002]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:11.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:11 smithi173 ceph-mon[30530]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:12.308 INFO:teuthology.orchestra.run.smithi017.stdout:231928234010 2024-09-17T16:10:12.308 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-17T16:10:12.461 INFO:teuthology.orchestra.run.smithi017.stdout:34359738430 2024-09-17T16:10:12.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-17T16:10:13.373 INFO:teuthology.orchestra.run.smithi017.stdout:111669149742 2024-09-17T16:10:13.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-17T16:10:13.523 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:13 smithi017 ceph-mon[27571]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:13 smithi071 ceph-mon[30002]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:13.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:13 smithi173 ceph-mon[30530]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:14.113 INFO:teuthology.orchestra.run.smithi017.stdout:197568495648 2024-09-17T16:10:14.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-17T16:10:15.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:15 smithi017 ceph-mon[27571]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:15.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:15 smithi071 ceph-mon[30002]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:15.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:15 smithi173 ceph-mon[30530]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:15.708 INFO:teuthology.orchestra.run.smithi017.stdout:261993005077 2024-09-17T16:10:15.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-17T16:10:15.985 INFO:teuthology.orchestra.run.smithi017.stdout:77309411380 2024-09-17T16:10:15.986 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-17T16:10:16.818 INFO:teuthology.orchestra.run.smithi017.stdout:356482285575 2024-09-17T16:10:16.818 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-17T16:10:17.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:17 smithi017 ceph-mon[27571]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:17.394 INFO:teuthology.orchestra.run.smithi017.stdout:133143986218 2024-09-17T16:10:17.395 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-17T16:10:17.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:17 smithi071 ceph-mon[30002]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:17.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:17 smithi173 ceph-mon[30530]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:18.465 INFO:teuthology.orchestra.run.smithi017.stdout:55834574906 2024-09-17T16:10:18.465 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-17T16:10:18.643 INFO:teuthology.orchestra.run.smithi017.stdout:167503724579 2024-09-17T16:10:18.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-17T16:10:19.111 INFO:teuthology.orchestra.run.smithi017.stdout:296352743441 2024-09-17T16:10:19.112 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-17T16:10:19.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:19 smithi017 ceph-mon[27571]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:19.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:19 smithi071 ceph-mon[30002]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:19.619 INFO:teuthology.orchestra.run.smithi017.stdout:322122547212 2024-09-17T16:10:19.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-17T16:10:19.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:19 smithi173 ceph-mon[30530]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:19.681 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:19.689 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:19.691 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:19.702 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:19.703 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:19.709 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:19.710 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:19.711 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:19.711 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:20.900 INFO:teuthology.orchestra.run.smithi017.stdout:197568495649 2024-09-17T16:10:21.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:21 smithi017 ceph-mon[27571]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:21.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3156467045' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-17T16:10:21.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1576016119' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-17T16:10:21.551 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495648 got 197568495649 for osd.6 2024-09-17T16:10:21.551 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:21.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:21 smithi071 ceph-mon[30002]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:21.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3156467045' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-17T16:10:21.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1576016119' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-17T16:10:21.633 INFO:teuthology.orchestra.run.smithi017.stdout:111669149743 2024-09-17T16:10:21.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:21 smithi173 ceph-mon[30530]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:21.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3156467045' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-17T16:10:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1576016119' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-17T16:10:22.233 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149742 got 111669149743 for osd.3 2024-09-17T16:10:22.233 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:22.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1283930269' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-17T16:10:22.464 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:22.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1283930269' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-17T16:10:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1283930269' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-17T16:10:23.098 INFO:teuthology.orchestra.run.smithi017.stdout:231928234013 2024-09-17T16:10:23.423 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:23 smithi017 ceph-mon[27571]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:23.423 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2314015928' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-17T16:10:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:23 smithi071 ceph-mon[30002]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2314015928' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-17T16:10:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:23 smithi173 ceph-mon[30530]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2314015928' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-17T16:10:23.836 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234010 got 231928234013 for osd.7 2024-09-17T16:10:23.836 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:24.386 INFO:teuthology.orchestra.run.smithi017.stdout:356482285577 2024-09-17T16:10:24.583 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:24.940 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482285575 got 356482285577 for osd.11 2024-09-17T16:10:24.940 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:25.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:25 smithi017 ceph-mon[27571]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:25 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:10:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:25 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:10:25.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:25 smithi071 ceph-mon[30002]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:25 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:10:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:25 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:10:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:25 smithi173 ceph-mon[30530]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:25 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:10:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:25 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:10:25.908 INFO:teuthology.orchestra.run.smithi017.stdout:261993005080 2024-09-17T16:10:26.092 INFO:teuthology.orchestra.run.smithi017.stdout:133143986220 2024-09-17T16:10:26.437 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:26 smithi017 ceph-mon[27571]: osdmap e88: 12 total, 12 up, 12 in 2024-09-17T16:10:26.437 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4136809094' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-17T16:10:26.438 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2224772838' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-17T16:10:26.438 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1807138840' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-17T16:10:26.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:26 smithi071 ceph-mon[30002]: osdmap e88: 12 total, 12 up, 12 in 2024-09-17T16:10:26.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4136809094' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-17T16:10:26.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2224772838' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-17T16:10:26.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1807138840' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-17T16:10:26.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:26 smithi173 ceph-mon[30530]: osdmap e88: 12 total, 12 up, 12 in 2024-09-17T16:10:26.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4136809094' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-17T16:10:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2224772838' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-17T16:10:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1807138840' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-17T16:10:26.836 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986218 got 133143986220 for osd.4 2024-09-17T16:10:26.836 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:26.929 INFO:teuthology.orchestra.run.smithi017.stdout:34359738433 2024-09-17T16:10:27.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:27 smithi017 ceph-mon[27571]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:27.561 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738430 got 34359738433 for osd.0 2024-09-17T16:10:27.561 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:27 smithi071 ceph-mon[30002]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:27.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:27 smithi173 ceph-mon[30530]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:28.259 INFO:teuthology.orchestra.run.smithi017.stdout:296352743444 2024-09-17T16:10:28.451 INFO:teuthology.orchestra.run.smithi017.stdout:77309411384 2024-09-17T16:10:28.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3834738425' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-17T16:10:28.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1245437870' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-17T16:10:28.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3834738425' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-17T16:10:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1245437870' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-17T16:10:28.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3834738425' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-17T16:10:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1245437870' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-17T16:10:29.179 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743441 got 296352743444 for osd.9 2024-09-17T16:10:29.179 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:29.268 INFO:teuthology.orchestra.run.smithi017.stdout:167503724583 2024-09-17T16:10:29.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:29 smithi017 ceph-mon[27571]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3158876754' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-17T16:10:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:29 smithi071 ceph-mon[30002]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3158876754' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-17T16:10:29.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:29 smithi173 ceph-mon[30530]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:29.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3158876754' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-17T16:10:29.869 INFO:teuthology.orchestra.run.smithi017.stdout:55834574909 2024-09-17T16:10:30.059 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411380 got 77309411384 for osd.2 2024-09-17T16:10:30.059 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:30.317 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:30.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2167333908' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-17T16:10:30.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2167333908' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-17T16:10:30.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2167333908' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-17T16:10:30.794 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005077 got 261993005080 for osd.8 2024-09-17T16:10:30.794 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:31.415 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574906 got 55834574909 for osd.1 2024-09-17T16:10:31.416 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:31 smithi071 ceph-mon[30002]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:31 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:10:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:31 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:10:31.564 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:31 smithi017 ceph-mon[27571]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:31.564 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:31 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:10:31.564 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:31 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:10:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:31 smithi173 ceph-mon[30530]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:31 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:10:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:31 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:10:32.097 INFO:teuthology.orchestra.run.smithi017.stdout:322122547215 2024-09-17T16:10:32.437 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724579 got 167503724583 for osd.5 2024-09-17T16:10:32.438 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:32.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:32 smithi071 ceph-mon[30002]: osdmap e89: 12 total, 12 up, 12 in 2024-09-17T16:10:32.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/67952350' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-17T16:10:32.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:32 smithi173 ceph-mon[30530]: osdmap e89: 12 total, 12 up, 12 in 2024-09-17T16:10:32.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/67952350' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-17T16:10:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:32 smithi017 ceph-mon[27571]: osdmap e89: 12 total, 12 up, 12 in 2024-09-17T16:10:32.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/67952350' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-17T16:10:33.131 INFO:tasks.cephadm.ceph_manager.ceph:need seq 322122547212 got 322122547215 for osd.10 2024-09-17T16:10:33.132 DEBUG:teuthology.parallel:result is None 2024-09-17T16:10:33.132 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-17T16:10:33.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:10:33.393 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:33.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:33 smithi017 ceph-mon[27571]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:33.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:33 smithi071 ceph-mon[30002]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:33.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:33 smithi173 ceph-mon[30530]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:34.086 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:10:34.086 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:10:34.851 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-17T16:10:33.717653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":341932,"kb_used_data":9276,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599892,"statfs":{"total":1151940427776,"available":1151590289408,"internally_reserved":0,"allocated":9498624,"data_stored":5635639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":340374842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.449529"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":120,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T16:10:27.892921+0000","last_change":"2024-09-17T16:09:51.306176+0000","last_active":"2024-09-17T16:10:27.892921+0000","last_peered":"2024-09-17T16:10:27.892921+0000","last_clean":"2024-09-17T16:10:27.892921+0000","last_became_active":"2024-09-17T16:09:51.301092+0000","last_became_peered":"2024-09-17T16:09:51.301092+0000","last_unstale":"2024-09-17T16:10:27.892921+0000","last_undegraded":"2024-09-17T16:10:27.892921+0000","last_fullsized":"2024-09-17T16:10:27.892921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:09:43.462563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:09:43.462563+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:01.928744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285580,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28180,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716972,"statfs":{"total":95995035648,"available":95966179328,"internally_reserved":0,"allocated":1179648,"data_stored":859552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547216,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1191936,"data_stored":866949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495652,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1191936,"data_stored":866949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724584,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986222,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713320,"statfs":{"total":95995035648,"available":95962439680,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713320,"statfs":{"total":95995035648,"available":95962439680,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:10:34.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:10:35.112 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:35 smithi017 ceph-mon[27571]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:35 smithi017 ceph-mon[27571]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:35 smithi071 ceph-mon[30002]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:35 smithi071 ceph-mon[30002]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:35.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:35 smithi173 ceph-mon[30530]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:35 smithi173 ceph-mon[30530]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:35.798 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:10:35.798 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:10:36.584 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-17T16:10:35.717938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":342016,"kb_used_data":9360,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599808,"statfs":{"total":1151940427776,"available":1151590203392,"internally_reserved":0,"allocated":9584640,"data_stored":5687278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":340374842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.011942"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":120,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T16:10:27.892921+0000","last_change":"2024-09-17T16:09:51.306176+0000","last_active":"2024-09-17T16:10:27.892921+0000","last_peered":"2024-09-17T16:10:27.892921+0000","last_clean":"2024-09-17T16:10:27.892921+0000","last_became_active":"2024-09-17T16:09:51.301092+0000","last_became_peered":"2024-09-17T16:09:51.301092+0000","last_unstale":"2024-09-17T16:10:27.892921+0000","last_undegraded":"2024-09-17T16:10:27.892921+0000","last_fullsized":"2024-09-17T16:10:27.892921+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:09:43.462563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:09:43.462563+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:01.928744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285581,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1204224,"data_stored":874306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547217,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":345398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":345398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28192,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716960,"statfs":{"total":95995035648,"available":95966167040,"internally_reserved":0,"allocated":1191936,"data_stored":866949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495653,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1204224,"data_stored":874326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724585,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":345398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":345398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713320,"statfs":{"total":95995035648,"available":95962439680,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713320,"statfs":{"total":95995035648,"available":95962439680,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":659456,"data_stored":338021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:10:36.585 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-17T16:10:36.585 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T16:10:36.585 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-17T16:10:36.585 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-17T16:10:36.847 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:37 smithi017 ceph-mon[27571]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:37 smithi017 ceph-mon[27571]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:37.553 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:10:37.553 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T16:10:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:37 smithi173 ceph-mon[30530]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:37 smithi173 ceph-mon[30530]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:37 smithi071 ceph-mon[30002]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:37 smithi071 ceph-mon[30002]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:38.343 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-17T16:10:38.344 INFO:tasks.cephadm:Setup complete, yielding 2024-09-17T16:10:38.344 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-17T16:10:38.354 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-17T16:10:38.355 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-bharath10-testing-2024-09-17-1348'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-bharath10-testing-2024-09-17-1348', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T16:10:38.355 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-09-17T16:10:38.355 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-17T16:10:38.617 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3742600541' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-17T16:10:38.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3742600541' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-17T16:10:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3742600541' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-17T16:10:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:39 smithi173 ceph-mon[30530]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:39 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:10:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:39 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:10:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:39 smithi173 ceph-mon[30530]: osdmap e90: 12 total, 12 up, 12 in 2024-09-17T16:10:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1580806344' entity='client.admin' 2024-09-17T16:10:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:39 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:39 smithi017 ceph-mon[27571]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:39 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:10:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:39 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:10:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:39 smithi017 ceph-mon[27571]: osdmap e90: 12 total, 12 up, 12 in 2024-09-17T16:10:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1580806344' entity='client.admin' 2024-09-17T16:10:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:39 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:39.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:39 smithi071 ceph-mon[30002]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:39.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:39 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:10:39.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:39 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:10:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:39 smithi071 ceph-mon[30002]: osdmap e90: 12 total, 12 up, 12 in 2024-09-17T16:10:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1580806344' entity='client.admin' 2024-09-17T16:10:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:39 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:10:40.102 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-17T16:10:40.102 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-17T16:10:40.103 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:10:40.359 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:40.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:40.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:40.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:40.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:40.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:40.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:40.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:10:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:10:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:10:41.035 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:10:41.035 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:10:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:41 smithi173 ceph-mon[30530]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:41 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:10:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:41 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:10:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:41 smithi173 ceph-mon[30530]: osdmap e91: 12 total, 12 up, 12 in 2024-09-17T16:10:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:41 smithi017 ceph-mon[27571]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:41 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:10:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:41 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:10:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:41 smithi017 ceph-mon[27571]: osdmap e91: 12 total, 12 up, 12 in 2024-09-17T16:10:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:41 smithi071 ceph-mon[30002]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:41 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:10:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:41 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:10:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:41 smithi071 ceph-mon[30002]: osdmap e91: 12 total, 12 up, 12 in 2024-09-17T16:10:41.899 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-09-17T16:10:40.657165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":342136,"kb_used_data":9480,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599688,"statfs":{"total":1151940427776,"available":1151590080512,"internally_reserved":0,"allocated":9707520,"data_stored":5761048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":340374842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.939805"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":122,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-17T16:10:31.470527+0000","last_change":"2024-09-17T16:09:51.306176+0000","last_active":"2024-09-17T16:10:31.470527+0000","last_peered":"2024-09-17T16:10:31.470527+0000","last_clean":"2024-09-17T16:10:31.470527+0000","last_became_active":"2024-09-17T16:09:51.301092+0000","last_became_peered":"2024-09-17T16:09:51.301092+0000","last_unstale":"2024-09-17T16:10:31.470527+0000","last_undegraded":"2024-09-17T16:10:31.470527+0000","last_fullsized":"2024-09-17T16:10:31.470527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:09:43.462563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:09:43.462563+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:01.928744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285582,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1204224,"data_stored":874306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547217,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":345398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":345398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":352775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1204224,"data_stored":874326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1216512,"data_stored":881703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724586,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":352775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":352775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":345398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713308,"statfs":{"total":95995035648,"available":95962427392,"internally_reserved":0,"allocated":671744,"data_stored":345398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713308,"statfs":{"total":95995035648,"available":95962427392,"internally_reserved":0,"allocated":671744,"data_stored":345398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":345398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:10:41.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:10:42.163 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:42.557 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:42 smithi017 ceph-mon[27571]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:42.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:42 smithi173 ceph-mon[30530]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:42.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:42 smithi071 ceph-mon[30002]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:42.839 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:10:42.839 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:10:43.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:43 smithi017 ceph-mon[27571]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:43.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:43 smithi017 ceph-mon[27571]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:43.643 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-17T16:10:41.719208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":342244,"kb_used_data":9588,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124599580,"statfs":{"total":1151940427776,"available":1151589969920,"internally_reserved":0,"allocated":9818112,"data_stored":5827441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":340374842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001554"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":126,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T16:10:40.719985+0000","last_change":"2024-09-17T16:09:51.306176+0000","last_active":"2024-09-17T16:10:40.719985+0000","last_peered":"2024-09-17T16:10:40.719985+0000","last_clean":"2024-09-17T16:10:40.719985+0000","last_became_active":"2024-09-17T16:09:51.301092+0000","last_became_peered":"2024-09-17T16:09:51.301092+0000","last_unstale":"2024-09-17T16:10:40.719985+0000","last_undegraded":"2024-09-17T16:10:40.719985+0000","last_fullsized":"2024-09-17T16:10:40.719985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:09:43.462563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:09:43.462563+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:01.928744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285582,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1204224,"data_stored":874306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":352775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717444,"statfs":{"total":95995035648,"available":95966662656,"internally_reserved":0,"allocated":696320,"data_stored":360152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":352775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234017,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1216512,"data_stored":881703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495654,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1216512,"data_stored":881703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724586,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":352775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":352775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":352775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713296,"statfs":{"total":95995035648,"available":95962415104,"internally_reserved":0,"allocated":684032,"data_stored":352775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":696320,"data_stored":360152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":352775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:10:43.644 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-17T16:10:43.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-17T16:10:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:43 smithi173 ceph-mon[30530]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:43 smithi173 ceph-mon[30530]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:43.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:43 smithi071 ceph-mon[30002]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:43 smithi071 ceph-mon[30002]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:10:43.904 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:44.558 INFO:teuthology.orchestra.run.smithi017.stdout:epoch 92 2024-09-17T16:10:44.558 INFO:teuthology.orchestra.run.smithi017.stdout:fsid 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:created 2024-09-17T16:02:46.978716+0000 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:modified 2024-09-17T16:10:44.332780+0000 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:crush_version 26 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:full_ratio 0.95 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:backfillfull_ratio 0.9 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:nearfull_ratio 0.85 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:require_min_compat_client luminous 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:min_compat_client jewel 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:require_osd_release squid 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:stretch_mode_enabled false 2024-09-17T16:10:44.559 INFO:teuthology.orchestra.run.smithi017.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:max_osd 12 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 up in weight 1 up_from 8 up_thru 54 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730] [v2:172.21.15.17:6804/3081446730,v1:172.21.15.17:6805/3081446730] exists,up 139c41b7-e38b-42d2-ae0b-70b94041dbfb 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 up in weight 1 up_from 13 up_thru 32 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826] [v2:172.21.15.17:6812/603840826,v1:172.21.15.17:6813/603840826] exists,up ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780] [v2:172.21.15.17:6820/2706983780,v1:172.21.15.17:6821/2706983780] exists,up efb6c3f1-6157-4581-a39d-bda978aea6b2 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275] [v2:172.21.15.17:6828/3512825275,v1:172.21.15.17:6829/3512825275] exists,up 37609e4c-d0cb-464f-a3c6-2d6e530149e1 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020] [v2:172.21.15.71:6802/146827020,v1:172.21.15.71:6803/146827020] exists,up bd9e473c-bf74-4621-92d0-7031d7db10e5 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 up in weight 1 up_from 39 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307] [v2:172.21.15.71:6810/1837749307,v1:172.21.15.71:6811/1837749307] exists,up f08464f8-610f-4775-87d0-803b569eb211 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 up in weight 1 up_from 46 up_thru 47 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628] [v2:172.21.15.71:6818/2384780628,v1:172.21.15.71:6819/2384780628] exists,up d05170c6-0258-43da-b81b-cc9cc64b5d31 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 up in weight 1 up_from 54 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174] [v2:172.21.15.71:6826/1816911174,v1:172.21.15.71:6827/1816911174] exists,up 0d34db33-5d4d-4761-887d-cf1a63dbf3d2 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865] [v2:172.21.15.173:6802/527945865,v1:172.21.15.173:6803/527945865] exists,up f419836d-0416-4789-80ad-a45ae2ccf035 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9 up in weight 1 up_from 69 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352] [v2:172.21.15.173:6810/3589162352,v1:172.21.15.173:6811/3589162352] exists,up 9dcaa1d8-e921-427c-8a2c-a368658a5bfe 2024-09-17T16:10:44.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10 up in weight 1 up_from 75 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719] [v2:172.21.15.173:6818/2428879719,v1:172.21.15.173:6819/2428879719] exists,up ee08b73f-f02b-4795-87bc-ab559b7faf61 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11 up in weight 1 up_from 83 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377] [v2:172.21.15.173:6826/1731820377,v1:172.21.15.173:6827/1731820377] exists,up ce178d21-377e-40ac-90fa-46009901114d 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:blocklist 172.21.15.17:0/3205448177 expires 2024-09-18T16:03:49.423969+0000 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:blocklist 172.21.15.17:6801/250768364 expires 2024-09-18T16:03:49.423969+0000 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:blocklist 172.21.15.17:6800/250768364 expires 2024-09-18T16:03:49.423969+0000 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:blocklist 172.21.15.17:0/394168293 expires 2024-09-18T16:03:49.423969+0000 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:blocklist 172.21.15.17:0/4225461221 expires 2024-09-18T16:03:49.423969+0000 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:blocklist 172.21.15.17:0/3672343901 expires 2024-09-18T16:03:21.713930+0000 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:blocklist 172.21.15.17:0/419827434 expires 2024-09-18T16:03:21.713930+0000 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:blocklist 172.21.15.17:6801/365103448 expires 2024-09-18T16:03:21.713930+0000 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:blocklist 172.21.15.17:6800/365103448 expires 2024-09-18T16:03:21.713930+0000 2024-09-17T16:10:44.561 INFO:teuthology.orchestra.run.smithi017.stdout:blocklist 172.21.15.17:0/2688765754 expires 2024-09-18T16:03:21.713930+0000 2024-09-17T16:10:44.562 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:44 smithi017 ceph-mon[27571]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:44 smithi173 ceph-mon[30530]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:44.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:44 smithi071 ceph-mon[30002]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:45.192 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 54 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6802/3081446730,v1:172.21.15.17:6803/3081446730] [v2:172.21.15.17:6804/3081446730,v1:172.21.15.17:6805/3081446730] exists,up 139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'osd.1 up in weight 1 up_from 13 up_thru 32 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6810/603840826,v1:172.21.15.17:6811/603840826] [v2:172.21.15.17:6812/603840826,v1:172.21.15.17:6813/603840826] exists,up ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6818/2706983780,v1:172.21.15.17:6819/2706983780] [v2:172.21.15.17:6820/2706983780,v1:172.21.15.17:6821/2706983780] exists,up efb6c3f1-6157-4581-a39d-bda978aea6b2', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6826/3512825275,v1:172.21.15.17:6827/3512825275] [v2:172.21.15.17:6828/3512825275,v1:172.21.15.17:6829/3512825275] exists,up 37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'osd.4 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6800/146827020,v1:172.21.15.71:6801/146827020] [v2:172.21.15.71:6802/146827020,v1:172.21.15.71:6803/146827020] exists,up bd9e473c-bf74-4621-92d0-7031d7db10e5', 'osd.5 up in weight 1 up_from 39 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6808/1837749307,v1:172.21.15.71:6809/1837749307] [v2:172.21.15.71:6810/1837749307,v1:172.21.15.71:6811/1837749307] exists,up f08464f8-610f-4775-87d0-803b569eb211', 'osd.6 up in weight 1 up_from 46 up_thru 47 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6816/2384780628,v1:172.21.15.71:6817/2384780628] [v2:172.21.15.71:6818/2384780628,v1:172.21.15.71:6819/2384780628] exists,up d05170c6-0258-43da-b81b-cc9cc64b5d31', 'osd.7 up in weight 1 up_from 54 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6824/1816911174,v1:172.21.15.71:6825/1816911174] [v2:172.21.15.71:6826/1816911174,v1:172.21.15.71:6827/1816911174] exists,up 0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'osd.8 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/527945865,v1:172.21.15.173:6801/527945865] [v2:172.21.15.173:6802/527945865,v1:172.21.15.173:6803/527945865] exists,up f419836d-0416-4789-80ad-a45ae2ccf035', 'osd.9 up in weight 1 up_from 69 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/3589162352,v1:172.21.15.173:6809/3589162352] [v2:172.21.15.173:6810/3589162352,v1:172.21.15.173:6811/3589162352] exists,up 9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'osd.10 up in weight 1 up_from 75 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/2428879719,v1:172.21.15.173:6817/2428879719] [v2:172.21.15.173:6818/2428879719,v1:172.21.15.173:6819/2428879719] exists,up ee08b73f-f02b-4795-87bc-ab559b7faf61', 'osd.11 up in weight 1 up_from 83 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/1731820377,v1:172.21.15.173:6825/1731820377] [v2:172.21.15.173:6826/1731820377,v1:172.21.15.173:6827/1731820377] exists,up ce178d21-377e-40ac-90fa-46009901114d'] 2024-09-17T16:10:45.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-17T16:10:45.448 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:45.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:45 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:10:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:45 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:10:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:45 smithi017 ceph-mon[27571]: osdmap e92: 12 total, 12 up, 12 in 2024-09-17T16:10:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:45 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2102962643' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-17T16:10:45.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:45 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:10:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:45 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:10:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:45 smithi173 ceph-mon[30530]: osdmap e92: 12 total, 12 up, 12 in 2024-09-17T16:10:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:45 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2102962643' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-17T16:10:45.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:45 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:10:45.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:45 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:10:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:45 smithi071 ceph-mon[30002]: osdmap e92: 12 total, 12 up, 12 in 2024-09-17T16:10:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:45 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2102962643' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-17T16:10:45.955 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:10:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "mon.a", 2024-09-17T16:10:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster": "ceph", 2024-09-17T16:10:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-17T16:10:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "admin_socket_mode": "", 2024-09-17T16:10:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-17T16:10:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_client_required": "cephx, none", 2024-09-17T16:10:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_cluster_required": "cephx", 2024-09-17T16:10:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_debug": "true", 2024-09-17T16:10:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-17T16:10:45.956 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_service_required": "cephx", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_supported": "", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_aio": "true", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_aio_poll_ms": "250", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_aio_reap_max": "16", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_async_discard_threads": "0", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_block_size": "4096", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_debug_aio": "false", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-17T16:10:45.957 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_debug_inflight_ios": "false", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_enable_discard": "false", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_flock_retry": "3", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_inject_crash": "0", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_ioring": "false", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_ioring_hipri": "false", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-17T16:10:45.958 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_type": "", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_alloc_size": "1048576", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_allocator": "hybrid", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_buffered_io": "true", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_check_for_zeros": "false", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_compact_log_sync": "false", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-17T16:10:45.959 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_max_log_runway": "4194304", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_max_prefetch": "1048576", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_min_flush_size": "524288", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_min_log_runway": "1048576", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_replay_recovery": "false", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_sync_write": "false", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_allocation_from_file": "true", 2024-09-17T16:10:45.960 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_allocator": "hybrid", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_async_db_compaction": "true", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bdev_label_multi": "true", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_create": "true", 2024-09-17T16:10:45.961 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_db_create": "false", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_db_path": "", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_db_size": "0", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_path": "", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_preallocate_file": "false", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_size": "107374182400", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_wal_create": "false", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_wal_path": "", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_wal_size": "100663296", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bluefs": "true", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-17T16:10:45.962 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_autotune": "true", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_size": "0", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-17T16:10:45.963 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_type": "2q", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_clone_cow": "true", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-17T16:10:45.964 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_mode": "none", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_csum_type": "crc32c", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_freelist": "false", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-17T16:10:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_small_allocations": "0", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_default_buffered_read": "true", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_default_buffered_write": "false", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-17T16:10:45.966 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fail_eio": "false", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_mount": "false", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_umount": "false", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-17T16:10:45.967 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_ignore_data_csum": "false", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_log_op_age": "5.000000", 2024-09-17T16:10:45.968 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_alloc_size": "0", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_blob_size": "0", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_deferred_txc": "32", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_min_alloc_size": "0", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_nid_prealloc": "1024", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-17T16:10:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_retry_disk_reads": "3", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_rocksdb_cf": "true", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_spdk_mem": "512", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-17T16:10:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_tracing": "false", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-17T16:10:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_write_v2": "false", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_write_v2_random": "false", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_zero_block_detection": "false", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-17T16:10:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_cluster_require_signatures": "false", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_cluster_require_version": "2", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_require_signatures": "false", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_require_version": "2", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_service_require_signatures": "false", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_service_require_version": "2", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_sign_messages": "true", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "chdir": "", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "client_acl_type": "", 2024-09-17T16:10:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "client_asio_thread_count": "2", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_cache_mid": "0.750000", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_cache_size": "16384", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_caps_release_delay": "5", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_check_pool_perm": "true", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_debug_force_sync_read": "false", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_debug_getattr_caps": "false", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_debug_inject_features": "", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_debug_inject_tick_delay": "0", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_die_on_failed_remount": "false", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_dirsize_rbytes": "true", 2024-09-17T16:10:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_force_lazyio": "false", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_fs": "", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_inject_release_failure": "false", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_max_inline_size": "4096", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_mds_namespace": "", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_metadata": "", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_mount_gid": "-1", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_mount_timeout": "300", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_mount_uid": "-1", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_mountpoint": "/", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_notify_timeout": "10", 2024-09-17T16:10:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc": "true", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc_max_dirty": "104857600", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc_max_objects": "1000", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc_size": "209715200", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc_target_dirty": "8388608", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_permissions": "true", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_quota": "true", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_quota_df": "true", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_readahead_max_bytes": "0", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_readahead_max_periods": "4", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_readahead_min": "131072", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_reconnect_stale": "false", 2024-09-17T16:10:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_shutdown_timeout": "30", 2024-09-17T16:10:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "client_snapdir": ".snap", 2024-09-17T16:10:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "client_tick_interval": "1", 2024-09-17T16:10:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "client_trace": "", 2024-09-17T16:10:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "client_try_dentry_invalidate": "false", 2024-09-17T16:10:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "client_use_faked_inos": "false", 2024-09-17T16:10:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "client_use_random_mds": "false", 2024-09-17T16:10:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_graylog": "false", 2024-09-17T16:10:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-17T16:10:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_graylog_port": "12201", 2024-09-17T16:10:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_monitors": "default=true", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_syslog": "false", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_syslog_level": "info", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "-", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_network": "", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_network_interface": "", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "compressor_zlib_isal": "false", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "compressor_zlib_level": "5", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "compressor_zlib_winsize": "-15", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "compressor_zstd_level": "1", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-17T16:10:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_alien_op_num_threads": "6", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_osd_stat_interval": "0", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_seastar_cpu_cores": "", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_seastar_num_threads": "0", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location_hook": "", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location_hook_timeout": "10", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "daemonize": "false", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "daos_pool": "tank", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-17T16:10:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_alienstore": "0/5", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_allow_any_pool_priority": "false", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_asok": "1/5", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_asok_assert_abort": "false", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_asserts_on_shutdown": "true", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_auth": "1/5", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_bdev": "1/3", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_bluefs": "1/5", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_bluestore": "1/5", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_buffer": "0/1", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_ceph_dedup": "0/5", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_ceph_exporter": "1/5", 2024-09-17T16:10:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_cephfs_mirror": "0/5", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_cephsqlite": "0/5", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_client": "0/5", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_compressor": "1/5", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_context": "0/1", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_crimson_interrupt": "0/5", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_crush": "1/1", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_crypto": "1/5", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_cyanstore": "0/5", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_deliberately_leak_memory": "false", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_disable_randomized_ping": "false", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_dpdk": "1/5", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_eventtrace": "1/5", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_filer": "0/1", 2024-09-17T16:10:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_filestore": "1/3", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_finisher": "1/1", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_fuse": "1/5", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_heartbeat_testing_span": "0", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_heartbeatmap": "1/5", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_javaclient": "1/5", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_journal": "1/3", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_journaler": "0/5", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_kstore": "1/5", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_lockdep": "0/1", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mclock": "1/5", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds": "1/5", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_balancer": "1/5", 2024-09-17T16:10:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_locker": "1/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_log": "1/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_log_expire": "1/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_migrator": "1/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_quiesce": "3/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_memstore": "1/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mgr": "2/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mgrc": "1/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mon": "20/20", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_monc": "0/10", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_ms": "1/1", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_none": "0/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_objclass": "0/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_objectcacher": "0/5", 2024-09-17T16:10:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_objecter": "0/1", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_optracker": "0/5", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_osd": "1/5", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_paxos": "20/20", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_perfcounter": "1/5", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_prioritycache": "1/5", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rados": "0/5", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rbd": "0/5", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rbd_mirror": "0/5", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rbd_pwl": "0/5", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rbd_replay": "0/5", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_refs": "0/0", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_reserver": "1/1", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw": "1/5", 2024-09-17T16:10:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_access": "1/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_datacache": "1/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_dbstore": "1/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_flight": "1/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_notification": "1/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_sync": "1/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rocksdb": "4/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore": "0/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_backref": "0/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_cache": "0/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_cleaner": "0/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_device": "0/5", 2024-09-17T16:10:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_epm": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_journal": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_lba": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_odata": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_omap": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_onode": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_t": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_tm": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_striper": "0/1", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_test": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_throttle": "1/1", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_timer": "0/1", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_tp": "0/5", 2024-09-17T16:10:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_trace": "1/5", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "device_failure_prediction_mode": "none", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "err_to_graylog": "false", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "err_to_journald": "false", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "err_to_stderr": "true", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "err_to_syslog": "false", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event_tracing": "false", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_addr": "0.0.0.0", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_cert_file": "", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_http_port": "9926", 2024-09-17T16:10:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_key_file": "", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_prio_limit": "5", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_sort_metrics": "true", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_stats_period": "5", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "fake_statfs_for_testing": "0", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "fatal_signal_handlers": "true", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "filer_max_purge_ops": "10", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "filer_max_truncate_ops": "128", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_apply_finisher_threads": "1", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_blackhole": "false", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_btrfs_clone_range": "true", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_btrfs_snap": "true", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_caller_concurrency": "10", 2024-09-17T16:10:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_collect_device_partition_information": "true", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_commit_timeout": "600.000000", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_debug_inject_read_err": "false", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_debug_omap_check": "false", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_debug_verify_split": "false", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_dump_file": "", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fadvise": "true", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fail_eio": "true", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fd_cache_shards": "16", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fd_cache_size": "128", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fiemap": "false", 2024-09-17T16:10:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fiemap_threshold": "4096", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_inject_stall": "0", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_journal_parallel": "false", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_journal_trailing": "false", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_journal_writeahead": "false", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_kill_at": "0", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-17T16:10:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattrs": "0", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_xattr_value_size": "0", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_merge_threshold": "-10", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_odsync_write": "false", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_omap_backend": "rocksdb", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_omap_backend_path": "", 2024-09-17T16:10:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_op_thread_timeout": "60", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_op_threads": "2", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_punch_hole": "false", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-17T16:10:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_max_ops": "50", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_seek_data_hole": "false", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_sloppy_crc": "false", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_splice": "false", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_split_multiple": "2", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_split_rand_factor": "20", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_update_to": "1000", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-17T16:10:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_enable": "true", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_xfs_extsize": "false", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_zfs_snap": "false", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "fio_dir": "/tmp/fio", 2024-09-17T16:10:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_allow_other": "true", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_atomic_o_trunc": "true", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_big_writes": "true", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_debug": "false", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_default_permissions": "false", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_disable_pagecache": "false", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_max_write": "0", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_multithreaded": "true", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_require_active_mds": "true", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_set_user_groups": "true", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_splice_move": "true", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_splice_read": "true", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_splice_write": "true", 2024-09-17T16:10:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_use_invalidate_cb": "true", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "gss_target_name": "ceph", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_file": "", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_inject_failure": "0", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_interval": "5", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "host": "smithi017", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-17T16:10:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "inject_early_sigterm": "false", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "jaeger_agent_port": "6799", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "jaeger_tracing_enable": "false", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_aio": "true", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_align_min_size": "65536", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_block_align": "true", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_block_size": "4096", 2024-09-17T16:10:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_dio": "true", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_discard": "false", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_force_aio": "false", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_ignore_corruption": "false", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_max_write_bytes": "10485760", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_max_write_entries": "100", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_replay_from": "0", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_write_header_frequency": "0", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_zero_on_create": "false", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journaler_prefetch_periods": "10", 2024-09-17T16:10:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "journaler_prezero_periods": "5", 2024-09-17T16:10:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "journaler_write_head_interval": "15", 2024-09-17T16:10:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "", 2024-09-17T16:10:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "keyfile": "", 2024-09-17T16:10:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-17T16:10:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_backend": "rocksdb", 2024-09-17T16:10:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_default_stripe_size": "65536", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_fsck_on_mount": "false", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_max_bytes": "67108864", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_max_ops": "512", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_nid_prealloc": "1024", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_onode_map_size": "1024", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_sync_submit_transaction": "false", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_sync_transaction": "false", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "librados_thread_count": "2", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "lockdep": "false", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "lockdep_force_backtrace": "false", 2024-09-17T16:10:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "log_coarse_timestamps": "true", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_flush_on_exit": "false", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_graylog_host": "127.0.0.1", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_graylog_port": "12201", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_max_new": "1000", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_max_recent": "10000", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_stderr_prefix": "", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_stop_at_utilization": "0.970000", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_to_file": "true", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_to_graylog": "false", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_to_journald": "false", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_to_stderr": "false", 2024-09-17T16:10:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "log_to_syslog": "false", 2024-09-17T16:10:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "max_rotating_auth_attempts": "10", 2024-09-17T16:10:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-17T16:10:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_action_on_write_error": "1", 2024-09-17T16:10:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_alternate_name_max": "8192", 2024-09-17T16:10:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_asio_thread_count": "2", 2024-09-17T16:10:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_export_pin": "true", 2024-09-17T16:10:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_fragment_dirs": "true", 2024-09-17T16:10:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-17T16:10:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_fragment_interval": "5", 2024-09-17T16:10:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_interval": "10", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_max": "-1", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_max_until": "-1", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_merge_size": "50", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_midchunk": "0.300000", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_min_start": "0.200000", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_minchunk": "0.001000", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_mode": "0", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_need_max": "1.200000", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_need_min": "0.800000", 2024-09-17T16:10:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_overload_epochs": "2", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_split_bits": "3", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_split_size": "10000", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_target_decay": "10.000000", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_beacon_grace": "15.000000", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_beacon_interval": "4.000000", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-17T16:10:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_mid": "0.700000", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_quiesce_delay": "0", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_release_free_interval": "10", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_reservation": "0.050000", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_trim_interval": "1", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_trim_threshold": "262144", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-17T16:10:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_client_prealloc_inos": "1000", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_connect_bootstrapping": "false", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_damage_table_max_entries": "10000", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_debug_auth_pins": "false", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_debug_frag": "false", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_debug_scatterstat": "false", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_debug_subtrees": "false", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_decay_halflife": "5.000000", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_default_dir_hash": "2", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_defer_session_stale": "true", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_deny_all_reconnect": "false", 2024-09-17T16:10:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dir_keys_per_op": "16384", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dir_max_commit_size": "10", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dir_max_entries": "0", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dir_prefetch": "true", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dump_cache_on_map": "false", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_early_reply": "true", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_enable_op_tracker": "true", 2024-09-17T16:10:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_enforce_unique_name": "true", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_export_ephemeral_random": "true", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_health_summarize_threshold": "10", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-17T16:10:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_health_dummy": "false", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_migrator_session_race": "false", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_join_fs": "", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_journal_format": "1", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_create_at": "0", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_dirfrag_at": "0", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_export_at": "0", 2024-09-17T16:10:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_import_at": "0", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_journal_at": "0", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_journal_expire_at": "0", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_journal_replay_at": "0", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_link_at": "0", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_mdstable_at": "0", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_openc_at": "0", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_rename_at": "0", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_shutdown_at": "0", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_event_large_threshold": "524288", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_events_per_segment": "1024", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_max_events": "-1", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_max_segments": "128", 2024-09-17T16:10:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_pause": "false", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_segment_size": "0", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_trim_threshold": "128", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_warn_factor": "2.000000", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_caps_per_client": "1048576", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_completed_flushes": "100000", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_completed_requests": "100000", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_export_size": "20971520", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_file_recover": "32", 2024-09-17T16:10:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_purge_files": "64", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_purge_ops": "8192", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_snaps_per_dir": "100", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_metrics_update_interval": "2", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_min_caps_per_client": "100", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_min_caps_working_set": "10000", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_numa_node": "-1", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_complaint_time": "30.000000", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_history_duration": "600", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_history_size": "20", 2024-09-17T16:10:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_history_slow_op_size": "20", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_log_threshold": "5", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_ping_grace": "15", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_ping_interval": "5", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_max_caps": "30000", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_warning_threshold": "262144", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_replay_interval": "1.000000", 2024-09-17T16:10:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_root_ino_gid": "0", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_root_ino_uid": "0", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_scrub_stats_review_period": "1", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-17T16:10:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_shutdown_check": "0", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_skip_ino": "0", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_snap_max_uid": "4294967294", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_snap_min_uid": "0", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_snap_rstat": "false", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_standby_replay_damaged": "false", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_symlink_recovery": "true", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_thrash_exports": "0", 2024-09-17T16:10:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_thrash_fragments": "0", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_tick_interval": "5.000000", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_valgrind_exit": "false", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_verify_backtrace": "1", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_verify_scatter": "false", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_wipe_sessions": "false", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "mempool_debug": "false", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "memstore_device_bytes": "1073741824", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "memstore_page_set": "false", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "memstore_page_size": "65536", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_client_bytes": "134217728", 2024-09-17T16:10:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_client_messages": "512", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_disabled_modules": "", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_enable_op_tracker": "true", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_max_pg_creating": "1024", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_max_pg_num_change": "128", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_mds_bytes": "134217728", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_mds_messages": "128", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_mon_bytes": "134217728", 2024-09-17T16:10:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_mon_messages": "128", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_history_duration": "600", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_history_size": "20", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_log_threshold": "5", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_osd_bytes": "536870912", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_osd_messages": "8192", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_pool": "true", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_standby_modules": "true", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_stats_period": "5", 2024-09-17T16:10:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_stats_threshold": "5", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_tick_period": "2", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_allow_pool_delete": "true", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_allow_pool_size_one": "false", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_auth_validate_all_caps": "true", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_bytes": "104857600", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_directed_command_retry": "5", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-17T16:10:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_hunt_parallel": "3", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_log_interval": "1.000000", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_ping_interval": "10.000000", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_target_rank": "-1", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_level": "debug", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_file": "true", 2024-09-17T16:10:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_journald": "true", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_compact_on_bootstrap": "false", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_compact_on_start": "false", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_compact_on_trim": "true", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cpu_threads": "4", 2024-09-17T16:10:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_crush_min_required_version": "hammer", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_daemon_bytes": "419430400", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_data_avail_crit": "5", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_data_avail_warn": "5", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_data_size_warn": "16106127360", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_dump_json": "false", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_dump_transactions": "false", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_extra_checks": "false", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-17T16:10:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_no_require_reef": "false", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_no_require_squid": "false", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_down_mkfs_grace": "60", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_down_uptime_grace": "60", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_election_default_strategy": "3", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_election_timeout": "5.000000", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_elector_ping_divisor": "2", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-17T16:10:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_enable_op_tracker": "true", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_fake_pool_delete": "false", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_force_quorum_join": "false", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_globalid_prealloc": "10000", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_detail_to_clog": "true", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_log_update_period": "5", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_max_detail": "50", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_to_clog": "true", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_to_clog_interval": "600", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_host": "[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_host_override": "", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_initial_members": "", 2024-09-17T16:10:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_lease": "5.000000", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_log_full_interval": "50", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_log_max": "10000", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_log_max_summary": "50", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_log_epochs": "500", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-17T16:10:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_osd": "10000", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_pg_per_osd": "10000", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_pool_pg_num": "65536", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mds_force_trim_to": "0", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mds_skip_sanity": "false", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_memory_autotune": "true", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_memory_target": "2147483648", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_beacon_grace": "30", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_digest_period": "5", 2024-09-17T16:10:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_inactive_grace": "60", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_min_osdmap_epochs": "50", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_complaint_time": "30", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_history_duration": "600", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_history_size": "20", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_history_slow_op_size": "20", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_log_threshold": "5", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-17T16:10:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_auto_mark_in": "false", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_cache_size": "500", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_down_out_interval": "600", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_force_trim_to": "0", 2024-09-17T16:10:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_min_down_reporters": "2", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_report_timeout": "900", 2024-09-17T16:10:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_stuck_threshold": "60", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-17T16:10:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_probe_timeout": "2.000000", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_reweight_max_change": "0.050000", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_reweight_max_osds": "4", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_scrub_interval": "300", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_scrub_max_keys": "100", 2024-09-17T16:10:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_scrub_timeout": "300", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_session_timeout": "300", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_smart_report_timeout": "5", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_stat_smooth_intervals": "6", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_stretch_pool_min_size": "2", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_stretch_pool_size": "4", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_debug": "false", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_provider_kill_at": "0", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_requester_kill_at": "0", 2024-09-17T16:10:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_timeout": "60.000000", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_target_pg_per_osd": "100", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_tick_interval": "5", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_timecheck_interval": "300.000000", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_use_min_delay_socket": "false", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_older_version_delay": "604800", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-17T16:10:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_misplaced": "false", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_older_version": "true", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": "", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-17T16:10:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_tracing_enabled": "false", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_op_threads": "3", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_cm": "false", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_device_name": "", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_dscp": "96", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-17T16:10:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_local_gid": "", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_port_num": "1", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_sl": "3", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_support_srq": "true", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_type": "ib", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_reap_threshold": "5", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_before_connect": "false", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_ipv4": "true", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_ipv6": "false", 2024-09-17T16:10:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_msgr1": "true", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_msgr2": "true", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_port_max": "7568", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_port_min": "6800", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_retry_count": "3", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_retry_delay": "5", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_blackhole_client": "false", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_blackhole_mds": "false", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_blackhole_mgr": "false", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_blackhole_mon": "false", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_blackhole_osd": "false", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_client_mode": "crc secure", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-17T16:10:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_cluster_mode": "crc secure", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_cluster_type": "", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_compress_secure": "false", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_connection_idle_timeout": "900", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_connection_ready_timeout": "10", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_crc_data": "true", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_crc_header": "true", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_die_on_bad_msg": "false", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_die_on_bug": "true", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_die_on_old_message": "true", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_die_on_skipped_message": "false", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-17T16:10:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_coremask": "0xF", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_enable_tso": "true", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_hugepages": "", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_lro": "true", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_memory_channel": "4", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_pmd": "", 2024-09-17T16:10:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_port_id": "0", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dump_on_send": "false", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_initial_backoff": "0.200000", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_delay_max": "1.000000", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_delay_probability": "0.005000", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_delay_type": "osd", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_internal_delays": "0.002000", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_network_congestion": "0", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_socket_failures": "2500", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_learn_addr_from_peer": "true", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_max_accept_failures": "4", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_max_backoff": "15.000000", 2024-09-17T16:10:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_mon_client_mode": "secure crc", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_mon_service_mode": "secure crc", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_osd_compress_min_size": "1024", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_osd_compress_mode": "none", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_pq_min_cost": "65536", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_public_type": "", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_service_mode": "crc secure", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_tcp_listen_backlog": "512", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_tcp_nodelay": "true", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_tcp_rcvbuf": "0", 2024-09-17T16:10:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_type": "async+posix", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "no_config_file": "false", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_completion_locks_per_session": "32", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_inflight_ops": "1024", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_tick_interval": "5.000000", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_timeout": "10.000000", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "objectstore_blackhole": "false", 2024-09-17T16:10:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "openssl_engine_opts": "", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_delay_time": "5.000000", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_hist_halflife": "1000", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_max_low_ops": "2", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_max_ops": "4", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_slop": "0.020000", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_asio_thread_count": "2", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_async_recovery_min_cost": "100", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-17T16:10:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backfill_scan_max": "512", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backfill_scan_min": "64", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backoff_on_degraded": "false", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backoff_on_peering": "false", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backoff_on_unfound": "true", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_beacon_report_interval": "300", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_bench_duration": "30", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_bench_max_block_size": "67108864", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_blkin_trace_all": "false", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-17T16:10:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_check_for_log_corruption": "false", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_class_update_on_start": "true", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_client_message_cap": "256", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_client_message_size_cap": "524288000", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_client_op_priority": "63", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_client_watch_timeout": "30", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_command_max_records": "256", 2024-09-17T16:10:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_compact_on_start": "false", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_crush_update_on_start": "true", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_crush_update_weight_set": "true", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_feed_pullee": "-1", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-17T16:10:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_misdirected_ops": "false", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_no_acting_change": "false", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_no_purge_strays": "false", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_op_order": "false", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_shutdown": "false", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_trim_objects": "false", 2024-09-17T16:10:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_verify_snaps": "false", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_keys": "1024", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_stride": "524288", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_default_notify_timeout": "30", 2024-09-17T16:10:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_delete_sleep": "0.000000", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_discard_disconnected_ops": "true", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_ec_partial_reads": "true", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_enable_op_tracker": "true", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_fast_info": "true", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_fast_shutdown": "true", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-17T16:10:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_function_tracing": "false", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_grace": "20", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_interval": "6", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_min_peers": "10", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_min_size": "2000", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_stale": "600", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_hit_set_max_size": "100000", 2024-09-17T16:10:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_hit_set_min_size": "1000", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_journal_size": "5120", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_kill_backfill_at": "0", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_map_cache_size": "50", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_map_dedup": "true", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_map_message_max": "40", 2024-09-17T16:10:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_map_share_max_epochs": "40", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_attr_name_len": "100", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_attr_size": "0", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_backfills": "1", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_markdown_count": "5", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_markdown_period": "600", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_object_name_len": "2048", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_object_namespace_len": "256", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_object_size": "134217728", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_pg_blocked_by": "16", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_pg_log_entries": "10000", 2024-09-17T16:10:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_pgls": "1024", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_push_cost": "8388608", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_push_objects": "10", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_scrubs": "3", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_trimming_pgs": "2", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_write_op_reply_len": "64", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_write_size": "90", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-17T16:10:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_profile": "balanced", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-17T16:10:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_base": "805306368", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_cache_min": "134217728", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_target": "4294967296", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_target_autotune": "false", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_min_pg_log_entries": "250", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_min_recovery_priority": "0", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mon_report_interval": "5", 2024-09-17T16:10:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_num_cache_shards": "32", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_num_op_tracker_shard": "32", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_numa_auto_affinity": "true", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_numa_node": "-1", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_numa_prefer_iface": "true", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_objecter_finishers": "1", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_objectstore": "bluestore", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_objectstore_fuse": "false", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_objectstore_tracing": "false", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_complaint_time": "30.000000", 2024-09-17T16:10:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_history_duration": "600", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_history_size": "20", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_history_slow_op_size": "20", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_log_threshold": "5", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_shards": "0", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_shards_hdd": "5", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_shards_ssd": "8", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_pq_min_cost": "65536", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-17T16:10:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_queue_cut_off": "high", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_thread_timeout": "15", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_open_classes_on_start": "true", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_os_flags": "0", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_peering_op_priority": "255", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_delete_cost": "1048576", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_delete_priority": "5", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_log_trim_max": "10000", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_log_trim_min": "100", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-17T16:10:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_crimson": "false", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-17T16:10:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flags": "0", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_min_size": "2", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_pg_num": "32", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_pgp_num": "0", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_read_ratio": "70", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_size": "3", 2024-09-17T16:10:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_type": "replicated", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_push_per_object_cost": "1000", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recover_clone_overlap": "true", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_cost": "20971520", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_active": "0", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-17T16:10:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_single_start": "1", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_op_priority": "3", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_priority": "5", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_sleep": "0.000000", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_repair_during_recovery": "false", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_requested_scrub_priority": "120", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-17T16:10:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_auto_repair": "false", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_begin_hour": "0", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_begin_week_day": "0", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_chunk_max": "15", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_chunk_min": "5", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_cost": "52428800", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_during_recovery": "false", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_end_hour": "0", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_end_week_day": "0", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_event_cost": "4096", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-17T16:10:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_invalid_stats": "true", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_max_preemptions": "5", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_priority": "5", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_retry_delay": "30", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_retry_trimming": "10", 2024-09-17T16:10:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_sleep": "0.000000", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_skip_data_digest": "false", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_smart_report_timeout": "5", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_cost": "1048576", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_priority": "5", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-17T16:10:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_target_transaction_size": "30", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-17T16:10:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tracing": "false", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_use_stale_snap": "false", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "osdc_blkin_trace_all": "false", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_kill_at": "0", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_max_join_drift": "10", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_min": "500", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_min_wait": "0.050000", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_propose_interval": "1.000000", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_service_trim_max": "500", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_service_trim_min": "10", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_stash_full_interval": "25", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_trim_max": "500", 2024-09-17T16:10:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_trim_min": "250", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "perf": "true", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "pid_file": "", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "-", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrv": "", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "public_bind_addr": "-", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "public_network": "", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "public_network_interface": "", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "qat_compressor_busy_polling": "false", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "qat_compressor_enabled": "false", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "qat_compressor_session_max_number": "256", 2024-09-17T16:10:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "rados_mon_op_timeout": "0", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rados_osd_op_timeout": "0", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rados_tracing": "false", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_atime_update_interval": "60", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_balance_parent_reads": "false", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_balance_snap_reads": "false", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_blkin_trace_all": "false", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache": "true", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-17T16:10:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_policy": "writearound", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_size": "33554432", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_clone_copy_on_read": "false", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_compression_hint": "none", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_concurrent_management_ops": "10", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_clone_format": "auto", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_data_pool": "", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_features": "61", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_format": "2", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_map_options": "", 2024-09-17T16:10:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_order": "22", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_pool": "rbd", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_stripe_count": "0", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_stripe_unit": "0", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_enable_alloc_hint": "true", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_io_scheduler": "simple", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-17T16:10:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_order": "24", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_pool": "", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_splay_width": "4", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_localize_parent_reads": "false", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_localize_snap_reads": "false", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-17T16:10:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-17T16:10:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-17T16:10:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mtime_update_interval": "60", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_non_blocking_aio": "true", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_op_thread_timeout": "60", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_op_threads": "1", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_parent_cache_enabled": "false", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_plugins": "", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_bps_burst": "0", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_bps_limit": "0", 2024-09-17T16:10:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_exclude_ops": "0", 2024-09-17T16:10:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_iops_burst": "0", 2024-09-17T16:10:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-17T16:10:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_iops_limit": "0", 2024-09-17T16:10:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-17T16:10:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-17T16:10:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-17T16:10:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-17T16:10:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-17T16:10:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_read_from_replica_policy": "default", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-17T16:10:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_skip_partial_discard": "true", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_tracing": "false", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_validate_names": "true", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_validate_pool": "true", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "restapi_base_url": "", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "restapi_log_level": "", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_acl_grants_max_num": "100", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_admin_entry": "admin", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_asio_assert_yielding": "false", 2024-09-17T16:10:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_backend_store": "rados", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_barbican_url": "", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_beast_enable_async": "true", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_counters_cache": "false", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_cache_enabled": "true", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_cache_expiry_interval": "900", 2024-09-17T16:10:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_cache_lru_size": "25000", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_config_store": "rados", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_content_length_compat": "false", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_copy_obj_progress": "true", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_cors_rules_max_num": "100", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_cross_domain_policy": "", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_addr": "", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-17T16:10:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_password": "", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_username": "", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_require_ssl": "true", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-17T16:10:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_addr": "", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_auth": "token", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_namespace": "", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_prefix": "", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-17T16:10:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_token_file": "", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_curl_buffersize": "524288", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_curl_low_speed_time": "300", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-17T16:10:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_log_changes_size": "1000", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_log_num_shards": "128", 2024-09-17T16:10:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_log_window": "30", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-17T16:10:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_disable_s3select": "false", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-17T16:10:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dns_name": "", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dns_s3website_name": "", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dynamic_resharding": "true", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_gc_threads": "true", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_lc_threads": "true", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_mdsearch": "true", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_ops_log": "false", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_quota_threads": "true", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_static_website": "false", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_usage_log": "false", 2024-09-17T16:10:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enforce_swift_acls": "true", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_exit_timeout_secs": "120", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_expose_bucket": "false", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_extended_http_attrs": "", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_filter": "none", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_frontends": "beast port=7480", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_deferred": "50", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_objs": "32", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-17T16:10:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_processor_period": "3600", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_init_timeout": "300", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_kafka_connection_idle": "300", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-17T16:10:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_domain": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_password": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_password_path": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_project": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_tenant": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_token": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_token_path": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_user": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_barbican_domain": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_barbican_password": "", 2024-09-17T16:10:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_barbican_project": "", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_barbican_user": "", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_url": "", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_debug_interval": "-1", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_lock_max_time": "90", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_max_objs": "32", 2024-09-17T16:10:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_max_rules": "1000", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_max_worker": "3", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_thread_delay": "0", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_dnattr": "uid", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_searchfilter": "", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_uri": "", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-17T16:10:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_log_http_headers": "", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_log_object_name_utc": "false", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_attr_name_len": "0", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_attr_size": "0", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_chunk_size": "4194304", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-17T16:10:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_listing_results": "5000", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_notify_retries": "10", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_put_param_size": "1048576", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_put_size": "5368709120", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_slo_entries": "1000", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_md_log_max_shards": "64", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_mp_lock_max_time": "600", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-17T16:10:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_max_gc": "300", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-17T16:10:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_num_async_rados_threads": "32", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_num_control_oids": "8", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_numa_node": "-1", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_objexp_chunk_size": "100", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_objexp_gc_interval": "600", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_op_thread_timeout": "600", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_op_tracing": "false", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_opa_token": "", 2024-09-17T16:10:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_opa_url": "", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_opa_verify_ssl": "true", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ops_log_rados": "false", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ops_log_socket_path": "", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_period_push_interval": "2.000000", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-17T16:10:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_cache_lanes": "3", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_cache_partitions": "3", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_print_continue": "true", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_rados_tracing": "false", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_realm": "", 2024-09-17T16:10:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_realm_id": "", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_region": "", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_request_uri": "", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_batch_size": "64", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_max_aio": "128", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_num_logs": "16", 2024-09-17T16:10:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_thread_interval": "600", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_resolve_cname": "false", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_run_sync_thread": "true", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-17T16:10:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_scheduler_type": "throttler", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_script_uri": "", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_service_provider_name": "", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_client_id": "", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_client_secret": "", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_entry": "sts", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_key": "sts", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_min_session_duration": "900", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_token_introspection_url": "", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_account_in_url": "false", 2024-09-17T16:10:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_auth_entry": "auth", 2024-09-17T16:10:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_auth_url": "", 2024-09-17T16:10:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_custom_header": "", 2024-09-17T16:10:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-17T16:10:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_need_stats": "true", 2024-09-17T16:10:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_tenant_name": "", 2024-09-17T16:10:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_token_expiration": "86400", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_url": "", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_url_prefix": "swift", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_lease_period": "120", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-17T16:10:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_thread_pool_size": "512", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_comment": "", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_createby": "", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_encoding": "", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_flag": "false", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-17T16:10:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_origin": "", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_tracker": "", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_trust_forwarded_https": "false", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_usage_max_shards": "32", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_usage_max_user_shards": "1", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_use_opa_authz": "false", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_counters_cache": "false", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-17T16:10:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_max_buckets": "1000", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_policies_max_num": "100", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_unique_email": "true", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_verify_ssl": "true", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zone": "", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zone_id": "", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zonegroup": "", 2024-09-17T16:10:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zonegroup_id": "", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rgwlc_auto_session_clear": "true", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_block_size": "4096", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_size": "536870912", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-17T16:10:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_index_type": "binary_search", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_partition_filters": "false", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_perf": "false", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-17T16:10:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "run_dir": "/var/run/ceph", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_block_create": "true", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_cache_lru_size": "67108864", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_default_max_object_size": "16777216", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_device_size": "53687091200", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_full_integrity_check": "false", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_journal_batch_capacity": "16", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-17T16:10:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_main_device_type": "SSD", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_segment_size": "67108864", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "set_keepcaps": "false", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "setgroup": "ceph", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "setuser": "ceph", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "setuser_match_path": "", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "thp": "false", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "threadpool_default_timeout": "60", 2024-09-17T16:10:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-17T16:10:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "throttler_perf_counter": "true", 2024-09-17T16:10:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "tmp_dir": "/tmp", 2024-09-17T16:10:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-17T16:10:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "uadk_compressor_enabled": "false", 2024-09-17T16:10:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-17T16:10:46.112 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:10:46.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:46 smithi173 ceph-mon[30530]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:46.727 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:46 smithi017 ceph-mon[27571]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:46.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-17T16:10:46.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:46 smithi071 ceph-mon[30002]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:47.010 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:47.489 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a: {} 2024-09-17T16:10:47.490 INFO:teuthology.orchestra.run.smithi017.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:10:47.495 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c: {} 2024-09-17T16:10:47.496 INFO:teuthology.orchestra.run.smithi017.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:10:47.510 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b: {} 2024-09-17T16:10:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:10:47.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/923405910' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:10:47.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/923405910' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:10:47.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/923405910' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:10:48.125 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-17T16:10:48.134 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T16:10:48.135 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T16:10:48.136 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-17T16:10:48.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-17T16:10:48.136 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-17T16:10:48.136 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-17T16:10:48.136 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:10:48.137 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-17T16:10:48.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:48.137 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-17T16:10:48.137 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:10:48.138 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-17T16:10:48.138 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:10:48.138 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-17T16:10:48.138 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:10:48.164 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:48.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:48.266 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:48.292 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:48.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:48.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:10:48.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:48.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:10:48.444 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:48.444 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:48.444 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:48.450 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:48.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:48.548 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:48.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:48.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:48.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:48.677 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:48.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:48.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:10:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:48.801 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:48.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:10:48.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:48 smithi071 ceph-mon[30002]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:48.864 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:48 smithi017 ceph-mon[27571]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:48 smithi173 ceph-mon[30530]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:48.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:10:48.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:48.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:10:49.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:49.052 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:49.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:49.154 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:10:49.180 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:49.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:10:49.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:10:49.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:49.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:49.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:10:49.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:10:49.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:49.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:10:49.525 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:10:49.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:49.562 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:49.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:49.604 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:10:49.604 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:10:49.621 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:10:49.621 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:10:49.642 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:10:49.642 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:10:49.662 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:10:49.663 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:10:49.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:10:49.685 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:10:49.686 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:10:49.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:49.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:10:49.707 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:10:49.707 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:10:49.729 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:10:49.730 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:10:49.752 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:10:49.752 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:10:49.774 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:10:49.775 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:10:49.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:10:49.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:49.818 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:10:49.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:10:49.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:49.946 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:49.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:10:49.999 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:10:50.000 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:10:50.022 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:10:50.022 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:10:50.047 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:50.071 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:50.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:50.169 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:10:50.169 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:10:50.173 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:50.197 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:50.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:50.299 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:10:50.300 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:50 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:10:50.300 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:50 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:10:50.300 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:50 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:10:50.300 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:50 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:10:50.300 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3634374345' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:10:50.300 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:50 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:10:50.300 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3634374345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:10:50.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:50 smithi017 ceph-mon[27571]: osdmap e93: 12 total, 12 up, 12 in 2024-09-17T16:10:50.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3130242368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:10:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:50.539 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:50.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:10:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:50 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:10:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:50 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:10:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:50 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:10:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:50 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:10:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3634374345' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:10:50.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:50 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:10:50.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3634374345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:10:50.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:50 smithi071 ceph-mon[30002]: osdmap e93: 12 total, 12 up, 12 in 2024-09-17T16:10:50.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3130242368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:10:50.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:10:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:50 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:10:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:50 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:10:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:50 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:10:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:50 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:10:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3634374345' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:10:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:50 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:10:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3634374345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:10:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:50 smithi173 ceph-mon[30530]: osdmap e93: 12 total, 12 up, 12 in 2024-09-17T16:10:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3130242368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:10:50.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:50.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:10:50.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:10:50.796 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:50.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:10:50.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:10:50.925 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:50.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:10:51.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:51.052 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:51.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:51.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:51.178 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:51.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:51.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:51.251 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:10:51.251 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:10:51.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:10:51.307 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:51.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:10:51.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:51.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:51.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:51.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:51 smithi017 ceph-mon[27571]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:51.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:51.563 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:51 smithi071 ceph-mon[30002]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:51.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:51.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:51 smithi173 ceph-mon[30530]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:51.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:10:51.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:51.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:10:51.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:51.819 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:51.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:51.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:10:51.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:51.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:51.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:10:52.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:52.072 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:52.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:52.123 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:10:52.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:52.197 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:52.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:52.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:52.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:10:52.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:52.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:10:52.427 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:52.441 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:52.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:52.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:52.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:10:52.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:10:52.577 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:52.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:52.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:10:52.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:10:52.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:52.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:52.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:10:52.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:52.828 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:52.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:52.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:52.930 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:52.952 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:52.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:52.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:52.967 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:10:52.967 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:10:52.998 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:10:53.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:10:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:53.077 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:53.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:10:53.178 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:53.204 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:53.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:53.235 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:53.272 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_0' created 2024-09-17T16:10:53.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:10:53.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:53.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:10:53.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:10:53.456 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:53.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:10:53.531 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:53 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:10:53.531 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:53 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:10:53.532 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:53 smithi017 ceph-mon[27571]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:53.532 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:53 smithi017 ceph-mon[27571]: osdmap e94: 12 total, 12 up, 12 in 2024-09-17T16:10:53.532 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4059715447' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-17T16:10:53.558 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:10:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:53 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:10:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:53 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:10:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:53 smithi071 ceph-mon[30002]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:53 smithi071 ceph-mon[30002]: osdmap e94: 12 total, 12 up, 12 in 2024-09-17T16:10:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4059715447' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-17T16:10:53.580 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:53.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:10:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:53 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:10:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:53 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:10:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:53 smithi173 ceph-mon[30530]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:53 smithi173 ceph-mon[30530]: osdmap e94: 12 total, 12 up, 12 in 2024-09-17T16:10:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4059715447' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-17T16:10:53.683 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:10:53.705 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:53.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:10:53.784 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:10:53.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:10:53.832 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:53.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:53.833 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:10:53.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:10:53.956 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:53.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:10:54.058 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:10:54.084 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:54.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:54.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:10:54.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:54.211 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:54.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:54.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:54.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:10:54.336 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:54.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:54.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:10:54.347 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4059715447' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-17T16:10:54.347 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:54 smithi017 ceph-mon[27571]: osdmap e95: 12 total, 12 up, 12 in 2024-09-17T16:10:54.438 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:54.462 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:54.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:54.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:54.482 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:54.520 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:54.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4059715447' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-17T16:10:54.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:54 smithi071 ceph-mon[30002]: osdmap e95: 12 total, 12 up, 12 in 2024-09-17T16:10:54.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:10:54.586 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:54.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:10:54.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4059715447' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-17T16:10:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:54 smithi173 ceph-mon[30530]: osdmap e95: 12 total, 12 up, 12 in 2024-09-17T16:10:54.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:10:54.713 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:54.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:10:54.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:10:54.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:54.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:10:54.943 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:54.968 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:54.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:54.981 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:10:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:10:55.002 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:10:55.002 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:10:55.022 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:10:55.023 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:10:55.043 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:10:55.043 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:10:55.065 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:10:55.065 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:10:55.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:10:55.089 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:10:55.089 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:10:55.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:55.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:10:55.110 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:10:55.111 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:10:55.132 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:10:55.133 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:10:55.154 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:10:55.154 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:10:55.176 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:10:55.176 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:10:55.197 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:55.198 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:10:55.198 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:10:55.219 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:10:55.219 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:10:55.223 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:55.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:55.282 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:10:55.325 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:55.347 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:55.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:55.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:55 smithi017 ceph-mon[27571]: pgmap v297: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:55.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:55 smithi017 ceph-mon[27571]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:10:55.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:55 smithi017 ceph-mon[27571]: osdmap e96: 12 total, 12 up, 12 in 2024-09-17T16:10:55.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/854040009' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:10:55.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/770652681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:10:55.449 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:10:55.472 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-17T16:10:55.472 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:55.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:10:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:55 smithi071 ceph-mon[30002]: pgmap v297: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:55 smithi071 ceph-mon[30002]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:10:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:55 smithi071 ceph-mon[30002]: osdmap e96: 12 total, 12 up, 12 in 2024-09-17T16:10:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/854040009' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:10:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/770652681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:10:55.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:10:55.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:55.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:10:55.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:55 smithi173 ceph-mon[30530]: pgmap v297: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:55 smithi173 ceph-mon[30530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:10:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:55 smithi173 ceph-mon[30530]: osdmap e96: 12 total, 12 up, 12 in 2024-09-17T16:10:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/854040009' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:10:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/770652681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:10:55.701 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:10:55.723 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:55.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:10:55.826 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:10:55.848 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:55.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:10:55.951 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:55.973 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:55.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:56.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:56.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:56.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:56.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:10:56.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:56.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:10:56.327 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:56.350 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:56.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:56.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:56 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:10:56.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/854040009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:10:56.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:56 smithi017 ceph-mon[27571]: osdmap e97: 12 total, 12 up, 12 in 2024-09-17T16:10:56.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:56.478 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:56.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:56.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:56 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:10:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/854040009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:10:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:56 smithi071 ceph-mon[30002]: osdmap e97: 12 total, 12 up, 12 in 2024-09-17T16:10:56.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:56.606 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:56.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:56.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:56 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:10:56.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/854040009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:10:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:56 smithi173 ceph-mon[30530]: osdmap e97: 12 total, 12 up, 12 in 2024-09-17T16:10:56.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:56.730 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:56.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:56.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:56.855 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:56.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:56.958 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:10:56.984 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:56.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:56.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:10:57.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:57.108 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:57.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:57.207 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:10:57.207 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:10:57.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:10:57.207 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:10:57.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.207 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:21 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'11, mlcod=20'8)", 2024-09-17T16:10:57.207 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:08.956005+0000", 2024-09-17T16:10:57.207 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.24830638999998, 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00056660499999999997, 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.956005+0000", 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.956005+0000", 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.956006+0000", 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0999999999999998e-07 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.956050+0000", 2024-09-17T16:10:57.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3723000000000001e-05 2024-09-17T16:10:57.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.956086+0000", 2024-09-17T16:10:57.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6755999999999998e-05 2024-09-17T16:10:57.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.956094+0000", 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6290000000000001e-06 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.956135+0000", 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0933000000000003e-05 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.956155+0000", 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0367e-05 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.956534+0000", 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00037826800000000002 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.956572+0000", 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8219000000000001e-05 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:40 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'22, mlcod=20'19)", 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.032607+0000", 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.17170493899999, 2024-09-17T16:10:57.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062279999999999996, 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032607+0000", 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032607+0000", 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032607+0000", 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7299999999999996e-07 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032612+0000", 2024-09-17T16:10:57.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6550000000000003e-06 2024-09-17T16:10:57.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032641+0000", 2024-09-17T16:10:57.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9519e-05 2024-09-17T16:10:57.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032649+0000", 2024-09-17T16:10:57.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2320000000000004e-06 2024-09-17T16:10:57.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032715+0000", 2024-09-17T16:10:57.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6248999999999995e-05 2024-09-17T16:10:57.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032739+0000", 2024-09-17T16:10:57.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3759000000000001e-05 2024-09-17T16:10:57.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.033195+0000", 2024-09-17T16:10:57.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045582700000000001 2024-09-17T16:10:57.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.033230+0000", 2024-09-17T16:10:57.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4986e-05 2024-09-17T16:10:57.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:41 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'23, mlcod=20'19)", 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.032870+0000", 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.17144200899997, 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000813969, 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032870+0000", 2024-09-17T16:10:57.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032870+0000", 2024-09-17T16:10:57.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032870+0000", 2024-09-17T16:10:57.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3900000000000004e-07 2024-09-17T16:10:57.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032913+0000", 2024-09-17T16:10:57.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3037999999999997e-05 2024-09-17T16:10:57.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032949+0000", 2024-09-17T16:10:57.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5318000000000003e-05 2024-09-17T16:10:57.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032955+0000", 2024-09-17T16:10:57.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1349999999999997e-06 2024-09-17T16:10:57.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.033004+0000", 2024-09-17T16:10:57.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8977000000000002e-05 2024-09-17T16:10:57.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.033029+0000", 2024-09-17T16:10:57.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5066999999999999e-05 2024-09-17T16:10:57.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.033648+0000", 2024-09-17T16:10:57.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061857800000000005 2024-09-17T16:10:57.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.033684+0000", 2024-09-17T16:10:57.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6117e-05 2024-09-17T16:10:57.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.226 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.227 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:44 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'26, mlcod=20'23)", 2024-09-17T16:10:57.227 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.034916+0000", 2024-09-17T16:10:57.227 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.169395554, 2024-09-17T16:10:57.227 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:10:57.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072683600000000002, 2024-09-17T16:10:57.230 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.230 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.230 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.230 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.034916+0000", 2024-09-17T16:10:57.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.034916+0000", 2024-09-17T16:10:57.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.034917+0000", 2024-09-17T16:10:57.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0399999999999995e-07 2024-09-17T16:10:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.034958+0000", 2024-09-17T16:10:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1610000000000003e-05 2024-09-17T16:10:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.035008+0000", 2024-09-17T16:10:57.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9932000000000002e-05 2024-09-17T16:10:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.035018+0000", 2024-09-17T16:10:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5340000000000005e-06 2024-09-17T16:10:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.035064+0000", 2024-09-17T16:10:57.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5754000000000003e-05 2024-09-17T16:10:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.035110+0000", 2024-09-17T16:10:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6186000000000001e-05 2024-09-17T16:10:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.035606+0000", 2024-09-17T16:10:57.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00049588400000000004 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.035643+0000", 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7231999999999999e-05 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.235 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:47 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'29, mlcod=20'26)", 2024-09-17T16:10:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.038861+0000", 2024-09-17T16:10:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.16545078000001, 2024-09-17T16:10:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001098471, 2024-09-17T16:10:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038861+0000", 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038861+0000", 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038862+0000", 2024-09-17T16:10:57.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7599999999999997e-07 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038912+0000", 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0719e-05 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039084+0000", 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017162699999999999 2024-09-17T16:10:57.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039089+0000", 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3460000000000001e-06 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039135+0000", 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5682000000000001e-05 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039148+0000", 2024-09-17T16:10:57.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3122000000000001e-05 2024-09-17T16:10:57.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039917+0000", 2024-09-17T16:10:57.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076887199999999998 2024-09-17T16:10:57.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039959+0000", 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2426999999999999e-05 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:48 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'30, mlcod=20'26)", 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.038915+0000", 2024-09-17T16:10:57.241 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.16539645500001, 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0013632130000000001, 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038915+0000", 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038915+0000", 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038916+0000", 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1399999999999998e-07 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038924+0000", 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4859999999999997e-06 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039102+0000", 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000177815 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039104+0000", 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.379e-06 2024-09-17T16:10:57.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039310+0000", 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000205398 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039339+0000", 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9624999999999998e-05 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040247+0000", 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090822699999999995 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040278+0000", 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0969000000000001e-05 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:49 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'31, mlcod=20'26)", 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.038926+0000", 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.16538574600003, 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001378974, 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038926+0000", 2024-09-17T16:10:57.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038926+0000", 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038926+0000", 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.11e-07 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038933+0000", 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.815e-06 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039134+0000", 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020065299999999999 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039137+0000", 2024-09-17T16:10:57.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1190000000000001e-06 2024-09-17T16:10:57.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039514+0000", 2024-09-17T16:10:57.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000377458 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039526+0000", 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2043e-05 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040280+0000", 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075403500000000001 2024-09-17T16:10:57.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040305+0000", 2024-09-17T16:10:57.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.474e-05 2024-09-17T16:10:57.252 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.252 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.252 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:50 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'32, mlcod=20'26)", 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.038934+0000", 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.16537795599999, 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001406125, 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038934+0000", 2024-09-17T16:10:57.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038934+0000", 2024-09-17T16:10:57.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038934+0000", 2024-09-17T16:10:57.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3999999999999995e-08 2024-09-17T16:10:57.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039050+0000", 2024-09-17T16:10:57.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116233 2024-09-17T16:10:57.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039143+0000", 2024-09-17T16:10:57.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2971000000000001e-05 2024-09-17T16:10:57.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039152+0000", 2024-09-17T16:10:57.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0399999999999998e-06 2024-09-17T16:10:57.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039703+0000", 2024-09-17T16:10:57.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00055059100000000001 2024-09-17T16:10:57.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039722+0000", 2024-09-17T16:10:57.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9284999999999999e-05 2024-09-17T16:10:57.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040307+0000", 2024-09-17T16:10:57.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00058539200000000001 2024-09-17T16:10:57.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040340+0000", 2024-09-17T16:10:57.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2518999999999998e-05 2024-09-17T16:10:57.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.259 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.260 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.260 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:51 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'33, mlcod=20'26)", 2024-09-17T16:10:57.260 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.039052+0000", 2024-09-17T16:10:57.260 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.16526003299998, 2024-09-17T16:10:57.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0013171179999999999, 2024-09-17T16:10:57.261 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.261 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.261 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039052+0000", 2024-09-17T16:10:57.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039052+0000", 2024-09-17T16:10:57.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039052+0000", 2024-09-17T16:10:57.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6500000000000001e-07 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039054+0000", 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6910000000000002e-06 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039160+0000", 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000105054 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039169+0000", 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.064e-06 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039987+0000", 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00081872800000000001 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039997+0000", 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0115000000000001e-05 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040341+0000", 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00034336700000000002 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040369+0000", 2024-09-17T16:10:57.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7934000000000001e-05 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:52 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'34, mlcod=20'26)", 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.039055+0000", 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.165256339, 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001347111, 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.266 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039055+0000", 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039055+0000", 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039055+0000", 2024-09-17T16:10:57.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0999999999999994e-08 2024-09-17T16:10:57.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039057+0000", 2024-09-17T16:10:57.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2619999999999999e-06 2024-09-17T16:10:57.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039179+0000", 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000122576 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.039183+0000", 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6789999999999998e-06 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040114+0000", 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093078400000000004 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040136+0000", 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2076e-05 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040370+0000", 2024-09-17T16:10:57.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023450200000000001 2024-09-17T16:10:57.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040402+0000", 2024-09-17T16:10:57.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2141000000000003e-05 2024-09-17T16:10:57.271 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.271 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.271 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:53 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'35, mlcod=20'33)", 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.048560+0000", 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.15575176900001, 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076348599999999996, 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.048560+0000", 2024-09-17T16:10:57.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.048560+0000", 2024-09-17T16:10:57.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.048561+0000", 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2780000000000001e-06 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.048611+0000", 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9413000000000003e-05 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.048648+0000", 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7580000000000003e-05 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.048657+0000", 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1419999999999999e-06 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.048713+0000", 2024-09-17T16:10:57.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5578000000000002e-05 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.048735+0000", 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1869999999999999e-05 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.049277+0000", 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054198799999999997 2024-09-17T16:10:57.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.049323+0000", 2024-09-17T16:10:57.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6637000000000001e-05 2024-09-17T16:10:57.277 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.277 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.277 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.278 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:55 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'37, mlcod=20'33)", 2024-09-17T16:10:57.278 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.049380+0000", 2024-09-17T16:10:57.278 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.154931793, 2024-09-17T16:10:57.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085702800000000002, 2024-09-17T16:10:57.278 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.278 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.278 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.278 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.049380+0000", 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.049380+0000", 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.049380+0000", 2024-09-17T16:10:57.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1600000000000001e-07 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.049404+0000", 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3934000000000002e-05 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.049512+0000", 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107194 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.049520+0000", 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9249999999999995e-06 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.049579+0000", 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9005000000000002e-05 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.049600+0000", 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1641000000000002e-05 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.050206+0000", 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060586500000000003 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.050237+0000", 2024-09-17T16:10:57.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0848e-05 2024-09-17T16:10:57.283 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.283 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.283 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.283 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:67 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'45, mlcod=21'42)", 2024-09-17T16:10:57.283 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.248341+0000", 2024-09-17T16:10:57.283 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 287.95597114999998, 2024-09-17T16:10:57.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00063060299999999998, 2024-09-17T16:10:57.283 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.248341+0000", 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.248341+0000", 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.248341+0000", 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1399999999999995e-07 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.248348+0000", 2024-09-17T16:10:57.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3260000000000001e-06 2024-09-17T16:10:57.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.248374+0000", 2024-09-17T16:10:57.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6565999999999999e-05 2024-09-17T16:10:57.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.248381+0000", 2024-09-17T16:10:57.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7780000000000002e-06 2024-09-17T16:10:57.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.248432+0000", 2024-09-17T16:10:57.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0640000000000003e-05 2024-09-17T16:10:57.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.248450+0000", 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8270999999999998e-05 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.248936+0000", 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048620500000000002 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.248971+0000", 2024-09-17T16:10:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4903000000000001e-05 2024-09-17T16:10:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:75 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'53, mlcod=21'50)", 2024-09-17T16:10:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.252940+0000", 2024-09-17T16:10:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 287.95137202199999, 2024-09-17T16:10:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080258300000000005, 2024-09-17T16:10:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.252940+0000", 2024-09-17T16:10:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.252940+0000", 2024-09-17T16:10:57.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.252941+0000", 2024-09-17T16:10:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2200000000000002e-07 2024-09-17T16:10:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.252969+0000", 2024-09-17T16:10:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7980999999999999e-05 2024-09-17T16:10:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.253003+0000", 2024-09-17T16:10:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4134999999999998e-05 2024-09-17T16:10:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.253009+0000", 2024-09-17T16:10:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0920000000000003e-06 2024-09-17T16:10:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.253064+0000", 2024-09-17T16:10:57.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5058e-05 2024-09-17T16:10:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.253084+0000", 2024-09-17T16:10:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9760999999999999e-05 2024-09-17T16:10:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.253705+0000", 2024-09-17T16:10:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000621157 2024-09-17T16:10:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.253742+0000", 2024-09-17T16:10:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7477e-05 2024-09-17T16:10:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:80 1.0 e21/19 1:30306672:devicehealth::main.db.0000000000000000:head v 21'58, mlcod=21'53)", 2024-09-17T16:10:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.256023+0000", 2024-09-17T16:10:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 287.94828877600003, 2024-09-17T16:10:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00057048999999999999, 2024-09-17T16:10:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.298 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.256023+0000", 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.256023+0000", 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.256024+0000", 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5600000000000005e-07 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.256046+0000", 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2279000000000001e-05 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.256092+0000", 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6174000000000002e-05 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.256099+0000", 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7610000000000002e-06 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.256147+0000", 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8099999999999997e-05 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.256171+0000", 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3635e-05 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.256557+0000", 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00038599499999999998 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.256593+0000", 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6789999999999998e-05 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:90 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'64, mlcod=21'60)", 2024-09-17T16:10:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.321760+0000", 2024-09-17T16:10:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 287.88255178700001, 2024-09-17T16:10:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073320299999999998, 2024-09-17T16:10:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.321760+0000", 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.321760+0000", 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.321761+0000", 2024-09-17T16:10:57.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1299999999999999e-07 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.321808+0000", 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7043999999999999e-05 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.321850+0000", 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2583999999999998e-05 2024-09-17T16:10:57.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.321858+0000", 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3490000000000003e-06 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.321926+0000", 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8590000000000006e-05 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.321946+0000", 2024-09-17T16:10:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9439000000000001e-05 2024-09-17T16:10:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.322445+0000", 2024-09-17T16:10:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00049943199999999996 2024-09-17T16:10:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.322493+0000", 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8052e-05 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:99 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'73, mlcod=21'68)", 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.327020+0000", 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 287.87729187899998, 2024-09-17T16:10:57.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062468599999999995, 2024-09-17T16:10:57.310 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.310 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.310 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.310 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.327020+0000", 2024-09-17T16:10:57.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.327020+0000", 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.327021+0000", 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999999e-07 2024-09-17T16:10:57.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.327047+0000", 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6203999999999998e-05 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.327091+0000", 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4487999999999998e-05 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.327098+0000", 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.533e-06 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.327144+0000", 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6173e-05 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.327163+0000", 2024-09-17T16:10:57.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9153000000000001e-05 2024-09-17T16:10:57.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.327604+0000", 2024-09-17T16:10:57.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000441369 2024-09-17T16:10:57.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.327644+0000", 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0046000000000002e-05 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:112 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'82, mlcod=21'80)", 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.340071+0000", 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 287.86424039799999, 2024-09-17T16:10:57.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062733199999999998, 2024-09-17T16:10:57.316 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.316 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.316 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.316 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.340071+0000", 2024-09-17T16:10:57.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.340071+0000", 2024-09-17T16:10:57.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.340072+0000", 2024-09-17T16:10:57.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5099999999999999e-07 2024-09-17T16:10:57.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.340079+0000", 2024-09-17T16:10:57.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.041e-06 2024-09-17T16:10:57.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.340106+0000", 2024-09-17T16:10:57.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6698000000000001e-05 2024-09-17T16:10:57.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.319 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.319 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.340112+0000", 2024-09-17T16:10:57.319 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4609999999999996e-06 2024-09-17T16:10:57.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.319 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.319 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.340168+0000", 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5849e-05 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.340190+0000", 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1518e-05 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.340677+0000", 2024-09-17T16:10:57.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048783799999999999 2024-09-17T16:10:57.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.340699+0000", 2024-09-17T16:10:57.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175999999999999e-05 2024-09-17T16:10:57.321 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.321 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.14158.0:117 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'87, mlcod=21'84)", 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.342740+0000", 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 287.86157133199998, 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091486400000000004, 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.342740+0000", 2024-09-17T16:10:57.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.342740+0000", 2024-09-17T16:10:57.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.342741+0000", 2024-09-17T16:10:57.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6000000000000003e-07 2024-09-17T16:10:57.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.342768+0000", 2024-09-17T16:10:57.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6911999999999999e-05 2024-09-17T16:10:57.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.342803+0000", 2024-09-17T16:10:57.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5425e-05 2024-09-17T16:10:57.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.342811+0000", 2024-09-17T16:10:57.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1829999999999997e-06 2024-09-17T16:10:57.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.342877+0000", 2024-09-17T16:10:57.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6031999999999994e-05 2024-09-17T16:10:57.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:10:57.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.342888+0000", 2024-09-17T16:10:57.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1486e-05 2024-09-17T16:10:57.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.327 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:10:57.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.343608+0000", 2024-09-17T16:10:57.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00071961300000000002 2024-09-17T16:10:57.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.343655+0000", 2024-09-17T16:10:57.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7453000000000001e-05 2024-09-17T16:10:57.328 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.328 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.328 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.329 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPull(1.0 e56/55 cost 1049576)", 2024-09-17T16:10:57.329 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:08:12.566373+0000", 2024-09-17T16:10:57.329 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 164.63793859699999, 2024-09-17T16:10:57.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000667203, 2024-09-17T16:10:57.329 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:10:57.330 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:10:57.330 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:10:57.330 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:10:57.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:10:57.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:08:12.566373+0000", 2024-09-17T16:10:57.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:10:57.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:08:12.566373+0000", 2024-09-17T16:10:57.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:10:57.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:10:57.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:08:12.566375+0000", 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2330000000000001e-06 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:08:12.566428+0000", 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2578999999999999e-05 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:08:12.566505+0000", 2024-09-17T16:10:57.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6879999999999996e-05 2024-09-17T16:10:57.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:10:57.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:08:12.566518+0000", 2024-09-17T16:10:57.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3016e-05 2024-09-17T16:10:57.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:10:57.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:08:12.566622+0000", 2024-09-17T16:10:57.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010406 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:08:12.567040+0000", 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00041843499999999999 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:10:57.334 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:10:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:57.336 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:57.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:10:57.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:10:57.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:57.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:10:57.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:57 smithi017 ceph-mon[27571]: pgmap v300: 17 pgs: 8 unknown, 9 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:57 smithi071 ceph-mon[30002]: pgmap v300: 17 pgs: 8 unknown, 9 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:57.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:57.591 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:57.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:57.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:57 smithi173 ceph-mon[30530]: pgmap v300: 17 pgs: 8 unknown, 9 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:57.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:57.720 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:57.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:57.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:57.822 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:57.847 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:57.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:57.948 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:57.974 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:57.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:57.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:58.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:10:58.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:10:58.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:58.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:58.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:10:58.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:10:58.135 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:58.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:58.226 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:58.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:58.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:58.353 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:58.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:58.455 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:58.477 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:58.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:58.579 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:10:58.605 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:58.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:10:58.699 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:58.706 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:10:58.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:10:58.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:58.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:10:58.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:10:58.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:58.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:10:58.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:58.987 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:58.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:58.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:59.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:10:59.088 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:10:59.114 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:59.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:10:59.155 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:10:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:10:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:10:59.155 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:10:59.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:59.237 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:59.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:59.288 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-17T16:10:59.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:10:59.362 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:59.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:10:59.399 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:59 smithi017 ceph-mon[27571]: pgmap v301: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:59.399 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:10:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/359865461' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:10:59.464 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:59.489 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:59.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:10:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:59 smithi071 ceph-mon[30002]: pgmap v301: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:10:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/359865461' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:10:59.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:10:59.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:10:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:59.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:10:59.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:59 smithi173 ceph-mon[30530]: pgmap v301: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:10:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:10:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/359865461' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:10:59.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:10:59.745 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:59.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:10:59.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:10:59.869 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:10:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:59.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:10:59.970 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:10:59.996 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:10:59.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:10:59.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:00.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:00.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:00.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:00.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:00.253 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:00.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:00.355 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:00.380 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:00.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:00.427 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:11:00.482 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:00.504 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:00.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:00.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/359865461' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:11:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:00 smithi071 ceph-mon[30002]: osdmap e98: 12 total, 12 up, 12 in 2024-09-17T16:11:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4259304695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:00.606 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:00.628 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:00.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:00.641 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:11:00.641 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:11:00.641 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/359865461' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:11:00.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:00 smithi017 ceph-mon[27571]: osdmap e98: 12 total, 12 up, 12 in 2024-09-17T16:11:00.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4259304695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:00.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/359865461' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:11:00.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:00 smithi173 ceph-mon[30530]: osdmap e98: 12 total, 12 up, 12 in 2024-09-17T16:11:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4259304695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:00.662 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:11:00.662 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:11:00.684 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:11:00.684 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:11:00.706 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:11:00.706 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:11:00.729 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:11:00.729 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:11:00.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:00.742 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:00.756 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:11:00.756 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:11:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:00.757 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:00.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:00.778 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:11:00.779 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:11:00.802 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:11:00.802 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:11:00.824 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:11:00.824 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:11:00.846 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:11:00.846 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:11:00.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:00.868 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:11:00.868 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:11:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:00.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:00.889 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:11:00.890 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:11:00.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:01.014 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:01.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:01.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:01.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:01.142 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:01.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:01.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:01.244 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:01.262 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:01.268 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:01.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:01.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:01.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:01.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:01.497 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:01.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:01.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:01.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:01 smithi017 ceph-mon[27571]: pgmap v303: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:01.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:01 smithi017 ceph-mon[27571]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:11:01.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:01 smithi071 ceph-mon[30002]: pgmap v303: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:01.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:01 smithi071 ceph-mon[30002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:11:01.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:01.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:01 smithi173 ceph-mon[30530]: pgmap v303: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:01 smithi173 ceph-mon[30530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:11:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:01.645 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:01.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:01.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:01.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:01.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:01.874 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:01.896 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:01.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:01.998 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:02.020 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:02.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:02.122 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:02.128 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-17T16:11:02.128 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-17T16:11:02.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:11:02.131 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T16:11:02.131 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T16:11:02.147 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:02.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:02.207 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-09-17T16:11:02.207 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T16:11:02.207 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_55121 2024-09-17T16:11:02.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:11:02.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-09-17T16:11:02.249 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:02.289 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:02.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:02.390 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:02.421 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:02.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:02.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:02.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2973875513' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:02.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:02.524 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:11:02.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:02.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:02.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2973875513' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:02.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2973875513' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:02.648 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:02.670 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:02.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:02.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:02.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:02.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:02.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:02.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:02.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:02.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:03.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:03.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:03.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:03.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:03.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:03.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:03.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:03.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 1819 1803 112.681 112.688 0.0105504 0.00874198 2024-09-17T16:11:03.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:03.289 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:03.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:03.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:03.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:03.436 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:03.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:03.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:03.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:03 smithi017 ceph-mon[27571]: pgmap v304: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:03.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:03 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:03.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2973875513' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:11:03.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:03 smithi017 ceph-mon[27571]: osdmap e99: 12 total, 12 up, 12 in 2024-09-17T16:11:03.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:03.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:03 smithi071 ceph-mon[30002]: pgmap v304: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:03.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:03 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:03.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2973875513' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:11:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:03 smithi071 ceph-mon[30002]: osdmap e99: 12 total, 12 up, 12 in 2024-09-17T16:11:03.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:03.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:03.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:03.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:03 smithi173 ceph-mon[30530]: pgmap v304: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:03 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2973875513' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:11:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:03 smithi173 ceph-mon[30530]: osdmap e99: 12 total, 12 up, 12 in 2024-09-17T16:11:03.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:03.693 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:03.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:03.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:03.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:03.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:03.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:03.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:03.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:03.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:04.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:04.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:04.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:04.095 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:04.095 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:04.169 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:04.193 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:04.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:04.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:04.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 2896 2880 89.9941 67.3125 0.00446039 0.0110648 2024-09-17T16:11:04.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:04.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:04.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:04.366 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:11:04.421 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:04.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:04.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:04.551 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:04.577 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:04.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:04.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:04.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:04.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:04.783 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:11:04.783 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:11:04.803 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:04.824 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:04.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:04.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:04.948 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:04.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:04.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:05.050 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:05.077 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:05.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:05.179 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:05.205 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:05.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:05.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 3983 3967 82.64 67.9375 0.00125841 0.0120452 2024-09-17T16:11:05.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:11:05.307 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:05.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:05.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:05.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:05.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:05.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:05.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:05.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:05 smithi017 ceph-mon[27571]: pgmap v306: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:05 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:11:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:05 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:11:05.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:05 smithi071 ceph-mon[30002]: pgmap v306: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:05 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:11:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:05 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:11:05.591 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:05.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:05.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:05 smithi173 ceph-mon[30530]: pgmap v306: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:05.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:05 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:11:05.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:05 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:11:05.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:05.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:05.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:05.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:05.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:05.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:05.947 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:05.971 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:05.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:06.059 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-09-17T16:11:03.723553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":343348,"kb_used_data":10692,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124598476,"statfs":{"total":1151940427776,"available":1151588839424,"internally_reserved":0,"allocated":10948608,"data_stored":6532648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":340374842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001587"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.288752+0000","last_change":"2024-09-17T16:10:54.283333+0000","last_active":"2024-09-17T16:10:59.288752+0000","last_peered":"2024-09-17T16:10:59.288752+0000","last_clean":"2024-09-17T16:10:59.288752+0000","last_became_active":"2024-09-17T16:10:54.282931+0000","last_became_peered":"2024-09-17T16:10:54.282931+0000","last_unstale":"2024-09-17T16:10:59.288752+0000","last_undegraded":"2024-09-17T16:10:59.288752+0000","last_fullsized":"2024-09-17T16:10:59.288752+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:57.200088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.769607+0000","last_change":"2024-09-17T16:10:54.283952+0000","last_active":"2024-09-17T16:10:59.769607+0000","last_peered":"2024-09-17T16:10:59.769607+0000","last_clean":"2024-09-17T16:10:59.769607+0000","last_became_active":"2024-09-17T16:10:54.283208+0000","last_became_peered":"2024-09-17T16:10:54.283208+0000","last_unstale":"2024-09-17T16:10:59.769607+0000","last_undegraded":"2024-09-17T16:10:59.769607+0000","last_fullsized":"2024-09-17T16:10:59.769607+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:11.177395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.526348+0000","last_change":"2024-09-17T16:10:54.283371+0000","last_active":"2024-09-17T16:10:59.526348+0000","last_peered":"2024-09-17T16:10:59.526348+0000","last_clean":"2024-09-17T16:10:59.526348+0000","last_became_active":"2024-09-17T16:10:54.283016+0000","last_became_peered":"2024-09-17T16:10:54.283016+0000","last_unstale":"2024-09-17T16:10:59.526348+0000","last_undegraded":"2024-09-17T16:10:59.526348+0000","last_fullsized":"2024-09-17T16:10:59.526348+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:05.847220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.308841+0000","last_change":"2024-09-17T16:10:54.282319+0000","last_active":"2024-09-17T16:10:59.308841+0000","last_peered":"2024-09-17T16:10:59.308841+0000","last_clean":"2024-09-17T16:10:59.308841+0000","last_became_active":"2024-09-17T16:10:54.282069+0000","last_became_peered":"2024-09-17T16:10:54.282069+0000","last_unstale":"2024-09-17T16:10:59.308841+0000","last_undegraded":"2024-09-17T16:10:59.308841+0000","last_fullsized":"2024-09-17T16:10:59.308841+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:55.441797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":140,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.770381+0000","last_change":"2024-09-17T16:09:51.306176+0000","last_active":"2024-09-17T16:10:59.770381+0000","last_peered":"2024-09-17T16:10:59.770381+0000","last_clean":"2024-09-17T16:10:59.770381+0000","last_became_active":"2024-09-17T16:09:51.301092+0000","last_became_peered":"2024-09-17T16:09:51.301092+0000","last_unstale":"2024-09-17T16:10:59.770381+0000","last_undegraded":"2024-09-17T16:10:59.770381+0000","last_fullsized":"2024-09-17T16:10:59.770381+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:09:43.462563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:09:43.462563+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:01.928744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.770119+0000","last_change":"2024-09-17T16:10:54.287397+0000","last_active":"2024-09-17T16:10:59.770119+0000","last_peered":"2024-09-17T16:10:59.770119+0000","last_clean":"2024-09-17T16:10:59.770119+0000","last_became_active":"2024-09-17T16:10:54.287032+0000","last_became_peered":"2024-09-17T16:10:54.287032+0000","last_unstale":"2024-09-17T16:10:59.770119+0000","last_undegraded":"2024-09-17T16:10:59.770119+0000","last_fullsized":"2024-09-17T16:10:59.770119+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:04.381607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.770386+0000","last_change":"2024-09-17T16:10:54.342019+0000","last_active":"2024-09-17T16:10:59.770386+0000","last_peered":"2024-09-17T16:10:59.770386+0000","last_clean":"2024-09-17T16:10:59.770386+0000","last_became_active":"2024-09-17T16:10:54.341667+0000","last_became_peered":"2024-09-17T16:10:54.341667+0000","last_unstale":"2024-09-17T16:10:59.770386+0000","last_undegraded":"2024-09-17T16:10:59.770386+0000","last_fullsized":"2024-09-17T16:10:59.770386+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:16.075485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.769592+0000","last_change":"2024-09-17T16:10:54.283572+0000","last_active":"2024-09-17T16:10:59.769592+0000","last_peered":"2024-09-17T16:10:59.769592+0000","last_clean":"2024-09-17T16:10:59.769592+0000","last_became_active":"2024-09-17T16:10:54.283241+0000","last_became_peered":"2024-09-17T16:10:54.283241+0000","last_unstale":"2024-09-17T16:10:59.769592+0000","last_undegraded":"2024-09-17T16:10:59.769592+0000","last_fullsized":"2024-09-17T16:10:59.769592+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:20.676266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.770119+0000","last_change":"2024-09-17T16:10:54.287658+0000","last_active":"2024-09-17T16:10:59.770119+0000","last_peered":"2024-09-17T16:10:59.770119+0000","last_clean":"2024-09-17T16:10:59.770119+0000","last_became_active":"2024-09-17T16:10:54.287147+0000","last_became_peered":"2024-09-17T16:10:54.287147+0000","last_unstale":"2024-09-17T16:10:59.770119+0000","last_undegraded":"2024-09-17T16:10:59.770119+0000","last_fullsized":"2024-09-17T16:10:59.770119+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:07.347265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:11:00.248584+0000","last_change":"2024-09-17T16:10:54.341903+0000","last_active":"2024-09-17T16:11:00.248584+0000","last_peered":"2024-09-17T16:11:00.248584+0000","last_clean":"2024-09-17T16:11:00.248584+0000","last_became_active":"2024-09-17T16:10:54.341559+0000","last_became_peered":"2024-09-17T16:10:54.341559+0000","last_unstale":"2024-09-17T16:11:00.248584+0000","last_undegraded":"2024-09-17T16:11:00.248584+0000","last_fullsized":"2024-09-17T16:11:00.248584+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:01.693664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.288750+0000","last_change":"2024-09-17T16:10:54.283988+0000","last_active":"2024-09-17T16:10:59.288750+0000","last_peered":"2024-09-17T16:10:59.288750+0000","last_clean":"2024-09-17T16:10:59.288750+0000","last_became_active":"2024-09-17T16:10:54.283202+0000","last_became_peered":"2024-09-17T16:10:54.283202+0000","last_unstale":"2024-09-17T16:10:59.288750+0000","last_undegraded":"2024-09-17T16:10:59.288750+0000","last_fullsized":"2024-09-17T16:10:59.288750+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:11.916246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:11:00.248578+0000","last_change":"2024-09-17T16:10:54.297867+0000","last_active":"2024-09-17T16:11:00.248578+0000","last_peered":"2024-09-17T16:11:00.248578+0000","last_clean":"2024-09-17T16:11:00.248578+0000","last_became_active":"2024-09-17T16:10:54.297528+0000","last_became_peered":"2024-09-17T16:10:54.297528+0000","last_unstale":"2024-09-17T16:11:00.248578+0000","last_undegraded":"2024-09-17T16:11:00.248578+0000","last_fullsized":"2024-09-17T16:11:00.248578+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:19.295333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.770174+0000","last_change":"2024-09-17T16:10:54.342105+0000","last_active":"2024-09-17T16:10:59.770174+0000","last_peered":"2024-09-17T16:10:59.770174+0000","last_clean":"2024-09-17T16:10:59.770174+0000","last_became_active":"2024-09-17T16:10:54.341697+0000","last_became_peered":"2024-09-17T16:10:54.341697+0000","last_unstale":"2024-09-17T16:10:59.770174+0000","last_undegraded":"2024-09-17T16:10:59.770174+0000","last_fullsized":"2024-09-17T16:10:59.770174+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:23.128622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.770216+0000","last_change":"2024-09-17T16:10:54.287366+0000","last_active":"2024-09-17T16:10:59.770216+0000","last_peered":"2024-09-17T16:10:59.770216+0000","last_clean":"2024-09-17T16:10:59.770216+0000","last_became_active":"2024-09-17T16:10:54.287046+0000","last_became_peered":"2024-09-17T16:10:54.287046+0000","last_unstale":"2024-09-17T16:10:59.770216+0000","last_undegraded":"2024-09-17T16:10:59.770216+0000","last_fullsized":"2024-09-17T16:10:59.770216+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:59.280480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:11:00.248587+0000","last_change":"2024-09-17T16:10:54.286599+0000","last_active":"2024-09-17T16:11:00.248587+0000","last_peered":"2024-09-17T16:11:00.248587+0000","last_clean":"2024-09-17T16:11:00.248587+0000","last_became_active":"2024-09-17T16:10:54.286304+0000","last_became_peered":"2024-09-17T16:10:54.286304+0000","last_unstale":"2024-09-17T16:11:00.248587+0000","last_undegraded":"2024-09-17T16:11:00.248587+0000","last_fullsized":"2024-09-17T16:11:00.248587+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:10.446781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.308831+0000","last_change":"2024-09-17T16:10:54.283470+0000","last_active":"2024-09-17T16:10:59.308831+0000","last_peered":"2024-09-17T16:10:59.308831+0000","last_clean":"2024-09-17T16:10:59.308831+0000","last_became_active":"2024-09-17T16:10:54.283026+0000","last_became_peered":"2024-09-17T16:10:54.283026+0000","last_unstale":"2024-09-17T16:10:59.308831+0000","last_undegraded":"2024-09-17T16:10:59.308831+0000","last_fullsized":"2024-09-17T16:10:59.308831+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:04.432087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":17,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-17T16:10:59.288737+0000","last_change":"2024-09-17T16:10:54.286746+0000","last_active":"2024-09-17T16:10:59.288737+0000","last_peered":"2024-09-17T16:10:59.288737+0000","last_clean":"2024-09-17T16:10:59.288737+0000","last_became_active":"2024-09-17T16:10:54.286405+0000","last_became_peered":"2024-09-17T16:10:54.286405+0000","last_unstale":"2024-09-17T16:10:59.288737+0000","last_undegraded":"2024-09-17T16:10:59.288737+0000","last_fullsized":"2024-09-17T16:10:59.288737+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:14.052241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285586,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28300,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716852,"statfs":{"total":95995035648,"available":95966056448,"internally_reserved":0,"allocated":1302528,"data_stored":935041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547222,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":782336,"data_stored":414165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743452,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":782336,"data_stored":414165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005088,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27780,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717372,"statfs":{"total":95995035648,"available":95966588928,"internally_reserved":0,"allocated":770048,"data_stored":406133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234021,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28312,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716840,"statfs":{"total":95995035648,"available":95966044160,"internally_reserved":0,"allocated":1314816,"data_stored":943093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28300,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716852,"statfs":{"total":95995035648,"available":95966056448,"internally_reserved":0,"allocated":1302528,"data_stored":935061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":782336,"data_stored":414165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986228,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":782336,"data_stored":414165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":782336,"data_stored":414165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":782336,"data_stored":414165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":782336,"data_stored":414165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":782336,"data_stored":414165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:11:06.061 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-17T16:11:06.061 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pg_num 26 2024-09-17T16:11:06.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:06.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:06.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:06.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:06.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:06.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 5593 5577 87.1345 100.625 0.0692998 0.0112745 2024-09-17T16:11:06.226 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:06.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:06.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:06.251 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:06.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:11:06.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:06.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:06.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:06.353 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:06.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:06.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:06.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:06.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:06.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:06.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:06.605 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:06.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:06 smithi173 ceph-mon[30530]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:11:06.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:06 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:11:06.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:06 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:11:06.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:06 smithi173 ceph-mon[30530]: osdmap e100: 12 total, 12 up, 12 in 2024-09-17T16:11:06.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:06 smithi017 ceph-mon[27571]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:11:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:06 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:11:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:06 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:11:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:06 smithi017 ceph-mon[27571]: osdmap e100: 12 total, 12 up, 12 in 2024-09-17T16:11:06.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:06.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:06.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:06.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:06 smithi071 ceph-mon[30002]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:11:06.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:06 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:11:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:06 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:11:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:06 smithi071 ceph-mon[30002]: osdmap e100: 12 total, 12 up, 12 in 2024-09-17T16:11:06.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:06.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:06.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:06.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:06.980 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:06.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:07.013 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:07.038 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:07.082 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:07.104 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:07.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:07.206 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:07.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 6924 6908 86.3438 83.1875 0.00146752 0.0115641 2024-09-17T16:11:07.237 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:07.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:07.325 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:11:07.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:07.360 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:07.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:07.462 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:07.489 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:07.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:07.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:07.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:07.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:07.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:07 smithi173 ceph-mon[30530]: pgmap v308: 17 pgs: 17 active+clean; 105 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 260 op/s 2024-09-17T16:11:07.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/813587968' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:07 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:07.715 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:07.723 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:07 smithi017 ceph-mon[27571]: pgmap v308: 17 pgs: 17 active+clean; 105 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 260 op/s 2024-09-17T16:11:07.723 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/813587968' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:07.723 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:07 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:07.744 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:07.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:07 smithi071 ceph-mon[30002]: pgmap v308: 17 pgs: 17 active+clean; 105 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 260 op/s 2024-09-17T16:11:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/813587968' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:07 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:07.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:07.867 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:07.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:07.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:07.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:07.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:07.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:08.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:08.115 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:08.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:08.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:08.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 7866 7850 81.765 58.875 0.00423352 0.0121905 2024-09-17T16:11:08.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:08.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:08.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:08.340 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:08.366 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:08.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:08.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:08 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:11:08.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:08 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:11:08.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:08 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:08.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:08 smithi017 ceph-mon[27571]: osdmap e101: 12 total, 12 up, 12 in 2024-09-17T16:11:08.468 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:08.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:08.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:08.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:08.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:08.623 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:08.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:08 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:11:08.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:08 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:11:08.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:08 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:08 smithi173 ceph-mon[30530]: osdmap e101: 12 total, 12 up, 12 in 2024-09-17T16:11:08.715 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:11:08.715 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:11:08.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:08.737 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:11:08.737 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:11:08.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:08.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:08.757 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:11:08.757 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:11:08.778 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:11:08.779 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:11:08.800 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:11:08.800 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:11:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:08 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:11:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:08 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:11:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:08 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:08 smithi071 ceph-mon[30002]: osdmap e101: 12 total, 12 up, 12 in 2024-09-17T16:11:08.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:11:08.822 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:11:08.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:08.855 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:11:08.855 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:11:08.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:08.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:08.890 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:11:08.890 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:11:08.924 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:11:08.924 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:11:08.947 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:08.953 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:11:08.953 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:11:08.977 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:08.986 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:11:08.986 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:11:08.995 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:08.079913+0000", 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 300.90943354900003, 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86337366999999998, 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:08.996 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.14158", 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.17:0/2484390953", 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 1 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079913+0000", 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079913+0000", 2024-09-17T16:11:08.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079915+0000", 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0330000000000001e-06 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079920+0000", 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5210000000000002e-06 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:08.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079929+0000", 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4490000000000005e-06 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079939+0000", 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3519999999999996e-06 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.080004+0000", 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5324e-05 2024-09-17T16:11:08.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.080011+0000", 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5300000000000002e-06 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.942061+0000", 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86205003199999997 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.942214+0000", 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000153287 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.942381+0000", 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016673 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943022+0000", 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00064124199999999998 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943096+0000", 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3516999999999994e-05 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943190+0000", 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4339000000000006e-05 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943247+0000", 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6779e-05 2024-09-17T16:11:09.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943287+0000", 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0534999999999997e-05 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.14158.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.032711+0000", 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 299.95663647100002, 2024-09-17T16:11:09.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015120609, 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.14158", 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.17:0/2484390953", 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032711+0000", 2024-09-17T16:11:09.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032711+0000", 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032711+0000", 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1200000000000003e-07 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032788+0000", 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-05 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032795+0000", 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6900000000000003e-06 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032799+0000", 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6709999999999998e-06 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.037716+0000", 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049166219999999998 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.037786+0000", 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9882999999999995e-05 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.037861+0000", 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5547e-05 2024-09-17T16:11:09.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038657+0000", 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079570900000000002 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040540+0000", 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001882508 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.047736+0000", 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071965680000000004 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.047789+0000", 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3049e-05 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.047831+0000", 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1869999999999997e-05 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.14158.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.032807+0000", 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 299.95653966200001, 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015136002000000001, 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.14158", 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.17:0/2484390953", 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-17T16:11:09.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032807+0000", 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032807+0000", 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032808+0000", 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4999999999999998e-07 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032810+0000", 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0420000000000001e-06 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032813+0000", 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5480000000000002e-06 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032816+0000", 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7219999999999999e-06 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038075+0000", 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052588629999999999 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038127+0000", 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2389000000000003e-05 2024-09-17T16:11:09.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038195+0000", 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7421000000000004e-05 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038659+0000", 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046454200000000003 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040584+0000", 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001924299 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.047883+0000", 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007299463 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.047918+0000", 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5086000000000003e-05 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.047943+0000", 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5276999999999999e-05 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.14158.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:09.032820+0000", 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 299.956526844, 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015245229000000001, 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.016 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.14158", 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.17:0/2484390953", 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032820+0000", 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032820+0000", 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032820+0000", 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.17e-07 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032823+0000", 2024-09-17T16:11:09.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.108e-06 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032826+0000", 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3280000000000002e-06 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.032828+0000", 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038361+0000", 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005532978 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038409+0000", 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8149999999999998e-05 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038496+0000", 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6306999999999999e-05 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.038688+0000", 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000192134 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.040611+0000", 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019229010000000001 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.047993+0000", 2024-09-17T16:11:09.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00738253 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.048042+0000", 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8772e-05 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:09.048065+0000", 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3323e-05 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:8416 2.2 e101/95 2:42c9549b:::benchmark_data_smithi173_55121_object8415:head v 101'548, mlcod=101'545)", 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.672067+0000", 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.317279696, 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.10194278700000001, 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.672067+0000", 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.672067+0000", 2024-09-17T16:11:09.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.672068+0000", 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5700000000000001e-07 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.672114+0000", 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6168000000000002e-05 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.773523+0000", 2024-09-17T16:11:09.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.101408501 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.773531+0000", 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0660000000000004e-06 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.773579+0000", 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7700000000000001e-05 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.773593+0000", 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3879e-05 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.773981+0000", 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000388296 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.774010+0000", 2024-09-17T16:11:09.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.932e-05 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:8428 2.c e101/95 2:3eee65e5:::benchmark_data_smithi173_55121_object8427:head v 101'530, mlcod=101'528)", 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.719802+0000", 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.26954477500000001, 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.051433487999999999, 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.719802+0000", 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.719802+0000", 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.719803+0000", 2024-09-17T16:11:09.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4499999999999997e-07 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.719899+0000", 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6482000000000002e-05 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.770564+0000", 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.050665121 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.770572+0000", 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0609999999999996e-06 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.770632+0000", 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9375e-05 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.770648+0000", 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6481e-05 2024-09-17T16:11:09.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.771194+0000", 2024-09-17T16:11:09.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054563699999999997 2024-09-17T16:11:09.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.771236+0000", 2024-09-17T16:11:09.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1786000000000003e-05 2024-09-17T16:11:09.032 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:8430 2.c e101/95 2:3e4b8e81:::benchmark_data_smithi173_55121_object8429:head v 101'532, mlcod=101'529)", 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.771936+0000", 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.217410565, 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.067690420000000001, 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.033 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.771936+0000", 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.771936+0000", 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.771937+0000", 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6499999999999999e-07 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.772000+0000", 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2665999999999999e-05 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.837895+0000", 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.065895033000000006 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.837901+0000", 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5230000000000001e-06 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.837979+0000", 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7954999999999995e-05 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.838001+0000", 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1946e-05 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.839578+0000", 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015764539999999999 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.839627+0000", 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9277999999999998e-05 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.037 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:8432 2.6 2:691a695d:::benchmark_data_smithi173_55121_object8431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.774558+0000", 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.214789497, 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.059827847000000003, 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:11:09.038 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:09.039 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 8432 2024-09-17T16:11:09.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.039 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.774558+0000", 2024-09-17T16:11:09.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:09.041 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:09.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:09.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.774558+0000", 2024-09-17T16:11:09.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.774559+0000", 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.858e-06 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.774593+0000", 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3540000000000001e-05 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.774609+0000", 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5951999999999999e-05 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.774617+0000", 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5769999999999994e-06 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.774677+0000", 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9864000000000001e-05 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.774837+0000", 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000160159 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,8", 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.775006+0000", 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016847300000000001 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.775508+0000", 2024-09-17T16:11:09.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00050178199999999999 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.776029+0000", 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00052105000000000001 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.834259+0000", 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.058230674000000003 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.834334+0000", 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4299000000000001e-05 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.834385+0000", 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1619000000000002e-05 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.046 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:8435 2.2 e101/95 2:4e81512b:::benchmark_data_smithi173_55121_object8434:head v 101'550, mlcod=101'547)", 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.779351+0000", 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.20999585500000001, 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.05782408, 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.779351+0000", 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.779351+0000", 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.779352+0000", 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9500000000000001e-07 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.779389+0000", 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6822000000000001e-05 2024-09-17T16:11:09.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.836052+0000", 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.056662682999999998 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.836059+0000", 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3309999999999998e-06 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.836140+0000", 2024-09-17T16:11:09.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1358000000000003e-05 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.836159+0000", 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8505999999999999e-05 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.837011+0000", 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085216399999999998 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.837175+0000", 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016432099999999999 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:8581 2.6 2:63d27e46:::benchmark_data_smithi173_55121_object8580:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.875516+0000", 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.113831214, 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015080005, 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.051 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 8581 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.875516+0000", 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.875516+0000", 2024-09-17T16:11:09.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.875517+0000", 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4720000000000001e-06 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.875614+0000", 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.679e-05 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.875630+0000", 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5472999999999999e-05 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.875637+0000", 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2189999999999998e-06 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.875686+0000", 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8925999999999999e-05 2024-09-17T16:11:09.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.875828+0000", 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142113 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,8", 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.875994+0000", 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016622499999999999 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:11:09.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.876726+0000", 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073157799999999998 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.876974+0000", 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024797399999999998 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.890471+0000", 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013497512 2024-09-17T16:11:09.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.890541+0000", 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9882999999999995e-05 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.890596+0000", 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4839999999999997e-05 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:8591 2.c e101/95 2:3a122bf1:::benchmark_data_smithi173_55121_object8590:head v 101'541, mlcod=101'538)", 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.879041+0000", 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.11030614599999999, 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015157409, 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.879041+0000", 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.879041+0000", 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.879042+0000", 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5099999999999998e-07 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.879075+0000", 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2784000000000003e-05 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.893231+0000", 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014156007 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.893237+0000", 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4799999999999998e-06 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.893305+0000", 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7664999999999997e-05 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.893329+0000", 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4702000000000002e-05 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.894158+0000", 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082905700000000001 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.894198+0000", 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9863000000000001e-05 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:8638 2.c e101/95 2:3c37b32a:::benchmark_data_smithi173_55121_object8637:head v 101'542, mlcod=101'539)", 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.893033+0000", 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.096313723000000004, 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.018456464999999998, 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.893033+0000", 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.893033+0000", 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.893034+0000", 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5499999999999997e-07 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.893210+0000", 2024-09-17T16:11:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017570300000000001 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.910723+0000", 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017513309000000001 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.910730+0000", 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4389999999999997e-06 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.910870+0000", 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014076200000000001 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.910895+0000", 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4422000000000001e-05 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.911447+0000", 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00055197299999999998 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.911490+0000", 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3001999999999999e-05 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:8654 2.c e101/95 2:33876257:::benchmark_data_smithi173_55121_object8653:head v 101'543, mlcod=101'539)", 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.897495+0000", 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.091851854999999996, 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.043730935999999998, 2024-09-17T16:11:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.897495+0000", 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.897495+0000", 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.897496+0000", 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8400000000000003e-07 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.897615+0000", 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118644 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.940065+0000", 2024-09-17T16:11:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.042450580000000002 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.940072+0000", 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6440000000000003e-06 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.940602+0000", 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00053048400000000001 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:11:09.071 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:11:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.940627+0000", 2024-09-17T16:11:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4464000000000002e-05 2024-09-17T16:11:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.941195+0000", 2024-09-17T16:11:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00056814599999999995 2024-09-17T16:11:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.941226+0000", 2024-09-17T16:11:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1189999999999998e-05 2024-09-17T16:11:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:8661 2.6 2:6095000c:::benchmark_data_smithi173_55121_object8660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.899566+0000", 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.089780578, 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015471894, 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 8661 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.899566+0000", 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.899566+0000", 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.899568+0000", 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.054e-06 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.899622+0000", 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4698999999999999e-05 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.899633+0000", 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0933e-05 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.899640+0000", 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2359999999999998e-06 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.899739+0000", 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8614000000000001e-05 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.899873+0000", 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000133788 2024-09-17T16:11:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,8", 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.900148+0000", 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027472400000000001 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.901520+0000", 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0013726929999999999 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:11:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.901841+0000", 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032035000000000001 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.914906+0000", 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013065838 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.914956+0000", 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9932999999999998e-05 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.915038+0000", 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2032000000000003e-05 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:8691 2.c e101/95 2:31e43fb6:::benchmark_data_smithi173_55121_object8690:head v 101'544, mlcod=101'539)", 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.910614+0000", 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.078732924999999995, 2024-09-17T16:11:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.037195021000000002, 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.910614+0000", 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.910614+0000", 2024-09-17T16:11:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.910615+0000", 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1000000000000001e-06 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.910702+0000", 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6334000000000004e-05 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946105+0000", 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.035403423000000003 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946113+0000", 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1729999999999995e-06 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946307+0000", 2024-09-17T16:11:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193566 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946338+0000", 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0862999999999999e-05 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.947767+0000", 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0014293750000000001 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.947809+0000", 2024-09-17T16:11:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2187000000000001e-05 2024-09-17T16:11:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:8718 2.6 2:6f5092b2:::benchmark_data_smithi173_55121_object8717:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-17T16:11:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.920725+0000", 2024-09-17T16:11:09.086 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.068621850999999998, 2024-09-17T16:11:09.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.026723500000000001, 2024-09-17T16:11:09.086 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.086 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.086 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 8718 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.920725+0000", 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.920725+0000", 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.920726+0000", 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.153e-06 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.920754+0000", 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7250000000000002e-05 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.920767+0000", 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3353000000000001e-05 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.920792+0000", 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5279e-05 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.920859+0000", 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6490999999999997e-05 2024-09-17T16:11:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.920949+0000", 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9963999999999994e-05 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,8", 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.921081+0000", 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013267300000000001 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922506+0000", 2024-09-17T16:11:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0014251000000000001 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922545+0000", 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8843000000000001e-05 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.947385+0000", 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.024840168999999999 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.947419+0000", 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3445999999999998e-05 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.947449+0000", 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9779000000000001e-05 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:8724 2.4 2:299664e7:::benchmark_data_smithi173_55121_object8723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.922680+0000", 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.066667137000000001, 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015291193999999999, 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 8724 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922680+0000", 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922680+0000", 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922681+0000", 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4000000000000001e-07 2024-09-17T16:11:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922703+0000", 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2552999999999999e-05 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922712+0000", 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2969999999999992e-06 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922716+0000", 2024-09-17T16:11:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9979999999999997e-06 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922761+0000", 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4385000000000003e-05 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922856+0000", 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5189e-05 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:11:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.922943+0000", 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6649000000000003e-05 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.924068+0000", 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001125638 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.924346+0000", 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027778299999999998 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.937868+0000", 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013522152000000001 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.937927+0000", 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8381999999999998e-05 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.937971+0000", 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4428000000000002e-05 2024-09-17T16:11:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:8754 2.4 2:250bc09d:::benchmark_data_smithi173_55121_object8753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.936298+0000", 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.053049234000000001, 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016040368999999999, 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:11:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 8754 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.936298+0000", 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.936298+0000", 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.936300+0000", 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.317e-06 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.936442+0000", 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014187800000000001 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.936463+0000", 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1152999999999999e-05 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.936472+0000", 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-06 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.936550+0000", 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8178999999999998e-05 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.936699+0000", 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014829599999999999 2024-09-17T16:11:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.936850+0000", 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015136899999999999 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.937725+0000", 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087549999999999998 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.938066+0000", 2024-09-17T16:11:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00034098 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.952226+0000", 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014160069000000001 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.952287+0000", 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0906000000000003e-05 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.952338+0000", 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0822000000000002e-05 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:8766 2.c e101/95 2:33da71e7:::benchmark_data_smithi173_55121_object8765:head v 101'547, mlcod=101'541)", 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.946059+0000", 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.043287645, 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016096288, 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946059+0000", 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946059+0000", 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946060+0000", 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7999999999999996e-07 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946171+0000", 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000111085 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.960529+0000", 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014357553 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.960536+0000", 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3259999999999998e-06 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.961089+0000", 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00055352799999999996 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.961107+0000", 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7776000000000001e-05 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.962123+0000", 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010155170000000001 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.962156+0000", 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3022999999999999e-05 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:8767 2.c e101/95 2:355a4d5c:::benchmark_data_smithi173_55121_object8766:head v 101'548, mlcod=101'541)", 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:08.946541+0000", 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.042806074, 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017856277, 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946541+0000", 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946541+0000", 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946542+0000", 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7399999999999991e-07 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.946601+0000", 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8700999999999999e-05 2024-09-17T16:11:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.962763+0000", 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016162507 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.962767+0000", 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.07e-06 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.962813+0000", 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5974999999999999e-05 2024-09-17T16:11:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.962831+0000", 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8258999999999999e-05 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.964218+0000", 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0013861730000000001 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:08.964397+0000", 2024-09-17T16:11:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017961799999999999 2024-09-17T16:11:09.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.114 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:09.114 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:09.114 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:11:09.142 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:09.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:09.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:09.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 9393 9377 83.7172 95.4375 0.00126699 0.0119282 2024-09-17T16:11:09.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:09.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:09.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:09.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:09.429 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:09.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:09.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:09.560 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:09.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:09.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:09 smithi173 ceph-mon[30530]: pgmap v310: 17 pgs: 17 active+clean; 296 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 786 op/s 2024-09-17T16:11:09.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1332698575' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:09 smithi173 ceph-mon[30530]: osdmap e102: 12 total, 12 up, 12 in 2024-09-17T16:11:09.662 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:09.685 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:09.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:09.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:09 smithi017 ceph-mon[27571]: pgmap v310: 17 pgs: 17 active+clean; 296 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 786 op/s 2024-09-17T16:11:09.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1332698575' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:09.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:09 smithi017 ceph-mon[27571]: osdmap e102: 12 total, 12 up, 12 in 2024-09-17T16:11:09.786 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:09 smithi071 ceph-mon[30002]: pgmap v310: 17 pgs: 17 active+clean; 296 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 786 op/s 2024-09-17T16:11:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1332698575' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:09 smithi071 ceph-mon[30002]: osdmap e102: 12 total, 12 up, 12 in 2024-09-17T16:11:09.814 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:09.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:09.916 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:09.938 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:09.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:10.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:10.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:10.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:10.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:10.186 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:11:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:10.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:10.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:10.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 10903 10887 85.0485 94.375 0.0251896 0.0117254 2024-09-17T16:11:10.289 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:10.311 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:10.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:10.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:10.441 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:10.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:10.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:10.543 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:10.564 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:10.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:10.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:10 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:11:10.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:10 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:11:10.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:11:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:10 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:11:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:10 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:11:10.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:10.690 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:10.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:10.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:10 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:11:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:10 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:11:10.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:10.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:10.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:10.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:10.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:11.039 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:11.063 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:11.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:11.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:11.189 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:11.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:11.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:11.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 12053 12037 83.5839 71.875 0.0206746 0.0119231 2024-09-17T16:11:11.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:11.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:11.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:11.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:11.333 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 2 pg_num to 26 2024-09-17T16:11:11.416 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:11.438 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:11.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:11.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:11 smithi017 ceph-mon[27571]: pgmap v312: 17 pgs: 17 active+clean; 422 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:11:11.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/90750144' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:11:11.539 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:11.565 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:11.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:11.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:11 smithi173 ceph-mon[30530]: pgmap v312: 17 pgs: 17 active+clean; 422 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:11:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/90750144' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:11:11.667 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:11.691 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:11.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:11.779 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:11.787 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:11.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:11.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:11 smithi071 ceph-mon[30002]: pgmap v312: 17 pgs: 17 active+clean; 422 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:11:11.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/90750144' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:11:11.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:11.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:11.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:11.942 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:11.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:12.044 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:12.068 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:12.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:12.170 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:12.194 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:12.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:12.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 13475 13459 84.112 88.875 0.0422027 0.0118466 2024-09-17T16:11:12.296 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:12.321 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:12.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:12.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:12.423 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:12.444 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:12.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:12.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:12.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/90750144' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:11:12.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:12 smithi017 ceph-mon[27571]: osdmap e103: 12 total, 12 up, 12 in 2024-09-17T16:11:12.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:12 smithi017 ceph-mon[27571]: pgmap v314: 17 pgs: 17 active+clean; 703 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.59k op/s 2024-09-17T16:11:12.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:12 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:11:12.546 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:12.567 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:12.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:12.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/90750144' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:11:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:12 smithi173 ceph-mon[30530]: osdmap e103: 12 total, 12 up, 12 in 2024-09-17T16:11:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:12 smithi173 ceph-mon[30530]: pgmap v314: 17 pgs: 17 active+clean; 703 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.59k op/s 2024-09-17T16:11:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:12 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:11:12.668 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:12.691 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:12.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:12.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:12.793 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/90750144' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:11:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:12 smithi071 ceph-mon[30002]: osdmap e103: 12 total, 12 up, 12 in 2024-09-17T16:11:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:12 smithi071 ceph-mon[30002]: pgmap v314: 17 pgs: 17 active+clean; 703 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.59k op/s 2024-09-17T16:11:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:12 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:11:12.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:12.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:12.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:12.939 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:12.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:13.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:13.063 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:13.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:13.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:13.186 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:13.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:13.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 13540 13524 76.8345 4.0625 0.508303 0.0120969 2024-09-17T16:11:13.261 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:11:13.261 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:11:13.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:13.261 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:13535 2.11 2.63551011 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:11:13.261 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:11:13.261 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.92353427499999996, 2024-09-17T16:11:13.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.92355901500000004, 2024-09-17T16:11:13.261 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:11:13.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:11:13.261 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for peered", 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 13535 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:11:13.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335349+0000", 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3599999999999999e-06 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335378+0000", 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8965000000000001e-05 2024-09-17T16:11:13.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335391+0000", 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.257e-05 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335398+0000", 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3499999999999999e-06 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.367429+0000", 2024-09-17T16:11:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.032030422000000003 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.367439+0000", 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0451e-05 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 1 2024-09-17T16:11:13.265 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:11:13.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:13.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:13.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:13.345 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:11:13.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:13.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:13.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:13.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:13.556 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:11:13.576 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:13.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:13.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:13.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:13 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:11:13.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:13 smithi173 ceph-mon[30530]: osdmap e104: 12 total, 12 up, 12 in 2024-09-17T16:11:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3274873206' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:11:13.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:13.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:13.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:13.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:13.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:13 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:11:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:13 smithi017 ceph-mon[27571]: osdmap e104: 12 total, 12 up, 12 in 2024-09-17T16:11:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3274873206' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:11:13.804 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:13 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:11:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:13 smithi071 ceph-mon[30002]: osdmap e104: 12 total, 12 up, 12 in 2024-09-17T16:11:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3274873206' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:11:13.828 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:13.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:13.930 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:13.953 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:13.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:14.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:14.076 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:14.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:14.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:11:14.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:14.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:14.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:14.208 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 14892 14876 77.4726 84.5 0.00267741 0.0127652 2024-09-17T16:11:14.302 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:14.325 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:14.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:14.427 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:14.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:14.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:14.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:14 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3274873206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:11:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:14 smithi017 ceph-mon[27571]: osdmap e105: 12 total, 12 up, 12 in 2024-09-17T16:11:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:14 smithi017 ceph-mon[27571]: 2.8 scrub starts 2024-09-17T16:11:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:14 smithi017 ceph-mon[27571]: 2.7 scrub starts 2024-09-17T16:11:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:14 smithi017 ceph-mon[27571]: pgmap v317: 27 pgs: 10 unknown, 17 active+clean; 703 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-17T16:11:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:11:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:14 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:11:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:14 smithi017 ceph-mon[27571]: osdmap e106: 12 total, 12 up, 12 in 2024-09-17T16:11:14.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:14.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:14.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:14.573 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:14.623 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:14 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3274873206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:11:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:14 smithi173 ceph-mon[30530]: osdmap e105: 12 total, 12 up, 12 in 2024-09-17T16:11:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:14 smithi173 ceph-mon[30530]: 2.8 scrub starts 2024-09-17T16:11:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:14 smithi173 ceph-mon[30530]: 2.7 scrub starts 2024-09-17T16:11:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:14 smithi173 ceph-mon[30530]: pgmap v317: 27 pgs: 10 unknown, 17 active+clean; 703 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-17T16:11:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:14 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:11:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:14 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:11:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:14 smithi173 ceph-mon[30530]: osdmap e106: 12 total, 12 up, 12 in 2024-09-17T16:11:14.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:14.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:14.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:14.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:14.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:14 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3274873206' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:11:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:14 smithi071 ceph-mon[30002]: osdmap e105: 12 total, 12 up, 12 in 2024-09-17T16:11:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:14 smithi071 ceph-mon[30002]: 2.8 scrub starts 2024-09-17T16:11:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:14 smithi071 ceph-mon[30002]: 2.7 scrub starts 2024-09-17T16:11:14.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:14 smithi071 ceph-mon[30002]: pgmap v317: 27 pgs: 10 unknown, 17 active+clean; 703 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-17T16:11:14.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:14 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:11:14.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:14 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:11:14.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:14 smithi071 ceph-mon[30002]: osdmap e106: 12 total, 12 up, 12 in 2024-09-17T16:11:14.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:14.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:14.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:14.948 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:14.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:14.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:15.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:15.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:15.073 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:15.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:15.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:15.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:15.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:15.209 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 14968 14952 71.8785 4.75 0.00155988 0.0131581 2024-09-17T16:11:15.298 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:15.323 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:15.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:15.339 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:11:15.339 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:11:15.361 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:11:15.362 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:11:15.378 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:15.384 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:11:15.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:11:15.406 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:11:15.407 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:11:15.424 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:15.428 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:11:15.428 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:11:15.447 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:15.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:15.450 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:11:15.450 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:11:15.472 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:11:15.472 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:11:15.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:15 smithi017 ceph-mon[27571]: 2.5 scrub starts 2024-09-17T16:11:15.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:15 smithi017 ceph-mon[27571]: 2.1 scrub starts 2024-09-17T16:11:15.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:15 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:11:15.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:15 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:11:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:15 smithi017 ceph-mon[27571]: 2.4 scrub starts 2024-09-17T16:11:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:15 smithi017 ceph-mon[27571]: 2.3 scrub starts 2024-09-17T16:11:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:15 smithi017 ceph-mon[27571]: 2.8 scrub starts 2024-09-17T16:11:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:15 smithi017 ceph-mon[27571]: 2.17 scrub starts 2024-09-17T16:11:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:15 smithi017 ceph-mon[27571]: osdmap e107: 12 total, 12 up, 12 in 2024-09-17T16:11:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3018044675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:15.493 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:11:15.494 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:11:15.515 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:11:15.515 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:11:15.536 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:11:15.536 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:11:15.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:15.557 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:11:15.558 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:11:15.570 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:15.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:15.579 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:11:15.579 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:11:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:15 smithi173 ceph-mon[30530]: 2.5 scrub starts 2024-09-17T16:11:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:15 smithi173 ceph-mon[30530]: 2.1 scrub starts 2024-09-17T16:11:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:15 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:11:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:15 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:11:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:15 smithi173 ceph-mon[30530]: 2.4 scrub starts 2024-09-17T16:11:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:15 smithi173 ceph-mon[30530]: 2.3 scrub starts 2024-09-17T16:11:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:15 smithi173 ceph-mon[30530]: 2.8 scrub starts 2024-09-17T16:11:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:15 smithi173 ceph-mon[30530]: 2.17 scrub starts 2024-09-17T16:11:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:15 smithi173 ceph-mon[30530]: osdmap e107: 12 total, 12 up, 12 in 2024-09-17T16:11:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3018044675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:15.672 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:15.698 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:15.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:15.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:15.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:15 smithi071 ceph-mon[30002]: 2.5 scrub starts 2024-09-17T16:11:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:15 smithi071 ceph-mon[30002]: 2.1 scrub starts 2024-09-17T16:11:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:15 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:11:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:15 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:11:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:15 smithi071 ceph-mon[30002]: 2.4 scrub starts 2024-09-17T16:11:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:15 smithi071 ceph-mon[30002]: 2.3 scrub starts 2024-09-17T16:11:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:15 smithi071 ceph-mon[30002]: 2.8 scrub starts 2024-09-17T16:11:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:15 smithi071 ceph-mon[30002]: 2.17 scrub starts 2024-09-17T16:11:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:15 smithi071 ceph-mon[30002]: osdmap e107: 12 total, 12 up, 12 in 2024-09-17T16:11:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3018044675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:15.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:15.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:15.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:15.946 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:15.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:16.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:16.073 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:16.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:16.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:16.200 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:16.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:16.209 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 14968 14952 66.7439 0 - 0.0131581 2024-09-17T16:11:16.302 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:16.334 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:16.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:16.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:16.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:16.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:16.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:16.569 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:16.596 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:16.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:16.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:16 smithi017 ceph-mon[27571]: 2.5 scrub starts 2024-09-17T16:11:16.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:16 smithi017 ceph-mon[27571]: 2.1 scrub starts 2024-09-17T16:11:16.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:16 smithi017 ceph-mon[27571]: 2.9 scrub starts 2024-09-17T16:11:16.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:16 smithi017 ceph-mon[27571]: 2.2 scrub starts 2024-09-17T16:11:16.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:16 smithi017 ceph-mon[27571]: 2.0 scrub starts 2024-09-17T16:11:16.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:16 smithi017 ceph-mon[27571]: 2.7 scrub starts 2024-09-17T16:11:16.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:16 smithi017 ceph-mon[27571]: pgmap v320: 27 pgs: 1 active+clean+scrubbing, 9 unknown, 17 active+clean; 766 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 252 op/s 2024-09-17T16:11:16.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:16 smithi017 ceph-mon[27571]: osdmap e108: 12 total, 12 up, 12 in 2024-09-17T16:11:16.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:11:16.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:16 smithi173 ceph-mon[30530]: 2.5 scrub starts 2024-09-17T16:11:16.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:16 smithi173 ceph-mon[30530]: 2.1 scrub starts 2024-09-17T16:11:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:16 smithi173 ceph-mon[30530]: 2.9 scrub starts 2024-09-17T16:11:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:16 smithi173 ceph-mon[30530]: 2.2 scrub starts 2024-09-17T16:11:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:16 smithi173 ceph-mon[30530]: 2.0 scrub starts 2024-09-17T16:11:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:16 smithi173 ceph-mon[30530]: 2.7 scrub starts 2024-09-17T16:11:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:16 smithi173 ceph-mon[30530]: pgmap v320: 27 pgs: 1 active+clean+scrubbing, 9 unknown, 17 active+clean; 766 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 252 op/s 2024-09-17T16:11:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:16 smithi173 ceph-mon[30530]: osdmap e108: 12 total, 12 up, 12 in 2024-09-17T16:11:16.698 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:16.734 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:16.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:16.775 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:16.775 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:16.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:16 smithi071 ceph-mon[30002]: 2.5 scrub starts 2024-09-17T16:11:16.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:16 smithi071 ceph-mon[30002]: 2.1 scrub starts 2024-09-17T16:11:16.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:16 smithi071 ceph-mon[30002]: 2.9 scrub starts 2024-09-17T16:11:16.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:16 smithi071 ceph-mon[30002]: 2.2 scrub starts 2024-09-17T16:11:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:16 smithi071 ceph-mon[30002]: 2.0 scrub starts 2024-09-17T16:11:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:16 smithi071 ceph-mon[30002]: 2.7 scrub starts 2024-09-17T16:11:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:16 smithi071 ceph-mon[30002]: pgmap v320: 27 pgs: 1 active+clean+scrubbing, 9 unknown, 17 active+clean; 766 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s wr, 252 op/s 2024-09-17T16:11:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:16 smithi071 ceph-mon[30002]: osdmap e108: 12 total, 12 up, 12 in 2024-09-17T16:11:16.836 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:16.860 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:16.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:16.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:16.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:16.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:17.088 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:17.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:17.123 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:17.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:17.209 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 16404 16388 68.2771 44.875 0.0032245 0.0146335 2024-09-17T16:11:17.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:17.253 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:17.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:17.270 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-17T16:11:17.270 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:11:17.271 INFO:tasks.thrashosds.thrasher:Reweighting osd 10 to 0.7501252276656047 2024-09-17T16:11:17.271 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 10 0.7501252276656047 2024-09-17T16:11:17.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:17.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:17.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:17.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:17.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:17 smithi017 ceph-mon[27571]: 2.4 scrub starts 2024-09-17T16:11:17.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:17 smithi017 ceph-mon[27571]: 2.3 scrub starts 2024-09-17T16:11:17.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:17.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:17.601 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:11:17.610 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:17.617 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:17.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:17 smithi173 ceph-mon[30530]: 2.4 scrub starts 2024-09-17T16:11:17.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:17 smithi173 ceph-mon[30530]: 2.3 scrub starts 2024-09-17T16:11:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:17.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:17.762 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:17.786 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:17.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:17.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:17 smithi071 ceph-mon[30002]: 2.4 scrub starts 2024-09-17T16:11:17.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:17 smithi071 ceph-mon[30002]: 2.3 scrub starts 2024-09-17T16:11:17.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:17.912 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:17.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:18.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:18.040 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:18.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:18.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:18.142 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:18.173 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:18.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:18.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:18.209 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 17773 17757 69.3569 85.5625 0.00381441 0.0144024 2024-09-17T16:11:18.275 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:18.305 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:18.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:18.317 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:18.356 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:18.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:18 smithi017 ceph-mon[27571]: 2.6 scrub starts 2024-09-17T16:11:18.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:18 smithi017 ceph-mon[27571]: pgmap v322: 27 pgs: 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 remapped+peering, 17 active+clean; 947 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 892 op/s; 5229/45444 objects misplaced (11.506%); 1.1 MiB/s, 17 objects/s recovering 2024-09-17T16:11:18.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:18.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:18.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:18.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:11:18.534 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:18.566 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:18.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:18.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:18 smithi173 ceph-mon[30530]: 2.6 scrub starts 2024-09-17T16:11:18.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:18 smithi173 ceph-mon[30530]: pgmap v322: 27 pgs: 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 remapped+peering, 17 active+clean; 947 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 892 op/s; 5229/45444 objects misplaced (11.506%); 1.1 MiB/s, 17 objects/s recovering 2024-09-17T16:11:18.668 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:18.704 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:18.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:18.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:18.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:18.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:18 smithi071 ceph-mon[30002]: 2.6 scrub starts 2024-09-17T16:11:18.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:18 smithi071 ceph-mon[30002]: pgmap v322: 27 pgs: 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 remapped+peering, 17 active+clean; 947 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 892 op/s; 5229/45444 objects misplaced (11.506%); 1.1 MiB/s, 17 objects/s recovering 2024-09-17T16:11:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:18.843 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:18.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:18.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:18.968 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:18.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:19.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:19.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:19.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:19.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:19.209 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 18942 18926 69.5746 73.0625 0.00293603 0.0143445 2024-09-17T16:11:19.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:19.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:19.262 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:11:19.337 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:19.361 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:19.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:19.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:19.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:19.486 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:19.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:19.588 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:19.620 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:19.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:19.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:19.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:19 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:11:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:19 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:11:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:19 smithi173 ceph-mon[30530]: osdmap e109: 12 total, 12 up, 12 in 2024-09-17T16:11:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/694096453' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:19 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:19 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:19 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:19 smithi173 ceph-mon[30530]: osdmap e110: 12 total, 12 up, 12 in 2024-09-17T16:11:19.677 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:19.677 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:19.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:19 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:11:19.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:19 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:11:19.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:19 smithi017 ceph-mon[27571]: osdmap e109: 12 total, 12 up, 12 in 2024-09-17T16:11:19.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/694096453' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:19.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:19 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:19.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:19 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:19.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:19 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:19.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:19 smithi017 ceph-mon[27571]: osdmap e110: 12 total, 12 up, 12 in 2024-09-17T16:11:19.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:19.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:19.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:19.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:19 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:11:19.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:19 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:11:19.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:19 smithi071 ceph-mon[30002]: osdmap e109: 12 total, 12 up, 12 in 2024-09-17T16:11:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/694096453' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:19 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:19 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:19 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:19 smithi071 ceph-mon[30002]: osdmap e110: 12 total, 12 up, 12 in 2024-09-17T16:11:19.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:19.867 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:19.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:19.969 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:19.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:19.991 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:19.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:20.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:20.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:20.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:20.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:20.209 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 19361 19345 67.1641 26.1875 0.0174417 0.014547 2024-09-17T16:11:20.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:20.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:20.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:20.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:20.260 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.10 to 0.750125 (c008) 2024-09-17T16:11:20.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:20.363 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:20.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:20.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:20.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:11:20.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:20.504 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:20.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:20.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:20.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:20.629 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:20.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:20.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:20.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2706547947' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 0.7501252276656047} : dispatch 2024-09-17T16:11:20.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:20 smithi173 ceph-mon[30530]: pgmap v325: 27 pgs: 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 remapped+peering, 17 active+clean; 1023 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 881 op/s; 5229/49086 objects misplaced (10.653%); 1.0 MiB/s, 16 objects/s recovering 2024-09-17T16:11:20.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2706547947' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 0.7501252276656047}]': finished 2024-09-17T16:11:20.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:20 smithi173 ceph-mon[30530]: osdmap e111: 12 total, 12 up, 12 in 2024-09-17T16:11:20.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:20.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2706547947' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 0.7501252276656047} : dispatch 2024-09-17T16:11:20.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:20 smithi017 ceph-mon[27571]: pgmap v325: 27 pgs: 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 remapped+peering, 17 active+clean; 1023 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 881 op/s; 5229/49086 objects misplaced (10.653%); 1.0 MiB/s, 16 objects/s recovering 2024-09-17T16:11:20.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2706547947' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 0.7501252276656047}]': finished 2024-09-17T16:11:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:20 smithi017 ceph-mon[27571]: osdmap e111: 12 total, 12 up, 12 in 2024-09-17T16:11:20.758 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:20.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2706547947' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 0.7501252276656047} : dispatch 2024-09-17T16:11:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:20 smithi071 ceph-mon[30002]: pgmap v325: 27 pgs: 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 remapped+peering, 17 active+clean; 1023 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 881 op/s; 5229/49086 objects misplaced (10.653%); 1.0 MiB/s, 16 objects/s recovering 2024-09-17T16:11:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2706547947' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 0.7501252276656047}]': finished 2024-09-17T16:11:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:20 smithi071 ceph-mon[30002]: osdmap e111: 12 total, 12 up, 12 in 2024-09-17T16:11:20.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:20.883 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:20.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:20.985 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:21.006 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:21.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:21.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:21.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:21.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:21.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:21.209 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 19595 19579 64.3988 14.625 0.00167423 0.0149161 2024-09-17T16:11:21.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:21.253 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:21.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:21.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:21.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:21.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:21.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:21.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:21.539 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:21.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:21.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:21.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:21.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:21.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:21.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:21.787 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:21.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:21.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:21.911 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:21.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:22.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:22.035 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:22.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:22.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:22.158 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:22.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:22.209 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:11:22.208811+0000 min lat: 0.00105621 max lat: 2.09646 avg lat: 0.0153113 2024-09-17T16:11:22.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:11:22.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 19709 19693 61.535 7.125 0.00159387 0.0153113 2024-09-17T16:11:22.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:22.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:22.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:22.385 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:22.413 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:22.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:22.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:22.539 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:22.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:22.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:22 smithi071 ceph-mon[30002]: 2.18 scrub starts 2024-09-17T16:11:22.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:22 smithi071 ceph-mon[30002]: osdmap e112: 12 total, 12 up, 12 in 2024-09-17T16:11:22.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:22 smithi071 ceph-mon[30002]: 2.10 scrub starts 2024-09-17T16:11:22.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:22 smithi173 ceph-mon[30530]: 2.18 scrub starts 2024-09-17T16:11:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:22 smithi173 ceph-mon[30530]: osdmap e112: 12 total, 12 up, 12 in 2024-09-17T16:11:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:22 smithi173 ceph-mon[30530]: 2.10 scrub starts 2024-09-17T16:11:22.674 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:22.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:22.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:22 smithi017 ceph-mon[27571]: 2.18 scrub starts 2024-09-17T16:11:22.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:22 smithi017 ceph-mon[27571]: osdmap e112: 12 total, 12 up, 12 in 2024-09-17T16:11:22.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:22 smithi017 ceph-mon[27571]: 2.10 scrub starts 2024-09-17T16:11:22.776 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:22.798 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:22.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:22.900 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:22.921 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:22.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:23.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:23.044 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:23.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:23.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:23.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:23.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:23.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 20092 20076 59.7444 23.9375 0.0704953 0.0162207 2024-09-17T16:11:23.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:23.297 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:23.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:23.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:23.419 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:23.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:23.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:23.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:23.546 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:23 smithi071 ceph-mon[30002]: pgmap v328: 27 pgs: 1 remapped+peering, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 18 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s; 1008/57930 objects degraded (1.740%); 3506/57930 objects misplaced (6.052%); 38 MiB/s, 613 objects/s recovering 2024-09-17T16:11:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:23 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 1008/57930 objects degraded (1.740%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:11:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:23 smithi071 ceph-mon[30002]: osdmap e113: 12 total, 12 up, 12 in 2024-09-17T16:11:23.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:23 smithi173 ceph-mon[30530]: pgmap v328: 27 pgs: 1 remapped+peering, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 18 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s; 1008/57930 objects degraded (1.740%); 3506/57930 objects misplaced (6.052%); 38 MiB/s, 613 objects/s recovering 2024-09-17T16:11:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:23 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 1008/57930 objects degraded (1.740%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:11:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:23 smithi173 ceph-mon[30530]: osdmap e113: 12 total, 12 up, 12 in 2024-09-17T16:11:23.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:23.672 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:23.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:23.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:23 smithi017 ceph-mon[27571]: pgmap v328: 27 pgs: 1 remapped+peering, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 18 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s; 1008/57930 objects degraded (1.740%); 3506/57930 objects misplaced (6.052%); 38 MiB/s, 613 objects/s recovering 2024-09-17T16:11:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:23 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 1008/57930 objects degraded (1.740%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:11:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:23 smithi017 ceph-mon[27571]: osdmap e113: 12 total, 12 up, 12 in 2024-09-17T16:11:23.774 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:23.804 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:23.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:23.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:23.927 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:23.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:23.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:24.029 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:24.052 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:24.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:24.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:24.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:24.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:24.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 20250 20234 57.4775 9.875 0.00159469 0.0168512 2024-09-17T16:11:24.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:24.304 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:24.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:24.407 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:24.429 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:24.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:24.531 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:24.552 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:24.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:24.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:24 smithi071 ceph-mon[30002]: 2.13 scrub starts 2024-09-17T16:11:24.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:24 smithi071 ceph-mon[30002]: osdmap e114: 12 total, 12 up, 12 in 2024-09-17T16:11:24.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:24 smithi071 ceph-mon[30002]: osdmap e115: 12 total, 12 up, 12 in 2024-09-17T16:11:24.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:24 smithi173 ceph-mon[30530]: 2.13 scrub starts 2024-09-17T16:11:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:24 smithi173 ceph-mon[30530]: osdmap e114: 12 total, 12 up, 12 in 2024-09-17T16:11:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:24 smithi173 ceph-mon[30530]: osdmap e115: 12 total, 12 up, 12 in 2024-09-17T16:11:24.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:24.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:24.678 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:24.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:24 smithi017 ceph-mon[27571]: 2.13 scrub starts 2024-09-17T16:11:24.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:24 smithi017 ceph-mon[27571]: osdmap e114: 12 total, 12 up, 12 in 2024-09-17T16:11:24.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:24 smithi017 ceph-mon[27571]: osdmap e115: 12 total, 12 up, 12 in 2024-09-17T16:11:24.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:24.801 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:24.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:24.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:24.924 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:24.924 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:24.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:25.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:25.048 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:25.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:25.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:25.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:25.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:25.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:25.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 20640 20624 56.0381 24.375 0.00798881 0.0174037 2024-09-17T16:11:25.274 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:25.311 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:25.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:25.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:25 smithi017 ceph-mon[27571]: pgmap v331: 27 pgs: 1 remapped+peering, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 18 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 736 op/s; 1008/57930 objects degraded (1.740%); 3506/57930 objects misplaced (6.052%); 38 MiB/s, 613 objects/s recovering 2024-09-17T16:11:25.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:25 smithi017 ceph-mon[27571]: 2.15 scrub starts 2024-09-17T16:11:25.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:25 smithi017 ceph-mon[27571]: 2.12 scrub starts 2024-09-17T16:11:25.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:25 smithi017 ceph-mon[27571]: osdmap e116: 12 total, 12 up, 12 in 2024-09-17T16:11:25.414 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:25.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:25.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:25.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:25.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:11:25.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:11:25.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:11:25.530 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:11:25.530 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:11:25.530 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:11:25.533 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:11:25.533 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:11:25.538 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:25.556 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:11:25.557 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:11:25.561 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:25.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:25.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:25 smithi071 ceph-mon[30002]: pgmap v331: 27 pgs: 1 remapped+peering, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 18 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 736 op/s; 1008/57930 objects degraded (1.740%); 3506/57930 objects misplaced (6.052%); 38 MiB/s, 613 objects/s recovering 2024-09-17T16:11:25.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:25 smithi071 ceph-mon[30002]: 2.15 scrub starts 2024-09-17T16:11:25.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:25 smithi071 ceph-mon[30002]: 2.12 scrub starts 2024-09-17T16:11:25.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:25 smithi071 ceph-mon[30002]: osdmap e116: 12 total, 12 up, 12 in 2024-09-17T16:11:25.578 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:11:25.578 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:11:25.600 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:11:25.600 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:11:25.622 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:11:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:11:25.631 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:25.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:25 smithi173 ceph-mon[30530]: pgmap v331: 27 pgs: 1 remapped+peering, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 18 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 736 op/s; 1008/57930 objects degraded (1.740%); 3506/57930 objects misplaced (6.052%); 38 MiB/s, 613 objects/s recovering 2024-09-17T16:11:25.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:25 smithi173 ceph-mon[30530]: 2.15 scrub starts 2024-09-17T16:11:25.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:25 smithi173 ceph-mon[30530]: 2.12 scrub starts 2024-09-17T16:11:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:25 smithi173 ceph-mon[30530]: osdmap e116: 12 total, 12 up, 12 in 2024-09-17T16:11:25.645 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:11:25.645 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:11:25.663 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:25.670 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:11:25.670 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:11:25.683 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:25.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:25.691 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:11:25.692 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:11:25.716 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:11:25.717 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:11:25.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:25.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:25.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:25.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:25.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:25.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:26.050 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:26.077 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:26.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:26.179 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:26.213 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 21452 21436 55.8176 50.75 0.00239573 0.0178953 2024-09-17T16:11:26.217 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:26.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:26.291 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:11:26.291 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:11:26.292 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:11:26.292 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:11:26.319 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:26.341 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:26.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:26.443 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:26.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:26.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:26 smithi017 ceph-mon[27571]: 2.19 scrub starts 2024-09-17T16:11:26.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:26 smithi017 ceph-mon[27571]: 2.16 scrub starts 2024-09-17T16:11:26.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2110080092' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:26.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:26 smithi071 ceph-mon[30002]: 2.19 scrub starts 2024-09-17T16:11:26.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:26 smithi071 ceph-mon[30002]: 2.16 scrub starts 2024-09-17T16:11:26.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2110080092' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:26.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:26.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:26.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:26.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:26 smithi173 ceph-mon[30530]: 2.19 scrub starts 2024-09-17T16:11:26.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:26 smithi173 ceph-mon[30530]: 2.16 scrub starts 2024-09-17T16:11:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2110080092' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:26.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:26.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:26.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:26.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:11:26.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:26.841 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:26.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:26.943 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:26.964 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:26.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:27.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:27.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:27.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:27.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:27.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 21856 21840 54.5948 25.25 0.00126333 0.0180185 2024-09-17T16:11:27.211 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:27.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:27.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:27.342 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:27.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:27.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:27.471 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:27 smithi071 ceph-mon[30002]: pgmap v334: 27 pgs: 1 remapped+peering, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 193 op/s; 1008/60258 objects degraded (1.673%); 2162/60258 objects misplaced (3.588%); 9.2 MiB/s, 147 objects/s recovering 2024-09-17T16:11:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:27 smithi071 ceph-mon[30002]: osdmap e117: 12 total, 12 up, 12 in 2024-09-17T16:11:27.572 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:27.600 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:27.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:27.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:27.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:27 smithi173 ceph-mon[30530]: pgmap v334: 27 pgs: 1 remapped+peering, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 193 op/s; 1008/60258 objects degraded (1.673%); 2162/60258 objects misplaced (3.588%); 9.2 MiB/s, 147 objects/s recovering 2024-09-17T16:11:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:27 smithi173 ceph-mon[30530]: osdmap e117: 12 total, 12 up, 12 in 2024-09-17T16:11:27.702 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:27.728 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:27.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:27.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:27.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:27 smithi017 ceph-mon[27571]: pgmap v334: 27 pgs: 1 remapped+peering, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 21 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 193 op/s; 1008/60258 objects degraded (1.673%); 2162/60258 objects misplaced (3.588%); 9.2 MiB/s, 147 objects/s recovering 2024-09-17T16:11:27.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:27 smithi017 ceph-mon[27571]: osdmap e117: 12 total, 12 up, 12 in 2024-09-17T16:11:27.812 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:11:27.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:27.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:27.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:27.856 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:27.868 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:27.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:27.978 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:27.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:28.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:28.103 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:28.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:28.205 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:28.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 23420 23404 56.2543 97.75 0.0233395 0.0177644 2024-09-17T16:11:28.236 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:28.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:28.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:28.366 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:28.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:28.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:28.442 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:28.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:28.491 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:28.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:28.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:28.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:28 smithi071 ceph-mon[30002]: osdmap e118: 12 total, 12 up, 12 in 2024-09-17T16:11:28.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:28 smithi071 ceph-mon[30002]: 2.14 scrub starts 2024-09-17T16:11:28.569 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:28 smithi017 ceph-mon[27571]: osdmap e118: 12 total, 12 up, 12 in 2024-09-17T16:11:28.569 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:28 smithi017 ceph-mon[27571]: 2.14 scrub starts 2024-09-17T16:11:28.593 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:28.615 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:28.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:28.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:28 smithi173 ceph-mon[30530]: osdmap e118: 12 total, 12 up, 12 in 2024-09-17T16:11:28.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:28 smithi173 ceph-mon[30530]: 2.14 scrub starts 2024-09-17T16:11:28.717 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:28.742 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:28.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:28.844 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:28.866 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:28.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:28.967 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:28.977 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:11:28.977 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:11:28.977 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:11:28.977 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:11:28.977 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:11:28.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:28.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:29.052 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:11:29.093 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:29.115 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:29.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:29.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 23944 23928 55.3837 32.75 0.00155156 0.0178887 2024-09-17T16:11:29.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:29.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:29.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:29.265 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:11:29.341 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:29.371 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:29.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:29.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:29.497 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:29.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:29 smithi071 ceph-mon[30002]: pgmap v337: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 442 op/s; 1020/63246 objects misplaced (1.613%); 30 MiB/s, 486 objects/s recovering 2024-09-17T16:11:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:29 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1008/60258 objects degraded (1.673%), 1 pg degraded) 2024-09-17T16:11:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:29 smithi071 ceph-mon[30002]: osdmap e119: 12 total, 12 up, 12 in 2024-09-17T16:11:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2929985948' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:29 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:29 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:29 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:11:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:29 smithi071 ceph-mon[30002]: osdmap e120: 12 total, 12 up, 12 in 2024-09-17T16:11:29.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:29.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:29.622 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:29 smithi173 ceph-mon[30530]: pgmap v337: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 442 op/s; 1020/63246 objects misplaced (1.613%); 30 MiB/s, 486 objects/s recovering 2024-09-17T16:11:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:29 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1008/60258 objects degraded (1.673%), 1 pg degraded) 2024-09-17T16:11:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:29 smithi173 ceph-mon[30530]: osdmap e119: 12 total, 12 up, 12 in 2024-09-17T16:11:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2929985948' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:29 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:29 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:29 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:11:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:29 smithi173 ceph-mon[30530]: osdmap e120: 12 total, 12 up, 12 in 2024-09-17T16:11:29.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:29.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:29 smithi017 ceph-mon[27571]: pgmap v337: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 442 op/s; 1020/63246 objects misplaced (1.613%); 30 MiB/s, 486 objects/s recovering 2024-09-17T16:11:29.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:29 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1008/60258 objects degraded (1.673%), 1 pg degraded) 2024-09-17T16:11:29.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:29 smithi017 ceph-mon[27571]: osdmap e119: 12 total, 12 up, 12 in 2024-09-17T16:11:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2929985948' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:29 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:29 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:29 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:11:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:29 smithi017 ceph-mon[27571]: osdmap e120: 12 total, 12 up, 12 in 2024-09-17T16:11:29.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:29.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:29.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:29.869 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:29.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:29.922 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-17T16:11:29.922 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:11:29.922 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T16:11:29.922 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-09-17T16:11:29.922 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 0 2024-09-17T16:11:29.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:29.992 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:29.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:30.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:30.120 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:30.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:30.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 24414 24398 54.4547 29.375 0.00162299 0.0179959 2024-09-17T16:11:30.222 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:30.244 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:30.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:30.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:30.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:30.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:30.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:30.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:30.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:30.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:30.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:30.603 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:30.634 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:30.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:30.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:30.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:30.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:30.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:30.862 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:30.883 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:30.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:30.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:31.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:31.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:31.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:31.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:31.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:31.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 25275 25259 54.4323 53.8125 0.00182205 0.0181984 2024-09-17T16:11:31.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:31.254 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:31.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:31.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:31.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:31.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:31.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:31.510 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:31.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:31 smithi071 ceph-mon[30002]: pgmap v340: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 534 op/s; 1020/66678 objects misplaced (1.530%); 21 MiB/s, 338 objects/s recovering 2024-09-17T16:11:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:31 smithi071 ceph-mon[30002]: 2.4 scrub starts 2024-09-17T16:11:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:31 smithi071 ceph-mon[30002]: 2.4 scrub ok 2024-09-17T16:11:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:31 smithi071 ceph-mon[30002]: osdmap e121: 12 total, 12 up, 12 in 2024-09-17T16:11:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:31 smithi071 ceph-mon[30002]: 2.11 scrub starts 2024-09-17T16:11:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:31 smithi071 ceph-mon[30002]: 2.11 scrub ok 2024-09-17T16:11:31.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:31.637 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:31.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:31.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:31 smithi173 ceph-mon[30530]: pgmap v340: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 534 op/s; 1020/66678 objects misplaced (1.530%); 21 MiB/s, 338 objects/s recovering 2024-09-17T16:11:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:31 smithi173 ceph-mon[30530]: 2.4 scrub starts 2024-09-17T16:11:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:31 smithi173 ceph-mon[30530]: 2.4 scrub ok 2024-09-17T16:11:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:31 smithi173 ceph-mon[30530]: osdmap e121: 12 total, 12 up, 12 in 2024-09-17T16:11:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:31 smithi173 ceph-mon[30530]: 2.11 scrub starts 2024-09-17T16:11:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:31 smithi173 ceph-mon[30530]: 2.11 scrub ok 2024-09-17T16:11:31.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:31 smithi017 ceph-mon[27571]: pgmap v340: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 534 op/s; 1020/66678 objects misplaced (1.530%); 21 MiB/s, 338 objects/s recovering 2024-09-17T16:11:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:31 smithi017 ceph-mon[27571]: 2.4 scrub starts 2024-09-17T16:11:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:31 smithi017 ceph-mon[27571]: 2.4 scrub ok 2024-09-17T16:11:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:31 smithi017 ceph-mon[27571]: osdmap e121: 12 total, 12 up, 12 in 2024-09-17T16:11:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:31 smithi017 ceph-mon[27571]: 2.11 scrub starts 2024-09-17T16:11:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:31 smithi017 ceph-mon[27571]: 2.11 scrub ok 2024-09-17T16:11:31.739 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:31.761 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:31.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:31.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:31.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:31.891 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:31.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:32.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:32.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:32.118 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:32.148 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:32.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:32.210 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 26801 26785 55.7967 95.375 0.00154755 0.0179116 2024-09-17T16:11:32.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:32.274 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:32.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:32.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:32.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:32.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:32.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:32.499 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:32.521 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:32.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:32.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:32.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:32.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:32.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:32.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:32.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:32.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:32.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:32.895 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:32.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:32.927 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:11:32.944 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:32.948 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:32.998 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:33.019 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:33.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:33.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:33.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:33.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:33.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:33.211 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 27009 26993 54.4161 13 0.0014093 0.0179985 2024-09-17T16:11:33.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:33.272 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:33.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:33.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:33.374 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:33.383 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:33 smithi017 ceph-mon[27571]: pgmap v342: 27 pgs: 27 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 687 op/s 2024-09-17T16:11:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:33.399 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:33.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:33.500 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:33.525 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:33.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:33.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:33 smithi071 ceph-mon[30002]: pgmap v342: 27 pgs: 27 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 687 op/s 2024-09-17T16:11:33.626 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:33.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:33 smithi173 ceph-mon[30530]: pgmap v342: 27 pgs: 27 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 687 op/s 2024-09-17T16:11:33.676 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:33.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:33.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:33.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:33.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:33.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:33.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:33.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:33.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:33.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:33.977 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:11:33.978 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:11:33.999 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:11:33.999 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:11:34.019 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:11:34.019 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:11:34.024 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:34.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:11:34.042 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:11:34.049 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:34.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:34.063 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:11:34.063 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:11:34.084 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:11:34.085 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:11:34.107 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:11:34.107 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:11:34.128 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:11:34.129 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:11:34.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:34.152 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:11:34.152 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:11:34.154 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:34.175 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:34.175 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:11:34.176 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:11:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:34.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:34.197 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:11:34.197 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:11:34.211 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 27049 27033 52.7937 2.5 0.00146527 0.0180236 2024-09-17T16:11:34.219 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:11:34.219 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:11:34.262 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.0 primary-affinity to 0 (802) 2024-09-17T16:11:34.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:34.303 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:34.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:34.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:34.405 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:11:34.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:34.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:34.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:34.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:34.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:34.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/513987812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2788594215' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-17T16:11:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2788594215' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-17T16:11:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/513987812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2788594215' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-17T16:11:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2788594215' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-17T16:11:34.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:34.680 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:34.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/513987812' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2788594215' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-17T16:11:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2788594215' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-17T16:11:34.782 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:34.809 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:34.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:34.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:34.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:34.938 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:34.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:35.040 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:35.064 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:35.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:35.165 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:35.190 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:35.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:35.211 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 27049 27033 51.1939 0 - 0.0180236 2024-09-17T16:11:35.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:35.301 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:35.301 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:35.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:35.318 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:35.419 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:35.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:35.446 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:35.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:35.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:35 smithi017 ceph-mon[27571]: pgmap v343: 27 pgs: 27 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 508 op/s 2024-09-17T16:11:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:35 smithi017 ceph-mon[27571]: osdmap e122: 12 total, 12 up, 12 in 2024-09-17T16:11:35.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:35 smithi173 ceph-mon[30530]: pgmap v343: 27 pgs: 27 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 508 op/s 2024-09-17T16:11:35.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:35 smithi173 ceph-mon[30530]: osdmap e122: 12 total, 12 up, 12 in 2024-09-17T16:11:35.546 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:35 smithi071 ceph-mon[30002]: pgmap v343: 27 pgs: 27 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 508 op/s 2024-09-17T16:11:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:35 smithi071 ceph-mon[30002]: osdmap e122: 12 total, 12 up, 12 in 2024-09-17T16:11:35.572 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:35.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:35.674 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:35.699 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:35.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:35.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:35.801 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:35.824 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:35.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:35.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:35.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:35.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:35.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:11:36.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:36.080 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:36.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:36.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:36.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:36.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:36.211 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 27049 27033 49.6881 0 - 0.0180236 2024-09-17T16:11:36.236 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:36.309 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:36.333 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:36.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:36.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:36.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:36.457 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:36.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:36.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:36.580 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:36.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:36.682 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:36.704 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:36.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:36.753 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:36.753 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:36.806 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:36.827 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:36.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:36.930 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:36.954 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:36.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:36.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:37.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:37.081 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:37.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:37.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:37.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:11:37.212 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 27049 27033 48.2685 0 - 0.0180236 2024-09-17T16:11:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:37.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:37.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:37.279 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:11:37.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:37.350 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:37.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:37.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:37.457 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:11:37.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:37.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:37 smithi017 ceph-mon[27571]: pgmap v345: 27 pgs: 27 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 579 op/s 2024-09-17T16:11:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1925188672' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:37 smithi071 ceph-mon[30002]: pgmap v345: 27 pgs: 27 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 579 op/s 2024-09-17T16:11:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1925188672' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:37.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:37.605 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:37.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:37.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:37 smithi173 ceph-mon[30530]: pgmap v345: 27 pgs: 27 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 579 op/s 2024-09-17T16:11:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1925188672' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:37.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:37.708 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:37.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:37.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:37.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:37.854 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:37.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:37.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:37.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:37.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:38.032 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:38.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:38.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:38.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:38.211 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 27049 27033 46.9276 0 - 0.0180236 2024-09-17T16:11:38.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:38.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:38.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:38.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:38.285 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:38.285 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:38.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:38.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:38.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:38.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:38.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:38.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:38.555 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:11:38.556 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:11:38.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:38 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:11:38.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:38 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:11:38.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1925188672' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:38.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:38 smithi017 ceph-mon[27571]: osdmap e123: 12 total, 12 up, 12 in 2024-09-17T16:11:38.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:38 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:11:38.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:38 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:11:38.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1925188672' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:38.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:38 smithi071 ceph-mon[30002]: osdmap e123: 12 total, 12 up, 12 in 2024-09-17T16:11:38.576 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:11:38.577 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:11:38.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:38.597 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:11:38.598 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:11:38.617 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:11:38.618 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:11:38.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:38.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:38.639 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:11:38.639 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:11:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:38 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:11:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:38 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:11:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1925188672' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:38.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:38 smithi173 ceph-mon[30530]: osdmap e123: 12 total, 12 up, 12 in 2024-09-17T16:11:38.661 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:11:38.661 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:11:38.682 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:11:38.683 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:11:38.704 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:11:38.705 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:11:38.722 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:38.727 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:11:38.727 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:11:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:38.745 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:38.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:38.749 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:11:38.749 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:11:38.770 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:11:38.770 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:11:38.793 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:11:38.793 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:11:38.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:38.868 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:38.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:38.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:38.933 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:11:38.969 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:38.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:38.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:39.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:39.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:39.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:39.175 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:39.211 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 27049 27033 45.6593 0 - 0.0180236 2024-09-17T16:11:39.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:39.242 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:39.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:39.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:39.365 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:39.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:39.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:39 smithi017 ceph-mon[27571]: pgmap v347: 27 pgs: 27 active+clean; 1.7 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 434 op/s 2024-09-17T16:11:39.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1317769450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:39.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:39 smithi017 ceph-mon[27571]: osdmap e124: 12 total, 12 up, 12 in 2024-09-17T16:11:39.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:39.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:39.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:39.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:39 smithi071 ceph-mon[30002]: pgmap v347: 27 pgs: 27 active+clean; 1.7 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 434 op/s 2024-09-17T16:11:39.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1317769450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:39.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:39 smithi071 ceph-mon[30002]: osdmap e124: 12 total, 12 up, 12 in 2024-09-17T16:11:39.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:39.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:39.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:39 smithi173 ceph-mon[30530]: pgmap v347: 27 pgs: 27 active+clean; 1.7 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 434 op/s 2024-09-17T16:11:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1317769450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:39 smithi173 ceph-mon[30530]: osdmap e124: 12 total, 12 up, 12 in 2024-09-17T16:11:39.683 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:11:39.683 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:11:39.683 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:11:39.683 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:11:39.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.683 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:11670 2.d e102/95 2:b281e889:::benchmark_data_smithi173_55121_object11669:head v 102'728, mlcod=102'723)", 2024-09-17T16:11:39.683 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:10.813980+0000", 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.866584396, 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17845166100000001, 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.813980+0000", 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.684 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.813980+0000", 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.813981+0000", 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8100000000000003e-07 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.814040+0000", 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9367000000000003e-05 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.991682+0000", 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17764131599999999 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.991687+0000", 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9470000000000002e-06 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.991768+0000", 2024-09-17T16:11:39.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1669999999999996e-05 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.991791+0000", 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2493999999999998e-05 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.992398+0000", 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00060709800000000003 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.687 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.992432+0000", 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4288000000000001e-05 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:11696 2.5 e102/95 2:adf77b71:::benchmark_data_smithi173_55121_object11695:head v 102'708, mlcod=102'706)", 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:10.834186+0000", 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.846378856000001, 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16647804899999999, 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.688 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.834186+0000", 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.834186+0000", 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.834187+0000", 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5600000000000004e-07 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.834218+0000", 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0858999999999997e-05 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000102+0000", 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16588424700000001 2024-09-17T16:11:39.690 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000108+0000", 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8869999999999997e-06 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000149+0000", 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1377e-05 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.691 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000163+0000", 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3624e-05 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000631+0000", 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00046845300000000002 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000664+0000", 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2645999999999999e-05 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.692 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:12407 2.d e103/95 2:ba34e8ef:::benchmark_data_smithi173_55121_object12406:head v 103'782, mlcod=103'778)", 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:11.554977+0000", 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.125587800000002, 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.056554451999999998, 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.693 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.554977+0000", 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.554977+0000", 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.554978+0000", 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7599999999999997e-07 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.694 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.555042+0000", 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4605999999999998e-05 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.610768+0000", 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.055726181999999999 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.610781+0000", 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.234e-05 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.695 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.610921+0000", 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000140364 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.610957+0000", 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5830000000000001e-05 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.611501+0000", 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054424200000000001 2024-09-17T16:11:39.696 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.611532+0000", 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0312e-05 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:12415 2.f e103/95 2:f191ced9:::benchmark_data_smithi173_55121_object12414:head v 103'733, mlcod=103'731)", 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:11.562095+0000", 2024-09-17T16:11:39.697 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.118469637, 2024-09-17T16:11:39.698 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.055535470000000003, 2024-09-17T16:11:39.698 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.562095+0000", 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.562095+0000", 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.699 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.562096+0000", 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.145e-06 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.562142+0000", 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5565999999999997e-05 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.700 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.616980+0000", 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.054837992000000002 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.616985+0000", 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6090000000000003e-06 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.617032+0000", 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7848e-05 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.701 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.617052+0000", 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9745000000000002e-05 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.617600+0000", 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054780200000000001 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.617631+0000", 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0762999999999997e-05 2024-09-17T16:11:39.702 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:23607 2.f e119/95 2:fcca4bae:::benchmark_data_smithi173_55121_object23606:head v 119'1443, mlcod=119'1439)", 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.334241+0000", 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.346324268, 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.110738323, 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.703 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.334241+0000", 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.334241+0000", 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.334241+0000", 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0500000000000002e-07 2024-09-17T16:11:39.704 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.334302+0000", 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0234e-05 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444185+0000", 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.109883548 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444191+0000", 2024-09-17T16:11:39.705 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8499999999999999e-06 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444233+0000", 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1516e-05 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444246+0000", 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.378e-05 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.706 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444940+0000", 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00069336899999999999 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444979+0000", 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9221000000000003e-05 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:23622 2.5 e119/104 2:a0150b5d:::benchmark_data_smithi173_55121_object23621:head v 119'1165, mlcod=118'1163)", 2024-09-17T16:11:39.707 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.347194+0000", 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.333371076000001, 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10002479, 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.347194+0000", 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.708 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.347194+0000", 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.347194+0000", 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7800000000000001e-07 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.347277+0000", 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2132999999999994e-05 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.446585+0000", 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.099308289999999994 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.446589+0000", 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5079999999999999e-06 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.446633+0000", 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3229999999999998e-05 2024-09-17T16:11:39.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.446653+0000", 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0471000000000001e-05 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.447190+0000", 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00053740199999999998 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.447219+0000", 2024-09-17T16:11:39.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8178e-05 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:23628 2.18 e119/110 2:1945c29a:::benchmark_data_smithi173_55121_object23627:head v 119'1177, mlcod=119'1175)", 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.356382+0000", 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.324183186000001, 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.120117801, 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.712 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.356382+0000", 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.356382+0000", 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.356382+0000", 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1299999999999999e-07 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.356481+0000", 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8696000000000005e-05 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.475914+0000", 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.11943327300000001 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.714 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.475919+0000", 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2769999999999999e-06 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.475967+0000", 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8578999999999997e-05 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.475985+0000", 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8221000000000001e-05 2024-09-17T16:11:39.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.476461+0000", 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000475723 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.476500+0000", 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8318999999999997e-05 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.716 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:23688 2.5 e119/104 2:a36fc0da:::benchmark_data_smithi173_55121_object23687:head v 119'1170, mlcod=119'1168)", 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.495369+0000", 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.185195431, 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.054840761000000002, 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.495369+0000", 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.717 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.495369+0000", 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.495370+0000", 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7299999999999995e-07 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.718 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.495440+0000", 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9734999999999996e-05 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.549693+0000", 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.054252718999999998 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.549696+0000", 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8689999999999997e-06 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.549729+0000", 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2237999999999999e-05 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.549742+0000", 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3137e-05 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.550185+0000", 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00044293700000000002 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.550210+0000", 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5452999999999998e-05 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:23703 2.f e119/95 2:f239cad7:::benchmark_data_smithi173_55121_object23702:head v 119'1449, mlcod=119'1446)", 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.549630+0000", 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.130934892000001, 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.093026670000000006, 2024-09-17T16:11:39.721 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.549630+0000", 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.549630+0000", 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.549630+0000", 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9800000000000004e-07 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.549682+0000", 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1873999999999998e-05 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.641904+0000", 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.092221803000000005 2024-09-17T16:11:39.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.641911+0000", 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4910000000000002e-06 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.641955+0000", 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4366999999999997e-05 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.641969+0000", 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3589e-05 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.642623+0000", 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00065424400000000001 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.642657+0000", 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3803999999999997e-05 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:23707 2.f e119/95 2:ff591525:::benchmark_data_smithi173_55121_object23706:head v 119'1450, mlcod=119'1447)", 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.551262+0000", 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.129302901000001, 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.094429227000000004, 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.551262+0000", 2024-09-17T16:11:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.551262+0000", 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.551262+0000", 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4200000000000001e-07 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.551330+0000", 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7786000000000005e-05 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.645116+0000", 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.093785513000000001 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.645122+0000", 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2179999999999996e-06 2024-09-17T16:11:39.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.645161+0000", 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9360000000000003e-05 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.645174+0000", 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2534000000000001e-05 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.729 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.645660+0000", 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000486565 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.645691+0000", 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0809000000000003e-05 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:23786 2.f e119/95 2:f3145a88:::benchmark_data_smithi173_55121_object23785:head v 119'1456, mlcod=119'1452)", 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.694407+0000", 2024-09-17T16:11:39.730 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.986158228000001, 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.057794910999999997, 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.694407+0000", 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.694407+0000", 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.694407+0000", 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3499999999999995e-07 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.694461+0000", 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3924000000000001e-05 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.751272+0000", 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.056810225999999998 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.751276+0000", 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3590000000000001e-06 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.751341+0000", 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4904000000000001e-05 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.734 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.751361+0000", 2024-09-17T16:11:39.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0214e-05 2024-09-17T16:11:39.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.734 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.752167+0000", 2024-09-17T16:11:39.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00080581900000000004 2024-09-17T16:11:39.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.752202+0000", 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4730000000000001e-05 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:24683 2.11 e121/120 2:895e3609:::benchmark_data_smithi173_55121_object24682:head v 121'1219, mlcod=121'1217)", 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.353904+0000", 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.3266610360000008, 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12774153799999999, 2024-09-17T16:11:39.735 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.353904+0000", 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.353904+0000", 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.353904+0000", 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1699999999999998e-07 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.353926+0000", 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1628999999999999e-05 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481041+0000", 2024-09-17T16:11:39.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12711526000000001 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481047+0000", 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0689999999999998e-06 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481128+0000", 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0317999999999999e-05 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481145+0000", 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6840000000000001e-05 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481615+0000", 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00047066399999999998 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481645+0000", 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0141000000000002e-05 2024-09-17T16:11:39.739 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:24753 2.11 e121/120 2:8cf09c3e:::benchmark_data_smithi173_55121_object24752:head v 121'1222, mlcod=121'1219)", 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.502202+0000", 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.1783632990000008, 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.081531169000000001, 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.502202+0000", 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.502202+0000", 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.502202+0000", 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8599999999999997e-07 2024-09-17T16:11:39.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.502224+0000", 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1444e-05 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.583093+0000", 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.08086873 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.583099+0000", 2024-09-17T16:11:39.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5250000000000002e-06 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.583153+0000", 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3942e-05 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.583173+0000", 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0066000000000001e-05 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.583707+0000", 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00053356599999999997 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.583733+0000", 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.601e-05 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:24802 2.11 e121/120 2:88f591bc:::benchmark_data_smithi173_55121_object24801:head v 121'1224, mlcod=121'1221)", 2024-09-17T16:11:39.744 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.588941+0000", 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.0916236900000005, 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.051171803000000002, 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.588941+0000", 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.588941+0000", 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.588942+0000", 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1800000000000004e-07 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.588966+0000", 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.406e-05 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.639417+0000", 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.050450377999999997 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.639423+0000", 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1739999999999997e-06 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.639493+0000", 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0573000000000003e-05 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.639515+0000", 2024-09-17T16:11:39.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1971e-05 2024-09-17T16:11:39.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.640080+0000", 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00056493899999999998 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.640113+0000", 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2790000000000003e-05 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:25244 2.5 e121/104 2:a10ca3e5:::benchmark_data_smithi173_55121_object25243:head v 121'1224, mlcod=121'1221)", 2024-09-17T16:11:39.749 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.807427+0000", 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.8731380000000009, 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32591590799999998, 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.807427+0000", 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.807427+0000", 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.807428+0000", 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4900000000000005e-07 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.807473+0000", 2024-09-17T16:11:39.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5410000000000001e-05 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.132702+0000", 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32522888799999999 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.132706+0000", 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2490000000000003e-06 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.132779+0000", 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2461999999999998e-05 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.132793+0000", 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4729000000000001e-05 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.133317+0000", 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00052318600000000003 2024-09-17T16:11:39.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.133343+0000", 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6135e-05 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:25248 2.18 e121/110 2:1d4c9d7f:::benchmark_data_smithi173_55121_object25247:head v 121'1230, mlcod=121'1228)", 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.813696+0000", 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.8668687720000001, 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10322632900000001, 2024-09-17T16:11:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.813696+0000", 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.813696+0000", 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.813697+0000", 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8999999999999996e-07 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.813785+0000", 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8428000000000001e-05 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916258+0000", 2024-09-17T16:11:39.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10247242300000001 2024-09-17T16:11:39.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:39.758 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916279+0000", 2024-09-17T16:11:39.758 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1573000000000002e-05 2024-09-17T16:11:39.758 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.758 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916345+0000", 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5747000000000006e-05 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916361+0000", 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6337e-05 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916889+0000", 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00052784299999999995 2024-09-17T16:11:39.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916922+0000", 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3287999999999997e-05 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:25251 2.d e121/95 2:bcbcc18b:::benchmark_data_smithi173_55121_object25250:head v 121'1581, mlcod=121'1579)", 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.916220+0000", 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.7643446249999997, 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.134344825, 2024-09-17T16:11:39.760 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916220+0000", 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916220+0000", 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916222+0000", 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.263e-06 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916244+0000", 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1985999999999999e-05 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.049691+0000", 2024-09-17T16:11:39.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13344773700000001 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.049696+0000", 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6179999999999997e-06 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.049739+0000", 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3294999999999998e-05 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.049760+0000", 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0976999999999999e-05 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.050539+0000", 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00077899899999999995 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.050565+0000", 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5950000000000001e-05 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.764 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:25264 2.d e121/95 2:b6131135:::benchmark_data_smithi173_55121_object25263:head v 121'1582, mlcod=121'1580)", 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:31.061757+0000", 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.6188078990000001, 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16186235199999999, 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.765 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.061757+0000", 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.061757+0000", 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.061758+0000", 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6400000000000001e-07 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.061780+0000", 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2152000000000001e-05 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.222683+0000", 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16090269700000001 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.222687+0000", 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0389999999999998e-06 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.222756+0000", 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8868999999999994e-05 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.222772+0000", 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6725e-05 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.223587+0000", 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00081504699999999995 2024-09-17T16:11:39.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.223619+0000", 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2058999999999999e-05 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:25271 2.5 e121/104 2:a186deeb:::benchmark_data_smithi173_55121_object25270:head v 121'1225, mlcod=121'1222)", 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:31.135631+0000", 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.5449343320000004, 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.094167759000000004, 2024-09-17T16:11:39.769 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.135631+0000", 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.135631+0000", 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.770 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.135631+0000", 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7700000000000004e-07 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.135699+0000", 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7620000000000006e-05 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.229165+0000", 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.093466260999999995 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.229170+0000", 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.865e-06 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.229221+0000", 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1156000000000003e-05 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.229238+0000", 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6381000000000001e-05 2024-09-17T16:11:39.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.229771+0000", 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00053379199999999995 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.229798+0000", 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6906999999999998e-05 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:25280 2.f e121/95 2:fb0648a6:::benchmark_data_smithi173_55121_object25279:head v 121'1551, mlcod=121'1549)", 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:31.229117+0000", 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.4514481200000002, 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102913609, 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.229117+0000", 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.229117+0000", 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.229118+0000", 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0599999999999999e-07 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.229152+0000", 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4198e-05 2024-09-17T16:11:39.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.331101+0000", 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101949374 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.331113+0000", 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1853e-05 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.331234+0000", 2024-09-17T16:11:39.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120865 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.331249+0000", 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5162e-05 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.331997+0000", 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00074835799999999997 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.332030+0000", 2024-09-17T16:11:39.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2993000000000001e-05 2024-09-17T16:11:39.778 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.778 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.778 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.778 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:39.778 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:39.778 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:11:39.779 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:39.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:39.881 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:39.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:39.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:39.986 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:11:40.004 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:40.052 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:40.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:40.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:40.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:40.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:40.211 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 27049 27033 44.4576 0 - 0.0180236 2024-09-17T16:11:40.244 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:40.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:40.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:40.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:40.400 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:40.422 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:40.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:40.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:40.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:11:40.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:11:40.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:11:40.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:11:40.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:40.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:40.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:11:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:11:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:11:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:11:40.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:11:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:11:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:11:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:11:40.648 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:40.663 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:11:40.672 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:40.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:40.774 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:40.795 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:40.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:40.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:40.915 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:40.920 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:40.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:41.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:41.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:41.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:41.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:41.174 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:41.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:41.212 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 27049 27033 43.3176 0 - 0.0180236 2024-09-17T16:11:41.276 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:41.298 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:11:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:41.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:41.363 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:11:41.363 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:11:41.363 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:11:41.363 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:11:41.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:41.422 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:41.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:41.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:11:41.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:41.545 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:41.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:41.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:41 smithi071 ceph-mon[30002]: pgmap v349: 27 pgs: 27 active+clean; 1.7 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 223 op/s 2024-09-17T16:11:41.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3163280591' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:11:41.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:41 smithi173 ceph-mon[30530]: pgmap v349: 27 pgs: 27 active+clean; 1.7 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 223 op/s 2024-09-17T16:11:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3163280591' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:11:41.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:41.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:41.669 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:41.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:41.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:41 smithi017 ceph-mon[27571]: pgmap v349: 27 pgs: 27 active+clean; 1.7 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 223 op/s 2024-09-17T16:11:41.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3163280591' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:11:41.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:41.793 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:41.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:41.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:41.916 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:41.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:42.017 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:42.024 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:42.038 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:42.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:42.047 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-17T16:11:42.047 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:11:42.048 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T16:11:42.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:11:42.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:11:42.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:42.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:42.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:42.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:42.212 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:11:42.211128+0000 min lat: 0.00105621 max lat: 2.09646 avg lat: 0.0180236 2024-09-17T16:11:42.212 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:11:42.212 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 27049 27033 42.2347 0 - 0.0180236 2024-09-17T16:11:42.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:42.288 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:42.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:42.379 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:42.390 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:42.414 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:42.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:42.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:42.537 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:42.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:42.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:42.547 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:11:42.547 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:11:42.548 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:42 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:42.548 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3163280591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:11:42.548 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:42 smithi017 ceph-mon[27571]: osdmap e125: 12 total, 12 up, 12 in 2024-09-17T16:11:42.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:42 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:42.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3163280591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:11:42.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:42 smithi071 ceph-mon[30002]: osdmap e125: 12 total, 12 up, 12 in 2024-09-17T16:11:42.570 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:11:42.570 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:11:42.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:11:42.591 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:11:42.612 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:11:42.613 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:11:42.634 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:11:42.635 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:11:42.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:42.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:42 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3163280591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:11:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:42 smithi173 ceph-mon[30530]: osdmap e125: 12 total, 12 up, 12 in 2024-09-17T16:11:42.658 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:11:42.658 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:11:42.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:42.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:42.678 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:11:42.678 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:11:42.699 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:11:42.700 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:11:42.724 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:11:42.724 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:11:42.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:11:42.744 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:11:42.766 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:11:42.766 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:11:42.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:42.787 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:11:42.787 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:11:42.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:42.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:42.862 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:42.862 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:42.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:42.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:42.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:42.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:43.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:43.046 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:43.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:43.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:43.169 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:43.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:43.212 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 27049 27033 41.2045 0 - 0.0180236 2024-09-17T16:11:43.269 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:43.292 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:43.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:43.387 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:43 smithi017 ceph-mon[27571]: pgmap v351: 27 pgs: 27 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s wr, 107 op/s 2024-09-17T16:11:43.387 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:43 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:11:43.387 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:43 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:11:43.387 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:43 smithi017 ceph-mon[27571]: osdmap e126: 12 total, 12 up, 12 in 2024-09-17T16:11:43.388 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/781283666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:43.388 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:11:43.388 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:11:43.388 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 126, 2024-09-17T16:11:43.388 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:11:43.388 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:11:43.388 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:11:42.297369+0000", 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:09:49.234769+0000", 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:09:29.753083+0000", 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5801984, 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "noscrub", 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 26, 2024-09-17T16:11:43.389 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 2, 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "jewel", 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.390 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:11:43.391 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:11:43.391 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:11:43.391 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:11:43.391 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:11:43.391 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:11:43.391 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:11:43.391 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:11:43.391 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:11:43.391 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:11:43.392 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "21", 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:11:43.393 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:11:43.394 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:11:43.395 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.396 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-17T16:11:43.397 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-17T16:11:43.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.398 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 2, 2024-09-17T16:11:43.398 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_0", 2024-09-17T16:11:43.398 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:10:52.564134+0000", 2024-09-17T16:11:43.398 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:11:43.398 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:11:43.398 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 26, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 26, 2024-09-17T16:11:43.399 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 26, 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 26, 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 26, 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "106", 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:11:43.400 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "104", 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:11:43.401 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:11:43.402 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:11:43.403 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 1.7000000476837158, 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 1.7000000476837158, 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1.0900000333786011, 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 1.8500000238418579, 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 1.8500000238418579, 2024-09-17T16:11:43.404 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.405 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 119, 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:11:43.406 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:11:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:11:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:11:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.407 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:11:43.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.411 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 119, 2024-09-17T16:11:43.413 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.414 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:11:43.415 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.416 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.417 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.418 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 111, 2024-09-17T16:11:43.419 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:11:43.420 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.421 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:11:43.423 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:11:43.424 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.425 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 114, 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.426 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.427 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:11:43.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.430 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.431 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 0, 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.432 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:11:43.434 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:11:43.435 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:11:43.436 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.437 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 115, 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.438 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.439 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:11:43.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:11:43.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.442 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.443 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 120, 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.444 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:11:43.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:11:43.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:11:43.447 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:11:43.448 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.449 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 54, 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 110, 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1816911174 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1816911174 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1816911174 2024-09-17T16:11:43.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1816911174 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.452 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1816911174 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1816911174 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.453 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1816911174 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1816911174 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.454 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/1816911174", 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/1816911174", 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/1816911174", 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/1816911174", 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.455 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 111, 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:11:43.456 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.459 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.461 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 113, 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.462 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.464 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.465 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.467 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:11:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:11:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.468 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.7501220703125, 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 114, 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.469 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.471 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.472 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:11:43.473 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.474 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 95, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:11:43.475 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.476 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:11:43.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:11:43.479 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:11:43.480 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:11:13.919602+0000", 2024-09-17T16:11:43.481 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:11:41.856401+0000", 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:10:51.481428+0000", 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.483 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:10:48.832510+0000", 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.484 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:10:44.307629+0000", 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.485 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:11:18.306885+0000", 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:10:31.118235+0000", 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:10:49.050300+0000", 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.487 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:11:09.238282+0000", 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:10:24.429208+0000", 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:10:38.157851+0000", 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:11:43.489 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:11:05.175632+0000", 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [], 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-17T16:11:43.490 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:11:43.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:11:43.492 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:11:43.493 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:11:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:43.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:43.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:43.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:43 smithi071 ceph-mon[30002]: pgmap v351: 27 pgs: 27 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s wr, 107 op/s 2024-09-17T16:11:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:43 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:11:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:43 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:11:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:43 smithi071 ceph-mon[30002]: osdmap e126: 12 total, 12 up, 12 in 2024-09-17T16:11:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/781283666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:43.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:43.619 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:43.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:43.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:43.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:11:43.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:43 smithi173 ceph-mon[30530]: pgmap v351: 27 pgs: 27 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s wr, 107 op/s 2024-09-17T16:11:43.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:43 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:11:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:43 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:11:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:43 smithi173 ceph-mon[30530]: osdmap e126: 12 total, 12 up, 12 in 2024-09-17T16:11:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/781283666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:43.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:43.743 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:43.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:43.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:43.867 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:43.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:43.905 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:43.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:43.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:11:43.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:43.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:44.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:44.120 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:44.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:44.212 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 27049 27033 40.2234 0 - 0.0180236 2024-09-17T16:11:44.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:44.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:44.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:44.263 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:44.348 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:44.363 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:44.364 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:44.370 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:44.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:44.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:44.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:44.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:44.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:44.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:44 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:11:44.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:44 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:11:44.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2401438139' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:11:44.598 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:44.622 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:44.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:44.640 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:44 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:11:44.640 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:44 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:11:44.640 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2401438139' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:11:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:44 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:11:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:44 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:11:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2401438139' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:11:44.668 INFO:tasks.thrashosds.thrasher:j is {'epoch': 126, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:11:42.297369+0000', 'last_up_change': '2024-09-17T16:09:49.234769+0000', 'last_in_change': '2024-09-17T16:09:29.753083+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-17T16:10:52.564134+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '104', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7000000476837158, 'score_stable': 1.7000000476837158, 'optimal_score': 1.090000033378601, 'raw_score_acting': 1.850000023841858, 'raw_score_stable': 1.850000023841858, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 119, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 119, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 114, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 1816911174}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 1816911174}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 1816911174}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 1816911174}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 1816911174}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 1816911174}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 1816911174}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 1816911174}]}, 'public_addr': '172.21.15.71:6825/1816911174', 'cluster_addr': '172.21.15.71:6827/1816911174', 'heartbeat_back_addr': '172.21.15.71:6831/1816911174', 'heartbeat_front_addr': '172.21.15.71:6829/1816911174', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 113, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 0.7501220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 114, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:11:13.919602+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:11:41.856401+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:10:51.481428+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:10:48.832510+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:10:44.307629+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:11:18.306885+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:10:31.118235+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:10:49.050300+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:11:09.238282+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:10:24.429208+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:10:38.157851+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:11:05.175632+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:11:44.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:11:44.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:44.746 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:44.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:44.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:44.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:44.869 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:44.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:44.969 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:11:44.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:44.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:44.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:45.100 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:45.122 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:45.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:45.204 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:45.210 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:45.212 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 27049 27033 39.2879 0 - 0.0180236 2024-09-17T16:11:45.224 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:45.247 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:45.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:45.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:45.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:45.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:45.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:45.477 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:45.536 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:45.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:45.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:45 smithi071 ceph-mon[30002]: pgmap v353: 27 pgs: 27 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:45 smithi071 ceph-mon[30002]: osdmap e127: 12 total, 12 up, 12 in 2024-09-17T16:11:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:45 smithi071 ceph-mon[30002]: 2.8 scrub starts 2024-09-17T16:11:45.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:45 smithi017 ceph-mon[27571]: pgmap v353: 27 pgs: 27 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:45.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:45 smithi017 ceph-mon[27571]: osdmap e127: 12 total, 12 up, 12 in 2024-09-17T16:11:45.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:45 smithi017 ceph-mon[27571]: 2.8 scrub starts 2024-09-17T16:11:45.638 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:45.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:45 smithi173 ceph-mon[30530]: pgmap v353: 27 pgs: 27 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:45.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:45 smithi173 ceph-mon[30530]: osdmap e127: 12 total, 12 up, 12 in 2024-09-17T16:11:45.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:45 smithi173 ceph-mon[30530]: 2.8 scrub starts 2024-09-17T16:11:45.659 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:45.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:45.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:45.681 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:45.681 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:45.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:45.783 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:45.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:45.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:11:45.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:45.886 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:11:45.887 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:11:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:45.909 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:45.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:46.011 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:46.031 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:46.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:46.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:46.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:46.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:46.212 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 27049 27033 38.395 0 - 0.0180236 2024-09-17T16:11:46.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:46.284 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:46.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:46.318 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:11:46.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:46.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:46.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:46.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:46.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:46 smithi017 ceph-mon[27571]: 2.5 scrub starts 2024-09-17T16:11:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:46 smithi017 ceph-mon[27571]: 2.1 scrub starts 2024-09-17T16:11:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:46 smithi017 ceph-mon[27571]: 2.9 scrub starts 2024-09-17T16:11:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4270976459' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:46 smithi017 ceph-mon[27571]: 2.0 scrub starts 2024-09-17T16:11:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:46 smithi017 ceph-mon[27571]: 2.2 scrub starts 2024-09-17T16:11:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:46 smithi017 ceph-mon[27571]: 2.17 scrub starts 2024-09-17T16:11:46.513 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:46.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:46.537 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:46.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:46.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:46 smithi071 ceph-mon[30002]: 2.5 scrub starts 2024-09-17T16:11:46.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:46 smithi071 ceph-mon[30002]: 2.1 scrub starts 2024-09-17T16:11:46.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:46 smithi071 ceph-mon[30002]: 2.9 scrub starts 2024-09-17T16:11:46.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4270976459' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:46.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:46 smithi071 ceph-mon[30002]: 2.0 scrub starts 2024-09-17T16:11:46.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:46 smithi071 ceph-mon[30002]: 2.2 scrub starts 2024-09-17T16:11:46.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:46 smithi071 ceph-mon[30002]: 2.17 scrub starts 2024-09-17T16:11:46.570 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-09-17T16:11:45.731851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1774420512,"num_objects":27068,"num_object_clones":0,"num_object_copies":81204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27068,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":25596,"num_write_kb":1632454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3744,"num_bytes_recovered":246560352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35699,"ondisk_log_size":35699,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":90,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5696316,"kb_used_data":5283196,"kb_used_omap":18,"kb_used_meta":412845,"kb_avail":1119245508,"statfs":{"total":1151940427776,"available":1146107400192,"internally_reserved":0,"allocated":5409992704,"data_stored":5404113362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":422753594},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,21],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001043"},"pg_stats":[{"pgid":"2.19","version":"121'1284","reported_seq":2770,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T16:11:39.830118+0000","last_change":"2024-09-17T16:11:25.259980+0000","last_active":"2024-09-17T16:11:39.830118+0000","last_peered":"2024-09-17T16:11:39.830118+0000","last_clean":"2024-09-17T16:11:39.830118+0000","last_became_active":"2024-09-17T16:11:24.264470+0000","last_became_peered":"2024-09-17T16:11:24.264470+0000","last_unstale":"2024-09-17T16:11:39.830118+0000","last_undegraded":"2024-09-17T16:11:39.830118+0000","last_fullsized":"2024-09-17T16:11:39.830118+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:55.154506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55672832,"num_objects":850,"num_object_clones":0,"num_object_copies":2550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":850,"num_write_kb":54368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":30474240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"121'1293","reported_seq":2361,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T16:11:39.830311+0000","last_change":"2024-09-17T16:11:20.907193+0000","last_active":"2024-09-17T16:11:39.830311+0000","last_peered":"2024-09-17T16:11:39.830311+0000","last_clean":"2024-09-17T16:11:39.830311+0000","last_became_active":"2024-09-17T16:11:20.266775+0000","last_became_peered":"2024-09-17T16:11:20.266775+0000","last_unstale":"2024-09-17T16:11:39.830311+0000","last_undegraded":"2024-09-17T16:11:39.830311+0000","last_fullsized":"2024-09-17T16:11:39.830311+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1293,"log_dups_size":0,"ondisk_log_size":1293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:50.888284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56590336,"num_objects":864,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":864,"num_write_kb":55264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":488,"num_bytes_recovered":31981568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"121'1319","reported_seq":1406,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-17T16:11:44.311950+0000","last_change":"2024-09-17T16:11:14.833681+0000","last_active":"2024-09-17T16:11:44.311950+0000","last_peered":"2024-09-17T16:11:44.311950+0000","last_clean":"2024-09-17T16:11:44.311950+0000","last_became_active":"2024-09-17T16:11:13.358625+0000","last_became_peered":"2024-09-17T16:11:13.358625+0000","last_unstale":"2024-09-17T16:11:44.311950+0000","last_undegraded":"2024-09-17T16:11:44.311950+0000","last_fullsized":"2024-09-17T16:11:44.311950+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1319,"log_dups_size":0,"ondisk_log_size":1319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:44.602710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58032128,"num_objects":886,"num_object_clones":0,"num_object_copies":2658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":886,"num_write_kb":56672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"121'1230","reported_seq":3156,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-17T16:11:44.430147+0000","last_change":"2024-09-17T16:11:25.314564+0000","last_active":"2024-09-17T16:11:44.430147+0000","last_peered":"2024-09-17T16:11:44.430147+0000","last_clean":"2024-09-17T16:11:44.430147+0000","last_became_active":"2024-09-17T16:11:25.266055+0000","last_became_peered":"2024-09-17T16:11:25.266055+0000","last_unstale":"2024-09-17T16:11:44.430147+0000","last_undegraded":"2024-09-17T16:11:44.430147+0000","last_fullsized":"2024-09-17T16:11:44.430147+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1230,"log_dups_size":0,"ondisk_log_size":1230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:55.285237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54001664,"num_objects":824,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":824,"num_write_kb":52736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":29949952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"121'1232","reported_seq":3179,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T16:11:39.830257+0000","last_change":"2024-09-17T16:11:24.268096+0000","last_active":"2024-09-17T16:11:39.830257+0000","last_peered":"2024-09-17T16:11:39.830257+0000","last_clean":"2024-09-17T16:11:39.830257+0000","last_became_active":"2024-09-17T16:11:23.302908+0000","last_became_peered":"2024-09-17T16:11:23.302908+0000","last_unstale":"2024-09-17T16:11:39.830257+0000","last_undegraded":"2024-09-17T16:11:39.830257+0000","last_fullsized":"2024-09-17T16:11:39.830257+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1232,"log_dups_size":0,"ondisk_log_size":1232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:53.903655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53346304,"num_objects":814,"num_object_clones":0,"num_object_copies":2442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":814,"num_write_kb":52096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":32178176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"121'1324","reported_seq":3424,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-17T16:11:44.311878+0000","last_change":"2024-09-17T16:11:27.594530+0000","last_active":"2024-09-17T16:11:44.311878+0000","last_peered":"2024-09-17T16:11:44.311878+0000","last_clean":"2024-09-17T16:11:44.311878+0000","last_became_active":"2024-09-17T16:11:27.290863+0000","last_became_peered":"2024-09-17T16:11:27.290863+0000","last_unstale":"2024-09-17T16:11:44.311878+0000","last_undegraded":"2024-09-17T16:11:44.311878+0000","last_fullsized":"2024-09-17T16:11:44.311878+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1324,"log_dups_size":0,"ondisk_log_size":1324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:57.591735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57475072,"num_objects":877,"num_object_clones":0,"num_object_copies":2631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":877,"num_write_kb":56128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":32768000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"121'1224","reported_seq":3080,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T16:11:39.830425+0000","last_change":"2024-09-17T16:11:23.269341+0000","last_active":"2024-09-17T16:11:39.830425+0000","last_peered":"2024-09-17T16:11:39.830425+0000","last_clean":"2024-09-17T16:11:39.830425+0000","last_became_active":"2024-09-17T16:11:22.310367+0000","last_became_peered":"2024-09-17T16:11:22.310367+0000","last_unstale":"2024-09-17T16:11:39.830425+0000","last_undegraded":"2024-09-17T16:11:39.830425+0000","last_fullsized":"2024-09-17T16:11:39.830425+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:52.891681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52625408,"num_objects":803,"num_object_clones":0,"num_object_copies":2409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":803,"num_write_kb":51392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":29163520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"121'1306","reported_seq":3246,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-17T16:11:44.653297+0000","last_change":"2024-09-17T16:11:24.322624+0000","last_active":"2024-09-17T16:11:44.653297+0000","last_peered":"2024-09-17T16:11:44.653297+0000","last_clean":"2024-09-17T16:11:44.653297+0000","last_became_active":"2024-09-17T16:11:24.263795+0000","last_became_peered":"2024-09-17T16:11:24.263795+0000","last_unstale":"2024-09-17T16:11:44.653297+0000","last_undegraded":"2024-09-17T16:11:44.653297+0000","last_fullsized":"2024-09-17T16:11:44.653297+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1306,"log_dups_size":0,"ondisk_log_size":1306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:54.315071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57049088,"num_objects":871,"num_object_clones":0,"num_object_copies":2613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":871,"num_write_kb":55712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":460,"num_bytes_recovered":30146560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,7],"acting":[10,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"121'1281","reported_seq":3415,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-17T16:11:44.311690+0000","last_change":"2024-09-17T16:11:30.818220+0000","last_active":"2024-09-17T16:11:44.311690+0000","last_peered":"2024-09-17T16:11:44.311690+0000","last_clean":"2024-09-17T16:11:44.311690+0000","last_became_active":"2024-09-17T16:11:30.264463+0000","last_became_peered":"2024-09-17T16:11:30.264463+0000","last_unstale":"2024-09-17T16:11:44.311690+0000","last_undegraded":"2024-09-17T16:11:44.311690+0000","last_fullsized":"2024-09-17T16:11:44.311690+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":5,"last_scrub":"121'1234","last_scrub_stamp":"2024-09-17T16:11:30.818169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:30.818169+0000","objects_scrubbed":816,"log_size":1281,"log_dups_size":0,"ondisk_log_size":1281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:44.239265+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56623104,"num_objects":864,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"121'1224","reported_seq":2614,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-17T16:11:42.306279+0000","last_change":"2024-09-17T16:11:22.270299+0000","last_active":"2024-09-17T16:11:42.306279+0000","last_peered":"2024-09-17T16:11:42.306279+0000","last_clean":"2024-09-17T16:11:42.306279+0000","last_became_active":"2024-09-17T16:11:21.266149+0000","last_became_peered":"2024-09-17T16:11:21.266149+0000","last_unstale":"2024-09-17T16:11:42.306279+0000","last_undegraded":"2024-09-17T16:11:42.306279+0000","last_fullsized":"2024-09-17T16:11:42.306279+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:51.555777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53313536,"num_objects":814,"num_object_clones":0,"num_object_copies":2442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":814,"num_write_kb":52064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":28246016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"121'1663","reported_seq":1739,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-17T16:11:42.306290+0000","last_change":"2024-09-17T16:10:54.283333+0000","last_active":"2024-09-17T16:11:42.306290+0000","last_peered":"2024-09-17T16:11:42.306290+0000","last_clean":"2024-09-17T16:11:42.306290+0000","last_became_active":"2024-09-17T16:10:54.282931+0000","last_became_peered":"2024-09-17T16:10:54.282931+0000","last_unstale":"2024-09-17T16:11:42.306290+0000","last_undegraded":"2024-09-17T16:11:42.306290+0000","last_fullsized":"2024-09-17T16:11:42.306290+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:20.701254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108986368,"num_objects":1663,"num_object_clones":0,"num_object_copies":4989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1663,"num_write_kb":106432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"121'1612","reported_seq":1684,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T16:11:39.829928+0000","last_change":"2024-09-17T16:10:54.283952+0000","last_active":"2024-09-17T16:11:39.829928+0000","last_peered":"2024-09-17T16:11:39.829928+0000","last_clean":"2024-09-17T16:11:39.829928+0000","last_became_active":"2024-09-17T16:10:54.283208+0000","last_became_peered":"2024-09-17T16:10:54.283208+0000","last_unstale":"2024-09-17T16:11:39.829928+0000","last_undegraded":"2024-09-17T16:11:39.829928+0000","last_fullsized":"2024-09-17T16:11:39.829928+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1612,"log_dups_size":0,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:56.650770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105644032,"num_objects":1612,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1612,"num_write_kb":103168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"121'1687","reported_seq":1759,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T16:11:39.830377+0000","last_change":"2024-09-17T16:10:54.283371+0000","last_active":"2024-09-17T16:11:39.830377+0000","last_peered":"2024-09-17T16:11:39.830377+0000","last_clean":"2024-09-17T16:11:39.830377+0000","last_became_active":"2024-09-17T16:10:54.283016+0000","last_became_peered":"2024-09-17T16:10:54.283016+0000","last_unstale":"2024-09-17T16:11:39.830377+0000","last_undegraded":"2024-09-17T16:11:39.830377+0000","last_fullsized":"2024-09-17T16:11:39.830377+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1687,"log_dups_size":0,"ondisk_log_size":1687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:15.690355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110559232,"num_objects":1687,"num_object_clones":0,"num_object_copies":5061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1687,"num_write_kb":107968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"121'1703","reported_seq":1778,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-17T16:11:42.376039+0000","last_change":"2024-09-17T16:10:54.282319+0000","last_active":"2024-09-17T16:11:42.376039+0000","last_peered":"2024-09-17T16:11:42.376039+0000","last_clean":"2024-09-17T16:11:42.376039+0000","last_became_active":"2024-09-17T16:10:54.282069+0000","last_became_peered":"2024-09-17T16:10:54.282069+0000","last_unstale":"2024-09-17T16:11:42.376039+0000","last_undegraded":"2024-09-17T16:11:42.376039+0000","last_fullsized":"2024-09-17T16:11:42.376039+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1703,"log_dups_size":0,"ondisk_log_size":1703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:58.910840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111607808,"num_objects":1703,"num_object_clones":0,"num_object_copies":5109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1703,"num_write_kb":108992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":200,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T16:11:39.304474+0000","last_change":"2024-09-17T16:11:04.478292+0000","last_active":"2024-09-17T16:11:39.304474+0000","last_peered":"2024-09-17T16:11:39.304474+0000","last_clean":"2024-09-17T16:11:39.304474+0000","last_became_active":"2024-09-17T16:09:51.301092+0000","last_became_peered":"2024-09-17T16:09:51.301092+0000","last_unstale":"2024-09-17T16:11:39.304474+0000","last_undegraded":"2024-09-17T16:11:39.304474+0000","last_fullsized":"2024-09-17T16:11:39.304474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:11:04.478242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:11:04.478242+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:12.203778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"121'1307","reported_seq":1397,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-17T16:11:44.430123+0000","last_change":"2024-09-17T16:11:16.348055+0000","last_active":"2024-09-17T16:11:44.430123+0000","last_peered":"2024-09-17T16:11:44.430123+0000","last_clean":"2024-09-17T16:11:44.430123+0000","last_became_active":"2024-09-17T16:11:13.351214+0000","last_became_peered":"2024-09-17T16:11:13.351214+0000","last_unstale":"2024-09-17T16:11:44.430123+0000","last_undegraded":"2024-09-17T16:11:44.430123+0000","last_fullsized":"2024-09-17T16:11:44.430123+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:46.342447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58064896,"num_objects":886,"num_object_clones":0,"num_object_copies":2658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":886,"num_write_kb":56704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"121'1244","reported_seq":1326,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T16:11:39.304405+0000","last_change":"2024-09-17T16:11:16.281105+0000","last_active":"2024-09-17T16:11:39.304405+0000","last_peered":"2024-09-17T16:11:39.304405+0000","last_clean":"2024-09-17T16:11:39.304405+0000","last_became_active":"2024-09-17T16:11:13.350104+0000","last_became_peered":"2024-09-17T16:11:13.350104+0000","last_unstale":"2024-09-17T16:11:39.304405+0000","last_undegraded":"2024-09-17T16:11:39.304405+0000","last_fullsized":"2024-09-17T16:11:39.304405+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1244,"log_dups_size":0,"ondisk_log_size":1244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:45.358144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54624256,"num_objects":833,"num_object_clones":0,"num_object_copies":2499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":833,"num_write_kb":53344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"121'1297","reported_seq":1381,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T16:11:39.829805+0000","last_change":"2024-09-17T16:11:15.262090+0000","last_active":"2024-09-17T16:11:39.829805+0000","last_peered":"2024-09-17T16:11:39.829805+0000","last_clean":"2024-09-17T16:11:39.829805+0000","last_became_active":"2024-09-17T16:11:13.356367+0000","last_became_peered":"2024-09-17T16:11:13.356367+0000","last_unstale":"2024-09-17T16:11:39.829805+0000","last_undegraded":"2024-09-17T16:11:39.829805+0000","last_fullsized":"2024-09-17T16:11:39.829805+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1297,"log_dups_size":0,"ondisk_log_size":1297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:44.800853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55934976,"num_objects":853,"num_object_clones":0,"num_object_copies":2559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":853,"num_write_kb":54624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"121'1274","reported_seq":1362,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-17T16:11:44.430132+0000","last_change":"2024-09-17T16:11:16.286400+0000","last_active":"2024-09-17T16:11:44.430132+0000","last_peered":"2024-09-17T16:11:44.430132+0000","last_clean":"2024-09-17T16:11:44.430132+0000","last_became_active":"2024-09-17T16:11:13.352761+0000","last_became_peered":"2024-09-17T16:11:13.352761+0000","last_unstale":"2024-09-17T16:11:44.430132+0000","last_undegraded":"2024-09-17T16:11:44.430132+0000","last_fullsized":"2024-09-17T16:11:44.430132+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:45.304144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54951936,"num_objects":838,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":838,"num_write_kb":53664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"121'1318","reported_seq":2688,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T16:11:41.351300+0000","last_change":"2024-09-17T16:11:30.184652+0000","last_active":"2024-09-17T16:11:41.351300+0000","last_peered":"2024-09-17T16:11:41.351300+0000","last_clean":"2024-09-17T16:11:41.351300+0000","last_became_active":"2024-09-17T16:11:29.267304+0000","last_became_peered":"2024-09-17T16:11:29.267304+0000","last_unstale":"2024-09-17T16:11:41.351300+0000","last_undegraded":"2024-09-17T16:11:41.351300+0000","last_fullsized":"2024-09-17T16:11:41.351300+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'1233","last_scrub_stamp":"2024-09-17T16:11:30.184562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:30.184562+0000","objects_scrubbed":777,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:44.063339+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56492032,"num_objects":862,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":5440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"121'1290","reported_seq":1378,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-17T16:11:42.306270+0000","last_change":"2024-09-17T16:11:15.258069+0000","last_active":"2024-09-17T16:11:42.306270+0000","last_peered":"2024-09-17T16:11:42.306270+0000","last_clean":"2024-09-17T16:11:42.306270+0000","last_became_active":"2024-09-17T16:11:13.348959+0000","last_became_peered":"2024-09-17T16:11:13.348959+0000","last_unstale":"2024-09-17T16:11:42.306270+0000","last_undegraded":"2024-09-17T16:11:42.306270+0000","last_fullsized":"2024-09-17T16:11:42.306270+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:44.587493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57147392,"num_objects":872,"num_object_clones":0,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":872,"num_write_kb":55808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"121'1233","reported_seq":1316,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T16:11:41.351228+0000","last_change":"2024-09-17T16:11:17.090352+0000","last_active":"2024-09-17T16:11:41.351228+0000","last_peered":"2024-09-17T16:11:41.351228+0000","last_clean":"2024-09-17T16:11:41.351228+0000","last_became_active":"2024-09-17T16:11:13.354186+0000","last_became_peered":"2024-09-17T16:11:13.354186+0000","last_unstale":"2024-09-17T16:11:41.351228+0000","last_undegraded":"2024-09-17T16:11:41.351228+0000","last_fullsized":"2024-09-17T16:11:41.351228+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1233,"log_dups_size":0,"ondisk_log_size":1233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:47.075004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54198272,"num_objects":827,"num_object_clones":0,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":827,"num_write_kb":52928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"121'1297","reported_seq":1387,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-17T16:11:44.312004+0000","last_change":"2024-09-17T16:11:16.310667+0000","last_active":"2024-09-17T16:11:44.312004+0000","last_peered":"2024-09-17T16:11:44.312004+0000","last_clean":"2024-09-17T16:11:44.312004+0000","last_became_active":"2024-09-17T16:11:13.352397+0000","last_became_peered":"2024-09-17T16:11:13.352397+0000","last_unstale":"2024-09-17T16:11:44.312004+0000","last_undegraded":"2024-09-17T16:11:44.312004+0000","last_fullsized":"2024-09-17T16:11:44.312004+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1297,"log_dups_size":0,"ondisk_log_size":1297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:45.568297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56590336,"num_objects":863,"num_object_clones":0,"num_object_copies":2589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":863,"num_write_kb":55264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"121'1345","reported_seq":1429,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T16:11:39.304401+0000","last_change":"2024-09-17T16:11:15.263053+0000","last_active":"2024-09-17T16:11:39.304401+0000","last_peered":"2024-09-17T16:11:39.304401+0000","last_clean":"2024-09-17T16:11:39.304401+0000","last_became_active":"2024-09-17T16:11:13.350241+0000","last_became_peered":"2024-09-17T16:11:13.350241+0000","last_unstale":"2024-09-17T16:11:39.304401+0000","last_undegraded":"2024-09-17T16:11:39.304401+0000","last_fullsized":"2024-09-17T16:11:39.304401+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1345,"log_dups_size":0,"ondisk_log_size":1345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:44.361555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59998208,"num_objects":915,"num_object_clones":0,"num_object_copies":2745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":915,"num_write_kb":58592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"121'1291","reported_seq":1378,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T16:11:41.351218+0000","last_change":"2024-09-17T16:11:15.257775+0000","last_active":"2024-09-17T16:11:41.351218+0000","last_peered":"2024-09-17T16:11:41.351218+0000","last_clean":"2024-09-17T16:11:41.351218+0000","last_became_active":"2024-09-17T16:11:13.358640+0000","last_became_peered":"2024-09-17T16:11:13.358640+0000","last_unstale":"2024-09-17T16:11:41.351218+0000","last_undegraded":"2024-09-17T16:11:41.351218+0000","last_fullsized":"2024-09-17T16:11:41.351218+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:11:45.039939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56131584,"num_objects":856,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":856,"num_write_kb":54816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"121'1658","reported_seq":1733,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-17T16:11:42.376032+0000","last_change":"2024-09-17T16:10:54.283470+0000","last_active":"2024-09-17T16:11:42.376032+0000","last_peered":"2024-09-17T16:11:42.376032+0000","last_clean":"2024-09-17T16:11:42.376032+0000","last_became_active":"2024-09-17T16:10:54.283026+0000","last_became_peered":"2024-09-17T16:10:54.283026+0000","last_unstale":"2024-09-17T16:11:42.376032+0000","last_undegraded":"2024-09-17T16:11:42.376032+0000","last_fullsized":"2024-09-17T16:11:42.376032+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1658,"log_dups_size":0,"ondisk_log_size":1658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:11:58.325578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108658688,"num_objects":1658,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1658,"num_write_kb":106112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"121'1671","reported_seq":1731,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-17T16:11:42.306349+0000","last_change":"2024-09-17T16:10:54.286746+0000","last_active":"2024-09-17T16:11:42.306349+0000","last_peered":"2024-09-17T16:11:42.306349+0000","last_clean":"2024-09-17T16:11:42.306349+0000","last_became_active":"2024-09-17T16:10:54.286405+0000","last_became_peered":"2024-09-17T16:10:54.286405+0000","last_unstale":"2024-09-17T16:11:42.306349+0000","last_undegraded":"2024-09-17T16:11:42.306349+0000","last_fullsized":"2024-09-17T16:11:42.306349+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:21.645423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109510656,"num_objects":1671,"num_object_clones":0,"num_object_copies":5013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1671,"num_write_kb":106944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":1773830144,"num_objects":27066,"num_object_clones":0,"num_object_copies":81198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25473,"num_write_kb":1630304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3738,"num_bytes_recovered":244973568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5390532608,"data_stored":5390532608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35607,"ondisk_log_size":35607,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285595,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":411432,"kb_used_data":376084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93333720,"statfs":{"total":95995035648,"available":95573729280,"internally_reserved":0,"allocated":385110016,"data_stored":384613990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547231,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":260908,"kb_used_data":229720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93484244,"statfs":{"total":95995035648,"available":95727865856,"internally_reserved":0,"allocated":235233280,"data_stored":234736328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743460,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":854728,"kb_used_data":805492,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92890424,"statfs":{"total":95995035648,"available":95119794176,"internally_reserved":0,"allocated":824823808,"data_stored":824340400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005097,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":410788,"kb_used_data":371276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93334364,"statfs":{"total":95995035648,"available":95574388736,"internally_reserved":0,"allocated":380186624,"data_stored":379694170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,21],"upper_bound":32768},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234029,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":471952,"kb_used_data":444924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93273200,"statfs":{"total":95995035648,"available":95511756800,"internally_reserved":0,"allocated":455602176,"data_stored":455115216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495668,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422452,"kb_used_data":387104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93322700,"statfs":{"total":95995035648,"available":95562444800,"internally_reserved":0,"allocated":396394496,"data_stored":395893992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724599,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309168,"kb_used_data":277976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93435984,"statfs":{"total":95995035648,"available":95678447616,"internally_reserved":0,"allocated":284647424,"data_stored":284150472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986237,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":470436,"kb_used_data":430924,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93274716,"statfs":{"total":95995035648,"available":95513309184,"internally_reserved":0,"allocated":441266176,"data_stored":440773722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686728,"kb_used_data":651380,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93058424,"statfs":{"total":95995035648,"available":95291826176,"internally_reserved":0,"allocated":667013120,"data_stored":666529712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408972,"kb_used_data":381940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93336180,"statfs":{"total":95995035648,"available":95576248320,"internally_reserved":0,"allocated":391106560,"data_stored":390623152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":523976,"kb_used_data":488628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93221176,"statfs":{"total":95995035648,"available":95458484224,"internally_reserved":0,"allocated":500355072,"data_stored":499871664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464776,"kb_used_data":437748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93280376,"statfs":{"total":95995035648,"available":95519105024,"internally_reserved":0,"allocated":448253952,"data_stored":447770544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":445186048,"data_stored":445186048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":499253248,"data_stored":499253248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":390004736,"data_stored":390004736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":663420928,"data_stored":663420928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":440139776,"data_stored":440139776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":283508736,"data_stored":283508736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":394723328,"data_stored":394723328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":453967872,"data_stored":453967872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":379060224,"data_stored":379060224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":823721984,"data_stored":823721984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":234094592,"data_stored":234094592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":383451136,"data_stored":383451136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:11:46.572 INFO:tasks.thrashosds.thrasher:Setting 2.12 to [6, 10, 3, 1, 9, 7] 2024-09-17T16:11:46.572 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.12', '6', '10', '3', '1', '9', '7'] 2024-09-17T16:11:46.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 2.12 6 10 3 1 9 7 2024-09-17T16:11:46.638 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:46.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:46 smithi173 ceph-mon[30530]: 2.5 scrub starts 2024-09-17T16:11:46.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:46 smithi173 ceph-mon[30530]: 2.1 scrub starts 2024-09-17T16:11:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:46 smithi173 ceph-mon[30530]: 2.9 scrub starts 2024-09-17T16:11:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4270976459' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:46 smithi173 ceph-mon[30530]: 2.0 scrub starts 2024-09-17T16:11:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:46 smithi173 ceph-mon[30530]: 2.2 scrub starts 2024-09-17T16:11:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:46 smithi173 ceph-mon[30530]: 2.17 scrub starts 2024-09-17T16:11:46.660 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:46.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:46.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:11:46.762 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:46.786 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:46.790 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:46.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:46.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:46.892 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:46.914 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:46.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:46.970 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:47.016 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:47.037 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:47.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:47.138 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:47.160 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:47.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:47.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 27129 27113 37.6529 0.384615 0.00315993 0.0265531 2024-09-17T16:11:47.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:47.283 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:47.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:47.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:47 smithi071 ceph-mon[30002]: pgmap v355: 27 pgs: 27 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:47.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:47 smithi071 ceph-mon[30002]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:11:47.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:47 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:47.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4270976459' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:47.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:47 smithi071 ceph-mon[30002]: osdmap e128: 12 total, 12 up, 12 in 2024-09-17T16:11:47.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:47 smithi071 ceph-mon[30002]: 2.3 scrub starts 2024-09-17T16:11:47.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:47 smithi071 ceph-mon[30002]: 2.7 scrub starts 2024-09-17T16:11:47.385 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:47.412 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:47.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:47.479 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:47.479 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:47 smithi017 ceph-mon[27571]: pgmap v355: 27 pgs: 27 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:47 smithi017 ceph-mon[27571]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:11:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:47 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4270976459' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:47 smithi017 ceph-mon[27571]: osdmap e128: 12 total, 12 up, 12 in 2024-09-17T16:11:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:47 smithi017 ceph-mon[27571]: 2.3 scrub starts 2024-09-17T16:11:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:47 smithi017 ceph-mon[27571]: 2.7 scrub starts 2024-09-17T16:11:47.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:47.540 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:47.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:47.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:47 smithi173 ceph-mon[30530]: pgmap v355: 27 pgs: 27 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:47 smithi173 ceph-mon[30530]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:11:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:47 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4270976459' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:47 smithi173 ceph-mon[30530]: osdmap e128: 12 total, 12 up, 12 in 2024-09-17T16:11:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:47 smithi173 ceph-mon[30530]: 2.3 scrub starts 2024-09-17T16:11:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:47 smithi173 ceph-mon[30530]: 2.7 scrub starts 2024-09-17T16:11:47.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:47.663 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:47.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:47.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:47.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:47.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:47.866 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:47.893 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:47.917 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:47.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:48.019 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:48.042 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:48.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:48.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:48.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:48.173 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:48.212 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 28477 28461 38.6657 84.25 0.00447343 0.0258507 2024-09-17T16:11:48.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:48.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:48.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:48.316 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:11:48.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:48.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:48.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:48.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:48.551 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:48.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:48.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:48 smithi071 ceph-mon[30002]: 2.6 scrub starts 2024-09-17T16:11:48.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3834924088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:48.578 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:48.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:48 smithi173 ceph-mon[30530]: 2.6 scrub starts 2024-09-17T16:11:48.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3834924088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:48.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:48.675 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:48.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:48.723 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:11:48.723 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:11:48.723 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:48 smithi017 ceph-mon[27571]: 2.6 scrub starts 2024-09-17T16:11:48.724 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3834924088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:48.747 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:11:48.747 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:11:48.772 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:11:48.772 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:11:48.777 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:48.795 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:11:48.795 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:11:48.800 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:48.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:48.818 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:11:48.818 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:11:48.839 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:11:48.839 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:11:48.860 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:11:48.860 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:11:48.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:48.902 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:11:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:11:48.925 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:11:48.925 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:11:48.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:48.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:48.948 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:11:48.948 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:11:48.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:11:48.974 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:11:49.002 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:11:49.003 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:11:49.029 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:49.056 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:49.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:13493 2.17 2:ef6c8979:::benchmark_data_smithi173_55121_object13492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:12.334922+0000", 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 36.721577273999998, 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0242685929999999, 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.060 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 13493 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334922+0000", 2024-09-17T16:11:49.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334922+0000", 2024-09-17T16:11:49.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334923+0000", 2024-09-17T16:11:49.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.192e-06 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334999+0000", 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5327999999999994e-05 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.335013+0000", 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.42e-05 2024-09-17T16:11:49.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.335019+0000", 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0920000000000003e-06 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.350269+0000", 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.01524939 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:11:49.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.350275+0000", 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7399999999999998e-06 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:13.359107+0000", 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0088320850000001 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:13.359191+0000", 2024-09-17T16:11:49.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3566000000000006e-05 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14929 2.11 2.db77d491 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.268492+0000", 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 34.788007657000001, 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.99314310400000005, 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.066 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14929 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268492+0000", 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268492+0000", 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268493+0000", 2024-09-17T16:11:49.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1900000000000001e-07 2024-09-17T16:11:49.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268552+0000", 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8807999999999997e-05 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268562+0000", 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0180000000000001e-05 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268567+0000", 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0549999999999999e-06 2024-09-17T16:11:49.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268612+0000", 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5719000000000001e-05 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268618+0000", 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5650000000000001e-06 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261624+0000", 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.993005519 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261635+0000", 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1439e-05 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14934 2.11 2.10e332f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:11:49.071 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.326376+0000", 2024-09-17T16:11:49.072 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 34.730123982000002, 2024-09-17T16:11:49.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93529751100000003, 2024-09-17T16:11:49.072 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.072 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.072 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:11:49.072 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.072 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.072 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.072 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14934 2024-09-17T16:11:49.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326376+0000", 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326376+0000", 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326378+0000", 2024-09-17T16:11:49.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1849999999999998e-06 2024-09-17T16:11:49.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326430+0000", 2024-09-17T16:11:49.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2686999999999999e-05 2024-09-17T16:11:49.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326448+0000", 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7518999999999999e-05 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326455+0000", 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2230000000000001e-06 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326508+0000", 2024-09-17T16:11:49.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2645000000000002e-05 2024-09-17T16:11:49.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:11:49.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326517+0000", 2024-09-17T16:11:49.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7679999999999997e-06 2024-09-17T16:11:49.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261665+0000", 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93514807200000005 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261673+0000", 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4120000000000001e-06 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14937 2.11 2.66ac0671 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.333944+0000", 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 34.722555372000002, 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.92790043600000005, 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14937 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.078 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.333944+0000", 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.333944+0000", 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.333945+0000", 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6499999999999998e-07 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334020+0000", 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5496999999999999e-05 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334029+0000", 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5909999999999996e-06 2024-09-17T16:11:49.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334033+0000", 2024-09-17T16:11:49.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2799999999999997e-06 2024-09-17T16:11:49.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334071+0000", 2024-09-17T16:11:49.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7719999999999998e-05 2024-09-17T16:11:49.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334076+0000", 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6659999999999999e-06 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261837+0000", 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.92776128899999999 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261845+0000", 2024-09-17T16:11:49.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6279999999999996e-06 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14955 2.11 2.20b690d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.701197+0000", 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 34.355302989000002, 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56068107599999994, 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.083 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.084 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:11:49.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.084 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14955 2024-09-17T16:11:49.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.084 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701197+0000", 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701197+0000", 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701206+0000", 2024-09-17T16:11:49.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6399999999999992e-06 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701258+0000", 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2159e-05 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701289+0000", 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0482999999999999e-05 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701303+0000", 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4576000000000001e-05 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701450+0000", 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014646099999999999 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701466+0000", 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5913000000000002e-05 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261871+0000", 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56040511900000001 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261878+0000", 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.725e-06 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.088 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14964 2.17 2:ead62e90:::benchmark_data_smithi173_55121_object14963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.870757+0000", 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 34.185742941999997, 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28433903799999999, 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.089 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14964 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870757+0000", 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870757+0000", 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870757+0000", 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0800000000000004e-07 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870849+0000", 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1188999999999997e-05 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870864+0000", 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4951e-05 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870871+0000", 2024-09-17T16:11:49.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4760000000000001e-06 2024-09-17T16:11:49.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870907+0000", 2024-09-17T16:11:49.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6145e-05 2024-09-17T16:11:49.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870982+0000", 2024-09-17T16:11:49.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4803999999999997e-05 2024-09-17T16:11:49.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:11:49.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871071+0000", 2024-09-17T16:11:49.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8721000000000007e-05 2024-09-17T16:11:49.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871587+0000", 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00051653300000000001 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872130+0000", 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054294599999999999 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.154876+0000", 2024-09-17T16:11:49.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28274554299999999 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.155026+0000", 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014985700000000001 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.155096+0000", 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0065000000000001e-05 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:14964 2.17 e106/104 ondisk, result = 0)", 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.871651+0000", 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 34.184848881000001, 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28345393800000002, 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871651+0000", 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871651+0000", 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871651+0000", 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2600000000000002e-07 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871655+0000", 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4790000000000001e-06 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.154629+0000", 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28297412399999999 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.154646+0000", 2024-09-17T16:11:49.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6677e-05 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.154832+0000", 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000186837 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.154858+0000", 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5276e-05 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.155105+0000", 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024691900000000001 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14965 2.11 2.79d92c71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.872030+0000", 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 34.184469368000002, 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.389896574, 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.101 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.102 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:11:49.102 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.102 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.102 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.102 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14965 2024-09-17T16:11:49.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.102 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872030+0000", 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872030+0000", 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872031+0000", 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6899999999999996e-07 2024-09-17T16:11:49.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872113+0000", 2024-09-17T16:11:49.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1584999999999999e-05 2024-09-17T16:11:49.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872126+0000", 2024-09-17T16:11:49.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3181e-05 2024-09-17T16:11:49.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872132+0000", 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5390000000000004e-06 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872179+0000", 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7058000000000002e-05 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:11:49.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872185+0000", 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7699999999999998e-06 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261919+0000", 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38973334999999998 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261927+0000", 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2220000000000002e-06 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.106 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:21738 2.7 2:e3b5b582:::benchmark_data_smithi173_55121_object21737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:26.656279+0000", 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.400220995000002, 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23206100299999999, 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 21738 2024-09-17T16:11:49.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.108 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.656279+0000", 2024-09-17T16:11:49.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.656279+0000", 2024-09-17T16:11:49.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.656280+0000", 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.88e-06 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.656323+0000", 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2224000000000001e-05 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.110 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.656334+0000", 2024-09-17T16:11:49.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0913999999999999e-05 2024-09-17T16:11:49.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.110 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.110 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.656339+0000", 2024-09-17T16:11:49.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.254e-06 2024-09-17T16:11:49.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.110 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.656388+0000", 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9295999999999997e-05 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.656469+0000", 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1032000000000006e-05 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:11:49.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.656550+0000", 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0969000000000004e-05 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.657093+0000", 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054334200000000004 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.825942+0000", 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16884837899999999 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.888234+0000", 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.062292509000000003 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.888295+0000", 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0909999999999998e-05 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.888340+0000", 2024-09-17T16:11:49.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4294e-05 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24310 2.7 2:e3e07458:::benchmark_data_smithi173_55121_object24309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.401789+0000", 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 19.654710309999999, 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.261667339, 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.114 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24310 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401789+0000", 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401789+0000", 2024-09-17T16:11:49.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401790+0000", 2024-09-17T16:11:49.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2399999999999998e-07 2024-09-17T16:11:49.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.117 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.117 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401863+0000", 2024-09-17T16:11:49.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2873000000000005e-05 2024-09-17T16:11:49.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.118 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401870+0000", 2024-09-17T16:11:49.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6000000000000001e-06 2024-09-17T16:11:49.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401874+0000", 2024-09-17T16:11:49.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8990000000000003e-06 2024-09-17T16:11:49.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401902+0000", 2024-09-17T16:11:49.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7807999999999999e-05 2024-09-17T16:11:49.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401975+0000", 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2803000000000001e-05 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.402056+0000", 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0778000000000005e-05 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.402499+0000", 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00044296800000000002 2024-09-17T16:11:49.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.402887+0000", 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00038852299999999999 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663392+0000", 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26050499300000002 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663430+0000", 2024-09-17T16:11:49.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8226000000000003e-05 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663457+0000", 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6244000000000001e-05 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24327 2.17 2:ecf8c341:::benchmark_data_smithi173_55121_object24326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.417487+0000", 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 19.639012730000001, 2024-09-17T16:11:49.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26755120999999998, 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24327 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417487+0000", 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417487+0000", 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417488+0000", 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0580000000000001e-06 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417538+0000", 2024-09-17T16:11:49.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9704999999999999e-05 2024-09-17T16:11:49.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417549+0000", 2024-09-17T16:11:49.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1037e-05 2024-09-17T16:11:49.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417556+0000", 2024-09-17T16:11:49.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0280000000000003e-06 2024-09-17T16:11:49.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417599+0000", 2024-09-17T16:11:49.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3739000000000003e-05 2024-09-17T16:11:49.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417683+0000", 2024-09-17T16:11:49.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3493999999999997e-05 2024-09-17T16:11:49.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.128 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:11:49.128 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417771+0000", 2024-09-17T16:11:49.128 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8325000000000006e-05 2024-09-17T16:11:49.128 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:11:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.418259+0000", 2024-09-17T16:11:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000487398 2024-09-17T16:11:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.418603+0000", 2024-09-17T16:11:49.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00034413100000000002 2024-09-17T16:11:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.684960+0000", 2024-09-17T16:11:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26635714300000002 2024-09-17T16:11:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.684997+0000", 2024-09-17T16:11:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7218999999999997e-05 2024-09-17T16:11:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.131 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.685038+0000", 2024-09-17T16:11:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0933000000000003e-05 2024-09-17T16:11:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:24310 2.7 e120/104 ondisk, result = 0)", 2024-09-17T16:11:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.418660+0000", 2024-09-17T16:11:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 19.637839141000001, 2024-09-17T16:11:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24480184699999999, 2024-09-17T16:11:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.133 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:11:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.418660+0000", 2024-09-17T16:11:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.418660+0000", 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.418661+0000", 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5499999999999998e-07 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.418665+0000", 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.726e-06 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663311+0000", 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24464601899999999 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663315+0000", 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4710000000000001e-06 2024-09-17T16:11:49.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663382+0000", 2024-09-17T16:11:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6765000000000002e-05 2024-09-17T16:11:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663389+0000", 2024-09-17T16:11:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.55e-06 2024-09-17T16:11:49.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.138 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.138 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.138 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663462+0000", 2024-09-17T16:11:49.138 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3660999999999998e-05 2024-09-17T16:11:49.138 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.138 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.138 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.139 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.139 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.139 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24378 2.17 2:eca9db3e:::benchmark_data_smithi173_55121_object24377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:11:49.139 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.695814+0000", 2024-09-17T16:11:49.139 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 19.360685178000001, 2024-09-17T16:11:49.139 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39380668000000002, 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24378 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.140 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695814+0000", 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695814+0000", 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695816+0000", 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.15e-06 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695864+0000", 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8797000000000001e-05 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695875+0000", 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0795000000000001e-05 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695881+0000", 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7470000000000002e-06 2024-09-17T16:11:49.142 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695926+0000", 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4669000000000002e-05 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696011+0000", 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5141000000000004e-05 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696097+0000", 2024-09-17T16:11:49.143 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5805999999999998e-05 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696594+0000", 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049760799999999997 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.697152+0000", 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055830099999999998 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.144 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089548+0000", 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39239545599999998 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089589+0000", 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1539999999999999e-05 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089621+0000", 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.167e-05 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.145 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:24378 2.17 e120/104 ondisk, result = 0)", 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.696958+0000", 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 19.359541221000001, 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39266769299999998, 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.146 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696958+0000", 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696958+0000", 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696959+0000", 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.99e-07 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.147 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696962+0000", 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5310000000000002e-06 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089489+0000", 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39252651 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089496+0000", 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.165e-06 2024-09-17T16:11:49.148 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089537+0000", 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0401000000000002e-05 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089543+0000", 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5130000000000002e-06 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089626+0000", 2024-09-17T16:11:49.149 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2973999999999995e-05 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:25247 2.14 2:2f1420ea:::benchmark_data_smithi173_55121_object25246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.811018+0000", 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 18.245481857000001, 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23636635, 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:49.150 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 25247 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.811018+0000", 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.811018+0000", 2024-09-17T16:11:49.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.811019+0000", 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4190000000000001e-06 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.811066+0000", 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6505999999999998e-05 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.152 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.811076+0000", 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0185e-05 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.811081+0000", 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2950000000000001e-06 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.811130+0000", 2024-09-17T16:11:49.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9283000000000002e-05 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.811226+0000", 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5989999999999994e-05 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 5,9", 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.811315+0000", 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9065000000000002e-05 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:11:49.154 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.811863+0000", 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000547257 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.812286+0000", 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00042353600000000002 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.047307+0000", 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.235020429 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.155 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.047345+0000", 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8738000000000001e-05 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.047384+0000", 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8646999999999998e-05 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.156 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:25247 2.14 e121/117 ondisk, result = 0)", 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.812276+0000", 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 18.244223238, 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.235113462, 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.812276+0000", 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.157 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.812276+0000", 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.812277+0000", 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8000000000000001e-07 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.812279+0000", 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.616e-06 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.158 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.047236+0000", 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23495709300000001 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.047242+0000", 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3449999999999996e-06 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.159 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.047295+0000", 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3313000000000003e-05 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.047301+0000", 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1650000000000003e-06 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.047390+0000", 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8549999999999998e-05 2024-09-17T16:11:49.160 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:26973 2.17 2:e9d3e7ef:::benchmark_data_smithi173_55121_object26972:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:32.894163+0000", 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 16.162336396000001, 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26820020100000003, 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.161 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 26973 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894163+0000", 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894163+0000", 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.162 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894164+0000", 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894189+0000", 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5344e-05 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894201+0000", 2024-09-17T16:11:49.163 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2116000000000001e-05 2024-09-17T16:11:49.164 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.164 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.164 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.164 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894208+0000", 2024-09-17T16:11:49.164 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.533e-06 2024-09-17T16:11:49.164 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.165 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.165 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.165 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894250+0000", 2024-09-17T16:11:49.165 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1740999999999999e-05 2024-09-17T16:11:49.165 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.165 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.165 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.165 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894332+0000", 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2132000000000006e-05 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894415+0000", 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3603000000000006e-05 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894846+0000", 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00043026300000000001 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.166 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.015482+0000", 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.120636205 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162286+0000", 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.14680454500000001 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162322+0000", 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5673999999999998e-05 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.167 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162363+0000", 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1230000000000003e-05 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:26973 2.17 e121/104 ondisk, result = 0)", 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:32.895491+0000", 2024-09-17T16:11:49.168 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 16.161008961, 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26687919500000001, 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.895491+0000", 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.169 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.895491+0000", 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.895491+0000", 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7599999999999997e-07 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.895494+0000", 2024-09-17T16:11:49.170 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7020000000000001e-06 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162227+0000", 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26673340000000001 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162232+0000", 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.171 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162278+0000", 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5797000000000002e-05 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162283+0000", 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0710000000000002e-06 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162370+0000", 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6615999999999999e-05 2024-09-17T16:11:49.172 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:27006 2.7 2:e4bc7e40:::benchmark_data_smithi173_55121_object27005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:33.164406+0000", 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 15.892093138, 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31356054900000002, 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:49.173 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 27006 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164406+0000", 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.174 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164406+0000", 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164407+0000", 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164434+0000", 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6551999999999999e-05 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.175 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164445+0000", 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1683e-05 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164468+0000", 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2708e-05 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164511+0000", 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2923000000000002e-05 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.176 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164588+0000", 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6625000000000006e-05 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164669+0000", 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1527999999999996e-05 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165334+0000", 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00066428899999999998 2024-09-17T16:11:49.177 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165780+0000", 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000446799 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477887+0000", 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31210646199999997 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.178 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477933+0000", 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6013999999999997e-05 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477967+0000", 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4167e-05 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.179 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:27006 2.7 e121/104 ondisk, result = 0)", 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 15.890741041, 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31221570399999998, 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.180 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2099999999999996e-07 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165764+0000", 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3159999999999998e-06 2024-09-17T16:11:49.181 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477726+0000", 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31196209800000002 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477766+0000", 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0238999999999999e-05 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477875+0000", 2024-09-17T16:11:49.182 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010957499999999999 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477883+0000", 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2860000000000002e-06 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477974+0000", 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1469000000000001e-05 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:11:49.183 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:11:49.184 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:11:49.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:49.208 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:49.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:49.213 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 29471 29455 39.1647 62.125 0.0265426 0.0255195 2024-09-17T16:11:49.269 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:11:49.309 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:49.323 INFO:teuthology.orchestra.run.smithi017.stderr:set 2.12 pg_upmap_items mapping to [6->10,3->1,9->7] 2024-09-17T16:11:49.339 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:49.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:49.441 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:49.463 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:49.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:49.566 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:49.590 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:49.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:49.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:49 smithi173 ceph-mon[30530]: pgmap v357: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1853240986' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [6, 10, 3, 1, 9, 7]} : dispatch 2024-09-17T16:11:49.692 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:49.715 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:49.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:49.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:49.735 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:49 smithi017 ceph-mon[27571]: pgmap v357: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:49.735 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1853240986' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [6, 10, 3, 1, 9, 7]} : dispatch 2024-09-17T16:11:49.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:49 smithi071 ceph-mon[30002]: pgmap v357: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:11:49.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1853240986' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [6, 10, 3, 1, 9, 7]} : dispatch 2024-09-17T16:11:49.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:49.838 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:49.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:49.940 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:49.962 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:49.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:49.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:11:50.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:50.085 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:50.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:50.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:50.210 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:50.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:50.213 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 29877 29861 38.8773 25.375 0.0016093 0.0254916 2024-09-17T16:11:50.216 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:50.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:50.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:50.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:50.439 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:50.459 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:50.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:50.561 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:50.584 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:50.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:50.621 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1853240986' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [6, 10, 3, 1, 9, 7]}]': finished 2024-09-17T16:11:50.621 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:50 smithi071 ceph-mon[30002]: osdmap e129: 12 total, 12 up, 12 in 2024-09-17T16:11:50.644 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:11:50.644 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:11:50.644 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:11:50.644 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:11:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1853240986' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [6, 10, 3, 1, 9, 7]}]': finished 2024-09-17T16:11:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:50 smithi173 ceph-mon[30530]: osdmap e129: 12 total, 12 up, 12 in 2024-09-17T16:11:50.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:50.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:50.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:50.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1853240986' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [6, 10, 3, 1, 9, 7]}]': finished 2024-09-17T16:11:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:50 smithi017 ceph-mon[27571]: osdmap e129: 12 total, 12 up, 12 in 2024-09-17T16:11:50.809 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:50.831 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:50.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:50.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:50.933 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:50.954 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:50.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:50.971 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:51.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:51.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:51.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:51.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:51.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:51.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:51.213 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 29901 29885 38.1145 1.5 0.00150992 0.0255123 2024-09-17T16:11:51.305 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:51.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:11:51.328 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:51.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:51.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:51.460 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:51.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:51.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:51 smithi017 ceph-mon[27571]: pgmap v359: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 128 op/s 2024-09-17T16:11:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:51 smithi017 ceph-mon[27571]: osdmap e130: 12 total, 12 up, 12 in 2024-09-17T16:11:51.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:51 smithi071 ceph-mon[30002]: pgmap v359: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 128 op/s 2024-09-17T16:11:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:51 smithi071 ceph-mon[30002]: osdmap e130: 12 total, 12 up, 12 in 2024-09-17T16:11:51.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:51.595 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:51.631 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:51.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:51 smithi173 ceph-mon[30530]: pgmap v359: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 1.7 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 128 op/s 2024-09-17T16:11:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:51 smithi173 ceph-mon[30530]: osdmap e130: 12 total, 12 up, 12 in 2024-09-17T16:11:51.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:51.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:51.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:51.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:51.765 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:11:51.829 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:51.854 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:51.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:51.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:51.984 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:51.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:51.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:52.085 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:52.086 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:52.114 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:52.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:52.213 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 31495 31479 39.3445 99.625 0.00167907 0.0254096 2024-09-17T16:11:52.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:52.241 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:52.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:52.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:52.357 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:11:52.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:52.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:52.407 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:11:52.407 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:11:52.408 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:11:52.408 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:11:52.408 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:11:52.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:52 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:11:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:52 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:11:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:52 smithi017 ceph-mon[27571]: osdmap e131: 12 total, 12 up, 12 in 2024-09-17T16:11:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:52 smithi017 ceph-mon[27571]: pgmap v362: 27 pgs: 1 remapped+peering, 26 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 521 op/s 2024-09-17T16:11:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3355527715' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:52 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:52.500 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:52.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:52.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:52 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:11:52.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:52 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:11:52.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:52 smithi071 ceph-mon[30002]: osdmap e131: 12 total, 12 up, 12 in 2024-09-17T16:11:52.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:52 smithi071 ceph-mon[30002]: pgmap v362: 27 pgs: 1 remapped+peering, 26 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 521 op/s 2024-09-17T16:11:52.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3355527715' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:52.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:52 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:52.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:52.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:52.628 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:52.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:52.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:52 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:11:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:52 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:11:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:52 smithi173 ceph-mon[30530]: osdmap e131: 12 total, 12 up, 12 in 2024-09-17T16:11:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:52 smithi173 ceph-mon[30530]: pgmap v362: 27 pgs: 1 remapped+peering, 26 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 521 op/s 2024-09-17T16:11:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3355527715' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:52 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:11:52.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:52.759 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:52.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:52.846 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:11:53.376 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:11:53.376 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:11:53.376 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:11:53.377 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:11:53.377 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:11:53.377 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:11:53.377 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:11:53.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:11:53.378 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:11:53.379 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:11:53.379 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:11:53.379 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:11:53.380 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:11:53.380 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:53.381 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:11:53.382 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 33332 33316 40.824 114.812 0.00171081 0.0244853 2024-09-17T16:11:53.382 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:11:53.382 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:11:53.383 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:11:53.383 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:11:53.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:11:53.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:11:53.384 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:11:53.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:11:53.385 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:11:53.385 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:11:53.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:53.450 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:53.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:53.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:53 smithi071 ceph-mon[30002]: 2.10 scrub starts 2024-09-17T16:11:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:53 smithi071 ceph-mon[30002]: 2.18 scrub starts 2024-09-17T16:11:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:53 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:53 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:11:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:53 smithi071 ceph-mon[30002]: osdmap e132: 12 total, 12 up, 12 in 2024-09-17T16:11:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1956809711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:53.574 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:53.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:53.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:53 smithi017 ceph-mon[27571]: 2.10 scrub starts 2024-09-17T16:11:53.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:53 smithi017 ceph-mon[27571]: 2.18 scrub starts 2024-09-17T16:11:53.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:53 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:53.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:53 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:11:53.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:53 smithi017 ceph-mon[27571]: osdmap e132: 12 total, 12 up, 12 in 2024-09-17T16:11:53.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1956809711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:53.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:53 smithi173 ceph-mon[30530]: 2.10 scrub starts 2024-09-17T16:11:53.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:53 smithi173 ceph-mon[30530]: 2.18 scrub starts 2024-09-17T16:11:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:53 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:11:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:53 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:11:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:53 smithi173 ceph-mon[30530]: osdmap e132: 12 total, 12 up, 12 in 2024-09-17T16:11:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1956809711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:53.658 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:11:53.676 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:53.703 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:53.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:53.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:53.805 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:53.828 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:53.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:53.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:53.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:53.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:53.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:54.055 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:54.082 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:54.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:54.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:54.188 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:54.188 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:54.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:54.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:54.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:54.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 34624 34608 41.5917 80.75 0.00187455 0.0240316 2024-09-17T16:11:54.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:54.338 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:54.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:54.440 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:54.467 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:54.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:54.569 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:54.593 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:54.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:54.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:54.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:54 smithi173 ceph-mon[30530]: 2.13 scrub starts 2024-09-17T16:11:54.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:54 smithi173 ceph-mon[30530]: 2.13 scrub ok 2024-09-17T16:11:54.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:54 smithi173 ceph-mon[30530]: pgmap v364: 27 pgs: 1 remapped+peering, 26 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 640 op/s 2024-09-17T16:11:54.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:11:54.718 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:54.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:11:54.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:54 smithi017 ceph-mon[27571]: 2.13 scrub starts 2024-09-17T16:11:54.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:54 smithi017 ceph-mon[27571]: 2.13 scrub ok 2024-09-17T16:11:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:54 smithi017 ceph-mon[27571]: pgmap v364: 27 pgs: 1 remapped+peering, 26 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 640 op/s 2024-09-17T16:11:54.808 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:54 smithi071 ceph-mon[30002]: 2.13 scrub starts 2024-09-17T16:11:54.808 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:54 smithi071 ceph-mon[30002]: 2.13 scrub ok 2024-09-17T16:11:54.808 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:54 smithi071 ceph-mon[30002]: pgmap v364: 27 pgs: 1 remapped+peering, 26 active+clean; 1.8 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 640 op/s 2024-09-17T16:11:54.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:54.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:54.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:54.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:54.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:11:54.976 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:54.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:55.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:55.102 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:55.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:55.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:55.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:55.213 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 35945 35929 42.3646 82.5625 0.0495747 0.0235826 2024-09-17T16:11:55.227 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:55.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:55.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:55.267 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:11:55.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:55.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:55.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:55.387 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:11:55.451 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:55.474 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:55.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:55.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:55.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:55.597 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:55.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:55.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:55.626 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:55 smithi173 ceph-mon[30530]: 2.15 scrub starts 2024-09-17T16:11:55.626 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:55 smithi173 ceph-mon[30530]: 2.15 scrub ok 2024-09-17T16:11:55.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:55 smithi017 ceph-mon[27571]: 2.15 scrub starts 2024-09-17T16:11:55.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:55 smithi017 ceph-mon[27571]: 2.15 scrub ok 2024-09-17T16:11:55.672 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:55.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:55.730 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:55.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:55.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:55.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:55 smithi071 ceph-mon[30002]: 2.15 scrub starts 2024-09-17T16:11:55.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:55 smithi071 ceph-mon[30002]: 2.15 scrub ok 2024-09-17T16:11:55.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:55.863 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:55.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:55.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:55.904 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:55.904 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:55.913 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-17T16:11:55.913 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:11:55.913 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-17T16:11:55.913 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-17T16:11:55.913 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.7 2024-09-17T16:11:55.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:55.968 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:11:56.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:56.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:56.106 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:56.141 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:56.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:56.213 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 37165 37149 42.992 76.25 0.0174984 0.0232537 2024-09-17T16:11:56.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:56.264 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:55 smithi071 systemd[1]: Stopping Ceph osd.7 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:11:56.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:56.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:56.377 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:56.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:56.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:56.543 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:56.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:56.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:56 smithi071 ceph-mon[30002]: 2.19 scrub starts 2024-09-17T16:11:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:56 smithi071 ceph-mon[30002]: 2.16 scrub starts 2024-09-17T16:11:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:56 smithi071 ceph-mon[30002]: pgmap v365: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 880 op/s; 357/99372 objects misplaced (0.359%); 5.9 MiB/s, 95 objects/s recovering 2024-09-17T16:11:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:56 smithi071 ceph-mon[30002]: osd.7 marked itself down and dead 2024-09-17T16:11:56.564 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:56 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[50487]: 2024-09-17T16:11:56.260+0000 7fbe407fc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:11:56.564 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:56 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[50487]: 2024-09-17T16:11:56.260+0000 7fbe407fc640 -1 osd.7 132 *** Got signal Terminated *** 2024-09-17T16:11:56.564 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:56 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[50487]: 2024-09-17T16:11:56.260+0000 7fbe407fc640 -1 osd.7 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:11:56.583 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:56 smithi173 ceph-mon[30530]: 2.19 scrub starts 2024-09-17T16:11:56.583 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:56 smithi173 ceph-mon[30530]: 2.16 scrub starts 2024-09-17T16:11:56.583 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:56 smithi173 ceph-mon[30530]: pgmap v365: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 880 op/s; 357/99372 objects misplaced (0.359%); 5.9 MiB/s, 95 objects/s recovering 2024-09-17T16:11:56.583 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:56 smithi173 ceph-mon[30530]: osd.7 marked itself down and dead 2024-09-17T16:11:56.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:11:56.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:56 smithi017 ceph-mon[27571]: 2.19 scrub starts 2024-09-17T16:11:56.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:56 smithi017 ceph-mon[27571]: 2.16 scrub starts 2024-09-17T16:11:56.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:56 smithi017 ceph-mon[27571]: pgmap v365: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 2.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 880 op/s; 357/99372 objects misplaced (0.359%); 5.9 MiB/s, 95 objects/s recovering 2024-09-17T16:11:56.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:56 smithi017 ceph-mon[27571]: osd.7 marked itself down and dead 2024-09-17T16:11:56.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:56.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:56.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:56.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:56.787 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:56.810 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:56.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:56.870 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:11:56.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:56.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:56.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:56.995 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:11:57.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:57.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:57.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:57.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:57.178 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:57.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:57.213 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:56 smithi071 podman[58654]: 2024-09-17 16:11:56.903862602 +0000 UTC m=+0.771257987 container died cf891d90ddf69512289871cd504df45fc2288ee6070576b0cbc8fb379d475b0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-17T16:11:57.213 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 37505 37489 42.5967 21.25 0.298557 0.023223 2024-09-17T16:11:57.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:57.306 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:57.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:57.367 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:11:57.378 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:11:57.378 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:11:57.406 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:57.431 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:57.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:57.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:57.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:57.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:57 smithi071 ceph-mon[30002]: 2.16 scrub ok 2024-09-17T16:11:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:57 smithi071 ceph-mon[30002]: 2.19 scrub ok 2024-09-17T16:11:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:57 smithi071 ceph-mon[30002]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T16:11:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:57 smithi071 ceph-mon[30002]: osdmap e133: 12 total, 11 up, 12 in 2024-09-17T16:11:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2857996967' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:57 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:11:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:57 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:11:57.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:57 smithi017 ceph-mon[27571]: 2.16 scrub ok 2024-09-17T16:11:57.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:57 smithi017 ceph-mon[27571]: 2.19 scrub ok 2024-09-17T16:11:57.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:57 smithi017 ceph-mon[27571]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T16:11:57.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:57 smithi017 ceph-mon[27571]: osdmap e133: 12 total, 11 up, 12 in 2024-09-17T16:11:57.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2857996967' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:57.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:57 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:11:57.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:57 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:11:57.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:57 smithi173 ceph-mon[30530]: 2.16 scrub ok 2024-09-17T16:11:57.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:57 smithi173 ceph-mon[30530]: 2.19 scrub ok 2024-09-17T16:11:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:57 smithi173 ceph-mon[30530]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T16:11:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:57 smithi173 ceph-mon[30530]: osdmap e133: 12 total, 11 up, 12 in 2024-09-17T16:11:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2857996967' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:11:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:57 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:11:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:57 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:11:57.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:11:57.717 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:57.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:11:57.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:57.842 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:57.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:57.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:57.965 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:57.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:58.035 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:11:58.035 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:11:58.058 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:11:58.058 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:11:58.063 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:57 smithi071 podman[58654]: 2024-09-17 16:11:57.619511807 +0000 UTC m=+1.486907233 container cleanup cf891d90ddf69512289871cd504df45fc2288ee6070576b0cbc8fb379d475b0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True) 2024-09-17T16:11:58.063 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:57 smithi071 bash[58654]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7 2024-09-17T16:11:58.063 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:57 smithi071 podman[58669]: 2024-09-17 16:11:57.820457246 +0000 UTC m=+0.916290460 container remove cf891d90ddf69512289871cd504df45fc2288ee6070576b0cbc8fb379d475b0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:11:58.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:58.078 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:11:58.078 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:11:58.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:11:58.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:58.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:58.090 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:58.099 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:11:58.099 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:11:58.121 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:11:58.121 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:11:58.144 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:11:58.144 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:11:58.165 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:11:58.165 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:11:58.165 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:11:58.187 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:11:58.187 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:11:58.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:58.209 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:11:58.209 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:11:58.213 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:58.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:11:58.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 37817 37801 42.1842 19.5 0.0672934 0.0233933 2024-09-17T16:11:58.231 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:11:58.231 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:11:58.253 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:11:58.254 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:11:58.315 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:11:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:58.337 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:58.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:11:58.342 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:11:58.438 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:58.467 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:58.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:58 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:11:58.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2857996967' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:58.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:58 smithi017 ceph-mon[27571]: osdmap e134: 12 total, 11 up, 12 in 2024-09-17T16:11:58.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:58 smithi017 ceph-mon[27571]: pgmap v368: 27 pgs: 1 peering, 3 stale+active+clean, 1 active+recovering+undersized+remapped, 22 active+clean; 2.2 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 526 op/s; 357/108768 objects misplaced (0.328%) 2024-09-17T16:11:58.469 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1491216613' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:58.469 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:11:58.469 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:11:58.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:58.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:58.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:58 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:11:58.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2857996967' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:58.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:58 smithi071 ceph-mon[30002]: osdmap e134: 12 total, 11 up, 12 in 2024-09-17T16:11:58.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:58 smithi071 ceph-mon[30002]: pgmap v368: 27 pgs: 1 peering, 3 stale+active+clean, 1 active+recovering+undersized+remapped, 22 active+clean; 2.2 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 526 op/s; 357/108768 objects misplaced (0.328%) 2024-09-17T16:11:58.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1491216613' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:58.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:11:58.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:11:58.506 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:58 smithi071 podman[58808]: 2024-09-17 16:11:58.139186681 +0000 UTC m=+0.016384846 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:11:58.506 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:58 smithi071 podman[58808]: 2024-09-17 16:11:58.254848921 +0000 UTC m=+0.132047089 container create 3e1894bf097a90d45b3fac5fe82d0b86fdc155408dbf55f7a7d701e836450c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T16:11:58.506 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:58 smithi071 podman[58808]: 2024-09-17 16:11:58.499959765 +0000 UTC m=+0.377157938 container init 3e1894bf097a90d45b3fac5fe82d0b86fdc155408dbf55f7a7d701e836450c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=) 2024-09-17T16:11:58.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:11:58.599 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:58.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:11:58.620 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:58 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:11:58.620 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2857996967' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:11:58.620 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:58 smithi173 ceph-mon[30530]: osdmap e134: 12 total, 11 up, 12 in 2024-09-17T16:11:58.620 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:58 smithi173 ceph-mon[30530]: pgmap v368: 27 pgs: 1 peering, 3 stale+active+clean, 1 active+recovering+undersized+remapped, 22 active+clean; 2.2 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 526 op/s; 357/108768 objects misplaced (0.328%) 2024-09-17T16:11:58.620 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1491216613' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:11:58.620 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:11:58.621 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:11:58.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:11:58.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:58.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:11:58.814 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:58 smithi071 podman[58808]: 2024-09-17 16:11:58.50521959 +0000 UTC m=+0.382417757 container start 3e1894bf097a90d45b3fac5fe82d0b86fdc155408dbf55f7a7d701e836450c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:11:58.814 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:58 smithi071 podman[58808]: 2024-09-17 16:11:58.60011559 +0000 UTC m=+0.477313757 container attach 3e1894bf097a90d45b3fac5fe82d0b86fdc155408dbf55f7a7d701e836450c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T16:11:58.814 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:11:58 smithi071 podman[58808]: 2024-09-17 16:11:58.729442712 +0000 UTC m=+0.606640881 container died 3e1894bf097a90d45b3fac5fe82d0b86fdc155408dbf55f7a7d701e836450c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-17T16:11:58.835 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:11:58.859 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:58.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:13514 2.f 2:f9c4878d:::benchmark_data_smithi173_55121_object13513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:12.318561+0000", 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.562602216999998, 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34231690599999998, 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.885 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 13514 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318561+0000", 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318561+0000", 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318562+0000", 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318587+0000", 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4413000000000002e-05 2024-09-17T16:11:58.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318599+0000", 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2449999999999999e-05 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318605+0000", 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0059999999999998e-06 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318661+0000", 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5299e-05 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318781+0000", 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120522 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318927+0000", 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000145594 2024-09-17T16:11:58.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.319619+0000", 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000692777 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.341876+0000", 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.022256811000000001 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.660723+0000", 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31884684699999999 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.660839+0000", 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115951 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.660878+0000", 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8899000000000001e-05 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.891 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:13492 2.15 2:a809b404:::benchmark_data_smithi173_55121_object13491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:12.334855+0000", 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.546308375999999, 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.019215271, 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.892 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 13492 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334855+0000", 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334855+0000", 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334856+0000", 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6700000000000002e-07 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334879+0000", 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3359000000000001e-05 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334901+0000", 2024-09-17T16:11:58.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2294999999999999e-05 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334911+0000", 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3649999999999993e-06 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.343520+0000", 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0086088640000000008 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:11:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.343525+0000", 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9930000000000003e-06 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.353911+0000", 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0103866720000001 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.354070+0000", 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000158856 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.896 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:14833 2.15 2:af93682f:::benchmark_data_smithi173_55121_object14832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:13.938688+0000", 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.942475399999999, 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30825458100000003, 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.897 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14833 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.938688+0000", 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.938688+0000", 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.938689+0000", 2024-09-17T16:11:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5899999999999995e-07 2024-09-17T16:11:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.938706+0000", 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7036000000000001e-05 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.938716+0000", 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0654e-05 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.938721+0000", 2024-09-17T16:11:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1580000000000004e-06 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.938759+0000", 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7728000000000002e-05 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.938828+0000", 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9292e-05 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:11:58.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.938918+0000", 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9313000000000004e-05 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.939298+0000", 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037980300000000001 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.939737+0000", 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043941800000000001 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.246833+0000", 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30709637200000001 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.246899+0000", 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5197999999999995e-05 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.246942+0000", 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3850000000000002e-05 2024-09-17T16:11:58.903 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:14833 2.15 e105/104 ondisk, result = 0)", 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:13.939551+0000", 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.941611784000003, 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.307398383, 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.939551+0000", 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.939551+0000", 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.939552+0000", 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9399999999999995e-07 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.939555+0000", 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1370000000000002e-06 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.246752+0000", 2024-09-17T16:11:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30719660599999998 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.246759+0000", 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9600000000000003e-06 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.246816+0000", 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7367000000000001e-05 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.246826+0000", 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9550000000000004e-06 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.246950+0000", 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012386400000000001 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:14845 2.b 2:d4296263:::benchmark_data_smithi173_55121_object14844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-17T16:11:58.908 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:13.947946+0000", 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.933217270999997, 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31393229, 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14845 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.947946+0000", 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.947946+0000", 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.947947+0000", 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8700000000000004e-07 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.947965+0000", 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8244999999999999e-05 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.947976+0000", 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0890000000000001e-05 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.947982+0000", 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7169999999999996e-06 2024-09-17T16:11:58.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.948030+0000", 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8761000000000003e-05 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.948148+0000", 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011716400000000001 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:11:58.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.948287+0000", 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139846 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.948795+0000", 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00050794900000000005 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.949064+0000", 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026895599999999999 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.261786+0000", 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31272181599999999 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.261838+0000", 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1405e-05 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.261878+0000", 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0654000000000001e-05 2024-09-17T16:11:58.914 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:20807 2.5 e116/104 ondisk, result = 0)", 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:25.330017+0000", 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 33.551145849999997, 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32339899300000002, 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.330017+0000", 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.330017+0000", 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.330018+0000", 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4000000000000001e-07 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.330022+0000", 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6009999999999999e-06 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.653242+0000", 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32321997099999999 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.653247+0000", 2024-09-17T16:11:58.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3149999999999999e-06 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.653327+0000", 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9862000000000002e-05 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.653334+0000", 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8009999999999998e-06 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.653416+0000", 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2803e-05 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24380 2.f 2:ff1cebfd:::benchmark_data_smithi173_55121_object24379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.697276+0000", 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.183887214999999, 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39030066800000002, 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.919 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24380 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697276+0000", 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697276+0000", 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697277+0000", 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.13e-06 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697302+0000", 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4533999999999999e-05 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697316+0000", 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4592e-05 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697324+0000", 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9109999999999993e-06 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697380+0000", 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6162999999999997e-05 2024-09-17T16:11:58.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697512+0000", 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013168899999999999 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697679+0000", 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016729599999999999 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.698584+0000", 2024-09-17T16:11:58.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090433600000000005 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.762716+0000", 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.064132583000000007 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087510+0000", 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32479419700000001 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087549+0000", 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8278999999999997e-05 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087577+0000", 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7957999999999999e-05 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24385 2.b 2:d672fe3e:::benchmark_data_smithi173_55121_object24384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:11:58.925 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.704153+0000", 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.177010052, 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53404779700000005, 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24385 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704153+0000", 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704153+0000", 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704154+0000", 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704177+0000", 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2127e-05 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704189+0000", 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2241e-05 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704195+0000", 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4280000000000001e-06 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704239+0000", 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3775e-05 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704326+0000", 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7198000000000001e-05 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704414+0000", 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7652999999999996e-05 2024-09-17T16:11:58.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705316+0000", 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090249000000000004 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705635+0000", 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031902200000000002 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.238124+0000", 2024-09-17T16:11:58.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53248838600000004 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.238158+0000", 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4789000000000002e-05 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.238201+0000", 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2468000000000001e-05 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.931 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24386 2.10 2:0afcc277:::benchmark_data_smithi173_55121_object24385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.705747+0000", 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.175416003999999, 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43386687200000001, 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.932 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24386 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705747+0000", 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705747+0000", 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705748+0000", 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5099999999999999e-07 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705770+0000", 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1563999999999999e-05 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705779+0000", 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3249999999999997e-06 2024-09-17T16:11:58.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705784+0000", 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.203e-06 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705832+0000", 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8180000000000003e-05 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705930+0000", 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7617999999999999e-05 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706020+0000", 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9890999999999997e-05 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706544+0000", 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00052466899999999996 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706882+0000", 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000337833 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139550+0000", 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43266755299999998 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139585+0000", 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5136999999999999e-05 2024-09-17T16:11:58.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139614+0000", 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9147999999999999e-05 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24388 2.f 2:f7f1158d:::benchmark_data_smithi173_55121_object24387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.761151+0000", 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.120012218999999, 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37970383499999999, 2024-09-17T16:11:58.938 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24388 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761151+0000", 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761151+0000", 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761152+0000", 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4619999999999999e-06 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761176+0000", 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3319000000000002e-05 2024-09-17T16:11:58.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761190+0000", 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3916e-05 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761198+0000", 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1280000000000008e-06 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761269+0000", 2024-09-17T16:11:58.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0862e-05 2024-09-17T16:11:58.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761423+0000", 2024-09-17T16:11:58.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015397499999999999 2024-09-17T16:11:58.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761596+0000", 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017338799999999999 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.762762+0000", 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001166325 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767453+0000", 2024-09-17T16:11:58.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046909999999999999 2024-09-17T16:11:58.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.140795+0000", 2024-09-17T16:11:58.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37334170999999999 2024-09-17T16:11:58.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.140830+0000", 2024-09-17T16:11:58.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4625999999999997e-05 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.140855+0000", 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5123999999999999e-05 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:24380 2.f e120/95 ondisk, result = 0)", 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.762985+0000", 2024-09-17T16:11:58.946 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.118178291, 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32459677300000001, 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.762985+0000", 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.762985+0000", 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.762986+0000", 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8400000000000004e-07 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.762989+0000", 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4860000000000002e-06 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087450+0000", 2024-09-17T16:11:58.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32446118600000001 2024-09-17T16:11:58.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087457+0000", 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.686e-06 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087502+0000", 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5040000000000002e-05 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087507+0000", 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2460000000000003e-06 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087582+0000", 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4544999999999999e-05 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24391 2.5 2:a6a580fc:::benchmark_data_smithi173_55121_object24390:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.766124+0000", 2024-09-17T16:11:58.951 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.115039427999999, 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36991581699999998, 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24391 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766124+0000", 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766124+0000", 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766125+0000", 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6300000000000004e-07 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766146+0000", 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1365e-05 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766157+0000", 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1294000000000001e-05 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766163+0000", 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3170000000000001e-06 2024-09-17T16:11:58.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766215+0000", 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2367999999999997e-05 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766306+0000", 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0994000000000003e-05 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766395+0000", 2024-09-17T16:11:58.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9352999999999996e-05 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766965+0000", 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056975199999999998 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767311+0000", 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000345621 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135856+0000", 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36854523 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135970+0000", 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114548 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.136040+0000", 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9111999999999998e-05 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.957 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:24391 2.5 e120/104 ondisk, result = 0)", 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.113770565999999, 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36866083100000002, 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5199999999999996e-07 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767397+0000", 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.27e-06 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135721+0000", 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36832436600000001 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135734+0000", 2024-09-17T16:11:58.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.323e-05 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135830+0000", 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5587999999999994e-05 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135843+0000", 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3699999999999999e-05 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.136053+0000", 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000210025 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25180 2.10 2:092fb28c:::benchmark_data_smithi173_55121_object25179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.740770+0000", 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.140393184000001, 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48461674700000001, 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.962 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25180 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740770+0000", 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740770+0000", 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740772+0000", 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5969999999999999e-06 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740793+0000", 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1379999999999999e-05 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740807+0000", 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4222e-05 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740814+0000", 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7830000000000001e-06 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740877+0000", 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2996999999999993e-05 2024-09-17T16:11:58.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740984+0000", 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106656 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.741069+0000", 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5752000000000002e-05 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.741666+0000", 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059629500000000005 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.742062+0000", 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000396301 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225328+0000", 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48326640700000001 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225362+0000", 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3317e-05 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225387+0000", 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5040000000000001e-05 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25215 2.10 2:0e4d5290:::benchmark_data_smithi173_55121_object25214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:11:58.968 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.775331+0000", 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.105832469999999, 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47406391599999997, 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25215 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775331+0000", 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775331+0000", 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775332+0000", 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9300000000000006e-07 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775355+0000", 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3303e-05 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775369+0000", 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3896999999999999e-05 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775377+0000", 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1699999999999997e-06 2024-09-17T16:11:58.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775446+0000", 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8745000000000007e-05 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775561+0000", 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115616 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775705+0000", 2024-09-17T16:11:58.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000143151 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776312+0000", 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00060773299999999995 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776660+0000", 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034724700000000002 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.249329+0000", 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47266967799999998 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.249367+0000", 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7233000000000001e-05 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.249395+0000", 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.815e-05 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:25180 2.10 e121/111 ondisk, result = 0)", 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.104690699999999, 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44891908400000002, 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3200000000000004e-07 2024-09-17T16:11:58.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776477+0000", 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6660000000000001e-06 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225282+0000", 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44880505700000001 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225287+0000", 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8659999999999997e-06 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225320+0000", 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3055000000000001e-05 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225324+0000", 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3599999999999998e-06 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225392+0000", 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7348000000000006e-05 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25237 2.5 2:a0e164cf:::benchmark_data_smithi173_55121_object25236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.800457+0000", 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.080706482, 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33304292899999999, 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.979 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25237 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800457+0000", 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800457+0000", 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800458+0000", 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800475+0000", 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7628000000000001e-05 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800486+0000", 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0669999999999999e-05 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800491+0000", 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.22e-06 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800536+0000", 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4712000000000002e-05 2024-09-17T16:11:58.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800616+0000", 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0365999999999996e-05 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800697+0000", 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0935999999999999e-05 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.801225+0000", 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000528128 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.801667+0000", 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044215299999999998 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133437+0000", 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33176945000000002 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133473+0000", 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6189999999999997e-05 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133500+0000", 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6701999999999999e-05 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25244 2.5 2:a10ca3e5:::benchmark_data_smithi173_55121_object25243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:11:58.985 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.806993+0000", 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.074170502000001, 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32972473499999999, 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25244 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.806993+0000", 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.806993+0000", 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.806994+0000", 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.105e-06 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807013+0000", 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9669000000000001e-05 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807025+0000", 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1605e-05 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807031+0000", 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7950000000000004e-06 2024-09-17T16:11:58.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807078+0000", 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7205e-05 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807163+0000", 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5242999999999997e-05 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807247+0000", 2024-09-17T16:11:58.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3404000000000003e-05 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807840+0000", 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059313499999999995 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808379+0000", 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053952500000000001 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:58.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.136661+0000", 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32828145800000003 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.136693+0000", 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2551000000000001e-05 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.136717+0000", 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.404e-05 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:25237 2.5 e121/104 ondisk, result = 0)", 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.808152+0000", 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.073011492999999, 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32535406, 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808152+0000", 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808152+0000", 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808152+0000", 2024-09-17T16:11:58.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.92e-07 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808157+0000", 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1450000000000001e-06 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133382+0000", 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32522591099999998 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133387+0000", 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7809999999999996e-06 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133428+0000", 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1057999999999999e-05 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133433+0000", 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1780000000000002e-06 2024-09-17T16:11:58.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133506+0000", 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2294999999999998e-05 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:26993 2.10 2:0b00f2f4:::benchmark_data_smithi173_55121_object26992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.983151+0000", 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 25.898011786000001, 2024-09-17T16:11:58.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51500823299999998, 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 26993 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983151+0000", 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983151+0000", 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983153+0000", 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.111e-06 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983170+0000", 2024-09-17T16:11:58.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7805e-05 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983182+0000", 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1391e-05 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983187+0000", 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4809999999999997e-06 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:11:58.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983229+0000", 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1817999999999998e-05 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983316+0000", 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6978999999999995e-05 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983399+0000", 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2720999999999996e-05 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:59.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983972+0000", 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057330599999999999 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.984444+0000", 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047245000000000002 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.498086+0000", 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51364169000000004 2024-09-17T16:11:59.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.498123+0000", 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6358e-05 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.498160+0000", 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7122999999999997e-05 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:11:59.002 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:11:59.003 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:11:59.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:11:59.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:59.025 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:11:59.126 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:59.149 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:59.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:11:59.164 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:11:59.213 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 39006 38990 42.7478 74.3125 0.00213438 0.0233859 2024-09-17T16:11:59.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:11:59.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:59.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:59.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:11:59.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:59 smithi017 ceph-mon[27571]: 2.14 scrub starts 2024-09-17T16:11:59.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:59 smithi017 ceph-mon[27571]: osdmap e135: 12 total, 11 up, 12 in 2024-09-17T16:11:59.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:59.411 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:59.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:59.513 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:11:59.541 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:59.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:11:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:59 smithi071 ceph-mon[30002]: 2.14 scrub starts 2024-09-17T16:11:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:59 smithi071 ceph-mon[30002]: osdmap e135: 12 total, 11 up, 12 in 2024-09-17T16:11:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:59 smithi071 ceph-mon[30002]: 2.a scrub starts 2024-09-17T16:11:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:59 smithi071 ceph-mon[30002]: 2.14 scrub ok 2024-09-17T16:11:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:59 smithi071 ceph-mon[30002]: 2.a scrub ok 2024-09-17T16:11:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:11:59 smithi071 ceph-mon[30002]: osdmap e136: 12 total, 11 up, 12 in 2024-09-17T16:11:59.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.7.service' 2024-09-17T16:11:59.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:11:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:59 smithi173 ceph-mon[30530]: 2.14 scrub starts 2024-09-17T16:11:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:59 smithi173 ceph-mon[30530]: osdmap e135: 12 total, 11 up, 12 in 2024-09-17T16:11:59.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:59 smithi173 ceph-mon[30530]: 2.a scrub starts 2024-09-17T16:11:59.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:59 smithi173 ceph-mon[30530]: 2.14 scrub ok 2024-09-17T16:11:59.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:59 smithi173 ceph-mon[30530]: 2.a scrub ok 2024-09-17T16:11:59.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:11:59 smithi173 ceph-mon[30530]: osdmap e136: 12 total, 11 up, 12 in 2024-09-17T16:11:59.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:11:59.663 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-17T16:11:59.664 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-09-17T16:11:59.664 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi071.front.sepia.ceph.com 2024-09-17T16:11:59.664 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.7 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-17T16:11:59.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:11:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:59.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:11:59.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:59 smithi017 ceph-mon[27571]: 2.a scrub starts 2024-09-17T16:11:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:59 smithi017 ceph-mon[27571]: 2.14 scrub ok 2024-09-17T16:11:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:59 smithi017 ceph-mon[27571]: 2.a scrub ok 2024-09-17T16:11:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:11:59 smithi017 ceph-mon[27571]: osdmap e136: 12 total, 11 up, 12 in 2024-09-17T16:11:59.769 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:11:59.793 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:11:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:59.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:11:59.878 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.7/config 2024-09-17T16:11:59.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:11:59.923 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:11:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:11:59.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:00.025 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:00.050 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:00.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:00.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:00.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:00.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:00.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 40567 40551 43.6927 97.5625 0.00255057 0.0228798 2024-09-17T16:12:00.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:00.303 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:00.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:00.406 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:00.429 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:00.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:00.513 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:12:00.531 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:00.554 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:00.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:00.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:00.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:00 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:12:00.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:00 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:12:00.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:00 smithi071 ceph-mon[30002]: pgmap v371: 27 pgs: 1 peering, 2 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+undersized+degraded, 21 active+clean; 2.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 340 op/s; 2475/113058 objects degraded (2.189%); 357/113058 objects misplaced (0.316%) 2024-09-17T16:12:00.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:12:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:00 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:12:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:00 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:12:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:00 smithi173 ceph-mon[30530]: pgmap v371: 27 pgs: 1 peering, 2 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+undersized+degraded, 21 active+clean; 2.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 340 op/s; 2475/113058 objects degraded (2.189%); 357/113058 objects misplaced (0.316%) 2024-09-17T16:12:00.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:00.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:00.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:00.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:00 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:12:00.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:00 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:12:00.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:00 smithi017 ceph-mon[27571]: pgmap v371: 27 pgs: 1 peering, 2 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+undersized+degraded, 21 active+clean; 2.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 340 op/s; 2475/113058 objects degraded (2.189%); 357/113058 objects misplaced (0.316%) 2024-09-17T16:12:00.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:00.843 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:00.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:00.851 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:00.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:00.969 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:00.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:01.044 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:12:01.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:01.100 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:01.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:01.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:12:01.201 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:01.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 42179 42163 44.6596 100.75 0.00640763 0.0223757 2024-09-17T16:12:01.222 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:01.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:01.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:01.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:01.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:01.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:01.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:01.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:01.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:01.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:01.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:01.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:01.634 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:01 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 2475/113058 objects degraded (2.189%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:01.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:01 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 2475/113058 objects degraded (2.189%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:01.679 INFO:teuthology.orchestra.run.smithi071.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:12:01.684 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:01 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 2475/113058 objects degraded (2.189%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:01.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:01.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:01.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:01.843 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:01.845 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:01.872 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:01.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:01.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:01.997 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:01.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:02.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:02.123 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:02.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:02.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:12:02.213222+0000 min lat: 0.00105621 max lat: 15.0115 avg lat: 0.0220805 2024-09-17T16:12:02.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:12:02.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 43448 43432 45.237 79.3125 0.054676 0.0220805 2024-09-17T16:12:02.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:02.245 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:02.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:02.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:02.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:02.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:02.387 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:12:02.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:02 smithi173 ceph-mon[30530]: 2.e scrub starts 2024-09-17T16:12:02.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:02 smithi173 ceph-mon[30530]: pgmap v372: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 7 active+undersized+degraded, 18 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 724 op/s; 10418/123339 objects degraded (8.447%); 236/123339 objects misplaced (0.191%) 2024-09-17T16:12:02.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4139480460' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:02.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:02.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:02.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:02.516 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:12:02.516 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:12:02.523 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:12:02.523 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:12:02.546 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:12:02.546 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:12:02.568 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:12:02.569 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:12:02.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:12:02.591 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:12:02.594 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:02.614 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:12:02.615 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:12:02.616 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:02.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:02.637 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:12:02.637 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:12:02.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:02 smithi017 ceph-mon[27571]: 2.e scrub starts 2024-09-17T16:12:02.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:02 smithi017 ceph-mon[27571]: pgmap v372: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 7 active+undersized+degraded, 18 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 724 op/s; 10418/123339 objects degraded (8.447%); 236/123339 objects misplaced (0.191%) 2024-09-17T16:12:02.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4139480460' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:02.638 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:12:02.700 INFO:teuthology.orchestra.run.smithi071.stdout:2.b 2024-09-17T16:12:02.700 INFO:teuthology.orchestra.run.smithi071.stdout:2.3 2024-09-17T16:12:02.700 INFO:teuthology.orchestra.run.smithi071.stdout:2.18 2024-09-17T16:12:02.700 INFO:teuthology.orchestra.run.smithi071.stdout:2.12 2024-09-17T16:12:02.700 INFO:teuthology.orchestra.run.smithi071.stdout:2.0 2024-09-17T16:12:02.700 INFO:teuthology.orchestra.run.smithi071.stdout:2.9 2024-09-17T16:12:02.701 INFO:teuthology.orchestra.run.smithi071.stdout:2.8 2024-09-17T16:12:02.701 INFO:teuthology.orchestra.run.smithi071.stdout:1.0 2024-09-17T16:12:02.701 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:02 smithi071 ceph-mon[30002]: 2.e scrub starts 2024-09-17T16:12:02.701 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:02 smithi071 ceph-mon[30002]: pgmap v372: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 7 active+undersized+degraded, 18 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 724 op/s; 10418/123339 objects degraded (8.447%); 236/123339 objects misplaced (0.191%) 2024-09-17T16:12:02.701 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4139480460' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:02.701 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:12:02.702 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:12:02.702 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:12:02.702 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:12:02.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:02.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:02.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:02.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:12:02.760 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:12:02.781 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:12:02.781 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:12:02.803 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:12:02.803 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:12:02.824 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:12:02.824 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:12:02.842 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:02.866 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:02.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:02.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:02.991 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:02.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:03.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:03.117 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:03.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:03.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:03.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 43595 43579 44.646 9.1875 0.00189418 0.0221508 2024-09-17T16:12:03.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:03.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:03.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:03.342 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:03.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:12:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:03.366 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:03.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:03.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:03.489 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:03.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:03.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:03.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:03.596 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:12:03.613 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:03.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:03.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:03 smithi173 ceph-mon[30530]: 2.e scrub ok 2024-09-17T16:12:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:03 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4139480460' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:12:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:03 smithi173 ceph-mon[30530]: osdmap e137: 12 total, 11 up, 12 in 2024-09-17T16:12:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/378533262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:03.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:03.736 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:03.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:03.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:03.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:03 smithi017 ceph-mon[27571]: 2.e scrub ok 2024-09-17T16:12:03.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:03 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:03.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4139480460' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:12:03.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:03 smithi017 ceph-mon[27571]: osdmap e137: 12 total, 11 up, 12 in 2024-09-17T16:12:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/378533262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:03.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:03 smithi071 ceph-mon[30002]: 2.e scrub ok 2024-09-17T16:12:03.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:03 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4139480460' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:12:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:03 smithi071 ceph-mon[30002]: osdmap e137: 12 total, 11 up, 12 in 2024-09-17T16:12:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/378533262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:03.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:03.861 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:03.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:03.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.7 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 2.8 --file /var/log/ceph/exp.2.8.7 2024-09-17T16:12:03.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:03.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:03.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:04.064 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:04.064 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:04.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:04.107 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:04.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:04.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:04.133 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.7/config 2024-09-17T16:12:04.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:04.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 43595 43579 43.9259 0 - 0.0221508 2024-09-17T16:12:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:12:04.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:12:04.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:04.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:04.333 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:04.357 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:04.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:04.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:04.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:04.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:04.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:04.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:04 smithi017 ceph-mon[27571]: osdmap e138: 12 total, 11 up, 12 in 2024-09-17T16:12:04.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:04 smithi017 ceph-mon[27571]: pgmap v375: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 7 active+undersized+degraded, 18 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 472 op/s; 10418/123339 objects degraded (8.447%); 236/123339 objects misplaced (0.191%) 2024-09-17T16:12:04.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:04 smithi071 ceph-mon[30002]: osdmap e138: 12 total, 11 up, 12 in 2024-09-17T16:12:04.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:04 smithi071 ceph-mon[30002]: pgmap v375: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 7 active+undersized+degraded, 18 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 472 op/s; 10418/123339 objects degraded (8.447%); 236/123339 objects misplaced (0.191%) 2024-09-17T16:12:04.582 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:04.607 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:04.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:04.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:04 smithi173 ceph-mon[30530]: osdmap e138: 12 total, 11 up, 12 in 2024-09-17T16:12:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:04 smithi173 ceph-mon[30530]: pgmap v375: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 7 active+undersized+degraded, 18 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 472 op/s; 10418/123339 objects degraded (8.447%); 236/123339 objects misplaced (0.191%) 2024-09-17T16:12:04.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:04.735 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:04.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:04.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:12:04.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:04.860 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:04.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:04.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:04.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:04.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:04.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:05.042 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:12:05.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:05.112 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:05.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:05.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:05.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 43595 43579 43.2286 0 - 0.0221508 2024-09-17T16:12:05.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:05.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:05.258 INFO:teuthology.orchestra.run.smithi071.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:12:05.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:05.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:05.361 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:05.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:12:05.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:05.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:05.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:05.517 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:05 smithi071 ceph-mon[30002]: osdmap e139: 12 total, 11 up, 12 in 2024-09-17T16:12:05.585 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:05.608 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:05.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:05.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:05.711 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:05.714 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:05 smithi173 ceph-mon[30530]: osdmap e139: 12 total, 11 up, 12 in 2024-09-17T16:12:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:05.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:05.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:05.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:05.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:05 smithi017 ceph-mon[27571]: osdmap e139: 12 total, 11 up, 12 in 2024-09-17T16:12:05.836 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:05.859 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:05.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:05.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:05.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:05.983 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:05.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:06.014 INFO:teuthology.orchestra.run.smithi071.stderr:Exporting 2.8 info 2.8( v 132'1644 (0'0,132'1644] local-lis/les=104/105 n=1214 ec=95/95 lis/c=104/104 les/c/f=105/105/0 sis=104) 2024-09-17T16:12:06.019 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:100174b5:::benchmark_data_smithi173_55121_object15916:head# 2024-09-17T16:12:06.020 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:100345db:::benchmark_data_smithi173_55121_object3902:head# 2024-09-17T16:12:06.020 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1009c2d5:::benchmark_data_smithi173_55121_object1579:head# 2024-09-17T16:12:06.021 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:100e262e:::benchmark_data_smithi173_55121_object21909:head# 2024-09-17T16:12:06.021 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:100f668e:::benchmark_data_smithi173_55121_object16593:head# 2024-09-17T16:12:06.022 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:101006a5:::benchmark_data_smithi173_55121_object2557:head# 2024-09-17T16:12:06.022 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10135051:::benchmark_data_smithi173_55121_object29933:head# 2024-09-17T16:12:06.023 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1013bc35:::benchmark_data_smithi173_55121_object33968:head# 2024-09-17T16:12:06.024 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10187720:::benchmark_data_smithi173_55121_object18603:head# 2024-09-17T16:12:06.024 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1018cab4:::benchmark_data_smithi173_55121_object22796:head# 2024-09-17T16:12:06.025 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:101b15df:::benchmark_data_smithi173_55121_object8581:head# 2024-09-17T16:12:06.025 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:101c462c:::benchmark_data_smithi173_55121_object999:head# 2024-09-17T16:12:06.025 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:101f2b2f:::benchmark_data_smithi173_55121_object10255:head# 2024-09-17T16:12:06.026 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1020b299:::benchmark_data_smithi173_55121_object34977:head# 2024-09-17T16:12:06.026 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:102381fe:::benchmark_data_smithi173_55121_object32823:head# 2024-09-17T16:12:06.027 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10276a83:::benchmark_data_smithi173_55121_object27040:head# 2024-09-17T16:12:06.027 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:102827da:::benchmark_data_smithi173_55121_object18772:head# 2024-09-17T16:12:06.028 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1028e588:::benchmark_data_smithi173_55121_object14770:head# 2024-09-17T16:12:06.028 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1029ddce:::benchmark_data_smithi173_55121_object5168:head# 2024-09-17T16:12:06.029 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:102a904b:::benchmark_data_smithi173_55121_object13433:head# 2024-09-17T16:12:06.029 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:102bd898:::benchmark_data_smithi173_55121_object29440:head# 2024-09-17T16:12:06.030 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:102d22aa:::benchmark_data_smithi173_55121_object15480:head# 2024-09-17T16:12:06.030 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:102d68eb:::benchmark_data_smithi173_55121_object27413:head# 2024-09-17T16:12:06.030 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:102f9859:::benchmark_data_smithi173_55121_object1385:head# 2024-09-17T16:12:06.031 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1032c5a6:::benchmark_data_smithi173_55121_object18866:head# 2024-09-17T16:12:06.031 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10330e94:::benchmark_data_smithi173_55121_object34648:head# 2024-09-17T16:12:06.032 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:103355cc:::benchmark_data_smithi173_55121_object8338:head# 2024-09-17T16:12:06.033 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1033c650:::benchmark_data_smithi173_55121_object23876:head# 2024-09-17T16:12:06.033 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1036ca55:::benchmark_data_smithi173_55121_object21843:head# 2024-09-17T16:12:06.033 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10386a84:::benchmark_data_smithi173_55121_object13150:head# 2024-09-17T16:12:06.034 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:103939a1:::benchmark_data_smithi173_55121_object27128:head# 2024-09-17T16:12:06.034 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:103a6f6a:::benchmark_data_smithi173_55121_object246:head# 2024-09-17T16:12:06.035 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:103b4fac:::benchmark_data_smithi173_55121_object25903:head# 2024-09-17T16:12:06.035 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:103cfd21:::benchmark_data_smithi173_55121_object29452:head# 2024-09-17T16:12:06.036 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10433941:::benchmark_data_smithi173_55121_object18851:head# 2024-09-17T16:12:06.036 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10433b8f:::benchmark_data_smithi173_55121_object12771:head# 2024-09-17T16:12:06.036 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:104438a4:::benchmark_data_smithi173_55121_object34557:head# 2024-09-17T16:12:06.037 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1045dd7b:::benchmark_data_smithi173_55121_object15118:head# 2024-09-17T16:12:06.037 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10475a66:::benchmark_data_smithi173_55121_object23692:head# 2024-09-17T16:12:06.038 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10482331:::benchmark_data_smithi173_55121_object21034:head# 2024-09-17T16:12:06.038 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10499626:::benchmark_data_smithi173_55121_object36327:head# 2024-09-17T16:12:06.039 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:104a9d47:::benchmark_data_smithi173_55121_object1567:head# 2024-09-17T16:12:06.039 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:104e3319:::benchmark_data_smithi173_55121_object12608:head# 2024-09-17T16:12:06.040 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:104e9add:::benchmark_data_smithi173_55121_object3895:head# 2024-09-17T16:12:06.040 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:104f1d12:::benchmark_data_smithi173_55121_object37261:head# 2024-09-17T16:12:06.041 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:104f7aaf:::benchmark_data_smithi173_55121_object962:head# 2024-09-17T16:12:06.041 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1052da9a:::benchmark_data_smithi173_55121_object27236:head# 2024-09-17T16:12:06.042 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10536e41:::benchmark_data_smithi173_55121_object27047:head# 2024-09-17T16:12:06.042 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10574661:::benchmark_data_smithi173_55121_object24753:head# 2024-09-17T16:12:06.043 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1058712e:::benchmark_data_smithi173_55121_object33443:head# 2024-09-17T16:12:06.043 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1058d865:::benchmark_data_smithi173_55121_object15559:head# 2024-09-17T16:12:06.044 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10592cf8:::benchmark_data_smithi173_55121_object12541:head# 2024-09-17T16:12:06.044 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:105c46a3:::benchmark_data_smithi173_55121_object4596:head# 2024-09-17T16:12:06.045 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:105d7009:::benchmark_data_smithi173_55121_object21328:head# 2024-09-17T16:12:06.045 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:105ec88d:::benchmark_data_smithi173_55121_object17726:head# 2024-09-17T16:12:06.046 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1062e034:::benchmark_data_smithi173_55121_object9394:head# 2024-09-17T16:12:06.046 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10641db1:::benchmark_data_smithi173_55121_object35008:head# 2024-09-17T16:12:06.047 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:106829a2:::benchmark_data_smithi173_55121_object35628:head# 2024-09-17T16:12:06.047 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10694afe:::benchmark_data_smithi173_55121_object8209:head# 2024-09-17T16:12:06.048 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:106f1b2b:::benchmark_data_smithi173_55121_object15921:head# 2024-09-17T16:12:06.048 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1070a2ef:::benchmark_data_smithi173_55121_object35247:head# 2024-09-17T16:12:06.049 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10712393:::benchmark_data_smithi173_55121_object36392:head# 2024-09-17T16:12:06.049 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10716a42:::benchmark_data_smithi173_55121_object3080:head# 2024-09-17T16:12:06.050 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:107320a5:::benchmark_data_smithi173_55121_object17709:head# 2024-09-17T16:12:06.050 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1075919d:::benchmark_data_smithi173_55121_object20857:head# 2024-09-17T16:12:06.051 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10766d69:::benchmark_data_smithi173_55121_object3331:head# 2024-09-17T16:12:06.051 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1079709b:::benchmark_data_smithi173_55121_object8582:head# 2024-09-17T16:12:06.052 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:107cdf5e:::benchmark_data_smithi173_55121_object33521:head# 2024-09-17T16:12:06.052 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10802558:::benchmark_data_smithi173_55121_object13982:head# 2024-09-17T16:12:06.052 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10829281:::benchmark_data_smithi173_55121_object30466:head# 2024-09-17T16:12:06.053 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:108401c3:::benchmark_data_smithi173_55121_object28384:head# 2024-09-17T16:12:06.053 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10861d74:::benchmark_data_smithi173_55121_object11300:head# 2024-09-17T16:12:06.054 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10864766:::benchmark_data_smithi173_55121_object13974:head# 2024-09-17T16:12:06.054 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:108bf821:::benchmark_data_smithi173_55121_object21667:head# 2024-09-17T16:12:06.055 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:108cb638:::benchmark_data_smithi173_55121_object13724:head# 2024-09-17T16:12:06.055 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:108e8052:::benchmark_data_smithi173_55121_object16785:head# 2024-09-17T16:12:06.056 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10923467:::benchmark_data_smithi173_55121_object34002:head# 2024-09-17T16:12:06.056 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1093652f:::benchmark_data_smithi173_55121_object4978:head# 2024-09-17T16:12:06.057 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1094af92:::benchmark_data_smithi173_55121_object10826:head# 2024-09-17T16:12:06.057 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1094d4dc:::benchmark_data_smithi173_55121_object22767:head# 2024-09-17T16:12:06.057 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1098a9eb:::benchmark_data_smithi173_55121_object9487:head# 2024-09-17T16:12:06.058 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:109b0c05:::benchmark_data_smithi173_55121_object7177:head# 2024-09-17T16:12:06.058 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:109c1f52:::benchmark_data_smithi173_55121_object29035:head# 2024-09-17T16:12:06.059 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:109caa6d:::benchmark_data_smithi173_55121_object7775:head# 2024-09-17T16:12:06.059 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:109ceb7b:::benchmark_data_smithi173_55121_object18972:head# 2024-09-17T16:12:06.060 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:109dc192:::benchmark_data_smithi173_55121_object8038:head# 2024-09-17T16:12:06.060 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10a457bd:::benchmark_data_smithi173_55121_object16419:head# 2024-09-17T16:12:06.061 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10a6bfa9:::benchmark_data_smithi173_55121_object22211:head# 2024-09-17T16:12:06.061 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10a8c544:::benchmark_data_smithi173_55121_object16437:head# 2024-09-17T16:12:06.062 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10ab1d31:::benchmark_data_smithi173_55121_object25070:head# 2024-09-17T16:12:06.062 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10ace813:::benchmark_data_smithi173_55121_object12155:head# 2024-09-17T16:12:06.062 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10ad1087:::benchmark_data_smithi173_55121_object29464:head# 2024-09-17T16:12:06.063 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10b05b30:::benchmark_data_smithi173_55121_object12875:head# 2024-09-17T16:12:06.063 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10b2ce18:::benchmark_data_smithi173_55121_object15110:head# 2024-09-17T16:12:06.064 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10b3314d:::benchmark_data_smithi173_55121_object10306:head# 2024-09-17T16:12:06.065 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10b40ebf:::benchmark_data_smithi173_55121_object29623:head# 2024-09-17T16:12:06.065 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10bbbe5c:::benchmark_data_smithi173_55121_object11897:head# 2024-09-17T16:12:06.066 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10c12933:::benchmark_data_smithi173_55121_object5390:head# 2024-09-17T16:12:06.066 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10c1a550:::benchmark_data_smithi173_55121_object16956:head# 2024-09-17T16:12:06.066 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10c209dc:::benchmark_data_smithi173_55121_object15680:head# 2024-09-17T16:12:06.067 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10c21785:::benchmark_data_smithi173_55121_object27310:head# 2024-09-17T16:12:06.067 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10c260e6:::benchmark_data_smithi173_55121_object24149:head# 2024-09-17T16:12:06.068 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10c378d6:::benchmark_data_smithi173_55121_object19927:head# 2024-09-17T16:12:06.068 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10c4f66f:::benchmark_data_smithi173_55121_object7715:head# 2024-09-17T16:12:06.069 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10c744c2:::benchmark_data_smithi173_55121_object25859:head# 2024-09-17T16:12:06.069 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10cc1c52:::benchmark_data_smithi173_55121_object32050:head# 2024-09-17T16:12:06.069 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10d00465:::benchmark_data_smithi173_55121_object1057:head# 2024-09-17T16:12:06.070 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10d390ea:::benchmark_data_smithi173_55121_object2914:head# 2024-09-17T16:12:06.070 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10d408b8:::benchmark_data_smithi173_55121_object11813:head# 2024-09-17T16:12:06.070 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10d4d99b:::benchmark_data_smithi173_55121_object29445:head# 2024-09-17T16:12:06.071 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10d52c61:::benchmark_data_smithi173_55121_object30930:head# 2024-09-17T16:12:06.072 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10d6b055:::benchmark_data_smithi173_55121_object24575:head# 2024-09-17T16:12:06.072 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10d84cd1:::benchmark_data_smithi173_55121_object19440:head# 2024-09-17T16:12:06.072 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10da1907:::benchmark_data_smithi173_55121_object13176:head# 2024-09-17T16:12:06.073 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10daa35d:::benchmark_data_smithi173_55121_object1601:head# 2024-09-17T16:12:06.073 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10df007b:::benchmark_data_smithi173_55121_object16331:head# 2024-09-17T16:12:06.074 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10df03bd:::benchmark_data_smithi173_55121_object22462:head# 2024-09-17T16:12:06.074 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10df1dbd:::benchmark_data_smithi173_55121_object14043:head# 2024-09-17T16:12:06.075 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10e0e2f3:::benchmark_data_smithi173_55121_object36713:head# 2024-09-17T16:12:06.075 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10e2d356:::benchmark_data_smithi173_55121_object32082:head# 2024-09-17T16:12:06.076 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10e3c86c:::benchmark_data_smithi173_55121_object9749:head# 2024-09-17T16:12:06.076 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10e47ad2:::benchmark_data_smithi173_55121_object4697:head# 2024-09-17T16:12:06.077 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10e759a0:::benchmark_data_smithi173_55121_object11574:head# 2024-09-17T16:12:06.077 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10e89c34:::benchmark_data_smithi173_55121_object23636:head# 2024-09-17T16:12:06.078 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10e8f986:::benchmark_data_smithi173_55121_object6106:head# 2024-09-17T16:12:06.078 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10ea2670:::benchmark_data_smithi173_55121_object15688:head# 2024-09-17T16:12:06.079 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10ec9938:::benchmark_data_smithi173_55121_object25605:head# 2024-09-17T16:12:06.079 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10eec2ca:::benchmark_data_smithi173_55121_object28522:head# 2024-09-17T16:12:06.079 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10f21596:::benchmark_data_smithi173_55121_object27830:head# 2024-09-17T16:12:06.080 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10f641ef:::benchmark_data_smithi173_55121_object25363:head# 2024-09-17T16:12:06.080 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10f7444f:::benchmark_data_smithi173_55121_object3349:head# 2024-09-17T16:12:06.081 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10f97f0d:::benchmark_data_smithi173_55121_object32103:head# 2024-09-17T16:12:06.081 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10f9ef05:::benchmark_data_smithi173_55121_object21520:head# 2024-09-17T16:12:06.082 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10fa22a7:::benchmark_data_smithi173_55121_object36197:head# 2024-09-17T16:12:06.082 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10fa99d6:::benchmark_data_smithi173_55121_object24653:head# 2024-09-17T16:12:06.083 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10fae88a:::benchmark_data_smithi173_55121_object15660:head# 2024-09-17T16:12:06.084 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10fb345f:::benchmark_data_smithi173_55121_object21739:head# 2024-09-17T16:12:06.084 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10fc8a2e:::benchmark_data_smithi173_55121_object31295:head# 2024-09-17T16:12:06.085 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:10fccbc0:::benchmark_data_smithi173_55121_object7373:head# 2024-09-17T16:12:06.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:06.086 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1100f24f:::benchmark_data_smithi173_55121_object9990:head# 2024-09-17T16:12:06.086 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11050221:::benchmark_data_smithi173_55121_object22559:head# 2024-09-17T16:12:06.086 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11060d59:::benchmark_data_smithi173_55121_object16996:head# 2024-09-17T16:12:06.087 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11078ef1:::benchmark_data_smithi173_55121_object25927:head# 2024-09-17T16:12:06.087 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1107c42a:::benchmark_data_smithi173_55121_object33052:head# 2024-09-17T16:12:06.088 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1108567e:::benchmark_data_smithi173_55121_object22290:head# 2024-09-17T16:12:06.088 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11093ca2:::benchmark_data_smithi173_55121_object14218:head# 2024-09-17T16:12:06.089 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:110f4ff1:::benchmark_data_smithi173_55121_object30909:head# 2024-09-17T16:12:06.089 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1111a0b5:::benchmark_data_smithi173_55121_object21142:head# 2024-09-17T16:12:06.090 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:111767b2:::benchmark_data_smithi173_55121_object15003:head# 2024-09-17T16:12:06.090 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:111986ca:::benchmark_data_smithi173_55121_object7837:head# 2024-09-17T16:12:06.090 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:111b0465:::benchmark_data_smithi173_55121_object25025:head# 2024-09-17T16:12:06.091 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:111e1035:::benchmark_data_smithi173_55121_object31349:head# 2024-09-17T16:12:06.091 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11218576:::benchmark_data_smithi173_55121_object12312:head# 2024-09-17T16:12:06.091 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1125aaa5:::benchmark_data_smithi173_55121_object30015:head# 2024-09-17T16:12:06.092 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1127da0c:::benchmark_data_smithi173_55121_object23374:head# 2024-09-17T16:12:06.092 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11288b6b:::benchmark_data_smithi173_55121_object19554:head# 2024-09-17T16:12:06.093 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1128e2e4:::benchmark_data_smithi173_55121_object33347:head# 2024-09-17T16:12:06.093 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:112a4083:::benchmark_data_smithi173_55121_object14336:head# 2024-09-17T16:12:06.093 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:112af029:::benchmark_data_smithi173_55121_object25165:head# 2024-09-17T16:12:06.094 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:112b76e5:::benchmark_data_smithi173_55121_object30447:head# 2024-09-17T16:12:06.094 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:112bb137:::benchmark_data_smithi173_55121_object2736:head# 2024-09-17T16:12:06.095 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:112bdb5d:::benchmark_data_smithi173_55121_object19127:head# 2024-09-17T16:12:06.095 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:112dbc65:::benchmark_data_smithi173_55121_object30258:head# 2024-09-17T16:12:06.096 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11312814:::benchmark_data_smithi173_55121_object20801:head# 2024-09-17T16:12:06.097 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11331096:::benchmark_data_smithi173_55121_object31657:head# 2024-09-17T16:12:06.097 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1134af16:::benchmark_data_smithi173_55121_object17720:head# 2024-09-17T16:12:06.097 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1134b4a4:::benchmark_data_smithi173_55121_object12783:head# 2024-09-17T16:12:06.098 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1136105f:::benchmark_data_smithi173_55121_object36772:head# 2024-09-17T16:12:06.098 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1136297c:::benchmark_data_smithi173_55121_object7537:head# 2024-09-17T16:12:06.099 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:113799ba:::benchmark_data_smithi173_55121_object9343:head# 2024-09-17T16:12:06.100 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11380459:::benchmark_data_smithi173_55121_object17549:head# 2024-09-17T16:12:06.100 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:113877ce:::benchmark_data_smithi173_55121_object7128:head# 2024-09-17T16:12:06.101 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1139bfb1:::benchmark_data_smithi173_55121_object339:head# 2024-09-17T16:12:06.101 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1139df8b:::benchmark_data_smithi173_55121_object1638:head# 2024-09-17T16:12:06.101 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:113cbd9a:::benchmark_data_smithi173_55121_object34327:head# 2024-09-17T16:12:06.102 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:113efad0:::benchmark_data_smithi173_55121_object19658:head# 2024-09-17T16:12:06.102 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1140d68c:::benchmark_data_smithi173_55121_object2577:head# 2024-09-17T16:12:06.103 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1141b240:::benchmark_data_smithi173_55121_object20873:head# 2024-09-17T16:12:06.103 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:114216c9:::benchmark_data_smithi173_55121_object4356:head# 2024-09-17T16:12:06.104 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1143d6bb:::benchmark_data_smithi173_55121_object29373:head# 2024-09-17T16:12:06.104 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1144a22c:::benchmark_data_smithi173_55121_object28374:head# 2024-09-17T16:12:06.105 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1144ef4a:::benchmark_data_smithi173_55121_object644:head# 2024-09-17T16:12:06.105 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11457c24:::benchmark_data_smithi173_55121_object36485:head# 2024-09-17T16:12:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1146bec9:::benchmark_data_smithi173_55121_object22464:head# 2024-09-17T16:12:06.106 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:114eaf3a:::benchmark_data_smithi173_55121_object14595:head# 2024-09-17T16:12:06.107 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:114edc28:::benchmark_data_smithi173_55121_object6768:head# 2024-09-17T16:12:06.107 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:114f726b:::benchmark_data_smithi173_55121_object30533:head# 2024-09-17T16:12:06.107 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:114fe870:::benchmark_data_smithi173_55121_object28364:head# 2024-09-17T16:12:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:06.109 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11533f3e:::benchmark_data_smithi173_55121_object22551:head# 2024-09-17T16:12:06.109 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11540543:::benchmark_data_smithi173_55121_object21928:head# 2024-09-17T16:12:06.110 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:115459c4:::benchmark_data_smithi173_55121_object10849:head# 2024-09-17T16:12:06.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:06.110 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11545d29:::benchmark_data_smithi173_55121_object26980:head# 2024-09-17T16:12:06.110 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1155ca6c:::benchmark_data_smithi173_55121_object25561:head# 2024-09-17T16:12:06.111 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11572e57:::benchmark_data_smithi173_55121_object10873:head# 2024-09-17T16:12:06.111 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:115a687b:::benchmark_data_smithi173_55121_object4205:head# 2024-09-17T16:12:06.112 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11602637:::benchmark_data_smithi173_55121_object24463:head# 2024-09-17T16:12:06.112 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11607be0:::benchmark_data_smithi173_55121_object26655:head# 2024-09-17T16:12:06.113 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11616751:::benchmark_data_smithi173_55121_object17516:head# 2024-09-17T16:12:06.113 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11619824:::benchmark_data_smithi173_55121_object19710:head# 2024-09-17T16:12:06.114 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:116bab10:::benchmark_data_smithi173_55121_object1646:head# 2024-09-17T16:12:06.114 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:116c65ee:::benchmark_data_smithi173_55121_object7241:head# 2024-09-17T16:12:06.115 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:116cdce5:::benchmark_data_smithi173_55121_object10522:head# 2024-09-17T16:12:06.116 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:116d1a41:::benchmark_data_smithi173_55121_object32407:head# 2024-09-17T16:12:06.116 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:116f4a47:::benchmark_data_smithi173_55121_object6801:head# 2024-09-17T16:12:06.116 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:116f9105:::benchmark_data_smithi173_55121_object21174:head# 2024-09-17T16:12:06.116 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11703d25:::benchmark_data_smithi173_55121_object31365:head# 2024-09-17T16:12:06.117 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11707bab:::benchmark_data_smithi173_55121_object4752:head# 2024-09-17T16:12:06.117 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11715783:::benchmark_data_smithi173_55121_object13608:head# 2024-09-17T16:12:06.118 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1171659a:::benchmark_data_smithi173_55121_object21170:head# 2024-09-17T16:12:06.118 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1172408f:::benchmark_data_smithi173_55121_object937:head# 2024-09-17T16:12:06.119 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1175001e:::benchmark_data_smithi173_55121_object37247:head# 2024-09-17T16:12:06.119 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11788134:::benchmark_data_smithi173_55121_object16542:head# 2024-09-17T16:12:06.119 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:117918e4:::benchmark_data_smithi173_55121_object203:head# 2024-09-17T16:12:06.120 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1179cb8c:::benchmark_data_smithi173_55121_object25615:head# 2024-09-17T16:12:06.120 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:117acac1:::benchmark_data_smithi173_55121_object13515:head# 2024-09-17T16:12:06.121 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:117ddb01:::benchmark_data_smithi173_55121_object3037:head# 2024-09-17T16:12:06.121 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1180789e:::benchmark_data_smithi173_55121_object35760:head# 2024-09-17T16:12:06.122 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11840406:::benchmark_data_smithi173_55121_object12633:head# 2024-09-17T16:12:06.122 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11845e7e:::benchmark_data_smithi173_55121_object8796:head# 2024-09-17T16:12:06.123 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1185abc7:::benchmark_data_smithi173_55121_object19359:head# 2024-09-17T16:12:06.123 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1186eeb5:::benchmark_data_smithi173_55121_object4321:head# 2024-09-17T16:12:06.124 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:118c045a:::benchmark_data_smithi173_55121_object18368:head# 2024-09-17T16:12:06.124 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:118d7109:::benchmark_data_smithi173_55121_object26201:head# 2024-09-17T16:12:06.125 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:118dbf4b:::benchmark_data_smithi173_55121_object36662:head# 2024-09-17T16:12:06.125 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:118f4536:::benchmark_data_smithi173_55121_object31093:head# 2024-09-17T16:12:06.125 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11910c6c:::benchmark_data_smithi173_55121_object14324:head# 2024-09-17T16:12:06.126 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1191cf26:::benchmark_data_smithi173_55121_object14579:head# 2024-09-17T16:12:06.127 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11923514:::benchmark_data_smithi173_55121_object27659:head# 2024-09-17T16:12:06.127 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1193bd12:::benchmark_data_smithi173_55121_object32150:head# 2024-09-17T16:12:06.128 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1193f619:::benchmark_data_smithi173_55121_object29465:head# 2024-09-17T16:12:06.128 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:119854a9:::benchmark_data_smithi173_55121_object31527:head# 2024-09-17T16:12:06.129 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11994494:::benchmark_data_smithi173_55121_object2612:head# 2024-09-17T16:12:06.129 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:119959ce:::benchmark_data_smithi173_55121_object6249:head# 2024-09-17T16:12:06.129 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:119d436d:::benchmark_data_smithi173_55121_object15675:head# 2024-09-17T16:12:06.130 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a08d92:::benchmark_data_smithi173_55121_object37240:head# 2024-09-17T16:12:06.131 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a0de21:::benchmark_data_smithi173_55121_object16642:head# 2024-09-17T16:12:06.131 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a53155:::benchmark_data_smithi173_55121_object26096:head# 2024-09-17T16:12:06.131 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a5bc3b:::benchmark_data_smithi173_55121_object10535:head# 2024-09-17T16:12:06.132 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a6461f:::benchmark_data_smithi173_55121_object27802:head# 2024-09-17T16:12:06.132 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a6f74e:::benchmark_data_smithi173_55121_object1258:head# 2024-09-17T16:12:06.133 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a8013e:::benchmark_data_smithi173_55121_object9308:head# 2024-09-17T16:12:06.133 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a85dd4:::benchmark_data_smithi173_55121_object1255:head# 2024-09-17T16:12:06.134 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a8c3ff:::benchmark_data_smithi173_55121_object14007:head# 2024-09-17T16:12:06.134 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a9b00c:::benchmark_data_smithi173_55121_object27826:head# 2024-09-17T16:12:06.135 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11a9fb00:::benchmark_data_smithi173_55121_object22363:head# 2024-09-17T16:12:06.135 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11aa82a8:::benchmark_data_smithi173_55121_object20347:head# 2024-09-17T16:12:06.136 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11ab2df1:::benchmark_data_smithi173_55121_object32388:head# 2024-09-17T16:12:06.136 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11ab4bc9:::benchmark_data_smithi173_55121_object33544:head# 2024-09-17T16:12:06.136 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11ac8476:::benchmark_data_smithi173_55121_object28459:head# 2024-09-17T16:12:06.137 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11af3e15:::benchmark_data_smithi173_55121_object25568:head# 2024-09-17T16:12:06.137 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11affa20:::benchmark_data_smithi173_55121_object27427:head# 2024-09-17T16:12:06.138 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11b1e9d7:::benchmark_data_smithi173_55121_object36378:head# 2024-09-17T16:12:06.139 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11b3774f:::benchmark_data_smithi173_55121_object20054:head# 2024-09-17T16:12:06.139 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11b43939:::benchmark_data_smithi173_55121_object13559:head# 2024-09-17T16:12:06.139 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11b4cd74:::benchmark_data_smithi173_55121_object6971:head# 2024-09-17T16:12:06.140 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11b5f83b:::benchmark_data_smithi173_55121_object26075:head# 2024-09-17T16:12:06.140 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11b8e195:::benchmark_data_smithi173_55121_object12485:head# 2024-09-17T16:12:06.141 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11b90522:::benchmark_data_smithi173_55121_object20497:head# 2024-09-17T16:12:06.141 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11b92366:::benchmark_data_smithi173_55121_object11095:head# 2024-09-17T16:12:06.141 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11bd353b:::benchmark_data_smithi173_55121_object26106:head# 2024-09-17T16:12:06.142 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11beacd3:::benchmark_data_smithi173_55121_object4533:head# 2024-09-17T16:12:06.142 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11c413d9:::benchmark_data_smithi173_55121_object15472:head# 2024-09-17T16:12:06.143 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11c64b56:::benchmark_data_smithi173_55121_object6866:head# 2024-09-17T16:12:06.143 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11c65cdf:::benchmark_data_smithi173_55121_object3179:head# 2024-09-17T16:12:06.144 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11ca2a66:::benchmark_data_smithi173_55121_object2471:head# 2024-09-17T16:12:06.144 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11cb577c:::benchmark_data_smithi173_55121_object19537:head# 2024-09-17T16:12:06.145 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11cb94e6:::benchmark_data_smithi173_55121_object29570:head# 2024-09-17T16:12:06.146 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11ce4de8:::benchmark_data_smithi173_55121_object6336:head# 2024-09-17T16:12:06.146 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11cfba28:::benchmark_data_smithi173_55121_object6785:head# 2024-09-17T16:12:06.146 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11d15fbb:::benchmark_data_smithi173_55121_object25558:head# 2024-09-17T16:12:06.147 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11d16de7:::benchmark_data_smithi173_55121_object1060:head# 2024-09-17T16:12:06.148 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11d3803b:::benchmark_data_smithi173_55121_object8274:head# 2024-09-17T16:12:06.148 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11d41faf:::benchmark_data_smithi173_55121_object33587:head# 2024-09-17T16:12:06.148 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11d5203f:::benchmark_data_smithi173_55121_object26803:head# 2024-09-17T16:12:06.149 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11d6d7f7:::benchmark_data_smithi173_55121_object16562:head# 2024-09-17T16:12:06.149 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11d79770:::benchmark_data_smithi173_55121_object19433:head# 2024-09-17T16:12:06.150 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11d7e74f:::benchmark_data_smithi173_55121_object22168:head# 2024-09-17T16:12:06.150 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11daecfa:::benchmark_data_smithi173_55121_object7116:head# 2024-09-17T16:12:06.151 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11dd7adf:::benchmark_data_smithi173_55121_object29700:head# 2024-09-17T16:12:06.151 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11ddafe0:::benchmark_data_smithi173_55121_object1498:head# 2024-09-17T16:12:06.152 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11dfb669:::benchmark_data_smithi173_55121_object12378:head# 2024-09-17T16:12:06.152 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11e3317e:::benchmark_data_smithi173_55121_object3379:head# 2024-09-17T16:12:06.152 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11e4271e:::benchmark_data_smithi173_55121_object21569:head# 2024-09-17T16:12:06.153 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11e6838b:::benchmark_data_smithi173_55121_object7682:head# 2024-09-17T16:12:06.153 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11e748e7:::benchmark_data_smithi173_55121_object9977:head# 2024-09-17T16:12:06.154 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11e91841:::benchmark_data_smithi173_55121_object1152:head# 2024-09-17T16:12:06.154 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11edcfb3:::benchmark_data_smithi173_55121_object24553:head# 2024-09-17T16:12:06.155 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11f09469:::benchmark_data_smithi173_55121_object21536:head# 2024-09-17T16:12:06.155 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11f1a343:::benchmark_data_smithi173_55121_object5070:head# 2024-09-17T16:12:06.155 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11f47931:::benchmark_data_smithi173_55121_object19704:head# 2024-09-17T16:12:06.156 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11f7854e:::benchmark_data_smithi173_55121_object19351:head# 2024-09-17T16:12:06.157 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11f9ab91:::benchmark_data_smithi173_55121_object27973:head# 2024-09-17T16:12:06.157 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11fe8bcc:::benchmark_data_smithi173_55121_object26087:head# 2024-09-17T16:12:06.158 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:11ffbea2:::benchmark_data_smithi173_55121_object36511:head# 2024-09-17T16:12:06.158 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1202d30e:::benchmark_data_smithi173_55121_object19573:head# 2024-09-17T16:12:06.159 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:120439c0:::benchmark_data_smithi173_55121_object31936:head# 2024-09-17T16:12:06.159 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12059d30:::benchmark_data_smithi173_55121_object21811:head# 2024-09-17T16:12:06.160 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12082966:::benchmark_data_smithi173_55121_object24138:head# 2024-09-17T16:12:06.161 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1208e078:::benchmark_data_smithi173_55121_object12051:head# 2024-09-17T16:12:06.161 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:120983ff:::benchmark_data_smithi173_55121_object821:head# 2024-09-17T16:12:06.161 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:120dc368:::benchmark_data_smithi173_55121_object13018:head# 2024-09-17T16:12:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:120e19d4:::benchmark_data_smithi173_55121_object36605:head# 2024-09-17T16:12:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:120e376d:::benchmark_data_smithi173_55121_object18455:head# 2024-09-17T16:12:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:120ebb70:::benchmark_data_smithi173_55121_object31624:head# 2024-09-17T16:12:06.163 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1215d5c2:::benchmark_data_smithi173_55121_object13782:head# 2024-09-17T16:12:06.163 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12165b65:::benchmark_data_smithi173_55121_object25629:head# 2024-09-17T16:12:06.164 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:121934bf:::benchmark_data_smithi173_55121_object32945:head# 2024-09-17T16:12:06.165 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1219f66f:::benchmark_data_smithi173_55121_object6800:head# 2024-09-17T16:12:06.166 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:121c730c:::benchmark_data_smithi173_55121_object13381:head# 2024-09-17T16:12:06.166 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:121d78a2:::benchmark_data_smithi173_55121_object3390:head# 2024-09-17T16:12:06.166 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:121f6650:::benchmark_data_smithi173_55121_object8119:head# 2024-09-17T16:12:06.167 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12211b7c:::benchmark_data_smithi173_55121_object30497:head# 2024-09-17T16:12:06.167 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1221b570:::benchmark_data_smithi173_55121_object17940:head# 2024-09-17T16:12:06.167 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:122640fc:::benchmark_data_smithi173_55121_object11722:head# 2024-09-17T16:12:06.168 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:122d1f7c:::benchmark_data_smithi173_55121_object4427:head# 2024-09-17T16:12:06.168 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:122f6ea9:::benchmark_data_smithi173_55121_object29750:head# 2024-09-17T16:12:06.169 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1230c4cc:::benchmark_data_smithi173_55121_object29105:head# 2024-09-17T16:12:06.169 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1231046c:::benchmark_data_smithi173_55121_object7680:head# 2024-09-17T16:12:06.169 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1231b636:::benchmark_data_smithi173_55121_object15812:head# 2024-09-17T16:12:06.170 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12370c9a:::benchmark_data_smithi173_55121_object10850:head# 2024-09-17T16:12:06.170 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1239a6b0:::benchmark_data_smithi173_55121_object57:head# 2024-09-17T16:12:06.171 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:123acad0:::benchmark_data_smithi173_55121_object12041:head# 2024-09-17T16:12:06.171 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:123fe9fd:::benchmark_data_smithi173_55121_object32636:head# 2024-09-17T16:12:06.172 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12404c95:::benchmark_data_smithi173_55121_object29275:head# 2024-09-17T16:12:06.172 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12416356:::benchmark_data_smithi173_55121_object2066:head# 2024-09-17T16:12:06.173 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1242a6d2:::benchmark_data_smithi173_55121_object27616:head# 2024-09-17T16:12:06.173 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:124345ad:::benchmark_data_smithi173_55121_object36269:head# 2024-09-17T16:12:06.174 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12463cca:::benchmark_data_smithi173_55121_object9187:head# 2024-09-17T16:12:06.174 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12465984:::benchmark_data_smithi173_55121_object29292:head# 2024-09-17T16:12:06.175 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:124a6cd0:::benchmark_data_smithi173_55121_object18948:head# 2024-09-17T16:12:06.175 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:124ae083:::benchmark_data_smithi173_55121_object15988:head# 2024-09-17T16:12:06.176 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:124b77d8:::benchmark_data_smithi173_55121_object16238:head# 2024-09-17T16:12:06.176 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:124bd779:::benchmark_data_smithi173_55121_object35065:head# 2024-09-17T16:12:06.177 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:124c8c24:::benchmark_data_smithi173_55121_object18831:head# 2024-09-17T16:12:06.177 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:124dc76a:::benchmark_data_smithi173_55121_object10840:head# 2024-09-17T16:12:06.178 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1252c8de:::benchmark_data_smithi173_55121_object12508:head# 2024-09-17T16:12:06.180 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12536015:::benchmark_data_smithi173_55121_object21439:head# 2024-09-17T16:12:06.180 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12550c64:::benchmark_data_smithi173_55121_object31915:head# 2024-09-17T16:12:06.180 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:125704dd:::benchmark_data_smithi173_55121_object34474:head# 2024-09-17T16:12:06.180 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1259888d:::benchmark_data_smithi173_55121_object25403:head# 2024-09-17T16:12:06.181 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:125d9e28:::benchmark_data_smithi173_55121_object13663:head# 2024-09-17T16:12:06.181 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:126314cf:::benchmark_data_smithi173_55121_object26349:head# 2024-09-17T16:12:06.182 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12639e3f:::benchmark_data_smithi173_55121_object6133:head# 2024-09-17T16:12:06.182 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:126735c9:::benchmark_data_smithi173_55121_object4213:head# 2024-09-17T16:12:06.183 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1267e718:::benchmark_data_smithi173_55121_object20325:head# 2024-09-17T16:12:06.183 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1268d5e5:::benchmark_data_smithi173_55121_object27636:head# 2024-09-17T16:12:06.183 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1269a582:::benchmark_data_smithi173_55121_object7542:head# 2024-09-17T16:12:06.188 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1269fbb9:::benchmark_data_smithi173_55121_object5005:head# 2024-09-17T16:12:06.188 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:126a76b9:::benchmark_data_smithi173_55121_object28908:head# 2024-09-17T16:12:06.188 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:126b46bb:::benchmark_data_smithi173_55121_object14772:head# 2024-09-17T16:12:06.188 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:126baddf:::benchmark_data_smithi173_55121_object15291:head# 2024-09-17T16:12:06.188 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:126cd91e:::benchmark_data_smithi173_55121_object31104:head# 2024-09-17T16:12:06.189 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:126d1554:::benchmark_data_smithi173_55121_object16154:head# 2024-09-17T16:12:06.189 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:126ecc20:::benchmark_data_smithi173_55121_object11801:head# 2024-09-17T16:12:06.189 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:126f8b5d:::benchmark_data_smithi173_55121_object28038:head# 2024-09-17T16:12:06.189 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1270e3c1:::benchmark_data_smithi173_55121_object24047:head# 2024-09-17T16:12:06.190 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1277095f:::benchmark_data_smithi173_55121_object23454:head# 2024-09-17T16:12:06.190 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12798dd0:::benchmark_data_smithi173_55121_object11492:head# 2024-09-17T16:12:06.190 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:127ac868:::benchmark_data_smithi173_55121_object6525:head# 2024-09-17T16:12:06.197 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:127aed42:::benchmark_data_smithi173_55121_object15822:head# 2024-09-17T16:12:06.197 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:127bdc13:::benchmark_data_smithi173_55121_object24856:head# 2024-09-17T16:12:06.197 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:127d60ea:::benchmark_data_smithi173_55121_object17998:head# 2024-09-17T16:12:06.197 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12808f27:::benchmark_data_smithi173_55121_object3453:head# 2024-09-17T16:12:06.198 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1281c135:::benchmark_data_smithi173_55121_object3040:head# 2024-09-17T16:12:06.198 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:128434e3:::benchmark_data_smithi173_55121_object33016:head# 2024-09-17T16:12:06.198 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12866bc0:::benchmark_data_smithi173_55121_object18530:head# 2024-09-17T16:12:06.198 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1286c8e1:::benchmark_data_smithi173_55121_object26586:head# 2024-09-17T16:12:06.198 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12887750:::benchmark_data_smithi173_55121_object2814:head# 2024-09-17T16:12:06.198 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12896178:::benchmark_data_smithi173_55121_object12223:head# 2024-09-17T16:12:06.199 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12898907:::benchmark_data_smithi173_55121_object1846:head# 2024-09-17T16:12:06.199 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:128a214e:::benchmark_data_smithi173_55121_object6060:head# 2024-09-17T16:12:06.199 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12924156:::benchmark_data_smithi173_55121_object7430:head# 2024-09-17T16:12:06.199 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1292ec9f:::benchmark_data_smithi173_55121_object15477:head# 2024-09-17T16:12:06.199 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1296b073:::benchmark_data_smithi173_55121_object26979:head# 2024-09-17T16:12:06.200 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:129743fd:::benchmark_data_smithi173_55121_object36360:head# 2024-09-17T16:12:06.200 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12992428:::benchmark_data_smithi173_55121_object31459:head# 2024-09-17T16:12:06.200 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:129a13c8:::benchmark_data_smithi173_55121_object14637:head# 2024-09-17T16:12:06.200 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:129d74bf:::benchmark_data_smithi173_55121_object20310:head# 2024-09-17T16:12:06.200 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:129da034:::benchmark_data_smithi173_55121_object32468:head# 2024-09-17T16:12:06.202 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12a08404:::benchmark_data_smithi173_55121_object8420:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12a634f1:::benchmark_data_smithi173_55121_object21289:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12a72343:::benchmark_data_smithi173_55121_object11223:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12aa6921:::benchmark_data_smithi173_55121_object18283:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12ad186e:::benchmark_data_smithi173_55121_object3025:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12ad3c1c:::benchmark_data_smithi173_55121_object15738:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12b59da0:::benchmark_data_smithi173_55121_object31335:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12b664ff:::benchmark_data_smithi173_55121_object8380:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12b72d19:::benchmark_data_smithi173_55121_object4336:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12b85bbe:::benchmark_data_smithi173_55121_object1126:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12b89de9:::benchmark_data_smithi173_55121_object12882:head# 2024-09-17T16:12:06.203 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12bb83e8:::benchmark_data_smithi173_55121_object29842:head# 2024-09-17T16:12:06.204 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12bd3c32:::benchmark_data_smithi173_55121_object9239:head# 2024-09-17T16:12:06.204 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12bdfde8:::benchmark_data_smithi173_55121_object8460:head# 2024-09-17T16:12:06.204 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12be3a6c:::benchmark_data_smithi173_55121_object16200:head# 2024-09-17T16:12:06.204 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12bff709:::benchmark_data_smithi173_55121_object26537:head# 2024-09-17T16:12:06.205 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12c1c1fb:::benchmark_data_smithi173_55121_object19701:head# 2024-09-17T16:12:06.205 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12c22b74:::benchmark_data_smithi173_55121_object4762:head# 2024-09-17T16:12:06.205 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12c65e03:::benchmark_data_smithi173_55121_object2222:head# 2024-09-17T16:12:06.206 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12c6fc34:::benchmark_data_smithi173_55121_object10641:head# 2024-09-17T16:12:06.206 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12c7fea8:::benchmark_data_smithi173_55121_object32871:head# 2024-09-17T16:12:06.207 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12c95f32:::benchmark_data_smithi173_55121_object2903:head# 2024-09-17T16:12:06.207 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12ca7da7:::benchmark_data_smithi173_55121_object3438:head# 2024-09-17T16:12:06.208 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12ce0422:::benchmark_data_smithi173_55121_object15479:head# 2024-09-17T16:12:06.208 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12ce49c4:::benchmark_data_smithi173_55121_object15426:head# 2024-09-17T16:12:06.209 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12cefbda:::benchmark_data_smithi173_55121_object34381:head# 2024-09-17T16:12:06.209 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12cf1253:::benchmark_data_smithi173_55121_object35597:head# 2024-09-17T16:12:06.209 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12d0230e:::benchmark_data_smithi173_55121_object946:head# 2024-09-17T16:12:06.210 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12d11b17:::benchmark_data_smithi173_55121_object20286:head# 2024-09-17T16:12:06.210 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:06.211 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12d3c795:::benchmark_data_smithi173_55121_object20601:head# 2024-09-17T16:12:06.211 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12d3f1a7:::benchmark_data_smithi173_55121_object32625:head# 2024-09-17T16:12:06.211 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12d4e766:::benchmark_data_smithi173_55121_object35743:head# 2024-09-17T16:12:06.212 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12d8ffb8:::benchmark_data_smithi173_55121_object36970:head# 2024-09-17T16:12:06.212 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12ddaee1:::benchmark_data_smithi173_55121_object24220:head# 2024-09-17T16:12:06.213 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12deb5fe:::benchmark_data_smithi173_55121_object5295:head# 2024-09-17T16:12:06.214 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12dffba9:::benchmark_data_smithi173_55121_object19009:head# 2024-09-17T16:12:06.214 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12e0166e:::benchmark_data_smithi173_55121_object20290:head# 2024-09-17T16:12:06.214 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12e096fb:::benchmark_data_smithi173_55121_object20882:head# 2024-09-17T16:12:06.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 43595 43579 42.5532 0 - 0.0221508 2024-09-17T16:12:06.215 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12e249b6:::benchmark_data_smithi173_55121_object28925:head# 2024-09-17T16:12:06.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:12:06.216 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12e24c44:::benchmark_data_smithi173_55121_object25688:head# 2024-09-17T16:12:06.216 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12e756ef:::benchmark_data_smithi173_55121_object34145:head# 2024-09-17T16:12:06.216 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12e759dd:::benchmark_data_smithi173_55121_object6401:head# 2024-09-17T16:12:06.217 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12e79435:::benchmark_data_smithi173_55121_object35091:head# 2024-09-17T16:12:06.217 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12e80ef3:::benchmark_data_smithi173_55121_object15424:head# 2024-09-17T16:12:06.217 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12e8c5f9:::benchmark_data_smithi173_55121_object759:head# 2024-09-17T16:12:06.218 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12e90b58:::benchmark_data_smithi173_55121_object10640:head# 2024-09-17T16:12:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12eb577c:::benchmark_data_smithi173_55121_object35163:head# 2024-09-17T16:12:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12ed1e55:::benchmark_data_smithi173_55121_object19428:head# 2024-09-17T16:12:06.219 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12f24834:::benchmark_data_smithi173_55121_object15273:head# 2024-09-17T16:12:06.220 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12f24854:::benchmark_data_smithi173_55121_object26185:head# 2024-09-17T16:12:06.220 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12f7a138:::benchmark_data_smithi173_55121_object7434:head# 2024-09-17T16:12:06.220 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12f89a21:::benchmark_data_smithi173_55121_object9929:head# 2024-09-17T16:12:06.221 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12f89af2:::benchmark_data_smithi173_55121_object20991:head# 2024-09-17T16:12:06.221 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12f95884:::benchmark_data_smithi173_55121_object12431:head# 2024-09-17T16:12:06.222 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12fc4b8e:::benchmark_data_smithi173_55121_object407:head# 2024-09-17T16:12:06.222 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12fee76b:::benchmark_data_smithi173_55121_object36730:head# 2024-09-17T16:12:06.223 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:12ffc9a6:::benchmark_data_smithi173_55121_object34699:head# 2024-09-17T16:12:06.223 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1308fdc8:::benchmark_data_smithi173_55121_object32780:head# 2024-09-17T16:12:06.224 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1309dc24:::benchmark_data_smithi173_55121_object29470:head# 2024-09-17T16:12:06.224 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:130a285f:::benchmark_data_smithi173_55121_object14406:head# 2024-09-17T16:12:06.225 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:130a701f:::benchmark_data_smithi173_55121_object23851:head# 2024-09-17T16:12:06.225 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:130d71a6:::benchmark_data_smithi173_55121_object6602:head# 2024-09-17T16:12:06.226 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:130e6c8c:::benchmark_data_smithi173_55121_object31317:head# 2024-09-17T16:12:06.226 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:130f3bfd:::benchmark_data_smithi173_55121_object29520:head# 2024-09-17T16:12:06.226 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:130f93d2:::benchmark_data_smithi173_55121_object2848:head# 2024-09-17T16:12:06.227 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13107f38:::benchmark_data_smithi173_55121_object6618:head# 2024-09-17T16:12:06.227 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1310af39:::benchmark_data_smithi173_55121_object27829:head# 2024-09-17T16:12:06.228 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1310dc98:::benchmark_data_smithi173_55121_object32383:head# 2024-09-17T16:12:06.228 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13135af7:::benchmark_data_smithi173_55121_object24870:head# 2024-09-17T16:12:06.229 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:131375f3:::benchmark_data_smithi173_55121_object21160:head# 2024-09-17T16:12:06.229 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1315723f:::benchmark_data_smithi173_55121_object4492:head# 2024-09-17T16:12:06.230 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13172a19:::benchmark_data_smithi173_55121_object21434:head# 2024-09-17T16:12:06.230 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13176735:::benchmark_data_smithi173_55121_object21220:head# 2024-09-17T16:12:06.230 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13181a33:::benchmark_data_smithi173_55121_object348:head# 2024-09-17T16:12:06.231 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:131840c4:::benchmark_data_smithi173_55121_object2587:head# 2024-09-17T16:12:06.231 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13216039:::benchmark_data_smithi173_55121_object20780:head# 2024-09-17T16:12:06.232 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:132262d1:::benchmark_data_smithi173_55121_object639:head# 2024-09-17T16:12:06.232 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1324c379:::benchmark_data_smithi173_55121_object15157:head# 2024-09-17T16:12:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13265bd1:::benchmark_data_smithi173_55121_object17803:head# 2024-09-17T16:12:06.233 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:132774f4:::benchmark_data_smithi173_55121_object10431:head# 2024-09-17T16:12:06.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:06.234 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:132a580b:::benchmark_data_smithi173_55121_object31566:head# 2024-09-17T16:12:06.234 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:132ec686:::benchmark_data_smithi173_55121_object22139:head# 2024-09-17T16:12:06.235 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:132f2169:::benchmark_data_smithi173_55121_object11186:head# 2024-09-17T16:12:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:06.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:132ff79b:::benchmark_data_smithi173_55121_object10818:head# 2024-09-17T16:12:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1330f6e7:::benchmark_data_smithi173_55121_object23299:head# 2024-09-17T16:12:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13316a28:::benchmark_data_smithi173_55121_object34330:head# 2024-09-17T16:12:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1331c5b5:::benchmark_data_smithi173_55121_object19096:head# 2024-09-17T16:12:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1333b8f7:::benchmark_data_smithi173_55121_object5253:head# 2024-09-17T16:12:06.238 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1333edf2:::benchmark_data_smithi173_55121_object36640:head# 2024-09-17T16:12:06.238 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1335b3b6:::benchmark_data_smithi173_55121_object9797:head# 2024-09-17T16:12:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13364278:::benchmark_data_smithi173_55121_object20958:head# 2024-09-17T16:12:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13379e83:::benchmark_data_smithi173_55121_object35018:head# 2024-09-17T16:12:06.239 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:133d86b8:::benchmark_data_smithi173_55121_object27447:head# 2024-09-17T16:12:06.240 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:133e36b5:::benchmark_data_smithi173_55121_object17425:head# 2024-09-17T16:12:06.240 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1345f97c:::benchmark_data_smithi173_55121_object19877:head# 2024-09-17T16:12:06.241 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13475e2e:::benchmark_data_smithi173_55121_object21693:head# 2024-09-17T16:12:06.241 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1349c4b3:::benchmark_data_smithi173_55121_object16264:head# 2024-09-17T16:12:06.242 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:134acb25:::benchmark_data_smithi173_55121_object17396:head# 2024-09-17T16:12:06.242 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:134c2f68:::benchmark_data_smithi173_55121_object37280:head# 2024-09-17T16:12:06.243 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:134d2de4:::benchmark_data_smithi173_55121_object4122:head# 2024-09-17T16:12:06.243 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1350e037:::benchmark_data_smithi173_55121_object6580:head# 2024-09-17T16:12:06.243 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13512b3a:::benchmark_data_smithi173_55121_object1169:head# 2024-09-17T16:12:06.244 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13521d71:::benchmark_data_smithi173_55121_object35393:head# 2024-09-17T16:12:06.245 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13535c0a:::benchmark_data_smithi173_55121_object33721:head# 2024-09-17T16:12:06.245 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1355fbe8:::benchmark_data_smithi173_55121_object3004:head# 2024-09-17T16:12:06.246 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1356961e:::benchmark_data_smithi173_55121_object19215:head# 2024-09-17T16:12:06.246 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:135773c0:::benchmark_data_smithi173_55121_object24367:head# 2024-09-17T16:12:06.246 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1357bee8:::benchmark_data_smithi173_55121_object15701:head# 2024-09-17T16:12:06.247 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:135920c6:::benchmark_data_smithi173_55121_object13424:head# 2024-09-17T16:12:06.247 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:135d3cae:::benchmark_data_smithi173_55121_object6351:head# 2024-09-17T16:12:06.248 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:135e9f37:::benchmark_data_smithi173_55121_object19263:head# 2024-09-17T16:12:06.248 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:135ea149:::benchmark_data_smithi173_55121_object27345:head# 2024-09-17T16:12:06.249 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:135f58e7:::benchmark_data_smithi173_55121_object17699:head# 2024-09-17T16:12:06.249 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13608c7d:::benchmark_data_smithi173_55121_object4313:head# 2024-09-17T16:12:06.250 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13629d80:::benchmark_data_smithi173_55121_object19087:head# 2024-09-17T16:12:06.251 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1363e2a1:::benchmark_data_smithi173_55121_object18942:head# 2024-09-17T16:12:06.251 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13647574:::benchmark_data_smithi173_55121_object2506:head# 2024-09-17T16:12:06.251 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1365245c:::benchmark_data_smithi173_55121_object31567:head# 2024-09-17T16:12:06.252 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13663aba:::benchmark_data_smithi173_55121_object32175:head# 2024-09-17T16:12:06.252 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1366deab:::benchmark_data_smithi173_55121_object21304:head# 2024-09-17T16:12:06.253 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13691d4a:::benchmark_data_smithi173_55121_object8575:head# 2024-09-17T16:12:06.253 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:136cedcb:::benchmark_data_smithi173_55121_object19725:head# 2024-09-17T16:12:06.254 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:136d0600:::benchmark_data_smithi173_55121_object10393:head# 2024-09-17T16:12:06.254 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:136f5023:::benchmark_data_smithi173_55121_object5678:head# 2024-09-17T16:12:06.255 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13705ecd:::benchmark_data_smithi173_55121_object18605:head# 2024-09-17T16:12:06.255 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13709ed7:::benchmark_data_smithi173_55121_object31398:head# 2024-09-17T16:12:06.256 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1371d9a2:::benchmark_data_smithi173_55121_object26759:head# 2024-09-17T16:12:06.256 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:137283fc:::benchmark_data_smithi173_55121_object20550:head# 2024-09-17T16:12:06.257 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1373f69d:::benchmark_data_smithi173_55121_object34349:head# 2024-09-17T16:12:06.257 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13754ff7:::benchmark_data_smithi173_55121_object21929:head# 2024-09-17T16:12:06.258 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13764800:::benchmark_data_smithi173_55121_object885:head# 2024-09-17T16:12:06.258 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1377965b:::benchmark_data_smithi173_55121_object3401:head# 2024-09-17T16:12:06.259 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1377bb68:::benchmark_data_smithi173_55121_object9358:head# 2024-09-17T16:12:06.259 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13796fd3:::benchmark_data_smithi173_55121_object11727:head# 2024-09-17T16:12:06.260 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:137d599f:::benchmark_data_smithi173_55121_object4380:head# 2024-09-17T16:12:06.260 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:137d69d5:::benchmark_data_smithi173_55121_object24887:head# 2024-09-17T16:12:06.261 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:137fce8a:::benchmark_data_smithi173_55121_object687:head# 2024-09-17T16:12:06.261 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:138131ac:::benchmark_data_smithi173_55121_object34212:head# 2024-09-17T16:12:06.262 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1382aeba:::benchmark_data_smithi173_55121_object16725:head# 2024-09-17T16:12:06.262 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13850090:::benchmark_data_smithi173_55121_object7988:head# 2024-09-17T16:12:06.263 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13853cfb:::benchmark_data_smithi173_55121_object13919:head# 2024-09-17T16:12:06.263 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1387daa0:::benchmark_data_smithi173_55121_object1320:head# 2024-09-17T16:12:06.264 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:138a48a4:::benchmark_data_smithi173_55121_object33613:head# 2024-09-17T16:12:06.264 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:138fe656:::benchmark_data_smithi173_55121_object3291:head# 2024-09-17T16:12:06.265 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:139471ea:::benchmark_data_smithi173_55121_object5399:head# 2024-09-17T16:12:06.265 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:139725cd:::benchmark_data_smithi173_55121_object5363:head# 2024-09-17T16:12:06.265 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13976581:::benchmark_data_smithi173_55121_object37366:head# 2024-09-17T16:12:06.266 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:139b271f:::benchmark_data_smithi173_55121_object29756:head# 2024-09-17T16:12:06.266 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:139b27dd:::benchmark_data_smithi173_55121_object5640:head# 2024-09-17T16:12:06.267 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:139f7def:::benchmark_data_smithi173_55121_object6827:head# 2024-09-17T16:12:06.267 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:139fbd52:::benchmark_data_smithi173_55121_object20018:head# 2024-09-17T16:12:06.268 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13a3d592:::benchmark_data_smithi173_55121_object16045:head# 2024-09-17T16:12:06.268 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13aaadb5:::benchmark_data_smithi173_55121_object12989:head# 2024-09-17T16:12:06.269 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13aafffb:::benchmark_data_smithi173_55121_object4625:head# 2024-09-17T16:12:06.269 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13ab4cc5:::benchmark_data_smithi173_55121_object19874:head# 2024-09-17T16:12:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13ae4fbf:::benchmark_data_smithi173_55121_object31949:head# 2024-09-17T16:12:06.270 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13afbe7a:::benchmark_data_smithi173_55121_object27253:head# 2024-09-17T16:12:06.271 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b2ebae:::benchmark_data_smithi173_55121_object8879:head# 2024-09-17T16:12:06.271 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b464c3:::benchmark_data_smithi173_55121_object18340:head# 2024-09-17T16:12:06.271 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b5b396:::benchmark_data_smithi173_55121_object33848:head# 2024-09-17T16:12:06.272 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b5dcf7:::benchmark_data_smithi173_55121_object28454:head# 2024-09-17T16:12:06.272 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b5f5f7:::benchmark_data_smithi173_55121_object31982:head# 2024-09-17T16:12:06.273 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b66ef6:::benchmark_data_smithi173_55121_object34347:head# 2024-09-17T16:12:06.273 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b6a037:::benchmark_data_smithi173_55121_object25478:head# 2024-09-17T16:12:06.274 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b7d140:::benchmark_data_smithi173_55121_object22619:head# 2024-09-17T16:12:06.274 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b806c4:::benchmark_data_smithi173_55121_object18008:head# 2024-09-17T16:12:06.274 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b9113c:::benchmark_data_smithi173_55121_object16439:head# 2024-09-17T16:12:06.275 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13b9557c:::benchmark_data_smithi173_55121_object3873:head# 2024-09-17T16:12:06.275 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13bf7dd1:::benchmark_data_smithi173_55121_object35211:head# 2024-09-17T16:12:06.276 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13bffffc:::benchmark_data_smithi173_55121_object23139:head# 2024-09-17T16:12:06.276 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13c0f122:::benchmark_data_smithi173_55121_object11314:head# 2024-09-17T16:12:06.277 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13c59a50:::benchmark_data_smithi173_55121_object26008:head# 2024-09-17T16:12:06.277 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13c69bd8:::benchmark_data_smithi173_55121_object1759:head# 2024-09-17T16:12:06.277 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13c6af85:::benchmark_data_smithi173_55121_object33750:head# 2024-09-17T16:12:06.278 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13c6ee06:::benchmark_data_smithi173_55121_object19959:head# 2024-09-17T16:12:06.278 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13c7619d:::benchmark_data_smithi173_55121_object22777:head# 2024-09-17T16:12:06.279 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13c7653a:::benchmark_data_smithi173_55121_object25966:head# 2024-09-17T16:12:06.279 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13c9886a:::benchmark_data_smithi173_55121_object35413:head# 2024-09-17T16:12:06.279 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13cad6a8:::benchmark_data_smithi173_55121_object24173:head# 2024-09-17T16:12:06.280 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13cb416d:::benchmark_data_smithi173_55121_object15488:head# 2024-09-17T16:12:06.280 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13ccc7d3:::benchmark_data_smithi173_55121_object28121:head# 2024-09-17T16:12:06.281 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13cd19e0:::benchmark_data_smithi173_55121_object16683:head# 2024-09-17T16:12:06.281 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13d02cbf:::benchmark_data_smithi173_55121_object16290:head# 2024-09-17T16:12:06.281 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13d0cf87:::benchmark_data_smithi173_55121_object12088:head# 2024-09-17T16:12:06.282 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13d3df4f:::benchmark_data_smithi173_55121_object5375:head# 2024-09-17T16:12:06.282 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13d4dc8c:::benchmark_data_smithi173_55121_object16189:head# 2024-09-17T16:12:06.282 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13d573c7:::benchmark_data_smithi173_55121_object30214:head# 2024-09-17T16:12:06.283 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13d5d9ea:::benchmark_data_smithi173_55121_object12576:head# 2024-09-17T16:12:06.283 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13d84d39:::benchmark_data_smithi173_55121_object3157:head# 2024-09-17T16:12:06.283 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13d89c05:::benchmark_data_smithi173_55121_object6946:head# 2024-09-17T16:12:06.284 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13da8a4c:::benchmark_data_smithi173_55121_object7182:head# 2024-09-17T16:12:06.284 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13db6325:::benchmark_data_smithi173_55121_object26600:head# 2024-09-17T16:12:06.285 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13dc9189:::benchmark_data_smithi173_55121_object25815:head# 2024-09-17T16:12:06.285 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13de04ba:::benchmark_data_smithi173_55121_object17004:head# 2024-09-17T16:12:06.286 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13e06e0e:::benchmark_data_smithi173_55121_object478:head# 2024-09-17T16:12:06.286 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13e450a4:::benchmark_data_smithi173_55121_object14296:head# 2024-09-17T16:12:06.286 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13e61a09:::benchmark_data_smithi173_55121_object34462:head# 2024-09-17T16:12:06.287 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13e629a8:::benchmark_data_smithi173_55121_object26331:head# 2024-09-17T16:12:06.287 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13e845d9:::benchmark_data_smithi173_55121_object28621:head# 2024-09-17T16:12:06.288 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13e9c073:::benchmark_data_smithi173_55121_object8337:head# 2024-09-17T16:12:06.288 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13e9ecb4:::benchmark_data_smithi173_55121_object27910:head# 2024-09-17T16:12:06.289 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13ea4c0c:::benchmark_data_smithi173_55121_object650:head# 2024-09-17T16:12:06.289 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13eac6a9:::benchmark_data_smithi173_55121_object13319:head# 2024-09-17T16:12:06.289 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13eb058f:::benchmark_data_smithi173_55121_object13798:head# 2024-09-17T16:12:06.290 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13eb90e9:::benchmark_data_smithi173_55121_object12417:head# 2024-09-17T16:12:06.290 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13ee691a:::benchmark_data_smithi173_55121_object28465:head# 2024-09-17T16:12:06.290 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13f3e3ff:::benchmark_data_smithi173_55121_object9746:head# 2024-09-17T16:12:06.291 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13f70767:::benchmark_data_smithi173_55121_object10601:head# 2024-09-17T16:12:06.291 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13f817f3:::benchmark_data_smithi173_55121_object7635:head# 2024-09-17T16:12:06.292 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13f83dc1:::benchmark_data_smithi173_55121_object36076:head# 2024-09-17T16:12:06.292 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13f919c6:::benchmark_data_smithi173_55121_object15042:head# 2024-09-17T16:12:06.293 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13fa2673:::benchmark_data_smithi173_55121_object19166:head# 2024-09-17T16:12:06.293 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13fbf047:::benchmark_data_smithi173_55121_object28385:head# 2024-09-17T16:12:06.293 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13fca8ad:::benchmark_data_smithi173_55121_object34129:head# 2024-09-17T16:12:06.294 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13fcecb9:::benchmark_data_smithi173_55121_object14854:head# 2024-09-17T16:12:06.294 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13fe2c75:::benchmark_data_smithi173_55121_object36498:head# 2024-09-17T16:12:06.295 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13fe615a:::benchmark_data_smithi173_55121_object14232:head# 2024-09-17T16:12:06.295 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13fec027:::benchmark_data_smithi173_55121_object10629:head# 2024-09-17T16:12:06.296 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13ffa1c1:::benchmark_data_smithi173_55121_object23263:head# 2024-09-17T16:12:06.296 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:13ffed11:::benchmark_data_smithi173_55121_object4257:head# 2024-09-17T16:12:06.297 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14048ee0:::benchmark_data_smithi173_55121_object17668:head# 2024-09-17T16:12:06.297 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1406d01b:::benchmark_data_smithi173_55121_object20907:head# 2024-09-17T16:12:06.298 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1407ee3a:::benchmark_data_smithi173_55121_object17796:head# 2024-09-17T16:12:06.298 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:140a6011:::benchmark_data_smithi173_55121_object33700:head# 2024-09-17T16:12:06.298 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:140ad613:::benchmark_data_smithi173_55121_object22615:head# 2024-09-17T16:12:06.299 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:140c026a:::benchmark_data_smithi173_55121_object304:head# 2024-09-17T16:12:06.299 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:140ffa98:::benchmark_data_smithi173_55121_object11979:head# 2024-09-17T16:12:06.300 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14132ea5:::benchmark_data_smithi173_55121_object31184:head# 2024-09-17T16:12:06.300 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1415c629:::benchmark_data_smithi173_55121_object6311:head# 2024-09-17T16:12:06.301 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14162493:::benchmark_data_smithi173_55121_object15200:head# 2024-09-17T16:12:06.302 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14187239:::benchmark_data_smithi173_55121_object6499:head# 2024-09-17T16:12:06.302 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:141af370:::benchmark_data_smithi173_55121_object15967:head# 2024-09-17T16:12:06.302 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:141e910b:::benchmark_data_smithi173_55121_object14699:head# 2024-09-17T16:12:06.302 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1420b645:::benchmark_data_smithi173_55121_object12420:head# 2024-09-17T16:12:06.303 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:142192fd:::benchmark_data_smithi173_55121_object13178:head# 2024-09-17T16:12:06.303 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14221fb8:::benchmark_data_smithi173_55121_object553:head# 2024-09-17T16:12:06.304 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14222259:::benchmark_data_smithi173_55121_object32155:head# 2024-09-17T16:12:06.304 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14231e31:::benchmark_data_smithi173_55121_object9926:head# 2024-09-17T16:12:06.304 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1423c4e1:::benchmark_data_smithi173_55121_object36077:head# 2024-09-17T16:12:06.305 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1428e724:::benchmark_data_smithi173_55121_object25418:head# 2024-09-17T16:12:06.305 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1429b0d3:::benchmark_data_smithi173_55121_object24479:head# 2024-09-17T16:12:06.306 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:142e6777:::benchmark_data_smithi173_55121_object13324:head# 2024-09-17T16:12:06.306 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:142f08bb:::benchmark_data_smithi173_55121_object26879:head# 2024-09-17T16:12:06.307 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1434af36:::benchmark_data_smithi173_55121_object20412:head# 2024-09-17T16:12:06.307 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1434b39b:::benchmark_data_smithi173_55121_object22176:head# 2024-09-17T16:12:06.308 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14370843:::benchmark_data_smithi173_55121_object16245:head# 2024-09-17T16:12:06.308 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14372295:::benchmark_data_smithi173_55121_object4912:head# 2024-09-17T16:12:06.309 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1438f582:::benchmark_data_smithi173_55121_object29572:head# 2024-09-17T16:12:06.309 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1439804f:::benchmark_data_smithi173_55121_object2348:head# 2024-09-17T16:12:06.310 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1439d5bd:::benchmark_data_smithi173_55121_object15312:head# 2024-09-17T16:12:06.310 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:143b4287:::benchmark_data_smithi173_55121_object6202:head# 2024-09-17T16:12:06.310 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:143bb8c9:::benchmark_data_smithi173_55121_object25190:head# 2024-09-17T16:12:06.311 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:143bcc43:::benchmark_data_smithi173_55121_object29398:head# 2024-09-17T16:12:06.311 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:143c2d11:::benchmark_data_smithi173_55121_object16231:head# 2024-09-17T16:12:06.311 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:143e33b9:::benchmark_data_smithi173_55121_object24251:head# 2024-09-17T16:12:06.312 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:143f8ea6:::benchmark_data_smithi173_55121_object7664:head# 2024-09-17T16:12:06.313 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14419391:::benchmark_data_smithi173_55121_object35437:head# 2024-09-17T16:12:06.313 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1441bc83:::benchmark_data_smithi173_55121_object15783:head# 2024-09-17T16:12:06.313 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1446b137:::benchmark_data_smithi173_55121_object33924:head# 2024-09-17T16:12:06.314 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:144777fa:::benchmark_data_smithi173_55121_object8065:head# 2024-09-17T16:12:06.314 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1448c033:::benchmark_data_smithi173_55121_object11366:head# 2024-09-17T16:12:06.315 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14490f6c:::benchmark_data_smithi173_55121_object12561:head# 2024-09-17T16:12:06.315 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:144aafe3:::benchmark_data_smithi173_55121_object11211:head# 2024-09-17T16:12:06.316 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:144bde5c:::benchmark_data_smithi173_55121_object15531:head# 2024-09-17T16:12:06.316 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:144ea865:::benchmark_data_smithi173_55121_object15283:head# 2024-09-17T16:12:06.317 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:145004c2:::benchmark_data_smithi173_55121_object30313:head# 2024-09-17T16:12:06.317 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14502b46:::benchmark_data_smithi173_55121_object10427:head# 2024-09-17T16:12:06.317 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1450e88d:::benchmark_data_smithi173_55121_object31746:head# 2024-09-17T16:12:06.318 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:145561f6:::benchmark_data_smithi173_55121_object18441:head# 2024-09-17T16:12:06.318 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14574c15:::benchmark_data_smithi173_55121_object35048:head# 2024-09-17T16:12:06.319 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1457d294:::benchmark_data_smithi173_55121_object3276:head# 2024-09-17T16:12:06.320 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14583a00:::benchmark_data_smithi173_55121_object22681:head# 2024-09-17T16:12:06.320 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14584de4:::benchmark_data_smithi173_55121_object13408:head# 2024-09-17T16:12:06.320 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14597804:::benchmark_data_smithi173_55121_object28984:head# 2024-09-17T16:12:06.321 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:145a4b59:::benchmark_data_smithi173_55121_object17050:head# 2024-09-17T16:12:06.321 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:145ac675:::benchmark_data_smithi173_55121_object20724:head# 2024-09-17T16:12:06.322 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:145b6e88:::benchmark_data_smithi173_55121_object29814:head# 2024-09-17T16:12:06.322 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:145c364e:::benchmark_data_smithi173_55121_object10056:head# 2024-09-17T16:12:06.323 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:145c419b:::benchmark_data_smithi173_55121_object19570:head# 2024-09-17T16:12:06.323 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1461037a:::benchmark_data_smithi173_55121_object891:head# 2024-09-17T16:12:06.324 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14649a30:::benchmark_data_smithi173_55121_object6380:head# 2024-09-17T16:12:06.325 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1464f223:::benchmark_data_smithi173_55121_object17883:head# 2024-09-17T16:12:06.325 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1465204e:::benchmark_data_smithi173_55121_object3261:head# 2024-09-17T16:12:06.325 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14663d0b:::benchmark_data_smithi173_55121_object18663:head# 2024-09-17T16:12:06.326 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14696061:::benchmark_data_smithi173_55121_object23972:head# 2024-09-17T16:12:06.326 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:146d7bf6:::benchmark_data_smithi173_55121_object18061:head# 2024-09-17T16:12:06.327 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:146ef9a4:::benchmark_data_smithi173_55121_object19145:head# 2024-09-17T16:12:06.327 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:146f0979:::benchmark_data_smithi173_55121_object34360:head# 2024-09-17T16:12:06.328 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1470dfd3:::benchmark_data_smithi173_55121_object5220:head# 2024-09-17T16:12:06.328 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14714364:::benchmark_data_smithi173_55121_object30392:head# 2024-09-17T16:12:06.329 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:147187df:::benchmark_data_smithi173_55121_object6174:head# 2024-09-17T16:12:06.329 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14723fd8:::benchmark_data_smithi173_55121_object20501:head# 2024-09-17T16:12:06.330 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:147a0adb:::benchmark_data_smithi173_55121_object19039:head# 2024-09-17T16:12:06.330 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1482838a:::benchmark_data_smithi173_55121_object24156:head# 2024-09-17T16:12:06.331 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:148627a0:::benchmark_data_smithi173_55121_object29026:head# 2024-09-17T16:12:06.331 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:148901a8:::benchmark_data_smithi173_55121_object7085:head# 2024-09-17T16:12:06.332 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1489c459:::benchmark_data_smithi173_55121_object34694:head# 2024-09-17T16:12:06.332 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:148a0876:::benchmark_data_smithi173_55121_object3698:head# 2024-09-17T16:12:06.333 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:148ad7cf:::benchmark_data_smithi173_55121_object1802:head# 2024-09-17T16:12:06.333 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:148ee069:::benchmark_data_smithi173_55121_object24398:head# 2024-09-17T16:12:06.334 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:148ee56d:::benchmark_data_smithi173_55121_object26326:head# 2024-09-17T16:12:06.334 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:148f1180:::benchmark_data_smithi173_55121_object34140:head# 2024-09-17T16:12:06.335 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:148f5585:::benchmark_data_smithi173_55121_object24053:head# 2024-09-17T16:12:06.335 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14921942:::benchmark_data_smithi173_55121_object14627:head# 2024-09-17T16:12:06.336 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14927897:::benchmark_data_smithi173_55121_object7684:head# 2024-09-17T16:12:06.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:06.337 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1492a7bd:::benchmark_data_smithi173_55121_object382:head# 2024-09-17T16:12:06.337 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14966653:::benchmark_data_smithi173_55121_object24953:head# 2024-09-17T16:12:06.337 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1498c070:::benchmark_data_smithi173_55121_object17971:head# 2024-09-17T16:12:06.338 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14996fd0:::benchmark_data_smithi173_55121_object12959:head# 2024-09-17T16:12:06.338 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1499a419:::benchmark_data_smithi173_55121_object1992:head# 2024-09-17T16:12:06.338 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1499a45f:::benchmark_data_smithi173_55121_object32061:head# 2024-09-17T16:12:06.339 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:149acb5e:::benchmark_data_smithi173_55121_object20730:head# 2024-09-17T16:12:06.340 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:149af195:::benchmark_data_smithi173_55121_object8130:head# 2024-09-17T16:12:06.340 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:149d9317:::benchmark_data_smithi173_55121_object3556:head# 2024-09-17T16:12:06.340 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14a03e63:::benchmark_data_smithi173_55121_object2054:head# 2024-09-17T16:12:06.341 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14a17395:::benchmark_data_smithi173_55121_object33460:head# 2024-09-17T16:12:06.342 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14a3bbe5:::benchmark_data_smithi173_55121_object37285:head# 2024-09-17T16:12:06.342 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14a86164:::benchmark_data_smithi173_55121_object31461:head# 2024-09-17T16:12:06.343 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14a875b9:::benchmark_data_smithi173_55121_object6799:head# 2024-09-17T16:12:06.343 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14abb0cc:::benchmark_data_smithi173_55121_object19093:head# 2024-09-17T16:12:06.343 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14ac2adb:::benchmark_data_smithi173_55121_object14184:head# 2024-09-17T16:12:06.344 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14ad3d38:::benchmark_data_smithi173_55121_object18310:head# 2024-09-17T16:12:06.345 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14ada928:::benchmark_data_smithi173_55121_object8953:head# 2024-09-17T16:12:06.345 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14adce74:::benchmark_data_smithi173_55121_object6693:head# 2024-09-17T16:12:06.345 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14afca7c:::benchmark_data_smithi173_55121_object12457:head# 2024-09-17T16:12:06.346 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14b0f346:::benchmark_data_smithi173_55121_object5040:head# 2024-09-17T16:12:06.346 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14b14af3:::benchmark_data_smithi173_55121_object1353:head# 2024-09-17T16:12:06.347 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14b16917:::benchmark_data_smithi173_55121_object30280:head# 2024-09-17T16:12:06.347 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14b1fe42:::benchmark_data_smithi173_55121_object164:head# 2024-09-17T16:12:06.347 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14b59eb8:::benchmark_data_smithi173_55121_object37196:head# 2024-09-17T16:12:06.348 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14b7a2dc:::benchmark_data_smithi173_55121_object24195:head# 2024-09-17T16:12:06.349 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14b86de9:::benchmark_data_smithi173_55121_object27869:head# 2024-09-17T16:12:06.349 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14b8b448:::benchmark_data_smithi173_55121_object19255:head# 2024-09-17T16:12:06.349 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14bc4a7f:::benchmark_data_smithi173_55121_object28697:head# 2024-09-17T16:12:06.350 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14bf1255:::benchmark_data_smithi173_55121_object24541:head# 2024-09-17T16:12:06.352 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14bf171d:::benchmark_data_smithi173_55121_object15505:head# 2024-09-17T16:12:06.352 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14bfa473:::benchmark_data_smithi173_55121_object32974:head# 2024-09-17T16:12:06.352 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14c25a15:::benchmark_data_smithi173_55121_object25387:head# 2024-09-17T16:12:06.352 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14c48c33:::benchmark_data_smithi173_55121_object27262:head# 2024-09-17T16:12:06.352 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14c5173a:::benchmark_data_smithi173_55121_object4691:head# 2024-09-17T16:12:06.353 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14c69611:::benchmark_data_smithi173_55121_object16763:head# 2024-09-17T16:12:06.354 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14c8baad:::benchmark_data_smithi173_55121_object12486:head# 2024-09-17T16:12:06.354 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14ca024d:::benchmark_data_smithi173_55121_object24337:head# 2024-09-17T16:12:06.355 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14cbfa8e:::benchmark_data_smithi173_55121_object32368:head# 2024-09-17T16:12:06.355 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14cc569d:::benchmark_data_smithi173_55121_object23369:head# 2024-09-17T16:12:06.356 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14cf8754:::benchmark_data_smithi173_55121_object18593:head# 2024-09-17T16:12:06.356 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14d16652:::benchmark_data_smithi173_55121_object6349:head# 2024-09-17T16:12:06.357 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14d2279d:::benchmark_data_smithi173_55121_object25367:head# 2024-09-17T16:12:06.357 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14d28eaf:::benchmark_data_smithi173_55121_object51:head# 2024-09-17T16:12:06.358 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14d4d275:::benchmark_data_smithi173_55121_object21884:head# 2024-09-17T16:12:06.358 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14d4de2d:::benchmark_data_smithi173_55121_object8981:head# 2024-09-17T16:12:06.359 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14d56b54:::benchmark_data_smithi173_55121_object26335:head# 2024-09-17T16:12:06.359 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14d6516e:::benchmark_data_smithi173_55121_object10480:head# 2024-09-17T16:12:06.359 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14d9ec1b:::benchmark_data_smithi173_55121_object4875:head# 2024-09-17T16:12:06.360 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14dab1a8:::benchmark_data_smithi173_55121_object23095:head# 2024-09-17T16:12:06.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:06.361 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14db98ef:::benchmark_data_smithi173_55121_object35113:head# 2024-09-17T16:12:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:06.361 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:06.362 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14de2308:::benchmark_data_smithi173_55121_object32350:head# 2024-09-17T16:12:06.362 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14de30d7:::benchmark_data_smithi173_55121_object32835:head# 2024-09-17T16:12:06.362 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14deaada:::benchmark_data_smithi173_55121_object20906:head# 2024-09-17T16:12:06.363 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14e02195:::benchmark_data_smithi173_55121_object21001:head# 2024-09-17T16:12:06.363 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14e0fc94:::benchmark_data_smithi173_55121_object8104:head# 2024-09-17T16:12:06.364 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14e206dc:::benchmark_data_smithi173_55121_object2221:head# 2024-09-17T16:12:06.364 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14e2d96a:::benchmark_data_smithi173_55121_object4520:head# 2024-09-17T16:12:06.365 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14e38033:::benchmark_data_smithi173_55121_object30343:head# 2024-09-17T16:12:06.365 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14e542d7:::benchmark_data_smithi173_55121_object14415:head# 2024-09-17T16:12:06.366 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14e75291:::benchmark_data_smithi173_55121_object21886:head# 2024-09-17T16:12:06.366 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14ea2843:::benchmark_data_smithi173_55121_object26806:head# 2024-09-17T16:12:06.366 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14ee4965:::benchmark_data_smithi173_55121_object7412:head# 2024-09-17T16:12:06.367 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14ef52b4:::benchmark_data_smithi173_55121_object30677:head# 2024-09-17T16:12:06.367 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14f21c3f:::benchmark_data_smithi173_55121_object5020:head# 2024-09-17T16:12:06.368 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14f3b0fe:::benchmark_data_smithi173_55121_object24134:head# 2024-09-17T16:12:06.369 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14f72e75:::benchmark_data_smithi173_55121_object29265:head# 2024-09-17T16:12:06.370 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14f7f5c0:::benchmark_data_smithi173_55121_object25343:head# 2024-09-17T16:12:06.370 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14f7fe11:::benchmark_data_smithi173_55121_object21229:head# 2024-09-17T16:12:06.370 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14f80f53:::benchmark_data_smithi173_55121_object33036:head# 2024-09-17T16:12:06.370 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14f8c35d:::benchmark_data_smithi173_55121_object26030:head# 2024-09-17T16:12:06.371 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14fab613:::benchmark_data_smithi173_55121_object24392:head# 2024-09-17T16:12:06.372 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14fae4bb:::benchmark_data_smithi173_55121_object14103:head# 2024-09-17T16:12:06.372 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14fb3220:::benchmark_data_smithi173_55121_object1271:head# 2024-09-17T16:12:06.373 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14fc6833:::benchmark_data_smithi173_55121_object26858:head# 2024-09-17T16:12:06.373 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:14fecccc:::benchmark_data_smithi173_55121_object16931:head# 2024-09-17T16:12:06.374 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15002fc2:::benchmark_data_smithi173_55121_object37213:head# 2024-09-17T16:12:06.374 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1501e40a:::benchmark_data_smithi173_55121_object1726:head# 2024-09-17T16:12:06.374 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1502ff15:::benchmark_data_smithi173_55121_object9029:head# 2024-09-17T16:12:06.375 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15030fb9:::benchmark_data_smithi173_55121_object34192:head# 2024-09-17T16:12:06.376 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:150332d0:::benchmark_data_smithi173_55121_object13942:head# 2024-09-17T16:12:06.376 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1506e21c:::benchmark_data_smithi173_55121_object32347:head# 2024-09-17T16:12:06.377 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:150779c0:::benchmark_data_smithi173_55121_object34455:head# 2024-09-17T16:12:06.378 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:150b4c44:::benchmark_data_smithi173_55121_object26828:head# 2024-09-17T16:12:06.378 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:150e2846:::benchmark_data_smithi173_55121_object30871:head# 2024-09-17T16:12:06.379 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15118b27:::benchmark_data_smithi173_55121_object9953:head# 2024-09-17T16:12:06.379 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1511d46a:::benchmark_data_smithi173_55121_object17367:head# 2024-09-17T16:12:06.379 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1514935e:::benchmark_data_smithi173_55121_object4330:head# 2024-09-17T16:12:06.380 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1518629e:::benchmark_data_smithi173_55121_object11275:head# 2024-09-17T16:12:06.380 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15188cf2:::benchmark_data_smithi173_55121_object300:head# 2024-09-17T16:12:06.381 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:151ac5b9:::benchmark_data_smithi173_55121_object18541:head# 2024-09-17T16:12:06.381 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:151b10cb:::benchmark_data_smithi173_55121_object3900:head# 2024-09-17T16:12:06.382 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:151bce1b:::benchmark_data_smithi173_55121_object21948:head# 2024-09-17T16:12:06.382 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:151e2b26:::benchmark_data_smithi173_55121_object35206:head# 2024-09-17T16:12:06.383 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:151e5d5d:::benchmark_data_smithi173_55121_object18874:head# 2024-09-17T16:12:06.383 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:151e7cfa:::benchmark_data_smithi173_55121_object24216:head# 2024-09-17T16:12:06.384 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:151ffbed:::benchmark_data_smithi173_55121_object5276:head# 2024-09-17T16:12:06.384 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1521fe1c:::benchmark_data_smithi173_55121_object2794:head# 2024-09-17T16:12:06.385 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:152234ea:::benchmark_data_smithi173_55121_object37032:head# 2024-09-17T16:12:06.385 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15244070:::benchmark_data_smithi173_55121_object20890:head# 2024-09-17T16:12:06.386 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15248bfc:::benchmark_data_smithi173_55121_object16680:head# 2024-09-17T16:12:06.386 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15253913:::benchmark_data_smithi173_55121_object9316:head# 2024-09-17T16:12:06.387 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:152616c9:::benchmark_data_smithi173_55121_object14188:head# 2024-09-17T16:12:06.387 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15261945:::benchmark_data_smithi173_55121_object9300:head# 2024-09-17T16:12:06.388 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:152633ec:::benchmark_data_smithi173_55121_object5889:head# 2024-09-17T16:12:06.388 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:152a38fe:::benchmark_data_smithi173_55121_object19759:head# 2024-09-17T16:12:06.389 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:152a3dca:::benchmark_data_smithi173_55121_object6869:head# 2024-09-17T16:12:06.389 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:152b1a40:::benchmark_data_smithi173_55121_object28878:head# 2024-09-17T16:12:06.390 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:152e2842:::benchmark_data_smithi173_55121_object16356:head# 2024-09-17T16:12:06.390 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:152e63b9:::benchmark_data_smithi173_55121_object11737:head# 2024-09-17T16:12:06.391 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:152e753a:::benchmark_data_smithi173_55121_object18747:head# 2024-09-17T16:12:06.391 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1530ed34:::benchmark_data_smithi173_55121_object28466:head# 2024-09-17T16:12:06.391 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1532ca94:::benchmark_data_smithi173_55121_object12059:head# 2024-09-17T16:12:06.392 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15350352:::benchmark_data_smithi173_55121_object30921:head# 2024-09-17T16:12:06.392 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1537328c:::benchmark_data_smithi173_55121_object8710:head# 2024-09-17T16:12:06.393 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15374f08:::benchmark_data_smithi173_55121_object11959:head# 2024-09-17T16:12:06.393 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1537930f:::benchmark_data_smithi173_55121_object35374:head# 2024-09-17T16:12:06.394 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15381c84:::benchmark_data_smithi173_55121_object26650:head# 2024-09-17T16:12:06.394 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1539958b:::benchmark_data_smithi173_55121_object4465:head# 2024-09-17T16:12:06.395 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:153ac907:::benchmark_data_smithi173_55121_object15554:head# 2024-09-17T16:12:06.395 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:153d3b7c:::benchmark_data_smithi173_55121_object19458:head# 2024-09-17T16:12:06.396 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15407f1f:::benchmark_data_smithi173_55121_object16304:head# 2024-09-17T16:12:06.396 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15454a63:::benchmark_data_smithi173_55121_object11657:head# 2024-09-17T16:12:06.397 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15481e77:::benchmark_data_smithi173_55121_object8797:head# 2024-09-17T16:12:06.397 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:154847a9:::benchmark_data_smithi173_55121_object36748:head# 2024-09-17T16:12:06.398 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1549feb5:::benchmark_data_smithi173_55121_object17634:head# 2024-09-17T16:12:06.398 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:154a35dd:::benchmark_data_smithi173_55121_object30271:head# 2024-09-17T16:12:06.399 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:154b2a32:::benchmark_data_smithi173_55121_object18042:head# 2024-09-17T16:12:06.399 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:154b7b29:::benchmark_data_smithi173_55121_object18527:head# 2024-09-17T16:12:06.400 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:154b8e49:::benchmark_data_smithi173_55121_object35154:head# 2024-09-17T16:12:06.400 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:154f98af:::benchmark_data_smithi173_55121_object21042:head# 2024-09-17T16:12:06.401 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:155163da:::benchmark_data_smithi173_55121_object28334:head# 2024-09-17T16:12:06.401 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15576e24:::benchmark_data_smithi173_55121_object29654:head# 2024-09-17T16:12:06.402 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1557ba33:::benchmark_data_smithi173_55121_object24350:head# 2024-09-17T16:12:06.402 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15586d86:::benchmark_data_smithi173_55121_object32174:head# 2024-09-17T16:12:06.403 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15587ce9:::benchmark_data_smithi173_55121_object15450:head# 2024-09-17T16:12:06.403 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:155b7bdd:::benchmark_data_smithi173_55121_object9645:head# 2024-09-17T16:12:06.404 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:155c2ec2:::benchmark_data_smithi173_55121_object10391:head# 2024-09-17T16:12:06.404 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:155d8f7c:::benchmark_data_smithi173_55121_object36321:head# 2024-09-17T16:12:06.404 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:155e5ac1:::benchmark_data_smithi173_55121_object19922:head# 2024-09-17T16:12:06.405 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15617fcc:::benchmark_data_smithi173_55121_object21523:head# 2024-09-17T16:12:06.405 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1561f169:::benchmark_data_smithi173_55121_object8372:head# 2024-09-17T16:12:06.406 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15629073:::benchmark_data_smithi173_55121_object35442:head# 2024-09-17T16:12:06.407 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:12:06.408 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15631760:::benchmark_data_smithi173_55121_object17447:head# 2024-09-17T16:12:06.409 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1563417e:::benchmark_data_smithi173_55121_object20511:head# 2024-09-17T16:12:06.409 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1566593c:::benchmark_data_smithi173_55121_object20299:head# 2024-09-17T16:12:06.409 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1569e85c:::benchmark_data_smithi173_55121_object15739:head# 2024-09-17T16:12:06.410 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:156a2abb:::benchmark_data_smithi173_55121_object3884:head# 2024-09-17T16:12:06.410 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:156b4f75:::benchmark_data_smithi173_55121_object24733:head# 2024-09-17T16:12:06.411 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:156e5dbe:::benchmark_data_smithi173_55121_object20781:head# 2024-09-17T16:12:06.411 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:156eb9d6:::benchmark_data_smithi173_55121_object9563:head# 2024-09-17T16:12:06.412 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:157395c3:::benchmark_data_smithi173_55121_object14618:head# 2024-09-17T16:12:06.412 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15742e2e:::benchmark_data_smithi173_55121_object5054:head# 2024-09-17T16:12:06.413 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1575269b:::benchmark_data_smithi173_55121_object13760:head# 2024-09-17T16:12:06.413 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1577ac4d:::benchmark_data_smithi173_55121_object15810:head# 2024-09-17T16:12:06.413 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1577b38b:::benchmark_data_smithi173_55121_object20787:head# 2024-09-17T16:12:06.414 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:157ceb88:::benchmark_data_smithi173_55121_object22738:head# 2024-09-17T16:12:06.414 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:157e8a43:::benchmark_data_smithi173_55121_object34385:head# 2024-09-17T16:12:06.415 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:157f1077:::benchmark_data_smithi173_55121_object15653:head# 2024-09-17T16:12:06.415 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:157fd8bc:::benchmark_data_smithi173_55121_object24106:head# 2024-09-17T16:12:06.416 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:158428f8:::benchmark_data_smithi173_55121_object29642:head# 2024-09-17T16:12:06.416 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:158577d8:::benchmark_data_smithi173_55121_object10286:head# 2024-09-17T16:12:06.417 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1585fea1:::benchmark_data_smithi173_55121_object20460:head# 2024-09-17T16:12:06.417 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1587de72:::benchmark_data_smithi173_55121_object7591:head# 2024-09-17T16:12:06.417 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15885dac:::benchmark_data_smithi173_55121_object29716:head# 2024-09-17T16:12:06.418 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:158a1e11:::benchmark_data_smithi173_55121_object32359:head# 2024-09-17T16:12:06.418 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:158ac562:::benchmark_data_smithi173_55121_object6470:head# 2024-09-17T16:12:06.419 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:158cb29a:::benchmark_data_smithi173_55121_object9693:head# 2024-09-17T16:12:06.419 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:158d7c97:::benchmark_data_smithi173_55121_object11133:head# 2024-09-17T16:12:06.420 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15907c92:::benchmark_data_smithi173_55121_object37036:head# 2024-09-17T16:12:06.420 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1591c986:::benchmark_data_smithi173_55121_object20982:head# 2024-09-17T16:12:06.421 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1594b689:::benchmark_data_smithi173_55121_object886:head# 2024-09-17T16:12:06.421 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1596db8b:::benchmark_data_smithi173_55121_object34691:head# 2024-09-17T16:12:06.422 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15986c6c:::benchmark_data_smithi173_55121_object33994:head# 2024-09-17T16:12:06.422 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1598e680:::benchmark_data_smithi173_55121_object3883:head# 2024-09-17T16:12:06.422 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1598e8f2:::benchmark_data_smithi173_55121_object21086:head# 2024-09-17T16:12:06.423 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:159bae42:::benchmark_data_smithi173_55121_object25858:head# 2024-09-17T16:12:06.423 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:159cc017:::benchmark_data_smithi173_55121_object14452:head# 2024-09-17T16:12:06.423 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:159d6593:::benchmark_data_smithi173_55121_object21653:head# 2024-09-17T16:12:06.424 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15a5bdca:::benchmark_data_smithi173_55121_object22849:head# 2024-09-17T16:12:06.425 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15a623cb:::benchmark_data_smithi173_55121_object27327:head# 2024-09-17T16:12:06.425 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15a8a1d7:::benchmark_data_smithi173_55121_object29346:head# 2024-09-17T16:12:06.426 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15abc3db:::benchmark_data_smithi173_55121_object14337:head# 2024-09-17T16:12:06.426 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15ac2342:::benchmark_data_smithi173_55121_object18131:head# 2024-09-17T16:12:06.427 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15af5f38:::benchmark_data_smithi173_55121_object23568:head# 2024-09-17T16:12:06.427 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15b1f2d4:::benchmark_data_smithi173_55121_object9588:head# 2024-09-17T16:12:06.428 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15b33cb9:::benchmark_data_smithi173_55121_object5699:head# 2024-09-17T16:12:06.428 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15b4f953:::benchmark_data_smithi173_55121_object31079:head# 2024-09-17T16:12:06.429 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15b7656d:::benchmark_data_smithi173_55121_object32620:head# 2024-09-17T16:12:06.429 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15b8401d:::benchmark_data_smithi173_55121_object25610:head# 2024-09-17T16:12:06.430 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15bbb346:::benchmark_data_smithi173_55121_object22315:head# 2024-09-17T16:12:06.430 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15bf0ae3:::benchmark_data_smithi173_55121_object35624:head# 2024-09-17T16:12:06.430 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15c1671f:::benchmark_data_smithi173_55121_object21759:head# 2024-09-17T16:12:06.431 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15c1b1e0:::benchmark_data_smithi173_55121_object13479:head# 2024-09-17T16:12:06.432 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15c1ba24:::benchmark_data_smithi173_55121_object17116:head# 2024-09-17T16:12:06.432 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15c29503:::benchmark_data_smithi173_55121_object14281:head# 2024-09-17T16:12:06.433 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15c36d7a:::benchmark_data_smithi173_55121_object27993:head# 2024-09-17T16:12:06.433 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15c3dc01:::benchmark_data_smithi173_55121_object33520:head# 2024-09-17T16:12:06.433 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15c4400f:::benchmark_data_smithi173_55121_object19493:head# 2024-09-17T16:12:06.434 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15c46104:::benchmark_data_smithi173_55121_object18846:head# 2024-09-17T16:12:06.435 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15ca2a9f:::benchmark_data_smithi173_55121_object5929:head# 2024-09-17T16:12:06.435 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15cb5d5d:::benchmark_data_smithi173_55121_object11805:head# 2024-09-17T16:12:06.435 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15cd67bd:::benchmark_data_smithi173_55121_object31352:head# 2024-09-17T16:12:06.436 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15cda41d:::benchmark_data_smithi173_55121_object21092:head# 2024-09-17T16:12:06.436 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15d23556:::benchmark_data_smithi173_55121_object7953:head# 2024-09-17T16:12:06.437 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15d31884:::benchmark_data_smithi173_55121_object7374:head# 2024-09-17T16:12:06.437 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15d46985:::benchmark_data_smithi173_55121_object29552:head# 2024-09-17T16:12:06.438 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15d48a51:::benchmark_data_smithi173_55121_object32223:head# 2024-09-17T16:12:06.438 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15d5a31b:::benchmark_data_smithi173_55121_object1734:head# 2024-09-17T16:12:06.439 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15d6f42e:::benchmark_data_smithi173_55121_object28845:head# 2024-09-17T16:12:06.439 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15d7488e:::benchmark_data_smithi173_55121_object137:head# 2024-09-17T16:12:06.439 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15d7c389:::benchmark_data_smithi173_55121_object10596:head# 2024-09-17T16:12:06.440 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15d8a6f9:::benchmark_data_smithi173_55121_object16740:head# 2024-09-17T16:12:06.440 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15d8d2de:::benchmark_data_smithi173_55121_object10411:head# 2024-09-17T16:12:06.441 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15da4597:::benchmark_data_smithi173_55121_object30917:head# 2024-09-17T16:12:06.441 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15db12ac:::benchmark_data_smithi173_55121_object25621:head# 2024-09-17T16:12:06.442 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15db4e3b:::benchmark_data_smithi173_55121_object862:head# 2024-09-17T16:12:06.442 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15dc87c5:::benchmark_data_smithi173_55121_object3661:head# 2024-09-17T16:12:06.443 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15dcc533:::benchmark_data_smithi173_55121_object33236:head# 2024-09-17T16:12:06.444 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15e125af:::benchmark_data_smithi173_55121_object36510:head# 2024-09-17T16:12:06.445 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15e2d8a1:::benchmark_data_smithi173_55121_object11400:head# 2024-09-17T16:12:06.445 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15e5d96a:::benchmark_data_smithi173_55121_object35176:head# 2024-09-17T16:12:06.445 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15e6b2e9:::benchmark_data_smithi173_55121_object37191:head# 2024-09-17T16:12:06.446 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15e7dc18:::benchmark_data_smithi173_55121_object25973:head# 2024-09-17T16:12:06.446 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15e7fa93:::benchmark_data_smithi173_55121_object8816:head# 2024-09-17T16:12:06.447 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15ec01c7:::benchmark_data_smithi173_55121_object28729:head# 2024-09-17T16:12:06.447 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15f0fce0:::benchmark_data_smithi173_55121_object19900:head# 2024-09-17T16:12:06.448 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15f11c6d:::benchmark_data_smithi173_55121_object35359:head# 2024-09-17T16:12:06.448 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15f17254:::benchmark_data_smithi173_55121_object9688:head# 2024-09-17T16:12:06.449 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15f1d3f1:::benchmark_data_smithi173_55121_object2479:head# 2024-09-17T16:12:06.449 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15f29082:::benchmark_data_smithi173_55121_object3970:head# 2024-09-17T16:12:06.449 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15f505d1:::benchmark_data_smithi173_55121_object13358:head# 2024-09-17T16:12:06.450 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15f972ec:::benchmark_data_smithi173_55121_object20633:head# 2024-09-17T16:12:06.450 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:15fdf25a:::benchmark_data_smithi173_55121_object1471:head# 2024-09-17T16:12:06.451 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1603d091:::benchmark_data_smithi173_55121_object26544:head# 2024-09-17T16:12:06.451 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1603dca6:::benchmark_data_smithi173_55121_object35664:head# 2024-09-17T16:12:06.451 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1603f506:::benchmark_data_smithi173_55121_object12509:head# 2024-09-17T16:12:06.452 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16078c08:::benchmark_data_smithi173_55121_object21998:head# 2024-09-17T16:12:06.453 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16090d1c:::benchmark_data_smithi173_55121_object22640:head# 2024-09-17T16:12:06.453 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:160a5ece:::benchmark_data_smithi173_55121_object9716:head# 2024-09-17T16:12:06.454 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:160b032a:::benchmark_data_smithi173_55121_object17332:head# 2024-09-17T16:12:06.454 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:160bb86d:::benchmark_data_smithi173_55121_object7083:head# 2024-09-17T16:12:06.455 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:160e33cd:::benchmark_data_smithi173_55121_object31011:head# 2024-09-17T16:12:06.455 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:160fe7d7:::benchmark_data_smithi173_55121_object896:head# 2024-09-17T16:12:06.456 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1612b85c:::benchmark_data_smithi173_55121_object9781:head# 2024-09-17T16:12:06.456 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1612bea5:::benchmark_data_smithi173_55121_object4497:head# 2024-09-17T16:12:06.457 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1614550c:::benchmark_data_smithi173_55121_object20134:head# 2024-09-17T16:12:06.457 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:161a1ae8:::benchmark_data_smithi173_55121_object24968:head# 2024-09-17T16:12:06.458 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:161ace95:::benchmark_data_smithi173_55121_object36918:head# 2024-09-17T16:12:06.458 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:161ebe1a:::benchmark_data_smithi173_55121_object23423:head# 2024-09-17T16:12:06.459 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:161f1fca:::benchmark_data_smithi173_55121_object7022:head# 2024-09-17T16:12:06.459 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:162130c2:::benchmark_data_smithi173_55121_object8083:head# 2024-09-17T16:12:06.459 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1623665b:::benchmark_data_smithi173_55121_object15543:head# 2024-09-17T16:12:06.460 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16269384:::benchmark_data_smithi173_55121_object15800:head# 2024-09-17T16:12:06.461 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1627ab82:::benchmark_data_smithi173_55121_object24273:head# 2024-09-17T16:12:06.461 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16282b85:::benchmark_data_smithi173_55121_object19472:head# 2024-09-17T16:12:06.462 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:06.462 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16287677:::benchmark_data_smithi173_55121_object37228:head# 2024-09-17T16:12:06.463 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1628fe1d:::benchmark_data_smithi173_55121_object23810:head# 2024-09-17T16:12:06.463 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16295cc3:::benchmark_data_smithi173_55121_object1993:head# 2024-09-17T16:12:06.464 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1629b02d:::benchmark_data_smithi173_55121_object32299:head# 2024-09-17T16:12:06.464 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:162a10b5:::benchmark_data_smithi173_55121_object9892:head# 2024-09-17T16:12:06.465 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:162ba2b4:::benchmark_data_smithi173_55121_object19724:head# 2024-09-17T16:12:06.465 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:162de0d9:::benchmark_data_smithi173_55121_object19136:head# 2024-09-17T16:12:06.466 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1630940f:::benchmark_data_smithi173_55121_object15113:head# 2024-09-17T16:12:06.466 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:163c08ca:::benchmark_data_smithi173_55121_object5772:head# 2024-09-17T16:12:06.466 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:163ca92b:::benchmark_data_smithi173_55121_object21460:head# 2024-09-17T16:12:06.467 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:163ea8af:::benchmark_data_smithi173_55121_object22973:head# 2024-09-17T16:12:06.467 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16400a14:::benchmark_data_smithi173_55121_object33665:head# 2024-09-17T16:12:06.467 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16411fda:::benchmark_data_smithi173_55121_object22509:head# 2024-09-17T16:12:06.468 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:164372fe:::benchmark_data_smithi173_55121_object31755:head# 2024-09-17T16:12:06.468 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1643d834:::benchmark_data_smithi173_55121_object31439:head# 2024-09-17T16:12:06.469 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16456935:::benchmark_data_smithi173_55121_object920:head# 2024-09-17T16:12:06.469 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:164a6b1c:::benchmark_data_smithi173_55121_object14285:head# 2024-09-17T16:12:06.470 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:164c26cc:::benchmark_data_smithi173_55121_object29302:head# 2024-09-17T16:12:06.470 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16504760:::benchmark_data_smithi173_55121_object33739:head# 2024-09-17T16:12:06.471 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1652ab48:::benchmark_data_smithi173_55121_object23918:head# 2024-09-17T16:12:06.471 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:165845c0:::benchmark_data_smithi173_55121_object22711:head# 2024-09-17T16:12:06.471 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1659b638:::benchmark_data_smithi173_55121_object6143:head# 2024-09-17T16:12:06.472 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:165ab9fe:::benchmark_data_smithi173_55121_object8812:head# 2024-09-17T16:12:06.473 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:165d1a11:::benchmark_data_smithi173_55121_object27763:head# 2024-09-17T16:12:06.473 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:165f9939:::benchmark_data_smithi173_55121_object17555:head# 2024-09-17T16:12:06.474 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:165fca20:::benchmark_data_smithi173_55121_object22060:head# 2024-09-17T16:12:06.474 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166178cc:::benchmark_data_smithi173_55121_object7614:head# 2024-09-17T16:12:06.475 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16628fc6:::benchmark_data_smithi173_55121_object11032:head# 2024-09-17T16:12:06.476 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1663d508:::benchmark_data_smithi173_55121_object15169:head# 2024-09-17T16:12:06.476 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16643d38:::benchmark_data_smithi173_55121_object31843:head# 2024-09-17T16:12:06.476 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16684c09:::benchmark_data_smithi173_55121_object20600:head# 2024-09-17T16:12:06.477 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1669b2dc:::benchmark_data_smithi173_55121_object30629:head# 2024-09-17T16:12:06.477 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166aef84:::benchmark_data_smithi173_55121_object9452:head# 2024-09-17T16:12:06.478 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166bee1b:::benchmark_data_smithi173_55121_object26651:head# 2024-09-17T16:12:06.478 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166c19b4:::benchmark_data_smithi173_55121_object24671:head# 2024-09-17T16:12:06.479 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166c3b21:::benchmark_data_smithi173_55121_object36347:head# 2024-09-17T16:12:06.479 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166c68bc:::benchmark_data_smithi173_55121_object5133:head# 2024-09-17T16:12:06.480 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166cf3f0:::benchmark_data_smithi173_55121_object19080:head# 2024-09-17T16:12:06.480 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166da625:::benchmark_data_smithi173_55121_object15328:head# 2024-09-17T16:12:06.481 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166dbe55:::benchmark_data_smithi173_55121_object13404:head# 2024-09-17T16:12:06.481 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166e19df:::benchmark_data_smithi173_55121_object502:head# 2024-09-17T16:12:06.482 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166e81fc:::benchmark_data_smithi173_55121_object33742:head# 2024-09-17T16:12:06.482 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:166f5f75:::benchmark_data_smithi173_55121_object14773:head# 2024-09-17T16:12:06.483 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16713ed8:::benchmark_data_smithi173_55121_object22142:head# 2024-09-17T16:12:06.483 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16733553:::benchmark_data_smithi173_55121_object4786:head# 2024-09-17T16:12:06.483 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1673fecc:::benchmark_data_smithi173_55121_object4085:head# 2024-09-17T16:12:06.484 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16745327:::benchmark_data_smithi173_55121_object14853:head# 2024-09-17T16:12:06.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:06.485 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:06.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:06.486 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1676a521:::benchmark_data_smithi173_55121_object1712:head# 2024-09-17T16:12:06.486 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16777371:::benchmark_data_smithi173_55121_object22736:head# 2024-09-17T16:12:06.486 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:167781d1:::benchmark_data_smithi173_55121_object10262:head# 2024-09-17T16:12:06.486 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16788145:::benchmark_data_smithi173_55121_object7099:head# 2024-09-17T16:12:06.486 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:167b6ba7:::benchmark_data_smithi173_55121_object8634:head# 2024-09-17T16:12:06.487 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:167b8a05:::benchmark_data_smithi173_55121_object35088:head# 2024-09-17T16:12:06.487 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1683c28d:::benchmark_data_smithi173_55121_object14450:head# 2024-09-17T16:12:06.488 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16860cd3:::benchmark_data_smithi173_55121_object17188:head# 2024-09-17T16:12:06.488 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:168a13fb:::benchmark_data_smithi173_55121_object10265:head# 2024-09-17T16:12:06.488 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:168ac31e:::benchmark_data_smithi173_55121_object7645:head# 2024-09-17T16:12:06.489 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:168b324a:::benchmark_data_smithi173_55121_object21539:head# 2024-09-17T16:12:06.489 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:168cd375:::benchmark_data_smithi173_55121_object14605:head# 2024-09-17T16:12:06.490 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:168dd287:::benchmark_data_smithi173_55121_object21257:head# 2024-09-17T16:12:06.490 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:168ea999:::benchmark_data_smithi173_55121_object22087:head# 2024-09-17T16:12:06.490 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:168fa22f:::benchmark_data_smithi173_55121_object10278:head# 2024-09-17T16:12:06.491 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:168faa94:::benchmark_data_smithi173_55121_object35662:head# 2024-09-17T16:12:06.491 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1693711f:::benchmark_data_smithi173_55121_object287:head# 2024-09-17T16:12:06.492 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1699176b:::benchmark_data_smithi173_55121_object5387:head# 2024-09-17T16:12:06.492 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1699d9fb:::benchmark_data_smithi173_55121_object2418:head# 2024-09-17T16:12:06.492 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1699db39:::benchmark_data_smithi173_55121_object11729:head# 2024-09-17T16:12:06.493 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:169a63ef:::benchmark_data_smithi173_55121_object36764:head# 2024-09-17T16:12:06.494 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:169ced6e:::benchmark_data_smithi173_55121_object21267:head# 2024-09-17T16:12:06.494 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:169eb35a:::benchmark_data_smithi173_55121_object267:head# 2024-09-17T16:12:06.494 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:169f5788:::benchmark_data_smithi173_55121_object32273:head# 2024-09-17T16:12:06.495 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:169fc84f:::benchmark_data_smithi173_55121_object30573:head# 2024-09-17T16:12:06.495 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16a1265f:::benchmark_data_smithi173_55121_object2538:head# 2024-09-17T16:12:06.496 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16a3579f:::benchmark_data_smithi173_55121_object22441:head# 2024-09-17T16:12:06.496 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16a386f1:::benchmark_data_smithi173_55121_object20838:head# 2024-09-17T16:12:06.497 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16a43780:::benchmark_data_smithi173_55121_object21629:head# 2024-09-17T16:12:06.497 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16a57c07:::benchmark_data_smithi173_55121_object24834:head# 2024-09-17T16:12:06.497 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16a8ec62:::benchmark_data_smithi173_55121_object11404:head# 2024-09-17T16:12:06.498 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16ad0484:::benchmark_data_smithi173_55121_object20131:head# 2024-09-17T16:12:06.498 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16ad85c9:::benchmark_data_smithi173_55121_object4653:head# 2024-09-17T16:12:06.499 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16ae1157:::benchmark_data_smithi173_55121_object25510:head# 2024-09-17T16:12:06.499 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16b08e19:::benchmark_data_smithi173_55121_object19398:head# 2024-09-17T16:12:06.500 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16b40911:::benchmark_data_smithi173_55121_object27006:head# 2024-09-17T16:12:06.500 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16b77e36:::benchmark_data_smithi173_55121_object4299:head# 2024-09-17T16:12:06.500 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16b9e198:::benchmark_data_smithi173_55121_object20169:head# 2024-09-17T16:12:06.501 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16bbd3d9:::benchmark_data_smithi173_55121_object2270:head# 2024-09-17T16:12:06.501 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16c49a5f:::benchmark_data_smithi173_55121_object6345:head# 2024-09-17T16:12:06.502 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16c90296:::benchmark_data_smithi173_55121_object30517:head# 2024-09-17T16:12:06.502 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16cad7b7:::benchmark_data_smithi173_55121_object11502:head# 2024-09-17T16:12:06.502 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16cbf34b:::benchmark_data_smithi173_55121_object13642:head# 2024-09-17T16:12:06.503 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16cc3eb2:::benchmark_data_smithi173_55121_object8031:head# 2024-09-17T16:12:06.503 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16cf259f:::benchmark_data_smithi173_55121_object27356:head# 2024-09-17T16:12:06.504 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d05e86:::benchmark_data_smithi173_55121_object35483:head# 2024-09-17T16:12:06.504 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d08a82:::benchmark_data_smithi173_55121_object23902:head# 2024-09-17T16:12:06.505 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d25e60:::benchmark_data_smithi173_55121_object28725:head# 2024-09-17T16:12:06.505 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d31c90:::benchmark_data_smithi173_55121_object6064:head# 2024-09-17T16:12:06.505 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d34f37:::benchmark_data_smithi173_55121_object27965:head# 2024-09-17T16:12:06.506 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d385df:::benchmark_data_smithi173_55121_object7653:head# 2024-09-17T16:12:06.506 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d51c10:::benchmark_data_smithi173_55121_object14834:head# 2024-09-17T16:12:06.507 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d52cb0:::benchmark_data_smithi173_55121_object21963:head# 2024-09-17T16:12:06.507 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d53482:::benchmark_data_smithi173_55121_object19627:head# 2024-09-17T16:12:06.508 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d595dc:::benchmark_data_smithi173_55121_object27406:head# 2024-09-17T16:12:06.508 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16d5ef27:::benchmark_data_smithi173_55121_object14471:head# 2024-09-17T16:12:06.509 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16db0734:::benchmark_data_smithi173_55121_object25828:head# 2024-09-17T16:12:06.509 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16dc16a1:::benchmark_data_smithi173_55121_object20032:head# 2024-09-17T16:12:06.509 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16dd69be:::benchmark_data_smithi173_55121_object1840:head# 2024-09-17T16:12:06.510 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16de44c2:::benchmark_data_smithi173_55121_object16965:head# 2024-09-17T16:12:06.510 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16dfa0ef:::benchmark_data_smithi173_55121_object16296:head# 2024-09-17T16:12:06.511 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16e149c9:::benchmark_data_smithi173_55121_object7521:head# 2024-09-17T16:12:06.511 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16e18974:::benchmark_data_smithi173_55121_object98:head# 2024-09-17T16:12:06.512 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16e21efd:::benchmark_data_smithi173_55121_object28686:head# 2024-09-17T16:12:06.512 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16e83c5f:::benchmark_data_smithi173_55121_object6429:head# 2024-09-17T16:12:06.513 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16ed4d33:::benchmark_data_smithi173_55121_object36147:head# 2024-09-17T16:12:06.513 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16ed7483:::benchmark_data_smithi173_55121_object6665:head# 2024-09-17T16:12:06.514 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16eec804:::benchmark_data_smithi173_55121_object19085:head# 2024-09-17T16:12:06.514 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:06 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:12:06.515 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:06 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:12:06.515 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:06 smithi071 ceph-mon[30002]: 2.12 scrub starts 2024-09-17T16:12:06.515 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:06 smithi071 ceph-mon[30002]: osdmap e140: 12 total, 11 up, 12 in 2024-09-17T16:12:06.515 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:06 smithi071 ceph-mon[30002]: pgmap v378: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+clean+scrubbing, 7 active+undersized+degraded, 18 active+clean; 2.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 513 op/s; 9507/129504 objects degraded (7.341%); 1414/129504 objects misplaced (1.092%); 3.7 MiB/s, 58 objects/s recovering 2024-09-17T16:12:06.515 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3008001317' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:06.516 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16eef1bf:::benchmark_data_smithi173_55121_object22560:head# 2024-09-17T16:12:06.516 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16efab65:::benchmark_data_smithi173_55121_object15716:head# 2024-09-17T16:12:06.516 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16f5a35e:::benchmark_data_smithi173_55121_object19499:head# 2024-09-17T16:12:06.516 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16f7f6ef:::benchmark_data_smithi173_55121_object32131:head# 2024-09-17T16:12:06.516 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16f96bb5:::benchmark_data_smithi173_55121_object1247:head# 2024-09-17T16:12:06.517 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16f994e6:::benchmark_data_smithi173_55121_object36889:head# 2024-09-17T16:12:06.517 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16fa4eea:::benchmark_data_smithi173_55121_object438:head# 2024-09-17T16:12:06.517 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16fab4ec:::benchmark_data_smithi173_55121_object28363:head# 2024-09-17T16:12:06.518 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16faf16e:::benchmark_data_smithi173_55121_object24931:head# 2024-09-17T16:12:06.519 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16fb7ea1:::benchmark_data_smithi173_55121_object32715:head# 2024-09-17T16:12:06.519 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:16fcf6dc:::benchmark_data_smithi173_55121_object27033:head# 2024-09-17T16:12:06.519 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:170728af:::benchmark_data_smithi173_55121_object3358:head# 2024-09-17T16:12:06.520 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17091cdd:::benchmark_data_smithi173_55121_object10343:head# 2024-09-17T16:12:06.520 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:170d78c2:::benchmark_data_smithi173_55121_object4102:head# 2024-09-17T16:12:06.520 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:170e001b:::benchmark_data_smithi173_55121_object27505:head# 2024-09-17T16:12:06.521 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:170f216f:::benchmark_data_smithi173_55121_object29668:head# 2024-09-17T16:12:06.521 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1715c6d6:::benchmark_data_smithi173_55121_object628:head# 2024-09-17T16:12:06.522 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17162b3e:::benchmark_data_smithi173_55121_object27226:head# 2024-09-17T16:12:06.522 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17196362:::benchmark_data_smithi173_55121_object22914:head# 2024-09-17T16:12:06.523 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:171abffe:::benchmark_data_smithi173_55121_object11580:head# 2024-09-17T16:12:06.523 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:171b42f0:::benchmark_data_smithi173_55121_object22136:head# 2024-09-17T16:12:06.523 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:171c611c:::benchmark_data_smithi173_55121_object22137:head# 2024-09-17T16:12:06.523 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:171ee01f:::benchmark_data_smithi173_55121_object20510:head# 2024-09-17T16:12:06.524 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:171f4c77:::benchmark_data_smithi173_55121_object27609:head# 2024-09-17T16:12:06.524 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1720bc84:::benchmark_data_smithi173_55121_object3356:head# 2024-09-17T16:12:06.525 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17249465:::benchmark_data_smithi173_55121_object3621:head# 2024-09-17T16:12:06.525 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:172c23cd:::benchmark_data_smithi173_55121_object25154:head# 2024-09-17T16:12:06.526 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:172cf7d4:::benchmark_data_smithi173_55121_object32355:head# 2024-09-17T16:12:06.526 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:172db588:::benchmark_data_smithi173_55121_object36958:head# 2024-09-17T16:12:06.526 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:172f5617:::benchmark_data_smithi173_55121_object6116:head# 2024-09-17T16:12:06.527 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1731585c:::benchmark_data_smithi173_55121_object31520:head# 2024-09-17T16:12:06.527 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17329e60:::benchmark_data_smithi173_55121_object10319:head# 2024-09-17T16:12:06.528 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:173398f8:::benchmark_data_smithi173_55121_object33628:head# 2024-09-17T16:12:06.528 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1734f29d:::benchmark_data_smithi173_55121_object4432:head# 2024-09-17T16:12:06.529 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1735dd4e:::benchmark_data_smithi173_55121_object4209:head# 2024-09-17T16:12:06.529 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1738d9dc:::benchmark_data_smithi173_55121_object27401:head# 2024-09-17T16:12:06.530 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1738f569:::benchmark_data_smithi173_55121_object21218:head# 2024-09-17T16:12:06.530 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:173bcae4:::benchmark_data_smithi173_55121_object244:head# 2024-09-17T16:12:06.530 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:173d14e9:::benchmark_data_smithi173_55121_object29300:head# 2024-09-17T16:12:06.531 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:173fbfa1:::benchmark_data_smithi173_55121_object27225:head# 2024-09-17T16:12:06.531 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17426680:::benchmark_data_smithi173_55121_object8126:head# 2024-09-17T16:12:06.532 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1742d75b:::benchmark_data_smithi173_55121_object1828:head# 2024-09-17T16:12:06.532 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:174340da:::benchmark_data_smithi173_55121_object30089:head# 2024-09-17T16:12:06.533 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1744720c:::benchmark_data_smithi173_55121_object36968:head# 2024-09-17T16:12:06.533 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1744cd97:::benchmark_data_smithi173_55121_object30878:head# 2024-09-17T16:12:06.534 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:174900c7:::benchmark_data_smithi173_55121_object8397:head# 2024-09-17T16:12:06.534 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:174a53aa:::benchmark_data_smithi173_55121_object18426:head# 2024-09-17T16:12:06.535 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:174c4b5f:::benchmark_data_smithi173_55121_object12692:head# 2024-09-17T16:12:06.535 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17532bb7:::benchmark_data_smithi173_55121_object36465:head# 2024-09-17T16:12:06.535 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17537ad6:::benchmark_data_smithi173_55121_object20843:head# 2024-09-17T16:12:06.536 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1754a97a:::benchmark_data_smithi173_55121_object17886:head# 2024-09-17T16:12:06.536 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1755c74e:::benchmark_data_smithi173_55121_object25707:head# 2024-09-17T16:12:06.537 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1756f249:::benchmark_data_smithi173_55121_object23192:head# 2024-09-17T16:12:06.537 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1758bce2:::benchmark_data_smithi173_55121_object22785:head# 2024-09-17T16:12:06.538 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1758d5e4:::benchmark_data_smithi173_55121_object4066:head# 2024-09-17T16:12:06.538 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1758e85e:::benchmark_data_smithi173_55121_object32667:head# 2024-09-17T16:12:06.539 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:175c8447:::benchmark_data_smithi173_55121_object36464:head# 2024-09-17T16:12:06.539 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:175cab30:::benchmark_data_smithi173_55121_object35650:head# 2024-09-17T16:12:06.539 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:175e4d57:::benchmark_data_smithi173_55121_object25141:head# 2024-09-17T16:12:06.540 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1761bc94:::benchmark_data_smithi173_55121_object5871:head# 2024-09-17T16:12:06.540 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17640e40:::benchmark_data_smithi173_55121_object20807:head# 2024-09-17T16:12:06.541 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:176594e8:::benchmark_data_smithi173_55121_object34975:head# 2024-09-17T16:12:06.541 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17676c75:::benchmark_data_smithi173_55121_object10823:head# 2024-09-17T16:12:06.541 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17682204:::benchmark_data_smithi173_55121_object9472:head# 2024-09-17T16:12:06.542 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1768b778:::benchmark_data_smithi173_55121_object20653:head# 2024-09-17T16:12:06.542 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:176dcad2:::benchmark_data_smithi173_55121_object12820:head# 2024-09-17T16:12:06.543 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:176ee98d:::benchmark_data_smithi173_55121_object33477:head# 2024-09-17T16:12:06.543 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:176f985a:::benchmark_data_smithi173_55121_object21560:head# 2024-09-17T16:12:06.543 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1771e225:::benchmark_data_smithi173_55121_object29186:head# 2024-09-17T16:12:06.544 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:177535bd:::benchmark_data_smithi173_55121_object8120:head# 2024-09-17T16:12:06.544 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1776518f:::benchmark_data_smithi173_55121_object25637:head# 2024-09-17T16:12:06.545 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17779a8f:::benchmark_data_smithi173_55121_object19928:head# 2024-09-17T16:12:06.545 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:177937f7:::benchmark_data_smithi173_55121_object33176:head# 2024-09-17T16:12:06.545 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:177d1f3f:::benchmark_data_smithi173_55121_object2473:head# 2024-09-17T16:12:06.546 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:177f52b3:::benchmark_data_smithi173_55121_object17718:head# 2024-09-17T16:12:06.546 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:178038a3:::benchmark_data_smithi173_55121_object16519:head# 2024-09-17T16:12:06.546 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:178153c8:::benchmark_data_smithi173_55121_object28772:head# 2024-09-17T16:12:06.547 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17827585:::benchmark_data_smithi173_55121_object5167:head# 2024-09-17T16:12:06.547 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1782a5dc:::benchmark_data_smithi173_55121_object15670:head# 2024-09-17T16:12:06.548 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17837ebf:::benchmark_data_smithi173_55121_object13209:head# 2024-09-17T16:12:06.548 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1783dc86:::benchmark_data_smithi173_55121_object17520:head# 2024-09-17T16:12:06.549 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17851b8b:::benchmark_data_smithi173_55121_object26270:head# 2024-09-17T16:12:06.549 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1785efdb:::benchmark_data_smithi173_55121_object17821:head# 2024-09-17T16:12:06.550 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17861710:::benchmark_data_smithi173_55121_object24625:head# 2024-09-17T16:12:06.550 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:178889bd:::benchmark_data_smithi173_55121_object31931:head# 2024-09-17T16:12:06.551 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1788f491:::benchmark_data_smithi173_55121_object16324:head# 2024-09-17T16:12:06.551 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:178944c9:::benchmark_data_smithi173_55121_object21184:head# 2024-09-17T16:12:06.551 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:178c636b:::benchmark_data_smithi173_55121_object2288:head# 2024-09-17T16:12:06.552 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:178eed9e:::benchmark_data_smithi173_55121_object7290:head# 2024-09-17T16:12:06.552 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17914229:::benchmark_data_smithi173_55121_object26128:head# 2024-09-17T16:12:06.553 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:179274a6:::benchmark_data_smithi173_55121_object6108:head# 2024-09-17T16:12:06.553 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:179370c0:::benchmark_data_smithi173_55121_object27476:head# 2024-09-17T16:12:06.553 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:179390b6:::benchmark_data_smithi173_55121_object16620:head# 2024-09-17T16:12:06.554 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1799709d:::benchmark_data_smithi173_55121_object34951:head# 2024-09-17T16:12:06.554 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:1799e643:::benchmark_data_smithi173_55121_object24215:head# 2024-09-17T16:12:06.555 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17a0db94:::benchmark_data_smithi173_55121_object9580:head# 2024-09-17T16:12:06.555 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17a8aeb2:::benchmark_data_smithi173_55121_object25127:head# 2024-09-17T16:12:06.555 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17a8e4ab:::benchmark_data_smithi173_55121_object28227:head# 2024-09-17T16:12:06.556 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17a8f7a3:::benchmark_data_smithi173_55121_object25016:head# 2024-09-17T16:12:06.556 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17a9c0ce:::benchmark_data_smithi173_55121_object19064:head# 2024-09-17T16:12:06.557 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17aab4c1:::benchmark_data_smithi173_55121_object26511:head# 2024-09-17T16:12:06.557 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17ab1a53:::benchmark_data_smithi173_55121_object19734:head# 2024-09-17T16:12:06.558 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17ab72e4:::benchmark_data_smithi173_55121_object15945:head# 2024-09-17T16:12:06.558 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17acb3e9:::benchmark_data_smithi173_55121_object8037:head# 2024-09-17T16:12:06.558 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17b22614:::benchmark_data_smithi173_55121_object17660:head# 2024-09-17T16:12:06.559 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17b34a8c:::benchmark_data_smithi173_55121_object14771:head# 2024-09-17T16:12:06.559 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17b406b5:::benchmark_data_smithi173_55121_object31040:head# 2024-09-17T16:12:06.560 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17b713d4:::benchmark_data_smithi173_55121_object29180:head# 2024-09-17T16:12:06.560 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17b8de45:::benchmark_data_smithi173_55121_object33895:head# 2024-09-17T16:12:06.561 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17b91f1a:::benchmark_data_smithi173_55121_object12102:head# 2024-09-17T16:12:06.561 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17c0ed84:::benchmark_data_smithi173_55121_object24002:head# 2024-09-17T16:12:06.562 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17c33ea7:::benchmark_data_smithi173_55121_object11917:head# 2024-09-17T16:12:06.562 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17c4b287:::benchmark_data_smithi173_55121_object766:head# 2024-09-17T16:12:06.563 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17c4e44a:::benchmark_data_smithi173_55121_object9205:head# 2024-09-17T16:12:06.563 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17c88cfe:::benchmark_data_smithi173_55121_object11800:head# 2024-09-17T16:12:06.564 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17c95945:::benchmark_data_smithi173_55121_object22370:head# 2024-09-17T16:12:06.564 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17ca5ebf:::benchmark_data_smithi173_55121_object14599:head# 2024-09-17T16:12:06.564 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17cb88cc:::benchmark_data_smithi173_55121_object1396:head# 2024-09-17T16:12:06.565 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17cee0cb:::benchmark_data_smithi173_55121_object32595:head# 2024-09-17T16:12:06.565 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17ceefa5:::benchmark_data_smithi173_55121_object22959:head# 2024-09-17T16:12:06.566 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17d33506:::benchmark_data_smithi173_55121_object19260:head# 2024-09-17T16:12:06.566 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17d4a1f6:::benchmark_data_smithi173_55121_object22461:head# 2024-09-17T16:12:06.566 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17d543e7:::benchmark_data_smithi173_55121_object10529:head# 2024-09-17T16:12:06.567 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17d9d890:::benchmark_data_smithi173_55121_object21272:head# 2024-09-17T16:12:06.567 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17de257b:::benchmark_data_smithi173_55121_object22100:head# 2024-09-17T16:12:06.568 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17de3be6:::benchmark_data_smithi173_55121_object4833:head# 2024-09-17T16:12:06.569 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17de9d2d:::benchmark_data_smithi173_55121_object15490:head# 2024-09-17T16:12:06.569 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17df3f6b:::benchmark_data_smithi173_55121_object13596:head# 2024-09-17T16:12:06.569 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17df4ade:::benchmark_data_smithi173_55121_object15703:head# 2024-09-17T16:12:06.570 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17e03d2c:::benchmark_data_smithi173_55121_object1347:head# 2024-09-17T16:12:06.570 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17e2dc5c:::benchmark_data_smithi173_55121_object12558:head# 2024-09-17T16:12:06.571 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17e464e4:::benchmark_data_smithi173_55121_object15268:head# 2024-09-17T16:12:06.571 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17e62a85:::benchmark_data_smithi173_55121_object22903:head# 2024-09-17T16:12:06.572 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17ea9bf8:::benchmark_data_smithi173_55121_object15833:head# 2024-09-17T16:12:06.572 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17ec8ef6:::benchmark_data_smithi173_55121_object33423:head# 2024-09-17T16:12:06.573 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17ecce6d:::benchmark_data_smithi173_55121_object27850:head# 2024-09-17T16:12:06.573 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17ee15e1:::benchmark_data_smithi173_55121_object19766:head# 2024-09-17T16:12:06.574 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17eed0d1:::benchmark_data_smithi173_55121_object25350:head# 2024-09-17T16:12:06.574 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17f00105:::benchmark_data_smithi173_55121_object33621:head# 2024-09-17T16:12:06.574 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17f28099:::benchmark_data_smithi173_55121_object17841:head# 2024-09-17T16:12:06.575 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17f3876b:::benchmark_data_smithi173_55121_object25641:head# 2024-09-17T16:12:06.576 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17f72219:::benchmark_data_smithi173_55121_object10165:head# 2024-09-17T16:12:06.576 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17f9649d:::benchmark_data_smithi173_55121_object20104:head# 2024-09-17T16:12:06.576 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17fc2e87:::benchmark_data_smithi173_55121_object15061:head# 2024-09-17T16:12:06.577 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17fd5239:::benchmark_data_smithi173_55121_object14073:head# 2024-09-17T16:12:06.578 INFO:teuthology.orchestra.run.smithi071.stderr:Read #2:17fed838:::benchmark_data_smithi173_55121_object2570:head# 2024-09-17T16:12:06.578 INFO:teuthology.orchestra.run.smithi071.stderr:Export successful 2024-09-17T16:12:06.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:06.608 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:06.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:06.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:06 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:12:06.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:06 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:12:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:06 smithi173 ceph-mon[30530]: 2.12 scrub starts 2024-09-17T16:12:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:06 smithi173 ceph-mon[30530]: osdmap e140: 12 total, 11 up, 12 in 2024-09-17T16:12:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:06 smithi173 ceph-mon[30530]: pgmap v378: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+clean+scrubbing, 7 active+undersized+degraded, 18 active+clean; 2.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 513 op/s; 9507/129504 objects degraded (7.341%); 1414/129504 objects misplaced (1.092%); 3.7 MiB/s, 58 objects/s recovering 2024-09-17T16:12:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3008001317' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:06.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:06.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:06.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:06.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:06 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:12:06.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:06 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:12:06.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:06 smithi017 ceph-mon[27571]: 2.12 scrub starts 2024-09-17T16:12:06.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:06 smithi017 ceph-mon[27571]: osdmap e140: 12 total, 11 up, 12 in 2024-09-17T16:12:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:06 smithi017 ceph-mon[27571]: pgmap v378: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+clean+scrubbing, 7 active+undersized+degraded, 18 active+clean; 2.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 513 op/s; 9507/129504 objects degraded (7.341%); 1414/129504 objects misplaced (1.092%); 3.7 MiB/s, 58 objects/s recovering 2024-09-17T16:12:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3008001317' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:06.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:06.860 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:06.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:06.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:06.983 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:06.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:07.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:12:07.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:07.110 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:07.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:07.211 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:07.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 43595 43579 41.8985 0 - 0.0221508 2024-09-17T16:12:07.233 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:07.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:07.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:07.310 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:12:07.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:07.355 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:07.358 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:07.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:07.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:07.460 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:07.482 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:07.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:07.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:07 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 9507/129504 objects degraded (7.341%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:07.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3008001317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:07.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:07 smithi017 ceph-mon[27571]: osdmap e141: 12 total, 11 up, 12 in 2024-09-17T16:12:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:07 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 9507/129504 objects degraded (7.341%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3008001317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:07 smithi071 ceph-mon[30002]: osdmap e141: 12 total, 11 up, 12 in 2024-09-17T16:12:07.585 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:07.607 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:07.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:07.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.7 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 2.8 2024-09-17T16:12:07.670 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:07 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 9507/129504 objects degraded (7.341%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:07.670 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3008001317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:07.670 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:07 smithi173 ceph-mon[30530]: osdmap e141: 12 total, 11 up, 12 in 2024-09-17T16:12:07.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:07.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:07.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:07.770 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:07.770 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:07.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:07.841 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.7/config 2024-09-17T16:12:07.855 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:07.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:07.872 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:12:07.873 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:12:07.894 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:12:07.894 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:12:07.914 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:12:07.915 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:12:07.935 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:12:07.935 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:12:07.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:07.965 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:12:07.965 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:12:07.985 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:07.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:07.998 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:12:07.998 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:12:08.027 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:12:08.027 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:12:08.027 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:12:08.064 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:12:08.065 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:12:08.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:08.097 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:12:08.098 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:12:08.118 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:08.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:08.132 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:12:08.132 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:12:08.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:12:08.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:12:08.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 44271 44255 41.9038 8.45 0.00381313 0.0238584 2024-09-17T16:12:08.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:08.251 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:08.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:08.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:08.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:08.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:08.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:08.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:12:08.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:08.521 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:08.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:08.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:08 smithi071 ceph-mon[30002]: pgmap v380: 27 pgs: 1 active+clean+remapped+scrubbing, 7 active+undersized+degraded, 19 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 580 op/s; 9624/130914 objects degraded (7.351%); 1414/130914 objects misplaced (1.080%); 3.4 MiB/s, 54 objects/s recovering 2024-09-17T16:12:08.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/854641958' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:08.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:08.647 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:08.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:08.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:08 smithi017 ceph-mon[27571]: pgmap v380: 27 pgs: 1 active+clean+remapped+scrubbing, 7 active+undersized+degraded, 19 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 580 op/s; 9624/130914 objects degraded (7.351%); 1414/130914 objects misplaced (1.080%); 3.4 MiB/s, 54 objects/s recovering 2024-09-17T16:12:08.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/854641958' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:08.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:08.768 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:12:08.772 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:08.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:08.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:08.866 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:12:08.866 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:12:08.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:08.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:08 smithi173 ceph-mon[30530]: pgmap v380: 27 pgs: 1 active+clean+remapped+scrubbing, 7 active+undersized+degraded, 19 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 580 op/s; 9624/130914 objects degraded (7.351%); 1414/130914 objects misplaced (1.080%); 3.4 MiB/s, 54 objects/s recovering 2024-09-17T16:12:08.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/854641958' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:08.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:08.906 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:09.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:09.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:09.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:09.127 INFO:teuthology.orchestra.run.smithi071.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:12:09.134 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:09.157 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:09.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:09.214 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 45835 45819 42.7372 97.75 0.00247562 0.0233883 2024-09-17T16:12:09.259 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:09.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:09.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:09.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:09.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:12:09.378 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:12:09.378 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:12:09.378 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:12:09.378 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:12:09.378 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:12:09.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:09.418 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:09.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:09.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:09.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:09.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:09.607 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:09 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:09.608 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:09 smithi071 ceph-mon[30002]: osdmap e142: 12 total, 11 up, 12 in 2024-09-17T16:12:09.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:09 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:09.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:09 smithi173 ceph-mon[30530]: osdmap e142: 12 total, 11 up, 12 in 2024-09-17T16:12:09.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:09.667 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:09.679 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:09.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:09.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:09 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:09 smithi017 ceph-mon[27571]: osdmap e142: 12 total, 11 up, 12 in 2024-09-17T16:12:09.780 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:09.801 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:09.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:09.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:09.929 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:09.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:10.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:10.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:10.061 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:10.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:10.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:10.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:10.192 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:10.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 47415 47399 43.5608 98.75 0.020872 0.0229489 2024-09-17T16:12:10.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:12:10.293 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:10.316 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:10.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:10.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:10.419 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:10.438 INFO:teuthology.orchestra.run.smithi071.stdout: marking collection for removal 2024-09-17T16:12:10.438 INFO:teuthology.orchestra.run.smithi071.stdout:setting '_remove' omap key 2024-09-17T16:12:10.438 INFO:teuthology.orchestra.run.smithi071.stdout:finish_remove_pgs 2.8_head removing 2.8 2024-09-17T16:12:10.447 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:10.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:10.453 INFO:teuthology.orchestra.run.smithi071.stdout:Remove successful 2024-09-17T16:12:10.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:10 smithi071 ceph-mon[30002]: pgmap v382: 27 pgs: 1 active+clean+remapped+scrubbing, 7 active+undersized+degraded, 19 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 367 op/s; 9777/133197 objects degraded (7.340%); 1414/133197 objects misplaced (1.062%); 2.8 MiB/s, 44 objects/s recovering 2024-09-17T16:12:10.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:10.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:10.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:10.584 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:12:10.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:10 smithi173 ceph-mon[30530]: pgmap v382: 27 pgs: 1 active+clean+remapped+scrubbing, 7 active+undersized+degraded, 19 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 367 op/s; 9777/133197 objects degraded (7.340%); 1414/133197 objects misplaced (1.062%); 2.8 MiB/s, 44 objects/s recovering 2024-09-17T16:12:10.674 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:10.696 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:10.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:10.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:10 smithi017 ceph-mon[27571]: pgmap v382: 27 pgs: 1 active+clean+remapped+scrubbing, 7 active+undersized+degraded, 19 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 367 op/s; 9777/133197 objects degraded (7.340%); 1414/133197 objects misplaced (1.062%); 2.8 MiB/s, 44 objects/s recovering 2024-09-17T16:12:10.797 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:10.823 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:10.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:10.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:10.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:12:10.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:10.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:10.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:11.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:11.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:11.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:11.096 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:12:11.096 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:12:11.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:12:11.096 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:12:11.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.096 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:11694 2.a 2:54727e25:::benchmark_data_smithi173_55121_object11693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:10.833354+0000", 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 60.258562961999999, 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16965316999999999, 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 11694 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.833354+0000", 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.833354+0000", 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.833358+0000", 2024-09-17T16:12:11.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.544e-06 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.833379+0000", 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0849e-05 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.833393+0000", 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4469e-05 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.833400+0000", 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8279999999999997e-06 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.833446+0000", 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5402999999999999e-05 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.833571+0000", 2024-09-17T16:12:11.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012549100000000001 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.833728+0000", 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015650199999999999 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.834493+0000", 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00076544999999999998 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.834822+0000", 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032936499999999997 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.002921+0000", 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16809819100000001 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.002964+0000", 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3887999999999997e-05 2024-09-17T16:12:11.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.003007+0000", 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2190000000000001e-05 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:11701 2.b e102/95 2:dcae3ad2:::benchmark_data_smithi173_55121_object11700:head v 102'724, mlcod=102'721)", 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:10.839023+0000", 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 60.252893884000002, 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17218850099999999, 2024-09-17T16:12:11.103 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.839023+0000", 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.839023+0000", 2024-09-17T16:12:11.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.839023+0000", 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0800000000000004e-07 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.839042+0000", 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8836000000000001e-05 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.010115+0000", 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17107333399999999 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.010120+0000", 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7419999999999997e-06 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.010161+0000", 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0429999999999997e-05 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.010181+0000", 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0483e-05 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.011170+0000", 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098878500000000005 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.011211+0000", 2024-09-17T16:12:11.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1183000000000002e-05 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:13400 2.a 2:5cc2a738:::benchmark_data_smithi173_55121_object13399:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:12.131876+0000", 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.960040782999997, 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18475019600000001, 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.108 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 13400 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.131876+0000", 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.131876+0000", 2024-09-17T16:12:11.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.131877+0000", 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7300000000000005e-07 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.131904+0000", 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7169e-05 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.131920+0000", 2024-09-17T16:12:11.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6606e-05 2024-09-17T16:12:11.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.131930+0000", 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2949999999999999e-06 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.131976+0000", 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6164e-05 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.132088+0000", 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112038 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.132176+0000", 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7783999999999998e-05 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.132806+0000", 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00063076600000000005 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.133260+0000", 2024-09-17T16:12:11.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000453385 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.316531+0000", 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18327074900000001 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.316583+0000", 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2438999999999997e-05 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.316626+0000", 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2928e-05 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:13531 2.a 2:59aa3d24:::benchmark_data_smithi173_55121_object13530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:12.331033+0000", 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.760883481999997, 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50813131099999997, 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.115 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 13531 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331033+0000", 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331033+0000", 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331034+0000", 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0200000000000001e-07 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331051+0000", 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7229999999999999e-05 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331061+0000", 2024-09-17T16:12:11.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.041e-05 2024-09-17T16:12:11.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331067+0000", 2024-09-17T16:12:11.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1730000000000003e-06 2024-09-17T16:12:11.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331115+0000", 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8053999999999998e-05 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331233+0000", 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118643 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331371+0000", 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000137973 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331939+0000", 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056774899999999999 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.332293+0000", 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000353698 2024-09-17T16:12:11.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.839080+0000", 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50678687099999997 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.839127+0000", 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7236e-05 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.839164+0000", 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7472000000000003e-05 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:14940 2.a 2:57eacb6e:::benchmark_data_smithi173_55121_object14939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:14.352400+0000", 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.739516301999998, 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25555733000000003, 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.122 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14940 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.352400+0000", 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.352400+0000", 2024-09-17T16:12:11.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.352401+0000", 2024-09-17T16:12:11.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.23e-07 2024-09-17T16:12:11.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.352419+0000", 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7685000000000001e-05 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.352428+0000", 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.251e-06 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.352434+0000", 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4200000000000004e-06 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.352471+0000", 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.659e-05 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.352553+0000", 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2484999999999994e-05 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.352636+0000", 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2700000000000004e-05 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.353315+0000", 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00067847299999999997 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.353599+0000", 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028405599999999998 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.607882+0000", 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25428302200000003 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.607929+0000", 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7178999999999997e-05 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.607958+0000", 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8745999999999999e-05 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:14949 2.a 2:58bfb129:::benchmark_data_smithi173_55121_object14948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:12:11.129 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:14.606118+0000", 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.485798455000001, 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25790651199999998, 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14949 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.606118+0000", 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.606118+0000", 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.606120+0000", 2024-09-17T16:12:11.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2790000000000001e-06 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.606283+0000", 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016256699999999999 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.606300+0000", 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7370000000000001e-05 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.606311+0000", 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0433e-05 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.606366+0000", 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5452999999999999e-05 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.606479+0000", 2024-09-17T16:12:11.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011243000000000001 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.606576+0000", 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7262999999999994e-05 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.607235+0000", 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065908099999999995 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.614287+0000", 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0070518919999999997 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.863953+0000", 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.249666467 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.863993+0000", 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9971000000000001e-05 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.864025+0000", 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1306000000000001e-05 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:20813 2.a e116/95 ondisk, result = 0)", 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:25.337708+0000", 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 45.754208697999999, 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.445626886, 2024-09-17T16:12:11.136 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.337708+0000", 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.337708+0000", 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.337708+0000", 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5200000000000002e-07 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.337719+0000", 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.027e-05 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.783227+0000", 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44550846599999999 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.783230+0000", 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9229999999999998e-06 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.783259+0000", 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8943999999999999e-05 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.783265+0000", 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9750000000000004e-06 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.783335+0000", 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9856000000000003e-05 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.140 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:21708 2.c 2:368f8462:::benchmark_data_smithi173_55121_object21707:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.630174+0000", 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.461742143000002, 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25529118000000001, 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21708 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.630174+0000", 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.630174+0000", 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.630176+0000", 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2024-09-17T16:12:11.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.630196+0000", 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9763e-05 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.630211+0000", 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5089000000000001e-05 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.630219+0000", 2024-09-17T16:12:11.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6329999999999996e-06 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.630267+0000", 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8807000000000002e-05 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.630354+0000", 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6110000000000001e-05 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.630436+0000", 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2231000000000006e-05 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.631205+0000", 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00076928299999999995 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.632043+0000", 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083821700000000004 2024-09-17T16:12:11.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.885407+0000", 2024-09-17T16:12:11.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25336380800000002 2024-09-17T16:12:11.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.885440+0000", 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2722999999999998e-05 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.885466+0000", 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5686000000000001e-05 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:21708 2.c e117/95 ondisk, result = 0)", 2024-09-17T16:12:11.147 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.631870+0000", 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.460046923999997, 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25360107900000001, 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.631870+0000", 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.631870+0000", 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.631870+0000", 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9100000000000004e-07 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.631873+0000", 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0520000000000002e-06 2024-09-17T16:12:11.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.885357+0000", 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25348356300000002 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.885361+0000", 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8800000000000001e-06 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.885396+0000", 2024-09-17T16:12:11.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5336999999999997e-05 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.885404+0000", 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8660000000000006e-06 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.885471+0000", 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6790000000000003e-05 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.151 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:21746 2.a 2:547291d7:::benchmark_data_smithi173_55121_object21745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.668911+0000", 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.423005121999999, 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20019793299999999, 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21746 2024-09-17T16:12:11.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.668911+0000", 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.668911+0000", 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.668913+0000", 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.328e-06 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.668931+0000", 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8661e-05 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.668941+0000", 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4930000000000004e-06 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.668946+0000", 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8269999999999997e-06 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.668996+0000", 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9932999999999998e-05 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.669079+0000", 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3279e-05 2024-09-17T16:12:11.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:12:11.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.669165+0000", 2024-09-17T16:12:11.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6244999999999998e-05 2024-09-17T16:12:11.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.669863+0000", 2024-09-17T16:12:11.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069793199999999996 2024-09-17T16:12:11.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.670159+0000", 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029630599999999998 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.869042+0000", 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19888287800000001 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.869080+0000", 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8118999999999999e-05 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.869109+0000", 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8932e-05 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:23546 2.a 2:5e3192fa:::benchmark_data_smithi173_55121_object23545:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:28.266515+0000", 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 42.825401378999999, 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20256827599999999, 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.159 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 23546 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.266515+0000", 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.266515+0000", 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.266516+0000", 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.266533+0000", 2024-09-17T16:12:11.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6523000000000001e-05 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.266545+0000", 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1929e-05 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.266551+0000", 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7289999999999997e-06 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.266597+0000", 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6069000000000002e-05 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.266685+0000", 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8108000000000004e-05 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.266769+0000", 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4645e-05 2024-09-17T16:12:11.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.267707+0000", 2024-09-17T16:12:11.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093802600000000001 2024-09-17T16:12:11.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.268008+0000", 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030092299999999998 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.469019+0000", 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20101077000000001 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.469056+0000", 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6377000000000001e-05 2024-09-17T16:12:11.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.469083+0000", 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7840000000000001e-05 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:23582 2.a 2:5fb0e99e:::benchmark_data_smithi173_55121_object23581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:28.317108+0000", 2024-09-17T16:12:11.166 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 42.774808567000001, 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.188323139, 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 23582 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.317108+0000", 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.317108+0000", 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.317109+0000", 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.418e-06 2024-09-17T16:12:11.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.317132+0000", 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2201e-05 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.317144+0000", 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2969000000000001e-05 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.317152+0000", 2024-09-17T16:12:11.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7510000000000008e-06 2024-09-17T16:12:11.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.317206+0000", 2024-09-17T16:12:11.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3795999999999998e-05 2024-09-17T16:12:11.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.317286+0000", 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9948000000000001e-05 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.317437+0000", 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015135699999999999 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.318347+0000", 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090976100000000003 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.318820+0000", 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000472535 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.505371+0000", 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18655089899999999 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.505404+0000", 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3101e-05 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.505431+0000", 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7402999999999998e-05 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.173 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:23802 2.a 2:57420e81:::benchmark_data_smithi173_55121_object23801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:28.715769+0000", 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 42.376147496999998, 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25018458500000001, 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.174 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 23802 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.715769+0000", 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.715769+0000", 2024-09-17T16:12:11.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.715770+0000", 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8299999999999996e-07 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.715789+0000", 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8802999999999999e-05 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.715799+0000", 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0246e-05 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.715804+0000", 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1399999999999999e-06 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.715842+0000", 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8136999999999998e-05 2024-09-17T16:12:11.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.715927+0000", 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4493999999999994e-05 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.716007+0000", 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0364000000000005e-05 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.716587+0000", 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000580427 2024-09-17T16:12:11.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.717011+0000", 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042334499999999999 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.965881+0000", 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24887045099999999 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.965927+0000", 2024-09-17T16:12:11.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5593000000000002e-05 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:28.965954+0000", 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6801999999999999e-05 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25245 2.c 2:3feb22e3:::benchmark_data_smithi173_55121_object25244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.807906+0000", 2024-09-17T16:12:11.180 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.284010123000002, 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32403302699999997, 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25245 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807906+0000", 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807906+0000", 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807907+0000", 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4799999999999997e-07 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807926+0000", 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8859e-05 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807936+0000", 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0071999999999999e-05 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807941+0000", 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1270000000000002e-06 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807980+0000", 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8279999999999999e-05 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808053+0000", 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3112000000000001e-05 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808131+0000", 2024-09-17T16:12:11.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8548999999999996e-05 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808765+0000", 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00063386300000000003 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.809542+0000", 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077723799999999997 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131686+0000", 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.322143974 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131783+0000", 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6248000000000004e-05 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131939+0000", 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015675699999999999 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:25245 2.c e121/95 ondisk, result = 0)", 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.809328+0000", 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.282588222000001, 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32261866900000002, 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.809328+0000", 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.809328+0000", 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.809329+0000", 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:12:11.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.809333+0000", 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7069999999999998e-06 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131494+0000", 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32216098300000001 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131509+0000", 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5313000000000001e-05 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131662+0000", 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015320000000000001 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131677+0000", 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5063e-05 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131947+0000", 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026969400000000001 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:26904 2.a 2:5574ee04:::benchmark_data_smithi173_55121_object26903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.318265+0000", 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.773651157000003, 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45754640299999999, 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.191 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 26904 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318265+0000", 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318265+0000", 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318266+0000", 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.15e-06 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318283+0000", 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6872e-05 2024-09-17T16:12:11.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318293+0000", 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0019e-05 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318298+0000", 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8600000000000001e-06 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318339+0000", 2024-09-17T16:12:11.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0398999999999997e-05 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318420+0000", 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1275999999999999e-05 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318505+0000", 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4759000000000006e-05 2024-09-17T16:12:11.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319324+0000", 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081926600000000003 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319648+0000", 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032444199999999998 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775709+0000", 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45606089100000002 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775767+0000", 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8139e-05 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775812+0000", 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4329999999999997e-05 2024-09-17T16:12:11.197 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:26904 2.a e121/95 ondisk, result = 0)", 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.319630+0000", 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.772286557000001, 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45618829300000002, 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319630+0000", 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319630+0000", 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319630+0000", 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5999999999999999e-07 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319634+0000", 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1300000000000001e-06 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.774978+0000", 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.455344099 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.774982+0000", 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7330000000000003e-06 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775696+0000", 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00071385899999999996 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775704+0000", 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6930000000000007e-06 2024-09-17T16:12:11.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775818+0000", 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114319 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:29670 2.b e129/95 2:d0572430:::benchmark_data_smithi173_55121_object29669:head v 129'1845, mlcod=129'1842)", 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:49.601443+0000", 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.490473676000001, 2024-09-17T16:12:11.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.241005844, 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.601443+0000", 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.601443+0000", 2024-09-17T16:12:11.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.601444+0000", 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6599999999999995e-07 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.601466+0000", 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2903000000000001e-05 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.841286+0000", 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23981946700000001 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.841294+0000", 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9060000000000002e-06 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.841354+0000", 2024-09-17T16:12:11.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9675e-05 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.841377+0000", 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3354999999999999e-05 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.842413+0000", 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001036101 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.842449+0000", 2024-09-17T16:12:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5670999999999998e-05 2024-09-17T16:12:11.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:29700 2.c 2:34e550dc:::benchmark_data_smithi173_55121_object29699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:49.619798+0000", 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.47211871, 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23592313100000001, 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.209 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29700 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.619798+0000", 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.619798+0000", 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.619799+0000", 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4279999999999999e-06 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.619820+0000", 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0387e-05 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.619831+0000", 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1365999999999999e-05 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.619836+0000", 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4559999999999999e-06 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.619881+0000", 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4653000000000001e-05 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.619973+0000", 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2114999999999995e-05 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.620053+0000", 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0055000000000005e-05 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.621021+0000", 2024-09-17T16:12:11.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096740400000000005 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.621469+0000", 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044796000000000001 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.855651+0000", 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23418247 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.855687+0000", 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6094999999999999e-05 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.855721+0000", 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3742000000000003e-05 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.215 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:29776 2.c 2:3146662f:::benchmark_data_smithi173_55121_object29775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:49.884678+0000", 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.207238347000001, 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21201687899999999, 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 29776 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.884678+0000", 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.884678+0000", 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.884679+0000", 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1790000000000001e-06 2024-09-17T16:12:11.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.884701+0000", 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1350999999999999e-05 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.884714+0000", 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3314e-05 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:12:11.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.884721+0000", 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5190000000000003e-06 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.884763+0000", 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1476000000000001e-05 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.884880+0000", 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000116747 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.884969+0000", 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9271000000000006e-05 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.885740+0000", 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000770684 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:49.886760+0000", 2024-09-17T16:12:11.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010202849999999999 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:50.096590+0000", 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20983045 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:50.096667+0000", 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6508000000000007e-05 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:50.096695+0000", 2024-09-17T16:12:11.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8095000000000001e-05 2024-09-17T16:12:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:12:11.222 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:12:11.222 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:12:11.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:11.223 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:11.224 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 48335 48319 43.7627 57.5 0.00167504 0.0228425 2024-09-17T16:12:11.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:11.246 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:11.347 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:11.369 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:11.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:11.425 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:12:11.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:11.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:11.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:11.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:11.621 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:11.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:11.680 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1045794841' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:12:11.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.7 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.8.7 2024-09-17T16:12:11.713 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1045794841' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:12:11.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:11.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1045794841' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:12:11.778 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:11.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:11.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:11.904 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:11.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:11.911 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:12:11.913 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:12:11.942 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.7/config 2024-09-17T16:12:11.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:12:11.945 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:12:11.976 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:12:11.976 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:12:12.003 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:12:12.005 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:12:12.005 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:12:12.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:12.040 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:12.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:12.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:12:12.043 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:12:12.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:12:12.081 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:12:12.107 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:12:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:12:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:12:12.135 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:12:12.135 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:12:12.143 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:12.170 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:12:12.170 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:12:12.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:12.176 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:12.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:12.203 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:12:12.203 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:12:12.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 49867 49851 44.5053 95.75 0.00333272 0.0224626 2024-09-17T16:12:12.231 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:12:12.231 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:12:12.277 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:12.299 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:12.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:12.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:12.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:12.422 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:12.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:12.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:12.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:12.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:12.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:12.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:12.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:12.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:12.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1045794841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:12:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:12 smithi017 ceph-mon[27571]: osdmap e143: 12 total, 11 up, 12 in 2024-09-17T16:12:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:12 smithi017 ceph-mon[27571]: pgmap v384: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 735 op/s; 10527/142743 objects degraded (7.375%); 1549/142743 objects misplaced (1.085%) 2024-09-17T16:12:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3907308390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:12.777 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:12.807 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:12.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1045794841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:12:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:12 smithi071 ceph-mon[30002]: osdmap e143: 12 total, 11 up, 12 in 2024-09-17T16:12:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:12 smithi071 ceph-mon[30002]: pgmap v384: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 735 op/s; 10527/142743 objects degraded (7.375%); 1549/142743 objects misplaced (1.085%) 2024-09-17T16:12:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3907308390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:12.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1045794841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:12:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:12 smithi173 ceph-mon[30530]: osdmap e143: 12 total, 11 up, 12 in 2024-09-17T16:12:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:12 smithi173 ceph-mon[30530]: pgmap v384: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 735 op/s; 10527/142743 objects degraded (7.375%); 1549/142743 objects misplaced (1.085%) 2024-09-17T16:12:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3907308390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:12.909 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:12.935 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:12.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:13.037 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:13.060 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:13.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:13.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:13.175 INFO:teuthology.orchestra.run.smithi071.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:12:13.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:13.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:13.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 51086 51070 44.9514 76.1875 0.00186372 0.0222393 2024-09-17T16:12:13.291 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:13.314 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:13.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:13.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:13.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:13.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:13.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:13.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:13 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 10527/142743 objects degraded (7.375%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:13.537 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:13.546 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:12:13.560 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:12:13.580 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:13.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:13.683 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:13.710 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:13.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:13.813 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:13 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 10527/142743 objects degraded (7.375%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:13.848 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:13.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:13 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 10527/142743 objects degraded (7.375%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:13.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:13.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:13.973 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:13.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:14.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:14.101 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:14.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:14.105 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:14.105 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:14.203 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:14.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 52332 52316 45.4086 77.875 0.0580041 0.0220064 2024-09-17T16:12:14.225 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:14.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:14.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:14.348 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:14.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:14.431 INFO:teuthology.orchestra.run.smithi071.stdout:get_pg_num_history pg_num_history pg_num_history(e132 pg_nums {1={19=1},2={95=16,104=26}} deleted_pools ) 2024-09-17T16:12:14.431 INFO:teuthology.orchestra.run.smithi071.stdout:Importing pgid 2.8 2024-09-17T16:12:14.431 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:100174b5:::benchmark_data_smithi173_55121_object15916:head# 2024-09-17T16:12:14.432 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.433 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:100345db:::benchmark_data_smithi173_55121_object3902:head# 2024-09-17T16:12:14.433 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.433 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1009c2d5:::benchmark_data_smithi173_55121_object1579:head# 2024-09-17T16:12:14.433 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.433 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:100e262e:::benchmark_data_smithi173_55121_object21909:head# 2024-09-17T16:12:14.433 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.433 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:100f668e:::benchmark_data_smithi173_55121_object16593:head# 2024-09-17T16:12:14.433 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.434 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:101006a5:::benchmark_data_smithi173_55121_object2557:head# 2024-09-17T16:12:14.434 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.435 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10135051:::benchmark_data_smithi173_55121_object29933:head# 2024-09-17T16:12:14.435 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.435 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1013bc35:::benchmark_data_smithi173_55121_object33968:head# 2024-09-17T16:12:14.435 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.435 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10187720:::benchmark_data_smithi173_55121_object18603:head# 2024-09-17T16:12:14.435 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.435 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1018cab4:::benchmark_data_smithi173_55121_object22796:head# 2024-09-17T16:12:14.435 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.436 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:101b15df:::benchmark_data_smithi173_55121_object8581:head# 2024-09-17T16:12:14.436 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.436 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:101c462c:::benchmark_data_smithi173_55121_object999:head# 2024-09-17T16:12:14.436 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.436 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:101f2b2f:::benchmark_data_smithi173_55121_object10255:head# 2024-09-17T16:12:14.436 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.437 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1020b299:::benchmark_data_smithi173_55121_object34977:head# 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:102381fe:::benchmark_data_smithi173_55121_object32823:head# 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10276a83:::benchmark_data_smithi173_55121_object27040:head# 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:102827da:::benchmark_data_smithi173_55121_object18772:head# 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1028e588:::benchmark_data_smithi173_55121_object14770:head# 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1029ddce:::benchmark_data_smithi173_55121_object5168:head# 2024-09-17T16:12:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.441 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:102a904b:::benchmark_data_smithi173_55121_object13433:head# 2024-09-17T16:12:14.441 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.441 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:102bd898:::benchmark_data_smithi173_55121_object29440:head# 2024-09-17T16:12:14.441 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.441 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:102d22aa:::benchmark_data_smithi173_55121_object15480:head# 2024-09-17T16:12:14.441 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.441 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:102d68eb:::benchmark_data_smithi173_55121_object27413:head# 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:102f9859:::benchmark_data_smithi173_55121_object1385:head# 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1032c5a6:::benchmark_data_smithi173_55121_object18866:head# 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10330e94:::benchmark_data_smithi173_55121_object34648:head# 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:103355cc:::benchmark_data_smithi173_55121_object8338:head# 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1033c650:::benchmark_data_smithi173_55121_object23876:head# 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.442 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1036ca55:::benchmark_data_smithi173_55121_object21843:head# 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10386a84:::benchmark_data_smithi173_55121_object13150:head# 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:103939a1:::benchmark_data_smithi173_55121_object27128:head# 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:103a6f6a:::benchmark_data_smithi173_55121_object246:head# 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:103b4fac:::benchmark_data_smithi173_55121_object25903:head# 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:103cfd21:::benchmark_data_smithi173_55121_object29452:head# 2024-09-17T16:12:14.443 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.446 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10433941:::benchmark_data_smithi173_55121_object18851:head# 2024-09-17T16:12:14.446 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.446 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10433b8f:::benchmark_data_smithi173_55121_object12771:head# 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:104438a4:::benchmark_data_smithi173_55121_object34557:head# 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1045dd7b:::benchmark_data_smithi173_55121_object15118:head# 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10475a66:::benchmark_data_smithi173_55121_object23692:head# 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10482331:::benchmark_data_smithi173_55121_object21034:head# 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10499626:::benchmark_data_smithi173_55121_object36327:head# 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:104a9d47:::benchmark_data_smithi173_55121_object1567:head# 2024-09-17T16:12:14.447 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:104e3319:::benchmark_data_smithi173_55121_object12608:head# 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:104e9add:::benchmark_data_smithi173_55121_object3895:head# 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:104f1d12:::benchmark_data_smithi173_55121_object37261:head# 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:104f7aaf:::benchmark_data_smithi173_55121_object962:head# 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1052da9a:::benchmark_data_smithi173_55121_object27236:head# 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10536e41:::benchmark_data_smithi173_55121_object27047:head# 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.448 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10574661:::benchmark_data_smithi173_55121_object24753:head# 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1058712e:::benchmark_data_smithi173_55121_object33443:head# 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1058d865:::benchmark_data_smithi173_55121_object15559:head# 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10592cf8:::benchmark_data_smithi173_55121_object12541:head# 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:105c46a3:::benchmark_data_smithi173_55121_object4596:head# 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:105d7009:::benchmark_data_smithi173_55121_object21328:head# 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:105ec88d:::benchmark_data_smithi173_55121_object17726:head# 2024-09-17T16:12:14.449 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1062e034:::benchmark_data_smithi173_55121_object9394:head# 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10641db1:::benchmark_data_smithi173_55121_object35008:head# 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:106829a2:::benchmark_data_smithi173_55121_object35628:head# 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10694afe:::benchmark_data_smithi173_55121_object8209:head# 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:106f1b2b:::benchmark_data_smithi173_55121_object15921:head# 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.465 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1070a2ef:::benchmark_data_smithi173_55121_object35247:head# 2024-09-17T16:12:14.466 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.466 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10712393:::benchmark_data_smithi173_55121_object36392:head# 2024-09-17T16:12:14.466 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.466 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10716a42:::benchmark_data_smithi173_55121_object3080:head# 2024-09-17T16:12:14.466 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.466 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:107320a5:::benchmark_data_smithi173_55121_object17709:head# 2024-09-17T16:12:14.466 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.467 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1075919d:::benchmark_data_smithi173_55121_object20857:head# 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10766d69:::benchmark_data_smithi173_55121_object3331:head# 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1079709b:::benchmark_data_smithi173_55121_object8582:head# 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:107cdf5e:::benchmark_data_smithi173_55121_object33521:head# 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10802558:::benchmark_data_smithi173_55121_object13982:head# 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10829281:::benchmark_data_smithi173_55121_object30466:head# 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:108401c3:::benchmark_data_smithi173_55121_object28384:head# 2024-09-17T16:12:14.468 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10861d74:::benchmark_data_smithi173_55121_object11300:head# 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10864766:::benchmark_data_smithi173_55121_object13974:head# 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:108bf821:::benchmark_data_smithi173_55121_object21667:head# 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:108cb638:::benchmark_data_smithi173_55121_object13724:head# 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:108e8052:::benchmark_data_smithi173_55121_object16785:head# 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10923467:::benchmark_data_smithi173_55121_object34002:head# 2024-09-17T16:12:14.469 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1093652f:::benchmark_data_smithi173_55121_object4978:head# 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1094af92:::benchmark_data_smithi173_55121_object10826:head# 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1094d4dc:::benchmark_data_smithi173_55121_object22767:head# 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1098a9eb:::benchmark_data_smithi173_55121_object9487:head# 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:109b0c05:::benchmark_data_smithi173_55121_object7177:head# 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:109c1f52:::benchmark_data_smithi173_55121_object29035:head# 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.470 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:109caa6d:::benchmark_data_smithi173_55121_object7775:head# 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:109ceb7b:::benchmark_data_smithi173_55121_object18972:head# 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:109dc192:::benchmark_data_smithi173_55121_object8038:head# 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10a457bd:::benchmark_data_smithi173_55121_object16419:head# 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10a6bfa9:::benchmark_data_smithi173_55121_object22211:head# 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10a8c544:::benchmark_data_smithi173_55121_object16437:head# 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.471 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10ab1d31:::benchmark_data_smithi173_55121_object25070:head# 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10ace813:::benchmark_data_smithi173_55121_object12155:head# 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10ad1087:::benchmark_data_smithi173_55121_object29464:head# 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10b05b30:::benchmark_data_smithi173_55121_object12875:head# 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10b2ce18:::benchmark_data_smithi173_55121_object15110:head# 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10b3314d:::benchmark_data_smithi173_55121_object10306:head# 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10b40ebf:::benchmark_data_smithi173_55121_object29623:head# 2024-09-17T16:12:14.472 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10bbbe5c:::benchmark_data_smithi173_55121_object11897:head# 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10c12933:::benchmark_data_smithi173_55121_object5390:head# 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10c1a550:::benchmark_data_smithi173_55121_object16956:head# 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10c209dc:::benchmark_data_smithi173_55121_object15680:head# 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10c21785:::benchmark_data_smithi173_55121_object27310:head# 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10c260e6:::benchmark_data_smithi173_55121_object24149:head# 2024-09-17T16:12:14.473 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10c378d6:::benchmark_data_smithi173_55121_object19927:head# 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10c4f66f:::benchmark_data_smithi173_55121_object7715:head# 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10c744c2:::benchmark_data_smithi173_55121_object25859:head# 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10cc1c52:::benchmark_data_smithi173_55121_object32050:head# 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10d00465:::benchmark_data_smithi173_55121_object1057:head# 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10d390ea:::benchmark_data_smithi173_55121_object2914:head# 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.474 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10d408b8:::benchmark_data_smithi173_55121_object11813:head# 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10d4d99b:::benchmark_data_smithi173_55121_object29445:head# 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10d52c61:::benchmark_data_smithi173_55121_object30930:head# 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10d6b055:::benchmark_data_smithi173_55121_object24575:head# 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10d84cd1:::benchmark_data_smithi173_55121_object19440:head# 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10da1907:::benchmark_data_smithi173_55121_object13176:head# 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10daa35d:::benchmark_data_smithi173_55121_object1601:head# 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.475 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10df007b:::benchmark_data_smithi173_55121_object16331:head# 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10df03bd:::benchmark_data_smithi173_55121_object22462:head# 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10df1dbd:::benchmark_data_smithi173_55121_object14043:head# 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10e0e2f3:::benchmark_data_smithi173_55121_object36713:head# 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10e2d356:::benchmark_data_smithi173_55121_object32082:head# 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10e3c86c:::benchmark_data_smithi173_55121_object9749:head# 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10e47ad2:::benchmark_data_smithi173_55121_object4697:head# 2024-09-17T16:12:14.476 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10e759a0:::benchmark_data_smithi173_55121_object11574:head# 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10e89c34:::benchmark_data_smithi173_55121_object23636:head# 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10e8f986:::benchmark_data_smithi173_55121_object6106:head# 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10ea2670:::benchmark_data_smithi173_55121_object15688:head# 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10ec9938:::benchmark_data_smithi173_55121_object25605:head# 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10eec2ca:::benchmark_data_smithi173_55121_object28522:head# 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.477 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10f21596:::benchmark_data_smithi173_55121_object27830:head# 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10f641ef:::benchmark_data_smithi173_55121_object25363:head# 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10f7444f:::benchmark_data_smithi173_55121_object3349:head# 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10f97f0d:::benchmark_data_smithi173_55121_object32103:head# 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10f9ef05:::benchmark_data_smithi173_55121_object21520:head# 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10fa22a7:::benchmark_data_smithi173_55121_object36197:head# 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10fa99d6:::benchmark_data_smithi173_55121_object24653:head# 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10fae88a:::benchmark_data_smithi173_55121_object15660:head# 2024-09-17T16:12:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.479 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10fb345f:::benchmark_data_smithi173_55121_object21739:head# 2024-09-17T16:12:14.479 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.479 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10fc8a2e:::benchmark_data_smithi173_55121_object31295:head# 2024-09-17T16:12:14.479 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.479 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:10fccbc0:::benchmark_data_smithi173_55121_object7373:head# 2024-09-17T16:12:14.479 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.479 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1100f24f:::benchmark_data_smithi173_55121_object9990:head# 2024-09-17T16:12:14.479 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.479 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11050221:::benchmark_data_smithi173_55121_object22559:head# 2024-09-17T16:12:14.479 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:14.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:14.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:14.505 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11060d59:::benchmark_data_smithi173_55121_object16996:head# 2024-09-17T16:12:14.505 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.505 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11078ef1:::benchmark_data_smithi173_55121_object25927:head# 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1107c42a:::benchmark_data_smithi173_55121_object33052:head# 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1108567e:::benchmark_data_smithi173_55121_object22290:head# 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11093ca2:::benchmark_data_smithi173_55121_object14218:head# 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:110f4ff1:::benchmark_data_smithi173_55121_object30909:head# 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1111a0b5:::benchmark_data_smithi173_55121_object21142:head# 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:111767b2:::benchmark_data_smithi173_55121_object15003:head# 2024-09-17T16:12:14.506 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.507 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:111986ca:::benchmark_data_smithi173_55121_object7837:head# 2024-09-17T16:12:14.507 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.507 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:111b0465:::benchmark_data_smithi173_55121_object25025:head# 2024-09-17T16:12:14.507 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.507 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:111e1035:::benchmark_data_smithi173_55121_object31349:head# 2024-09-17T16:12:14.507 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.507 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11218576:::benchmark_data_smithi173_55121_object12312:head# 2024-09-17T16:12:14.507 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.507 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1125aaa5:::benchmark_data_smithi173_55121_object30015:head# 2024-09-17T16:12:14.507 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.508 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1127da0c:::benchmark_data_smithi173_55121_object23374:head# 2024-09-17T16:12:14.508 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.508 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11288b6b:::benchmark_data_smithi173_55121_object19554:head# 2024-09-17T16:12:14.508 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.508 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1128e2e4:::benchmark_data_smithi173_55121_object33347:head# 2024-09-17T16:12:14.508 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:112a4083:::benchmark_data_smithi173_55121_object14336:head# 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:112af029:::benchmark_data_smithi173_55121_object25165:head# 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:112b76e5:::benchmark_data_smithi173_55121_object30447:head# 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:112bb137:::benchmark_data_smithi173_55121_object2736:head# 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:112bdb5d:::benchmark_data_smithi173_55121_object19127:head# 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:112dbc65:::benchmark_data_smithi173_55121_object30258:head# 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.509 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11312814:::benchmark_data_smithi173_55121_object20801:head# 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11331096:::benchmark_data_smithi173_55121_object31657:head# 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1134af16:::benchmark_data_smithi173_55121_object17720:head# 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1134b4a4:::benchmark_data_smithi173_55121_object12783:head# 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1136105f:::benchmark_data_smithi173_55121_object36772:head# 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1136297c:::benchmark_data_smithi173_55121_object7537:head# 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.510 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:113799ba:::benchmark_data_smithi173_55121_object9343:head# 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11380459:::benchmark_data_smithi173_55121_object17549:head# 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:113877ce:::benchmark_data_smithi173_55121_object7128:head# 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1139bfb1:::benchmark_data_smithi173_55121_object339:head# 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1139df8b:::benchmark_data_smithi173_55121_object1638:head# 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:113cbd9a:::benchmark_data_smithi173_55121_object34327:head# 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:113efad0:::benchmark_data_smithi173_55121_object19658:head# 2024-09-17T16:12:14.511 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1140d68c:::benchmark_data_smithi173_55121_object2577:head# 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1141b240:::benchmark_data_smithi173_55121_object20873:head# 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:114216c9:::benchmark_data_smithi173_55121_object4356:head# 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1143d6bb:::benchmark_data_smithi173_55121_object29373:head# 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1144a22c:::benchmark_data_smithi173_55121_object28374:head# 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1144ef4a:::benchmark_data_smithi173_55121_object644:head# 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.512 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11457c24:::benchmark_data_smithi173_55121_object36485:head# 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1146bec9:::benchmark_data_smithi173_55121_object22464:head# 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:114eaf3a:::benchmark_data_smithi173_55121_object14595:head# 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:114edc28:::benchmark_data_smithi173_55121_object6768:head# 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:114f726b:::benchmark_data_smithi173_55121_object30533:head# 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:114fe870:::benchmark_data_smithi173_55121_object28364:head# 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11533f3e:::benchmark_data_smithi173_55121_object22551:head# 2024-09-17T16:12:14.513 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11540543:::benchmark_data_smithi173_55121_object21928:head# 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:115459c4:::benchmark_data_smithi173_55121_object10849:head# 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11545d29:::benchmark_data_smithi173_55121_object26980:head# 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1155ca6c:::benchmark_data_smithi173_55121_object25561:head# 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11572e57:::benchmark_data_smithi173_55121_object10873:head# 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:115a687b:::benchmark_data_smithi173_55121_object4205:head# 2024-09-17T16:12:14.514 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11602637:::benchmark_data_smithi173_55121_object24463:head# 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11607be0:::benchmark_data_smithi173_55121_object26655:head# 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11616751:::benchmark_data_smithi173_55121_object17516:head# 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11619824:::benchmark_data_smithi173_55121_object19710:head# 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:116bab10:::benchmark_data_smithi173_55121_object1646:head# 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:116c65ee:::benchmark_data_smithi173_55121_object7241:head# 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.515 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:116cdce5:::benchmark_data_smithi173_55121_object10522:head# 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:116d1a41:::benchmark_data_smithi173_55121_object32407:head# 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:116f4a47:::benchmark_data_smithi173_55121_object6801:head# 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:116f9105:::benchmark_data_smithi173_55121_object21174:head# 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11703d25:::benchmark_data_smithi173_55121_object31365:head# 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11707bab:::benchmark_data_smithi173_55121_object4752:head# 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11715783:::benchmark_data_smithi173_55121_object13608:head# 2024-09-17T16:12:14.516 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1171659a:::benchmark_data_smithi173_55121_object21170:head# 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1172408f:::benchmark_data_smithi173_55121_object937:head# 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1175001e:::benchmark_data_smithi173_55121_object37247:head# 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11788134:::benchmark_data_smithi173_55121_object16542:head# 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:117918e4:::benchmark_data_smithi173_55121_object203:head# 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1179cb8c:::benchmark_data_smithi173_55121_object25615:head# 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.517 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:117acac1:::benchmark_data_smithi173_55121_object13515:head# 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:117ddb01:::benchmark_data_smithi173_55121_object3037:head# 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1180789e:::benchmark_data_smithi173_55121_object35760:head# 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11840406:::benchmark_data_smithi173_55121_object12633:head# 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11845e7e:::benchmark_data_smithi173_55121_object8796:head# 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1185abc7:::benchmark_data_smithi173_55121_object19359:head# 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.518 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1186eeb5:::benchmark_data_smithi173_55121_object4321:head# 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:118c045a:::benchmark_data_smithi173_55121_object18368:head# 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:118d7109:::benchmark_data_smithi173_55121_object26201:head# 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:118dbf4b:::benchmark_data_smithi173_55121_object36662:head# 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:118f4536:::benchmark_data_smithi173_55121_object31093:head# 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11910c6c:::benchmark_data_smithi173_55121_object14324:head# 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1191cf26:::benchmark_data_smithi173_55121_object14579:head# 2024-09-17T16:12:14.519 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11923514:::benchmark_data_smithi173_55121_object27659:head# 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1193bd12:::benchmark_data_smithi173_55121_object32150:head# 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1193f619:::benchmark_data_smithi173_55121_object29465:head# 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:119854a9:::benchmark_data_smithi173_55121_object31527:head# 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11994494:::benchmark_data_smithi173_55121_object2612:head# 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:119959ce:::benchmark_data_smithi173_55121_object6249:head# 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.520 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:119d436d:::benchmark_data_smithi173_55121_object15675:head# 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a08d92:::benchmark_data_smithi173_55121_object37240:head# 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a0de21:::benchmark_data_smithi173_55121_object16642:head# 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a53155:::benchmark_data_smithi173_55121_object26096:head# 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a5bc3b:::benchmark_data_smithi173_55121_object10535:head# 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a6461f:::benchmark_data_smithi173_55121_object27802:head# 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a6f74e:::benchmark_data_smithi173_55121_object1258:head# 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.521 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a8013e:::benchmark_data_smithi173_55121_object9308:head# 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a85dd4:::benchmark_data_smithi173_55121_object1255:head# 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a8c3ff:::benchmark_data_smithi173_55121_object14007:head# 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a9b00c:::benchmark_data_smithi173_55121_object27826:head# 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11a9fb00:::benchmark_data_smithi173_55121_object22363:head# 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11aa82a8:::benchmark_data_smithi173_55121_object20347:head# 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.522 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11ab2df1:::benchmark_data_smithi173_55121_object32388:head# 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11ab4bc9:::benchmark_data_smithi173_55121_object33544:head# 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11ac8476:::benchmark_data_smithi173_55121_object28459:head# 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11af3e15:::benchmark_data_smithi173_55121_object25568:head# 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11affa20:::benchmark_data_smithi173_55121_object27427:head# 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11b1e9d7:::benchmark_data_smithi173_55121_object36378:head# 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11b3774f:::benchmark_data_smithi173_55121_object20054:head# 2024-09-17T16:12:14.523 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11b43939:::benchmark_data_smithi173_55121_object13559:head# 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11b4cd74:::benchmark_data_smithi173_55121_object6971:head# 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11b5f83b:::benchmark_data_smithi173_55121_object26075:head# 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11b8e195:::benchmark_data_smithi173_55121_object12485:head# 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11b90522:::benchmark_data_smithi173_55121_object20497:head# 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11b92366:::benchmark_data_smithi173_55121_object11095:head# 2024-09-17T16:12:14.524 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11bd353b:::benchmark_data_smithi173_55121_object26106:head# 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11beacd3:::benchmark_data_smithi173_55121_object4533:head# 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11c413d9:::benchmark_data_smithi173_55121_object15472:head# 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11c64b56:::benchmark_data_smithi173_55121_object6866:head# 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11c65cdf:::benchmark_data_smithi173_55121_object3179:head# 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11ca2a66:::benchmark_data_smithi173_55121_object2471:head# 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.525 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11cb577c:::benchmark_data_smithi173_55121_object19537:head# 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11cb94e6:::benchmark_data_smithi173_55121_object29570:head# 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11ce4de8:::benchmark_data_smithi173_55121_object6336:head# 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11cfba28:::benchmark_data_smithi173_55121_object6785:head# 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11d15fbb:::benchmark_data_smithi173_55121_object25558:head# 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11d16de7:::benchmark_data_smithi173_55121_object1060:head# 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.526 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11d3803b:::benchmark_data_smithi173_55121_object8274:head# 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11d41faf:::benchmark_data_smithi173_55121_object33587:head# 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11d5203f:::benchmark_data_smithi173_55121_object26803:head# 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11d6d7f7:::benchmark_data_smithi173_55121_object16562:head# 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11d79770:::benchmark_data_smithi173_55121_object19433:head# 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11d7e74f:::benchmark_data_smithi173_55121_object22168:head# 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11daecfa:::benchmark_data_smithi173_55121_object7116:head# 2024-09-17T16:12:14.527 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11dd7adf:::benchmark_data_smithi173_55121_object29700:head# 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11ddafe0:::benchmark_data_smithi173_55121_object1498:head# 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11dfb669:::benchmark_data_smithi173_55121_object12378:head# 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11e3317e:::benchmark_data_smithi173_55121_object3379:head# 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11e4271e:::benchmark_data_smithi173_55121_object21569:head# 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11e6838b:::benchmark_data_smithi173_55121_object7682:head# 2024-09-17T16:12:14.528 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11e748e7:::benchmark_data_smithi173_55121_object9977:head# 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11e91841:::benchmark_data_smithi173_55121_object1152:head# 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11edcfb3:::benchmark_data_smithi173_55121_object24553:head# 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11f09469:::benchmark_data_smithi173_55121_object21536:head# 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11f1a343:::benchmark_data_smithi173_55121_object5070:head# 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11f47931:::benchmark_data_smithi173_55121_object19704:head# 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.529 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11f7854e:::benchmark_data_smithi173_55121_object19351:head# 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11f9ab91:::benchmark_data_smithi173_55121_object27973:head# 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11fe8bcc:::benchmark_data_smithi173_55121_object26087:head# 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:11ffbea2:::benchmark_data_smithi173_55121_object36511:head# 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1202d30e:::benchmark_data_smithi173_55121_object19573:head# 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:120439c0:::benchmark_data_smithi173_55121_object31936:head# 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12059d30:::benchmark_data_smithi173_55121_object21811:head# 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.530 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12082966:::benchmark_data_smithi173_55121_object24138:head# 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1208e078:::benchmark_data_smithi173_55121_object12051:head# 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:120983ff:::benchmark_data_smithi173_55121_object821:head# 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:120dc368:::benchmark_data_smithi173_55121_object13018:head# 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:120e19d4:::benchmark_data_smithi173_55121_object36605:head# 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:120e376d:::benchmark_data_smithi173_55121_object18455:head# 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:120ebb70:::benchmark_data_smithi173_55121_object31624:head# 2024-09-17T16:12:14.531 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1215d5c2:::benchmark_data_smithi173_55121_object13782:head# 2024-09-17T16:12:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12165b65:::benchmark_data_smithi173_55121_object25629:head# 2024-09-17T16:12:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:121934bf:::benchmark_data_smithi173_55121_object32945:head# 2024-09-17T16:12:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1219f66f:::benchmark_data_smithi173_55121_object6800:head# 2024-09-17T16:12:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:121c730c:::benchmark_data_smithi173_55121_object13381:head# 2024-09-17T16:12:14.532 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:121d78a2:::benchmark_data_smithi173_55121_object3390:head# 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:121f6650:::benchmark_data_smithi173_55121_object8119:head# 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12211b7c:::benchmark_data_smithi173_55121_object30497:head# 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1221b570:::benchmark_data_smithi173_55121_object17940:head# 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:122640fc:::benchmark_data_smithi173_55121_object11722:head# 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:122d1f7c:::benchmark_data_smithi173_55121_object4427:head# 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:122f6ea9:::benchmark_data_smithi173_55121_object29750:head# 2024-09-17T16:12:14.533 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.534 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1230c4cc:::benchmark_data_smithi173_55121_object29105:head# 2024-09-17T16:12:14.534 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.558 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1231046c:::benchmark_data_smithi173_55121_object7680:head# 2024-09-17T16:12:14.558 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.558 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1231b636:::benchmark_data_smithi173_55121_object15812:head# 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12370c9a:::benchmark_data_smithi173_55121_object10850:head# 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1239a6b0:::benchmark_data_smithi173_55121_object57:head# 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:123acad0:::benchmark_data_smithi173_55121_object12041:head# 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:123fe9fd:::benchmark_data_smithi173_55121_object32636:head# 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12404c95:::benchmark_data_smithi173_55121_object29275:head# 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.559 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12416356:::benchmark_data_smithi173_55121_object2066:head# 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1242a6d2:::benchmark_data_smithi173_55121_object27616:head# 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:124345ad:::benchmark_data_smithi173_55121_object36269:head# 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12463cca:::benchmark_data_smithi173_55121_object9187:head# 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12465984:::benchmark_data_smithi173_55121_object29292:head# 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:124a6cd0:::benchmark_data_smithi173_55121_object18948:head# 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:124ae083:::benchmark_data_smithi173_55121_object15988:head# 2024-09-17T16:12:14.560 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:124b77d8:::benchmark_data_smithi173_55121_object16238:head# 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:124bd779:::benchmark_data_smithi173_55121_object35065:head# 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:124c8c24:::benchmark_data_smithi173_55121_object18831:head# 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:124dc76a:::benchmark_data_smithi173_55121_object10840:head# 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1252c8de:::benchmark_data_smithi173_55121_object12508:head# 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12536015:::benchmark_data_smithi173_55121_object21439:head# 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.561 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12550c64:::benchmark_data_smithi173_55121_object31915:head# 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:125704dd:::benchmark_data_smithi173_55121_object34474:head# 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1259888d:::benchmark_data_smithi173_55121_object25403:head# 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:125d9e28:::benchmark_data_smithi173_55121_object13663:head# 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:126314cf:::benchmark_data_smithi173_55121_object26349:head# 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12639e3f:::benchmark_data_smithi173_55121_object6133:head# 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:126735c9:::benchmark_data_smithi173_55121_object4213:head# 2024-09-17T16:12:14.562 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1267e718:::benchmark_data_smithi173_55121_object20325:head# 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1268d5e5:::benchmark_data_smithi173_55121_object27636:head# 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1269a582:::benchmark_data_smithi173_55121_object7542:head# 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1269fbb9:::benchmark_data_smithi173_55121_object5005:head# 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:126a76b9:::benchmark_data_smithi173_55121_object28908:head# 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:126b46bb:::benchmark_data_smithi173_55121_object14772:head# 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.563 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:126baddf:::benchmark_data_smithi173_55121_object15291:head# 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:126cd91e:::benchmark_data_smithi173_55121_object31104:head# 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:126d1554:::benchmark_data_smithi173_55121_object16154:head# 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:126ecc20:::benchmark_data_smithi173_55121_object11801:head# 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:126f8b5d:::benchmark_data_smithi173_55121_object28038:head# 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1270e3c1:::benchmark_data_smithi173_55121_object24047:head# 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.564 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1277095f:::benchmark_data_smithi173_55121_object23454:head# 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12798dd0:::benchmark_data_smithi173_55121_object11492:head# 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:127ac868:::benchmark_data_smithi173_55121_object6525:head# 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:127aed42:::benchmark_data_smithi173_55121_object15822:head# 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:127bdc13:::benchmark_data_smithi173_55121_object24856:head# 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:127d60ea:::benchmark_data_smithi173_55121_object17998:head# 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12808f27:::benchmark_data_smithi173_55121_object3453:head# 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.565 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1281c135:::benchmark_data_smithi173_55121_object3040:head# 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:128434e3:::benchmark_data_smithi173_55121_object33016:head# 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12866bc0:::benchmark_data_smithi173_55121_object18530:head# 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1286c8e1:::benchmark_data_smithi173_55121_object26586:head# 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12887750:::benchmark_data_smithi173_55121_object2814:head# 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12896178:::benchmark_data_smithi173_55121_object12223:head# 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12898907:::benchmark_data_smithi173_55121_object1846:head# 2024-09-17T16:12:14.566 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:128a214e:::benchmark_data_smithi173_55121_object6060:head# 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12924156:::benchmark_data_smithi173_55121_object7430:head# 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1292ec9f:::benchmark_data_smithi173_55121_object15477:head# 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1296b073:::benchmark_data_smithi173_55121_object26979:head# 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:129743fd:::benchmark_data_smithi173_55121_object36360:head# 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12992428:::benchmark_data_smithi173_55121_object31459:head# 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.567 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:129a13c8:::benchmark_data_smithi173_55121_object14637:head# 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:129d74bf:::benchmark_data_smithi173_55121_object20310:head# 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:129da034:::benchmark_data_smithi173_55121_object32468:head# 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12a08404:::benchmark_data_smithi173_55121_object8420:head# 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12a634f1:::benchmark_data_smithi173_55121_object21289:head# 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12a72343:::benchmark_data_smithi173_55121_object11223:head# 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12aa6921:::benchmark_data_smithi173_55121_object18283:head# 2024-09-17T16:12:14.568 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12ad186e:::benchmark_data_smithi173_55121_object3025:head# 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12ad3c1c:::benchmark_data_smithi173_55121_object15738:head# 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12b59da0:::benchmark_data_smithi173_55121_object31335:head# 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12b664ff:::benchmark_data_smithi173_55121_object8380:head# 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12b72d19:::benchmark_data_smithi173_55121_object4336:head# 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12b85bbe:::benchmark_data_smithi173_55121_object1126:head# 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.569 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12b89de9:::benchmark_data_smithi173_55121_object12882:head# 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12bb83e8:::benchmark_data_smithi173_55121_object29842:head# 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12bd3c32:::benchmark_data_smithi173_55121_object9239:head# 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12bdfde8:::benchmark_data_smithi173_55121_object8460:head# 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12be3a6c:::benchmark_data_smithi173_55121_object16200:head# 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12bff709:::benchmark_data_smithi173_55121_object26537:head# 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12c1c1fb:::benchmark_data_smithi173_55121_object19701:head# 2024-09-17T16:12:14.570 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12c22b74:::benchmark_data_smithi173_55121_object4762:head# 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12c65e03:::benchmark_data_smithi173_55121_object2222:head# 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12c6fc34:::benchmark_data_smithi173_55121_object10641:head# 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12c7fea8:::benchmark_data_smithi173_55121_object32871:head# 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12c95f32:::benchmark_data_smithi173_55121_object2903:head# 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12ca7da7:::benchmark_data_smithi173_55121_object3438:head# 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12ce0422:::benchmark_data_smithi173_55121_object15479:head# 2024-09-17T16:12:14.571 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12ce49c4:::benchmark_data_smithi173_55121_object15426:head# 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12cefbda:::benchmark_data_smithi173_55121_object34381:head# 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12cf1253:::benchmark_data_smithi173_55121_object35597:head# 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12d0230e:::benchmark_data_smithi173_55121_object946:head# 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12d11b17:::benchmark_data_smithi173_55121_object20286:head# 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12d3c795:::benchmark_data_smithi173_55121_object20601:head# 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12d3f1a7:::benchmark_data_smithi173_55121_object32625:head# 2024-09-17T16:12:14.572 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12d4e766:::benchmark_data_smithi173_55121_object35743:head# 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12d8ffb8:::benchmark_data_smithi173_55121_object36970:head# 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12ddaee1:::benchmark_data_smithi173_55121_object24220:head# 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12deb5fe:::benchmark_data_smithi173_55121_object5295:head# 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12dffba9:::benchmark_data_smithi173_55121_object19009:head# 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12e0166e:::benchmark_data_smithi173_55121_object20290:head# 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12e096fb:::benchmark_data_smithi173_55121_object20882:head# 2024-09-17T16:12:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12e249b6:::benchmark_data_smithi173_55121_object28925:head# 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12e24c44:::benchmark_data_smithi173_55121_object25688:head# 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12e756ef:::benchmark_data_smithi173_55121_object34145:head# 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12e759dd:::benchmark_data_smithi173_55121_object6401:head# 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12e79435:::benchmark_data_smithi173_55121_object35091:head# 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12e80ef3:::benchmark_data_smithi173_55121_object15424:head# 2024-09-17T16:12:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12e8c5f9:::benchmark_data_smithi173_55121_object759:head# 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12e90b58:::benchmark_data_smithi173_55121_object10640:head# 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12eb577c:::benchmark_data_smithi173_55121_object35163:head# 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12ed1e55:::benchmark_data_smithi173_55121_object19428:head# 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12f24834:::benchmark_data_smithi173_55121_object15273:head# 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12f24854:::benchmark_data_smithi173_55121_object26185:head# 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.575 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12f7a138:::benchmark_data_smithi173_55121_object7434:head# 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12f89a21:::benchmark_data_smithi173_55121_object9929:head# 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12f89af2:::benchmark_data_smithi173_55121_object20991:head# 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12f95884:::benchmark_data_smithi173_55121_object12431:head# 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12fc4b8e:::benchmark_data_smithi173_55121_object407:head# 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12fee76b:::benchmark_data_smithi173_55121_object36730:head# 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:12ffc9a6:::benchmark_data_smithi173_55121_object34699:head# 2024-09-17T16:12:14.576 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1308fdc8:::benchmark_data_smithi173_55121_object32780:head# 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1309dc24:::benchmark_data_smithi173_55121_object29470:head# 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:130a285f:::benchmark_data_smithi173_55121_object14406:head# 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:130a701f:::benchmark_data_smithi173_55121_object23851:head# 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:130d71a6:::benchmark_data_smithi173_55121_object6602:head# 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:130e6c8c:::benchmark_data_smithi173_55121_object31317:head# 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.577 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:130f3bfd:::benchmark_data_smithi173_55121_object29520:head# 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:130f93d2:::benchmark_data_smithi173_55121_object2848:head# 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13107f38:::benchmark_data_smithi173_55121_object6618:head# 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1310af39:::benchmark_data_smithi173_55121_object27829:head# 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1310dc98:::benchmark_data_smithi173_55121_object32383:head# 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13135af7:::benchmark_data_smithi173_55121_object24870:head# 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.578 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:131375f3:::benchmark_data_smithi173_55121_object21160:head# 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1315723f:::benchmark_data_smithi173_55121_object4492:head# 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13172a19:::benchmark_data_smithi173_55121_object21434:head# 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13176735:::benchmark_data_smithi173_55121_object21220:head# 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13181a33:::benchmark_data_smithi173_55121_object348:head# 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:131840c4:::benchmark_data_smithi173_55121_object2587:head# 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13216039:::benchmark_data_smithi173_55121_object20780:head# 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.579 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:132262d1:::benchmark_data_smithi173_55121_object639:head# 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1324c379:::benchmark_data_smithi173_55121_object15157:head# 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13265bd1:::benchmark_data_smithi173_55121_object17803:head# 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:132774f4:::benchmark_data_smithi173_55121_object10431:head# 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:132a580b:::benchmark_data_smithi173_55121_object31566:head# 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:132ec686:::benchmark_data_smithi173_55121_object22139:head# 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:132f2169:::benchmark_data_smithi173_55121_object11186:head# 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.580 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:132ff79b:::benchmark_data_smithi173_55121_object10818:head# 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1330f6e7:::benchmark_data_smithi173_55121_object23299:head# 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13316a28:::benchmark_data_smithi173_55121_object34330:head# 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1331c5b5:::benchmark_data_smithi173_55121_object19096:head# 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1333b8f7:::benchmark_data_smithi173_55121_object5253:head# 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1333edf2:::benchmark_data_smithi173_55121_object36640:head# 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1335b3b6:::benchmark_data_smithi173_55121_object9797:head# 2024-09-17T16:12:14.581 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13364278:::benchmark_data_smithi173_55121_object20958:head# 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13379e83:::benchmark_data_smithi173_55121_object35018:head# 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:133d86b8:::benchmark_data_smithi173_55121_object27447:head# 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:133e36b5:::benchmark_data_smithi173_55121_object17425:head# 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1345f97c:::benchmark_data_smithi173_55121_object19877:head# 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13475e2e:::benchmark_data_smithi173_55121_object21693:head# 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.582 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1349c4b3:::benchmark_data_smithi173_55121_object16264:head# 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:134acb25:::benchmark_data_smithi173_55121_object17396:head# 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:134c2f68:::benchmark_data_smithi173_55121_object37280:head# 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:134d2de4:::benchmark_data_smithi173_55121_object4122:head# 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1350e037:::benchmark_data_smithi173_55121_object6580:head# 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13512b3a:::benchmark_data_smithi173_55121_object1169:head# 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13521d71:::benchmark_data_smithi173_55121_object35393:head# 2024-09-17T16:12:14.583 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13535c0a:::benchmark_data_smithi173_55121_object33721:head# 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1355fbe8:::benchmark_data_smithi173_55121_object3004:head# 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1356961e:::benchmark_data_smithi173_55121_object19215:head# 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:135773c0:::benchmark_data_smithi173_55121_object24367:head# 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1357bee8:::benchmark_data_smithi173_55121_object15701:head# 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:135920c6:::benchmark_data_smithi173_55121_object13424:head# 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.584 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:135d3cae:::benchmark_data_smithi173_55121_object6351:head# 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:135e9f37:::benchmark_data_smithi173_55121_object19263:head# 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:135ea149:::benchmark_data_smithi173_55121_object27345:head# 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:135f58e7:::benchmark_data_smithi173_55121_object17699:head# 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13608c7d:::benchmark_data_smithi173_55121_object4313:head# 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13629d80:::benchmark_data_smithi173_55121_object19087:head# 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1363e2a1:::benchmark_data_smithi173_55121_object18942:head# 2024-09-17T16:12:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13647574:::benchmark_data_smithi173_55121_object2506:head# 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1365245c:::benchmark_data_smithi173_55121_object31567:head# 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13663aba:::benchmark_data_smithi173_55121_object32175:head# 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1366deab:::benchmark_data_smithi173_55121_object21304:head# 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13691d4a:::benchmark_data_smithi173_55121_object8575:head# 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:136cedcb:::benchmark_data_smithi173_55121_object19725:head# 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:136d0600:::benchmark_data_smithi173_55121_object10393:head# 2024-09-17T16:12:14.586 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:136f5023:::benchmark_data_smithi173_55121_object5678:head# 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13705ecd:::benchmark_data_smithi173_55121_object18605:head# 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13709ed7:::benchmark_data_smithi173_55121_object31398:head# 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1371d9a2:::benchmark_data_smithi173_55121_object26759:head# 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:137283fc:::benchmark_data_smithi173_55121_object20550:head# 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1373f69d:::benchmark_data_smithi173_55121_object34349:head# 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.587 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13754ff7:::benchmark_data_smithi173_55121_object21929:head# 2024-09-17T16:12:14.588 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.588 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13764800:::benchmark_data_smithi173_55121_object885:head# 2024-09-17T16:12:14.588 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.588 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1377965b:::benchmark_data_smithi173_55121_object3401:head# 2024-09-17T16:12:14.588 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:14.616 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1377bb68:::benchmark_data_smithi173_55121_object9358:head# 2024-09-17T16:12:14.616 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.616 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13796fd3:::benchmark_data_smithi173_55121_object11727:head# 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:137d599f:::benchmark_data_smithi173_55121_object4380:head# 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:137d69d5:::benchmark_data_smithi173_55121_object24887:head# 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:137fce8a:::benchmark_data_smithi173_55121_object687:head# 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:138131ac:::benchmark_data_smithi173_55121_object34212:head# 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1382aeba:::benchmark_data_smithi173_55121_object16725:head# 2024-09-17T16:12:14.617 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13850090:::benchmark_data_smithi173_55121_object7988:head# 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13853cfb:::benchmark_data_smithi173_55121_object13919:head# 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1387daa0:::benchmark_data_smithi173_55121_object1320:head# 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:138a48a4:::benchmark_data_smithi173_55121_object33613:head# 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:138fe656:::benchmark_data_smithi173_55121_object3291:head# 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:139471ea:::benchmark_data_smithi173_55121_object5399:head# 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:139725cd:::benchmark_data_smithi173_55121_object5363:head# 2024-09-17T16:12:14.618 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13976581:::benchmark_data_smithi173_55121_object37366:head# 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:139b271f:::benchmark_data_smithi173_55121_object29756:head# 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:139b27dd:::benchmark_data_smithi173_55121_object5640:head# 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:139f7def:::benchmark_data_smithi173_55121_object6827:head# 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:139fbd52:::benchmark_data_smithi173_55121_object20018:head# 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13a3d592:::benchmark_data_smithi173_55121_object16045:head# 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13aaadb5:::benchmark_data_smithi173_55121_object12989:head# 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.619 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13aafffb:::benchmark_data_smithi173_55121_object4625:head# 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13ab4cc5:::benchmark_data_smithi173_55121_object19874:head# 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13ae4fbf:::benchmark_data_smithi173_55121_object31949:head# 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13afbe7a:::benchmark_data_smithi173_55121_object27253:head# 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b2ebae:::benchmark_data_smithi173_55121_object8879:head# 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b464c3:::benchmark_data_smithi173_55121_object18340:head# 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b5b396:::benchmark_data_smithi173_55121_object33848:head# 2024-09-17T16:12:14.620 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b5dcf7:::benchmark_data_smithi173_55121_object28454:head# 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b5f5f7:::benchmark_data_smithi173_55121_object31982:head# 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b66ef6:::benchmark_data_smithi173_55121_object34347:head# 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b6a037:::benchmark_data_smithi173_55121_object25478:head# 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b7d140:::benchmark_data_smithi173_55121_object22619:head# 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b806c4:::benchmark_data_smithi173_55121_object18008:head# 2024-09-17T16:12:14.621 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b9113c:::benchmark_data_smithi173_55121_object16439:head# 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13b9557c:::benchmark_data_smithi173_55121_object3873:head# 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13bf7dd1:::benchmark_data_smithi173_55121_object35211:head# 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13bffffc:::benchmark_data_smithi173_55121_object23139:head# 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13c0f122:::benchmark_data_smithi173_55121_object11314:head# 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13c59a50:::benchmark_data_smithi173_55121_object26008:head# 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.622 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13c69bd8:::benchmark_data_smithi173_55121_object1759:head# 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13c6af85:::benchmark_data_smithi173_55121_object33750:head# 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13c6ee06:::benchmark_data_smithi173_55121_object19959:head# 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13c7619d:::benchmark_data_smithi173_55121_object22777:head# 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13c7653a:::benchmark_data_smithi173_55121_object25966:head# 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13c9886a:::benchmark_data_smithi173_55121_object35413:head# 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13cad6a8:::benchmark_data_smithi173_55121_object24173:head# 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.623 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13cb416d:::benchmark_data_smithi173_55121_object15488:head# 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13ccc7d3:::benchmark_data_smithi173_55121_object28121:head# 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13cd19e0:::benchmark_data_smithi173_55121_object16683:head# 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13d02cbf:::benchmark_data_smithi173_55121_object16290:head# 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13d0cf87:::benchmark_data_smithi173_55121_object12088:head# 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13d3df4f:::benchmark_data_smithi173_55121_object5375:head# 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13d4dc8c:::benchmark_data_smithi173_55121_object16189:head# 2024-09-17T16:12:14.624 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13d573c7:::benchmark_data_smithi173_55121_object30214:head# 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13d5d9ea:::benchmark_data_smithi173_55121_object12576:head# 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13d84d39:::benchmark_data_smithi173_55121_object3157:head# 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13d89c05:::benchmark_data_smithi173_55121_object6946:head# 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13da8a4c:::benchmark_data_smithi173_55121_object7182:head# 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13db6325:::benchmark_data_smithi173_55121_object26600:head# 2024-09-17T16:12:14.625 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13dc9189:::benchmark_data_smithi173_55121_object25815:head# 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13de04ba:::benchmark_data_smithi173_55121_object17004:head# 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13e06e0e:::benchmark_data_smithi173_55121_object478:head# 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13e450a4:::benchmark_data_smithi173_55121_object14296:head# 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13e61a09:::benchmark_data_smithi173_55121_object34462:head# 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13e629a8:::benchmark_data_smithi173_55121_object26331:head# 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.626 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13e845d9:::benchmark_data_smithi173_55121_object28621:head# 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13e9c073:::benchmark_data_smithi173_55121_object8337:head# 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13e9ecb4:::benchmark_data_smithi173_55121_object27910:head# 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13ea4c0c:::benchmark_data_smithi173_55121_object650:head# 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13eac6a9:::benchmark_data_smithi173_55121_object13319:head# 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13eb058f:::benchmark_data_smithi173_55121_object13798:head# 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13eb90e9:::benchmark_data_smithi173_55121_object12417:head# 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.627 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13ee691a:::benchmark_data_smithi173_55121_object28465:head# 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13f3e3ff:::benchmark_data_smithi173_55121_object9746:head# 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13f70767:::benchmark_data_smithi173_55121_object10601:head# 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13f817f3:::benchmark_data_smithi173_55121_object7635:head# 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13f83dc1:::benchmark_data_smithi173_55121_object36076:head# 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13f919c6:::benchmark_data_smithi173_55121_object15042:head# 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13fa2673:::benchmark_data_smithi173_55121_object19166:head# 2024-09-17T16:12:14.628 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13fbf047:::benchmark_data_smithi173_55121_object28385:head# 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13fca8ad:::benchmark_data_smithi173_55121_object34129:head# 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13fcecb9:::benchmark_data_smithi173_55121_object14854:head# 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13fe2c75:::benchmark_data_smithi173_55121_object36498:head# 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13fe615a:::benchmark_data_smithi173_55121_object14232:head# 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13fec027:::benchmark_data_smithi173_55121_object10629:head# 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.629 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13ffa1c1:::benchmark_data_smithi173_55121_object23263:head# 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:13ffed11:::benchmark_data_smithi173_55121_object4257:head# 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14048ee0:::benchmark_data_smithi173_55121_object17668:head# 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1406d01b:::benchmark_data_smithi173_55121_object20907:head# 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1407ee3a:::benchmark_data_smithi173_55121_object17796:head# 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:140a6011:::benchmark_data_smithi173_55121_object33700:head# 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:140ad613:::benchmark_data_smithi173_55121_object22615:head# 2024-09-17T16:12:14.630 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:140c026a:::benchmark_data_smithi173_55121_object304:head# 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:140ffa98:::benchmark_data_smithi173_55121_object11979:head# 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14132ea5:::benchmark_data_smithi173_55121_object31184:head# 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1415c629:::benchmark_data_smithi173_55121_object6311:head# 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14162493:::benchmark_data_smithi173_55121_object15200:head# 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14187239:::benchmark_data_smithi173_55121_object6499:head# 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:141af370:::benchmark_data_smithi173_55121_object15967:head# 2024-09-17T16:12:14.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:141e910b:::benchmark_data_smithi173_55121_object14699:head# 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1420b645:::benchmark_data_smithi173_55121_object12420:head# 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:142192fd:::benchmark_data_smithi173_55121_object13178:head# 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14221fb8:::benchmark_data_smithi173_55121_object553:head# 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14222259:::benchmark_data_smithi173_55121_object32155:head# 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14231e31:::benchmark_data_smithi173_55121_object9926:head# 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.632 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1423c4e1:::benchmark_data_smithi173_55121_object36077:head# 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1428e724:::benchmark_data_smithi173_55121_object25418:head# 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1429b0d3:::benchmark_data_smithi173_55121_object24479:head# 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:142e6777:::benchmark_data_smithi173_55121_object13324:head# 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:142f08bb:::benchmark_data_smithi173_55121_object26879:head# 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1434af36:::benchmark_data_smithi173_55121_object20412:head# 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1434b39b:::benchmark_data_smithi173_55121_object22176:head# 2024-09-17T16:12:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14370843:::benchmark_data_smithi173_55121_object16245:head# 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14372295:::benchmark_data_smithi173_55121_object4912:head# 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1438f582:::benchmark_data_smithi173_55121_object29572:head# 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1439804f:::benchmark_data_smithi173_55121_object2348:head# 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1439d5bd:::benchmark_data_smithi173_55121_object15312:head# 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:143b4287:::benchmark_data_smithi173_55121_object6202:head# 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:143bb8c9:::benchmark_data_smithi173_55121_object25190:head# 2024-09-17T16:12:14.634 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:143bcc43:::benchmark_data_smithi173_55121_object29398:head# 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:143c2d11:::benchmark_data_smithi173_55121_object16231:head# 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:143e33b9:::benchmark_data_smithi173_55121_object24251:head# 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:143f8ea6:::benchmark_data_smithi173_55121_object7664:head# 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14419391:::benchmark_data_smithi173_55121_object35437:head# 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1441bc83:::benchmark_data_smithi173_55121_object15783:head# 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1446b137:::benchmark_data_smithi173_55121_object33924:head# 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:144777fa:::benchmark_data_smithi173_55121_object8065:head# 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1448c033:::benchmark_data_smithi173_55121_object11366:head# 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14490f6c:::benchmark_data_smithi173_55121_object12561:head# 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:144aafe3:::benchmark_data_smithi173_55121_object11211:head# 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:144bde5c:::benchmark_data_smithi173_55121_object15531:head# 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:144ea865:::benchmark_data_smithi173_55121_object15283:head# 2024-09-17T16:12:14.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:145004c2:::benchmark_data_smithi173_55121_object30313:head# 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14502b46:::benchmark_data_smithi173_55121_object10427:head# 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1450e88d:::benchmark_data_smithi173_55121_object31746:head# 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:145561f6:::benchmark_data_smithi173_55121_object18441:head# 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14574c15:::benchmark_data_smithi173_55121_object35048:head# 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1457d294:::benchmark_data_smithi173_55121_object3276:head# 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14583a00:::benchmark_data_smithi173_55121_object22681:head# 2024-09-17T16:12:14.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14584de4:::benchmark_data_smithi173_55121_object13408:head# 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14597804:::benchmark_data_smithi173_55121_object28984:head# 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:145a4b59:::benchmark_data_smithi173_55121_object17050:head# 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:145ac675:::benchmark_data_smithi173_55121_object20724:head# 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:145b6e88:::benchmark_data_smithi173_55121_object29814:head# 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:145c364e:::benchmark_data_smithi173_55121_object10056:head# 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.638 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:145c419b:::benchmark_data_smithi173_55121_object19570:head# 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1461037a:::benchmark_data_smithi173_55121_object891:head# 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14649a30:::benchmark_data_smithi173_55121_object6380:head# 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1464f223:::benchmark_data_smithi173_55121_object17883:head# 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1465204e:::benchmark_data_smithi173_55121_object3261:head# 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14663d0b:::benchmark_data_smithi173_55121_object18663:head# 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14696061:::benchmark_data_smithi173_55121_object23972:head# 2024-09-17T16:12:14.639 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:146d7bf6:::benchmark_data_smithi173_55121_object18061:head# 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:146ef9a4:::benchmark_data_smithi173_55121_object19145:head# 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:146f0979:::benchmark_data_smithi173_55121_object34360:head# 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1470dfd3:::benchmark_data_smithi173_55121_object5220:head# 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14714364:::benchmark_data_smithi173_55121_object30392:head# 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:147187df:::benchmark_data_smithi173_55121_object6174:head# 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.640 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14723fd8:::benchmark_data_smithi173_55121_object20501:head# 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:147a0adb:::benchmark_data_smithi173_55121_object19039:head# 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1482838a:::benchmark_data_smithi173_55121_object24156:head# 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:148627a0:::benchmark_data_smithi173_55121_object29026:head# 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:148901a8:::benchmark_data_smithi173_55121_object7085:head# 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1489c459:::benchmark_data_smithi173_55121_object34694:head# 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:148a0876:::benchmark_data_smithi173_55121_object3698:head# 2024-09-17T16:12:14.641 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:148ad7cf:::benchmark_data_smithi173_55121_object1802:head# 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:148ee069:::benchmark_data_smithi173_55121_object24398:head# 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:148ee56d:::benchmark_data_smithi173_55121_object26326:head# 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:148f1180:::benchmark_data_smithi173_55121_object34140:head# 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:148f5585:::benchmark_data_smithi173_55121_object24053:head# 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14921942:::benchmark_data_smithi173_55121_object14627:head# 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.642 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14927897:::benchmark_data_smithi173_55121_object7684:head# 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1492a7bd:::benchmark_data_smithi173_55121_object382:head# 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14966653:::benchmark_data_smithi173_55121_object24953:head# 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1498c070:::benchmark_data_smithi173_55121_object17971:head# 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14996fd0:::benchmark_data_smithi173_55121_object12959:head# 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1499a419:::benchmark_data_smithi173_55121_object1992:head# 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.643 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1499a45f:::benchmark_data_smithi173_55121_object32061:head# 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:149acb5e:::benchmark_data_smithi173_55121_object20730:head# 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:149af195:::benchmark_data_smithi173_55121_object8130:head# 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:149d9317:::benchmark_data_smithi173_55121_object3556:head# 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14a03e63:::benchmark_data_smithi173_55121_object2054:head# 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14a17395:::benchmark_data_smithi173_55121_object33460:head# 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14a3bbe5:::benchmark_data_smithi173_55121_object37285:head# 2024-09-17T16:12:14.644 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14a86164:::benchmark_data_smithi173_55121_object31461:head# 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14a875b9:::benchmark_data_smithi173_55121_object6799:head# 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14abb0cc:::benchmark_data_smithi173_55121_object19093:head# 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14ac2adb:::benchmark_data_smithi173_55121_object14184:head# 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14ad3d38:::benchmark_data_smithi173_55121_object18310:head# 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14ada928:::benchmark_data_smithi173_55121_object8953:head# 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14adce74:::benchmark_data_smithi173_55121_object6693:head# 2024-09-17T16:12:14.645 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14afca7c:::benchmark_data_smithi173_55121_object12457:head# 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14b0f346:::benchmark_data_smithi173_55121_object5040:head# 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14b14af3:::benchmark_data_smithi173_55121_object1353:head# 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14b16917:::benchmark_data_smithi173_55121_object30280:head# 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14b1fe42:::benchmark_data_smithi173_55121_object164:head# 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14b59eb8:::benchmark_data_smithi173_55121_object37196:head# 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14b7a2dc:::benchmark_data_smithi173_55121_object24195:head# 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.646 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14b86de9:::benchmark_data_smithi173_55121_object27869:head# 2024-09-17T16:12:14.647 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.647 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14b8b448:::benchmark_data_smithi173_55121_object19255:head# 2024-09-17T16:12:14.647 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14bc4a7f:::benchmark_data_smithi173_55121_object28697:head# 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14bf1255:::benchmark_data_smithi173_55121_object24541:head# 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14bf171d:::benchmark_data_smithi173_55121_object15505:head# 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14bfa473:::benchmark_data_smithi173_55121_object32974:head# 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14c25a15:::benchmark_data_smithi173_55121_object25387:head# 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14c48c33:::benchmark_data_smithi173_55121_object27262:head# 2024-09-17T16:12:14.658 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14c5173a:::benchmark_data_smithi173_55121_object4691:head# 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14c69611:::benchmark_data_smithi173_55121_object16763:head# 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14c8baad:::benchmark_data_smithi173_55121_object12486:head# 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14ca024d:::benchmark_data_smithi173_55121_object24337:head# 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14cbfa8e:::benchmark_data_smithi173_55121_object32368:head# 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14cc569d:::benchmark_data_smithi173_55121_object23369:head# 2024-09-17T16:12:14.659 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14cf8754:::benchmark_data_smithi173_55121_object18593:head# 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14d16652:::benchmark_data_smithi173_55121_object6349:head# 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14d2279d:::benchmark_data_smithi173_55121_object25367:head# 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14d28eaf:::benchmark_data_smithi173_55121_object51:head# 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14d4d275:::benchmark_data_smithi173_55121_object21884:head# 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14d4de2d:::benchmark_data_smithi173_55121_object8981:head# 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.660 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14d56b54:::benchmark_data_smithi173_55121_object26335:head# 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14d6516e:::benchmark_data_smithi173_55121_object10480:head# 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14d9ec1b:::benchmark_data_smithi173_55121_object4875:head# 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14dab1a8:::benchmark_data_smithi173_55121_object23095:head# 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14db98ef:::benchmark_data_smithi173_55121_object35113:head# 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14de2308:::benchmark_data_smithi173_55121_object32350:head# 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14de30d7:::benchmark_data_smithi173_55121_object32835:head# 2024-09-17T16:12:14.661 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14deaada:::benchmark_data_smithi173_55121_object20906:head# 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14e02195:::benchmark_data_smithi173_55121_object21001:head# 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14e0fc94:::benchmark_data_smithi173_55121_object8104:head# 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14e206dc:::benchmark_data_smithi173_55121_object2221:head# 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14e2d96a:::benchmark_data_smithi173_55121_object4520:head# 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14e38033:::benchmark_data_smithi173_55121_object30343:head# 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.662 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14e542d7:::benchmark_data_smithi173_55121_object14415:head# 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14e75291:::benchmark_data_smithi173_55121_object21886:head# 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14ea2843:::benchmark_data_smithi173_55121_object26806:head# 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14ee4965:::benchmark_data_smithi173_55121_object7412:head# 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14ef52b4:::benchmark_data_smithi173_55121_object30677:head# 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14f21c3f:::benchmark_data_smithi173_55121_object5020:head# 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14f3b0fe:::benchmark_data_smithi173_55121_object24134:head# 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.663 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14f72e75:::benchmark_data_smithi173_55121_object29265:head# 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14f7f5c0:::benchmark_data_smithi173_55121_object25343:head# 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14f7fe11:::benchmark_data_smithi173_55121_object21229:head# 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14f80f53:::benchmark_data_smithi173_55121_object33036:head# 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14f8c35d:::benchmark_data_smithi173_55121_object26030:head# 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14fab613:::benchmark_data_smithi173_55121_object24392:head# 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14fae4bb:::benchmark_data_smithi173_55121_object14103:head# 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.664 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14fb3220:::benchmark_data_smithi173_55121_object1271:head# 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14fc6833:::benchmark_data_smithi173_55121_object26858:head# 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:14fecccc:::benchmark_data_smithi173_55121_object16931:head# 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15002fc2:::benchmark_data_smithi173_55121_object37213:head# 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1501e40a:::benchmark_data_smithi173_55121_object1726:head# 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1502ff15:::benchmark_data_smithi173_55121_object9029:head# 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15030fb9:::benchmark_data_smithi173_55121_object34192:head# 2024-09-17T16:12:14.665 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:150332d0:::benchmark_data_smithi173_55121_object13942:head# 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1506e21c:::benchmark_data_smithi173_55121_object32347:head# 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:150779c0:::benchmark_data_smithi173_55121_object34455:head# 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:150b4c44:::benchmark_data_smithi173_55121_object26828:head# 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:150e2846:::benchmark_data_smithi173_55121_object30871:head# 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15118b27:::benchmark_data_smithi173_55121_object9953:head# 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.666 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1511d46a:::benchmark_data_smithi173_55121_object17367:head# 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1514935e:::benchmark_data_smithi173_55121_object4330:head# 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1518629e:::benchmark_data_smithi173_55121_object11275:head# 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15188cf2:::benchmark_data_smithi173_55121_object300:head# 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:151ac5b9:::benchmark_data_smithi173_55121_object18541:head# 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:151b10cb:::benchmark_data_smithi173_55121_object3900:head# 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:151bce1b:::benchmark_data_smithi173_55121_object21948:head# 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.667 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:151e2b26:::benchmark_data_smithi173_55121_object35206:head# 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:151e5d5d:::benchmark_data_smithi173_55121_object18874:head# 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:151e7cfa:::benchmark_data_smithi173_55121_object24216:head# 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:151ffbed:::benchmark_data_smithi173_55121_object5276:head# 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1521fe1c:::benchmark_data_smithi173_55121_object2794:head# 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:152234ea:::benchmark_data_smithi173_55121_object37032:head# 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15244070:::benchmark_data_smithi173_55121_object20890:head# 2024-09-17T16:12:14.668 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.669 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15248bfc:::benchmark_data_smithi173_55121_object16680:head# 2024-09-17T16:12:14.669 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.669 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15253913:::benchmark_data_smithi173_55121_object9316:head# 2024-09-17T16:12:14.669 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.671 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:152616c9:::benchmark_data_smithi173_55121_object14188:head# 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15261945:::benchmark_data_smithi173_55121_object9300:head# 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:152633ec:::benchmark_data_smithi173_55121_object5889:head# 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:152a38fe:::benchmark_data_smithi173_55121_object19759:head# 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:152a3dca:::benchmark_data_smithi173_55121_object6869:head# 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:152b1a40:::benchmark_data_smithi173_55121_object28878:head# 2024-09-17T16:12:14.672 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:152e2842:::benchmark_data_smithi173_55121_object16356:head# 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:152e63b9:::benchmark_data_smithi173_55121_object11737:head# 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:152e753a:::benchmark_data_smithi173_55121_object18747:head# 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1530ed34:::benchmark_data_smithi173_55121_object28466:head# 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1532ca94:::benchmark_data_smithi173_55121_object12059:head# 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15350352:::benchmark_data_smithi173_55121_object30921:head# 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1537328c:::benchmark_data_smithi173_55121_object8710:head# 2024-09-17T16:12:14.673 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15374f08:::benchmark_data_smithi173_55121_object11959:head# 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1537930f:::benchmark_data_smithi173_55121_object35374:head# 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15381c84:::benchmark_data_smithi173_55121_object26650:head# 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1539958b:::benchmark_data_smithi173_55121_object4465:head# 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:153ac907:::benchmark_data_smithi173_55121_object15554:head# 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:153d3b7c:::benchmark_data_smithi173_55121_object19458:head# 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15407f1f:::benchmark_data_smithi173_55121_object16304:head# 2024-09-17T16:12:14.674 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.675 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:14.675 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15454a63:::benchmark_data_smithi173_55121_object11657:head# 2024-09-17T16:12:14.675 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:14.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:14.701 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15481e77:::benchmark_data_smithi173_55121_object8797:head# 2024-09-17T16:12:14.701 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.701 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:154847a9:::benchmark_data_smithi173_55121_object36748:head# 2024-09-17T16:12:14.701 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.701 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1549feb5:::benchmark_data_smithi173_55121_object17634:head# 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:154a35dd:::benchmark_data_smithi173_55121_object30271:head# 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:154b2a32:::benchmark_data_smithi173_55121_object18042:head# 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:154b7b29:::benchmark_data_smithi173_55121_object18527:head# 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:154b8e49:::benchmark_data_smithi173_55121_object35154:head# 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:154f98af:::benchmark_data_smithi173_55121_object21042:head# 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:155163da:::benchmark_data_smithi173_55121_object28334:head# 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15576e24:::benchmark_data_smithi173_55121_object29654:head# 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1557ba33:::benchmark_data_smithi173_55121_object24350:head# 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15586d86:::benchmark_data_smithi173_55121_object32174:head# 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15587ce9:::benchmark_data_smithi173_55121_object15450:head# 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:155b7bdd:::benchmark_data_smithi173_55121_object9645:head# 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:155c2ec2:::benchmark_data_smithi173_55121_object10391:head# 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:155d8f7c:::benchmark_data_smithi173_55121_object36321:head# 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:155e5ac1:::benchmark_data_smithi173_55121_object19922:head# 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15617fcc:::benchmark_data_smithi173_55121_object21523:head# 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1561f169:::benchmark_data_smithi173_55121_object8372:head# 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15629073:::benchmark_data_smithi173_55121_object35442:head# 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15631760:::benchmark_data_smithi173_55121_object17447:head# 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1563417e:::benchmark_data_smithi173_55121_object20511:head# 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1566593c:::benchmark_data_smithi173_55121_object20299:head# 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1569e85c:::benchmark_data_smithi173_55121_object15739:head# 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:156a2abb:::benchmark_data_smithi173_55121_object3884:head# 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:156b4f75:::benchmark_data_smithi173_55121_object24733:head# 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:156e5dbe:::benchmark_data_smithi173_55121_object20781:head# 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:156eb9d6:::benchmark_data_smithi173_55121_object9563:head# 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:157395c3:::benchmark_data_smithi173_55121_object14618:head# 2024-09-17T16:12:14.705 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15742e2e:::benchmark_data_smithi173_55121_object5054:head# 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1575269b:::benchmark_data_smithi173_55121_object13760:head# 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1577ac4d:::benchmark_data_smithi173_55121_object15810:head# 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1577b38b:::benchmark_data_smithi173_55121_object20787:head# 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:157ceb88:::benchmark_data_smithi173_55121_object22738:head# 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:157e8a43:::benchmark_data_smithi173_55121_object34385:head# 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.706 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:157f1077:::benchmark_data_smithi173_55121_object15653:head# 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:157fd8bc:::benchmark_data_smithi173_55121_object24106:head# 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:158428f8:::benchmark_data_smithi173_55121_object29642:head# 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:158577d8:::benchmark_data_smithi173_55121_object10286:head# 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1585fea1:::benchmark_data_smithi173_55121_object20460:head# 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1587de72:::benchmark_data_smithi173_55121_object7591:head# 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15885dac:::benchmark_data_smithi173_55121_object29716:head# 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.707 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:158a1e11:::benchmark_data_smithi173_55121_object32359:head# 2024-09-17T16:12:14.708 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.708 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:158ac562:::benchmark_data_smithi173_55121_object6470:head# 2024-09-17T16:12:14.708 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.708 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:158cb29a:::benchmark_data_smithi173_55121_object9693:head# 2024-09-17T16:12:14.708 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.708 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:158d7c97:::benchmark_data_smithi173_55121_object11133:head# 2024-09-17T16:12:14.708 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.708 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15907c92:::benchmark_data_smithi173_55121_object37036:head# 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1591c986:::benchmark_data_smithi173_55121_object20982:head# 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1594b689:::benchmark_data_smithi173_55121_object886:head# 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1596db8b:::benchmark_data_smithi173_55121_object34691:head# 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15986c6c:::benchmark_data_smithi173_55121_object33994:head# 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1598e680:::benchmark_data_smithi173_55121_object3883:head# 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.709 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1598e8f2:::benchmark_data_smithi173_55121_object21086:head# 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:159bae42:::benchmark_data_smithi173_55121_object25858:head# 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:159cc017:::benchmark_data_smithi173_55121_object14452:head# 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:159d6593:::benchmark_data_smithi173_55121_object21653:head# 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15a5bdca:::benchmark_data_smithi173_55121_object22849:head# 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15a623cb:::benchmark_data_smithi173_55121_object27327:head# 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15a8a1d7:::benchmark_data_smithi173_55121_object29346:head# 2024-09-17T16:12:14.710 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15abc3db:::benchmark_data_smithi173_55121_object14337:head# 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15ac2342:::benchmark_data_smithi173_55121_object18131:head# 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15af5f38:::benchmark_data_smithi173_55121_object23568:head# 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15b1f2d4:::benchmark_data_smithi173_55121_object9588:head# 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15b33cb9:::benchmark_data_smithi173_55121_object5699:head# 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15b4f953:::benchmark_data_smithi173_55121_object31079:head# 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15b7656d:::benchmark_data_smithi173_55121_object32620:head# 2024-09-17T16:12:14.711 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15b8401d:::benchmark_data_smithi173_55121_object25610:head# 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15bbb346:::benchmark_data_smithi173_55121_object22315:head# 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15bf0ae3:::benchmark_data_smithi173_55121_object35624:head# 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15c1671f:::benchmark_data_smithi173_55121_object21759:head# 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15c1b1e0:::benchmark_data_smithi173_55121_object13479:head# 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15c1ba24:::benchmark_data_smithi173_55121_object17116:head# 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.712 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15c29503:::benchmark_data_smithi173_55121_object14281:head# 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15c36d7a:::benchmark_data_smithi173_55121_object27993:head# 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15c3dc01:::benchmark_data_smithi173_55121_object33520:head# 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15c4400f:::benchmark_data_smithi173_55121_object19493:head# 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15c46104:::benchmark_data_smithi173_55121_object18846:head# 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15ca2a9f:::benchmark_data_smithi173_55121_object5929:head# 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15cb5d5d:::benchmark_data_smithi173_55121_object11805:head# 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.713 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15cd67bd:::benchmark_data_smithi173_55121_object31352:head# 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15cda41d:::benchmark_data_smithi173_55121_object21092:head# 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15d23556:::benchmark_data_smithi173_55121_object7953:head# 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15d31884:::benchmark_data_smithi173_55121_object7374:head# 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15d46985:::benchmark_data_smithi173_55121_object29552:head# 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15d48a51:::benchmark_data_smithi173_55121_object32223:head# 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15d5a31b:::benchmark_data_smithi173_55121_object1734:head# 2024-09-17T16:12:14.714 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15d6f42e:::benchmark_data_smithi173_55121_object28845:head# 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15d7488e:::benchmark_data_smithi173_55121_object137:head# 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15d7c389:::benchmark_data_smithi173_55121_object10596:head# 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15d8a6f9:::benchmark_data_smithi173_55121_object16740:head# 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15d8d2de:::benchmark_data_smithi173_55121_object10411:head# 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15da4597:::benchmark_data_smithi173_55121_object30917:head# 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15db12ac:::benchmark_data_smithi173_55121_object25621:head# 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.715 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15db4e3b:::benchmark_data_smithi173_55121_object862:head# 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15dc87c5:::benchmark_data_smithi173_55121_object3661:head# 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15dcc533:::benchmark_data_smithi173_55121_object33236:head# 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15e125af:::benchmark_data_smithi173_55121_object36510:head# 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15e2d8a1:::benchmark_data_smithi173_55121_object11400:head# 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15e5d96a:::benchmark_data_smithi173_55121_object35176:head# 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15e6b2e9:::benchmark_data_smithi173_55121_object37191:head# 2024-09-17T16:12:14.716 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15e7dc18:::benchmark_data_smithi173_55121_object25973:head# 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15e7fa93:::benchmark_data_smithi173_55121_object8816:head# 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15ec01c7:::benchmark_data_smithi173_55121_object28729:head# 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15f0fce0:::benchmark_data_smithi173_55121_object19900:head# 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15f11c6d:::benchmark_data_smithi173_55121_object35359:head# 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15f17254:::benchmark_data_smithi173_55121_object9688:head# 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15f1d3f1:::benchmark_data_smithi173_55121_object2479:head# 2024-09-17T16:12:14.717 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15f29082:::benchmark_data_smithi173_55121_object3970:head# 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15f505d1:::benchmark_data_smithi173_55121_object13358:head# 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15f972ec:::benchmark_data_smithi173_55121_object20633:head# 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:15fdf25a:::benchmark_data_smithi173_55121_object1471:head# 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1603d091:::benchmark_data_smithi173_55121_object26544:head# 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1603dca6:::benchmark_data_smithi173_55121_object35664:head# 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1603f506:::benchmark_data_smithi173_55121_object12509:head# 2024-09-17T16:12:14.718 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16078c08:::benchmark_data_smithi173_55121_object21998:head# 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16090d1c:::benchmark_data_smithi173_55121_object22640:head# 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:160a5ece:::benchmark_data_smithi173_55121_object9716:head# 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:160b032a:::benchmark_data_smithi173_55121_object17332:head# 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:160bb86d:::benchmark_data_smithi173_55121_object7083:head# 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:160e33cd:::benchmark_data_smithi173_55121_object31011:head# 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.719 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:160fe7d7:::benchmark_data_smithi173_55121_object896:head# 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1612b85c:::benchmark_data_smithi173_55121_object9781:head# 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1612bea5:::benchmark_data_smithi173_55121_object4497:head# 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1614550c:::benchmark_data_smithi173_55121_object20134:head# 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:161a1ae8:::benchmark_data_smithi173_55121_object24968:head# 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:161ace95:::benchmark_data_smithi173_55121_object36918:head# 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:161ebe1a:::benchmark_data_smithi173_55121_object23423:head# 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:161f1fca:::benchmark_data_smithi173_55121_object7022:head# 2024-09-17T16:12:14.720 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:162130c2:::benchmark_data_smithi173_55121_object8083:head# 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1623665b:::benchmark_data_smithi173_55121_object15543:head# 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16269384:::benchmark_data_smithi173_55121_object15800:head# 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1627ab82:::benchmark_data_smithi173_55121_object24273:head# 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16282b85:::benchmark_data_smithi173_55121_object19472:head# 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16287677:::benchmark_data_smithi173_55121_object37228:head# 2024-09-17T16:12:14.721 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1628fe1d:::benchmark_data_smithi173_55121_object23810:head# 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16295cc3:::benchmark_data_smithi173_55121_object1993:head# 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1629b02d:::benchmark_data_smithi173_55121_object32299:head# 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:162a10b5:::benchmark_data_smithi173_55121_object9892:head# 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:162ba2b4:::benchmark_data_smithi173_55121_object19724:head# 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:162de0d9:::benchmark_data_smithi173_55121_object19136:head# 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1630940f:::benchmark_data_smithi173_55121_object15113:head# 2024-09-17T16:12:14.722 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:163c08ca:::benchmark_data_smithi173_55121_object5772:head# 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:163ca92b:::benchmark_data_smithi173_55121_object21460:head# 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:163ea8af:::benchmark_data_smithi173_55121_object22973:head# 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16400a14:::benchmark_data_smithi173_55121_object33665:head# 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16411fda:::benchmark_data_smithi173_55121_object22509:head# 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:164372fe:::benchmark_data_smithi173_55121_object31755:head# 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.723 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1643d834:::benchmark_data_smithi173_55121_object31439:head# 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16456935:::benchmark_data_smithi173_55121_object920:head# 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:164a6b1c:::benchmark_data_smithi173_55121_object14285:head# 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:164c26cc:::benchmark_data_smithi173_55121_object29302:head# 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16504760:::benchmark_data_smithi173_55121_object33739:head# 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1652ab48:::benchmark_data_smithi173_55121_object23918:head# 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:165845c0:::benchmark_data_smithi173_55121_object22711:head# 2024-09-17T16:12:14.724 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1659b638:::benchmark_data_smithi173_55121_object6143:head# 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:165ab9fe:::benchmark_data_smithi173_55121_object8812:head# 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:165d1a11:::benchmark_data_smithi173_55121_object27763:head# 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:165f9939:::benchmark_data_smithi173_55121_object17555:head# 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:165fca20:::benchmark_data_smithi173_55121_object22060:head# 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166178cc:::benchmark_data_smithi173_55121_object7614:head# 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.725 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16628fc6:::benchmark_data_smithi173_55121_object11032:head# 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1663d508:::benchmark_data_smithi173_55121_object15169:head# 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16643d38:::benchmark_data_smithi173_55121_object31843:head# 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16684c09:::benchmark_data_smithi173_55121_object20600:head# 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1669b2dc:::benchmark_data_smithi173_55121_object30629:head# 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166aef84:::benchmark_data_smithi173_55121_object9452:head# 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166bee1b:::benchmark_data_smithi173_55121_object26651:head# 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.726 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166c19b4:::benchmark_data_smithi173_55121_object24671:head# 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166c3b21:::benchmark_data_smithi173_55121_object36347:head# 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166c68bc:::benchmark_data_smithi173_55121_object5133:head# 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166cf3f0:::benchmark_data_smithi173_55121_object19080:head# 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166da625:::benchmark_data_smithi173_55121_object15328:head# 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166dbe55:::benchmark_data_smithi173_55121_object13404:head# 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166e19df:::benchmark_data_smithi173_55121_object502:head# 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.727 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166e81fc:::benchmark_data_smithi173_55121_object33742:head# 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:166f5f75:::benchmark_data_smithi173_55121_object14773:head# 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16713ed8:::benchmark_data_smithi173_55121_object22142:head# 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16733553:::benchmark_data_smithi173_55121_object4786:head# 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1673fecc:::benchmark_data_smithi173_55121_object4085:head# 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16745327:::benchmark_data_smithi173_55121_object14853:head# 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1676a521:::benchmark_data_smithi173_55121_object1712:head# 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.728 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16777371:::benchmark_data_smithi173_55121_object22736:head# 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:167781d1:::benchmark_data_smithi173_55121_object10262:head# 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16788145:::benchmark_data_smithi173_55121_object7099:head# 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:167b6ba7:::benchmark_data_smithi173_55121_object8634:head# 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:167b8a05:::benchmark_data_smithi173_55121_object35088:head# 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1683c28d:::benchmark_data_smithi173_55121_object14450:head# 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16860cd3:::benchmark_data_smithi173_55121_object17188:head# 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.729 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:168a13fb:::benchmark_data_smithi173_55121_object10265:head# 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:168ac31e:::benchmark_data_smithi173_55121_object7645:head# 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:168b324a:::benchmark_data_smithi173_55121_object21539:head# 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:168cd375:::benchmark_data_smithi173_55121_object14605:head# 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:168dd287:::benchmark_data_smithi173_55121_object21257:head# 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:168ea999:::benchmark_data_smithi173_55121_object22087:head# 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:168fa22f:::benchmark_data_smithi173_55121_object10278:head# 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:168faa94:::benchmark_data_smithi173_55121_object35662:head# 2024-09-17T16:12:14.730 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.731 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1693711f:::benchmark_data_smithi173_55121_object287:head# 2024-09-17T16:12:14.731 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.731 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1699176b:::benchmark_data_smithi173_55121_object5387:head# 2024-09-17T16:12:14.731 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.731 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:14 smithi071 ceph-mon[30002]: pgmap v385: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 560 op/s; 10527/142743 objects degraded (7.375%); 1549/142743 objects misplaced (1.085%) 2024-09-17T16:12:14.731 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:14 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:14.731 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:14 smithi071 ceph-mon[30002]: osdmap e144: 12 total, 11 up, 12 in 2024-09-17T16:12:14.732 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:14 smithi017 ceph-mon[27571]: pgmap v385: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 560 op/s; 10527/142743 objects degraded (7.375%); 1549/142743 objects misplaced (1.085%) 2024-09-17T16:12:14.733 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:14 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:14.733 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:14 smithi017 ceph-mon[27571]: osdmap e144: 12 total, 11 up, 12 in 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1699d9fb:::benchmark_data_smithi173_55121_object2418:head# 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1699db39:::benchmark_data_smithi173_55121_object11729:head# 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:169a63ef:::benchmark_data_smithi173_55121_object36764:head# 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:169ced6e:::benchmark_data_smithi173_55121_object21267:head# 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:169eb35a:::benchmark_data_smithi173_55121_object267:head# 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:169f5788:::benchmark_data_smithi173_55121_object32273:head# 2024-09-17T16:12:14.754 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:169fc84f:::benchmark_data_smithi173_55121_object30573:head# 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16a1265f:::benchmark_data_smithi173_55121_object2538:head# 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16a3579f:::benchmark_data_smithi173_55121_object22441:head# 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16a386f1:::benchmark_data_smithi173_55121_object20838:head# 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16a43780:::benchmark_data_smithi173_55121_object21629:head# 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16a57c07:::benchmark_data_smithi173_55121_object24834:head# 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16a8ec62:::benchmark_data_smithi173_55121_object11404:head# 2024-09-17T16:12:14.755 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16ad0484:::benchmark_data_smithi173_55121_object20131:head# 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16ad85c9:::benchmark_data_smithi173_55121_object4653:head# 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16ae1157:::benchmark_data_smithi173_55121_object25510:head# 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16b08e19:::benchmark_data_smithi173_55121_object19398:head# 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16b40911:::benchmark_data_smithi173_55121_object27006:head# 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16b77e36:::benchmark_data_smithi173_55121_object4299:head# 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16b9e198:::benchmark_data_smithi173_55121_object20169:head# 2024-09-17T16:12:14.756 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16bbd3d9:::benchmark_data_smithi173_55121_object2270:head# 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16c49a5f:::benchmark_data_smithi173_55121_object6345:head# 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16c90296:::benchmark_data_smithi173_55121_object30517:head# 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16cad7b7:::benchmark_data_smithi173_55121_object11502:head# 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16cbf34b:::benchmark_data_smithi173_55121_object13642:head# 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16cc3eb2:::benchmark_data_smithi173_55121_object8031:head# 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.757 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16cf259f:::benchmark_data_smithi173_55121_object27356:head# 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d05e86:::benchmark_data_smithi173_55121_object35483:head# 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d08a82:::benchmark_data_smithi173_55121_object23902:head# 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d25e60:::benchmark_data_smithi173_55121_object28725:head# 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d31c90:::benchmark_data_smithi173_55121_object6064:head# 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d34f37:::benchmark_data_smithi173_55121_object27965:head# 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d385df:::benchmark_data_smithi173_55121_object7653:head# 2024-09-17T16:12:14.758 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d51c10:::benchmark_data_smithi173_55121_object14834:head# 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d52cb0:::benchmark_data_smithi173_55121_object21963:head# 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d53482:::benchmark_data_smithi173_55121_object19627:head# 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d595dc:::benchmark_data_smithi173_55121_object27406:head# 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16d5ef27:::benchmark_data_smithi173_55121_object14471:head# 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16db0734:::benchmark_data_smithi173_55121_object25828:head# 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16dc16a1:::benchmark_data_smithi173_55121_object20032:head# 2024-09-17T16:12:14.759 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16dd69be:::benchmark_data_smithi173_55121_object1840:head# 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16de44c2:::benchmark_data_smithi173_55121_object16965:head# 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16dfa0ef:::benchmark_data_smithi173_55121_object16296:head# 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16e149c9:::benchmark_data_smithi173_55121_object7521:head# 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16e18974:::benchmark_data_smithi173_55121_object98:head# 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16e21efd:::benchmark_data_smithi173_55121_object28686:head# 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16e83c5f:::benchmark_data_smithi173_55121_object6429:head# 2024-09-17T16:12:14.760 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16ed4d33:::benchmark_data_smithi173_55121_object36147:head# 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16ed7483:::benchmark_data_smithi173_55121_object6665:head# 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16eec804:::benchmark_data_smithi173_55121_object19085:head# 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16eef1bf:::benchmark_data_smithi173_55121_object22560:head# 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16efab65:::benchmark_data_smithi173_55121_object15716:head# 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16f5a35e:::benchmark_data_smithi173_55121_object19499:head# 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16f7f6ef:::benchmark_data_smithi173_55121_object32131:head# 2024-09-17T16:12:14.761 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16f96bb5:::benchmark_data_smithi173_55121_object1247:head# 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16f994e6:::benchmark_data_smithi173_55121_object36889:head# 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16fa4eea:::benchmark_data_smithi173_55121_object438:head# 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16fab4ec:::benchmark_data_smithi173_55121_object28363:head# 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16faf16e:::benchmark_data_smithi173_55121_object24931:head# 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16fb7ea1:::benchmark_data_smithi173_55121_object32715:head# 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.762 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:16fcf6dc:::benchmark_data_smithi173_55121_object27033:head# 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:170728af:::benchmark_data_smithi173_55121_object3358:head# 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17091cdd:::benchmark_data_smithi173_55121_object10343:head# 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:170d78c2:::benchmark_data_smithi173_55121_object4102:head# 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:170e001b:::benchmark_data_smithi173_55121_object27505:head# 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:170f216f:::benchmark_data_smithi173_55121_object29668:head# 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1715c6d6:::benchmark_data_smithi173_55121_object628:head# 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.763 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17162b3e:::benchmark_data_smithi173_55121_object27226:head# 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17196362:::benchmark_data_smithi173_55121_object22914:head# 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:171abffe:::benchmark_data_smithi173_55121_object11580:head# 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:171b42f0:::benchmark_data_smithi173_55121_object22136:head# 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:171c611c:::benchmark_data_smithi173_55121_object22137:head# 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:171ee01f:::benchmark_data_smithi173_55121_object20510:head# 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:171f4c77:::benchmark_data_smithi173_55121_object27609:head# 2024-09-17T16:12:14.764 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1720bc84:::benchmark_data_smithi173_55121_object3356:head# 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17249465:::benchmark_data_smithi173_55121_object3621:head# 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:172c23cd:::benchmark_data_smithi173_55121_object25154:head# 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:172cf7d4:::benchmark_data_smithi173_55121_object32355:head# 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:172db588:::benchmark_data_smithi173_55121_object36958:head# 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:172f5617:::benchmark_data_smithi173_55121_object6116:head# 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.765 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1731585c:::benchmark_data_smithi173_55121_object31520:head# 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17329e60:::benchmark_data_smithi173_55121_object10319:head# 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:173398f8:::benchmark_data_smithi173_55121_object33628:head# 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1734f29d:::benchmark_data_smithi173_55121_object4432:head# 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1735dd4e:::benchmark_data_smithi173_55121_object4209:head# 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1738d9dc:::benchmark_data_smithi173_55121_object27401:head# 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1738f569:::benchmark_data_smithi173_55121_object21218:head# 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.766 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:173bcae4:::benchmark_data_smithi173_55121_object244:head# 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:173d14e9:::benchmark_data_smithi173_55121_object29300:head# 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:173fbfa1:::benchmark_data_smithi173_55121_object27225:head# 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17426680:::benchmark_data_smithi173_55121_object8126:head# 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1742d75b:::benchmark_data_smithi173_55121_object1828:head# 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:174340da:::benchmark_data_smithi173_55121_object30089:head# 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1744720c:::benchmark_data_smithi173_55121_object36968:head# 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.767 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1744cd97:::benchmark_data_smithi173_55121_object30878:head# 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:174900c7:::benchmark_data_smithi173_55121_object8397:head# 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:174a53aa:::benchmark_data_smithi173_55121_object18426:head# 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:174c4b5f:::benchmark_data_smithi173_55121_object12692:head# 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17532bb7:::benchmark_data_smithi173_55121_object36465:head# 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17537ad6:::benchmark_data_smithi173_55121_object20843:head# 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1754a97a:::benchmark_data_smithi173_55121_object17886:head# 2024-09-17T16:12:14.768 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1755c74e:::benchmark_data_smithi173_55121_object25707:head# 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1756f249:::benchmark_data_smithi173_55121_object23192:head# 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1758bce2:::benchmark_data_smithi173_55121_object22785:head# 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1758d5e4:::benchmark_data_smithi173_55121_object4066:head# 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1758e85e:::benchmark_data_smithi173_55121_object32667:head# 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:175c8447:::benchmark_data_smithi173_55121_object36464:head# 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.769 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:175cab30:::benchmark_data_smithi173_55121_object35650:head# 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:175e4d57:::benchmark_data_smithi173_55121_object25141:head# 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1761bc94:::benchmark_data_smithi173_55121_object5871:head# 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17640e40:::benchmark_data_smithi173_55121_object20807:head# 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:176594e8:::benchmark_data_smithi173_55121_object34975:head# 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17676c75:::benchmark_data_smithi173_55121_object10823:head# 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17682204:::benchmark_data_smithi173_55121_object9472:head# 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.770 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1768b778:::benchmark_data_smithi173_55121_object20653:head# 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:176dcad2:::benchmark_data_smithi173_55121_object12820:head# 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:176ee98d:::benchmark_data_smithi173_55121_object33477:head# 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:176f985a:::benchmark_data_smithi173_55121_object21560:head# 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1771e225:::benchmark_data_smithi173_55121_object29186:head# 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:177535bd:::benchmark_data_smithi173_55121_object8120:head# 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1776518f:::benchmark_data_smithi173_55121_object25637:head# 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.771 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17779a8f:::benchmark_data_smithi173_55121_object19928:head# 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:177937f7:::benchmark_data_smithi173_55121_object33176:head# 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:177d1f3f:::benchmark_data_smithi173_55121_object2473:head# 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:177f52b3:::benchmark_data_smithi173_55121_object17718:head# 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:178038a3:::benchmark_data_smithi173_55121_object16519:head# 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:178153c8:::benchmark_data_smithi173_55121_object28772:head# 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17827585:::benchmark_data_smithi173_55121_object5167:head# 2024-09-17T16:12:14.772 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1782a5dc:::benchmark_data_smithi173_55121_object15670:head# 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17837ebf:::benchmark_data_smithi173_55121_object13209:head# 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1783dc86:::benchmark_data_smithi173_55121_object17520:head# 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17851b8b:::benchmark_data_smithi173_55121_object26270:head# 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1785efdb:::benchmark_data_smithi173_55121_object17821:head# 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17861710:::benchmark_data_smithi173_55121_object24625:head# 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:178889bd:::benchmark_data_smithi173_55121_object31931:head# 2024-09-17T16:12:14.773 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1788f491:::benchmark_data_smithi173_55121_object16324:head# 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:178944c9:::benchmark_data_smithi173_55121_object21184:head# 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:178c636b:::benchmark_data_smithi173_55121_object2288:head# 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:178eed9e:::benchmark_data_smithi173_55121_object7290:head# 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17914229:::benchmark_data_smithi173_55121_object26128:head# 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:179274a6:::benchmark_data_smithi173_55121_object6108:head# 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.774 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:179370c0:::benchmark_data_smithi173_55121_object27476:head# 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:179390b6:::benchmark_data_smithi173_55121_object16620:head# 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1799709d:::benchmark_data_smithi173_55121_object34951:head# 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:1799e643:::benchmark_data_smithi173_55121_object24215:head# 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17a0db94:::benchmark_data_smithi173_55121_object9580:head# 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17a8aeb2:::benchmark_data_smithi173_55121_object25127:head# 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17a8e4ab:::benchmark_data_smithi173_55121_object28227:head# 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17a8f7a3:::benchmark_data_smithi173_55121_object25016:head# 2024-09-17T16:12:14.775 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17a9c0ce:::benchmark_data_smithi173_55121_object19064:head# 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17aab4c1:::benchmark_data_smithi173_55121_object26511:head# 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17ab1a53:::benchmark_data_smithi173_55121_object19734:head# 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17ab72e4:::benchmark_data_smithi173_55121_object15945:head# 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17acb3e9:::benchmark_data_smithi173_55121_object8037:head# 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17b22614:::benchmark_data_smithi173_55121_object17660:head# 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.776 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17b34a8c:::benchmark_data_smithi173_55121_object14771:head# 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17b406b5:::benchmark_data_smithi173_55121_object31040:head# 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17b713d4:::benchmark_data_smithi173_55121_object29180:head# 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17b8de45:::benchmark_data_smithi173_55121_object33895:head# 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17b91f1a:::benchmark_data_smithi173_55121_object12102:head# 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17c0ed84:::benchmark_data_smithi173_55121_object24002:head# 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17c33ea7:::benchmark_data_smithi173_55121_object11917:head# 2024-09-17T16:12:14.777 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17c4b287:::benchmark_data_smithi173_55121_object766:head# 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17c4e44a:::benchmark_data_smithi173_55121_object9205:head# 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17c88cfe:::benchmark_data_smithi173_55121_object11800:head# 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17c95945:::benchmark_data_smithi173_55121_object22370:head# 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17ca5ebf:::benchmark_data_smithi173_55121_object14599:head# 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17cb88cc:::benchmark_data_smithi173_55121_object1396:head# 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17cee0cb:::benchmark_data_smithi173_55121_object32595:head# 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.778 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17ceefa5:::benchmark_data_smithi173_55121_object22959:head# 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17d33506:::benchmark_data_smithi173_55121_object19260:head# 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17d4a1f6:::benchmark_data_smithi173_55121_object22461:head# 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17d543e7:::benchmark_data_smithi173_55121_object10529:head# 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17d9d890:::benchmark_data_smithi173_55121_object21272:head# 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17de257b:::benchmark_data_smithi173_55121_object22100:head# 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17de3be6:::benchmark_data_smithi173_55121_object4833:head# 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17de9d2d:::benchmark_data_smithi173_55121_object15490:head# 2024-09-17T16:12:14.779 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17df3f6b:::benchmark_data_smithi173_55121_object13596:head# 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17df4ade:::benchmark_data_smithi173_55121_object15703:head# 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17e03d2c:::benchmark_data_smithi173_55121_object1347:head# 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17e2dc5c:::benchmark_data_smithi173_55121_object12558:head# 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17e464e4:::benchmark_data_smithi173_55121_object15268:head# 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17e62a85:::benchmark_data_smithi173_55121_object22903:head# 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17ea9bf8:::benchmark_data_smithi173_55121_object15833:head# 2024-09-17T16:12:14.780 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17ec8ef6:::benchmark_data_smithi173_55121_object33423:head# 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17ecce6d:::benchmark_data_smithi173_55121_object27850:head# 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17ee15e1:::benchmark_data_smithi173_55121_object19766:head# 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17eed0d1:::benchmark_data_smithi173_55121_object25350:head# 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17f00105:::benchmark_data_smithi173_55121_object33621:head# 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17f28099:::benchmark_data_smithi173_55121_object17841:head# 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17f3876b:::benchmark_data_smithi173_55121_object25641:head# 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.781 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17f72219:::benchmark_data_smithi173_55121_object10165:head# 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17f9649d:::benchmark_data_smithi173_55121_object20104:head# 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17fc2e87:::benchmark_data_smithi173_55121_object15061:head# 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17fd5239:::benchmark_data_smithi173_55121_object14073:head# 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:Write #2:17fed838:::benchmark_data_smithi173_55121_object2570:head# 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:write_pg epoch 132 info 2.8( v 132'1644 (0'0,132'1644] local-lis/les=104/105 n=1214 ec=95/95 lis/c=104/104 les/c/f=105/105/0 sis=104) 2024-09-17T16:12:14.782 INFO:teuthology.orchestra.run.smithi071.stdout:Import successful 2024-09-17T16:12:14.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:14.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:14.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:14.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:14.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:12:14.844 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:12:14.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:14 smithi173 ceph-mon[30530]: pgmap v385: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 2.9 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 560 op/s; 10527/142743 objects degraded (7.375%); 1549/142743 objects misplaced (1.085%) 2024-09-17T16:12:14.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:14 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:14.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:14 smithi173 ceph-mon[30530]: osdmap e144: 12 total, 11 up, 12 in 2024-09-17T16:12:14.911 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:14.955 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:14.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:15.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:15.101 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:15.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:15.189 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:15.203 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:15.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 53465 53449 45.7565 70.8125 0.00606535 0.0218472 2024-09-17T16:12:15.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:15.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:15.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:15.360 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:15.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:15.462 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:15.469 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:15.486 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:15.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:15.561 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:12:15.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:15.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:15.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:15.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:15.755 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:15.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:15.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:15.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:15.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:15.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:15.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:15 smithi173 ceph-mon[30530]: 2.5 scrub starts 2024-09-17T16:12:15.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:15 smithi173 ceph-mon[30530]: 2.5 scrub ok 2024-09-17T16:12:15.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:15 smithi173 ceph-mon[30530]: 2.2 scrub starts 2024-09-17T16:12:15.926 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:15 smithi017 ceph-mon[27571]: 2.5 scrub starts 2024-09-17T16:12:15.926 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:15 smithi017 ceph-mon[27571]: 2.5 scrub ok 2024-09-17T16:12:15.926 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:15 smithi017 ceph-mon[27571]: 2.2 scrub starts 2024-09-17T16:12:15.986 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:16.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:16.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:16.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:16.011 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:15 smithi071 ceph-mon[30002]: 2.5 scrub starts 2024-09-17T16:12:16.011 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:15 smithi071 ceph-mon[30002]: 2.5 scrub ok 2024-09-17T16:12:16.011 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:15 smithi071 ceph-mon[30002]: 2.2 scrub starts 2024-09-17T16:12:16.042 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/exp.2.8.7 2024-09-17T16:12:16.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:16.180 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:16.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:16.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:16.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 55072 55056 46.4953 100.438 0.00292799 0.0214997 2024-09-17T16:12:16.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:16.316 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:16.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:16.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:16.418 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:16.418 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:16.419 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:16.447 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:16.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:16.548 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:16.574 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:16.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:16.609 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:12:16.676 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:16.705 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:16.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:16.807 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:16.833 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:16.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:16.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:16 smithi173 ceph-mon[30530]: pgmap v387: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s; 11349/153942 objects degraded (7.372%); 1742/153942 objects misplaced (1.132%) 2024-09-17T16:12:16.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:16 smithi173 ceph-mon[30530]: 2.2 scrub ok 2024-09-17T16:12:16.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2262113349' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:16 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:16 smithi173 ceph-mon[30530]: 2.1 scrub starts 2024-09-17T16:12:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:16 smithi173 ceph-mon[30530]: 2.1 scrub ok 2024-09-17T16:12:16.936 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:16.960 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:16.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:16.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:16 smithi017 ceph-mon[27571]: pgmap v387: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s; 11349/153942 objects degraded (7.372%); 1742/153942 objects misplaced (1.132%) 2024-09-17T16:12:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:16 smithi017 ceph-mon[27571]: 2.2 scrub ok 2024-09-17T16:12:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2262113349' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:16 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:16 smithi017 ceph-mon[27571]: 2.1 scrub starts 2024-09-17T16:12:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:16 smithi017 ceph-mon[27571]: 2.1 scrub ok 2024-09-17T16:12:17.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:17.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:16 smithi071 ceph-mon[30002]: pgmap v387: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.1 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s; 11349/153942 objects degraded (7.372%); 1742/153942 objects misplaced (1.132%) 2024-09-17T16:12:17.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:16 smithi071 ceph-mon[30002]: 2.2 scrub ok 2024-09-17T16:12:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2262113349' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:16 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:16 smithi071 ceph-mon[30002]: 2.1 scrub starts 2024-09-17T16:12:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:16 smithi071 ceph-mon[30002]: 2.1 scrub ok 2024-09-17T16:12:17.093 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:17.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:17.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:17.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:17.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 56476 56460 47.0452 87.75 0.0621731 0.0212469 2024-09-17T16:12:17.216 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:17.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:17.319 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:17.342 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:17.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:17.444 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:17.469 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:17.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:17.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:17.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:17.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:17.695 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:17.720 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:17.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:17.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:17.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:17.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:17.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:17 smithi173 ceph-mon[30530]: 2.17 scrub starts 2024-09-17T16:12:17.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:17 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:12:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:17 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:17 smithi173 ceph-mon[30530]: osdmap e145: 12 total, 11 up, 12 in 2024-09-17T16:12:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:17 smithi173 ceph-mon[30530]: 2.17 scrub ok 2024-09-17T16:12:17.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:17.971 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:17.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:17 smithi017 ceph-mon[27571]: 2.17 scrub starts 2024-09-17T16:12:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:17 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:12:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:17 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:17 smithi017 ceph-mon[27571]: osdmap e145: 12 total, 11 up, 12 in 2024-09-17T16:12:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:17 smithi017 ceph-mon[27571]: 2.17 scrub ok 2024-09-17T16:12:18.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:17 smithi071 ceph-mon[30002]: 2.17 scrub starts 2024-09-17T16:12:18.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:17 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:12:18.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:17 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:17 smithi071 ceph-mon[30002]: osdmap e145: 12 total, 11 up, 12 in 2024-09-17T16:12:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:17 smithi071 ceph-mon[30002]: 2.17 scrub ok 2024-09-17T16:12:18.072 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:18.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:18.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:18.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:18.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:12:18.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 57573 57557 47.3283 68.5625 0.00305964 0.021123 2024-09-17T16:12:18.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:18.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:18.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:18.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:18.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:18.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:18.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:18.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:18.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:18.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:18.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:18.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:18.746 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:18.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:18.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:18.877 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:18.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:18.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:18 smithi173 ceph-mon[30530]: 2.7 scrub starts 2024-09-17T16:12:18.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:18 smithi173 ceph-mon[30530]: pgmap v389: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s; 12062/162204 objects degraded (7.436%); 1742/162204 objects misplaced (1.074%) 2024-09-17T16:12:18.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:18 smithi173 ceph-mon[30530]: 2.7 scrub ok 2024-09-17T16:12:18.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4213852604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:18.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:18.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:18 smithi017 ceph-mon[27571]: 2.7 scrub starts 2024-09-17T16:12:18.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:18 smithi017 ceph-mon[27571]: pgmap v389: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s; 12062/162204 objects degraded (7.436%); 1742/162204 objects misplaced (1.074%) 2024-09-17T16:12:18.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:18 smithi017 ceph-mon[27571]: 2.7 scrub ok 2024-09-17T16:12:18.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4213852604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:19.001 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:19.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:19.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:18 smithi071 ceph-mon[30002]: 2.7 scrub starts 2024-09-17T16:12:19.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:18 smithi071 ceph-mon[30002]: pgmap v389: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s; 12062/162204 objects degraded (7.436%); 1742/162204 objects misplaced (1.074%) 2024-09-17T16:12:19.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:18 smithi071 ceph-mon[30002]: 2.7 scrub ok 2024-09-17T16:12:19.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4213852604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:19.103 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:19.133 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:19.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:19.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 59163 59147 48.0041 99.375 0.00651787 0.0208263 2024-09-17T16:12:19.235 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:19.263 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:19.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:19.265 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:12:19.265 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:12:19.293 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:12:19.293 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:12:19.319 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:12:19.319 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:12:19.352 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:12:19.352 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:12:19.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:19.377 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:12:19.377 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:12:19.390 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:19.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:19.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:12:19.400 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:12:19.429 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:12:19.430 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:12:19.430 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:12:19.461 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:12:19.462 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:12:19.470 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:19.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:19.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:19.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:19.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:19.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:19.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:19.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:19.695 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:12:19.696 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:12:19.725 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:12:19.726 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:12:19.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:19.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:12:19.760 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:12:19.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:19.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:19.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:19.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:19 smithi173 ceph-mon[30530]: 2.d scrub starts 2024-09-17T16:12:19.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:19 smithi173 ceph-mon[30530]: 2.6 scrub starts 2024-09-17T16:12:19.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:19 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 12062/162204 objects degraded (7.436%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:19 smithi173 ceph-mon[30530]: 2.6 scrub ok 2024-09-17T16:12:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:19 smithi173 ceph-mon[30530]: osdmap e146: 12 total, 11 up, 12 in 2024-09-17T16:12:19.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:19.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:19 smithi017 ceph-mon[27571]: 2.d scrub starts 2024-09-17T16:12:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:19 smithi017 ceph-mon[27571]: 2.6 scrub starts 2024-09-17T16:12:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:19 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 12062/162204 objects degraded (7.436%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:19 smithi017 ceph-mon[27571]: 2.6 scrub ok 2024-09-17T16:12:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:19 smithi017 ceph-mon[27571]: osdmap e146: 12 total, 11 up, 12 in 2024-09-17T16:12:20.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:20.043 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:20.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:20.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:19 smithi071 ceph-mon[30002]: 2.d scrub starts 2024-09-17T16:12:20.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:19 smithi071 ceph-mon[30002]: 2.6 scrub starts 2024-09-17T16:12:20.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:19 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 12062/162204 objects degraded (7.436%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:20.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:19 smithi071 ceph-mon[30002]: 2.6 scrub ok 2024-09-17T16:12:20.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:19 smithi071 ceph-mon[30002]: osdmap e146: 12 total, 11 up, 12 in 2024-09-17T16:12:20.144 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:20.168 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:20.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:20.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 15 60947 60932 48.8188 111.562 0.00173629 0.0204769 2024-09-17T16:12:20.236 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:12:20.236 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:11672 2.1 e102/95 2:8c8823c2:::benchmark_data_smithi173_55121_object11671:head v 102'766, mlcod=102'762)", 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:10.815447+0000", 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.417607648000001, 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.189146908, 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.815447+0000", 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.815447+0000", 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.815448+0000", 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:12:20.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.815471+0000", 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2438000000000001e-05 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:11.003706+0000", 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.18823552299999999 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:11.003713+0000", 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9249999999999998e-06 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:11.003777+0000", 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4046000000000003e-05 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:11.003797+0000", 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9320000000000001e-05 2024-09-17T16:12:20.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:11.004565+0000", 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076825000000000005 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:11.004594+0000", 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9618e-05 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:21583 2.1 e117/104 2:836ecb44:::benchmark_data_smithi173_55121_object21582:head v 117'1119, mlcod=117'1116)", 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:26.421536+0000", 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 53.811518962999997, 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.11794297199999999, 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.421536+0000", 2024-09-17T16:12:20.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.421536+0000", 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.421537+0000", 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.421563+0000", 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.637e-05 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.538644+0000", 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.117080343 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.538652+0000", 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5089999999999993e-06 2024-09-17T16:12:20.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.538725+0000", 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2407e-05 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.538746+0000", 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1807e-05 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.539433+0000", 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00068611099999999999 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.539479+0000", 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6554999999999997e-05 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.246 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(2.11 118/107 [PushReplyOp(2:8afd0151:::benchmark_data_smithi173_55121_object9596:head)])", 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:27.897565+0000", 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 52.335489854999999, 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.091020158000000004, 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:27.897565+0000", 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:27.897565+0000", 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:27.897566+0000", 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8000000000000001e-07 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:27.897569+0000", 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3139999999999999e-06 2024-09-17T16:12:20.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:27.988159+0000", 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.090589978000000002 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:27.988168+0000", 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6289999999999999e-06 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:27.988238+0000", 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0382999999999993e-05 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:27.988585+0000", 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00034747400000000001 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:24657 2.19 e121/114 2:9911b308:::benchmark_data_smithi173_55121_object24656:head v 121'1224, mlcod=121'1221)", 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.346037+0000", 2024-09-17T16:12:20.250 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.887018214000001, 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.2241959, 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.346037+0000", 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.346037+0000", 2024-09-17T16:12:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.346038+0000", 2024-09-17T16:12:20.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8100000000000002e-07 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.346059+0000", 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1831000000000002e-05 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.569624+0000", 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.223564295 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.569628+0000", 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.442e-06 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.569673+0000", 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4765999999999998e-05 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.569692+0000", 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8853999999999999e-05 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.570206+0000", 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051373300000000005 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.570233+0000", 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7297999999999999e-05 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25217 2.19 e121/114 2:9a6d6ce1:::benchmark_data_smithi173_55121_object25216:head v 121'1238, mlcod=121'1235)", 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.777156+0000", 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.455899156000001, 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25947270300000003, 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777156+0000", 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777156+0000", 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777157+0000", 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7300000000000005e-07 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777182+0000", 2024-09-17T16:12:20.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5307e-05 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035767+0000", 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25858533500000003 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035781+0000", 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3567000000000001e-05 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035903+0000", 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012237899999999999 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035938+0000", 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4215999999999999e-05 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.036602+0000", 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00066465100000000004 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.036629+0000", 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6475e-05 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25228 2.1 e121/104 2:8352dd7d:::benchmark_data_smithi173_55121_object25227:head v 121'1242, mlcod=121'1239)", 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.786569+0000", 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.446485887999998, 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27958702099999999, 2024-09-17T16:12:20.260 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.261 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.261 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786569+0000", 2024-09-17T16:12:20.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786569+0000", 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786570+0000", 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1399999999999997e-07 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786589+0000", 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8754999999999999e-05 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065420+0000", 2024-09-17T16:12:20.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.278831679 2024-09-17T16:12:20.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065429+0000", 2024-09-17T16:12:20.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4149999999999999e-06 2024-09-17T16:12:20.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065499+0000", 2024-09-17T16:12:20.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9944000000000007e-05 2024-09-17T16:12:20.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065517+0000", 2024-09-17T16:12:20.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8499e-05 2024-09-17T16:12:20.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066120+0000", 2024-09-17T16:12:20.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060275699999999999 2024-09-17T16:12:20.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066156+0000", 2024-09-17T16:12:20.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6358e-05 2024-09-17T16:12:20.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.267 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.267 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25266 2.1 e121/104 2:82d1421d:::benchmark_data_smithi173_55121_object25265:head v 121'1243, mlcod=121'1240)", 2024-09-17T16:12:20.267 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:31.065373+0000", 2024-09-17T16:12:20.267 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.167682499000001, 2024-09-17T16:12:20.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.170985634, 2024-09-17T16:12:20.267 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.267 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065373+0000", 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065373+0000", 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065374+0000", 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2410000000000001e-06 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065398+0000", 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4576999999999999e-05 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.235531+0000", 2024-09-17T16:12:20.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.17013220100000001 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.235537+0000", 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1399999999999997e-06 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.235610+0000", 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3301999999999997e-05 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.235627+0000", 2024-09-17T16:12:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7067000000000001e-05 2024-09-17T16:12:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.236307+0000", 2024-09-17T16:12:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067963300000000002 2024-09-17T16:12:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.236358+0000", 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1473e-05 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25284 2.1 e121/104 2:800217e1:::benchmark_data_smithi173_55121_object25283:head v 121'1244, mlcod=121'1241)", 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:31.235476+0000", 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 48.997579381000001, 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.162725433, 2024-09-17T16:12:20.272 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.235476+0000", 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.235476+0000", 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.235477+0000", 2024-09-17T16:12:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5200000000000001e-06 2024-09-17T16:12:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.235505+0000", 2024-09-17T16:12:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.739e-05 2024-09-17T16:12:20.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.397432+0000", 2024-09-17T16:12:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.16192688399999999 2024-09-17T16:12:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.397438+0000", 2024-09-17T16:12:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3489999999999997e-06 2024-09-17T16:12:20.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.397502+0000", 2024-09-17T16:12:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4441000000000002e-05 2024-09-17T16:12:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.397525+0000", 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2866000000000001e-05 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.398156+0000", 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00063067400000000001 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.398201+0000", 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5309000000000003e-05 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:28922 2.1 e128/104 2:82213792:::benchmark_data_smithi173_55121_object28921:head v 128'1355, mlcod=128'1353)", 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:48.757614+0000", 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 31.475441623999998, 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.114584171, 2024-09-17T16:12:20.278 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.757614+0000", 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.757614+0000", 2024-09-17T16:12:20.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.757614+0000", 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5000000000000002e-07 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.757635+0000", 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1140000000000001e-05 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.871475+0000", 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.113839127 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.871479+0000", 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2699999999999998e-06 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.871529+0000", 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0455999999999999e-05 2024-09-17T16:12:20.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.871550+0000", 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.048e-05 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.872168+0000", 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061809400000000002 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.872198+0000", 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9853999999999999e-05 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:29460 2.1 e128/104 2:83aee601:::benchmark_data_smithi173_55121_object29459:head v 128'1369, mlcod=128'1366)", 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.198210+0000", 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 31.034844854999999, 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31777874900000003, 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.283 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198210+0000", 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198210+0000", 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198211+0000", 2024-09-17T16:12:20.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-17T16:12:20.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198235+0000", 2024-09-17T16:12:20.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3884999999999999e-05 2024-09-17T16:12:20.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515223+0000", 2024-09-17T16:12:20.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:20.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31698817499999998 2024-09-17T16:12:20.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515231+0000", 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9189999999999999e-06 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515321+0000", 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9240000000000006e-05 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515344+0000", 2024-09-17T16:12:20.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3362000000000001e-05 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515954+0000", 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000610291 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515989+0000", 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4758000000000002e-05 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.289 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:29625 2.5 e129/104 2:a21e4127:::benchmark_data_smithi173_55121_object29624:head v 129'1363, mlcod=129'1360)", 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.564207+0000", 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.668848094000001, 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29265457499999997, 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564207+0000", 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564207+0000", 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564208+0000", 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2399999999999998e-07 2024-09-17T16:12:20.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564411+0000", 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020328800000000001 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856209+0000", 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29179771999999998 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856213+0000", 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0579999999999999e-06 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856281+0000", 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7911999999999996e-05 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856295+0000", 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4559000000000001e-05 2024-09-17T16:12:20.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856830+0000", 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00053516399999999997 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856862+0000", 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1350000000000003e-05 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:29763 2.5 e129/104 2:a5e91dae:::benchmark_data_smithi173_55121_object29762:head v 129'1365, mlcod=129'1362)", 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.874594+0000", 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.358461482999999, 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.12601459300000001, 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.874594+0000", 2024-09-17T16:12:20.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.874594+0000", 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.874594+0000", 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3600000000000003e-07 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.874692+0000", 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8026e-05 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.999834+0000", 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.125141474 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.999841+0000", 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7560000000000002e-06 2024-09-17T16:12:20.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.999888+0000", 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7419000000000001e-05 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.999907+0000", 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9423999999999999e-05 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.000553+0000", 2024-09-17T16:12:20.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00064563900000000004 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.000608+0000", 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5219000000000001e-05 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:30890 2.c e131/95 2:3121263d:::benchmark_data_smithi173_55121_object30889:head v 131'1945, mlcod=131'1941)", 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:51.796812+0000", 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.436243211000001, 2024-09-17T16:12:20.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.102687853, 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.796812+0000", 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.796812+0000", 2024-09-17T16:12:20.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.796813+0000", 2024-09-17T16:12:20.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.35e-06 2024-09-17T16:12:20.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.796907+0000", 2024-09-17T16:12:20.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3640000000000005e-05 2024-09-17T16:12:20.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.898304+0000", 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.10139696300000001 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.898312+0000", 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9209999999999992e-06 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.898381+0000", 2024-09-17T16:12:20.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9381000000000005e-05 2024-09-17T16:12:20.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.898404+0000", 2024-09-17T16:12:20.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2872999999999999e-05 2024-09-17T16:12:20.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.899449+0000", 2024-09-17T16:12:20.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001045378 2024-09-17T16:12:20.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.899500+0000", 2024-09-17T16:12:20.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0346999999999998e-05 2024-09-17T16:12:20.306 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.306 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.307 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:38612 2.5 e135/104 2:a07c47e7:::benchmark_data_smithi173_55121_object38611:head v 135'1641, mlcod=135'1638)", 2024-09-17T16:12:20.307 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:58.822496+0000", 2024-09-17T16:12:20.307 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 21.410559421999999, 2024-09-17T16:12:20.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.13895849099999999, 2024-09-17T16:12:20.308 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.308 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.308 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.308 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.822496+0000", 2024-09-17T16:12:20.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.822496+0000", 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.822501+0000", 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.887e-06 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.822591+0000", 2024-09-17T16:12:20.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0723999999999994e-05 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.960660+0000", 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.13806882600000001 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.960664+0000", 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9129999999999996e-06 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.960712+0000", 2024-09-17T16:12:20.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8238999999999997e-05 2024-09-17T16:12:20.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.960726+0000", 2024-09-17T16:12:20.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4032000000000001e-05 2024-09-17T16:12:20.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.961426+0000", 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00069920300000000002 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.961454+0000", 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8666999999999998e-05 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:38725 2.5 e135/104 2:a46f9753:::benchmark_data_smithi173_55121_object38724:head v 135'1642, mlcod=135'1639)", 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:58.960574+0000", 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 21.272481234000001, 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.167981611, 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.960574+0000", 2024-09-17T16:12:20.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.960574+0000", 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.960574+0000", 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8299999999999997e-07 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:58.960646+0000", 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1946000000000005e-05 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:59.127591+0000", 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.16694473800000001 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:59.127598+0000", 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0260000000000002e-06 2024-09-17T16:12:20.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:59.127668+0000", 2024-09-17T16:12:20.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9703999999999995e-05 2024-09-17T16:12:20.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:59.127690+0000", 2024-09-17T16:12:20.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1679e-05 2024-09-17T16:12:20.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:59.128522+0000", 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083272800000000003 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:59.128556+0000", 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3207000000000002e-05 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:41577 2.5 e136/104 2:a5eda23c:::benchmark_data_smithi173_55121_object41576:head v 136'1740, mlcod=136'1737)", 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:00.860950+0000", 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 19.372104849999999, 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.106295915, 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:00.860950+0000", 2024-09-17T16:12:20.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:00.860950+0000", 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:00.860951+0000", 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2699999999999998e-07 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.319 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:00.861014+0000", 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3372000000000002e-05 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:00.966434+0000", 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.10541956299999999 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:00.966438+0000", 2024-09-17T16:12:20.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2350000000000001e-06 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:00.966494+0000", 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6186e-05 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:00.966509+0000", 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.438e-05 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:00.967209+0000", 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070006799999999998 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:00.967246+0000", 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7283999999999997e-05 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:42646 2.4 e136/119 2:27a11e99:::benchmark_data_smithi173_55121_object42645:head v 136'1796, mlcod=136'1793)", 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:01.496233+0000", 2024-09-17T16:12:20.322 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.736822096000001, 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.094998916000000003, 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.496233+0000", 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.496233+0000", 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.496234+0000", 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.004e-06 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.496260+0000", 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6026000000000001e-05 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.590105+0000", 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.093844736999999998 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.590112+0000", 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.187e-06 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.590179+0000", 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6659e-05 2024-09-17T16:12:20.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.590200+0000", 2024-09-17T16:12:20.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1382999999999998e-05 2024-09-17T16:12:20.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.591200+0000", 2024-09-17T16:12:20.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099962300000000005 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.591232+0000", 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2296999999999999e-05 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:43085 2.4 e136/119 2:20089549:::benchmark_data_smithi173_55121_object43084:head v 136'1807, mlcod=136'1804)", 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:01.800298+0000", 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.432757475999999, 2024-09-17T16:12:20.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.10756911299999999, 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.800298+0000", 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.800298+0000", 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.800298+0000", 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2200000000000003e-07 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.800327+0000", 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8869000000000001e-05 2024-09-17T16:12:20.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.906775+0000", 2024-09-17T16:12:20.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.106447654 2024-09-17T16:12:20.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.906782+0000", 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.72e-06 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.906859+0000", 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7643000000000002e-05 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.906886+0000", 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6435e-05 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.907825+0000", 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093897700000000002 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:01.907867+0000", 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1993000000000003e-05 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:47692 2.8 e142/133 2:16c73e1c:::benchmark_data_smithi173_55121_object47691:head v 142'1980, mlcod=142'1977)", 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:10.456120+0000", 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.7769348320000002, 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.208404231, 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.456120+0000", 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.456120+0000", 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.456121+0000", 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3200000000000005e-07 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.456230+0000", 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108797 2024-09-17T16:12:20.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.663492+0000", 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.20726259399999999 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.663505+0000", 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2853999999999999e-05 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.663642+0000", 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000136909 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.663707+0000", 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4958999999999999e-05 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.664494+0000", 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078647299999999999 2024-09-17T16:12:20.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.664525+0000", 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1013000000000003e-05 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:47793 2.4 e142/119 2:26db6692:::benchmark_data_smithi173_55121_object47792:head v 142'1952, mlcod=142'1949)", 2024-09-17T16:12:20.337 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:10.581653+0000", 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.6514017780000003, 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.190504216, 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.581653+0000", 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.581653+0000", 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.581654+0000", 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4099999999999998e-07 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.581678+0000", 2024-09-17T16:12:20.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3824000000000001e-05 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.771275+0000", 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.189597196 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.771282+0000", 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.939e-06 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.771367+0000", 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5309000000000006e-05 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.771382+0000", 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4931e-05 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.772125+0000", 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00074277700000000002 2024-09-17T16:12:20.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:20.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:20.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:20.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.772158+0000", 2024-09-17T16:12:20.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2499000000000002e-05 2024-09-17T16:12:20.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.342 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:20.342 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:20.342 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:12:20.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:20.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:20.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:20.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:20.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:20.599 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:20.623 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:20.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:20.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:20.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:20.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:12:20.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:12:20.850 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:20.878 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:20.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:20.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:20 smithi173 ceph-mon[30530]: 2.d scrub ok 2024-09-17T16:12:20.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:20 smithi173 ceph-mon[30530]: pgmap v391: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s; 12485/168501 objects degraded (7.409%); 1742/168501 objects misplaced (1.034%) 2024-09-17T16:12:20.980 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:20.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:20 smithi017 ceph-mon[27571]: 2.d scrub ok 2024-09-17T16:12:20.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:20 smithi017 ceph-mon[27571]: pgmap v391: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s; 12485/168501 objects degraded (7.409%); 1742/168501 objects misplaced (1.034%) 2024-09-17T16:12:21.006 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:21.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:21.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:20 smithi071 ceph-mon[30002]: 2.d scrub ok 2024-09-17T16:12:21.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:20 smithi071 ceph-mon[30002]: pgmap v391: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s; 12485/168501 objects degraded (7.409%); 1742/168501 objects misplaced (1.034%) 2024-09-17T16:12:21.108 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:21.120 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] 2024-09-17T16:12:21.121 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:12:21.121 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T16:12:21.121 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-09-17T16:12:21.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-17T16:12:21.131 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:21.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:21.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 62276 62260 49.2514 83 0.00329747 0.0202976 2024-09-17T16:12:21.233 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:21.256 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:21.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:21.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:21.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:21.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:21.455 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:12:21.491 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:21.530 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:21.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:21.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:21.553 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:12:21.623 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:21.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:21.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:21.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:21.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:21.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:21.795 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:21.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:21.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:21.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:21.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:22.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:22.057 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:22.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:22.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:22.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:22.192 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:22.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:22.215 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:12:22.214975+0000 min lat: 0.00105621 max lat: 15.0115 avg lat: 0.020068 2024-09-17T16:12:22.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:12:22.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 63779 63763 49.8099 93.9375 0.00501273 0.020068 2024-09-17T16:12:22.218 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:22.223 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:22.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:22.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:22.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:22.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:22.424 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:22.456 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:22.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:22.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:22.558 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:22.582 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:22.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:22.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:22.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:22.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:22.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:12:22.810 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-17T16:12:22.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:22.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:22.839 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:22.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:22.941 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:22.973 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:22.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:22.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:22 smithi017 ceph-mon[27571]: pgmap v392: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 434 op/s; 13589/183771 objects degraded (7.395%); 1944/183771 objects misplaced (1.058%) 2024-09-17T16:12:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:22 smithi017 ceph-mon[27571]: 2.10 scrub starts 2024-09-17T16:12:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:22 smithi017 ceph-mon[27571]: 2.10 scrub ok 2024-09-17T16:12:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2589480717' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:12:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:22 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:12:23.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:22 smithi071 ceph-mon[30002]: pgmap v392: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 434 op/s; 13589/183771 objects degraded (7.395%); 1944/183771 objects misplaced (1.058%) 2024-09-17T16:12:23.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:22 smithi071 ceph-mon[30002]: 2.10 scrub starts 2024-09-17T16:12:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:22 smithi071 ceph-mon[30002]: 2.10 scrub ok 2024-09-17T16:12:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2589480717' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:12:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:22 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:12:23.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:23.098 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:23.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:23.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:22 smithi173 ceph-mon[30530]: pgmap v392: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 434 op/s; 13589/183771 objects degraded (7.395%); 1944/183771 objects misplaced (1.058%) 2024-09-17T16:12:23.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:22 smithi173 ceph-mon[30530]: 2.10 scrub starts 2024-09-17T16:12:23.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:22 smithi173 ceph-mon[30530]: 2.10 scrub ok 2024-09-17T16:12:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2589480717' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:12:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:22 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:12:23.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:23.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 65349 65333 50.4062 98.125 0.00289321 0.0198301 2024-09-17T16:12:23.223 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:23.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:23.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:23.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:23.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:23.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:23.451 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:23.472 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:23.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:23.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:23.596 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:23.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:23.698 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:23.727 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:23.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:23.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:23.822 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:12:23.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:23.861 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:23.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:23.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:23.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:23.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:24.091 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:24.113 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:24.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:24.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:23 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:12:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:23 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:12:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:23 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:12:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:23 smithi173 ceph-mon[30530]: osdmap e147: 12 total, 11 up, 12 in 2024-09-17T16:12:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:23 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:12:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:23 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:12:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4220238508' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:23 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:24.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:23 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:12:24.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:23 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:12:24.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:23 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:12:24.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:23 smithi017 ceph-mon[27571]: osdmap e147: 12 total, 11 up, 12 in 2024-09-17T16:12:24.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:23 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:12:24.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:23 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:12:24.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4220238508' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:24.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:23 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:24.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:24.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 66641 66625 50.7762 80.75 0.0259091 0.0196841 2024-09-17T16:12:24.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:24.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:24.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:24.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:23 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:12:24.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:23 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:12:24.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:23 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:12:24.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:23 smithi071 ceph-mon[30002]: osdmap e147: 12 total, 11 up, 12 in 2024-09-17T16:12:24.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:23 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:12:24.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:23 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:12:24.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4220238508' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:24.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:23 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:24.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:24.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:24.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:24.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:24.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:24.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:24.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:24.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:24.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:24.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:24.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:24.759 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:24.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:24.861 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:24.886 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:24.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:24.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:24.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:24 smithi017 ceph-mon[27571]: pgmap v394: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13589/183771 objects degraded (7.395%); 1944/183771 objects misplaced (1.058%) 2024-09-17T16:12:24.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:24 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:24.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:24 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:12:24.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:24 smithi017 ceph-mon[27571]: osdmap e148: 12 total, 11 up, 12 in 2024-09-17T16:12:24.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:24 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 13589/183771 objects degraded (7.395%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:24.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:25.010 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:25.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:25.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:25.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:25.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:25.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:25.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:25.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:24 smithi173 ceph-mon[30530]: pgmap v394: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13589/183771 objects degraded (7.395%); 1944/183771 objects misplaced (1.058%) 2024-09-17T16:12:25.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:24 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:25.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:24 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:12:25.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:24 smithi173 ceph-mon[30530]: osdmap e148: 12 total, 11 up, 12 in 2024-09-17T16:12:25.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:24 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 13589/183771 objects degraded (7.395%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:25.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 67884 67868 51.1004 77.6875 0.0303995 0.0195628 2024-09-17T16:12:25.236 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:12:25.236 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:12:25.236 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:12:25.236 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:12:25.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:25.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:25.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:25.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:24 smithi071 ceph-mon[30002]: pgmap v394: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13589/183771 objects degraded (7.395%); 1944/183771 objects misplaced (1.058%) 2024-09-17T16:12:25.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:24 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:24 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:12:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:24 smithi071 ceph-mon[30002]: osdmap e148: 12 total, 11 up, 12 in 2024-09-17T16:12:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:24 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 13589/183771 objects degraded (7.395%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:25.367 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:25.396 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:25.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:25.484 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:25.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:25.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:25.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:25.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:25.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:25.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:25.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:25.788 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:25.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:25.890 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:25.918 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:25.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:25.985 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:12:26.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:26.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:26.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:26.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:26.173 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:26.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:26.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:26.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 69451 69435 51.6578 97.9375 0.0104066 0.0193521 2024-09-17T16:12:26.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:26.298 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:26.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:26.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:26.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:26.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:26.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:26.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:26.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:26.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:26.559 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:12:26.559 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:12:26.597 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:12:26.597 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:12:26.598 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:26.635 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:12:26.635 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:12:26.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:26.672 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:12:26.672 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:12:26.674 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:26.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:26.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:26.707 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:12:26.707 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:12:26.730 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:12:26.730 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:12:26.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:26.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:26.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:26.848 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:12:26.849 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:12:26.849 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:12:26.888 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:12:26.888 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:12:26.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:26.925 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:12:26.925 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:12:26.938 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:26.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:26.946 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:12:26.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:12:26.974 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:12:27.011 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:12:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:12:27.013 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:26 smithi017 ceph-mon[27571]: pgmap v396: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s wr, 105 op/s; 14465/196245 objects degraded (7.371%); 2162/196245 objects misplaced (1.102%) 2024-09-17T16:12:27.014 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3311541660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:27.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:27.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:27.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:27.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:26 smithi173 ceph-mon[30530]: pgmap v396: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s wr, 105 op/s; 14465/196245 objects degraded (7.371%); 2162/196245 objects misplaced (1.102%) 2024-09-17T16:12:27.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3311541660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:27.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:27.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:27.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:27.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:27.215 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:27.215 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:27.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 70600 70584 51.8949 71.8125 0.00346657 0.019256 2024-09-17T16:12:27.291 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:27.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:26 smithi071 ceph-mon[30002]: pgmap v396: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s wr, 105 op/s; 14465/196245 objects degraded (7.371%); 2162/196245 objects misplaced (1.102%) 2024-09-17T16:12:27.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3311541660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:27.314 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:27.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:27.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:27.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:27.441 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:27.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:27.543 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:27.567 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:27.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:27.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:27.692 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:27.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:27.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:27.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:27.815 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:27.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:27.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:12:27.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:12:27.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:27.943 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:27.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:28.045 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:28.066 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:28.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:28.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:28.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:28.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:28.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 71521 71505 51.9607 57.5625 0.0236019 0.0192405 2024-09-17T16:12:28.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:28.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:28.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:28.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:28.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:28.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:28.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:28.587 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:28.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:28.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:28.711 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:28.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:28.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:28.837 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:28.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:28.884 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:12:28.899 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:28.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:28.965 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:28.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:29.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:29.099 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:29.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:29.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:28 smithi173 ceph-mon[30530]: pgmap v397: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 406 op/s; 15132/205200 objects degraded (7.374%); 2162/205200 objects misplaced (1.054%) 2024-09-17T16:12:29.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:28 smithi017 ceph-mon[27571]: pgmap v397: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 406 op/s; 15132/205200 objects degraded (7.374%); 2162/205200 objects misplaced (1.054%) 2024-09-17T16:12:29.201 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:29.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 73009 72993 52.4323 93 0.00467224 0.0190592 2024-09-17T16:12:29.225 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:29.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:29.235 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11] 2024-09-17T16:12:29.235 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-17T16:12:29.235 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-09-17T16:12:29.235 DEBUG:teuthology.orchestra.run.smithi071:osd.7> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.7.service 2024-09-17T16:12:29.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl start ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.7 2024-09-17T16:12:29.257 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:28 smithi071 ceph-mon[30002]: pgmap v397: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 406 op/s; 15132/205200 objects degraded (7.374%); 2162/205200 objects misplaced (1.054%) 2024-09-17T16:12:29.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:29.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:29.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:29.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:29.454 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:29.493 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:29.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:29.514 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:29.521 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:29 smithi071 systemd[1]: Starting Ceph osd.7 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:12:29.596 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:29.628 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:29.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:29.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:29.759 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:29.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:29.813 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:29 smithi071 podman[61762]: 2024-09-17 16:12:29.423392158 +0000 UTC m=+0.019689847 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:12:29.814 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:29 smithi071 podman[61762]: 2024-09-17 16:12:29.553916816 +0000 UTC m=+0.150214497 container create f4e42daf467addffd3d9eecb59a13881dfbeb895953248fa1019d4e3c42076a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-17T16:12:29.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:29.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:12:29.887 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:29.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:29.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:30.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:30.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:30.115 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:30.140 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:30.141 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:30.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:29 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 15132/205200 objects degraded (7.374%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:30.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4253948804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:30.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:29 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:30.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:29 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 15132/205200 objects degraded (7.374%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:30.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4253948804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:30.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:29 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:30.149 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:30.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:30.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 74391 74375 52.8179 86.375 0.0115559 0.0189283 2024-09-17T16:12:30.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:30.265 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:29 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 15132/205200 objects degraded (7.374%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:30.265 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4253948804' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:30.265 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:29 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:30.280 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:30.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:30.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:30.286 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:12:30.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:30.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:30.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:30.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:30.508 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:30.530 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:30.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:30.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:30.563 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 podman[61762]: 2024-09-17 16:12:30.264156564 +0000 UTC m=+0.860454246 container init f4e42daf467addffd3d9eecb59a13881dfbeb895953248fa1019d4e3c42076a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:12:30.564 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 podman[61762]: 2024-09-17 16:12:30.268431629 +0000 UTC m=+0.864729314 container start f4e42daf467addffd3d9eecb59a13881dfbeb895953248fa1019d4e3c42076a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T16:12:30.564 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 podman[61762]: 2024-09-17 16:12:30.364366994 +0000 UTC m=+0.960664690 container attach f4e42daf467addffd3d9eecb59a13881dfbeb895953248fa1019d4e3c42076a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-17T16:12:30.564 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate[61778]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:12:30.564 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 bash[61762]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:12:30.564 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate[61778]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:12:30.564 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 bash[61762]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:12:30.632 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:30.656 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:30.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:30.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:30.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:30.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:30.890 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:30.919 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:30.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:30 smithi017 ceph-mon[27571]: pgmap v398: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 897 op/s; 15525/211137 objects degraded (7.353%); 2162/211137 objects misplaced (1.024%) 2024-09-17T16:12:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:30 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:30 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:30 smithi017 ceph-mon[27571]: osdmap e149: 12 total, 11 up, 12 in 2024-09-17T16:12:31.021 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:31.047 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:31.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:31.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:30 smithi173 ceph-mon[30530]: pgmap v398: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 897 op/s; 15525/211137 objects degraded (7.353%); 2162/211137 objects misplaced (1.024%) 2024-09-17T16:12:31.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:30 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:31.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:30 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:31.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:30 smithi173 ceph-mon[30530]: osdmap e149: 12 total, 11 up, 12 in 2024-09-17T16:12:31.148 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:31.173 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:31.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:31.192 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:30 smithi071 ceph-mon[30002]: pgmap v398: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 897 op/s; 15525/211137 objects degraded (7.353%); 2162/211137 objects misplaced (1.024%) 2024-09-17T16:12:31.192 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:30 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:31.192 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:30 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:31.192 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:30 smithi071 ceph-mon[30002]: osdmap e149: 12 total, 11 up, 12 in 2024-09-17T16:12:31.193 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate[61778]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T16:12:31.193 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 bash[61762]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T16:12:31.193 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate[61778]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T16:12:31.193 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:30 smithi071 bash[61762]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T16:12:31.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 75468 75452 52.9807 67.3125 0.00837665 0.0188669 2024-09-17T16:12:31.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:31.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:31.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:31.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:31.422 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:31.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:31.520 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate[61778]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T16:12:31.520 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 bash[61762]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T16:12:31.520 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate[61778]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T16:12:31.520 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 bash[61762]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T16:12:31.520 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate[61778]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-17T16:12:31.520 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 bash[61762]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-17T16:12:31.520 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate[61778]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T16:12:31.520 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 bash[61762]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T16:12:31.521 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate[61778]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-17T16:12:31.521 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 bash[61762]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-17T16:12:31.521 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 podman[61762]: 2024-09-17 16:12:31.221303133 +0000 UTC m=+1.817600827 container died f4e42daf467addffd3d9eecb59a13881dfbeb895953248fa1019d4e3c42076a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-17T16:12:31.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:31.549 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:31.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:31.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:31.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:12:31.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:31.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:31.733 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:31.786 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:31.818 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:31.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:31.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:31.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:31.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:32.049 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:32.079 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:32.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:32.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:32.207 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:32.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:32.216 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 14 76486 76472 53.1003 63.75 0.00330804 0.0188195 2024-09-17T16:12:32.305 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:12:32.306 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:12:32.309 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:32.314 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:31 smithi071 podman[61914]: 2024-09-17 16:12:31.953826166 +0000 UTC m=+0.726229350 container remove f4e42daf467addffd3d9eecb59a13881dfbeb895953248fa1019d4e3c42076a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-17T16:12:32.314 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:32 smithi071 podman[61988]: 2024-09-17 16:12:32.071785583 +0000 UTC m=+0.017293177 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:12:32.314 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:32 smithi071 podman[61988]: 2024-09-17 16:12:32.19679198 +0000 UTC m=+0.142299599 container create 012a121b3a81f37bedecd93d965ba29462235ac649273f031f204dcefc4ab090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:12:32.331 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:32.333 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:32.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:32.336 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:12:32.337 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:12:32.358 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:12:32.358 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:12:32.380 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:12:32.380 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:12:32.401 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:12:32.401 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:12:32.422 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:12:32.422 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:12:32.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:32.443 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:12:32.443 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:12:32.443 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:12:32.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:32.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:32.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:32.464 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:12:32.465 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:12:32.486 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:12:32.486 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:12:32.508 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:12:32.508 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:12:32.530 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:12:32.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:32.594 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:32.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:32.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:32.696 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:32.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:32.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:32.747 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:12:32.747 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:12:32.780 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:12:32.814 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:32 smithi071 podman[61988]: 2024-09-17 16:12:32.424307597 +0000 UTC m=+0.369815179 container init 012a121b3a81f37bedecd93d965ba29462235ac649273f031f204dcefc4ab090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:12:32.814 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:32 smithi071 podman[61988]: 2024-09-17 16:12:32.427837055 +0000 UTC m=+0.373344629 container start 012a121b3a81f37bedecd93d965ba29462235ac649273f031f204dcefc4ab090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T16:12:32.814 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:32 smithi071 ceph-osd[62007]: -- 172.21.15.71:0/3385158455 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5581a653da40 con 0x5581a730c400 2024-09-17T16:12:32.814 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:32 smithi071 bash[61988]: 012a121b3a81f37bedecd93d965ba29462235ac649273f031f204dcefc4ab090 2024-09-17T16:12:32.814 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:32 smithi071 systemd[1]: Started Ceph osd.7 for 19341aec-750e-11ef-bceb-c7b262605968. 2024-09-17T16:12:32.818 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:12:32.818 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:12:32.818 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:12:32.818 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:12:32.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:32.842 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:32.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:32.944 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:32.966 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:32.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:32.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:32 smithi017 ceph-mon[27571]: pgmap v400: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s; 16468/223617 objects degraded (7.364%); 2382/223617 objects misplaced (1.065%) 2024-09-17T16:12:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4278983743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:33.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:33.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:12:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:33.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:33.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:33.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:32 smithi173 ceph-mon[30530]: pgmap v400: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s; 16468/223617 objects degraded (7.364%); 2382/223617 objects misplaced (1.065%) 2024-09-17T16:12:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4278983743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:33.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:32 smithi071 ceph-mon[30002]: pgmap v400: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s; 16468/223617 objects degraded (7.364%); 2382/223617 objects misplaced (1.065%) 2024-09-17T16:12:33.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4278983743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:33.193 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:33.216 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:33.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:33.217 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.8725 2024-09-17T16:12:33.217 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 76486 2024-09-17T16:12:33.217 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-09-17T16:12:33.217 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-09-17T16:12:33.217 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 52.6053 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 39.6327 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 114.812 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 841 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 634.128 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 1837 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 0 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0188836 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.231686 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 15.0115 2024-09-17T16:12:33.218 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00105621 2024-09-17T16:12:33.223 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:0 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:1 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:2 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:abort 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:assert 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:config diff 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:config diff get 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:config help [] 2024-09-17T16:12:33.314 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2024-09-17T16:12:33.315 INFO:teuthology.orchestra.run.smithi071.stderr:admin_socket: invalid command 2024-09-17T16:12:33.318 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:33.345 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:33.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:12:33.410 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:12:33.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:33.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:33.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:33.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:33.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:33.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:33.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:33.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:33.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:33.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:33.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:33.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:33.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:33.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:33.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:33.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:34.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:12:34.100 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-17T16:12:34.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:34.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:34.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:34.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:34.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:34.313 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:34 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:12:34.058+0000 7f29ffac0640 -1 Falling back to public interface 2024-09-17T16:12:34.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:34.381 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:34.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:34.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:34.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:34.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:34.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:34.613 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:34.642 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:34.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:34.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:34.769 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:34.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:34.870 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:34.897 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:34.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:35.000 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:35.023 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:35.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:35.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:34 smithi071 ceph-mon[30002]: pgmap v401: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s; 16468/223617 objects degraded (7.364%); 2382/223617 objects misplaced (1.065%) 2024-09-17T16:12:35.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:34 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 16468/223617 objects degraded (7.364%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:35.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:34 smithi071 ceph-mon[30002]: osdmap e150: 12 total, 11 up, 12 in 2024-09-17T16:12:35.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:35.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:34 smithi173 ceph-mon[30530]: pgmap v401: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s; 16468/223617 objects degraded (7.364%); 2382/223617 objects misplaced (1.065%) 2024-09-17T16:12:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:34 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 16468/223617 objects degraded (7.364%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:34 smithi173 ceph-mon[30530]: osdmap e150: 12 total, 11 up, 12 in 2024-09-17T16:12:35.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:35.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:34 smithi017 ceph-mon[27571]: pgmap v401: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s; 16468/223617 objects degraded (7.364%); 2382/223617 objects misplaced (1.065%) 2024-09-17T16:12:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:34 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 16468/223617 objects degraded (7.364%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:34 smithi017 ceph-mon[27571]: osdmap e150: 12 total, 11 up, 12 in 2024-09-17T16:12:35.248 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:35.272 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:35.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:35.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:35.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:35.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:35.411 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:12:35.465 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:12:35.483 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:35.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:35.537 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:35.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:35.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:35.672 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:35.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:35.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:35.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:35.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:35.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:35.928 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:35.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:36.028 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:35 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:12:36.028 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:35 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:12:36.028 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:35 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:12:36.028 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:35 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:12:36.031 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:36.052 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:36.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:36.079 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:36.080 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:36.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:35 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:12:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:35 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:12:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:35 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:12:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:35 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:12:36.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:36.179 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:36.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:36.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:36.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:36.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:36.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:36.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:35 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:12:36.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:35 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:12:36.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:35 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:12:36.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:35 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:12:36.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:36.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:36.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:36.538 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:36.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:36.566 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:36.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:36.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:36.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:36.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:36.800 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:36.830 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:36.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:36.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:36.855 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:12:36.855 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:36 smithi017 ceph-mon[27571]: 2.12 scrub starts 2024-09-17T16:12:36.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:36 smithi017 ceph-mon[27571]: pgmap v403: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 1.16k op/s; 16458/223659 objects degraded (7.359%); 2330/223659 objects misplaced (1.042%) 2024-09-17T16:12:36.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:36 smithi017 ceph-mon[27571]: osdmap e151: 12 total, 11 up, 12 in 2024-09-17T16:12:36.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2224388649' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:12:36.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:36 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:12:36.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/400031436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:36.864 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:12:36.864 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:12:36.899 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:12:36.899 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:12:36.926 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:12:36.926 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:12:36.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:36.954 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:36.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:36.958 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:12:36.958 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:12:36.992 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:12:36.992 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:12:37.027 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:12:37.027 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:12:37.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:37.070 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:12:37.070 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:12:37.070 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:12:37.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:37.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:37.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:37.090 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:37.090 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:37.109 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:12:37.109 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:12:37.143 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:12:37.143 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:12:37.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:36 smithi173 ceph-mon[30530]: 2.12 scrub starts 2024-09-17T16:12:37.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:36 smithi173 ceph-mon[30530]: pgmap v403: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 1.16k op/s; 16458/223659 objects degraded (7.359%); 2330/223659 objects misplaced (1.042%) 2024-09-17T16:12:37.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:36 smithi173 ceph-mon[30530]: osdmap e151: 12 total, 11 up, 12 in 2024-09-17T16:12:37.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2224388649' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:12:37.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:36 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:12:37.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/400031436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:37.171 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:12:37.171 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:12:37.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:37.198 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:12:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:12:37.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:37.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:37.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:36 smithi071 ceph-mon[30002]: 2.12 scrub starts 2024-09-17T16:12:37.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:36 smithi071 ceph-mon[30002]: pgmap v403: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 1.16k op/s; 16458/223659 objects degraded (7.359%); 2330/223659 objects misplaced (1.042%) 2024-09-17T16:12:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:36 smithi071 ceph-mon[30002]: osdmap e151: 12 total, 11 up, 12 in 2024-09-17T16:12:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2224388649' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:12:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:36 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:12:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/400031436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:37.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:37.343 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:37.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:37.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:37.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:37.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:37.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:37.608 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:37.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:37.710 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:37.734 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:37.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:37.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:37.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:37.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:37.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:37.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:37.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:37 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:37.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:37 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:12:37.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:37 smithi017 ceph-mon[27571]: osdmap e152: 12 total, 11 up, 12 in 2024-09-17T16:12:37.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:37 smithi017 ceph-mon[27571]: from='osd.7 [v2:172.21.15.71:6824/3827414222,v1:172.21.15.71:6825/3827414222]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:12:37.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:37 smithi017 ceph-mon[27571]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:12:37.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:37.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:37.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:38.063 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:37 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:12:37.760+0000 7f29ffac0640 -1 osd.7 132 log_to_monitors true 2024-09-17T16:12:38.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:12:37 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:12:37.881+0000 7f29f786a640 -1 osd.7 132 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:12:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:37 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:37 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:12:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:37 smithi071 ceph-mon[30002]: osdmap e152: 12 total, 11 up, 12 in 2024-09-17T16:12:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:37 smithi071 ceph-mon[30002]: from='osd.7 [v2:172.21.15.71:6824/3827414222,v1:172.21.15.71:6825/3827414222]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:12:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:37 smithi071 ceph-mon[30002]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:12:38.094 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:38.116 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:38.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:38.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:37 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:38.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:37 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:12:38.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:37 smithi173 ceph-mon[30530]: osdmap e152: 12 total, 11 up, 12 in 2024-09-17T16:12:38.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:37 smithi173 ceph-mon[30530]: from='osd.7 [v2:172.21.15.71:6824/3827414222,v1:172.21.15.71:6825/3827414222]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:12:38.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:37 smithi173 ceph-mon[30530]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-17T16:12:38.218 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:38.242 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:38.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:38.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:12:38.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:12:38.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:38.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:38.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:38.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:38.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:38.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:38.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:38.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:38.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:38.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:38.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:38.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:38.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:38.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:38.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:38 smithi017 ceph-mon[27571]: pgmap v406: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 20 MiB/s wr, 893 op/s; 16057/219054 objects degraded (7.330%); 2330/219054 objects misplaced (1.064%) 2024-09-17T16:12:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:38 smithi017 ceph-mon[27571]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T16:12:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:38 smithi017 ceph-mon[27571]: osdmap e153: 12 total, 11 up, 12 in 2024-09-17T16:12:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:38 smithi017 ceph-mon[27571]: from='osd.7 [v2:172.21.15.71:6824/3827414222,v1:172.21.15.71:6825/3827414222]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:12:38.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:38 smithi017 ceph-mon[27571]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:12:39.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:39.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:39.055 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:12:39.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:12:39.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:39.123 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:38 smithi071 ceph-mon[30002]: pgmap v406: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 20 MiB/s wr, 893 op/s; 16057/219054 objects degraded (7.330%); 2330/219054 objects misplaced (1.064%) 2024-09-17T16:12:39.123 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:38 smithi071 ceph-mon[30002]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T16:12:39.123 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:38 smithi071 ceph-mon[30002]: osdmap e153: 12 total, 11 up, 12 in 2024-09-17T16:12:39.124 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:38 smithi071 ceph-mon[30002]: from='osd.7 [v2:172.21.15.71:6824/3827414222,v1:172.21.15.71:6825/3827414222]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:12:39.124 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:38 smithi071 ceph-mon[30002]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:12:39.144 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:39.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:38 smithi173 ceph-mon[30530]: pgmap v406: 27 pgs: 1 active+clean+remapped, 7 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 20 MiB/s wr, 893 op/s; 16057/219054 objects degraded (7.330%); 2330/219054 objects misplaced (1.064%) 2024-09-17T16:12:39.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:38 smithi173 ceph-mon[30530]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T16:12:39.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:38 smithi173 ceph-mon[30530]: osdmap e153: 12 total, 11 up, 12 in 2024-09-17T16:12:39.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:38 smithi173 ceph-mon[30530]: from='osd.7 [v2:172.21.15.71:6824/3827414222,v1:172.21.15.71:6825/3827414222]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:12:39.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:38 smithi173 ceph-mon[30530]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:12:39.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:39.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:39.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:39.271 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:39.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:39.351 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:12:39.372 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:39.394 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:39.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:39.496 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:39.517 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:39.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:39.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:39.643 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:39.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:39.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:39.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:12:39.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:39.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:39.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:39.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:39.831 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:39.832 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:39.849 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:12:39.849 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:12:39.849 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:12:39.849 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:12:39.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:39.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:39.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:39 smithi017 ceph-mon[27571]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T16:12:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:39 smithi017 ceph-mon[27571]: osd.7 [v2:172.21.15.71:6824/3827414222,v1:172.21.15.71:6825/3827414222] boot 2024-09-17T16:12:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:39 smithi017 ceph-mon[27571]: osdmap e154: 12 total, 12 up, 12 in 2024-09-17T16:12:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:39 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:12:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:39 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 16057/219054 objects degraded (7.330%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:39 smithi017 ceph-mon[27571]: osdmap e155: 12 total, 12 up, 12 in 2024-09-17T16:12:39.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:40.012 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:40.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:40.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:39 smithi071 ceph-mon[30002]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T16:12:40.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:39 smithi071 ceph-mon[30002]: osd.7 [v2:172.21.15.71:6824/3827414222,v1:172.21.15.71:6825/3827414222] boot 2024-09-17T16:12:40.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:39 smithi071 ceph-mon[30002]: osdmap e154: 12 total, 12 up, 12 in 2024-09-17T16:12:40.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:39 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:12:40.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:39 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 16057/219054 objects degraded (7.330%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:40.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:39 smithi071 ceph-mon[30002]: osdmap e155: 12 total, 12 up, 12 in 2024-09-17T16:12:40.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:40.134 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:40.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:40.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:39 smithi173 ceph-mon[30530]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T16:12:40.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:39 smithi173 ceph-mon[30530]: osd.7 [v2:172.21.15.71:6824/3827414222,v1:172.21.15.71:6825/3827414222] boot 2024-09-17T16:12:40.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:39 smithi173 ceph-mon[30530]: osdmap e154: 12 total, 12 up, 12 in 2024-09-17T16:12:40.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:39 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-17T16:12:40.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:39 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 16057/219054 objects degraded (7.330%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:39 smithi173 ceph-mon[30530]: osdmap e155: 12 total, 12 up, 12 in 2024-09-17T16:12:40.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:40.262 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:40.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:40.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:40.369 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:12:40.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:40.386 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:40.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:40.509 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:40.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:40.547 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T16:12:40.610 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:40.636 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:40.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:40.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:40.772 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:40.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:40.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:40.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:40.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:40.915 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:12:40.977 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:40.985 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:41.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:41.029 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:41.062 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:41.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:41.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:41.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:40 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:12:41.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:40 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:12:41.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:40 smithi017 ceph-mon[27571]: pgmap v410: 27 pgs: 1 remapped+peering, 1 active+clean+remapped, 6 active+undersized+degraded, 19 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 262 B/s rd, 5.3 MiB/s wr, 1.05k op/s; 11358/213339 objects degraded (5.324%); 2330/213339 objects misplaced (1.092%) 2024-09-17T16:12:41.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:12:41.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:40 smithi017 ceph-mon[27571]: osdmap e156: 12 total, 12 up, 12 in 2024-09-17T16:12:41.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:40 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:12:41.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:40 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:12:41.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:40 smithi173 ceph-mon[30530]: pgmap v410: 27 pgs: 1 remapped+peering, 1 active+clean+remapped, 6 active+undersized+degraded, 19 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 262 B/s rd, 5.3 MiB/s wr, 1.05k op/s; 11358/213339 objects degraded (5.324%); 2330/213339 objects misplaced (1.092%) 2024-09-17T16:12:41.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:12:41.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:40 smithi173 ceph-mon[30530]: osdmap e156: 12 total, 12 up, 12 in 2024-09-17T16:12:41.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:41.193 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:41.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:41.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:41.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:40 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:12:41.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:40 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:12:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:40 smithi071 ceph-mon[30002]: pgmap v410: 27 pgs: 1 remapped+peering, 1 active+clean+remapped, 6 active+undersized+degraded, 19 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 262 B/s rd, 5.3 MiB/s wr, 1.05k op/s; 11358/213339 objects degraded (5.324%); 2330/213339 objects misplaced (1.092%) 2024-09-17T16:12:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:12:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:40 smithi071 ceph-mon[30002]: osdmap e156: 12 total, 12 up, 12 in 2024-09-17T16:12:41.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:41.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:41.421 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:41.442 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:41.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:41.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:41.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:41.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:41.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:41.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:41.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:41.795 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:41.819 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:41.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:41.912 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:12:41.913 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:12:41.913 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:12:41.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:41.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:41.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:42.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:42.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:42.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:42.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:12:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:12:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:12:42.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:42.211 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:42.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:42.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:42.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:12:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:12:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:12:42.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:42.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:42.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:42.473 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:42.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:42.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:42.589 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:42.609 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:42.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:42.711 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:42.744 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:42.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:42.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:42.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:42.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:42.871 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:42.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:42.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:42.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:43.019 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:12:43.019 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:12:43.054 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:12:43.054 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:12:43.087 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:12:43.087 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:12:43.099 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:43.121 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:12:43.121 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:12:43.134 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:43.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:43.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:12:43.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:12:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:12:43.156 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T16:12:43.164 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:12:43.164 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:87667 2.e e157/95 2:75596fb6:::benchmark_data_smithi173_55121_object11178:head v 157'5450, mlcod=156'5447)", 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:43.160231+0000", 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.0021421109999999999, 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021708280000000001, 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.160231+0000", 2024-09-17T16:12:43.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.160231+0000", 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.160231+0000", 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7000000000000001e-07 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:43.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.160235+0000", 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.89e-06 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.160284+0000", 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8979e-05 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.160292+0000", 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4980000000000006e-06 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:43.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.160773+0000", 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048116800000000003 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.160794+0000", 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0959e-05 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:12:43.168 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 1 2024-09-17T16:12:43.169 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:12:43.200 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:12:43.200 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:12:43.232 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:12:43.232 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:12:43.236 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:43.261 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:12:43.261 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:12:43.263 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:43.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:43.293 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:12:43.293 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:12:43.324 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:12:43.324 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:12:43.359 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:12:43.359 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:12:43.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:43.387 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:12:43.387 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:12:43.393 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:43.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:43.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:43 smithi017 ceph-mon[27571]: pgmap v412: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+undersized+degraded, 19 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 7130/204024 objects degraded (3.495%); 1213/204024 objects misplaced (0.595%); 1.6 MiB/s, 25 objects/s recovering 2024-09-17T16:12:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3418890102' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3396109675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:12:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2176153893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:43.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:43.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:43.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:43.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:43 smithi071 ceph-mon[30002]: pgmap v412: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+undersized+degraded, 19 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 7130/204024 objects degraded (3.495%); 1213/204024 objects misplaced (0.595%); 1.6 MiB/s, 25 objects/s recovering 2024-09-17T16:12:43.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3418890102' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3396109675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:12:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2176153893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:43.626 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:43.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:43 smithi173 ceph-mon[30530]: pgmap v412: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 active+undersized+degraded, 19 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.25k op/s; 7130/204024 objects degraded (3.495%); 1213/204024 objects misplaced (0.595%); 1.6 MiB/s, 25 objects/s recovering 2024-09-17T16:12:43.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3418890102' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:43.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3396109675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:12:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2176153893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:43.653 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:43.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:43.754 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:43.780 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:43.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:43.883 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:43.907 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:43.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:44.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:44.032 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:44.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:44.132 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:44.155 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:44.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:44.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:44.257 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:44.283 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:44.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:44.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:44.409 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:44.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:44.410 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:12:44.434 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:44 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:12:44.434 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3396109675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:12:44.434 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2176153893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:44.434 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:44 smithi017 ceph-mon[27571]: osdmap e157: 12 total, 12 up, 12 in 2024-09-17T16:12:44.510 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:44.538 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:44.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:44.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:44 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:12:44.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3396109675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:12:44.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2176153893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:44.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:44 smithi071 ceph-mon[30002]: osdmap e157: 12 total, 12 up, 12 in 2024-09-17T16:12:44.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:44 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:12:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3396109675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:12:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2176153893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:44 smithi173 ceph-mon[30530]: osdmap e157: 12 total, 12 up, 12 in 2024-09-17T16:12:44.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:44.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:44.777 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:44.803 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:44.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:44.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:44.928 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:44.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:45.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:45.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:45.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:45.158 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:45.183 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:45.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:45.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:45.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:45.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:45.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:45.414 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:45.442 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:45.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:45.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:45 smithi017 ceph-mon[27571]: pgmap v414: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 20 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 7128/204024 objects degraded (3.494%); 1213/204024 objects misplaced (0.595%); 1.2 MiB/s, 20 objects/s recovering 2024-09-17T16:12:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:45 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7128/204024 objects degraded (3.494%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:45 smithi017 ceph-mon[27571]: osdmap e158: 12 total, 12 up, 12 in 2024-09-17T16:12:45.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:45.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:45 smithi071 ceph-mon[30002]: pgmap v414: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 20 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 7128/204024 objects degraded (3.494%); 1213/204024 objects misplaced (0.595%); 1.2 MiB/s, 20 objects/s recovering 2024-09-17T16:12:45.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:45 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7128/204024 objects degraded (3.494%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:45 smithi071 ceph-mon[30002]: osdmap e158: 12 total, 12 up, 12 in 2024-09-17T16:12:45.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:45.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:45.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:45 smithi173 ceph-mon[30530]: pgmap v414: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 20 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 7128/204024 objects degraded (3.494%); 1213/204024 objects misplaced (0.595%); 1.2 MiB/s, 20 objects/s recovering 2024-09-17T16:12:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:45 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7128/204024 objects degraded (3.494%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:45 smithi173 ceph-mon[30530]: osdmap e158: 12 total, 12 up, 12 in 2024-09-17T16:12:45.656 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T16:12:45.672 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:45.727 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:45.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:45.829 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:45.854 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:45.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:45.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:45.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:45.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:46.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:46.105 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:46.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:46.207 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:46.239 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:46.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:46.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:46.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:46.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:46.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:46.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:46.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:46.598 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:46.622 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:46.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:46.716 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:46.723 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:46.752 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:46.753 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:46.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:46.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:46.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:46.881 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:46.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:46.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:47.008 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:47.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:47.110 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:47.135 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:47.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:47.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:47.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:47.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:47.300 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:47 smithi017 ceph-mon[27571]: 2.11 scrub starts 2024-09-17T16:12:47.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:47 smithi017 ceph-mon[27571]: 2.11 scrub ok 2024-09-17T16:12:47.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:47 smithi017 ceph-mon[27571]: pgmap v416: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 9269/196044 objects degraded (4.728%); 782/196044 objects misplaced (0.399%); 11 MiB/s, 176 objects/s recovering 2024-09-17T16:12:47.365 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:47.388 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:47.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:47.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:47 smithi173 ceph-mon[30530]: 2.11 scrub starts 2024-09-17T16:12:47.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:47 smithi173 ceph-mon[30530]: 2.11 scrub ok 2024-09-17T16:12:47.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:47 smithi173 ceph-mon[30530]: pgmap v416: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 9269/196044 objects degraded (4.728%); 782/196044 objects misplaced (0.399%); 11 MiB/s, 176 objects/s recovering 2024-09-17T16:12:47.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:47.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:47.518 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:47 smithi071 ceph-mon[30002]: 2.11 scrub starts 2024-09-17T16:12:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:47 smithi071 ceph-mon[30002]: 2.11 scrub ok 2024-09-17T16:12:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:47 smithi071 ceph-mon[30002]: pgmap v416: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 960 op/s; 9269/196044 objects degraded (4.728%); 782/196044 objects misplaced (0.399%); 11 MiB/s, 176 objects/s recovering 2024-09-17T16:12:47.608 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:12:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:12:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:12:47.609 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:12:47.609 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:12:47.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:47.653 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:47.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:47.755 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:47.781 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:47.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:47.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:47.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:47.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:47.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:48.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:48.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:48.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:48.133 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:48.159 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:48.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:48.197 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:12:48.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:48.285 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:48.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:48.387 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:48.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:48.419 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:48.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:48.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:48.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:48.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:48.572 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:12:48.651 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:48.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:48.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:48.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:48.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:12:48.780 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:48.801 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:48.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:48.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:48.927 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:48.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:49.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:49.050 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:49.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:49.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:49.173 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T16:12:49.176 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:49.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:49.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:49.309 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:49.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:49.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:49.442 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:49.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:49.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:49 smithi017 ceph-mon[27571]: pgmap v417: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 8886/189156 objects degraded (4.698%); 782/189156 objects misplaced (0.413%); 12 MiB/s, 193 objects/s recovering 2024-09-17T16:12:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2308178936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:12:49.544 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:49.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:49 smithi071 ceph-mon[30002]: pgmap v417: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 8886/189156 objects degraded (4.698%); 782/189156 objects misplaced (0.413%); 12 MiB/s, 193 objects/s recovering 2024-09-17T16:12:49.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2308178936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:12:49.570 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:49.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:49.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:49 smithi173 ceph-mon[30530]: pgmap v417: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 8886/189156 objects degraded (4.698%); 782/189156 objects misplaced (0.413%); 12 MiB/s, 193 objects/s recovering 2024-09-17T16:12:49.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2308178936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:12:49.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:49.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:49.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:49.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:49.834 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:49.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:49.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:49.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:49.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:49.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:50.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:50.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:50.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:50.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:50.207 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:50.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:50.235 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2308178936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:12:50.235 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:50 smithi017 ceph-mon[27571]: osdmap e159: 12 total, 12 up, 12 in 2024-09-17T16:12:50.235 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:50 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 8886/189156 objects degraded (4.698%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:50.235 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:50 smithi017 ceph-mon[27571]: osdmap e160: 12 total, 12 up, 12 in 2024-09-17T16:12:50.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:50 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:12:50.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:50 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:12:50.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3152084724' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:50.292 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:12:50.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:50.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:50.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:50.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:50.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:50.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:50.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:50.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:50.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2308178936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:12:50.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:50 smithi071 ceph-mon[30002]: osdmap e159: 12 total, 12 up, 12 in 2024-09-17T16:12:50.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:50 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 8886/189156 objects degraded (4.698%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:50.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:50 smithi071 ceph-mon[30002]: osdmap e160: 12 total, 12 up, 12 in 2024-09-17T16:12:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:50 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:12:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:50 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:12:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3152084724' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:50.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:50.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2308178936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:12:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:50 smithi173 ceph-mon[30530]: osdmap e159: 12 total, 12 up, 12 in 2024-09-17T16:12:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:50 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 8886/189156 objects degraded (4.698%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:50 smithi173 ceph-mon[30530]: osdmap e160: 12 total, 12 up, 12 in 2024-09-17T16:12:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:50 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:12:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:50 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:12:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3152084724' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:12:50.686 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:50.714 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:50.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:50.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:50.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:50.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:50.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:50.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:50.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:51.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:51.099 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:51.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:51.201 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:51.223 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:51.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:51.325 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:51.350 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:51.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:51.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:51.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:51.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:51.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:51 smithi017 ceph-mon[27571]: pgmap v420: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 8886/185172 objects degraded (4.799%); 782/185172 objects misplaced (0.422%); 14 MiB/s, 223 objects/s recovering 2024-09-17T16:12:51.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:51 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3152084724' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:12:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:51 smithi017 ceph-mon[27571]: osdmap e161: 12 total, 12 up, 12 in 2024-09-17T16:12:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:51 smithi071 ceph-mon[30002]: pgmap v420: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 8886/185172 objects degraded (4.799%); 782/185172 objects misplaced (0.422%); 14 MiB/s, 223 objects/s recovering 2024-09-17T16:12:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:51 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3152084724' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:12:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:51 smithi071 ceph-mon[30002]: osdmap e161: 12 total, 12 up, 12 in 2024-09-17T16:12:51.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:51.605 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:51.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:51.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:51.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:51 smithi173 ceph-mon[30530]: pgmap v420: 27 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 8886/185172 objects degraded (4.799%); 782/185172 objects misplaced (0.422%); 14 MiB/s, 223 objects/s recovering 2024-09-17T16:12:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:51 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3152084724' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:12:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:51 smithi173 ceph-mon[30530]: osdmap e161: 12 total, 12 up, 12 in 2024-09-17T16:12:51.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:51.729 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:51.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:51.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T16:12:51.831 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:51.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:51.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:51.963 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:51.984 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:51.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:51.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:52.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:52.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:52.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:52.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:52.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:52.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:52.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:52.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:52.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:52.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:52 smithi017 ceph-mon[27571]: osdmap e162: 12 total, 12 up, 12 in 2024-09-17T16:12:52.475 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:52.478 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:52.484 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:52.500 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:52.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:52.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:52 smithi071 ceph-mon[30002]: osdmap e162: 12 total, 12 up, 12 in 2024-09-17T16:12:52.602 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:52.629 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:52.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:52.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:52 smithi173 ceph-mon[30530]: osdmap e162: 12 total, 12 up, 12 in 2024-09-17T16:12:52.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:52.761 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:52.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:52.777 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:52.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:52.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:52.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:52.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:53.021 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:53.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:53.123 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:53.149 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:53.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:53.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:53.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:53.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:53.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:53.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:53.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:53.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:53.461 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:12:53.461 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:12:53.461 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:12:53.461 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:12:53.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.461 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:11670 2.d e102/95 2:b281e889:::benchmark_data_smithi173_55121_object11669:head v 102'728, mlcod=102'723)", 2024-09-17T16:12:53.461 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:10.813857+0000", 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 102.64409102899999, 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.178192506, 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.813857+0000", 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.813857+0000", 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.813858+0000", 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7399999999999998e-07 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.814029+0000", 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000171645 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.991492+0000", 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.17746303299999999 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.991499+0000", 2024-09-17T16:12:53.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6089999999999997e-06 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.991549+0000", 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0167000000000003e-05 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.991565+0000", 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6283999999999999e-05 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.992026+0000", 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046056800000000001 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:10.992050+0000", 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3726e-05 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.467 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:13400 2.a e103/95 2:5cc2a738:::benchmark_data_smithi173_55121_object13399:head v 103'822, mlcod=103'819)", 2024-09-17T16:12:53.467 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.132258+0000", 2024-09-17T16:12:53.467 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 101.325690489, 2024-09-17T16:12:53.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.18362804399999999, 2024-09-17T16:12:53.467 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.467 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.467 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.467 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.132258+0000", 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.132258+0000", 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.132258+0000", 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1500000000000004e-07 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.132323+0000", 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4548999999999995e-05 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.315216+0000", 2024-09-17T16:12:53.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.18289282900000001 2024-09-17T16:12:53.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.315224+0000", 2024-09-17T16:12:53.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6300000000000004e-06 2024-09-17T16:12:53.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.315281+0000", 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.66e-05 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.315302+0000", 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0760000000000001e-05 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.315844+0000", 2024-09-17T16:12:53.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054228499999999999 2024-09-17T16:12:53.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.472 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.472 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.315886+0000", 2024-09-17T16:12:53.472 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1675999999999999e-05 2024-09-17T16:12:53.472 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.472 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.472 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:13526 2.d e103/95 2:b276e7f5:::benchmark_data_smithi173_55121_object13525:head v 103'845, mlcod=103'842)", 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.327883+0000", 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 101.13006494299999, 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41382291199999999, 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.327883+0000", 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.327883+0000", 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.327884+0000", 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5899999999999996e-07 2024-09-17T16:12:53.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.328029+0000", 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014476699999999999 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.740919+0000", 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41289076499999999 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.740932+0000", 2024-09-17T16:12:53.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2248e-05 2024-09-17T16:12:53.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741023+0000", 2024-09-17T16:12:53.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1221e-05 2024-09-17T16:12:53.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741050+0000", 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7132e-05 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741673+0000", 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062324600000000004 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741706+0000", 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2974e-05 2024-09-17T16:12:53.477 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:13531 2.a e103/95 2:59aa3d24:::benchmark_data_smithi173_55121_object13530:head v 103'829, mlcod=103'826)", 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.331447+0000", 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 101.12650078, 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.507364445, 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.478 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331447+0000", 2024-09-17T16:12:53.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331447+0000", 2024-09-17T16:12:53.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331448+0000", 2024-09-17T16:12:53.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8899999999999999e-07 2024-09-17T16:12:53.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331493+0000", 2024-09-17T16:12:53.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4728999999999998e-05 2024-09-17T16:12:53.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838043+0000", 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.50655012099999996 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838053+0000", 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7850000000000003e-06 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838135+0000", 2024-09-17T16:12:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2113000000000005e-05 2024-09-17T16:12:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838163+0000", 2024-09-17T16:12:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8181999999999999e-05 2024-09-17T16:12:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838781+0000", 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061825400000000005 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838812+0000", 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0571999999999998e-05 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:14940 2.a e106/95 2:57eacb6e:::benchmark_data_smithi173_55121_object14939:head v 106'904, mlcod=105'901)", 2024-09-17T16:12:53.483 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.352719+0000", 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 99.105229648000005, 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25501401899999998, 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.352719+0000", 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.352719+0000", 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.352719+0000", 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0900000000000001e-07 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.352797+0000", 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8271999999999999e-05 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606767+0000", 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25397041100000001 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606776+0000", 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5609999999999999e-06 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606852+0000", 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6020999999999996e-05 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606884+0000", 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2354000000000002e-05 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.607694+0000", 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080928700000000001 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.607733+0000", 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8903999999999999e-05 2024-09-17T16:12:53.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:14949 2.a e106/95 2:58bfb129:::benchmark_data_smithi173_55121_object14948:head v 106'905, mlcod=105'902)", 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.606700+0000", 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 98.851247916000005, 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25693972399999998, 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.489 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606700+0000", 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606700+0000", 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606702+0000", 2024-09-17T16:12:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8810000000000001e-06 2024-09-17T16:12:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606738+0000", 2024-09-17T16:12:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5660000000000001e-05 2024-09-17T16:12:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.862886+0000", 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25614775400000001 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.862891+0000", 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.057e-06 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.862948+0000", 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7484000000000001e-05 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.862968+0000", 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9629000000000001e-05 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.863600+0000", 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000632629 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.863640+0000", 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9629999999999998e-05 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:20838 2.e e116/95 2:7e96b7fd:::benchmark_data_smithi173_55121_object20837:head v 116'1252, mlcod=116'1249)", 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:25.455525+0000", 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 88.002423344999997, 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31922061699999998, 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.494 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.495 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.495 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455525+0000", 2024-09-17T16:12:53.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455525+0000", 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455526+0000", 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.33e-07 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455549+0000", 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3686999999999999e-05 2024-09-17T16:12:53.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773668+0000", 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31811887700000002 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773674+0000", 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.037e-06 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773734+0000", 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9619e-05 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773758+0000", 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4097999999999999e-05 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.774701+0000", 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00094260999999999998 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.774745+0000", 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4756000000000003e-05 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:21746 2.a e117/95 2:547291d7:::benchmark_data_smithi173_55121_object21745:head v 117'1352, mlcod=117'1349)", 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:26.669244+0000", 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 86.788703963000003, 2024-09-17T16:12:53.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.19967753599999999, 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.669244+0000", 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.669244+0000", 2024-09-17T16:12:53.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.669245+0000", 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2800000000000005e-07 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.669301+0000", 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5535000000000003e-05 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.867974+0000", 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.198673552 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.867986+0000", 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1430000000000001e-05 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.868060+0000", 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4314999999999996e-05 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.868080+0000", 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9647e-05 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.868886+0000", 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080653099999999998 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.868922+0000", 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5598e-05 2024-09-17T16:12:53.504 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:24657 2.19 2:9911b308:::benchmark_data_smithi173_55121_object24656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.345526+0000", 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 83.112422476000006, 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22486350799999999, 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.505 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 24657 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.345526+0000", 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.345526+0000", 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.345527+0000", 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.097e-06 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.345625+0000", 2024-09-17T16:12:53.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8634000000000005e-05 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.345636+0000", 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0930999999999999e-05 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.345643+0000", 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8850000000000001e-06 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.345695+0000", 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1573999999999998e-05 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.345813+0000", 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011838299999999999 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.345957+0000", 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014369200000000001 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.346640+0000", 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00068294199999999997 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.346757+0000", 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011715 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.570331+0000", 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22357359399999999 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.570363+0000", 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2644000000000001e-05 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.570389+0000", 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5982e-05 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.511 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25217 2.19 2:9a6d6ce1:::benchmark_data_smithi173_55121_object25216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.776796+0000", 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 82.681151989, 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47785892200000002, 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25217 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776796+0000", 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776796+0000", 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776798+0000", 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8899999999999999e-06 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776843+0000", 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5306999999999998e-05 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776856+0000", 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2858e-05 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776862+0000", 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3120000000000001e-06 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776906+0000", 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4311000000000003e-05 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777001+0000", 2024-09-17T16:12:53.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4927999999999996e-05 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777093+0000", 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1878000000000004e-05 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777671+0000", 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00057851599999999995 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.036749+0000", 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.259078209 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254584+0000", 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21783461500000001 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254624+0000", 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9915e-05 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254655+0000", 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1183000000000003e-05 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:25217 2.19 e121/114 ondisk, result = 0)", 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.781044+0000", 2024-09-17T16:12:53.518 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 82.676904282999999, 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47361829999999999, 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781044+0000", 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781044+0000", 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781045+0000", 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4099999999999998e-07 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781049+0000", 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4859999999999999e-06 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254521+0000", 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47347151599999998 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254527+0000", 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5239999999999997e-06 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254576+0000", 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8445000000000001e-05 2024-09-17T16:12:53.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254581+0000", 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254662+0000", 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1454999999999999e-05 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25240 2.9 e121/104 2:9323c63b:::benchmark_data_smithi173_55121_object25239:head v 121'1240, mlcod=121'1236)", 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.804013+0000", 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 82.653935441000002, 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25673434699999997, 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804013+0000", 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804013+0000", 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804014+0000", 2024-09-17T16:12:53.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4499999999999996e-07 2024-09-17T16:12:53.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804036+0000", 2024-09-17T16:12:53.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2107e-05 2024-09-17T16:12:53.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.059951+0000", 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25591505199999998 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.059958+0000", 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8580000000000002e-06 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060030+0000", 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1942999999999999e-05 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060054+0000", 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4596e-05 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060719+0000", 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00066500100000000003 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060747+0000", 2024-09-17T16:12:53.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7945000000000001e-05 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25263 2.9 e121/104 2:916f8b10:::benchmark_data_smithi173_55121_object25262:head v 121'1241, mlcod=121'1237)", 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:31.059898+0000", 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 82.398050556000001, 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.181107147, 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.528 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.059898+0000", 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.059898+0000", 2024-09-17T16:12:53.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.059899+0000", 2024-09-17T16:12:53.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2399999999999996e-07 2024-09-17T16:12:53.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.059929+0000", 2024-09-17T16:12:53.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0521000000000001e-05 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.240203+0000", 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.18027383899999999 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.240243+0000", 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0466999999999997e-05 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.240301+0000", 2024-09-17T16:12:53.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7633000000000002e-05 2024-09-17T16:12:53.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.240328+0000", 2024-09-17T16:12:53.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6548999999999999e-05 2024-09-17T16:12:53.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.240955+0000", 2024-09-17T16:12:53.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062731500000000003 2024-09-17T16:12:53.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.241005+0000", 2024-09-17T16:12:53.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9898999999999998e-05 2024-09-17T16:12:53.533 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.533 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.533 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:26910 2.1 e121/104 2:838cca45:::benchmark_data_smithi173_55121_object26909:head v 121'1292, mlcod=121'1289)", 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.327897+0000", 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 81.130050792000006, 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23580620599999999, 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327897+0000", 2024-09-17T16:12:53.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327897+0000", 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327898+0000", 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4900000000000005e-07 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327921+0000", 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3317000000000001e-05 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.562770+0000", 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23484860900000001 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.562785+0000", 2024-09-17T16:12:53.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.484e-05 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.562965+0000", 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018036500000000001 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563003+0000", 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7696e-05 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563671+0000", 2024-09-17T16:12:53.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00066814399999999996 2024-09-17T16:12:53.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563704+0000", 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2486e-05 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:26911 2.e e121/95 2:7501b6f7:::benchmark_data_smithi173_55121_object26910:head v 121'1608, mlcod=121'1605)", 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.329368+0000", 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 81.128580131000007, 2024-09-17T16:12:53.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.260516003, 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329368+0000", 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329368+0000", 2024-09-17T16:12:53.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329369+0000", 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5400000000000001e-07 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329387+0000", 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8306000000000001e-05 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.588701+0000", 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.259314139 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.588719+0000", 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8053000000000001e-05 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.588863+0000", 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014388899999999999 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.588903+0000", 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0469000000000002e-05 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.589849+0000", 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00094574899999999996 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.589884+0000", 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4844e-05 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:47778 2.13 e142/112 2:cf64c7ca:::benchmark_data_smithi173_55121_object47777:head v 142'1859, mlcod=142'1857)", 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:10.556687+0000", 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.901261329999997, 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22771733999999999, 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.545 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.556687+0000", 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.556687+0000", 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.556688+0000", 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8000000000000005e-07 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.556761+0000", 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3238000000000006e-05 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.783341+0000", 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22657978500000001 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.783353+0000", 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2427e-05 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.783472+0000", 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011927200000000001 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.783506+0000", 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3176000000000002e-05 2024-09-17T16:12:53.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.784367+0000", 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086174799999999996 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.784404+0000", 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6913999999999999e-05 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:88568 2.19 2:9815a9f0:::benchmark_data_smithi173_55121_object12079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:43.869911+0000", 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.5880368100000002, 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24823031700000001, 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 88568 2024-09-17T16:12:53.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869911+0000", 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869911+0000", 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869913+0000", 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.57e-06 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869918+0000", 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7400000000000004e-06 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869927+0000", 2024-09-17T16:12:53.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7710000000000001e-06 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869933+0000", 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3129999999999998e-06 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869973+0000", 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0599999999999998e-05 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.870044+0000", 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0654999999999994e-05 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.870105+0000", 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0659999999999999e-05 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:12:53.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.871249+0000", 2024-09-17T16:12:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001144064 2024-09-17T16:12:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.872222+0000", 2024-09-17T16:12:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097360800000000005 2024-09-17T16:12:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.118065+0000", 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24584251600000001 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.118106+0000", 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1556e-05 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.118142+0000", 2024-09-17T16:12:53.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5264e-05 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:88853 2.19 2:99a057e6:::benchmark_data_smithi173_55121_object12364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:44.330874+0000", 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.1270743339999996, 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.174012899, 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.557 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:12:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:12:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 88853 2024-09-17T16:12:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.330874+0000", 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.330874+0000", 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.330875+0000", 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.330879+0000", 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7579999999999998e-06 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.330888+0000", 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5990000000000008e-06 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.330893+0000", 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5090000000000004e-06 2024-09-17T16:12:53.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.330930+0000", 2024-09-17T16:12:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7462999999999997e-05 2024-09-17T16:12:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.330995+0000", 2024-09-17T16:12:53.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4727999999999994e-05 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.331052+0000", 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7077999999999998e-05 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.331976+0000", 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092449199999999996 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.504572+0000", 2024-09-17T16:12:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.17259555800000001 2024-09-17T16:12:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.504811+0000", 2024-09-17T16:12:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023895899999999999 2024-09-17T16:12:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.504846+0000", 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5299999999999997e-05 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.504887+0000", 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0565000000000002e-05 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:94007 2.19 2:9e42b13d:::benchmark_data_smithi173_55121_object17518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.5702558720000002, 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.467128445, 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:53.565 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 94007 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887693+0000", 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887700+0000", 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887711+0000", 2024-09-17T16:12:53.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0739e-05 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887716+0000", 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2000000000000002e-06 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887759+0000", 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2898999999999997e-05 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887839+0000", 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9500999999999997e-05 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887937+0000", 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8531999999999998e-05 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.889187+0000", 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012492580000000001 2024-09-17T16:12:53.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.889505+0000", 2024-09-17T16:12:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031885699999999997 2024-09-17T16:12:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354736+0000", 2024-09-17T16:12:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46523019100000002 2024-09-17T16:12:53.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354784+0000", 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8776999999999997e-05 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354821+0000", 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6321e-05 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:95149 2.19 2:9c74ff86:::benchmark_data_smithi173_55121_object18660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-17T16:12:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:51.251455+0000", 2024-09-17T16:12:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 2.2064934950000001, 2024-09-17T16:12:53.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.186296763, 2024-09-17T16:12:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:12:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:12:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:12:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:12:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:12:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:12:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 95149 2024-09-17T16:12:53.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:12:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:12:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.251455+0000", 2024-09-17T16:12:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:12:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.251455+0000", 2024-09-17T16:12:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:12:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:12:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.251456+0000", 2024-09-17T16:12:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.282e-06 2024-09-17T16:12:53.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:12:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.251460+0000", 2024-09-17T16:12:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.51e-06 2024-09-17T16:12:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:12:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.251473+0000", 2024-09-17T16:12:53.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2069000000000001e-05 2024-09-17T16:12:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:12:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.251478+0000", 2024-09-17T16:12:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6699999999999999e-06 2024-09-17T16:12:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:12:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.251523+0000", 2024-09-17T16:12:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4471000000000002e-05 2024-09-17T16:12:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:12:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.251604+0000", 2024-09-17T16:12:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1728000000000001e-05 2024-09-17T16:12:53.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:12:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.251674+0000", 2024-09-17T16:12:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9793000000000001e-05 2024-09-17T16:12:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:12:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.252932+0000", 2024-09-17T16:12:53.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012578019999999999 2024-09-17T16:12:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.253241+0000", 2024-09-17T16:12:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00030903899999999999 2024-09-17T16:12:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:12:53.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.437689+0000", 2024-09-17T16:12:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.184447519 2024-09-17T16:12:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:12:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.437720+0000", 2024-09-17T16:12:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1659000000000003e-05 2024-09-17T16:12:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:12:53.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:12:53.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:12:53.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.437751+0000", 2024-09-17T16:12:53.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1220999999999998e-05 2024-09-17T16:12:53.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.582 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:12:53.582 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:12:53.583 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:12:53.583 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:53.584 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:53 smithi071 ceph-mon[30002]: pgmap v423: 27 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 7302/175842 objects degraded (4.153%); 454/175842 objects misplaced (0.258%); 26 MiB/s, 462 objects/s recovering 2024-09-17T16:12:53.584 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:53 smithi071 ceph-mon[30002]: osdmap e163: 12 total, 12 up, 12 in 2024-09-17T16:12:53.584 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:53 smithi071 ceph-mon[30002]: 2.3 scrub starts 2024-09-17T16:12:53.584 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1286165760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:53.608 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:53.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:53.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:53 smithi017 ceph-mon[27571]: pgmap v423: 27 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 7302/175842 objects degraded (4.153%); 454/175842 objects misplaced (0.258%); 26 MiB/s, 462 objects/s recovering 2024-09-17T16:12:53.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:53 smithi017 ceph-mon[27571]: osdmap e163: 12 total, 12 up, 12 in 2024-09-17T16:12:53.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:53 smithi017 ceph-mon[27571]: 2.3 scrub starts 2024-09-17T16:12:53.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1286165760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:53.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:53 smithi173 ceph-mon[30530]: pgmap v423: 27 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 7302/175842 objects degraded (4.153%); 454/175842 objects misplaced (0.258%); 26 MiB/s, 462 objects/s recovering 2024-09-17T16:12:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:53 smithi173 ceph-mon[30530]: osdmap e163: 12 total, 12 up, 12 in 2024-09-17T16:12:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:53 smithi173 ceph-mon[30530]: 2.3 scrub starts 2024-09-17T16:12:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1286165760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:12:53.711 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:53.737 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:53.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:53.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:53.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:53.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:53.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:53.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:53.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:54.085 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:12:54.085 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:12:54.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:54.109 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:12:54.110 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:12:54.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:54.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:54.136 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:12:54.137 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:12:54.167 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:12:54.168 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:12:54.196 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:12:54.196 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:12:54.213 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:54.232 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:12:54.232 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:12:54.244 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:54.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:54.268 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:12:54.269 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:12:54.301 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:12:54.301 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:12:54.335 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:12:54.335 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:12:54.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:54 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7302/175842 objects degraded (4.153%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:54.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:54 smithi017 ceph-mon[27571]: osdmap e164: 12 total, 12 up, 12 in 2024-09-17T16:12:54.346 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:54.368 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:12:54.368 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:12:54.372 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:54.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:54.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:54.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:12:54.401 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:12:54.430 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:12:54.430 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:12:54.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:12:54.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:54.507 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:54.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:54.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:54 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7302/175842 objects degraded (4.153%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:54 smithi071 ceph-mon[30002]: osdmap e164: 12 total, 12 up, 12 in 2024-09-17T16:12:54.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:54.633 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:54.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:54.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:54.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:54 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7302/175842 objects degraded (4.153%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:54.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:54 smithi173 ceph-mon[30530]: osdmap e164: 12 total, 12 up, 12 in 2024-09-17T16:12:54.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:54.760 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:54.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:54.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:54.887 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:54.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:54.989 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:54.999 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:12:55.015 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:55.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:55.117 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:55.147 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:55.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:55.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:55.339 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:12:55.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:55.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:55.466 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:55.497 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:55.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:55 smithi071 ceph-mon[30002]: pgmap v425: 27 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 994 op/s; 7302/175842 objects degraded (4.153%); 454/175842 objects misplaced (0.258%); 23 MiB/s, 414 objects/s recovering 2024-09-17T16:12:55.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:55 smithi017 ceph-mon[27571]: pgmap v425: 27 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 994 op/s; 7302/175842 objects degraded (4.153%); 454/175842 objects misplaced (0.258%); 23 MiB/s, 414 objects/s recovering 2024-09-17T16:12:55.599 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:55.626 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:55.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:55.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:55.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:55 smithi173 ceph-mon[30530]: pgmap v425: 27 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 994 op/s; 7302/175842 objects degraded (4.153%); 454/175842 objects misplaced (0.258%); 23 MiB/s, 414 objects/s recovering 2024-09-17T16:12:55.728 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:55.750 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:55.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:55.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:55.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:55.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:55.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:55.879 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:55.879 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:55.930 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:55.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:55.999 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:56.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:56.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:56.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:56.123 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:56.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:56.225 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:56.248 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:56.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:56.322 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T16:12:56.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:56.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:56.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:56.484 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:56.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:56 smithi017 ceph-mon[27571]: osdmap e165: 12 total, 12 up, 12 in 2024-09-17T16:12:56.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/796566238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:12:56.512 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:56.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:56.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:56.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:56.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:56.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:56 smithi173 ceph-mon[30530]: osdmap e165: 12 total, 12 up, 12 in 2024-09-17T16:12:56.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/796566238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:12:56.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:56.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:56.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:56 smithi071 ceph-mon[30002]: osdmap e165: 12 total, 12 up, 12 in 2024-09-17T16:12:56.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/796566238' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:12:56.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:56.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:56.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:56.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:56.987 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:57.013 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:57.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:57.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:57.136 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:57.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:57.237 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:12:57.260 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:57.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:12:57.362 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:57.397 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:57.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:57.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:57.499 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:57.523 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:57.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:57.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:57.637 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:57 smithi017 ceph-mon[27571]: pgmap v428: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 21 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 7302/167652 objects degraded (4.355%); 15 MiB/s, 313 objects/s recovering 2024-09-17T16:12:57.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/796566238' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:12:57.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:57 smithi017 ceph-mon[27571]: osdmap e166: 12 total, 12 up, 12 in 2024-09-17T16:12:57.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:57 smithi173 ceph-mon[30530]: pgmap v428: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 21 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 7302/167652 objects degraded (4.355%); 15 MiB/s, 313 objects/s recovering 2024-09-17T16:12:57.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/796566238' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:12:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:57 smithi173 ceph-mon[30530]: osdmap e166: 12 total, 12 up, 12 in 2024-09-17T16:12:57.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:57.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:57.677 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-17T16:12:57.756 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:57.786 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:57.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:57.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:57 smithi071 ceph-mon[30002]: pgmap v428: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 21 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s; 7302/167652 objects degraded (4.355%); 15 MiB/s, 313 objects/s recovering 2024-09-17T16:12:57.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/796566238' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:12:57.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:57 smithi071 ceph-mon[30002]: osdmap e166: 12 total, 12 up, 12 in 2024-09-17T16:12:57.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:12:57.910 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:57.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:12:57.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:12:58.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:58.034 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:58.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:58.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:58.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:58.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:12:58.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:58.189 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:58.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:58.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:58.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:58.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:58.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:58.341 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:12:58.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:58.449 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:12:58.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:58.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:58.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:12:58.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:58 smithi017 ceph-mon[27571]: 2.12 scrub starts 2024-09-17T16:12:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:12:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:12:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:58 smithi017 ceph-mon[27571]: osdmap e167: 12 total, 12 up, 12 in 2024-09-17T16:12:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:58 smithi017 ceph-mon[27571]: 2.9 scrub starts 2024-09-17T16:12:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1219740924' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:58.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:12:58.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:58 smithi071 ceph-mon[30002]: 2.12 scrub starts 2024-09-17T16:12:58.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:12:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:12:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:58 smithi071 ceph-mon[30002]: osdmap e167: 12 total, 12 up, 12 in 2024-09-17T16:12:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:58 smithi071 ceph-mon[30002]: 2.9 scrub starts 2024-09-17T16:12:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1219740924' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:58.578 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:58.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:12:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:58 smithi173 ceph-mon[30530]: 2.12 scrub starts 2024-09-17T16:12:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:12:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:12:58.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:58 smithi173 ceph-mon[30530]: osdmap e167: 12 total, 12 up, 12 in 2024-09-17T16:12:58.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:58 smithi173 ceph-mon[30530]: 2.9 scrub starts 2024-09-17T16:12:58.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1219740924' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:12:58.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:12:58.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:58.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:12:58.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:58.832 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:58.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:58.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:12:58.961 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:58.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:12:59.023 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:12:59.023 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:12:59.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:12:59.088 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:59.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:12:59.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:12:59.217 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:59.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:12:59.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:12:59.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:59.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:12:59.384 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:12:59.450 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:59.473 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:59.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:59.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:12:59.597 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:12:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:59.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:12:59.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:59 smithi173 ceph-mon[30530]: pgmap v431: 27 pgs: 2 peering, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 21 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 5675/161739 objects degraded (3.509%); 24 MiB/s, 540 objects/s recovering 2024-09-17T16:12:59.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:59 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1219740924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:59 smithi173 ceph-mon[30530]: osdmap e168: 12 total, 12 up, 12 in 2024-09-17T16:12:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:12:59 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 5675/161739 objects degraded (3.509%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:59.683 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:59 smithi017 ceph-mon[27571]: pgmap v431: 27 pgs: 2 peering, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 21 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 5675/161739 objects degraded (3.509%); 24 MiB/s, 540 objects/s recovering 2024-09-17T16:12:59.683 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:59 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:59.683 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1219740924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:59.683 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:59 smithi017 ceph-mon[27571]: osdmap e168: 12 total, 12 up, 12 in 2024-09-17T16:12:59.683 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:12:59 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 5675/161739 objects degraded (3.509%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:59.683 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:59 smithi071 ceph-mon[30002]: pgmap v431: 27 pgs: 2 peering, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 21 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.18k op/s; 5675/161739 objects degraded (3.509%); 24 MiB/s, 540 objects/s recovering 2024-09-17T16:12:59.683 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:59 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:12:59.683 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1219740924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:12:59.684 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:59 smithi071 ceph-mon[30002]: osdmap e168: 12 total, 12 up, 12 in 2024-09-17T16:12:59.684 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:12:59 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 5675/161739 objects degraded (3.509%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:12:59.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:12:59.723 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:12:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:59.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:12:59.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:12:59.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:59.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:12:59.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:12:59.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:12:59.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:12:59.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:00.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:00.105 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:00.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:00.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:00.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:00.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:00.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:00.337 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:00.339 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T16:13:00.361 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:00.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:00.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:13:00.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:00.507 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:00.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:00.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:00.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:00.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:00.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4062928167' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:13:00.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2664624964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:00.738 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:13:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4062928167' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:13:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2664624964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:00.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:00.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:00.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:00.781 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:13:00.781 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:13:00.808 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:13:00.808 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:13:00.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4062928167' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:13:00.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2664624964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:00.829 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:13:00.829 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:13:00.850 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:13:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:13:00.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:00.873 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:13:00.873 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:13:00.890 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:00.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:00.895 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:13:00.895 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:13:00.919 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:13:00.919 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:13:00.942 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:13:00.942 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:13:00.964 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:13:00.965 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:13:00.989 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:13:00.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:13:00.993 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:01.017 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:01.018 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:13:01.018 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:13:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:01.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:01.046 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:13:01.046 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:13:01.088 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:13:01.120 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:01.147 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:01.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:01.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:01.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:01.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:01.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:01.300 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:13:01.300 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:13:01.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:13:01.300 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:13:01.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.300 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:11670 2.d e102/95 2:b281e889:::benchmark_data_smithi173_55121_object11669:head v 102'728, mlcod=102'723)", 2024-09-17T16:13:01.300 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:10.813980+0000", 2024-09-17T16:13:01.300 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 110.482565709, 2024-09-17T16:13:01.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17845166100000001, 2024-09-17T16:13:01.300 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.813980+0000", 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.813980+0000", 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.813981+0000", 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8100000000000003e-07 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.814040+0000", 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9367000000000003e-05 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.991682+0000", 2024-09-17T16:13:01.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17764131599999999 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.991687+0000", 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9470000000000002e-06 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.991768+0000", 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1669999999999996e-05 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.991791+0000", 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2493999999999998e-05 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.992398+0000", 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00060709800000000003 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.992432+0000", 2024-09-17T16:13:01.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4288000000000001e-05 2024-09-17T16:13:01.305 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.305 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.305 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.305 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:11696 2.5 e102/95 2:adf77b71:::benchmark_data_smithi173_55121_object11695:head v 102'708, mlcod=102'706)", 2024-09-17T16:13:01.305 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:10.834186+0000", 2024-09-17T16:13:01.305 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 110.46236016899999, 2024-09-17T16:13:01.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16647804899999999, 2024-09-17T16:13:01.305 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.834186+0000", 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.834186+0000", 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.834187+0000", 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5600000000000004e-07 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:10.834218+0000", 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0858999999999997e-05 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000102+0000", 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16588424700000001 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000108+0000", 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8869999999999997e-06 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000149+0000", 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1377e-05 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000163+0000", 2024-09-17T16:13:01.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3624e-05 2024-09-17T16:13:01.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000631+0000", 2024-09-17T16:13:01.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00046845300000000002 2024-09-17T16:13:01.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:11.000664+0000", 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2645999999999999e-05 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:23607 2.f e119/95 2:fcca4bae:::benchmark_data_smithi173_55121_object23606:head v 119'1443, mlcod=119'1439)", 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.334241+0000", 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 92.962305580999995, 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.110738323, 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.313 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.334241+0000", 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.334241+0000", 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.334241+0000", 2024-09-17T16:13:01.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0500000000000002e-07 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.334302+0000", 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0234e-05 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444185+0000", 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.109883548 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444191+0000", 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8499999999999999e-06 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444233+0000", 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1516e-05 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444246+0000", 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.378e-05 2024-09-17T16:13:01.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444940+0000", 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00069336899999999999 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.444979+0000", 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9221000000000003e-05 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.318 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:23628 2.18 e119/110 2:1945c29a:::benchmark_data_smithi173_55121_object23627:head v 119'1177, mlcod=119'1175)", 2024-09-17T16:13:01.318 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.356382+0000", 2024-09-17T16:13:01.318 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 92.940164499000005, 2024-09-17T16:13:01.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.120117801, 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.356382+0000", 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.356382+0000", 2024-09-17T16:13:01.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.356382+0000", 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1299999999999999e-07 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.356481+0000", 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8696000000000005e-05 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.475914+0000", 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.11943327300000001 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.475919+0000", 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2769999999999999e-06 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.475967+0000", 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8578999999999997e-05 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.475985+0000", 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8221000000000001e-05 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.476461+0000", 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000475723 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.476500+0000", 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8318999999999997e-05 2024-09-17T16:13:01.322 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:24683 2.11 e121/120 2:895e3609:::benchmark_data_smithi173_55121_object24682:head v 121'1219, mlcod=121'1217)", 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.353904+0000", 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 90.942642348999996, 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12774153799999999, 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.353904+0000", 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.353904+0000", 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.353904+0000", 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1699999999999998e-07 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.353926+0000", 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1628999999999999e-05 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481041+0000", 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12711526000000001 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481047+0000", 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0689999999999998e-06 2024-09-17T16:13:01.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481128+0000", 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0317999999999999e-05 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481145+0000", 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6840000000000001e-05 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481615+0000", 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00047066399999999998 2024-09-17T16:13:01.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.481645+0000", 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0141000000000002e-05 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:25244 2.5 e121/104 2:a10ca3e5:::benchmark_data_smithi173_55121_object25243:head v 121'1224, mlcod=121'1221)", 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.807427+0000", 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 90.489119313000003, 2024-09-17T16:13:01.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32591590799999998, 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.807427+0000", 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.807427+0000", 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.807428+0000", 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4900000000000005e-07 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.807473+0000", 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5410000000000001e-05 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.132702+0000", 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32522888799999999 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.132706+0000", 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2490000000000003e-06 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.132779+0000", 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2461999999999998e-05 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.132793+0000", 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4729000000000001e-05 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.133317+0000", 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00052318600000000003 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.133343+0000", 2024-09-17T16:13:01.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6135e-05 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:25251 2.d e121/95 2:bcbcc18b:::benchmark_data_smithi173_55121_object25250:head v 121'1581, mlcod=121'1579)", 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:30.916220+0000", 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 90.380325937999999, 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.134344825, 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916220+0000", 2024-09-17T16:13:01.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916220+0000", 2024-09-17T16:13:01.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916222+0000", 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.263e-06 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.916244+0000", 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1985999999999999e-05 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.049691+0000", 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13344773700000001 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.049696+0000", 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6179999999999997e-06 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.049739+0000", 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3294999999999998e-05 2024-09-17T16:13:01.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.049760+0000", 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0976999999999999e-05 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.050539+0000", 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00077899899999999995 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.050565+0000", 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5950000000000001e-05 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:25264 2.d e121/95 2:b6131135:::benchmark_data_smithi173_55121_object25263:head v 121'1582, mlcod=121'1580)", 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:31.061757+0000", 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 90.234789211999995, 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16186235199999999, 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.337 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.061757+0000", 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.061757+0000", 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.061758+0000", 2024-09-17T16:13:01.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6400000000000001e-07 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.061780+0000", 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2152000000000001e-05 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.222683+0000", 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16090269700000001 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.222687+0000", 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0389999999999998e-06 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.222756+0000", 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8868999999999994e-05 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.222772+0000", 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6725e-05 2024-09-17T16:13:01.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.223587+0000", 2024-09-17T16:13:01.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00081504699999999995 2024-09-17T16:13:01.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:31.223619+0000", 2024-09-17T16:13:01.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2058999999999999e-05 2024-09-17T16:13:01.341 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:29472 2.f e128/95 2:fa36e0b8:::benchmark_data_smithi173_55121_object29471:head v 128'1826, mlcod=128'1823)", 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:49.212863+0000", 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 72.083683424, 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12538254400000001, 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.342 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.212863+0000", 2024-09-17T16:13:01.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.212863+0000", 2024-09-17T16:13:01.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.212866+0000", 2024-09-17T16:13:01.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.484e-06 2024-09-17T16:13:01.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.212929+0000", 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3202999999999998e-05 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.337240+0000", 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.124310979 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.337249+0000", 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4349999999999997e-06 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.337333+0000", 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3634999999999995e-05 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.337348+0000", 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5724000000000001e-05 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.338212+0000", 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00086353199999999997 2024-09-17T16:13:01.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.338245+0000", 2024-09-17T16:13:01.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3552e-05 2024-09-17T16:13:01.347 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.347 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.347 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.348 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:29475 2.5 e128/104 2:a3d66497:::benchmark_data_smithi173_55121_object29474:head v 128'1359, mlcod=128'1356)", 2024-09-17T16:13:01.348 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:49.216963+0000", 2024-09-17T16:13:01.348 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 72.079582915000003, 2024-09-17T16:13:01.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.121189192, 2024-09-17T16:13:01.348 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.348 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.348 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.348 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.216963+0000", 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.216963+0000", 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.216964+0000", 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3900000000000004e-07 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.217079+0000", 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000114888 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.337299+0000", 2024-09-17T16:13:01.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.120220649 2024-09-17T16:13:01.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.337304+0000", 2024-09-17T16:13:01.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8230000000000002e-06 2024-09-17T16:13:01.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.337357+0000", 2024-09-17T16:13:01.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2645999999999997e-05 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.337395+0000", 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8438e-05 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.338109+0000", 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00071325800000000003 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.338152+0000", 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3850999999999997e-05 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:29536 2.5 e129/104 2:a1fe3753:::benchmark_data_smithi173_55121_object29535:head v 129'1360, mlcod=128'1358)", 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:49.388622+0000", 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 71.907924344999998, 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.125382613, 2024-09-17T16:13:01.353 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.354 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.354 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.354 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.388622+0000", 2024-09-17T16:13:01.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.388622+0000", 2024-09-17T16:13:01.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.388623+0000", 2024-09-17T16:13:01.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7000000000000004e-07 2024-09-17T16:13:01.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.388716+0000", 2024-09-17T16:13:01.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3350000000000006e-05 2024-09-17T16:13:01.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.513330+0000", 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12461409800000001 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.513335+0000", 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5909999999999998e-06 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.513390+0000", 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5537e-05 2024-09-17T16:13:01.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.513405+0000", 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4691e-05 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.513975+0000", 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057031200000000001 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.514004+0000", 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9363999999999999e-05 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:29540 2.5 e129/104 2:a1cc521d:::benchmark_data_smithi173_55121_object29539:head v 129'1361, mlcod=128'1358)", 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:49.395196+0000", 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 71.901349730000007, 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12196525799999999, 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.359 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.395196+0000", 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.395196+0000", 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.395197+0000", 2024-09-17T16:13:01.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3099999999999997e-07 2024-09-17T16:13:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.395219+0000", 2024-09-17T16:13:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1532999999999999e-05 2024-09-17T16:13:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.516544+0000", 2024-09-17T16:13:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12132515000000001 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.516548+0000", 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3050000000000003e-06 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.516591+0000", 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2698999999999999e-05 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.516612+0000", 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0656e-05 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.517133+0000", 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00052191599999999998 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:49.517162+0000", 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8268000000000001e-05 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:67888 2.11 e148/120 2:8b79a4c5:::benchmark_data_smithi173_55121_object67887:head v 148'2559, mlcod=148'2555)", 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:25.217865+0000", 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 36.078681346000003, 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.105926117, 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:25.217865+0000", 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:25.217865+0000", 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:25.217865+0000", 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7599999999999997e-07 2024-09-17T16:13:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:25.217883+0000", 2024-09-17T16:13:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7557000000000001e-05 2024-09-17T16:13:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:25.322741+0000", 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.104857942 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:25.322745+0000", 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4730000000000002e-06 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:25.322796+0000", 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0997e-05 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:25.322811+0000", 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4541e-05 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:25.323741+0000", 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093036400000000004 2024-09-17T16:13:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:25.323791+0000", 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9767e-05 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:76391 2.11 e149/120 2:8d70a08e:::benchmark_data_smithi173_55121_object76390:head v 149'2818, mlcod=149'2815)", 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:32.084009+0000", 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 29.212536958000001, 2024-09-17T16:13:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58258129999999997, 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:32.084009+0000", 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:32.084009+0000", 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:32.084010+0000", 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1489999999999999e-06 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:32.084038+0000", 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7321000000000002e-05 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:32.665292+0000", 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58125423700000001 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:32.665317+0000", 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5333999999999998e-05 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:32.665371+0000", 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3773999999999997e-05 2024-09-17T16:13:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:32.665392+0000", 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0973999999999999e-05 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:32.666550+0000", 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00115835 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:32.666591+0000", 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0160999999999997e-05 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:77387 2.11 e149/120 2:8e0e6431:::benchmark_data_smithi173_55121_object898:head v 149'2851, mlcod=149'2847)", 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:33.725829+0000", 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.570717438999999, 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.11501781699999999, 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:33.725829+0000", 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:33.725829+0000", 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:33.725829+0000", 2024-09-17T16:13:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7999999999999995e-07 2024-09-17T16:13:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:33.725834+0000", 2024-09-17T16:13:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6450000000000004e-06 2024-09-17T16:13:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:33.839845+0000", 2024-09-17T16:13:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.114010836 2024-09-17T16:13:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:33.839851+0000", 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.336e-06 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:33.839896+0000", 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4369999999999997e-05 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:33.839908+0000", 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2867000000000001e-05 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:33.840815+0000", 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00090612600000000005 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:33.840847+0000", 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2057000000000002e-05 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:79880 2.11 e150/120 2:8ce06f57:::benchmark_data_smithi173_55121_object3391:head v 150'2942, mlcod=150'2939)", 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:35.500906+0000", 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 25.795640619, 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.168146507, 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.500906+0000", 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.500906+0000", 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.500906+0000", 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0699999999999996e-07 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.500911+0000", 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8169999999999998e-06 2024-09-17T16:13:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.667903+0000", 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16699181299999999 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.667910+0000", 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9210000000000003e-06 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.667959+0000", 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9490000000000002e-05 2024-09-17T16:13:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.667981+0000", 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1654999999999998e-05 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.669010+0000", 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010286449999999999 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.669052+0000", 2024-09-17T16:13:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2358999999999999e-05 2024-09-17T16:13:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:90333 2.18 e158/155 2:19d21309:::benchmark_data_smithi173_55121_object13844:head v 158'3326, mlcod=132'1615)", 2024-09-17T16:13:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:46.029751+0000", 2024-09-17T16:13:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 15.266794762, 2024-09-17T16:13:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.108304949, 2024-09-17T16:13:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:46.029751+0000", 2024-09-17T16:13:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:46.029751+0000", 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:46.029753+0000", 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.184e-06 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:46.029760+0000", 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4139999999999997e-06 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:46.136801+0000", 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10704092900000001 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:46.136810+0000", 2024-09-17T16:13:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7490000000000004e-06 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:46.136872+0000", 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2225999999999997e-05 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:46.136891+0000", 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8833000000000001e-05 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:46.138013+0000", 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011224570000000001 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:46.138056+0000", 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3157e-05 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:94053 2.18 e160/155 2:1a522ef7:::benchmark_data_smithi173_55121_object17564:head v 160'3459, mlcod=132'1615)", 2024-09-17T16:13:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:49.942283+0000", 2024-09-17T16:13:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.354263679000001, 2024-09-17T16:13:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.107184452, 2024-09-17T16:13:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.942283+0000", 2024-09-17T16:13:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.942283+0000", 2024-09-17T16:13:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.942284+0000", 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6670000000000001e-06 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.942295+0000", 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1246e-05 2024-09-17T16:13:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.048185+0000", 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.105889306 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.048191+0000", 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1040000000000003e-06 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.048241+0000", 2024-09-17T16:13:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.986e-05 2024-09-17T16:13:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.048258+0000", 2024-09-17T16:13:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7096999999999999e-05 2024-09-17T16:13:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.049422+0000", 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011643490000000001 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.049467+0000", 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4823000000000001e-05 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:94103 2.d e160/95 2:b0e5cb1e:::benchmark_data_smithi173_55121_object17614:head v 160'5838, mlcod=160'5836)", 2024-09-17T16:13:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:50.114426+0000", 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.182120636000001, 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16927261900000001, 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.114426+0000", 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.114426+0000", 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.114427+0000", 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8500000000000001e-06 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.114441+0000", 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3553000000000001e-05 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.281320+0000", 2024-09-17T16:13:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16687907199999999 2024-09-17T16:13:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.281333+0000", 2024-09-17T16:13:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2859e-05 2024-09-17T16:13:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.281495+0000", 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000162122 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.281512+0000", 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7172000000000001e-05 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.283666+0000", 2024-09-17T16:13:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021537380000000001 2024-09-17T16:13:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.283698+0000", 2024-09-17T16:13:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2252999999999998e-05 2024-09-17T16:13:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:95672 2.f e162/95 2:f5093e3c:::benchmark_data_smithi173_55121_object19183:head v 162'5950, mlcod=162'5947)", 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:52.203326+0000", 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.0932198720000006, 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.11586840700000001, 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:52.203326+0000", 2024-09-17T16:13:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:52.203326+0000", 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:52.203328+0000", 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.39e-06 2024-09-17T16:13:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:52.203334+0000", 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4960000000000001e-06 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:52.316516+0000", 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.113181738 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:52.316525+0000", 2024-09-17T16:13:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6899999999999998e-06 2024-09-17T16:13:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:52.316597+0000", 2024-09-17T16:13:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2768000000000005e-05 2024-09-17T16:13:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:52.316621+0000", 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3536999999999998e-05 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:52.319115+0000", 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024942559999999998 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:52.319195+0000", 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9531999999999997e-05 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:01.407 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:13:01.407 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:01.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:01.430 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:01.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:01.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:01.554 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:01.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:01.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:01 smithi071 ceph-mon[30002]: pgmap v433: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 3 active+recovering+undersized+degraded+remapped, 21 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 5675/157446 objects degraded (3.604%); 22 MiB/s, 488 objects/s recovering 2024-09-17T16:13:01.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4062928167' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:13:01.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:01 smithi071 ceph-mon[30002]: osdmap e169: 12 total, 12 up, 12 in 2024-09-17T16:13:01.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:01 smithi173 ceph-mon[30530]: pgmap v433: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 3 active+recovering+undersized+degraded+remapped, 21 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 5675/157446 objects degraded (3.604%); 22 MiB/s, 488 objects/s recovering 2024-09-17T16:13:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4062928167' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:13:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:01 smithi173 ceph-mon[30530]: osdmap e169: 12 total, 12 up, 12 in 2024-09-17T16:13:01.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:01.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:01.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:01.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:01 smithi017 ceph-mon[27571]: pgmap v433: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 3 active+recovering+undersized+degraded+remapped, 21 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 5675/157446 objects degraded (3.604%); 22 MiB/s, 488 objects/s recovering 2024-09-17T16:13:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4062928167' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:13:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:01 smithi017 ceph-mon[27571]: osdmap e169: 12 total, 12 up, 12 in 2024-09-17T16:13:01.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:01.811 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:01.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:01.913 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:01.940 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:01.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:02.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:02.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:02.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:02.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:02.154 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:13:02.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:02.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:02.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:02.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:02.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:02.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:02.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:02.424 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:02.450 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:02.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:02.461 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:13:02.471 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T16:13:02.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:02.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:02 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:13:02.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:02 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:13:02.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:02.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:02.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:02 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:13:02.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:02 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:13:02.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:02.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:02.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:02.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:02.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:02 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:13:02.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:02 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:13:02.805 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:02.830 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:02.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:02.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:02.955 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:02.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:02.969 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:13:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:13:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:105490 2.16 2:6e87e2b0:::benchmark_data_smithi173_55121_object29001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:02.937587+0000", 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.029755569999999999, 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.029780046000000001, 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 105490 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.937587+0000", 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.937587+0000", 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.937589+0000", 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.733e-06 2024-09-17T16:13:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.937596+0000", 2024-09-17T16:13:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1119999999999998e-06 2024-09-17T16:13:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.937612+0000", 2024-09-17T16:13:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5690999999999999e-05 2024-09-17T16:13:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.937620+0000", 2024-09-17T16:13:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.388e-06 2024-09-17T16:13:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.937677+0000", 2024-09-17T16:13:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6245000000000001e-05 2024-09-17T16:13:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.937785+0000", 2024-09-17T16:13:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010857 2024-09-17T16:13:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:13:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.937862+0000", 2024-09-17T16:13:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6318999999999999e-05 2024-09-17T16:13:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.939245+0000", 2024-09-17T16:13:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013836499999999999 2024-09-17T16:13:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.939599+0000", 2024-09-17T16:13:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000353822 2024-09-17T16:13:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:105495 2.2 2:47e62fd2:::benchmark_data_smithi173_55121_object29006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-17T16:13:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:02.950903+0000", 2024-09-17T16:13:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.016439927, 2024-09-17T16:13:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.016512684999999999, 2024-09-17T16:13:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:13:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 105495 2024-09-17T16:13:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950903+0000", 2024-09-17T16:13:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950903+0000", 2024-09-17T16:13:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950905+0000", 2024-09-17T16:13:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1330000000000002e-06 2024-09-17T16:13:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.985 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:02.985 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950912+0000", 2024-09-17T16:13:02.985 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8560000000000001e-06 2024-09-17T16:13:02.985 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.986 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:02.986 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950929+0000", 2024-09-17T16:13:02.986 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7039000000000001e-05 2024-09-17T16:13:02.986 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.986 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:02.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950936+0000", 2024-09-17T16:13:02.987 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0999999999999998e-06 2024-09-17T16:13:02.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:02.988 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950999+0000", 2024-09-17T16:13:02.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2403999999999994e-05 2024-09-17T16:13:02.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.988 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:02.989 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.951087+0000", 2024-09-17T16:13:02.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7913999999999999e-05 2024-09-17T16:13:02.989 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.989 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.989 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:13:02.989 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.951156+0000", 2024-09-17T16:13:02.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9014e-05 2024-09-17T16:13:02.990 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.990 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:02.990 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.952531+0000", 2024-09-17T16:13:02.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013754959999999999 2024-09-17T16:13:02.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:02.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:02.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:02.991 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.953080+0000", 2024-09-17T16:13:02.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054879500000000003 2024-09-17T16:13:02.991 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:02.992 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:02.992 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:02.992 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:02.992 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:13:02.992 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 2 2024-09-17T16:13:02.993 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:13:03.056 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:03.080 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:03.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:03.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:03.183 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:03.204 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:03.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:03.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:03.277 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:03.306 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:03.314 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:03.327 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:03.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:03.430 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:03.456 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:03.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:03.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:03 smithi071 ceph-mon[30002]: pgmap v435: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.0 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 4292/148443 objects degraded (2.891%); 26 MiB/s, 486 objects/s recovering 2024-09-17T16:13:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:03 smithi071 ceph-mon[30002]: osdmap e170: 12 total, 12 up, 12 in 2024-09-17T16:13:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:03 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:13:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:03 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:13:03.626 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:03.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:03.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:03 smithi173 ceph-mon[30530]: pgmap v435: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.0 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 4292/148443 objects degraded (2.891%); 26 MiB/s, 486 objects/s recovering 2024-09-17T16:13:03.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:03 smithi173 ceph-mon[30530]: osdmap e170: 12 total, 12 up, 12 in 2024-09-17T16:13:03.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:03 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:13:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:03 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:13:03.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:03 smithi017 ceph-mon[27571]: pgmap v435: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.0 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 4292/148443 objects degraded (2.891%); 26 MiB/s, 486 objects/s recovering 2024-09-17T16:13:03.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:03 smithi017 ceph-mon[27571]: osdmap e170: 12 total, 12 up, 12 in 2024-09-17T16:13:03.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:03 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:13:03.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:03 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:13:03.729 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:03.756 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:03.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:03.796 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:13:03.858 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:03.906 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:03.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:04.009 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:04.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:04.039 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:04.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:04.166 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:13:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:04.173 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:04.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:04.274 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:04.289 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:13:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:04.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:04.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:04.433 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:04.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:04.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:04 smithi071 ceph-mon[30002]: osdmap e171: 12 total, 12 up, 12 in 2024-09-17T16:13:04.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:04 smithi071 ceph-mon[30002]: 2.0 scrub starts 2024-09-17T16:13:04.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:04 smithi071 ceph-mon[30002]: pgmap v438: 27 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 823 op/s; 4292/148380 objects degraded (2.893%); 16 MiB/s, 303 objects/s recovering 2024-09-17T16:13:04.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/344631736' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:04.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:04 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 4292/148443 objects degraded (2.891%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:13:04.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:04 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:04.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/344631736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:13:04.437 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:04 smithi071 ceph-mon[30002]: osdmap e172: 12 total, 12 up, 12 in 2024-09-17T16:13:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:04 smithi017 ceph-mon[27571]: osdmap e171: 12 total, 12 up, 12 in 2024-09-17T16:13:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:04 smithi017 ceph-mon[27571]: 2.0 scrub starts 2024-09-17T16:13:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:04 smithi017 ceph-mon[27571]: pgmap v438: 27 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 823 op/s; 4292/148380 objects degraded (2.893%); 16 MiB/s, 303 objects/s recovering 2024-09-17T16:13:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/344631736' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:04 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 4292/148443 objects degraded (2.891%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:13:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:04 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/344631736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:13:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:04 smithi017 ceph-mon[27571]: osdmap e172: 12 total, 12 up, 12 in 2024-09-17T16:13:04.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:04.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:04.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:04.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:04 smithi173 ceph-mon[30530]: osdmap e171: 12 total, 12 up, 12 in 2024-09-17T16:13:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:04 smithi173 ceph-mon[30530]: 2.0 scrub starts 2024-09-17T16:13:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:04 smithi173 ceph-mon[30530]: pgmap v438: 27 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 23 active+clean; 3.0 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 823 op/s; 4292/148380 objects degraded (2.893%); 16 MiB/s, 303 objects/s recovering 2024-09-17T16:13:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/344631736' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:04 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 4292/148443 objects degraded (2.891%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:13:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:04 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/344631736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:13:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:04 smithi173 ceph-mon[30530]: osdmap e172: 12 total, 12 up, 12 in 2024-09-17T16:13:04.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:04.699 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:04.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:04.785 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:13:04.785 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:13:04.785 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:13:04.785 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:13:04.785 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:13:04.801 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:04.838 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:04.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:04.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:04.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:04.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:04.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:05.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:05.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:05.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:05.193 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:13:05.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:05.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:05.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:05.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:05.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:05.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:05.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:05.365 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T16:13:05.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:05.479 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:05.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:05.579 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4090788123' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:13:05.579 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:05 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:13:05.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:05.609 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:05.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:05.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:13:05.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4090788123' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:13:05.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:05 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:13:05.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4090788123' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:13:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:05 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:13:05.739 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:05.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:05.841 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:05.870 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:05.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:05.963 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:13:05.972 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:05.998 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:05.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:06.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:06.123 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:06.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:06.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T16:13:06.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:06.241 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:06.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:06.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:06.350 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:06.375 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:06.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:06.390 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:06 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:13:06.391 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:06 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:13:06.391 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:06 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:13:06.391 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:06 smithi017 ceph-mon[27571]: osdmap e173: 12 total, 12 up, 12 in 2024-09-17T16:13:06.391 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:06 smithi017 ceph-mon[27571]: pgmap v441: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 23 active+clean; 2.9 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 3657/140373 objects degraded (2.605%); 2.4 MiB/s, 85 objects/s recovering 2024-09-17T16:13:06.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:06.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:06.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:06.524 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:13:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:13:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:13:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:13:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:21501 2.2 2:47484336:::benchmark_data_smithi173_55121_object21500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:13:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:26.288763+0000", 2024-09-17T16:13:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 100.231876677, 2024-09-17T16:13:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22508275899999999, 2024-09-17T16:13:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 21501 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.288763+0000", 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.288763+0000", 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.288766+0000", 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4619999999999999e-06 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.288800+0000", 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4125999999999998e-05 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.288817+0000", 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7076999999999999e-05 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.288826+0000", 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1430000000000004e-06 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.288882+0000", 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5358000000000001e-05 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.288887+0000", 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3410000000000002e-06 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.291651+0000", 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0027636900000000001 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.291758+0000", 2024-09-17T16:13:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000106965 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.291889+0000", 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000131366 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.292673+0000", 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00078437400000000001 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.293265+0000", 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059179700000000003 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.513770+0000", 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22050476799999999 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.513806+0000", 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6202999999999999e-05 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.513846+0000", 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0089000000000002e-05 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:21741 2.3 2:c4bdb6e7:::benchmark_data_smithi173_55121_object21740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 99.857669971000007, 2024-09-17T16:13:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55429691999999997, 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 21741 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:13:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662972+0000", 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.722e-06 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663024+0000", 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2175000000000001e-05 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663036+0000", 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2131e-05 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663042+0000", 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9649999999999996e-06 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663084+0000", 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2354999999999997e-05 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663166+0000", 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1699999999999994e-05 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663249+0000", 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2714999999999997e-05 2024-09-17T16:13:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663804+0000", 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055499599999999998 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.664173+0000", 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000369181 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217193+0000", 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55302014600000005 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217234+0000", 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1149000000000002e-05 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217267+0000", 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2685000000000003e-05 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:23782 2.2 2:41257b37:::benchmark_data_smithi173_55121_object23781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 97.832351692000003, 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51828190799999996, 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 23782 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688289+0000", 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7400000000000002e-07 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688376+0000", 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6744999999999997e-05 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688387+0000", 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0499000000000001e-05 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688393+0000", 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9689999999999997e-06 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688434+0000", 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0694e-05 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688511+0000", 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6656000000000006e-05 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688594+0000", 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2682999999999994e-05 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689092+0000", 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049850100000000004 2024-09-17T16:13:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689369+0000", 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027721100000000003 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206489+0000", 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51712006899999996 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206524+0000", 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4516e-05 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206570+0000", 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6490999999999999e-05 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:23782 2.2 e119/104 ondisk, result = 0)", 2024-09-17T16:13:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 97.831258907000006, 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51719464199999998, 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689382+0000", 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689385+0000", 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3569999999999998e-06 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206448+0000", 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51706313699999995 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206453+0000", 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4360000000000004e-06 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206480+0000", 2024-09-17T16:13:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7770999999999999e-05 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206486+0000", 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6779999999999997e-06 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206576+0000", 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9701000000000001e-05 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24369 2.3 2:c67d9130:::benchmark_data_smithi173_55121_object24368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 96.833132371000005, 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55710953100000005, 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24369 2024-09-17T16:13:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687509+0000", 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1399999999999995e-07 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687596+0000", 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7288999999999997e-05 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687607+0000", 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1505000000000001e-05 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687613+0000", 2024-09-17T16:13:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1440000000000002e-06 2024-09-17T16:13:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687655+0000", 2024-09-17T16:13:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2240000000000002e-05 2024-09-17T16:13:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687782+0000", 2024-09-17T16:13:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000127043 2024-09-17T16:13:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687864+0000", 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2064000000000006e-05 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688444+0000", 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058032600000000004 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688721+0000", 2024-09-17T16:13:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000276734 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244551+0000", 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55582960199999998 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244588+0000", 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7004e-05 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244617+0000", 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9666e-05 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:24369 2.3 e120/104 ondisk, result = 0)", 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 96.831925179999999, 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55590799999999996, 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688716+0000", 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8999999999999996e-07 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688719+0000", 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5980000000000001e-06 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244478+0000", 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55575852299999995 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244483+0000", 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244541+0000", 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8377e-05 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244547+0000", 2024-09-17T16:13:06.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8980000000000001e-06 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244623+0000", 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5780999999999998e-05 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:29877 2.2 2:40729eb6:::benchmark_data_smithi173_55121_object29876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 76.316770794000007, 2024-09-17T16:13:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.874997425, 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 29877 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:13:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203870+0000", 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203910+0000", 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9477999999999997e-05 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203918+0000", 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203922+0000", 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6369999999999998e-06 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203958+0000", 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5045000000000001e-05 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204026+0000", 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8693000000000001e-05 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204105+0000", 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8458e-05 2024-09-17T16:13:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204758+0000", 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00065323300000000003 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.205073+0000", 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031491100000000002 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078782+0000", 2024-09-17T16:13:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87370871400000005 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078827+0000", 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5219999999999997e-05 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078867+0000", 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9953000000000002e-05 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:79524 2.15 e150/113 2:a890cf23:::benchmark_data_smithi173_55121_object3035:head v 150'2897, mlcod=150'2893)", 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:35.019124+0000", 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 31.501516531, 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24077440899999999, 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.019124+0000", 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.019124+0000", 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.019124+0000", 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0200000000000002e-07 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.019129+0000", 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5260000000000004e-06 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.258791+0000", 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.239661979 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.258795+0000", 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7929999999999997e-06 2024-09-17T16:13:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.258866+0000", 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.046e-05 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.258888+0000", 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2257000000000001e-05 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.259861+0000", 2024-09-17T16:13:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00097262499999999997 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.259898+0000", 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7166999999999998e-05 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:88614 2.3 2:c3dfe8aa:::benchmark_data_smithi173_55121_object12125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:43.927637+0000", 2024-09-17T16:13:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.593003528000001, 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21890425299999999, 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 88614 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.927637+0000", 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.927637+0000", 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.927638+0000", 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4050000000000001e-06 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.927643+0000", 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2719999999999997e-06 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.927654+0000", 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0988999999999999e-05 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.927659+0000", 2024-09-17T16:13:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9949999999999996e-06 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.929915+0000", 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022558309999999998 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.930025+0000", 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000110108 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:13:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.930143+0000", 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000117379 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.931999+0000", 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00185611 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.933027+0000", 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00102819 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.145966+0000", 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21293917600000001 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.146031+0000", 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4727999999999994e-05 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.146541+0000", 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00051006999999999995 2024-09-17T16:13:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:88652 2.2 2:45ffdfe0:::benchmark_data_smithi173_55121_object12163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:13:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:13:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.563885200000001, 2024-09-17T16:13:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29260301500000002, 2024-09-17T16:13:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 88652 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956757+0000", 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7859999999999999e-06 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956762+0000", 2024-09-17T16:13:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.31e-06 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956773+0000", 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0832999999999999e-05 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956778+0000", 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2320000000000001e-06 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956829+0000", 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1124e-05 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956915+0000", 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5541e-05 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.957003+0000", 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8332999999999996e-05 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.958148+0000", 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001144937 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.961994+0000", 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038457320000000001 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249267+0000", 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28727286400000002 2024-09-17T16:13:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249320+0000", 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3016000000000002e-05 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249358+0000", 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8306999999999998e-05 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(2.3 158/155 [PushReplyOp(2:c6916317:::benchmark_data_smithi173_55121_object76243:head)])", 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.797592950999999, 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45253040700000002, 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723048+0000", 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4300000000000003e-07 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723052+0000", 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9310000000000001e-06 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175226+0000", 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45217462899999999 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175233+0000", 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4099999999999996e-06 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175322+0000", 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9011000000000005e-05 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175578+0000", 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025588299999999999 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:94077 2.16 2:6e055a89:::benchmark_data_smithi173_55121_object17588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 16.442167328, 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28469862699999998, 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 94077 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078474+0000", 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.274e-06 2024-09-17T16:13:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078479+0000", 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6310000000000002e-06 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078490+0000", 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1799e-05 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078496+0000", 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7579999999999998e-06 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078540+0000", 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3686999999999997e-05 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078624+0000", 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4153000000000006e-05 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078694+0000", 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9533e-05 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.079805+0000", 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011117550000000001 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.085499+0000", 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0056938969999999998 2024-09-17T16:13:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363107+0000", 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27760825700000002 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363140+0000", 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2055999999999999e-05 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363171+0000", 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1826999999999999e-05 2024-09-17T16:13:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97425 2.3 2:c4562d4f:::benchmark_data_smithi173_55121_object20936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.673568974, 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29697053200000001, 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97425 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847072+0000", 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1399999999999997e-07 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847074+0000", 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7329999999999999e-06 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847082+0000", 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6830000000000008e-06 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847088+0000", 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0499999999999997e-06 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848028+0000", 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093989799999999999 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848133+0000", 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000104874 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848231+0000", 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8121999999999993e-05 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.849551+0000", 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00131985 2024-09-17T16:13:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143633+0000", 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29408256900000002 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143940+0000", 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030690399999999999 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143994+0000", 2024-09-17T16:13:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3161999999999997e-05 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.144042+0000", 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8072999999999999e-05 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97454 2.16 2:6cbc5315:::benchmark_data_smithi173_55121_object20965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.914416+0000", 2024-09-17T16:13:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.606224085000001, 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22414108499999999, 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97454 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.914416+0000", 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.914416+0000", 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.914418+0000", 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8080000000000001e-06 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.914425+0000", 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3579999999999997e-06 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.914442+0000", 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7022000000000001e-05 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.914450+0000", 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4209999999999998e-06 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.914511+0000", 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1799999999999998e-05 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.914651+0000", 2024-09-17T16:13:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013953500000000001 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.914783+0000", 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000132519 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.916017+0000", 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012333470000000001 2024-09-17T16:13:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.916371+0000", 2024-09-17T16:13:06.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00035391399999999999 2024-09-17T16:13:06.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.138459+0000", 2024-09-17T16:13:06.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22208856799999999 2024-09-17T16:13:06.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.138515+0000", 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5331000000000003e-05 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.138557+0000", 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2462000000000001e-05 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.609 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:97454 2.16 e163/115 ondisk, result = 0)", 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.916197+0000", 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.604442588, 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.222366117, 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.916197+0000", 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.916197+0000", 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.916198+0000", 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5300000000000003e-07 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.916202+0000", 2024-09-17T16:13:06.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0820000000000001e-06 2024-09-17T16:13:06.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.138393+0000", 2024-09-17T16:13:06.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22219087600000001 2024-09-17T16:13:06.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:06.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.138398+0000", 2024-09-17T16:13:06.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9189999999999998e-06 2024-09-17T16:13:06.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.138444+0000", 2024-09-17T16:13:06.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6338999999999998e-05 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.138453+0000", 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6209999999999993e-06 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.138564+0000", 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011052699999999999 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97456 2.2 2:4302bb4b:::benchmark_data_smithi173_55121_object20967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.918164+0000", 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.602475769, 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22776361000000001, 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97456 2024-09-17T16:13:06.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.918164+0000", 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.918164+0000", 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.918165+0000", 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2600000000000002e-07 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.918168+0000", 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2650000000000001e-06 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.918176+0000", 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4549999999999998e-06 2024-09-17T16:13:06.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.918180+0000", 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9940000000000002e-06 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.918211+0000", 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1349000000000001e-05 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.918280+0000", 2024-09-17T16:13:06.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9246999999999996e-05 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.918343+0000", 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2466999999999997e-05 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.919646+0000", 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001303589 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.919923+0000", 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027716999999999998 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.145848+0000", 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22592457599999999 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.145884+0000", 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5837000000000003e-05 2024-09-17T16:13:06.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.145928+0000", 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4035000000000001e-05 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:97456 2.2 e163/104 ondisk, result = 0)", 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.919736+0000", 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.600903607999999, 2024-09-17T16:13:06.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22619621000000001, 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.919736+0000", 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.919736+0000", 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.919737+0000", 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6899999999999997e-07 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.919741+0000", 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8840000000000004e-06 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.145799+0000", 2024-09-17T16:13:06.624 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.226057698 2024-09-17T16:13:06.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.625 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.145804+0000", 2024-09-17T16:13:06.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4199999999999998e-06 2024-09-17T16:13:06.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.625 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.145839+0000", 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4892999999999999e-05 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.145844+0000", 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4439999999999999e-06 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.145933+0000", 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8201999999999993e-05 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.626 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:104910 2.16 2:6ea0893e:::benchmark_data_smithi173_55121_object28421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:02.015154+0000", 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.5054857019999996, 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22116918599999999, 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 104910 2024-09-17T16:13:06.627 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.015154+0000", 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.015154+0000", 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.628 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.015155+0000", 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1200000000000001e-06 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.015160+0000", 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6979999999999997e-06 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.015170+0000", 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.029e-05 2024-09-17T16:13:06.629 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.015176+0000", 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1240000000000004e-06 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.015226+0000", 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0197000000000001e-05 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.015353+0000", 2024-09-17T16:13:06.630 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000127267 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.015474+0000", 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120679 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.016824+0000", 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001350125 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.051228+0000", 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.034404226000000003 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.236263+0000", 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18503476199999999 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.236297+0000", 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3708999999999999e-05 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.236324+0000", 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6988999999999999e-05 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:104910 2.16 e169/115 ondisk, result = 0)", 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:02.017002+0000", 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.5036377640000005, 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21932621699999999, 2024-09-17T16:13:06.633 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.017002+0000", 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.017002+0000", 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.017003+0000", 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7300000000000001e-07 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.017006+0000", 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0070000000000002e-06 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.236217+0000", 2024-09-17T16:13:06.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21921096900000001 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.236221+0000", 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4279999999999998e-06 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.236254+0000", 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2954000000000003e-05 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.236259+0000", 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1590000000000001e-06 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.236329+0000", 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9227000000000007e-05 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.637 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:105495 2.2 2:47e62fd2:::benchmark_data_smithi173_55121_object29006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:02.950903+0000", 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 3.5697369430000001, 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.236890983, 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 105495 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.638 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950903+0000", 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950903+0000", 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950905+0000", 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1330000000000002e-06 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.639 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950912+0000", 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8560000000000001e-06 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950929+0000", 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7039000000000001e-05 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950936+0000", 2024-09-17T16:13:06.640 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0999999999999998e-06 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950999+0000", 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2403999999999994e-05 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.951087+0000", 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7913999999999999e-05 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:13:06.641 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.951156+0000", 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9014e-05 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.952531+0000", 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013754959999999999 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.953080+0000", 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054879500000000003 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.642 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:03.187722+0000", 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23464195500000001 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:03.187764+0000", 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2058999999999998e-05 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:03.187794+0000", 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0218000000000001e-05 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.643 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.644 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.644 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:06.644 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:06.644 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:13:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:06.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:06.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:06.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:06 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:13:06.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:06 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:13:06.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:06 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:13:06.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:06 smithi173 ceph-mon[30530]: osdmap e173: 12 total, 12 up, 12 in 2024-09-17T16:13:06.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:06 smithi173 ceph-mon[30530]: pgmap v441: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 23 active+clean; 2.9 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 3657/140373 objects degraded (2.605%); 2.4 MiB/s, 85 objects/s recovering 2024-09-17T16:13:06.746 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:06.777 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:06.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:06.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:06.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:06 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:13:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:06 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:13:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:06 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:13:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:06 smithi071 ceph-mon[30002]: osdmap e173: 12 total, 12 up, 12 in 2024-09-17T16:13:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:06 smithi071 ceph-mon[30002]: pgmap v441: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 23 active+clean; 2.9 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 3657/140373 objects degraded (2.605%); 2.4 MiB/s, 85 objects/s recovering 2024-09-17T16:13:06.830 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:13:06.830 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:13:06.846 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:06.863 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:13:06.863 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:13:06.879 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:06.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:13:06.893 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:13:06.905 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:06.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:06.920 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:13:06.921 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:13:06.948 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:13:06.948 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:13:06.975 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:13:06.975 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:13:06.999 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:13:06.999 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:13:07.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:07.022 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:13:07.022 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:13:07.029 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:07.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:07.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:07.045 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:13:07.046 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:13:07.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:13:07.076 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:13:07.076 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:13:07.103 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:13:07.103 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:13:07.131 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:13:07.132 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:13:07.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:07.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:07.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:07.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:07.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:07.288 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:07.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:07.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:07.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:07.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:07.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:07.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:07.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:07.600 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:13:07.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:13:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:13:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:13:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:13:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:13:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:07 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:13:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:07 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:13:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1790541789' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:07.656 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:07.680 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:07.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:13:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:13:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:13:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:13:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:13:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:07 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:13:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:07 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:13:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1790541789' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:07.783 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:07.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:07.806 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:07.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:13:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:13:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:13:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:13:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:13:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:07 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:13:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:07 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:13:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1790541789' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:07.861 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:13:07.907 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:07.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:07.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:08.032 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:08.054 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:08.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:08.157 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:08.180 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:08.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:08.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:08.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:08.305 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:08.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:08.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:08.378 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:08.378 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:08.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:08.433 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:08.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:08.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:08.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:08 smithi071 ceph-mon[30002]: osdmap e174: 12 total, 12 up, 12 in 2024-09-17T16:13:08.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:08 smithi071 ceph-mon[30002]: pgmap v443: 27 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 23 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 2339/134268 objects degraded (1.742%); 21 MiB/s, 362 objects/s recovering 2024-09-17T16:13:08.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:08.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:08.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:08.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:08.667 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:08 smithi017 ceph-mon[27571]: osdmap e174: 12 total, 12 up, 12 in 2024-09-17T16:13:08.667 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:08 smithi017 ceph-mon[27571]: pgmap v443: 27 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 23 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 2339/134268 objects degraded (1.742%); 21 MiB/s, 362 objects/s recovering 2024-09-17T16:13:08.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:08.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:08.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:08.827 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:08.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:08.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:08 smithi173 ceph-mon[30530]: osdmap e174: 12 total, 12 up, 12 in 2024-09-17T16:13:08.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:08 smithi173 ceph-mon[30530]: pgmap v443: 27 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 23 active+clean; 2.7 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 2339/134268 objects degraded (1.742%); 21 MiB/s, 362 objects/s recovering 2024-09-17T16:13:08.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:08.954 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:08.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:09.016 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:09.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:09.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:09.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:09.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:13:09.188 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:09.219 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:09.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:09.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:09.292 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T16:13:09.321 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:09.352 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:09.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:09.408 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:13:09.454 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:09.478 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:09.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:09.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:09.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:09.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:09.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:09 smithi017 ceph-mon[27571]: 2.19 scrub starts 2024-09-17T16:13:09.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:09 smithi017 ceph-mon[27571]: osdmap e175: 12 total, 12 up, 12 in 2024-09-17T16:13:09.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:09 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:13:09.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:09 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:13:09.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:09 smithi017 ceph-mon[27571]: 2.8 scrub starts 2024-09-17T16:13:09.628 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/122134936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:13:09.628 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:09 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2339/134268 objects degraded (1.742%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:13:09.628 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/122134936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:13:09.628 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:09 smithi017 ceph-mon[27571]: osdmap e176: 12 total, 12 up, 12 in 2024-09-17T16:13:09.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:09.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:09.728 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:09 smithi071 ceph-mon[30002]: 2.19 scrub starts 2024-09-17T16:13:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:09 smithi071 ceph-mon[30002]: osdmap e175: 12 total, 12 up, 12 in 2024-09-17T16:13:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:09 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:13:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:09 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:13:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:09 smithi071 ceph-mon[30002]: 2.8 scrub starts 2024-09-17T16:13:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/122134936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:13:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:09 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2339/134268 objects degraded (1.742%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:13:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/122134936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:13:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:09 smithi071 ceph-mon[30002]: osdmap e176: 12 total, 12 up, 12 in 2024-09-17T16:13:09.830 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:09.852 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:09.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:09.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:09.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:09 smithi173 ceph-mon[30530]: 2.19 scrub starts 2024-09-17T16:13:09.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:09 smithi173 ceph-mon[30530]: osdmap e175: 12 total, 12 up, 12 in 2024-09-17T16:13:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:09 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:13:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:09 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:13:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:09 smithi173 ceph-mon[30530]: 2.8 scrub starts 2024-09-17T16:13:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/122134936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:13:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:09 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2339/134268 objects degraded (1.742%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:13:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/122134936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:13:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:09 smithi173 ceph-mon[30530]: osdmap e176: 12 total, 12 up, 12 in 2024-09-17T16:13:09.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:09.976 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:09.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:10.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:10.101 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:10.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:10.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:10.205 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:10.226 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:10.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:10.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T16:13:10.300 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:13:10.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:10.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:10.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:10.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:10.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:10.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:10 smithi017 ceph-mon[27571]: 2.19 scrub ok 2024-09-17T16:13:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:10 smithi017 ceph-mon[27571]: 2.8 scrub ok 2024-09-17T16:13:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:10 smithi017 ceph-mon[27571]: pgmap v446: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 24 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 2078/128769 objects degraded (1.614%); 22 MiB/s, 416 objects/s recovering 2024-09-17T16:13:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/39442552' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:10 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:13:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/39442552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:10 smithi017 ceph-mon[27571]: osdmap e177: 12 total, 12 up, 12 in 2024-09-17T16:13:10.562 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:10 smithi071 ceph-mon[30002]: 2.19 scrub ok 2024-09-17T16:13:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:10 smithi071 ceph-mon[30002]: 2.8 scrub ok 2024-09-17T16:13:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:10 smithi071 ceph-mon[30002]: pgmap v446: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 24 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 2078/128769 objects degraded (1.614%); 22 MiB/s, 416 objects/s recovering 2024-09-17T16:13:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/39442552' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:10 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:13:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/39442552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:10 smithi071 ceph-mon[30002]: osdmap e177: 12 total, 12 up, 12 in 2024-09-17T16:13:10.583 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:10.583 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:10.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:10.615 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:10.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:10.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:13:10.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:10.741 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:10.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:10.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:10.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:10.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:10.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:10.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:10 smithi173 ceph-mon[30530]: 2.19 scrub ok 2024-09-17T16:13:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:10 smithi173 ceph-mon[30530]: 2.8 scrub ok 2024-09-17T16:13:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:10 smithi173 ceph-mon[30530]: pgmap v446: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 24 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 2078/128769 objects degraded (1.614%); 22 MiB/s, 416 objects/s recovering 2024-09-17T16:13:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/39442552' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:10 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:13:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/39442552' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:10 smithi173 ceph-mon[30530]: osdmap e177: 12 total, 12 up, 12 in 2024-09-17T16:13:10.972 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:10.993 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:10.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:10.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:11.068 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:11.069 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:11.095 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:11.118 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:11.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:11.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:11.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:11.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:11.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:11.350 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:11.373 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:11.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:11.436 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:13:11.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:11.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:11.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:11.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:11.607 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:11.640 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:11.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:11.700 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:13:11.700 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:13:11.730 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:13:11.730 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:13:11.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:11.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:13:11.759 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:13:11.766 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:13:11.769 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:11.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:11.794 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:13:11.794 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:13:11.794 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1262155610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:11.820 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:13:11.820 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:13:11.842 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:13:11.843 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:13:11.869 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:13:11.869 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:13:11.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:11.892 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:13:11.892 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:13:11.896 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:11.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:11.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:11.922 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:13:11.923 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:13:11.954 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:13:11.954 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:13:11.981 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:13:11.981 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:13:11.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:12.005 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:13:12.006 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:13:12.024 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1262155610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:12.025 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:12.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:12.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:12.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1262155610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:12.149 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:12.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:12.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:12.269 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:13:12.269 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:13:12.269 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:13:12.269 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:13:12.269 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.269 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:13493 2.17 2:ef6c8979:::benchmark_data_smithi173_55121_object13492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:13:12.269 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:12.334922+0000", 2024-09-17T16:13:12.269 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 119.93125582899999, 2024-09-17T16:13:12.269 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0242685929999999, 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 13493 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334922+0000", 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334922+0000", 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334923+0000", 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.192e-06 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334999+0000", 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5327999999999994e-05 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.335013+0000", 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.42e-05 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.335019+0000", 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0920000000000003e-06 2024-09-17T16:13:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.350269+0000", 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.01524939 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.350275+0000", 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7399999999999998e-06 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:13.359107+0000", 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0088320850000001 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:13.359191+0000", 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3566000000000006e-05 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14929 2.11 2.db77d491 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.268492+0000", 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 117.997686212, 2024-09-17T16:13:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.99314310400000005, 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14929 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268492+0000", 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268492+0000", 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268493+0000", 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1900000000000001e-07 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268552+0000", 2024-09-17T16:13:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8807999999999997e-05 2024-09-17T16:13:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268562+0000", 2024-09-17T16:13:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0180000000000001e-05 2024-09-17T16:13:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268567+0000", 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0549999999999999e-06 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268612+0000", 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5719000000000001e-05 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268618+0000", 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5650000000000001e-06 2024-09-17T16:13:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261624+0000", 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.993005519 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261635+0000", 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1439e-05 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14934 2.11 2.10e332f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.326376+0000", 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 117.93980253700001, 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93529751100000003, 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14934 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326376+0000", 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326376+0000", 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326378+0000", 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1849999999999998e-06 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326430+0000", 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2686999999999999e-05 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326448+0000", 2024-09-17T16:13:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7518999999999999e-05 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326455+0000", 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2230000000000001e-06 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326508+0000", 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2645000000000002e-05 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:13:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326517+0000", 2024-09-17T16:13:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7679999999999997e-06 2024-09-17T16:13:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261665+0000", 2024-09-17T16:13:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93514807200000005 2024-09-17T16:13:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261673+0000", 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4120000000000001e-06 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14937 2.11 2.66ac0671 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.333944+0000", 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 117.932233927, 2024-09-17T16:13:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.92790043600000005, 2024-09-17T16:13:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:13:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14937 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.333944+0000", 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.333944+0000", 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.333945+0000", 2024-09-17T16:13:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6499999999999998e-07 2024-09-17T16:13:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334020+0000", 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5496999999999999e-05 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334029+0000", 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5909999999999996e-06 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334033+0000", 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2799999999999997e-06 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334071+0000", 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7719999999999998e-05 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334076+0000", 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6659999999999999e-06 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261837+0000", 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.92776128899999999 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261845+0000", 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6279999999999996e-06 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14955 2.11 2.20b690d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.701197+0000", 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 117.56498154400001, 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56068107599999994, 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14955 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701197+0000", 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701197+0000", 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701206+0000", 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6399999999999992e-06 2024-09-17T16:13:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701258+0000", 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2159e-05 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701289+0000", 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0482999999999999e-05 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701303+0000", 2024-09-17T16:13:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4576000000000001e-05 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701450+0000", 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014646099999999999 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701466+0000", 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5913000000000002e-05 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261871+0000", 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56040511900000001 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261878+0000", 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.725e-06 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14964 2.17 2:ead62e90:::benchmark_data_smithi173_55121_object14963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.870757+0000", 2024-09-17T16:13:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 117.395421497, 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28433903799999999, 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14964 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870757+0000", 2024-09-17T16:13:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870757+0000", 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870757+0000", 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0800000000000004e-07 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870849+0000", 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1188999999999997e-05 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870864+0000", 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4951e-05 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870871+0000", 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4760000000000001e-06 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870907+0000", 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6145e-05 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.870982+0000", 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4803999999999997e-05 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871071+0000", 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8721000000000007e-05 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871587+0000", 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00051653300000000001 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872130+0000", 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054294599999999999 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.154876+0000", 2024-09-17T16:13:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28274554299999999 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.155026+0000", 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014985700000000001 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.155096+0000", 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0065000000000001e-05 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:14964 2.17 e106/104 ondisk, result = 0)", 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.871651+0000", 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 117.394527436, 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28345393800000002, 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871651+0000", 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871651+0000", 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871651+0000", 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2600000000000002e-07 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.871655+0000", 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4790000000000001e-06 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.154629+0000", 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28297412399999999 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.154646+0000", 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6677e-05 2024-09-17T16:13:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.154832+0000", 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000186837 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.154858+0000", 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5276e-05 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.155105+0000", 2024-09-17T16:13:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024691900000000001 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14965 2.11 2.79d92c71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.872030+0000", 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 117.39414792300001, 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.389896574, 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:13:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14965 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872030+0000", 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872030+0000", 2024-09-17T16:13:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872031+0000", 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6899999999999996e-07 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872113+0000", 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1584999999999999e-05 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872126+0000", 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3181e-05 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872132+0000", 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5390000000000004e-06 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872179+0000", 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7058000000000002e-05 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872185+0000", 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7699999999999998e-06 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261919+0000", 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38973334999999998 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261927+0000", 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2220000000000002e-06 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24310 2.7 2:e3e07458:::benchmark_data_smithi173_55121_object24309:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.401789+0000", 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 102.864388865, 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.261667339, 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24310 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401789+0000", 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401789+0000", 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401790+0000", 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2399999999999998e-07 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401863+0000", 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2873000000000005e-05 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401870+0000", 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6000000000000001e-06 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401874+0000", 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8990000000000003e-06 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401902+0000", 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7807999999999999e-05 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.401975+0000", 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2803000000000001e-05 2024-09-17T16:13:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.402056+0000", 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0778000000000005e-05 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.402499+0000", 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00044296800000000002 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.402887+0000", 2024-09-17T16:13:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00038852299999999999 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663392+0000", 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26050499300000002 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663430+0000", 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8226000000000003e-05 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.663457+0000", 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6244000000000001e-05 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24327 2.17 2:ecf8c341:::benchmark_data_smithi173_55121_object24326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.417487+0000", 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 102.848691285, 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26755120999999998, 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24327 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417487+0000", 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417487+0000", 2024-09-17T16:13:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417488+0000", 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0580000000000001e-06 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417538+0000", 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9704999999999999e-05 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417549+0000", 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1037e-05 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417556+0000", 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0280000000000003e-06 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417599+0000", 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3739000000000003e-05 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417683+0000", 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3493999999999997e-05 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.417771+0000", 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8325000000000006e-05 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.418259+0000", 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000487398 2024-09-17T16:13:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.418603+0000", 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00034413100000000002 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.684960+0000", 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26635714300000002 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.684997+0000", 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7218999999999997e-05 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.685038+0000", 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0933000000000003e-05 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24378 2.17 2:eca9db3e:::benchmark_data_smithi173_55121_object24377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.695814+0000", 2024-09-17T16:13:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 102.57036373299999, 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39380668000000002, 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24378 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695814+0000", 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695814+0000", 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695816+0000", 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.15e-06 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695864+0000", 2024-09-17T16:13:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8797000000000001e-05 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695875+0000", 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0795000000000001e-05 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695881+0000", 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7470000000000002e-06 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:12.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695926+0000", 2024-09-17T16:13:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4669000000000002e-05 2024-09-17T16:13:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696011+0000", 2024-09-17T16:13:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5141000000000004e-05 2024-09-17T16:13:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:13:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696097+0000", 2024-09-17T16:13:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5805999999999998e-05 2024-09-17T16:13:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696594+0000", 2024-09-17T16:13:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049760799999999997 2024-09-17T16:13:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.697152+0000", 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055830099999999998 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089548+0000", 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39239545599999998 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089589+0000", 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1539999999999999e-05 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089621+0000", 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.167e-05 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:24378 2.17 e120/104 ondisk, result = 0)", 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.696958+0000", 2024-09-17T16:13:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 102.569219776, 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39266769299999998, 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696958+0000", 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696958+0000", 2024-09-17T16:13:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696959+0000", 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.99e-07 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696962+0000", 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5310000000000002e-06 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089489+0000", 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39252651 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089496+0000", 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.165e-06 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089537+0000", 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0401000000000002e-05 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089543+0000", 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5130000000000002e-06 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089626+0000", 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2973999999999995e-05 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:26973 2.17 2:e9d3e7ef:::benchmark_data_smithi173_55121_object26972:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:13:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:32.894163+0000", 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 99.372014950999997, 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26820020100000003, 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 26973 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894163+0000", 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894163+0000", 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894164+0000", 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894189+0000", 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5344e-05 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894201+0000", 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2116000000000001e-05 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894208+0000", 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.533e-06 2024-09-17T16:13:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894250+0000", 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1740999999999999e-05 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894332+0000", 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2132000000000006e-05 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894415+0000", 2024-09-17T16:13:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3603000000000006e-05 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.894846+0000", 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00043026300000000001 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.015482+0000", 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.120636205 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162286+0000", 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.14680454500000001 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162322+0000", 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5673999999999998e-05 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162363+0000", 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1230000000000003e-05 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:26973 2.17 e121/104 ondisk, result = 0)", 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:32.895491+0000", 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 99.370687516000004, 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26687919500000001, 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.895491+0000", 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.895491+0000", 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.895491+0000", 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7599999999999997e-07 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:32.895494+0000", 2024-09-17T16:13:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7020000000000001e-06 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162227+0000", 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26673340000000001 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162232+0000", 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162278+0000", 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5797000000000002e-05 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162283+0000", 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0710000000000002e-06 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.162370+0000", 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6615999999999999e-05 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:27006 2.7 2:e4bc7e40:::benchmark_data_smithi173_55121_object27005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:33.164406+0000", 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 99.101771693000003, 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31356054900000002, 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 27006 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164406+0000", 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164406+0000", 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164407+0000", 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164434+0000", 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6551999999999999e-05 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164445+0000", 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1683e-05 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164468+0000", 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2708e-05 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164511+0000", 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2923000000000002e-05 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164588+0000", 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6625000000000006e-05 2024-09-17T16:13:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164669+0000", 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1527999999999996e-05 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165334+0000", 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00066428899999999998 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165780+0000", 2024-09-17T16:13:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000446799 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477887+0000", 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31210646199999997 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477933+0000", 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6013999999999997e-05 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477967+0000", 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4167e-05 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:27006 2.7 e121/104 ondisk, result = 0)", 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 99.100419595999995, 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31221570399999998, 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:13:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:13:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2099999999999996e-07 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165764+0000", 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3159999999999998e-06 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477726+0000", 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31196209800000002 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477766+0000", 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0238999999999999e-05 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477875+0000", 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010957499999999999 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477883+0000", 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2860000000000002e-06 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477974+0000", 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1469000000000001e-05 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:94007 2.19 e160/114 2:9e42b13d:::benchmark_data_smithi173_55121_object17518:head v 160'3326, mlcod=160'3323)", 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:49.888089+0000", 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.378089462999998, 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.46654707200000001, 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.888089+0000", 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.888089+0000", 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.888089+0000", 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0699999999999997e-07 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.888094+0000", 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.938e-06 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.353318+0000", 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.46522339899999998 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.353323+0000", 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1800000000000004e-06 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.353396+0000", 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3294000000000007e-05 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.353411+0000", 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4942e-05 2024-09-17T16:13:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.354595+0000", 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011835630000000001 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.354636+0000", 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1149000000000002e-05 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97410 2.7 2:e74465cf:::benchmark_data_smithi173_55121_object20921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.828184+0000", 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 18.437993816999999, 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24855951400000001, 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97410 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.828184+0000", 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.828184+0000", 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.828186+0000", 2024-09-17T16:13:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.373e-06 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.828191+0000", 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8050000000000003e-06 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.828203+0000", 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1107999999999999e-05 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.828208+0000", 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6049999999999997e-06 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.828251+0000", 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3223999999999998e-05 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.828341+0000", 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9874e-05 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.828418+0000", 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6862999999999999e-05 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.829728+0000", 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013094249999999999 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.830204+0000", 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000476095 2024-09-17T16:13:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.076677+0000", 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.246473678 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.076714+0000", 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6415000000000003e-05 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.076744+0000", 2024-09-17T16:13:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0049e-05 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97421 2.14 2:2bd549c5:::benchmark_data_smithi173_55121_object20932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.844120+0000", 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 18.422058349, 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.341556152, 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:13:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97421 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844120+0000", 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844120+0000", 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844122+0000", 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9439999999999999e-06 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844128+0000", 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2029999999999997e-06 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844145+0000", 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7370000000000001e-05 2024-09-17T16:13:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844153+0000", 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7409999999999992e-06 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844213+0000", 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0003000000000001e-05 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844307+0000", 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4271000000000006e-05 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 5,9", 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844381+0000", 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3847e-05 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.845862+0000", 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014811150000000001 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.853013+0000", 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0071510469999999998 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185577+0000", 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33256385799999999 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185630+0000", 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2664000000000003e-05 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185676+0000", 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6088999999999999e-05 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:97421 2.14 e163/117 ondisk, result = 0)", 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.846233+0000", 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 18.419945203000001, 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33944879500000003, 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:13:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.846233+0000", 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.846233+0000", 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.846233+0000", 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6599999999999996e-07 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.846238+0000", 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3970000000000004e-06 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185505+0000", 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33926690199999998 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185511+0000", 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1489999999999999e-06 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185564+0000", 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3371000000000001e-05 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185571+0000", 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1640000000000004e-06 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:13:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:13:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:13:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185682+0000", 2024-09-17T16:13:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000110246 2024-09-17T16:13:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:13:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:13:12.379 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:13:12.430 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:12.453 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:12.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:12.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:12.578 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:12.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:12.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:12.705 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:12.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:12.725 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T16:13:12.807 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:12.834 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:12.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:12.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:12.962 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:12.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:12.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:12.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:12 smithi017 ceph-mon[27571]: 2.18 scrub starts 2024-09-17T16:13:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:12 smithi017 ceph-mon[27571]: 2.18 scrub ok 2024-09-17T16:13:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:12 smithi017 ceph-mon[27571]: pgmap v448: 27 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 2.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s; 2040/121440 objects degraded (1.680%); 4.3 MiB/s, 161 objects/s recovering 2024-09-17T16:13:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1245648726' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:13:13.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:13:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:12 smithi071 ceph-mon[30002]: 2.18 scrub starts 2024-09-17T16:13:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:12 smithi071 ceph-mon[30002]: 2.18 scrub ok 2024-09-17T16:13:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:12 smithi071 ceph-mon[30002]: pgmap v448: 27 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 2.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s; 2040/121440 objects degraded (1.680%); 4.3 MiB/s, 161 objects/s recovering 2024-09-17T16:13:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1245648726' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:13:13.064 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:13.103 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:13.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:13.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:12 smithi173 ceph-mon[30530]: 2.18 scrub starts 2024-09-17T16:13:13.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:12 smithi173 ceph-mon[30530]: 2.18 scrub ok 2024-09-17T16:13:13.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:12 smithi173 ceph-mon[30530]: pgmap v448: 27 pgs: 1 active+clean+scrubbing, 1 peering, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 2.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s; 2040/121440 objects degraded (1.680%); 4.3 MiB/s, 161 objects/s recovering 2024-09-17T16:13:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1245648726' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:13:13.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:13.230 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:13.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:13.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:13.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:13.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:13.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:13.462 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:13.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:13:13.489 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:13.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:13.591 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:13.613 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:13.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:13.715 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:13.742 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:13.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:13.807 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:13.843 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:13.865 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:13.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:13.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:13.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:13 smithi173 ceph-mon[30530]: 2.15 scrub starts 2024-09-17T16:13:13.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:13 smithi173 ceph-mon[30530]: 2.15 scrub ok 2024-09-17T16:13:13.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:13 smithi173 ceph-mon[30530]: 2.14 scrub starts 2024-09-17T16:13:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1245648726' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:13:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:13 smithi173 ceph-mon[30530]: osdmap e178: 12 total, 12 up, 12 in 2024-09-17T16:13:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:13 smithi173 ceph-mon[30530]: 2.14 scrub ok 2024-09-17T16:13:13.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:13.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:13 smithi017 ceph-mon[27571]: 2.15 scrub starts 2024-09-17T16:13:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:13 smithi017 ceph-mon[27571]: 2.15 scrub ok 2024-09-17T16:13:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:13 smithi017 ceph-mon[27571]: 2.14 scrub starts 2024-09-17T16:13:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1245648726' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:13:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:13 smithi017 ceph-mon[27571]: osdmap e178: 12 total, 12 up, 12 in 2024-09-17T16:13:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:13 smithi017 ceph-mon[27571]: 2.14 scrub ok 2024-09-17T16:13:13.991 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:13.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:14.060 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:14.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:13 smithi071 ceph-mon[30002]: 2.15 scrub starts 2024-09-17T16:13:14.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:13 smithi071 ceph-mon[30002]: 2.15 scrub ok 2024-09-17T16:13:14.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:13 smithi071 ceph-mon[30002]: 2.14 scrub starts 2024-09-17T16:13:14.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1245648726' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:13:14.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:13 smithi071 ceph-mon[30002]: osdmap e178: 12 total, 12 up, 12 in 2024-09-17T16:13:14.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:13 smithi071 ceph-mon[30002]: 2.14 scrub ok 2024-09-17T16:13:14.081 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T16:13:14.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:14.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:14.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:14.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:14.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:14.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:14.388 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:14.414 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:14.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:14.516 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:14.539 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:14.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:14.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:14.644 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:14.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:14.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:14.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:14.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:14.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:14.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:14.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:14.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:14.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:14.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:14 smithi017 ceph-mon[27571]: pgmap v450: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 2.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 227 op/s; 2040/119262 objects degraded (1.711%) 2024-09-17T16:13:14.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:14 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2040/121440 objects degraded (1.680%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:15.016 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:15.040 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:15.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:15.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:14 smithi071 ceph-mon[30002]: pgmap v450: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 2.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 227 op/s; 2040/119262 objects degraded (1.711%) 2024-09-17T16:13:15.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:14 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2040/121440 objects degraded (1.680%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:15.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:15.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:14 smithi173 ceph-mon[30530]: pgmap v450: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 2.4 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 227 op/s; 2040/119262 objects degraded (1.711%) 2024-09-17T16:13:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:14 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2040/121440 objects degraded (1.680%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:15.171 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:15.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:15.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:15.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:15.297 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:15.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:15.360 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:13:15.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:15.421 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:15.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:15.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:15.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:15.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:15.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:15.653 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:15.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:15.678 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:15.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:15.743 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T16:13:15.743 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:13:15.780 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:15.808 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:15.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:15.910 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:15.942 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:15.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:15.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1620732600' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:13:15.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1745046501' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:13:16.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:16.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1620732600' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:13:16.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1745046501' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:13:16.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:16.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:16.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1620732600' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:13:16.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1745046501' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:13:16.172 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:16.194 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:16.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:16.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:16.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:16.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:16.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:16.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:16.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:16.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:16.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:16.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:16.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:16.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:16.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:16.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:16.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:16.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:16.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:16.959 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:16.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:16.980 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:16 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:16.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1620732600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:13:16.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1745046501' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:13:16.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:16 smithi017 ceph-mon[27571]: osdmap e179: 12 total, 12 up, 12 in 2024-09-17T16:13:16.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:16 smithi017 ceph-mon[27571]: pgmap v452: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 2.3 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 1606/111027 objects degraded (1.446%) 2024-09-17T16:13:17.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:16 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:17.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1620732600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:13:17.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1745046501' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:13:17.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:16 smithi173 ceph-mon[30530]: osdmap e179: 12 total, 12 up, 12 in 2024-09-17T16:13:17.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:16 smithi173 ceph-mon[30530]: pgmap v452: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 2.3 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 1606/111027 objects degraded (1.446%) 2024-09-17T16:13:17.011 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:13:17.011 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:13:17.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.011 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:119318 2.b e179/155)", 2024-09-17T16:13:17.011 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:17.006731+0000", 2024-09-17T16:13:17.011 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0013039149999999999, 2024-09-17T16:13:17.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00133198, 2024-09-17T16:13:17.011 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006731+0000", 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006731+0000", 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:17.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006732+0000", 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9100000000000003e-07 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006737+0000", 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8389999999999997e-06 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:17.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006766+0000", 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9354000000000001e-05 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006773+0000", 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3300000000000004e-06 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.014 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:119318 2.b e179/155)", 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:17.006929+0000", 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0011058590000000001, 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001173019, 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006929+0000", 2024-09-17T16:13:17.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006929+0000", 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006930+0000", 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8199999999999999e-07 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006934+0000", 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3290000000000004e-06 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006971+0000", 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6860999999999998e-05 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:17.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.006978+0000", 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.392e-06 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:119317 2.b 2:dd2af632:::benchmark_data_smithi173_55121_object42828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:16.997609+0000", 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.010426279, 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010535595, 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:17.018 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 119317 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.997609+0000", 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:17.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.997609+0000", 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.997610+0000", 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4050000000000001e-06 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.997616+0000", 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.896e-06 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.997628+0000", 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1652e-05 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.997633+0000", 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0250000000000002e-06 2024-09-17T16:13:17.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.997696+0000", 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3233999999999998e-05 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.997818+0000", 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012156599999999999 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.997943+0000", 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000125377 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.003105+0000", 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0051616400000000003 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.003166+0000", 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1487000000000003e-05 2024-09-17T16:13:17.023 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:119318 2.b 2:d3d62f8b:::benchmark_data_smithi173_55121_object42829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:16.998321+0000", 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0097141720000000001, 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0098972950000000004, 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:17.024 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 119318 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.998321+0000", 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.998321+0000", 2024-09-17T16:13:17.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.998322+0000", 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7199999999999998e-07 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.998326+0000", 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.498e-06 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.998337+0000", 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0305999999999999e-05 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:16.998344+0000", 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1029999999999996e-06 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:17.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.003207+0000", 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048634560000000004 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.003360+0000", 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015314300000000001 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.003507+0000", 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014705799999999999 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.028 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:119324 2.b 2.d80583eb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:17.004084+0000", 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0039517160000000001, 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041935000000000002, 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:17.029 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 119324 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.004084+0000", 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:17.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.004084+0000", 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.004085+0000", 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1310000000000001e-06 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.004090+0000", 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8679999999999998e-06 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.004101+0000", 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1313e-05 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:17.004108+0000", 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4649999999999997e-06 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:17.032 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:13:17.033 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 5 2024-09-17T16:13:17.033 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:13:17.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:17.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:16 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:17.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1620732600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:13:17.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1745046501' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:13:17.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:16 smithi071 ceph-mon[30002]: osdmap e179: 12 total, 12 up, 12 in 2024-09-17T16:13:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:16 smithi071 ceph-mon[30002]: pgmap v452: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 2.3 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 1606/111027 objects degraded (1.446%) 2024-09-17T16:13:17.094 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:17.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:17.196 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:17.219 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:17.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:17.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:17.336 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-17T16:13:17.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:17.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:17.345 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:17.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:17.469 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:17.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:17.565 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:13:17.566 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:13:17.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:17.597 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:13:17.597 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:13:17.599 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:17.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:17.627 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:13:17.627 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:13:17.656 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:13:17.656 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:13:17.660 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:17.687 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:13:17.687 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:13:17.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:17.715 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:13:17.716 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:13:17.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:17.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:17.746 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:13:17.746 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:13:17.778 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:13:17.779 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:13:17.805 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:13:17.805 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:13:17.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:17.839 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:13:17.839 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:13:17.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1549203478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:17.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:17.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:17.863 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:13:17.863 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:13:17.886 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:13:17.886 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:13:17.958 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:17.980 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:17.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:18.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1549203478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:18.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:18.106 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:18.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:18.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1549203478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:18.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:18.231 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:18.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:18.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:18.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:18.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:18.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:18.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:13:18.471 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:18.495 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:18.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:18.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:18.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:18.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:18.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:13:18.724 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:18.735 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:18.753 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:18.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:18.855 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:18.877 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:18.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:18.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:18.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:18 smithi017 ceph-mon[27571]: pgmap v453: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 71 op/s; 1606/105552 objects degraded (1.522%) 2024-09-17T16:13:19.002 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:19.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:19.014 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:18 smithi173 ceph-mon[30530]: pgmap v453: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 71 op/s; 1606/105552 objects degraded (1.522%) 2024-09-17T16:13:19.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:18 smithi071 ceph-mon[30002]: pgmap v453: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 71 op/s; 1606/105552 objects degraded (1.522%) 2024-09-17T16:13:19.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:19.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:19.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:19.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:19.262 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:19.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:19.285 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:19.285 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:19.293 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.8 to 1 (10000) 2024-09-17T16:13:19.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:19.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:19.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:19.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:19.517 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:19.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:19.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:19.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:19.641 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:19.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:19.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:19.765 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:19.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:19.867 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:19.889 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:19.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:19.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/648776394' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:13:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:19 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1606/105552 objects degraded (1.522%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/648776394' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:13:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:19 smithi017 ceph-mon[27571]: osdmap e180: 12 total, 12 up, 12 in 2024-09-17T16:13:19.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:20.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:20.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:20.051 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-17T16:13:20.060 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/648776394' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:13:20.061 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:19 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1606/105552 objects degraded (1.522%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:20.061 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/648776394' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:13:20.061 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:19 smithi173 ceph-mon[30530]: osdmap e180: 12 total, 12 up, 12 in 2024-09-17T16:13:20.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/648776394' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:13:20.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:19 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1606/105552 objects degraded (1.522%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:20.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/648776394' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:13:20.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:19 smithi071 ceph-mon[30002]: osdmap e180: 12 total, 12 up, 12 in 2024-09-17T16:13:20.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:13:20.121 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:20.122 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:20.146 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:20.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:20.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:20.271 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:20.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:20.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:20.399 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:20.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:20.500 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:20.502 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:20.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:20.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:20.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:20.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:20.659 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:20.743 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:20.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:20.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:20.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:20.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:20.873 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:20 smithi017 ceph-mon[27571]: pgmap v455: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1177/100176 objects degraded (1.175%) 2024-09-17T16:13:20.890 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:20.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:20 smithi173 ceph-mon[30530]: pgmap v455: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1177/100176 objects degraded (1.175%) 2024-09-17T16:13:20.917 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:20.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:21.019 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:21.040 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:21.040 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:21.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:21.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:21.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:20 smithi071 ceph-mon[30002]: pgmap v455: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1177/100176 objects degraded (1.175%) 2024-09-17T16:13:21.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:21.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:21.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:21.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:21.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:13:21.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:21.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:21.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:21.444 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:21.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:21.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:21.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:21.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:21.636 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:21.672 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:21.694 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:21.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:21.765 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.9 to 1 (10000) 2024-09-17T16:13:21.769 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:13:21.796 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:21.817 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:21.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:21.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:13:21.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:21.942 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:21.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:22.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:22.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2813351165' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:22.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:21 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:22.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1216919078' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:13:22.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:22.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:22.114 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2813351165' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:22.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:21 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:22.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1216919078' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:13:22.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2813351165' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:22.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:21 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:22.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1216919078' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:13:22.155 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:22.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:22.203 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:22.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:22.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:22.305 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:22.330 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:22.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:22.432 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:22.456 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:22.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:22.558 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:22.582 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:22.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:22.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:22.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:22.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:22.812 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:22.833 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:22.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:22.935 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:22.963 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:22.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:23.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:22 smithi017 ceph-mon[27571]: pgmap v456: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 1177/90867 objects degraded (1.295%) 2024-09-17T16:13:23.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:22 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:23.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:22 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:23.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1216919078' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:13:23.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:22 smithi017 ceph-mon[27571]: osdmap e181: 12 total, 12 up, 12 in 2024-09-17T16:13:23.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:23.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:22 smithi071 ceph-mon[30002]: pgmap v456: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 1177/90867 objects degraded (1.295%) 2024-09-17T16:13:23.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:22 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:23.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:22 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:23.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1216919078' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:13:23.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:22 smithi071 ceph-mon[30002]: osdmap e181: 12 total, 12 up, 12 in 2024-09-17T16:13:23.093 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:23.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:23.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:22 smithi173 ceph-mon[30530]: pgmap v456: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.8 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 730 op/s; 1177/90867 objects degraded (1.295%) 2024-09-17T16:13:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:22 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:22 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1216919078' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:13:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:22 smithi173 ceph-mon[30530]: osdmap e181: 12 total, 12 up, 12 in 2024-09-17T16:13:23.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:23.226 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:23.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:23.327 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:23.356 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:23.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:23.458 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:23.481 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:23.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:23.583 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:23.606 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:23.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:23.654 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:23.654 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:23.708 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:23.734 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:23.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:23.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:23.862 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:23.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:23.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:23.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:23 smithi173 ceph-mon[30530]: 2.3 scrub starts 2024-09-17T16:13:23.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3253824213' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:23.905 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:13:23.906 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:13:23.906 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:13:23.906 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:13:23.906 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:13:23.906 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:13:23.906 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:13:23.906 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:13:23.906 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:13:23.906 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:13:23.907 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:13:23.907 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:13:23.907 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:13:23.907 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:13:23.907 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:13:23.907 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:13:23.907 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:13:23.907 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:13:23.908 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:13:23.908 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:13:23.908 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:13:23.908 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:13:23.908 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:13:23.908 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:13:23.909 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:23 smithi017 ceph-mon[27571]: 2.3 scrub starts 2024-09-17T16:13:23.909 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3253824213' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:23.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:23.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:23.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:24.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:23 smithi071 ceph-mon[30002]: 2.3 scrub starts 2024-09-17T16:13:24.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3253824213' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:24.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:24.121 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:24.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:24.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:24.247 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:24.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:24.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:24.374 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:24.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:13:24.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:24.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:24.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:24.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:24.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:24.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:24.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:24.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:24.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:24.727 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:24.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:24.769 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:24.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:24.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:24.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:24.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:24.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:24 smithi017 ceph-mon[27571]: pgmap v458: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 422 op/s; 1177/90027 objects degraded (1.307%) 2024-09-17T16:13:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:24 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:13:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:24 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:13:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:24 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1177/90027 objects degraded (1.307%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:24 smithi017 ceph-mon[27571]: osdmap e182: 12 total, 12 up, 12 in 2024-09-17T16:13:25.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:25.025 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:25.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:25.031 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:24 smithi173 ceph-mon[30530]: pgmap v458: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 422 op/s; 1177/90027 objects degraded (1.307%) 2024-09-17T16:13:25.031 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:24 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:13:25.031 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:24 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:13:25.031 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:24 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1177/90027 objects degraded (1.307%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:25.031 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:24 smithi173 ceph-mon[30530]: osdmap e182: 12 total, 12 up, 12 in 2024-09-17T16:13:25.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:24 smithi071 ceph-mon[30002]: pgmap v458: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 422 op/s; 1177/90027 objects degraded (1.307%) 2024-09-17T16:13:25.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:24 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:13:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:24 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:13:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:24 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1177/90027 objects degraded (1.307%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:24 smithi071 ceph-mon[30002]: osdmap e182: 12 total, 12 up, 12 in 2024-09-17T16:13:25.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:13:25.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-17T16:13:25.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:25.148 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:25.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:25.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:25.262 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:13:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:13:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:13:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:13:25.262 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:13:25.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:25.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:25.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:25.396 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:25.403 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:25.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:25.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:25.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:25.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:25.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:25.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:25.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:25.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:25.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:25.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:25.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:25.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:25.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:25.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:25.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:25.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:26.018 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:26.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:26.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:26.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:26.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:13:26.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:26.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:26.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:26.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:26.306 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:26.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:26.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:26.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:26.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:26.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:26.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:26.468 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:26.547 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:26.579 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:26.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:26.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:13:26.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:26.705 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:26.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:26.775 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:13:26.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:26.832 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:26.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:26.882 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:26 smithi173 ceph-mon[30530]: pgmap v460: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.7 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 782/82029 objects degraded (0.953%); 8.0 MiB/s, 127 objects/s recovering 2024-09-17T16:13:26.882 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1781021274' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:26.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:26.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:26.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:26.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:26 smithi017 ceph-mon[27571]: pgmap v460: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.7 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 782/82029 objects degraded (0.953%); 8.0 MiB/s, 127 objects/s recovering 2024-09-17T16:13:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1781021274' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:27.053 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:13526 2.d 2:b276e7f5:::benchmark_data_smithi173_55121_object13525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:12.327471+0000", 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 134.722042316, 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41466789399999998, 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 13526 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.327471+0000", 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.327471+0000", 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.327472+0000", 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3399999999999998e-07 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.327487+0000", 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5523e-05 2024-09-17T16:13:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.327497+0000", 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9960000000000005e-06 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.327502+0000", 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9339999999999997e-06 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.327558+0000", 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6227999999999998e-05 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.327674+0000", 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011592299999999999 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.327810+0000", 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013598500000000001 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.328380+0000", 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056976399999999995 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.328805+0000", 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042534300000000001 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.742025+0000", 2024-09-17T16:13:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41321945500000001 2024-09-17T16:13:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.742101+0000", 2024-09-17T16:13:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6663999999999996e-05 2024-09-17T16:13:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.742139+0000", 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7245e-05 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:14931 2.18 2.b4f18d38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:14.294157+0000", 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 132.75535564, 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.967921951, 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14931 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294157+0000", 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294157+0000", 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294159+0000", 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1039999999999999e-06 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294178+0000", 2024-09-17T16:13:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9079000000000001e-05 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294189+0000", 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0926e-05 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294194+0000", 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4759999999999997e-06 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294244+0000", 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9740000000000001e-05 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294251+0000", 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5290000000000003e-06 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:15.262048+0000", 2024-09-17T16:13:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96779710200000002 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:15.262079+0000", 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0994999999999997e-05 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:14951 2.18 2.cba96138 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:14.611191+0000", 2024-09-17T16:13:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 132.43832176800001, 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65113860700000004, 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14951 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.611191+0000", 2024-09-17T16:13:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.611191+0000", 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.611193+0000", 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.359e-06 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.611212+0000", 2024-09-17T16:13:27.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9831e-05 2024-09-17T16:13:27.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:27.070 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:26 smithi071 ceph-mon[30002]: pgmap v460: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.7 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.16k op/s; 782/82029 objects degraded (0.953%); 8.0 MiB/s, 127 objects/s recovering 2024-09-17T16:13:27.070 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1781021274' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.611224+0000", 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1905e-05 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.611232+0000", 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6029999999999999e-06 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.611280+0000", 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7515000000000002e-05 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.611285+0000", 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6350000000000003e-06 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:15.262321+0000", 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65103587799999996 2024-09-17T16:13:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:15.262330+0000", 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8810000000000002e-06 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:20781 2.8 e116/104 2:13216039:::benchmark_data_smithi173_55121_object20780:head v 116'1122, mlcod=116'1120)", 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:25.316642+0000", 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.732871367, 2024-09-17T16:13:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57580490299999998, 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.316642+0000", 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.316642+0000", 2024-09-17T16:13:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.316642+0000", 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7199999999999999e-07 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.316721+0000", 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9047000000000004e-05 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.891682+0000", 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57496040199999998 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.891693+0000", 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1477e-05 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.891839+0000", 2024-09-17T16:13:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000145792 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.891854+0000", 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5313999999999999e-05 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.892415+0000", 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056041000000000001 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.892447+0000", 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1888999999999999e-05 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:20801 2.18 e116/110 ondisk, result = 0)", 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:25.326806+0000", 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.72270745500001, 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59753896299999998, 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.326806+0000", 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.326806+0000", 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.326806+0000", 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7899999999999998e-07 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.326809+0000", 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1820000000000002e-06 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.924115+0000", 2024-09-17T16:13:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59730530599999998 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.924121+0000", 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0449999999999997e-06 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.924213+0000", 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2443999999999998e-05 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.924223+0000", 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6900000000000004e-06 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.924345+0000", 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121617 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:24386 2.10 e120/111 2:0afcc277:::benchmark_data_smithi173_55121_object24385:head v 120'1130, mlcod=120'1127)", 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.706116+0000", 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 117.343396735, 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38124361400000001, 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706116+0000", 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706116+0000", 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706117+0000", 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706138+0000", 2024-09-17T16:13:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0503e-05 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.086480+0000", 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38034230200000002 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.086493+0000", 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.289e-05 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.086661+0000", 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016800300000000001 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.086701+0000", 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0076000000000001e-05 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087328+0000", 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062703900000000003 2024-09-17T16:13:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087360+0000", 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2079000000000002e-05 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:24388 2.f e120/95 2:f7f1158d:::benchmark_data_smithi173_55121_object24387:head v 120'1500, mlcod=120'1497)", 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.761731+0000", 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 117.28778169900001, 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37769881399999999, 2024-09-17T16:13:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761731+0000", 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761731+0000", 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761732+0000", 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.009e-06 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761755+0000", 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3067000000000001e-05 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.138529+0000", 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37677365200000001 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.138536+0000", 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6490000000000002e-06 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.138597+0000", 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1198000000000006e-05 2024-09-17T16:13:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.138613+0000", 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6240999999999999e-05 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139403+0000", 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00079012099999999997 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139430+0000", 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6877e-05 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25248 2.18 2:1d4c9d7f:::benchmark_data_smithi173_55121_object25247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.813376+0000", 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 116.236137195, 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44896838500000003, 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25248 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.813376+0000", 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.813376+0000", 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.813377+0000", 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9800000000000003e-07 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.813393+0000", 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6155999999999999e-05 2024-09-17T16:13:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.813401+0000", 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2830000000000001e-06 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.813406+0000", 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4009999999999996e-06 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.813451+0000", 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4394999999999998e-05 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.813527+0000", 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6392999999999999e-05 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.813607+0000", 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9478e-05 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.814083+0000", 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047595200000000003 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.917024+0000", 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10294177 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.262284+0000", 2024-09-17T16:13:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34526000800000001 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.262319+0000", 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5000999999999999e-05 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.262344+0000", 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4850000000000001e-05 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:25248 2.18 e121/110 ondisk, result = 0)", 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.814456+0000", 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 116.235056564, 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.447892862, 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.814456+0000", 2024-09-17T16:13:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.814456+0000", 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.814457+0000", 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4799999999999999e-07 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.814459+0000", 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2809999999999998e-06 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.262229+0000", 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44776981799999999 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.262234+0000", 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4669999999999998e-06 2024-09-17T16:13:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.262276+0000", 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2274000000000002e-05 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.262280+0000", 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5469999999999998e-06 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.262349+0000", 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9027000000000002e-05 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:26888 2.15 2:abda253b:::benchmark_data_smithi173_55121_object26887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.296859+0000", 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 114.752654527, 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39129625699999998, 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 26888 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.296859+0000", 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.296859+0000", 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.296860+0000", 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2309999999999999e-06 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.296878+0000", 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8732e-05 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.296890+0000", 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1147e-05 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.296906+0000", 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5909e-05 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.296969+0000", 2024-09-17T16:13:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3047999999999997e-05 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.297092+0000", 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123559 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.297182+0000", 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9370999999999995e-05 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.297743+0000", 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056158300000000005 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.298270+0000", 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000527302 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.688073+0000", 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38980244200000003 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.688107+0000", 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4338999999999998e-05 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.688155+0000", 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7593999999999999e-05 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:26888 2.15 e121/113 ondisk, result = 0)", 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.298018+0000", 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 114.751495272, 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39014603199999998, 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.298018+0000", 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.298018+0000", 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.298018+0000", 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.51e-07 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.298022+0000", 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1889999999999999e-06 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.688023+0000", 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39000165399999998 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.688029+0000", 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9769999999999997e-06 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.688063+0000", 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3717000000000003e-05 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.688069+0000", 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2550000000000003e-06 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.688164+0000", 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4689000000000001e-05 2024-09-17T16:13:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:26993 2.10 e121/111 2:0b00f2f4:::benchmark_data_smithi173_55121_object26992:head v 121'1223, mlcod=121'1220)", 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.983495+0000", 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 114.0660183, 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51444622399999995, 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983495+0000", 2024-09-17T16:13:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983495+0000", 2024-09-17T16:13:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983495+0000", 2024-09-17T16:13:27.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9800000000000003e-07 2024-09-17T16:13:27.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983512+0000", 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6799e-05 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.497003+0000", 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51349118800000004 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.497021+0000", 2024-09-17T16:13:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7326e-05 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.497174+0000", 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015328 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.497213+0000", 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8593000000000001e-05 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.497905+0000", 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069217599999999997 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.497941+0000", 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6164000000000001e-05 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:27009 2.e e121/95 2:7c49ab26:::benchmark_data_smithi173_55121_object27008:head v 121'1611, mlcod=121'1609)", 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:33.169849+0000", 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 113.879663772, 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33100500199999999, 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.169849+0000", 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.169849+0000", 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.169850+0000", 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1800000000000005e-07 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.169933+0000", 2024-09-17T16:13:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2725999999999994e-05 2024-09-17T16:13:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.499858+0000", 2024-09-17T16:13:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.329925262 2024-09-17T16:13:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.499864+0000", 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2879999999999998e-06 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.499939+0000", 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4606999999999999e-05 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.499953+0000", 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4418e-05 2024-09-17T16:13:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.500821+0000", 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086723499999999995 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.500854+0000", 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3748000000000003e-05 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:76468 2.15 e149/113 ondisk, result = 0)", 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:32.182636+0000", 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 54.866877013, 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.74260242600000004, 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.182636+0000", 2024-09-17T16:13:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.182636+0000", 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.182637+0000", 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3499999999999996e-07 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.182640+0000", 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2490000000000001e-06 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.925050+0000", 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.74241045500000002 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.925060+0000", 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2930000000000006e-06 2024-09-17T16:13:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.925141+0000", 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1599000000000003e-05 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.925149+0000", 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6009999999999997e-06 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.925238+0000", 2024-09-17T16:13:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9693999999999999e-05 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:76476 2.d e149/95 ondisk, result = 0)", 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:32.191934+0000", 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 54.857579027, 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59274599100000003, 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:13:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.191934+0000", 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.191934+0000", 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.191935+0000", 2024-09-17T16:13:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2300000000000001e-07 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.191937+0000", 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8380000000000002e-06 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.784549+0000", 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59261121100000003 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.784562+0000", 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2943e-05 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.784665+0000", 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103182 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.784671+0000", 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4500000000000001e-06 2024-09-17T16:13:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.784680+0000", 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8440000000000004e-06 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:76476 2.d e149/95 ondisk, result = 0)", 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:32.192989+0000", 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 54.856524104999998, 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88641944299999997, 2024-09-17T16:13:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.192989+0000", 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.192989+0000", 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.192990+0000", 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5499999999999998e-07 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.192993+0000", 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1540000000000002e-06 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:33.079232+0000", 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88623938400000002 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:33.079239+0000", 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8510000000000001e-06 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:33.079303+0000", 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3652999999999995e-05 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:33.079310+0000", 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0160000000000003e-06 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:33.079408+0000", 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8830000000000001e-05 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:90894 2.0 2:01faad8d:::benchmark_data_smithi173_55121_object14405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:46.572451+0000", 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.477061837000001, 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29430099100000001, 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 90894 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.572451+0000", 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.572451+0000", 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.572452+0000", 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0100000000000003e-07 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.572456+0000", 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8029999999999998e-06 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.572466+0000", 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0074e-05 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.572472+0000", 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7209999999999999e-06 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.572514+0000", 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2284999999999999e-05 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.572606+0000", 2024-09-17T16:13:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1516999999999998e-05 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.572683+0000", 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7118000000000003e-05 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.574632+0000", 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001949201 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.575022+0000", 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039007899999999999 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.866026+0000", 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29100374400000001 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.866083+0000", 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7805e-05 2024-09-17T16:13:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.866752+0000", 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00066874299999999996 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:90894 2.0 e158/155 ondisk, result = 0)", 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:46.574355+0000", 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.475157840999998, 2024-09-17T16:13:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29240214399999997, 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.574355+0000", 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.574355+0000", 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.574356+0000", 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9999999999999997e-07 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.574360+0000", 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4299999999999999e-06 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.865923+0000", 2024-09-17T16:13:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29156285300000001 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.865929+0000", 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6899999999999997e-06 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.866009+0000", 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0536000000000003e-05 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.866018+0000", 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9609999999999994e-06 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.866757+0000", 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073897399999999999 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:94044 2.8 e160/155 2:165f9939:::benchmark_data_smithi173_55121_object17555:head v 160'3435, mlcod=132'1644)", 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:49.925352+0000", 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.124160857, 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35829445399999998, 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.925352+0000", 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.925352+0000", 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.925353+0000", 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.92e-07 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.925358+0000", 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3190000000000002e-06 2024-09-17T16:13:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.281939+0000", 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35658108599999999 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.281961+0000", 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1352000000000002e-05 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.282072+0000", 2024-09-17T16:13:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111075 2024-09-17T16:13:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.282116+0000", 2024-09-17T16:13:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3998999999999997e-05 2024-09-17T16:13:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.283611+0000", 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014952190000000001 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.283647+0000", 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5611999999999997e-05 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:95544 2.f e162/95 2:f3fe689c:::benchmark_data_smithi173_55121_object19055:head v 162'5936, mlcod=162'5934)", 2024-09-17T16:13:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:51.679441+0000", 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.370072483999998, 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30711341800000003, 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:51.679441+0000", 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:51.679441+0000", 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:51.679441+0000", 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4300000000000003e-07 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:51.679446+0000", 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.826e-06 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:51.984096+0000", 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30465023899999999 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:51.984103+0000", 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0650000000000001e-06 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:51.984153+0000", 2024-09-17T16:13:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9499000000000002e-05 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:51.984176+0000", 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3045e-05 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:51.986520+0000", 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023443140000000001 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:51.986554+0000", 2024-09-17T16:13:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3787e-05 2024-09-17T16:13:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:27.159 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:13:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:27.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:27.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:27.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:27.286 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:27.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:27.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:27.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:27.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:27.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:27.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:27.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:27.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:27.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:27.603 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:27.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:27.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:27.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:27.779 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.10 to 1 (10000) 2024-09-17T16:13:27.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:27.811 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:27.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:27.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:27 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1781021274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:13:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:27 smithi173 ceph-mon[30530]: osdmap e183: 12 total, 12 up, 12 in 2024-09-17T16:13:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1316455063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:13:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:27 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:13:27.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:27.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:27.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:27.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:27 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:27.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1781021274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:13:27.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:27 smithi017 ceph-mon[27571]: osdmap e183: 12 total, 12 up, 12 in 2024-09-17T16:13:27.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1316455063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:13:27.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:27 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:13:28.037 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:28.059 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:28.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:28.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:27 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:28.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1781021274' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:13:28.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:27 smithi071 ceph-mon[30002]: osdmap e183: 12 total, 12 up, 12 in 2024-09-17T16:13:28.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1316455063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:13:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:27 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:13:28.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:13:28.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:28.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:28.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:28.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:28.316 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:28.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:28.401 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:28.417 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:28.439 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:28.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:28.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:28.563 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:28.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:28.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:28.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:28.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:28.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:28.812 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:28.812 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:13:28.812 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:13:28.812 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:13:28.812 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:13:28.812 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:13:28.813 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:13:28.814 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:13:28.814 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:13:28.814 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:13:28.814 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:13:28.814 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:13:28.814 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:13:28.814 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:13:28.814 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:13:28.815 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:13:28.815 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:13:28.815 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:13:28.815 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:13:28.815 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:13:28.815 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:13:28.815 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:13:28.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:13:28.816 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:13:28.816 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:13:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:28.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:28.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:28.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:28.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:28 smithi017 ceph-mon[27571]: 2.12 scrub starts 2024-09-17T16:13:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:28 smithi017 ceph-mon[27571]: 2.12 scrub ok 2024-09-17T16:13:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:28 smithi017 ceph-mon[27571]: pgmap v462: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 782/76686 objects degraded (1.020%); 4.1 MiB/s, 65 objects/s recovering 2024-09-17T16:13:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:28 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:13:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:28 smithi017 ceph-mon[27571]: osdmap e184: 12 total, 12 up, 12 in 2024-09-17T16:13:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2871941457' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:29.043 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:29.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:28 smithi071 ceph-mon[30002]: 2.12 scrub starts 2024-09-17T16:13:29.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:28 smithi071 ceph-mon[30002]: 2.12 scrub ok 2024-09-17T16:13:29.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:28 smithi071 ceph-mon[30002]: pgmap v462: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 782/76686 objects degraded (1.020%); 4.1 MiB/s, 65 objects/s recovering 2024-09-17T16:13:29.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:28 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:13:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:28 smithi071 ceph-mon[30002]: osdmap e184: 12 total, 12 up, 12 in 2024-09-17T16:13:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2871941457' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:29.064 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:29.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:29.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:28 smithi173 ceph-mon[30530]: 2.12 scrub starts 2024-09-17T16:13:29.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:28 smithi173 ceph-mon[30530]: 2.12 scrub ok 2024-09-17T16:13:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:28 smithi173 ceph-mon[30530]: pgmap v462: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 787 op/s; 782/76686 objects degraded (1.020%); 4.1 MiB/s, 65 objects/s recovering 2024-09-17T16:13:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:28 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:13:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:28 smithi173 ceph-mon[30530]: osdmap e184: 12 total, 12 up, 12 in 2024-09-17T16:13:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2871941457' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:29.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:29.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:29.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:29.290 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:29.313 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:29.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:29.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:29.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:29.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:29.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:29.496 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:13:29.540 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:29.563 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:29.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:29.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:29.665 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:29.687 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:29.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:29.789 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:29.814 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:29.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:29.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:29.932 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:29.932 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:29.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:29.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:30.025 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-17T16:13:30.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:30.050 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:29 smithi017 ceph-mon[27571]: 2.9 scrub starts 2024-09-17T16:13:30.051 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:29 smithi017 ceph-mon[27571]: 2.9 scrub ok 2024-09-17T16:13:30.051 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:29 smithi017 ceph-mon[27571]: osdmap e185: 12 total, 12 up, 12 in 2024-09-17T16:13:30.051 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:29 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 782/76686 objects degraded (1.020%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:30.051 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:29 smithi017 ceph-mon[27571]: osdmap e186: 12 total, 12 up, 12 in 2024-09-17T16:13:30.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:29 smithi071 ceph-mon[30002]: 2.9 scrub starts 2024-09-17T16:13:30.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:29 smithi071 ceph-mon[30002]: 2.9 scrub ok 2024-09-17T16:13:30.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:29 smithi071 ceph-mon[30002]: osdmap e185: 12 total, 12 up, 12 in 2024-09-17T16:13:30.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:29 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 782/76686 objects degraded (1.020%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:29 smithi071 ceph-mon[30002]: osdmap e186: 12 total, 12 up, 12 in 2024-09-17T16:13:30.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:30.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:30.102 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:30.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:29 smithi173 ceph-mon[30530]: 2.9 scrub starts 2024-09-17T16:13:30.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:29 smithi173 ceph-mon[30530]: 2.9 scrub ok 2024-09-17T16:13:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:29 smithi173 ceph-mon[30530]: osdmap e185: 12 total, 12 up, 12 in 2024-09-17T16:13:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:29 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 782/76686 objects degraded (1.020%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:29 smithi173 ceph-mon[30530]: osdmap e186: 12 total, 12 up, 12 in 2024-09-17T16:13:30.170 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:30.209 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:30.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:30.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:30.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:30.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:30.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:30.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:30.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:30.572 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:30.602 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:30.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:30.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:30.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:30.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:30.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:30.729 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:30.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:13:30.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:30.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:30.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:30.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:30.984 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:30.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:31.056 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:31.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:31.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:31.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:31.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:31.238 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:31.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:31.314 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:13:31.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:31.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:31.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:31.468 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:31.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:31 smithi173 ceph-mon[30530]: pgmap v466: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 297/71982 objects degraded (0.413%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:13:31.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2676858606' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:31.469 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:31 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:31.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:13:31.492 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:31.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:31 smithi071 ceph-mon[30002]: pgmap v466: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 297/71982 objects degraded (0.413%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:13:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2676858606' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:31 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:31.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:31.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:31.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:31.701 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:31 smithi017 ceph-mon[27571]: pgmap v466: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s; 297/71982 objects degraded (0.413%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:13:31.701 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2676858606' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:31.701 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:31 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:31.719 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:31.753 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:31.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:31.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:31.878 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:31.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:31.898 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:31.898 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:31.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:32.005 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:32.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:32.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:32.129 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:32.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:32.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:32.154 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:32.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:32.261 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:32.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:32.315 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.11 to 1 (10000) 2024-09-17T16:13:32.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:32.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:32.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:32.493 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:32.515 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:32.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:32.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:32.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:32 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:13:32.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:32 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:32.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:32 smithi071 ceph-mon[30002]: osdmap e187: 12 total, 12 up, 12 in 2024-09-17T16:13:32.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1053901008' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:13:32.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:32.644 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:32.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:32 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:13:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:32 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:32 smithi173 ceph-mon[30530]: osdmap e187: 12 total, 12 up, 12 in 2024-09-17T16:13:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1053901008' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:13:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:32.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:32.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:32 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:13:32.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:32 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:32.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:32 smithi017 ceph-mon[27571]: osdmap e187: 12 total, 12 up, 12 in 2024-09-17T16:13:32.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1053901008' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:13:32.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:13:32.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:32.772 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:32.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:32.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:32.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:32.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:32.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:13:32.971 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:13:32.999 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:13:33.000 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:13:33.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:33.023 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:13:33.023 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:13:33.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:33.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:33.044 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:13:33.045 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:13:33.068 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:13:33.069 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:13:33.071 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:33.091 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:13:33.091 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:13:33.115 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:13:33.115 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:13:33.133 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:33.142 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:13:33.143 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:13:33.158 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:33.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:33.170 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:13:33.170 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:13:33.199 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:13:33.199 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:13:33.223 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:13:33.223 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:13:33.244 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:13:33.245 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:13:33.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:33.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:33.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:33.385 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:33.415 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:33.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:33.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:33 smithi017 ceph-mon[27571]: pgmap v468: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.3 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314/65634 objects degraded (0.478%); 467/65634 objects misplaced (0.712%) 2024-09-17T16:13:33.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1053901008' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:13:33.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:33 smithi017 ceph-mon[27571]: osdmap e188: 12 total, 12 up, 12 in 2024-09-17T16:13:33.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3948171045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:33.431 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T16:13:33.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:13:33.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:33.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:33.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:33 smithi173 ceph-mon[30530]: pgmap v468: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.3 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314/65634 objects degraded (0.478%); 467/65634 objects misplaced (0.712%) 2024-09-17T16:13:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1053901008' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:13:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:33 smithi173 ceph-mon[30530]: osdmap e188: 12 total, 12 up, 12 in 2024-09-17T16:13:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3948171045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:33.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:33.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:33.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:33.775 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:33.805 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:33.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:33 smithi071 ceph-mon[30002]: pgmap v468: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.3 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314/65634 objects degraded (0.478%); 467/65634 objects misplaced (0.712%) 2024-09-17T16:13:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1053901008' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:13:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:33 smithi071 ceph-mon[30002]: osdmap e188: 12 total, 12 up, 12 in 2024-09-17T16:13:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3948171045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:33.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:33.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:33.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:33.946 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:13:33.946 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:13:33.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:13:33.946 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:13:33.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.946 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:11693 2.7 e102/95 2:eb9c71a1:::benchmark_data_smithi173_55121_object11692:head v 102'755, mlcod=102'752)", 2024-09-17T16:13:33.946 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:10.832712+0000", 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 143.110376602, 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.171520639, 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.832712+0000", 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.832712+0000", 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.832713+0000", 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8999999999999996e-07 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.832794+0000", 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1312999999999999e-05 2024-09-17T16:13:33.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.003701+0000", 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17090686399999999 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.003706+0000", 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2209999999999996e-06 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:33.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.003748+0000", 2024-09-17T16:13:33.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1372000000000003e-05 2024-09-17T16:13:33.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:33.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.003762+0000", 2024-09-17T16:13:33.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.453e-05 2024-09-17T16:13:33.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.004207+0000", 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044488900000000002 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.004233+0000", 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.586e-05 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.951 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:21738 2.7 e117/104 2:e3b5b582:::benchmark_data_smithi173_55121_object21737:head v 117'1138, mlcod=117'1136)", 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.656805+0000", 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 127.28628353400001, 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.168970552, 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.656805+0000", 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.656805+0000", 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.656807+0000", 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1340000000000001e-06 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.656851+0000", 2024-09-17T16:13:33.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4484999999999998e-05 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.824917+0000", 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.168065465 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.824939+0000", 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2339000000000001e-05 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.825069+0000", 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000130307 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.825111+0000", 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1946000000000001e-05 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.825745+0000", 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00063373699999999999 2024-09-17T16:13:33.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.825776+0000", 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1139000000000001e-05 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:21834 2.12 2:4e087699:::benchmark_data_smithi173_55121_object21833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 126.955596978, 2024-09-17T16:13:33.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23867154600000001, 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21834 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987494+0000", 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2919999999999998e-06 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987518+0000", 2024-09-17T16:13:33.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3598e-05 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987534+0000", 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5896000000000002e-05 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987542+0000", 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1829999999999994e-06 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:33.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987592+0000", 2024-09-17T16:13:33.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0033e-05 2024-09-17T16:13:33.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:33.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987671+0000", 2024-09-17T16:13:33.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8993999999999996e-05 2024-09-17T16:13:33.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:13:33.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987759+0000", 2024-09-17T16:13:33.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7751999999999996e-05 2024-09-17T16:13:33.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:33.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988272+0000", 2024-09-17T16:13:33.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051340700000000001 2024-09-17T16:13:33.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:33.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988663+0000", 2024-09-17T16:13:33.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039071199999999998 2024-09-17T16:13:33.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226103+0000", 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23744021000000001 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226137+0000", 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3929e-05 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226164+0000", 2024-09-17T16:13:33.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.654e-05 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:21834 2.12 e117/114 ondisk, result = 0)", 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 126.954487418, 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23756740700000001, 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:13:33.964 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:13:33.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988605+0000", 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0079999999999998e-06 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226056+0000", 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23745111999999999 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226062+0000", 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9739999999999999e-06 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226095+0000", 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2503999999999999e-05 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226100+0000", 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9049999999999996e-06 2024-09-17T16:13:33.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226169+0000", 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9303999999999999e-05 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:25268 2.7 e121/104 2:e432f6fb:::benchmark_data_smithi173_55121_object25267:head v 121'1249, mlcod=121'1247)", 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:31.069057+0000", 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 122.874031676, 2024-09-17T16:13:33.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.163705394, 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.069057+0000", 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.069057+0000", 2024-09-17T16:13:33.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:33.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.069058+0000", 2024-09-17T16:13:33.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5899999999999996e-07 2024-09-17T16:13:33.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:33.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.069175+0000", 2024-09-17T16:13:33.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011732599999999999 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232110+0000", 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16293511299999999 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232117+0000", 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.46e-06 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232162+0000", 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5345e-05 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232181+0000", 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.855e-05 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232732+0000", 2024-09-17T16:13:33.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00055167900000000001 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232763+0000", 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0261999999999999e-05 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:26989 2.12 2:48690473:::benchmark_data_smithi173_55121_object26988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.021873281, 2024-09-17T16:13:33.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22401436399999999, 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 26989 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921217+0000", 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5e-06 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921237+0000", 2024-09-17T16:13:33.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9777000000000001e-05 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921250+0000", 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2704999999999999e-05 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921256+0000", 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4080000000000003e-06 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:33.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921298+0000", 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2169000000000002e-05 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921385+0000", 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6536e-05 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921467+0000", 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2657999999999994e-05 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922015+0000", 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054718699999999998 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922501+0000", 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048639400000000002 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145060+0000", 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.222559429 2024-09-17T16:13:33.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145161+0000", 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100058 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145230+0000", 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9542999999999995e-05 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:26989 2.12 e121/114 ondisk, result = 0)", 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.020611647, 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22276984499999999, 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:13:33.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:33.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:13:33.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:33.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922478+0000", 2024-09-17T16:13:33.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9100000000000004e-07 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922484+0000", 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.144e-06 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.144878+0000", 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.222394432 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:33.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.144910+0000", 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1365999999999998e-05 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145033+0000", 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122717 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145050+0000", 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.753e-05 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145247+0000", 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019706500000000001 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 136/134 [PushReplyOp(2:482f57f9:::benchmark_data_smithi173_55121_object24323:head)])", 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 91.779125313999998, 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.229086917, 2024-09-17T16:13:33.984 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:33.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0200000000000002e-07 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163968+0000", 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9369999999999997e-06 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392756+0000", 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22878752399999999 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392762+0000", 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0959999999999997e-06 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392828+0000", 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5746000000000004e-05 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.393051+0000", 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000223012 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:47735 2.7 e142/104 2:e2cd41e4:::benchmark_data_smithi173_55121_object47734:head v 142'1954, mlcod=142'1952)", 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:10.497154+0000", 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 83.445934885, 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16541420900000001, 2024-09-17T16:13:33.988 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.497154+0000", 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.497154+0000", 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.497156+0000", 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6670000000000001e-06 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.497194+0000", 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8078999999999999e-05 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.661482+0000", 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16428769600000001 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.661495+0000", 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2989999999999999e-05 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.661590+0000", 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5414000000000005e-05 2024-09-17T16:13:33.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.661624+0000", 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4266e-05 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.662538+0000", 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091425999999999996 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:33.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.662568+0000", 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9836999999999999e-05 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:61344 2.0 2:04323bed:::benchmark_data_smithi173_55121_object61343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:20.465911+0000", 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.477178330000001, 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15215703699999999, 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:33.993 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 61344 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.465911+0000", 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.465911+0000", 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.465913+0000", 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5940000000000001e-06 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:33.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.465946+0000", 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2670999999999999e-05 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.465966+0000", 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0420000000000001e-05 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.465974+0000", 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8339999999999999e-06 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.466041+0000", 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6604000000000001e-05 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.466190+0000", 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014962100000000001 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9", 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.466350+0000", 2024-09-17T16:13:33.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015918700000000001 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.467352+0000", 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001002281 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.617911+0000", 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15055896499999999 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.617974+0000", 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3513e-05 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:20.618068+0000", 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3346999999999999e-05 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:33.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:79589 2.0 2:00bb2590:::benchmark_data_smithi173_55121_object3100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:35.115215+0000", 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.827874014999999, 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20227213599999999, 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79589 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:34.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115215+0000", 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115215+0000", 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115216+0000", 2024-09-17T16:13:34.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.24e-06 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115221+0000", 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4989999999999996e-06 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115232+0000", 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.166e-05 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115239+0000", 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6069999999999996e-06 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115291+0000", 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2428e-05 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115411+0000", 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120098 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9", 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115547+0000", 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000135368 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.116715+0000", 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011682369999999999 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:34.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.317399+0000", 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20068405 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.317433+0000", 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4178999999999999e-05 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.317487+0000", 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3770000000000002e-05 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.005 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 157/155 [PushReplyOp(2:4c436b3d:::benchmark_data_smithi173_55121_object46284:head)])", 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:43.901089+0000", 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 50.042000244, 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19615692600000001, 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:43.901089+0000", 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:34.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:43.901089+0000", 2024-09-17T16:13:34.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:34.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:43.901090+0000", 2024-09-17T16:13:34.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5300000000000002e-07 2024-09-17T16:13:34.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:34.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:43.901095+0000", 2024-09-17T16:13:34.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4929999999999998e-06 2024-09-17T16:13:34.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.096430+0000", 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.195335068 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.096434+0000", 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2169999999999996e-06 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:34.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.096463+0000", 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8495000000000001e-05 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.097246+0000", 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0007827 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 158/155 [PushReplyOp(2:499d4037:::benchmark_data_smithi173_55121_object46772:head)])", 2024-09-17T16:13:34.010 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:44.315863+0000", 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 49.627225824999996, 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16739532600000001, 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.315863+0000", 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:34.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.315863+0000", 2024-09-17T16:13:34.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:34.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.315864+0000", 2024-09-17T16:13:34.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9999999999999997e-07 2024-09-17T16:13:34.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.315868+0000", 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6190000000000002e-06 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.482487+0000", 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16661815599999999 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.482491+0000", 2024-09-17T16:13:34.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2749999999999997e-06 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.482520+0000", 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9130999999999999e-05 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.483258+0000", 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073854499999999996 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.014 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:91009 2.7 e158/104 2:e0d66b6b:::benchmark_data_smithi173_55121_object14520:head v 158'3309, mlcod=158'3305)", 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:46.779887+0000", 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 47.163202372000001, 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20306453699999999, 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:34.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.779887+0000", 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.779887+0000", 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.779887+0000", 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9599999999999999e-07 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.779892+0000", 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5249999999999999e-06 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.981252+0000", 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20136042900000001 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:34.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.981260+0000", 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.824e-06 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.981325+0000", 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5195000000000002e-05 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.981351+0000", 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5935000000000002e-05 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.982913+0000", 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015619379999999999 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.982951+0000", 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8095e-05 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.019 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:93035 2.12 2:4eb811d6:::benchmark_data_smithi173_55121_object16546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.696790+0000", 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 45.246299006000001, 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22142989699999999, 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93035 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696790+0000", 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696790+0000", 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:34.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696792+0000", 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.592e-06 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696796+0000", 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4859999999999999e-06 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696806+0000", 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0261999999999999e-05 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696812+0000", 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4960000000000004e-06 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696851+0000", 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9588000000000001e-05 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696952+0000", 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100528 2024-09-17T16:13:34.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.697033+0000", 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1314000000000001e-05 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.698646+0000", 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016126459999999999 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.698655+0000", 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6290000000000005e-06 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917751+0000", 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21909500900000001 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917859+0000", 2024-09-17T16:13:34.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108412 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.918220+0000", 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000360935 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:93035 2.12 e158/155 ondisk, result = 0)", 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.699652+0000", 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 45.243436696000003, 2024-09-17T16:13:34.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21857501500000001, 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699652+0000", 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699652+0000", 2024-09-17T16:13:34.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699653+0000", 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5600000000000005e-07 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699657+0000", 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.092e-06 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917601+0000", 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21794412199999999 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917615+0000", 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3397000000000001e-05 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917723+0000", 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108371 2024-09-17T16:13:34.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917739+0000", 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6042999999999999e-05 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.918227+0000", 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048833399999999999 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 158/155 [PushReplyOp(2:4d5499b6:::benchmark_data_smithi173_55121_object58972:head)])", 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 45.181413786999997, 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46593107700000003, 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761676+0000", 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1900000000000002e-07 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:34.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761680+0000", 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1710000000000004e-06 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226484+0000", 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46480413500000001 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226502+0000", 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7895999999999999e-05 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226671+0000", 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016863300000000001 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.227606+0000", 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093562300000000001 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.034 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:97116 2.12 2:4e245c3d:::benchmark_data_smithi173_55121_object20627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:53.535158+0000", 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.407931013000002, 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15195112899999999, 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 97116 2024-09-17T16:13:34.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.535158+0000", 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.535158+0000", 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:34.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.535159+0000", 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.099e-06 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.535163+0000", 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8319999999999999e-06 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.535173+0000", 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8030000000000008e-06 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:34.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.535178+0000", 2024-09-17T16:13:34.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0560000000000004e-06 2024-09-17T16:13:34.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:34.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.535219+0000", 2024-09-17T16:13:34.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1369999999999999e-05 2024-09-17T16:13:34.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.535298+0000", 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8484000000000003e-05 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.535381+0000", 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3249000000000002e-05 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.536764+0000", 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001382791 2024-09-17T16:13:34.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.538177+0000", 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014137349999999999 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.686964+0000", 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14878628999999999 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:34.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.687002+0000", 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8760000000000002e-05 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.687109+0000", 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010666 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(2.12 e163,155 2:4c1d21f9:::benchmark_data_smithi173_55121_object3753:head,150'2969)", 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:53.886957+0000", 2024-09-17T16:13:34.041 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.056131772999997, 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16653584900000001, 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.886957+0000", 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:34.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.886957+0000", 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.886958+0000", 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2300000000000001e-07 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.886962+0000", 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1130000000000003e-06 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:54.053273+0000", 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16631127800000001 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:54.053282+0000", 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5930000000000006e-06 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:54.053330+0000", 2024-09-17T16:13:34.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7741999999999998e-05 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:54.053493+0000", 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016349999999999999 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:117822 2.12 2:4d80328a:::benchmark_data_smithi173_55121_object41333:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:15.355549+0000", 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 18.587540322999999, 2024-09-17T16:13:34.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17288575, 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 117822 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355549+0000", 2024-09-17T16:13:34.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355549+0000", 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355550+0000", 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.691e-06 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:13:34.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355556+0000", 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0519999999999998e-06 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355569+0000", 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2748000000000001e-05 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355577+0000", 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9189999999999999e-06 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355640+0000", 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3325999999999996e-05 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355767+0000", 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012693900000000001 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355879+0000", 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011152100000000001 2024-09-17T16:13:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.357392+0000", 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015128850000000001 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.357507+0000", 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115241 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.528162+0000", 2024-09-17T16:13:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.170655316 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.528385+0000", 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000222463 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.528434+0000", 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9648999999999998e-05 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:13:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:13:34.052 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:13:34.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:34.060 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:13:34.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:34.076 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:34.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:34.177 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:34.216 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:34.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:34.255 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:34.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:34.329 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:34 smithi071 ceph-mon[30002]: 2.2 scrub starts 2024-09-17T16:13:34.346 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:34.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:34.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:34.473 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:34.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:34.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:34.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:34 smithi017 ceph-mon[27571]: 2.2 scrub starts 2024-09-17T16:13:34.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:34 smithi017 ceph-mon[27571]: 2.2 scrub ok 2024-09-17T16:13:34.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:13:34.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:13:34.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:34 smithi017 ceph-mon[27571]: osdmap e189: 12 total, 12 up, 12 in 2024-09-17T16:13:34.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:34 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 314/65634 objects degraded (0.478%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:13:34.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:34 smithi017 ceph-mon[27571]: osdmap e190: 12 total, 12 up, 12 in 2024-09-17T16:13:34.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:34.604 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:34.608 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:34 smithi071 ceph-mon[30002]: 2.2 scrub ok 2024-09-17T16:13:34.608 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:13:34.608 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:13:34.608 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:34 smithi071 ceph-mon[30002]: osdmap e189: 12 total, 12 up, 12 in 2024-09-17T16:13:34.608 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:34 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 314/65634 objects degraded (0.478%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:13:34.608 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:34 smithi071 ceph-mon[30002]: osdmap e190: 12 total, 12 up, 12 in 2024-09-17T16:13:34.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:34 smithi173 ceph-mon[30530]: 2.2 scrub starts 2024-09-17T16:13:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:34 smithi173 ceph-mon[30530]: 2.2 scrub ok 2024-09-17T16:13:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:13:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:13:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:34 smithi173 ceph-mon[30530]: osdmap e189: 12 total, 12 up, 12 in 2024-09-17T16:13:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:34 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 314/65634 objects degraded (0.478%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:13:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:34 smithi173 ceph-mon[30530]: osdmap e190: 12 total, 12 up, 12 in 2024-09-17T16:13:34.706 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:34.736 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:34.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:34.838 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:34.862 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:34.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:34.888 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:34.963 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:34.993 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:34.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:34.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:35.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:13:35.094 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:13:35.118 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:13:35.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:35.120 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:35.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:35.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:35.244 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:35.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:35.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:35.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:35.374 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:35.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:35.379 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:35.476 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:35.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:35.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:35.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:35 smithi017 ceph-mon[27571]: pgmap v471: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.3 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314/64941 objects degraded (0.484%); 467/64941 objects misplaced (0.719%) 2024-09-17T16:13:35.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:35 smithi017 ceph-mon[27571]: 2.0 scrub starts 2024-09-17T16:13:35.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:35 smithi017 ceph-mon[27571]: 2.0 scrub ok 2024-09-17T16:13:35.598 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:35.621 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:35.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:35.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:35.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:35 smithi173 ceph-mon[30530]: pgmap v471: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.3 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314/64941 objects degraded (0.484%); 467/64941 objects misplaced (0.719%) 2024-09-17T16:13:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:35 smithi173 ceph-mon[30530]: 2.0 scrub starts 2024-09-17T16:13:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:35 smithi173 ceph-mon[30530]: 2.0 scrub ok 2024-09-17T16:13:35.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:35.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:35.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:35.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:35 smithi071 ceph-mon[30002]: pgmap v471: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 25 active+clean; 1.3 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314/64941 objects degraded (0.484%); 467/64941 objects misplaced (0.719%) 2024-09-17T16:13:35.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:35 smithi071 ceph-mon[30002]: 2.0 scrub starts 2024-09-17T16:13:35.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:35 smithi071 ceph-mon[30002]: 2.0 scrub ok 2024-09-17T16:13:35.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:35.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:35.873 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:35.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:35.876 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:13:35.877 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:13:35.877 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:13:35.877 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:13:35.928 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-17T16:13:34.298398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1419084344,"num_objects":21647,"num_object_clones":0,"num_object_copies":64941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":467,"num_objects_unfound":0,"num_objects_dirty":21647,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":96379,"num_write_kb":3023367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5366,"num_bytes_recovered":317592096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139947,"ondisk_log_size":139947,"up":81,"acting":79,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5213360,"kb_used_data":4297008,"kb_used_omap":58,"kb_used_meta":915973,"kb_avail":1119728464,"statfs":{"total":1151940427776,"available":1146601947136,"internally_reserved":0,"allocated":4400136192,"data_stored":4394570723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":937956515},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-153812992,"num_objects":-2347,"num_object_clones":0,"num_object_copies":-7041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":467,"num_objects_unfound":0,"num_objects_dirty":-2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-5964,"num_write_kb":-311616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4409,"num_bytes_recovered":-253558784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.541038"},"pg_stats":[{"pgid":"2.19","version":"186'4472","reported_seq":6124,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:31.167028+0000","last_change":"2024-09-17T16:13:08.775898+0000","last_active":"2024-09-17T16:13:31.167028+0000","last_peered":"2024-09-17T16:13:31.167028+0000","last_clean":"2024-09-17T16:13:31.167028+0000","last_became_active":"2024-09-17T16:11:24.264470+0000","last_became_peered":"2024-09-17T16:11:24.264470+0000","last_unstale":"2024-09-17T16:13:31.167028+0000","last_undegraded":"2024-09-17T16:13:31.167028+0000","last_fullsized":"2024-09-17T16:13:31.167028+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"175'3870","last_scrub_stamp":"2024-09-17T16:13:08.775814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:08.775814+0000","objects_scrubbed":1265,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:30.625192+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42729472,"num_objects":652,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2903,"num_write_kb":77952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"187'4696","reported_seq":8998,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-17T16:13:31.325063+0000","last_change":"2024-09-17T16:13:11.523551+0000","last_active":"2024-09-17T16:13:31.325063+0000","last_peered":"2024-09-17T16:13:31.325063+0000","last_clean":"2024-09-17T16:13:31.325063+0000","last_became_active":"2024-09-17T16:13:10.323458+0000","last_became_peered":"2024-09-17T16:13:10.323458+0000","last_unstale":"2024-09-17T16:13:31.325063+0000","last_undegraded":"2024-09-17T16:13:31.325063+0000","last_fullsized":"2024-09-17T16:13:31.325063+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"177'4073","last_scrub_stamp":"2024-09-17T16:13:11.523023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:11.523023+0000","objects_scrubbed":1267,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:25.569087+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":631,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"186'4522","reported_seq":4755,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:30.494579+0000","last_change":"2024-09-17T16:12:16.704276+0000","last_active":"2024-09-17T16:13:30.494579+0000","last_peered":"2024-09-17T16:13:30.494579+0000","last_clean":"2024-09-17T16:13:30.494579+0000","last_became_active":"2024-09-17T16:11:13.358625+0000","last_became_peered":"2024-09-17T16:11:13.358625+0000","last_unstale":"2024-09-17T16:13:30.494579+0000","last_undegraded":"2024-09-17T16:13:30.494579+0000","last_fullsized":"2024-09-17T16:13:30.494579+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":5,"last_scrub":"145'2177","last_scrub_stamp":"2024-09-17T16:12:16.704167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:16.704167+0000","objects_scrubbed":1705,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:25.584490+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45481984,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2357,"num_write_kb":43072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"186'4371","reported_seq":6434,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:30.420023+0000","last_change":"2024-09-17T16:11:55.554301+0000","last_active":"2024-09-17T16:13:30.420023+0000","last_peered":"2024-09-17T16:13:30.420023+0000","last_clean":"2024-09-17T16:13:30.420023+0000","last_became_active":"2024-09-17T16:11:25.266055+0000","last_became_peered":"2024-09-17T16:11:25.266055+0000","last_unstale":"2024-09-17T16:13:30.420023+0000","last_undegraded":"2024-09-17T16:13:30.420023+0000","last_fullsized":"2024-09-17T16:13:30.420023+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"132'1495","last_scrub_stamp":"2024-09-17T16:11:55.554231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:55.554231+0000","objects_scrubbed":1100,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:24.678149+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43778048,"num_objects":668,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2876,"num_write_kb":78208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"186'4539","reported_seq":6650,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-17T16:13:31.313184+0000","last_change":"2024-09-17T16:13:12.481602+0000","last_active":"2024-09-17T16:13:31.313184+0000","last_peered":"2024-09-17T16:13:31.313184+0000","last_clean":"2024-09-17T16:13:31.313184+0000","last_became_active":"2024-09-17T16:11:23.302908+0000","last_became_peered":"2024-09-17T16:11:23.302908+0000","last_unstale":"2024-09-17T16:13:31.313184+0000","last_undegraded":"2024-09-17T16:13:31.313184+0000","last_fullsized":"2024-09-17T16:13:31.313184+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"177'4029","last_scrub_stamp":"2024-09-17T16:13:12.481403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:12.481403+0000","objects_scrubbed":1189,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:39.732490+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44433408,"num_objects":678,"num_object_clones":0,"num_object_copies":2034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":83776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"186'4592","reported_seq":6850,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:30.510022+0000","last_change":"2024-09-17T16:13:12.834609+0000","last_active":"2024-09-17T16:13:30.510022+0000","last_peered":"2024-09-17T16:13:30.510022+0000","last_clean":"2024-09-17T16:13:30.510022+0000","last_became_active":"2024-09-17T16:11:27.290863+0000","last_became_peered":"2024-09-17T16:11:27.290863+0000","last_unstale":"2024-09-17T16:13:30.510022+0000","last_undegraded":"2024-09-17T16:13:30.510022+0000","last_fullsized":"2024-09-17T16:13:30.510022+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":5,"last_scrub":"178'4116","last_scrub_stamp":"2024-09-17T16:13:12.834492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:12.834492+0000","objects_scrubbed":1189,"log_size":4592,"log_dups_size":0,"ondisk_log_size":4592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:36.962652+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46530560,"num_objects":710,"num_object_clones":0,"num_object_copies":2130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2931,"num_write_kb":77376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"186'4409","reported_seq":6409,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-17T16:13:31.313238+0000","last_change":"2024-09-17T16:11:53.132205+0000","last_active":"2024-09-17T16:13:31.313238+0000","last_peered":"2024-09-17T16:13:31.313238+0000","last_clean":"2024-09-17T16:13:31.313238+0000","last_became_active":"2024-09-17T16:11:22.310367+0000","last_became_peered":"2024-09-17T16:11:22.310367+0000","last_unstale":"2024-09-17T16:13:31.313238+0000","last_undegraded":"2024-09-17T16:13:31.313238+0000","last_fullsized":"2024-09-17T16:13:31.313238+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"132'1415","last_scrub_stamp":"2024-09-17T16:11:53.132100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:53.132100+0000","objects_scrubbed":973,"log_size":4409,"log_dups_size":0,"ondisk_log_size":4409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:24.400146+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43319296,"num_objects":661,"num_object_clones":0,"num_object_copies":1983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3000,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"186'4603","reported_seq":12045,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:30.656702+0000","last_change":"2024-09-17T16:13:27.714585+0000","last_active":"2024-09-17T16:13:30.656702+0000","last_peered":"2024-09-17T16:13:30.656702+0000","last_clean":"2024-09-17T16:13:30.656702+0000","last_became_active":"2024-09-17T16:12:56.327408+0000","last_became_peered":"2024-09-17T16:12:56.327408+0000","last_unstale":"2024-09-17T16:13:30.656702+0000","last_undegraded":"2024-09-17T16:13:30.656702+0000","last_fullsized":"2024-09-17T16:13:30.656702+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'4560","last_scrub_stamp":"2024-09-17T16:13:27.714377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:27.714377+0000","objects_scrubbed":720,"log_size":4603,"log_dups_size":0,"ondisk_log_size":4603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:40.853484+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44236800,"num_objects":675,"num_object_clones":0,"num_object_copies":2025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,7],"acting":[10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"186'4552","reported_seq":6832,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:30.522710+0000","last_change":"2024-09-17T16:12:45.621594+0000","last_active":"2024-09-17T16:13:30.522710+0000","last_peered":"2024-09-17T16:13:30.522710+0000","last_clean":"2024-09-17T16:13:30.522710+0000","last_became_active":"2024-09-17T16:11:30.264463+0000","last_became_peered":"2024-09-17T16:11:30.264463+0000","last_unstale":"2024-09-17T16:13:30.522710+0000","last_undegraded":"2024-09-17T16:13:30.522710+0000","last_fullsized":"2024-09-17T16:13:30.522710+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":5,"last_scrub":"158'3284","last_scrub_stamp":"2024-09-17T16:12:45.621470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:45.621470+0000","objects_scrubbed":1937,"log_size":4552,"log_dups_size":0,"ondisk_log_size":4552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:00.545203+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43712512,"num_objects":667,"num_object_clones":0,"num_object_copies":2001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3319,"num_write_kb":101440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"185'4442","reported_seq":5982,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:29.298783+0000","last_change":"2024-09-17T16:12:22.182841+0000","last_active":"2024-09-17T16:13:29.298783+0000","last_peered":"2024-09-17T16:13:29.298783+0000","last_clean":"2024-09-17T16:13:29.298783+0000","last_became_active":"2024-09-17T16:11:21.266149+0000","last_became_peered":"2024-09-17T16:11:21.266149+0000","last_unstale":"2024-09-17T16:13:29.298783+0000","last_undegraded":"2024-09-17T16:13:29.298783+0000","last_fullsized":"2024-09-17T16:13:29.298783+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'2377","last_scrub_stamp":"2024-09-17T16:12:22.182710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:22.182710+0000","objects_scrubbed":1944,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:34.798898+0000","scrub_duration":332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49348608,"num_objects":753,"num_object_clones":0,"num_object_copies":2259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2071,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"185'8117","reported_seq":8314,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:29.298810+0000","last_change":"2024-09-17T16:10:54.283333+0000","last_active":"2024-09-17T16:13:29.298810+0000","last_peered":"2024-09-17T16:13:29.298810+0000","last_clean":"2024-09-17T16:13:29.298810+0000","last_became_active":"2024-09-17T16:10:54.282931+0000","last_became_peered":"2024-09-17T16:10:54.282931+0000","last_unstale":"2024-09-17T16:13:29.298810+0000","last_undegraded":"2024-09-17T16:13:29.298810+0000","last_fullsized":"2024-09-17T16:13:29.298810+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":8117,"log_dups_size":0,"ondisk_log_size":8117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:34.852831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93650944,"num_objects":1429,"num_object_clones":0,"num_object_copies":4287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":305472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"187'8180","reported_seq":8411,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-17T16:13:31.595447+0000","last_change":"2024-09-17T16:12:01.681365+0000","last_active":"2024-09-17T16:13:31.595447+0000","last_peered":"2024-09-17T16:13:31.595447+0000","last_clean":"2024-09-17T16:13:31.595447+0000","last_became_active":"2024-09-17T16:10:54.283208+0000","last_became_peered":"2024-09-17T16:10:54.283208+0000","last_unstale":"2024-09-17T16:13:31.595447+0000","last_undegraded":"2024-09-17T16:13:31.595447+0000","last_fullsized":"2024-09-17T16:13:31.595447+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'2642","last_scrub_stamp":"2024-09-17T16:12:01.681123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:01.681123+0000","objects_scrubbed":2614,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:17.223777+0000","scrub_duration":684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88342528,"num_objects":1348,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8180,"num_write_kb":304896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"187'8205","reported_seq":8449,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-17T16:13:31.313335+0000","last_change":"2024-09-17T16:12:18.766646+0000","last_active":"2024-09-17T16:13:31.313335+0000","last_peered":"2024-09-17T16:13:31.313335+0000","last_clean":"2024-09-17T16:13:31.313335+0000","last_became_active":"2024-09-17T16:10:54.283016+0000","last_became_peered":"2024-09-17T16:10:54.283016+0000","last_unstale":"2024-09-17T16:13:31.313335+0000","last_undegraded":"2024-09-17T16:13:31.313335+0000","last_fullsized":"2024-09-17T16:13:31.313335+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3627","last_scrub_stamp":"2024-09-17T16:12:18.766492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:18.766492+0000","objects_scrubbed":3600,"log_size":8205,"log_dups_size":0,"ondisk_log_size":8205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:46.143371+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86507544,"num_objects":1321,"num_object_clones":0,"num_object_copies":3963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8205,"num_write_kb":304769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"185'8087","reported_seq":8281,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T16:13:29.043387+0000","last_change":"2024-09-17T16:10:54.282319+0000","last_active":"2024-09-17T16:13:29.043387+0000","last_peered":"2024-09-17T16:13:29.043387+0000","last_clean":"2024-09-17T16:13:29.043387+0000","last_became_active":"2024-09-17T16:10:54.282069+0000","last_became_peered":"2024-09-17T16:10:54.282069+0000","last_unstale":"2024-09-17T16:13:29.043387+0000","last_undegraded":"2024-09-17T16:13:29.043387+0000","last_fullsized":"2024-09-17T16:13:29.043387+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":8087,"log_dups_size":0,"ondisk_log_size":8087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88539136,"num_objects":1351,"num_object_clones":0,"num_object_copies":4053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8087,"num_write_kb":302016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":229,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-17T16:13:32.316237+0000","last_change":"2024-09-17T16:12:39.292150+0000","last_active":"2024-09-17T16:13:32.316237+0000","last_peered":"2024-09-17T16:13:32.316237+0000","last_clean":"2024-09-17T16:13:32.316237+0000","last_became_active":"2024-09-17T16:12:39.286388+0000","last_became_peered":"2024-09-17T16:12:39.286388+0000","last_unstale":"2024-09-17T16:13:32.316237+0000","last_undegraded":"2024-09-17T16:13:32.316237+0000","last_fullsized":"2024-09-17T16:13:32.316237+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:11:04.478242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:11:04.478242+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"186'4600","reported_seq":7828,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:30.322805+0000","last_change":"2024-09-17T16:13:22.680448+0000","last_active":"2024-09-17T16:13:30.322805+0000","last_peered":"2024-09-17T16:13:30.322805+0000","last_clean":"2024-09-17T16:13:30.322805+0000","last_became_active":"2024-09-17T16:12:52.302562+0000","last_became_peered":"2024-09-17T16:12:52.302562+0000","last_unstale":"2024-09-17T16:13:30.322805+0000","last_undegraded":"2024-09-17T16:13:30.322805+0000","last_fullsized":"2024-09-17T16:13:30.322805+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":4600,"log_dups_size":0,"ondisk_log_size":4600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:13:52.659598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43581440,"num_objects":665,"num_object_clones":0,"num_object_copies":1995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4179,"num_write_kb":155008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1497,"num_bytes_recovered":80609280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"188'4479","reported_seq":7594,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-17T16:13:32.828608+0000","last_change":"2024-09-17T16:13:03.748291+0000","last_active":"2024-09-17T16:13:32.828608+0000","last_peered":"2024-09-17T16:13:32.828608+0000","last_clean":"2024-09-17T16:13:32.828608+0000","last_became_active":"2024-09-17T16:13:03.359631+0000","last_became_peered":"2024-09-17T16:13:03.359631+0000","last_unstale":"2024-09-17T16:13:32.828608+0000","last_undegraded":"2024-09-17T16:13:32.828608+0000","last_fullsized":"2024-09-17T16:13:32.828608+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:13:33.701447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36864000,"num_objects":562,"num_object_clones":0,"num_object_copies":1686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4068,"num_write_kb":148192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"187'4599","reported_seq":4839,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-17T16:13:31.638538+0000","last_change":"2024-09-17T16:12:16.414710+0000","last_active":"2024-09-17T16:13:31.638538+0000","last_peered":"2024-09-17T16:13:31.638538+0000","last_clean":"2024-09-17T16:13:31.638538+0000","last_became_active":"2024-09-17T16:11:13.356367+0000","last_became_peered":"2024-09-17T16:11:13.356367+0000","last_unstale":"2024-09-17T16:13:31.638538+0000","last_undegraded":"2024-09-17T16:13:31.638538+0000","last_fullsized":"2024-09-17T16:13:31.638538+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2190","last_scrub_stamp":"2024-09-17T16:12:16.414644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:16.414644+0000","objects_scrubbed":1711,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:20.196085+0000","scrub_duration":347,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45154304,"num_objects":689,"num_object_clones":0,"num_object_copies":2067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2418,"num_write_kb":44672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"186'4529","reported_seq":4765,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:30.397560+0000","last_change":"2024-09-17T16:12:15.918081+0000","last_active":"2024-09-17T16:13:30.397560+0000","last_peered":"2024-09-17T16:13:30.397560+0000","last_clean":"2024-09-17T16:13:30.397560+0000","last_became_active":"2024-09-17T16:11:13.352761+0000","last_became_peered":"2024-09-17T16:11:13.352761+0000","last_unstale":"2024-09-17T16:13:30.397560+0000","last_undegraded":"2024-09-17T16:13:30.397560+0000","last_fullsized":"2024-09-17T16:13:30.397560+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2150","last_scrub_stamp":"2024-09-17T16:12:15.917968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:15.917968+0000","objects_scrubbed":1721,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:30.932205+0000","scrub_duration":377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42991616,"num_objects":656,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2389,"num_write_kb":42368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"187'4580","reported_seq":6621,"reported_epoch":187,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-17T16:13:31.626507+0000","last_change":"2024-09-17T16:13:29.729752+0000","last_active":"2024-09-17T16:13:31.626507+0000","last_peered":"2024-09-17T16:13:31.626507+0000","last_clean":"2024-09-17T16:13:27.758735+0000","last_became_active":"2024-09-17T16:13:29.693012+0000","last_became_peered":"2024-09-17T16:13:29.693012+0000","last_unstale":"2024-09-17T16:13:31.626507+0000","last_undegraded":"2024-09-17T16:13:29.306887+0000","last_fullsized":"2024-09-17T16:13:29.296055+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'1233","last_scrub_stamp":"2024-09-17T16:11:30.184562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:30.184562+0000","objects_scrubbed":777,"log_size":4580,"log_dups_size":0,"ondisk_log_size":4580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45481984,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":467,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3347,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":17760256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"0,1","objects":210},{"shards":"0,1,2","objects":467}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"185'4587","reported_seq":4824,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:29.298783+0000","last_change":"2024-09-17T16:12:15.296063+0000","last_active":"2024-09-17T16:13:29.298783+0000","last_peered":"2024-09-17T16:13:29.298783+0000","last_clean":"2024-09-17T16:13:29.298783+0000","last_became_active":"2024-09-17T16:11:13.348959+0000","last_became_peered":"2024-09-17T16:11:13.348959+0000","last_unstale":"2024-09-17T16:13:29.298783+0000","last_undegraded":"2024-09-17T16:13:29.298783+0000","last_fullsized":"2024-09-17T16:13:29.298783+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2146","last_scrub_stamp":"2024-09-17T16:12:15.295952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:15.295952+0000","objects_scrubbed":1680,"log_size":4587,"log_dups_size":0,"ondisk_log_size":4587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":439,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47579136,"num_objects":726,"num_object_clones":0,"num_object_copies":2178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":48000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"187'4541","reported_seq":4777,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-17T16:13:31.355984+0000","last_change":"2024-09-17T16:12:18.628732+0000","last_active":"2024-09-17T16:13:31.355984+0000","last_peered":"2024-09-17T16:13:31.355984+0000","last_clean":"2024-09-17T16:13:31.355984+0000","last_became_active":"2024-09-17T16:11:13.354186+0000","last_became_peered":"2024-09-17T16:11:13.354186+0000","last_unstale":"2024-09-17T16:13:31.355984+0000","last_undegraded":"2024-09-17T16:13:31.355984+0000","last_fullsized":"2024-09-17T16:13:31.355984+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'2226","last_scrub_stamp":"2024-09-17T16:12:18.628468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:18.628468+0000","objects_scrubbed":1798,"log_size":4541,"log_dups_size":0,"ondisk_log_size":4541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46792704,"num_objects":714,"num_object_clones":0,"num_object_copies":2142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2326,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"186'4555","reported_seq":4794,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T16:13:30.485040+0000","last_change":"2024-09-17T16:12:17.792813+0000","last_active":"2024-09-17T16:13:30.485040+0000","last_peered":"2024-09-17T16:13:30.485040+0000","last_clean":"2024-09-17T16:13:30.485040+0000","last_became_active":"2024-09-17T16:11:13.352397+0000","last_became_peered":"2024-09-17T16:11:13.352397+0000","last_unstale":"2024-09-17T16:13:30.485040+0000","last_undegraded":"2024-09-17T16:13:30.485040+0000","last_fullsized":"2024-09-17T16:13:30.485040+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'2229","last_scrub_stamp":"2024-09-17T16:12:17.792731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:17.792731+0000","objects_scrubbed":1815,"log_size":4555,"log_dups_size":0,"ondisk_log_size":4555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:34.612063+0000","scrub_duration":495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42860544,"num_objects":654,"num_object_clones":0,"num_object_copies":1962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2333,"num_write_kb":37504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"188'4705","reported_seq":7922,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-17T16:13:32.763142+0000","last_change":"2024-09-17T16:13:08.898511+0000","last_active":"2024-09-17T16:13:32.763142+0000","last_peered":"2024-09-17T16:13:32.763142+0000","last_clean":"2024-09-17T16:13:32.763142+0000","last_became_active":"2024-09-17T16:13:08.416136+0000","last_became_peered":"2024-09-17T16:13:08.416136+0000","last_unstale":"2024-09-17T16:13:32.763142+0000","last_undegraded":"2024-09-17T16:13:32.763142+0000","last_fullsized":"2024-09-17T16:13:32.763142+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'4006","last_scrub_stamp":"2024-09-17T16:13:08.898311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:08.898311+0000","objects_scrubbed":1303,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:17.866499+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39452672,"num_objects":602,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":701,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"187'4674","reported_seq":8032,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-17T16:13:31.607286+0000","last_change":"2024-09-17T16:13:28.524306+0000","last_active":"2024-09-17T16:13:31.607286+0000","last_peered":"2024-09-17T16:13:31.607286+0000","last_clean":"2024-09-17T16:13:31.607286+0000","last_became_active":"2024-09-17T16:12:57.338076+0000","last_became_peered":"2024-09-17T16:12:57.338076+0000","last_unstale":"2024-09-17T16:13:31.607286+0000","last_undegraded":"2024-09-17T16:13:31.607286+0000","last_fullsized":"2024-09-17T16:13:31.607286+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'4624","last_scrub_stamp":"2024-09-17T16:13:28.524255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:28.524255+0000","objects_scrubbed":707,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:33.220937+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42926080,"num_objects":655,"num_object_clones":0,"num_object_copies":1965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"185'8022","reported_seq":8250,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T16:13:29.136387+0000","last_change":"2024-09-17T16:11:58.924449+0000","last_active":"2024-09-17T16:13:29.136387+0000","last_peered":"2024-09-17T16:13:29.136387+0000","last_clean":"2024-09-17T16:13:29.136387+0000","last_became_active":"2024-09-17T16:10:54.283026+0000","last_became_peered":"2024-09-17T16:10:54.283026+0000","last_unstale":"2024-09-17T16:13:29.136387+0000","last_undegraded":"2024-09-17T16:13:29.136387+0000","last_fullsized":"2024-09-17T16:13:29.136387+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'2421","last_scrub_stamp":"2024-09-17T16:11:58.924214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:58.924214+0000","objects_scrubbed":2408,"log_size":8022,"log_dups_size":0,"ondisk_log_size":8022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89784320,"num_objects":1370,"num_object_clones":0,"num_object_copies":4110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":300544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"185'8197","reported_seq":12712,"reported_epoch":186,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-17T16:13:29.387204+0000","last_change":"2024-09-17T16:13:02.054374+0000","last_active":"2024-09-17T16:13:29.387204+0000","last_peered":"2024-09-17T16:13:29.387204+0000","last_clean":"2024-09-17T16:11:56.354915+0000","last_became_active":"2024-09-17T16:12:40.626458+0000","last_became_peered":"2024-09-17T16:12:40.626458+0000","last_unstale":"2024-09-17T16:13:29.387204+0000","last_undegraded":"2024-09-17T16:12:40.298717+0000","last_fullsized":"2024-09-17T16:12:40.286628+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":8197,"log_dups_size":0,"ondisk_log_size":8197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92733440,"num_objects":1415,"num_object_clones":0,"num_object_copies":4245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8197,"num_write_kb":307584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":141230080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11],"avail_no_missing":["8","11"],"object_location_counts":[{"shards":"8,11","objects":1118}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":1418493976,"num_objects":21645,"num_object_clones":0,"num_object_copies":64935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":467,"num_objects_unfound":0,"num_objects_dirty":21645,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96256,"num_write_kb":3021217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5364,"num_bytes_recovered":317063168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4342558720,"data_stored":4342546504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38296,"internal_metadata":0},"log_size":139855,"ondisk_log_size":139855,"up":78,"acting":76,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285616,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":387748,"kb_used_data":312140,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93357404,"statfs":{"total":95995035648,"available":95597981696,"internally_reserved":0,"allocated":319631360,"data_stored":319170732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547252,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":249636,"kb_used_data":178192,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93495516,"statfs":{"total":95995035648,"available":95739408384,"internally_reserved":0,"allocated":182468608,"data_stored":182007099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743482,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741732,"kb_used_data":642516,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93003420,"statfs":{"total":95995035648,"available":95235502080,"internally_reserved":0,"allocated":657936384,"data_stored":657470803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005118,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":393432,"kb_used_data":326148,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93351720,"statfs":{"total":95995035648,"available":95592161280,"internally_reserved":0,"allocated":333975552,"data_stored":333518496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963596,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":427076,"kb_used_data":338288,"kb_used_omap":41,"kb_used_meta":88662,"kb_avail":93318076,"statfs":{"total":95995035648,"available":95557709824,"internally_reserved":0,"allocated":346406912,"data_stored":345932937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":90790195},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367212,"kb_used_data":304088,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93377940,"statfs":{"total":95995035648,"available":95619010560,"internally_reserved":0,"allocated":311386112,"data_stored":310921051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724620,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":274924,"kb_used_data":225680,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93470228,"statfs":{"total":95995035648,"available":95713513472,"internally_reserved":0,"allocated":231096320,"data_stored":230634811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986258,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":431020,"kb_used_data":363732,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93314132,"statfs":{"total":95995035648,"available":95553671168,"internally_reserved":0,"allocated":372461568,"data_stored":371995987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":656932,"kb_used_data":563284,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93088220,"statfs":{"total":95995035648,"available":95322337280,"internally_reserved":0,"allocated":576802816,"data_stored":576337235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":95877575},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":379816,"kb_used_data":308368,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93365336,"statfs":{"total":95995035648,"available":95606104064,"internally_reserved":0,"allocated":315768832,"data_stored":315307323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":520180,"kb_used_data":426524,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93224972,"statfs":{"total":95995035648,"available":95462371328,"internally_reserved":0,"allocated":436760576,"data_stored":436294606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":383652,"kb_used_data":308048,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93361500,"statfs":{"total":95995035648,"available":95602176000,"internally_reserved":0,"allocated":315441152,"data_stored":314979643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306118656,"data_stored":306118656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":434438144,"data_stored":434438144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312147968,"data_stored":312147968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":556666880,"data_stored":556662808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":371200000,"data_stored":371195928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":224067584,"data_stored":224067584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":309002240,"data_stored":309002240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342949888,"data_stored":342949888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38296,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":332726272,"data_stored":332726272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":655429632,"data_stored":655425560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":181207040,"data_stored":181207040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":316604416,"data_stored":316604416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:13:35.931 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:13:35.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:35.995 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:35.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:36.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:36.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:13:36.118 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:36.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:36.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:36.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:36.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:36.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:36.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:36.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:36.368 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:36.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:36.378 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:36.399 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:36.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:36.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:36 smithi017 ceph-mon[27571]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:36.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:36 smithi017 ceph-mon[27571]: osdmap e191: 12 total, 12 up, 12 in 2024-09-17T16:13:36.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:36 smithi017 ceph-mon[27571]: 2.7 scrub starts 2024-09-17T16:13:36.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:36 smithi017 ceph-mon[27571]: 2.7 scrub ok 2024-09-17T16:13:36.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:36.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:36.606 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:36.641 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:36.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:36.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:36 smithi173 ceph-mon[30530]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:36.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:36 smithi173 ceph-mon[30530]: osdmap e191: 12 total, 12 up, 12 in 2024-09-17T16:13:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:36 smithi173 ceph-mon[30530]: 2.7 scrub starts 2024-09-17T16:13:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:36 smithi173 ceph-mon[30530]: 2.7 scrub ok 2024-09-17T16:13:36.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:13:36.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:36.765 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:36.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:36.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:36.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:36 smithi071 ceph-mon[30002]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:36 smithi071 ceph-mon[30002]: osdmap e191: 12 total, 12 up, 12 in 2024-09-17T16:13:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:36 smithi071 ceph-mon[30002]: 2.7 scrub starts 2024-09-17T16:13:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:36 smithi071 ceph-mon[30002]: 2.7 scrub ok 2024-09-17T16:13:36.867 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:36.892 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:36.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:36.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:37.017 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:37.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:37.070 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:37.104 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:13:37.105 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:13:37.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:37.132 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:13:37.132 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:13:37.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:37.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:37.157 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:13:37.157 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:13:37.180 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:13:37.180 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:13:37.248 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:37.282 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:37.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:37.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:37.348 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:13:37.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:37 smithi017 ceph-mon[27571]: pgmap v474: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 17/57762 objects degraded (0.029%); 467/57762 objects misplaced (0.808%); 4.6 MiB/s, 181 objects/s recovering 2024-09-17T16:13:37.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:37 smithi017 ceph-mon[27571]: osdmap e192: 12 total, 12 up, 12 in 2024-09-17T16:13:37.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3458963095' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:13:37.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4276600901' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:37.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:37.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:37 smithi173 ceph-mon[30530]: pgmap v474: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 17/57762 objects degraded (0.029%); 467/57762 objects misplaced (0.808%); 4.6 MiB/s, 181 objects/s recovering 2024-09-17T16:13:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:37 smithi173 ceph-mon[30530]: osdmap e192: 12 total, 12 up, 12 in 2024-09-17T16:13:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3458963095' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:13:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4276600901' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:37.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:37.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:37.419 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:13:37.419 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:13:37.449 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:13:37.449 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:13:37.476 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:13:37.476 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:13:37.506 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:13:37.506 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:13:37.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:37.535 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:37.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:37.544 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:13:37.544 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:13:37.574 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:13:37.574 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:13:37.598 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:13:37.599 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:13:37.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:37.649 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:13:37.650 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:13:37.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:37.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:37.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:37.781 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:13:37.782 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:13:37.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:37.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:37.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:37 smithi071 ceph-mon[30002]: pgmap v474: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 24 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 17/57762 objects degraded (0.029%); 467/57762 objects misplaced (0.808%); 4.6 MiB/s, 181 objects/s recovering 2024-09-17T16:13:37.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:37 smithi071 ceph-mon[30002]: osdmap e192: 12 total, 12 up, 12 in 2024-09-17T16:13:37.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3458963095' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:13:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4276600901' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:37.892 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:37.893 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:37.893 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:37.915 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:37.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:37.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:38.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:38.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:38.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:38.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:38.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:38.181 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:38.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:38.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:38.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:38.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:38.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:38.418 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:38.441 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:38.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:38.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:38.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:38.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:38.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:38.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:38 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:13:38.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:38 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:13:38.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:38 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:38.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3458963095' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:13:38.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:38 smithi017 ceph-mon[27571]: osdmap e193: 12 total, 12 up, 12 in 2024-09-17T16:13:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:38 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:13:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:38 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:13:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:38 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:38.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3458963095' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:13:38.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:38 smithi173 ceph-mon[30530]: osdmap e193: 12 total, 12 up, 12 in 2024-09-17T16:13:38.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:38.676 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-17T16:13:37.759056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1165394488,"num_objects":17776,"num_object_clones":0,"num_object_copies":53328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17776,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":100250,"num_write_kb":3023367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6702,"num_bytes_recovered":367661600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143818,"ondisk_log_size":143818,"up":81,"acting":79,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":83,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":4548428,"kb_used_data":3595976,"kb_used_omap":58,"kb_used_meta":952069,"kb_avail":1120393396,"statfs":{"total":1151940427776,"available":1147282837504,"internally_reserved":0,"allocated":3682279424,"data_stored":3676491538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":974918819},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-253689856,"num_objects":-3871,"num_object_clones":0,"num_object_copies":-11613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-297,"num_objects_misplaced":-467,"num_objects_unfound":0,"num_objects_dirty":-3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3871,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1336,"num_bytes_recovered":50069504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000731"},"pg_stats":[{"pgid":"2.19","version":"191'4586","reported_seq":6248,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-17T16:13:35.554812+0000","last_change":"2024-09-17T16:13:08.775898+0000","last_active":"2024-09-17T16:13:35.554812+0000","last_peered":"2024-09-17T16:13:35.554812+0000","last_clean":"2024-09-17T16:13:35.554812+0000","last_became_active":"2024-09-17T16:11:24.264470+0000","last_became_peered":"2024-09-17T16:11:24.264470+0000","last_unstale":"2024-09-17T16:13:35.554812+0000","last_undegraded":"2024-09-17T16:13:35.554812+0000","last_fullsized":"2024-09-17T16:13:35.554812+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"175'3870","last_scrub_stamp":"2024-09-17T16:13:08.775814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:08.775814+0000","objects_scrubbed":1265,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:30.625192+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":538,"num_object_clones":0,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":77952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"191'4796","reported_seq":9109,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-17T16:13:36.364592+0000","last_change":"2024-09-17T16:13:11.523551+0000","last_active":"2024-09-17T16:13:36.364592+0000","last_peered":"2024-09-17T16:13:36.364592+0000","last_clean":"2024-09-17T16:13:36.364592+0000","last_became_active":"2024-09-17T16:13:10.323458+0000","last_became_peered":"2024-09-17T16:13:10.323458+0000","last_unstale":"2024-09-17T16:13:36.364592+0000","last_undegraded":"2024-09-17T16:13:36.364592+0000","last_fullsized":"2024-09-17T16:13:36.364592+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"177'4073","last_scrub_stamp":"2024-09-17T16:13:11.523023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:11.523023+0000","objects_scrubbed":1267,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:25.569087+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":536,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":731,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"191'4672","reported_seq":4915,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-17T16:13:35.440344+0000","last_change":"2024-09-17T16:12:16.704276+0000","last_active":"2024-09-17T16:13:35.440344+0000","last_peered":"2024-09-17T16:13:35.440344+0000","last_clean":"2024-09-17T16:13:35.440344+0000","last_became_active":"2024-09-17T16:11:13.358625+0000","last_became_peered":"2024-09-17T16:11:13.358625+0000","last_unstale":"2024-09-17T16:13:35.440344+0000","last_undegraded":"2024-09-17T16:13:35.440344+0000","last_fullsized":"2024-09-17T16:13:35.440344+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":5,"last_scrub":"145'2177","last_scrub_stamp":"2024-09-17T16:12:16.704167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:16.704167+0000","objects_scrubbed":1705,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:25.584490+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35651584,"num_objects":544,"num_object_clones":0,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2507,"num_write_kb":43072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"190'4521","reported_seq":6594,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-17T16:13:35.338281+0000","last_change":"2024-09-17T16:11:55.554301+0000","last_active":"2024-09-17T16:13:35.338281+0000","last_peered":"2024-09-17T16:13:35.338281+0000","last_clean":"2024-09-17T16:13:35.338281+0000","last_became_active":"2024-09-17T16:11:25.266055+0000","last_became_peered":"2024-09-17T16:11:25.266055+0000","last_unstale":"2024-09-17T16:13:35.338281+0000","last_undegraded":"2024-09-17T16:13:35.338281+0000","last_fullsized":"2024-09-17T16:13:35.338281+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"132'1495","last_scrub_stamp":"2024-09-17T16:11:55.554231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:55.554231+0000","objects_scrubbed":1100,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:24.678149+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33947648,"num_objects":518,"num_object_clones":0,"num_object_copies":1554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":78208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"191'4672","reported_seq":6793,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-17T16:13:36.364515+0000","last_change":"2024-09-17T16:13:12.481602+0000","last_active":"2024-09-17T16:13:36.364515+0000","last_peered":"2024-09-17T16:13:36.364515+0000","last_clean":"2024-09-17T16:13:36.364515+0000","last_became_active":"2024-09-17T16:11:23.302908+0000","last_became_peered":"2024-09-17T16:11:23.302908+0000","last_unstale":"2024-09-17T16:13:36.364515+0000","last_undegraded":"2024-09-17T16:13:36.364515+0000","last_fullsized":"2024-09-17T16:13:36.364515+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"177'4029","last_scrub_stamp":"2024-09-17T16:13:12.481403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:12.481403+0000","objects_scrubbed":1189,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:39.732490+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35717120,"num_objects":545,"num_object_clones":0,"num_object_copies":1635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3181,"num_write_kb":83776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"191'4744","reported_seq":7012,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-17T16:13:35.349766+0000","last_change":"2024-09-17T16:13:12.834609+0000","last_active":"2024-09-17T16:13:35.349766+0000","last_peered":"2024-09-17T16:13:35.349766+0000","last_clean":"2024-09-17T16:13:35.349766+0000","last_became_active":"2024-09-17T16:11:27.290863+0000","last_became_peered":"2024-09-17T16:11:27.290863+0000","last_unstale":"2024-09-17T16:13:35.349766+0000","last_undegraded":"2024-09-17T16:13:35.349766+0000","last_fullsized":"2024-09-17T16:13:35.349766+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":5,"last_scrub":"178'4116","last_scrub_stamp":"2024-09-17T16:13:12.834492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:12.834492+0000","objects_scrubbed":1189,"log_size":4744,"log_dups_size":0,"ondisk_log_size":4744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:36.962652+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36569088,"num_objects":558,"num_object_clones":0,"num_object_copies":1674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":77376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"191'4528","reported_seq":6538,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-17T16:13:36.364500+0000","last_change":"2024-09-17T16:11:53.132205+0000","last_active":"2024-09-17T16:13:36.364500+0000","last_peered":"2024-09-17T16:13:36.364500+0000","last_clean":"2024-09-17T16:13:36.364500+0000","last_became_active":"2024-09-17T16:11:22.310367+0000","last_became_peered":"2024-09-17T16:11:22.310367+0000","last_unstale":"2024-09-17T16:13:36.364500+0000","last_undegraded":"2024-09-17T16:13:36.364500+0000","last_fullsized":"2024-09-17T16:13:36.364500+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"132'1415","last_scrub_stamp":"2024-09-17T16:11:53.132100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:53.132100+0000","objects_scrubbed":973,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:24.400146+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35520512,"num_objects":542,"num_object_clones":0,"num_object_copies":1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3119,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"191'4727","reported_seq":12179,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-17T16:13:35.402608+0000","last_change":"2024-09-17T16:13:27.714585+0000","last_active":"2024-09-17T16:13:35.402608+0000","last_peered":"2024-09-17T16:13:35.402608+0000","last_clean":"2024-09-17T16:13:35.402608+0000","last_became_active":"2024-09-17T16:12:56.327408+0000","last_became_peered":"2024-09-17T16:12:56.327408+0000","last_unstale":"2024-09-17T16:13:35.402608+0000","last_undegraded":"2024-09-17T16:13:35.402608+0000","last_fullsized":"2024-09-17T16:13:35.402608+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'4560","last_scrub_stamp":"2024-09-17T16:13:27.714377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:27.714377+0000","objects_scrubbed":720,"log_size":4727,"log_dups_size":0,"ondisk_log_size":4727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:40.853484+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36110336,"num_objects":551,"num_object_clones":0,"num_object_copies":1653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,7],"acting":[10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"191'4682","reported_seq":6972,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-17T16:13:35.455186+0000","last_change":"2024-09-17T16:12:45.621594+0000","last_active":"2024-09-17T16:13:35.455186+0000","last_peered":"2024-09-17T16:13:35.455186+0000","last_clean":"2024-09-17T16:13:35.455186+0000","last_became_active":"2024-09-17T16:11:30.264463+0000","last_became_peered":"2024-09-17T16:11:30.264463+0000","last_unstale":"2024-09-17T16:13:35.455186+0000","last_undegraded":"2024-09-17T16:13:35.455186+0000","last_fullsized":"2024-09-17T16:13:35.455186+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":5,"last_scrub":"158'3284","last_scrub_stamp":"2024-09-17T16:12:45.621470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:45.621470+0000","objects_scrubbed":1937,"log_size":4682,"log_dups_size":0,"ondisk_log_size":4682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:00.545203+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35192832,"num_objects":537,"num_object_clones":0,"num_object_copies":1611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3449,"num_write_kb":101440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"190'4575","reported_seq":6124,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-17T16:13:34.377656+0000","last_change":"2024-09-17T16:12:22.182841+0000","last_active":"2024-09-17T16:13:34.377656+0000","last_peered":"2024-09-17T16:13:34.377656+0000","last_clean":"2024-09-17T16:13:34.377656+0000","last_became_active":"2024-09-17T16:11:21.266149+0000","last_became_peered":"2024-09-17T16:11:21.266149+0000","last_unstale":"2024-09-17T16:13:34.377656+0000","last_undegraded":"2024-09-17T16:13:34.377656+0000","last_fullsized":"2024-09-17T16:13:34.377656+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'2377","last_scrub_stamp":"2024-09-17T16:12:22.182710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:22.182710+0000","objects_scrubbed":1944,"log_size":4575,"log_dups_size":0,"ondisk_log_size":4575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:34.798898+0000","scrub_duration":332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40632320,"num_objects":620,"num_object_clones":0,"num_object_copies":1860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2204,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"190'8344","reported_seq":8548,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-17T16:13:34.377729+0000","last_change":"2024-09-17T16:10:54.283333+0000","last_active":"2024-09-17T16:13:34.377729+0000","last_peered":"2024-09-17T16:13:34.377729+0000","last_clean":"2024-09-17T16:13:34.377729+0000","last_became_active":"2024-09-17T16:10:54.282931+0000","last_became_peered":"2024-09-17T16:10:54.282931+0000","last_unstale":"2024-09-17T16:13:34.377729+0000","last_undegraded":"2024-09-17T16:13:34.377729+0000","last_fullsized":"2024-09-17T16:13:34.377729+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":8344,"log_dups_size":0,"ondisk_log_size":8344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:34.852831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78774272,"num_objects":1202,"num_object_clones":0,"num_object_copies":3606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8344,"num_write_kb":305472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"192'8441","reported_seq":8684,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-17T16:13:36.619971+0000","last_change":"2024-09-17T16:12:01.681365+0000","last_active":"2024-09-17T16:13:36.619971+0000","last_peered":"2024-09-17T16:13:36.619971+0000","last_clean":"2024-09-17T16:13:36.619971+0000","last_became_active":"2024-09-17T16:10:54.283208+0000","last_became_peered":"2024-09-17T16:10:54.283208+0000","last_unstale":"2024-09-17T16:13:36.619971+0000","last_undegraded":"2024-09-17T16:13:36.619971+0000","last_fullsized":"2024-09-17T16:13:36.619971+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'2642","last_scrub_stamp":"2024-09-17T16:12:01.681123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:01.681123+0000","objects_scrubbed":2614,"log_size":8441,"log_dups_size":0,"ondisk_log_size":8441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:17.223777+0000","scrub_duration":684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71237632,"num_objects":1087,"num_object_clones":0,"num_object_copies":3261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8441,"num_write_kb":304896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"191'8430","reported_seq":8684,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-17T16:13:36.364627+0000","last_change":"2024-09-17T16:12:18.766646+0000","last_active":"2024-09-17T16:13:36.364627+0000","last_peered":"2024-09-17T16:13:36.364627+0000","last_clean":"2024-09-17T16:13:36.364627+0000","last_became_active":"2024-09-17T16:10:54.283016+0000","last_became_peered":"2024-09-17T16:10:54.283016+0000","last_unstale":"2024-09-17T16:13:36.364627+0000","last_undegraded":"2024-09-17T16:13:36.364627+0000","last_fullsized":"2024-09-17T16:13:36.364627+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3627","last_scrub_stamp":"2024-09-17T16:12:18.766492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:18.766492+0000","objects_scrubbed":3600,"log_size":8430,"log_dups_size":0,"ondisk_log_size":8430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:46.143371+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71761944,"num_objects":1096,"num_object_clones":0,"num_object_copies":3288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1096,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8430,"num_write_kb":304769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"190'8349","reported_seq":8553,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-17T16:13:34.337555+0000","last_change":"2024-09-17T16:10:54.282319+0000","last_active":"2024-09-17T16:13:34.337555+0000","last_peered":"2024-09-17T16:13:34.337555+0000","last_clean":"2024-09-17T16:13:34.337555+0000","last_became_active":"2024-09-17T16:10:54.282069+0000","last_became_peered":"2024-09-17T16:10:54.282069+0000","last_unstale":"2024-09-17T16:13:34.337555+0000","last_undegraded":"2024-09-17T16:13:34.337555+0000","last_fullsized":"2024-09-17T16:13:34.337555+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":8349,"log_dups_size":0,"ondisk_log_size":8349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71368704,"num_objects":1089,"num_object_clones":0,"num_object_copies":3267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8349,"num_write_kb":302016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":229,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-17T16:13:32.316237+0000","last_change":"2024-09-17T16:12:39.292150+0000","last_active":"2024-09-17T16:13:32.316237+0000","last_peered":"2024-09-17T16:13:32.316237+0000","last_clean":"2024-09-17T16:13:32.316237+0000","last_became_active":"2024-09-17T16:12:39.286388+0000","last_became_peered":"2024-09-17T16:12:39.286388+0000","last_unstale":"2024-09-17T16:13:32.316237+0000","last_undegraded":"2024-09-17T16:13:32.316237+0000","last_fullsized":"2024-09-17T16:13:32.316237+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:11:04.478242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:11:04.478242+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"191'4726","reported_seq":7963,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-17T16:13:35.338308+0000","last_change":"2024-09-17T16:13:22.680448+0000","last_active":"2024-09-17T16:13:35.338308+0000","last_peered":"2024-09-17T16:13:35.338308+0000","last_clean":"2024-09-17T16:13:35.338308+0000","last_became_active":"2024-09-17T16:12:52.302562+0000","last_became_peered":"2024-09-17T16:12:52.302562+0000","last_unstale":"2024-09-17T16:13:35.338308+0000","last_undegraded":"2024-09-17T16:13:35.338308+0000","last_fullsized":"2024-09-17T16:13:35.338308+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":4726,"log_dups_size":0,"ondisk_log_size":4726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:13:52.659598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35323904,"num_objects":539,"num_object_clones":0,"num_object_copies":1617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4305,"num_write_kb":155008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1497,"num_bytes_recovered":80609280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"188'4479","reported_seq":7594,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-17T16:13:32.828608+0000","last_change":"2024-09-17T16:13:03.748291+0000","last_active":"2024-09-17T16:13:32.828608+0000","last_peered":"2024-09-17T16:13:32.828608+0000","last_clean":"2024-09-17T16:13:32.828608+0000","last_became_active":"2024-09-17T16:13:03.359631+0000","last_became_peered":"2024-09-17T16:13:03.359631+0000","last_unstale":"2024-09-17T16:13:32.828608+0000","last_undegraded":"2024-09-17T16:13:32.828608+0000","last_fullsized":"2024-09-17T16:13:32.828608+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:13:33.701447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36864000,"num_objects":562,"num_object_clones":0,"num_object_copies":1686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4068,"num_write_kb":148192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1441,"num_bytes_recovered":77463552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"192'4719","reported_seq":4969,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-17T16:13:36.596490+0000","last_change":"2024-09-17T16:12:16.414710+0000","last_active":"2024-09-17T16:13:36.596490+0000","last_peered":"2024-09-17T16:13:36.596490+0000","last_clean":"2024-09-17T16:13:36.596490+0000","last_became_active":"2024-09-17T16:11:13.356367+0000","last_became_peered":"2024-09-17T16:11:13.356367+0000","last_unstale":"2024-09-17T16:13:36.596490+0000","last_undegraded":"2024-09-17T16:13:36.596490+0000","last_fullsized":"2024-09-17T16:13:36.596490+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2190","last_scrub_stamp":"2024-09-17T16:12:16.414644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:16.414644+0000","objects_scrubbed":1711,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:20.196085+0000","scrub_duration":347,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37289984,"num_objects":569,"num_object_clones":0,"num_object_copies":1707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":44672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"191'4669","reported_seq":4928,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-17T16:13:35.340922+0000","last_change":"2024-09-17T16:13:32.993292+0000","last_active":"2024-09-17T16:13:35.340922+0000","last_peered":"2024-09-17T16:13:35.340922+0000","last_clean":"2024-09-17T16:13:35.340922+0000","last_became_active":"2024-09-17T16:11:13.352761+0000","last_became_peered":"2024-09-17T16:11:13.352761+0000","last_unstale":"2024-09-17T16:13:35.340922+0000","last_undegraded":"2024-09-17T16:13:35.340922+0000","last_fullsized":"2024-09-17T16:13:35.340922+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4605","last_scrub_stamp":"2024-09-17T16:13:32.993228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:32.993228+0000","objects_scrubbed":582,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:47.865099+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33816576,"num_objects":516,"num_object_clones":0,"num_object_copies":1548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":42368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"192'4719","reported_seq":7988,"reported_epoch":192,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-17T16:13:36.627133+0000","last_change":"2024-09-17T16:13:29.729752+0000","last_active":"2024-09-17T16:13:36.627133+0000","last_peered":"2024-09-17T16:13:36.627133+0000","last_clean":"2024-09-17T16:13:27.758735+0000","last_became_active":"2024-09-17T16:13:29.693012+0000","last_became_peered":"2024-09-17T16:13:29.693012+0000","last_unstale":"2024-09-17T16:13:36.627133+0000","last_undegraded":"2024-09-17T16:13:29.306887+0000","last_fullsized":"2024-09-17T16:13:29.296055+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'1233","last_scrub_stamp":"2024-09-17T16:11:30.184562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:30.184562+0000","objects_scrubbed":777,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36372480,"num_objects":555,"num_object_clones":0,"num_object_copies":1665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":880,"num_bytes_recovered":48365568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"0,1","objects":538}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"190'4719","reported_seq":4963,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-17T16:13:34.342323+0000","last_change":"2024-09-17T16:12:15.296063+0000","last_active":"2024-09-17T16:13:34.342323+0000","last_peered":"2024-09-17T16:13:34.342323+0000","last_clean":"2024-09-17T16:13:34.342323+0000","last_became_active":"2024-09-17T16:11:13.348959+0000","last_became_peered":"2024-09-17T16:11:13.348959+0000","last_unstale":"2024-09-17T16:13:34.342323+0000","last_undegraded":"2024-09-17T16:13:34.342323+0000","last_fullsized":"2024-09-17T16:13:34.342323+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2146","last_scrub_stamp":"2024-09-17T16:12:15.295952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:15.295952+0000","objects_scrubbed":1680,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":439,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38928384,"num_objects":594,"num_object_clones":0,"num_object_copies":1782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2586,"num_write_kb":48000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"192'4689","reported_seq":4935,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-17T16:13:36.599682+0000","last_change":"2024-09-17T16:12:18.628732+0000","last_active":"2024-09-17T16:13:36.599682+0000","last_peered":"2024-09-17T16:13:36.599682+0000","last_clean":"2024-09-17T16:13:36.599682+0000","last_became_active":"2024-09-17T16:11:13.354186+0000","last_became_peered":"2024-09-17T16:11:13.354186+0000","last_unstale":"2024-09-17T16:13:36.599682+0000","last_undegraded":"2024-09-17T16:13:36.599682+0000","last_fullsized":"2024-09-17T16:13:36.599682+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'2226","last_scrub_stamp":"2024-09-17T16:12:18.628468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:18.628468+0000","objects_scrubbed":1798,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37093376,"num_objects":566,"num_object_clones":0,"num_object_copies":1698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2474,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"191'4691","reported_seq":4943,"reported_epoch":191,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T16:13:35.538831+0000","last_change":"2024-09-17T16:13:35.519056+0000","last_active":"2024-09-17T16:13:35.538831+0000","last_peered":"2024-09-17T16:13:35.538831+0000","last_clean":"2024-09-17T16:13:35.538831+0000","last_became_active":"2024-09-17T16:11:13.352397+0000","last_became_peered":"2024-09-17T16:11:13.352397+0000","last_unstale":"2024-09-17T16:13:35.538831+0000","last_undegraded":"2024-09-17T16:13:35.538831+0000","last_fullsized":"2024-09-17T16:13:35.538831+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'2229","last_scrub_stamp":"2024-09-17T16:12:17.792731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:17.792731+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33947648,"num_objects":518,"num_object_clones":0,"num_object_copies":1554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2469,"num_write_kb":37504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"188'4705","reported_seq":7922,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-17T16:13:32.763142+0000","last_change":"2024-09-17T16:13:08.898511+0000","last_active":"2024-09-17T16:13:32.763142+0000","last_peered":"2024-09-17T16:13:32.763142+0000","last_clean":"2024-09-17T16:13:32.763142+0000","last_became_active":"2024-09-17T16:13:08.416136+0000","last_became_peered":"2024-09-17T16:13:08.416136+0000","last_unstale":"2024-09-17T16:13:32.763142+0000","last_undegraded":"2024-09-17T16:13:32.763142+0000","last_fullsized":"2024-09-17T16:13:32.763142+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'4006","last_scrub_stamp":"2024-09-17T16:13:08.898311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:08.898311+0000","objects_scrubbed":1303,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:17.866499+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39452672,"num_objects":602,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":701,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"192'4793","reported_seq":8161,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-17T16:13:36.613926+0000","last_change":"2024-09-17T16:13:28.524306+0000","last_active":"2024-09-17T16:13:36.613926+0000","last_peered":"2024-09-17T16:13:36.613926+0000","last_clean":"2024-09-17T16:13:36.613926+0000","last_became_active":"2024-09-17T16:12:57.338076+0000","last_became_peered":"2024-09-17T16:12:57.338076+0000","last_unstale":"2024-09-17T16:13:36.613926+0000","last_undegraded":"2024-09-17T16:13:36.613926+0000","last_fullsized":"2024-09-17T16:13:36.613926+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'4624","last_scrub_stamp":"2024-09-17T16:13:28.524255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:28.524255+0000","objects_scrubbed":707,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:33.220937+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":536,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"190'8289","reported_seq":8527,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-17T16:13:34.311457+0000","last_change":"2024-09-17T16:11:58.924449+0000","last_active":"2024-09-17T16:13:34.311457+0000","last_peered":"2024-09-17T16:13:34.311457+0000","last_clean":"2024-09-17T16:13:34.311457+0000","last_became_active":"2024-09-17T16:10:54.283026+0000","last_became_peered":"2024-09-17T16:10:54.283026+0000","last_unstale":"2024-09-17T16:13:34.311457+0000","last_undegraded":"2024-09-17T16:13:34.311457+0000","last_fullsized":"2024-09-17T16:13:34.311457+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'2421","last_scrub_stamp":"2024-09-17T16:11:58.924214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:58.924214+0000","objects_scrubbed":2408,"log_size":8289,"log_dups_size":0,"ondisk_log_size":8289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72286208,"num_objects":1103,"num_object_clones":0,"num_object_copies":3309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8289,"num_write_kb":300544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"190'8461","reported_seq":14436,"reported_epoch":190,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:13:34.392512+0000","last_change":"2024-09-17T16:13:31.563408+0000","last_active":"2024-09-17T16:13:34.392512+0000","last_peered":"2024-09-17T16:13:34.392512+0000","last_clean":"2024-09-17T16:11:56.354915+0000","last_became_active":"2024-09-17T16:12:40.626458+0000","last_became_peered":"2024-09-17T16:12:40.626458+0000","last_unstale":"2024-09-17T16:13:34.392512+0000","last_undegraded":"2024-09-17T16:13:34.392512+0000","last_fullsized":"2024-09-17T16:12:40.286628+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":8461,"log_dups_size":0,"ondisk_log_size":8461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75431936,"num_objects":1151,"num_object_clones":0,"num_object_copies":3453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8461,"num_write_kb":307584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2882,"num_bytes_recovered":160694272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11],"avail_no_missing":["8","11"],"object_location_counts":[{"shards":"8,11","objects":1151}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":1164804120,"num_objects":17774,"num_object_clones":0,"num_object_copies":53322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100127,"num_write_kb":3021217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6700,"num_bytes_recovered":367132672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3628544000,"data_stored":3628531784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38296,"internal_metadata":0},"log_size":143726,"ondisk_log_size":143726,"up":78,"acting":76,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285617,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321104,"kb_used_data":257980,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93424048,"statfs":{"total":95995035648,"available":95666225152,"internally_reserved":0,"allocated":264171520,"data_stored":263693114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547253,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":248992,"kb_used_data":173388,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93496160,"statfs":{"total":95995035648,"available":95740067840,"internally_reserved":0,"allocated":177549312,"data_stored":177065556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":77396416},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743483,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":632104,"kb_used_data":524560,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93113048,"statfs":{"total":95995035648,"available":95347761152,"internally_reserved":0,"allocated":537149440,"data_stored":536661612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005119,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":344336,"kb_used_data":272884,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93400816,"statfs":{"total":95995035648,"available":95642435584,"internally_reserved":0,"allocated":279433216,"data_stored":278958382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963596,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":427076,"kb_used_data":338288,"kb_used_omap":41,"kb_used_meta":88662,"kb_avail":93318076,"statfs":{"total":95995035648,"available":95557709824,"internally_reserved":0,"allocated":346406912,"data_stored":345932937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":90790195},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308072,"kb_used_data":240788,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93437080,"statfs":{"total":95995035648,"available":95679569920,"internally_reserved":0,"allocated":246566912,"data_stored":246079604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724621,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":251732,"kb_used_data":180288,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93493420,"statfs":{"total":95995035648,"available":95737262080,"internally_reserved":0,"allocated":184614912,"data_stored":184135617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986259,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":370584,"kb_used_data":299140,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93374568,"statfs":{"total":95995035648,"available":95615557632,"internally_reserved":0,"allocated":306319360,"data_stored":305835993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":535452,"kb_used_data":454276,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93209700,"statfs":{"total":95995035648,"available":95446732800,"internally_reserved":0,"allocated":465178624,"data_stored":464695257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":83098055},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":336288,"kb_used_data":260684,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93408864,"statfs":{"total":95995035648,"available":95650676736,"internally_reserved":0,"allocated":266940416,"data_stored":266456660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":445680,"kb_used_data":346456,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93299472,"statfs":{"total":95995035648,"available":95538659328,"internally_reserved":0,"allocated":354770944,"data_stored":354282706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327008,"kb_used_data":247244,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93418144,"statfs":{"total":95995035648,"available":95660179456,"internally_reserved":0,"allocated":253177856,"data_stored":252694100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":250413056,"data_stored":250413056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":349831168,"data_stored":349831168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":260702208,"data_stored":260702208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":458231808,"data_stored":458227736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":303697920,"data_stored":303693848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":176619520,"data_stored":176619520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":244449280,"data_stored":244449280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342949888,"data_stored":342949888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38296,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":273940480,"data_stored":273940480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":535695360,"data_stored":535691288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":176357376,"data_stored":176357376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":255655936,"data_stored":255655936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:13:38.678 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-17T16:13:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:38.699 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:38.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:38.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:13:38.800 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:38 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:13:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:38 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:13:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:38 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3458963095' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:13:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:38 smithi071 ceph-mon[30002]: osdmap e193: 12 total, 12 up, 12 in 2024-09-17T16:13:38.822 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:38.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:38.924 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:38.946 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:38.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:39.041 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:13:39.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:39.070 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:39.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:39.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:39.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:39.197 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:39.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:39.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:39.299 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:39.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:39.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:39.424 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:39.447 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:39.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:39.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:39.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:39.572 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:39.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:39 smithi173 ceph-mon[30530]: pgmap v477: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 24 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 967 op/s; 17/53328 objects degraded (0.032%); 12 MiB/s, 333 objects/s recovering 2024-09-17T16:13:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:39 smithi173 ceph-mon[30530]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:39 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 17/53328 objects degraded (0.032%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:39 smithi173 ceph-mon[30530]: osdmap e194: 12 total, 12 up, 12 in 2024-09-17T16:13:39.674 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:39.707 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:39.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:39.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:39.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:39 smithi017 ceph-mon[27571]: pgmap v477: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 24 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 967 op/s; 17/53328 objects degraded (0.032%); 12 MiB/s, 333 objects/s recovering 2024-09-17T16:13:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:39 smithi017 ceph-mon[27571]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:39 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 17/53328 objects degraded (0.032%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:39 smithi017 ceph-mon[27571]: osdmap e194: 12 total, 12 up, 12 in 2024-09-17T16:13:39.809 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:39.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:39 smithi071 ceph-mon[30002]: pgmap v477: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 24 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 967 op/s; 17/53328 objects degraded (0.032%); 12 MiB/s, 333 objects/s recovering 2024-09-17T16:13:39.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:39 smithi071 ceph-mon[30002]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:39.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:39 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 17/53328 objects degraded (0.032%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:39.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:39 smithi071 ceph-mon[30002]: osdmap e194: 12 total, 12 up, 12 in 2024-09-17T16:13:39.836 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:39.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:39.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:39.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:39.965 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:39.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:40.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:40.093 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:40.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:40.150 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:40.150 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:40.194 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:40.216 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:40.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:40.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:40.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:40.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:40.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:40.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:40.476 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:40.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:40.505 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:40.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:40.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:40.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:40.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:40.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:40.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:40.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:13:40.840 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:40.866 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:40.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:40.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:40.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:40.992 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:41.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:41.095 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:13:41.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:41.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:41.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:41.259 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:41.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:41.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:41.389 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:41.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:41.483 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:13:41.483 INFO:teuthology.orchestra.run.smithi017.stdout:{"fsid":"19341aec-750e-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17/48960 objects degraded (0.035%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":539,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":194,"num_osds":12,"num_up_osds":12,"osd_up_since":1726589558,"num_in_osds":12,"osd_in_since":1726589369,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":25},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":27,"num_pools":2,"num_objects":16320,"data_bytes":1069941304,"bytes_used":4404490240,"bytes_avail":1147535937536,"bytes_total":1151940427776,"degraded_objects":17,"degraded_total":48960,"degraded_ratio":0.00034722222222222224},"fsmap":{"epoch":1,"btime":"2024-09-17T16:02:46:978559+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.17:8443/"}},"servicemap":{"epoch":20,"modified":"2024-09-17T16:13:39.759509+0000","services":{}},"progress_events":{}} 2024-09-17T16:13:41.491 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:41.518 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:41 smithi071 ceph-mon[30002]: pgmap v479: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 1020 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17/48960 objects degraded (0.035%) 2024-09-17T16:13:41.522 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:41.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:41.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:41 smithi173 ceph-mon[30530]: pgmap v479: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 1020 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17/48960 objects degraded (0.035%) 2024-09-17T16:13:41.646 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:41.647 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:41.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:41.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:41 smithi017 ceph-mon[27571]: pgmap v479: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 25 active+clean; 1020 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17/48960 objects degraded (0.035%) 2024-09-17T16:13:41.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:41.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:41.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:41.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:41.890 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:41.916 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:41.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:42.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:42.042 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:42.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:42.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:42.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:42.171 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:42.232 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T16:13:42.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:13:42.272 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:42.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:13:42.295 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:42.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:42.315 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:13:42.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:42.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:42.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:42.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:13:42.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/599538388' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:13:42.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:42.543 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:42.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:42.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/599538388' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:13:42.567 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:42.615 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:42.626 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:13:42.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/599538388' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:13:42.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:42.671 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:42.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:42.773 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:42.795 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:42.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:42.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:42.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:42.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:43.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:43.053 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:43.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:43.150 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:43.150 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:43.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:43.184 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:43.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:43.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:43.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:43.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:43.367 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:43 smithi017 ceph-mon[27571]: pgmap v480: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 827 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 325 op/s; 17/39693 objects degraded (0.043%) 2024-09-17T16:13:43.419 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:43.442 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:43.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:43.544 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:43.546 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:13:43.548 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:13:43.583 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:43.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:43.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:43 smithi173 ceph-mon[30530]: pgmap v480: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 827 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 325 op/s; 17/39693 objects degraded (0.043%) 2024-09-17T16:13:43.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:43.716 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:43.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:43.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:43 smithi071 ceph-mon[30002]: pgmap v480: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 827 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 325 op/s; 17/39693 objects degraded (0.043%) 2024-09-17T16:13:43.810 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:13:43.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:43.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:43.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:43.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:43.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:43.973 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:43.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:44.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:44.094 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:13:44.098 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:44.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:44.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:44.222 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:44.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:44.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:44.271 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-17T16:13:41.759974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":867500600,"num_objects":13231,"num_object_clones":0,"num_object_copies":39693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13231,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":100725,"num_write_kb":2875175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5818,"num_bytes_recovered":290198048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148380,"ondisk_log_size":148380,"up":81,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":83,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":3661404,"kb_used_data":2694976,"kb_used_omap":58,"kb_used_meta":965957,"kb_avail":1121280420,"statfs":{"total":1151940427776,"available":1148191150080,"internally_reserved":0,"allocated":2759655424,"data_stored":2753679386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":989140131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-394756096,"num_objects":-6023,"num_object_clones":0,"num_object_copies":-18069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-467,"num_objects_unfound":0,"num_objects_dirty":-6023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":-148192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-275,"num_bytes_recovered":-46858240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001271"},"pg_stats":[{"pgid":"2.19","version":"194'4751","reported_seq":6420,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:41.175666+0000","last_change":"2024-09-17T16:13:08.775898+0000","last_active":"2024-09-17T16:13:41.175666+0000","last_peered":"2024-09-17T16:13:41.175666+0000","last_clean":"2024-09-17T16:13:41.175666+0000","last_became_active":"2024-09-17T16:11:24.264470+0000","last_became_peered":"2024-09-17T16:11:24.264470+0000","last_unstale":"2024-09-17T16:13:41.175666+0000","last_undegraded":"2024-09-17T16:13:41.175666+0000","last_fullsized":"2024-09-17T16:13:41.175666+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"175'3870","last_scrub_stamp":"2024-09-17T16:13:08.775814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:08.775814+0000","objects_scrubbed":1265,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:30.625192+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24444928,"num_objects":373,"num_object_clones":0,"num_object_copies":1119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3182,"num_write_kb":77952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"194'4950","reported_seq":9267,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:41.314049+0000","last_change":"2024-09-17T16:13:11.523551+0000","last_active":"2024-09-17T16:13:41.314049+0000","last_peered":"2024-09-17T16:13:41.314049+0000","last_clean":"2024-09-17T16:13:41.314049+0000","last_became_active":"2024-09-17T16:13:10.323458+0000","last_became_peered":"2024-09-17T16:13:10.323458+0000","last_unstale":"2024-09-17T16:13:41.314049+0000","last_undegraded":"2024-09-17T16:13:41.314049+0000","last_fullsized":"2024-09-17T16:13:41.314049+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"177'4073","last_scrub_stamp":"2024-09-17T16:13:11.523023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:11.523023+0000","objects_scrubbed":1267,"log_size":4950,"log_dups_size":0,"ondisk_log_size":4950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:25.569087+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25034752,"num_objects":382,"num_object_clones":0,"num_object_copies":1146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":885,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"194'4802","reported_seq":5052,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:40.472081+0000","last_change":"2024-09-17T16:12:16.704276+0000","last_active":"2024-09-17T16:13:40.472081+0000","last_peered":"2024-09-17T16:13:40.472081+0000","last_clean":"2024-09-17T16:13:40.472081+0000","last_became_active":"2024-09-17T16:11:13.358625+0000","last_became_peered":"2024-09-17T16:11:13.358625+0000","last_unstale":"2024-09-17T16:13:40.472081+0000","last_undegraded":"2024-09-17T16:13:40.472081+0000","last_fullsized":"2024-09-17T16:13:40.472081+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":5,"last_scrub":"145'2177","last_scrub_stamp":"2024-09-17T16:12:16.704167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:16.704167+0000","objects_scrubbed":1705,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:25.584490+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27131904,"num_objects":414,"num_object_clones":0,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2637,"num_write_kb":43072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"194'4657","reported_seq":6736,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:40.393310+0000","last_change":"2024-09-17T16:11:55.554301+0000","last_active":"2024-09-17T16:13:40.393310+0000","last_peered":"2024-09-17T16:13:40.393310+0000","last_clean":"2024-09-17T16:13:40.393310+0000","last_became_active":"2024-09-17T16:11:25.266055+0000","last_became_peered":"2024-09-17T16:11:25.266055+0000","last_unstale":"2024-09-17T16:13:40.393310+0000","last_undegraded":"2024-09-17T16:13:40.393310+0000","last_fullsized":"2024-09-17T16:13:40.393310+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"132'1495","last_scrub_stamp":"2024-09-17T16:11:55.554231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:55.554231+0000","objects_scrubbed":1100,"log_size":4657,"log_dups_size":0,"ondisk_log_size":4657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:24.678149+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25034752,"num_objects":382,"num_object_clones":0,"num_object_copies":1146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3162,"num_write_kb":78208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"194'4828","reported_seq":6952,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:41.364840+0000","last_change":"2024-09-17T16:13:12.481602+0000","last_active":"2024-09-17T16:13:41.364840+0000","last_peered":"2024-09-17T16:13:41.364840+0000","last_clean":"2024-09-17T16:13:41.364840+0000","last_became_active":"2024-09-17T16:11:23.302908+0000","last_became_peered":"2024-09-17T16:11:23.302908+0000","last_unstale":"2024-09-17T16:13:41.364840+0000","last_undegraded":"2024-09-17T16:13:41.364840+0000","last_fullsized":"2024-09-17T16:13:41.364840+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"177'4029","last_scrub_stamp":"2024-09-17T16:13:12.481403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:12.481403+0000","objects_scrubbed":1189,"log_size":4828,"log_dups_size":0,"ondisk_log_size":4828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:39.732490+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3337,"num_write_kb":83776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"194'4879","reported_seq":7153,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:40.435944+0000","last_change":"2024-09-17T16:13:12.834609+0000","last_active":"2024-09-17T16:13:40.435944+0000","last_peered":"2024-09-17T16:13:40.435944+0000","last_clean":"2024-09-17T16:13:40.435944+0000","last_became_active":"2024-09-17T16:11:27.290863+0000","last_became_peered":"2024-09-17T16:11:27.290863+0000","last_unstale":"2024-09-17T16:13:40.435944+0000","last_undegraded":"2024-09-17T16:13:40.435944+0000","last_fullsized":"2024-09-17T16:13:40.435944+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":5,"last_scrub":"178'4116","last_scrub_stamp":"2024-09-17T16:13:12.834492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:12.834492+0000","objects_scrubbed":1189,"log_size":4879,"log_dups_size":0,"ondisk_log_size":4879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:36.962652+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":423,"num_object_clones":0,"num_object_copies":1269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3218,"num_write_kb":77376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"194'4696","reported_seq":6711,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:41.330945+0000","last_change":"2024-09-17T16:11:53.132205+0000","last_active":"2024-09-17T16:13:41.330945+0000","last_peered":"2024-09-17T16:13:41.330945+0000","last_clean":"2024-09-17T16:13:41.330945+0000","last_became_active":"2024-09-17T16:11:22.310367+0000","last_became_peered":"2024-09-17T16:11:22.310367+0000","last_unstale":"2024-09-17T16:13:41.330945+0000","last_undegraded":"2024-09-17T16:13:41.330945+0000","last_fullsized":"2024-09-17T16:13:41.330945+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"132'1415","last_scrub_stamp":"2024-09-17T16:11:53.132100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:53.132100+0000","objects_scrubbed":973,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:24.400146+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24510464,"num_objects":374,"num_object_clones":0,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3287,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"194'4847","reported_seq":12305,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:40.636190+0000","last_change":"2024-09-17T16:13:27.714585+0000","last_active":"2024-09-17T16:13:40.636190+0000","last_peered":"2024-09-17T16:13:40.636190+0000","last_clean":"2024-09-17T16:13:40.636190+0000","last_became_active":"2024-09-17T16:12:56.327408+0000","last_became_peered":"2024-09-17T16:12:56.327408+0000","last_unstale":"2024-09-17T16:13:40.636190+0000","last_undegraded":"2024-09-17T16:13:40.636190+0000","last_fullsized":"2024-09-17T16:13:40.636190+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'4560","last_scrub_stamp":"2024-09-17T16:13:27.714377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:27.714377+0000","objects_scrubbed":720,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:40.853484+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28246016,"num_objects":431,"num_object_clones":0,"num_object_copies":1293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":289,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,7],"acting":[10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"194'4815","reported_seq":7111,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:40.461857+0000","last_change":"2024-09-17T16:12:45.621594+0000","last_active":"2024-09-17T16:13:40.461857+0000","last_peered":"2024-09-17T16:13:40.461857+0000","last_clean":"2024-09-17T16:13:40.461857+0000","last_became_active":"2024-09-17T16:11:30.264463+0000","last_became_peered":"2024-09-17T16:11:30.264463+0000","last_unstale":"2024-09-17T16:13:40.461857+0000","last_undegraded":"2024-09-17T16:13:40.461857+0000","last_fullsized":"2024-09-17T16:13:40.461857+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":5,"last_scrub":"158'3284","last_scrub_stamp":"2024-09-17T16:12:45.621470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:45.621470+0000","objects_scrubbed":1937,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:00.545203+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":101440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"193'4711","reported_seq":6268,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:39.299602+0000","last_change":"2024-09-17T16:12:22.182841+0000","last_active":"2024-09-17T16:13:39.299602+0000","last_peered":"2024-09-17T16:13:39.299602+0000","last_clean":"2024-09-17T16:13:39.299602+0000","last_became_active":"2024-09-17T16:11:21.266149+0000","last_became_peered":"2024-09-17T16:11:21.266149+0000","last_unstale":"2024-09-17T16:13:39.299602+0000","last_undegraded":"2024-09-17T16:13:39.299602+0000","last_fullsized":"2024-09-17T16:13:39.299602+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'2377","last_scrub_stamp":"2024-09-17T16:12:22.182710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:22.182710+0000","objects_scrubbed":1944,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":484,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2340,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"194'8591","reported_seq":8804,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:39.391627+0000","last_change":"2024-09-17T16:10:54.283333+0000","last_active":"2024-09-17T16:13:39.391627+0000","last_peered":"2024-09-17T16:13:39.391627+0000","last_clean":"2024-09-17T16:13:39.391627+0000","last_became_active":"2024-09-17T16:10:54.282931+0000","last_became_peered":"2024-09-17T16:10:54.282931+0000","last_unstale":"2024-09-17T16:13:39.391627+0000","last_undegraded":"2024-09-17T16:13:39.391627+0000","last_fullsized":"2024-09-17T16:13:39.391627+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62586880,"num_objects":955,"num_object_clones":0,"num_object_copies":2865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8591,"num_write_kb":305472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"194'8787","reported_seq":9034,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:41.649943+0000","last_change":"2024-09-17T16:12:01.681365+0000","last_active":"2024-09-17T16:13:41.649943+0000","last_peered":"2024-09-17T16:13:41.649943+0000","last_clean":"2024-09-17T16:13:41.649943+0000","last_became_active":"2024-09-17T16:10:54.283208+0000","last_became_peered":"2024-09-17T16:10:54.283208+0000","last_unstale":"2024-09-17T16:13:41.649943+0000","last_undegraded":"2024-09-17T16:13:41.649943+0000","last_fullsized":"2024-09-17T16:13:41.649943+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'2642","last_scrub_stamp":"2024-09-17T16:12:01.681123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:01.681123+0000","objects_scrubbed":2614,"log_size":8787,"log_dups_size":0,"ondisk_log_size":8787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:17.223777+0000","scrub_duration":684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48562176,"num_objects":741,"num_object_clones":0,"num_object_copies":2223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8787,"num_write_kb":304896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"194'8789","reported_seq":9047,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:41.358323+0000","last_change":"2024-09-17T16:12:18.766646+0000","last_active":"2024-09-17T16:13:41.358323+0000","last_peered":"2024-09-17T16:13:41.358323+0000","last_clean":"2024-09-17T16:13:41.358323+0000","last_became_active":"2024-09-17T16:10:54.283016+0000","last_became_peered":"2024-09-17T16:10:54.283016+0000","last_unstale":"2024-09-17T16:13:41.358323+0000","last_undegraded":"2024-09-17T16:13:41.358323+0000","last_fullsized":"2024-09-17T16:13:41.358323+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3627","last_scrub_stamp":"2024-09-17T16:12:18.766492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:18.766492+0000","objects_scrubbed":3600,"log_size":8789,"log_dups_size":0,"ondisk_log_size":8789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:13:46.143371+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48234520,"num_objects":737,"num_object_clones":0,"num_object_copies":2211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":737,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8789,"num_write_kb":304769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"194'8594","reported_seq":8806,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:39.380182+0000","last_change":"2024-09-17T16:10:54.282319+0000","last_active":"2024-09-17T16:13:39.380182+0000","last_peered":"2024-09-17T16:13:39.380182+0000","last_clean":"2024-09-17T16:13:39.380182+0000","last_became_active":"2024-09-17T16:10:54.282069+0000","last_became_peered":"2024-09-17T16:10:54.282069+0000","last_unstale":"2024-09-17T16:13:39.380182+0000","last_undegraded":"2024-09-17T16:13:39.380182+0000","last_fullsized":"2024-09-17T16:13:39.380182+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":8594,"log_dups_size":0,"ondisk_log_size":8594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:38.865424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55312384,"num_objects":844,"num_object_clones":0,"num_object_copies":2532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8594,"num_write_kb":302016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":247,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-17T16:13:37.354001+0000","last_change":"2024-09-17T16:13:36.826049+0000","last_active":"2024-09-17T16:13:37.354001+0000","last_peered":"2024-09-17T16:13:37.354001+0000","last_clean":"2024-09-17T16:13:37.354001+0000","last_became_active":"2024-09-17T16:12:39.286388+0000","last_became_peered":"2024-09-17T16:12:39.286388+0000","last_unstale":"2024-09-17T16:13:37.354001+0000","last_undegraded":"2024-09-17T16:13:37.354001+0000","last_fullsized":"2024-09-17T16:13:37.354001+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:13:36.825983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:13:36.825983+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:54.394135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"194'4856","reported_seq":8100,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:40.417028+0000","last_change":"2024-09-17T16:13:22.680448+0000","last_active":"2024-09-17T16:13:40.417028+0000","last_peered":"2024-09-17T16:13:40.417028+0000","last_clean":"2024-09-17T16:13:40.417028+0000","last_became_active":"2024-09-17T16:12:52.302562+0000","last_became_peered":"2024-09-17T16:12:52.302562+0000","last_unstale":"2024-09-17T16:13:40.417028+0000","last_undegraded":"2024-09-17T16:13:40.417028+0000","last_fullsized":"2024-09-17T16:13:40.417028+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:13:52.659598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26804224,"num_objects":409,"num_object_clones":0,"num_object_copies":1227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4435,"num_write_kb":155008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1497,"num_bytes_recovered":80609280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"193'4595","reported_seq":7734,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-17T16:13:37.707049+0000","last_change":"2024-09-17T16:13:33.904714+0000","last_active":"2024-09-17T16:13:37.707049+0000","last_peered":"2024-09-17T16:13:37.707049+0000","last_clean":"2024-09-17T16:13:37.707049+0000","last_became_active":"2024-09-17T16:13:03.359631+0000","last_became_peered":"2024-09-17T16:13:03.359631+0000","last_unstale":"2024-09-17T16:13:37.707049+0000","last_undegraded":"2024-09-17T16:13:37.707049+0000","last_fullsized":"2024-09-17T16:13:37.707049+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'4499","last_scrub_stamp":"2024-09-17T16:13:33.904633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:33.904633+0000","objects_scrubbed":560,"log_size":4595,"log_dups_size":0,"ondisk_log_size":4595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:35.002572+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343168,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"194'4893","reported_seq":5147,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:41.621260+0000","last_change":"2024-09-17T16:12:16.414710+0000","last_active":"2024-09-17T16:13:41.621260+0000","last_peered":"2024-09-17T16:13:41.621260+0000","last_clean":"2024-09-17T16:13:41.621260+0000","last_became_active":"2024-09-17T16:11:13.356367+0000","last_became_peered":"2024-09-17T16:11:13.356367+0000","last_unstale":"2024-09-17T16:13:41.621260+0000","last_undegraded":"2024-09-17T16:13:41.621260+0000","last_fullsized":"2024-09-17T16:13:41.621260+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2190","last_scrub_stamp":"2024-09-17T16:12:16.414644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:16.414644+0000","objects_scrubbed":1711,"log_size":4893,"log_dups_size":0,"ondisk_log_size":4893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:20.196085+0000","scrub_duration":347,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25886720,"num_objects":395,"num_object_clones":0,"num_object_copies":1185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2712,"num_write_kb":44672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"194'4784","reported_seq":5051,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:40.395122+0000","last_change":"2024-09-17T16:13:32.993292+0000","last_active":"2024-09-17T16:13:40.395122+0000","last_peered":"2024-09-17T16:13:40.395122+0000","last_clean":"2024-09-17T16:13:40.395122+0000","last_became_active":"2024-09-17T16:11:13.352761+0000","last_became_peered":"2024-09-17T16:11:13.352761+0000","last_unstale":"2024-09-17T16:13:40.395122+0000","last_undegraded":"2024-09-17T16:13:40.395122+0000","last_fullsized":"2024-09-17T16:13:40.395122+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4605","last_scrub_stamp":"2024-09-17T16:13:32.993228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:32.993228+0000","objects_scrubbed":582,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:47.865099+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26279936,"num_objects":401,"num_object_clones":0,"num_object_copies":1203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2644,"num_write_kb":42368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"194'4898","reported_seq":9177,"reported_epoch":194,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-17T16:13:41.632481+0000","last_change":"2024-09-17T16:13:29.729752+0000","last_active":"2024-09-17T16:13:41.632481+0000","last_peered":"2024-09-17T16:13:41.632481+0000","last_clean":"2024-09-17T16:13:27.758735+0000","last_became_active":"2024-09-17T16:13:29.693012+0000","last_became_peered":"2024-09-17T16:13:29.693012+0000","last_unstale":"2024-09-17T16:13:41.632481+0000","last_undegraded":"2024-09-17T16:13:29.306887+0000","last_fullsized":"2024-09-17T16:13:29.296055+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'1233","last_scrub_stamp":"2024-09-17T16:11:30.184562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:30.184562+0000","objects_scrubbed":777,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1383,"num_bytes_recovered":48365568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"0,1","objects":359}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"194'4835","reported_seq":5088,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:39.299633+0000","last_change":"2024-09-17T16:12:15.296063+0000","last_active":"2024-09-17T16:13:39.299633+0000","last_peered":"2024-09-17T16:13:39.299633+0000","last_clean":"2024-09-17T16:13:39.299633+0000","last_became_active":"2024-09-17T16:11:13.348959+0000","last_became_peered":"2024-09-17T16:11:13.348959+0000","last_unstale":"2024-09-17T16:13:39.299633+0000","last_undegraded":"2024-09-17T16:13:39.299633+0000","last_fullsized":"2024-09-17T16:13:39.299633+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2146","last_scrub_stamp":"2024-09-17T16:12:15.295952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:15.295952+0000","objects_scrubbed":1680,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":439,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31326208,"num_objects":478,"num_object_clones":0,"num_object_copies":1434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2702,"num_write_kb":48000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"194'4861","reported_seq":5111,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:41.549691+0000","last_change":"2024-09-17T16:12:18.628732+0000","last_active":"2024-09-17T16:13:41.549691+0000","last_peered":"2024-09-17T16:13:41.549691+0000","last_clean":"2024-09-17T16:13:41.549691+0000","last_became_active":"2024-09-17T16:11:13.354186+0000","last_became_peered":"2024-09-17T16:11:13.354186+0000","last_unstale":"2024-09-17T16:13:41.549691+0000","last_undegraded":"2024-09-17T16:13:41.549691+0000","last_fullsized":"2024-09-17T16:13:41.549691+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'2226","last_scrub_stamp":"2024-09-17T16:12:18.628468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:18.628468+0000","objects_scrubbed":1798,"log_size":4861,"log_dups_size":0,"ondisk_log_size":4861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25821184,"num_objects":394,"num_object_clones":0,"num_object_copies":1182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2646,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"194'4837","reported_seq":5106,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:40.421138+0000","last_change":"2024-09-17T16:13:35.606144+0000","last_active":"2024-09-17T16:13:40.421138+0000","last_peered":"2024-09-17T16:13:40.421138+0000","last_clean":"2024-09-17T16:13:40.421138+0000","last_became_active":"2024-09-17T16:11:13.352397+0000","last_became_peered":"2024-09-17T16:11:13.352397+0000","last_unstale":"2024-09-17T16:13:40.421138+0000","last_undegraded":"2024-09-17T16:13:40.421138+0000","last_fullsized":"2024-09-17T16:13:40.421138+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'4691","last_scrub_stamp":"2024-09-17T16:13:35.606083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:35.606083+0000","objects_scrubbed":518,"log_size":4837,"log_dups_size":0,"ondisk_log_size":4837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:44.819021+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379392,"num_objects":372,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2615,"num_write_kb":37504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"193'4820","reported_seq":8047,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-17T16:13:37.793694+0000","last_change":"2024-09-17T16:13:08.898511+0000","last_active":"2024-09-17T16:13:37.793694+0000","last_peered":"2024-09-17T16:13:37.793694+0000","last_clean":"2024-09-17T16:13:37.793694+0000","last_became_active":"2024-09-17T16:13:08.416136+0000","last_became_peered":"2024-09-17T16:13:08.416136+0000","last_unstale":"2024-09-17T16:13:37.793694+0000","last_undegraded":"2024-09-17T16:13:37.793694+0000","last_fullsized":"2024-09-17T16:13:37.793694+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'4006","last_scrub_stamp":"2024-09-17T16:13:08.898311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:08.898311+0000","objects_scrubbed":1303,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:17.866499+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31916032,"num_objects":487,"num_object_clones":0,"num_object_copies":1461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":816,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"194'4964","reported_seq":8335,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:41.605972+0000","last_change":"2024-09-17T16:13:28.524306+0000","last_active":"2024-09-17T16:13:41.605972+0000","last_peered":"2024-09-17T16:13:41.605972+0000","last_clean":"2024-09-17T16:13:41.605972+0000","last_became_active":"2024-09-17T16:12:57.338076+0000","last_became_peered":"2024-09-17T16:12:57.338076+0000","last_unstale":"2024-09-17T16:13:41.605972+0000","last_undegraded":"2024-09-17T16:13:41.605972+0000","last_fullsized":"2024-09-17T16:13:41.605972+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'4624","last_scrub_stamp":"2024-09-17T16:13:28.524255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:28.524255+0000","objects_scrubbed":707,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:33.220937+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23920640,"num_objects":365,"num_object_clones":0,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":342,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"193'8526","reported_seq":8771,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:39.372553+0000","last_change":"2024-09-17T16:11:58.924449+0000","last_active":"2024-09-17T16:13:39.372553+0000","last_peered":"2024-09-17T16:13:39.372553+0000","last_clean":"2024-09-17T16:13:39.372553+0000","last_became_active":"2024-09-17T16:10:54.283026+0000","last_became_peered":"2024-09-17T16:10:54.283026+0000","last_unstale":"2024-09-17T16:13:39.372553+0000","last_undegraded":"2024-09-17T16:13:39.372553+0000","last_fullsized":"2024-09-17T16:13:39.372553+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'2421","last_scrub_stamp":"2024-09-17T16:11:58.924214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:58.924214+0000","objects_scrubbed":2408,"log_size":8526,"log_dups_size":0,"ondisk_log_size":8526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56754176,"num_objects":866,"num_object_clones":0,"num_object_copies":2598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8526,"num_write_kb":300544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"194'8722","reported_seq":14819,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-17T16:13:39.299606+0000","last_change":"2024-09-17T16:13:36.362764+0000","last_active":"2024-09-17T16:13:39.299606+0000","last_peered":"2024-09-17T16:13:39.299606+0000","last_clean":"2024-09-17T16:13:39.299606+0000","last_became_active":"2024-09-17T16:13:36.345378+0000","last_became_peered":"2024-09-17T16:13:36.345378+0000","last_unstale":"2024-09-17T16:13:39.299606+0000","last_undegraded":"2024-09-17T16:13:39.299606+0000","last_fullsized":"2024-09-17T16:13:39.299606+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":8722,"log_dups_size":0,"ondisk_log_size":8722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58327040,"num_objects":890,"num_object_clones":0,"num_object_copies":2670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8722,"num_write_kb":307584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2936,"num_bytes_recovered":160694272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":866910232,"num_objects":13229,"num_object_clones":0,"num_object_copies":39687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100602,"num_write_kb":2873025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5816,"num_bytes_recovered":289669120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2682269696,"data_stored":2682257480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38296,"internal_metadata":0},"log_size":148288,"ondisk_log_size":148288,"up":78,"acting":77,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285618,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":266556,"kb_used_data":199276,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93478596,"statfs":{"total":95995035648,"available":95722082304,"internally_reserved":0,"allocated":204058624,"data_stored":203562324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":68876740},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547254,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":199556,"kb_used_data":132272,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93545596,"statfs":{"total":95995035648,"available":95790690304,"internally_reserved":0,"allocated":135446528,"data_stored":134949310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743484,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":467908,"kb_used_data":368628,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93277244,"statfs":{"total":95995035648,"available":95515897856,"internally_reserved":0,"allocated":377475072,"data_stored":376973782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005120,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":289016,"kb_used_data":213348,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93456136,"statfs":{"total":95995035648,"available":95699083264,"internally_reserved":0,"allocated":218468352,"data_stored":217975624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963597,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348988,"kb_used_data":256108,"kb_used_omap":41,"kb_used_meta":92822,"kb_avail":93396164,"statfs":{"total":95995035648,"available":95637671936,"internally_reserved":0,"allocated":262254592,"data_stored":261758312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":95050035},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":251468,"kb_used_data":180024,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93493684,"statfs":{"total":95995035648,"available":95737532416,"internally_reserved":0,"allocated":184344576,"data_stored":183843806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724622,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212416,"kb_used_data":140976,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93532736,"statfs":{"total":95995035648,"available":95777521664,"internally_reserved":0,"allocated":144359424,"data_stored":143862206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986260,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311372,"kb_used_data":235764,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93433780,"statfs":{"total":95995035648,"available":95676190720,"internally_reserved":0,"allocated":241422336,"data_stored":240921046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":450760,"kb_used_data":343220,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93294392,"statfs":{"total":95995035648,"available":95533457408,"internally_reserved":0,"allocated":351457280,"data_stored":350955990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":261248,"kb_used_data":198128,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93483904,"statfs":{"total":95995035648,"available":95727517696,"internally_reserved":0,"allocated":202883072,"data_stored":202385854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":346688,"kb_used_data":239152,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93398464,"statfs":{"total":95995035648,"available":95640027136,"internally_reserved":0,"allocated":244891648,"data_stored":244394430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":255428,"kb_used_data":188080,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93489724,"statfs":{"total":95995035648,"available":95733477376,"internally_reserved":0,"allocated":192593920,"data_stored":192096702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":178651136,"data_stored":178651136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242876416,"data_stored":242876416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194183168,"data_stored":194183168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331616256,"data_stored":331612184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":237506560,"data_stored":237502488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":135135232,"data_stored":135135232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":182190080,"data_stored":182190080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":259588096,"data_stored":259588096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38296,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":215285760,"data_stored":215285760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":375787520,"data_stored":375783448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":132055040,"data_stored":132055040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":197394432,"data_stored":197394432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:13:44.317 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:44.324 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:44.356 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:44.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:44.367 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:13:44.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:44.477 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/22146531' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:44.477 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:44.478 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:44 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 17/39693 objects degraded (0.043%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:44.487 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:44.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:44.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/22146531' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:44.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:44 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 17/39693 objects degraded (0.043%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:44.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:44.596 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:44.596 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:44.614 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:44.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:44.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/22146531' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:13:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:44 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 17/39693 objects degraded (0.043%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:44.716 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:44.747 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:44.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:44.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:44.849 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:44.875 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:44.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:44.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:45.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:45.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:45.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:45.133 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:45.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:45.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:45.243 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:13:45.257 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:45.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:45.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:45.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:45.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:45.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:45.507 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:45.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:45.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:45 smithi071 ceph-mon[30002]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:45.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:45 smithi071 ceph-mon[30002]: pgmap v481: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 808 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 17/38769 objects degraded (0.044%) 2024-09-17T16:13:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:45 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:45 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:45 smithi071 ceph-mon[30002]: osdmap e195: 12 total, 12 up, 12 in 2024-09-17T16:13:45.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:45.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:45.632 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:45.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:45.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:45 smithi173 ceph-mon[30530]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:45.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:45 smithi173 ceph-mon[30530]: pgmap v481: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 808 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 17/38769 objects degraded (0.044%) 2024-09-17T16:13:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:45 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:45 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:45 smithi173 ceph-mon[30530]: osdmap e195: 12 total, 12 up, 12 in 2024-09-17T16:13:45.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:45.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:45 smithi017 ceph-mon[27571]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:45.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:45 smithi017 ceph-mon[27571]: pgmap v481: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 808 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 17/38769 objects degraded (0.044%) 2024-09-17T16:13:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:45 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:45 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:13:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:45 smithi017 ceph-mon[27571]: osdmap e195: 12 total, 12 up, 12 in 2024-09-17T16:13:45.754 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:45.755 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:45.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:45.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:45.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:46.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:46.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:46.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:46.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:46.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:46.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:46.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:46.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:46.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:46.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:46.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:46.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:46.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:46.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:46.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:46.639 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:46.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:46 smithi173 ceph-mon[30530]: pgmap v483: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 621 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 17/29805 objects degraded (0.057%) 2024-09-17T16:13:46.664 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:46.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:46.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:46.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:46 smithi017 ceph-mon[27571]: pgmap v483: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 621 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 17/29805 objects degraded (0.057%) 2024-09-17T16:13:46.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:46.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:46.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:46.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:46.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:46 smithi071 ceph-mon[30002]: pgmap v483: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 621 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 17/29805 objects degraded (0.057%) 2024-09-17T16:13:46.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:46.916 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:46.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:46.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:47.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:47.042 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:47.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:47.144 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:47.170 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:47.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:47.272 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:47.273 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:13:47.298 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:47.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:47.400 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:47.425 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:47.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:47.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:47.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:47.551 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:47.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:47.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:47.674 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:47.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:47.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:47.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:47.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:47.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:47.931 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:47.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:48.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:48.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:48.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:48.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:48.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:48.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:48.292 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:48.319 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:48.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:48.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:48.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:48.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:48.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:48.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:48.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:48.585 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:48.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:48.709 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:48.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:48.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:48.811 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:48.842 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:48.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:48.905 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:13:48.905 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:13:48.939 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:13:48.939 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:13:48.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:48.966 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:48.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:48.972 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:13:48.973 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:13:49.005 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:13:49.005 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:13:49.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:48 smithi017 ceph-mon[27571]: pgmap v484: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 498 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 17/23901 objects degraded (0.071%); 0 B/s, 113 objects/s recovering 2024-09-17T16:13:49.040 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:13:49.040 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:13:49.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:49.078 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:13:49.078 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:13:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:49.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:49.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:49.114 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:13:49.114 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:13:49.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:48 smithi173 ceph-mon[30530]: pgmap v484: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 498 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 17/23901 objects degraded (0.071%); 0 B/s, 113 objects/s recovering 2024-09-17T16:13:49.145 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:49.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:13:49.157 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:13:49.168 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:49.192 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:13:49.192 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:13:49.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:49.220 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:49.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:49.227 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:13:49.228 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:13:49.261 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:13:49.261 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:13:49.294 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:13:49.294 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:13:49.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:48 smithi071 ceph-mon[30002]: pgmap v484: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 498 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.13k op/s; 17/23901 objects degraded (0.071%); 0 B/s, 113 objects/s recovering 2024-09-17T16:13:49.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:49.344 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:49.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:49.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:49.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:49.468 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:49.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:49.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:49.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:49.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:49.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:49.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:49.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:49.831 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:49.854 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:49.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:49.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:49.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:49.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:49.989 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:49.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:50.090 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:50.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:50.117 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:50.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:50.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3873602277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:50.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:49 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 17/23901 objects degraded (0.071%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:50.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:49 smithi173 ceph-mon[30530]: osdmap e196: 12 total, 12 up, 12 in 2024-09-17T16:13:50.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3873602277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:50.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:49 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 17/23901 objects degraded (0.071%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:50.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:49 smithi017 ceph-mon[27571]: osdmap e196: 12 total, 12 up, 12 in 2024-09-17T16:13:50.202 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:13:50.203 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:13:50.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.203 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:149775 2.c e196/95 2:3c1d57d3:::benchmark_data_smithi173_55121_object73286:head v 196'9249, mlcod=196'9247)", 2024-09-17T16:13:50.203 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:50.197161+0000", 2024-09-17T16:13:50.203 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.0033280240000000002, 2024-09-17T16:13:50.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033542680000000001, 2024-09-17T16:13:50.203 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:13:50.203 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.197161+0000", 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.197161+0000", 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.197162+0000", 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2e-07 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.197167+0000", 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7190000000000001e-06 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.197193+0000", 2024-09-17T16:13:50.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6690999999999999e-05 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.197199+0000", 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8080000000000001e-06 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.197295+0000", 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5580000000000003e-05 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:13:50.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.197313+0000", 2024-09-17T16:13:50.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8349e-05 2024-09-17T16:13:50.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:13:50.207 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:13:50.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:13:50.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.207 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:149776 2.1 e196/104 2:80fbb945:::benchmark_data_smithi173_55121_object73287:head v 196'5191, mlcod=196'5187)", 2024-09-17T16:13:50.207 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:50.199537+0000", 2024-09-17T16:13:50.207 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.000952438, 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001032769, 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.199537+0000", 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:13:50.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.199537+0000", 2024-09-17T16:13:50.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:13:50.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:13:50.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.199538+0000", 2024-09-17T16:13:50.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0999999999999997e-07 2024-09-17T16:13:50.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.199545+0000", 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1500000000000002e-06 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.199613+0000", 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8035999999999997e-05 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.199621+0000", 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9230000000000002e-06 2024-09-17T16:13:50.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.199838+0000", 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021657999999999999 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.199856+0000", 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8749e-05 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:13:50.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:149780 2.8 e196/174 2:10aa3bf3:::benchmark_data_smithi173_55121_object73291:head v 196'5224, mlcod=196'5221)", 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:50.200160+0000", 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.00032971500000000001, 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045722500000000001, 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.200160+0000", 2024-09-17T16:13:50.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.200160+0000", 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.200160+0000", 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5600000000000005e-07 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:13:50.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.200166+0000", 2024-09-17T16:13:50.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4349999999999996e-06 2024-09-17T16:13:50.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:13:50.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.200189+0000", 2024-09-17T16:13:50.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2787000000000001e-05 2024-09-17T16:13:50.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.200195+0000", 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6050000000000003e-06 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.200236+0000", 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0623000000000001e-05 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:13:50.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:50.200253+0000", 2024-09-17T16:13:50.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6850999999999998e-05 2024-09-17T16:13:50.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:13:50.216 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:13:50.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:13:50.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:13:50.216 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:13:50.216 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 3 2024-09-17T16:13:50.216 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:13:50.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:50.240 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:50.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:50.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3873602277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:49 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 17/23901 objects degraded (0.071%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:13:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:49 smithi071 ceph-mon[30002]: osdmap e196: 12 total, 12 up, 12 in 2024-09-17T16:13:50.343 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:50.369 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:50.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:50.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:50.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:50.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:50.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:50.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:50.623 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:50.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:50.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:50.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:50.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:50.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:50.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:50.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:50.972 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:50.994 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:50.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:50.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:51.003 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:13:51.003 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:13:51.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:51.101 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:13:51.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:51.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:51.220 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:13:51.242 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:51.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:13:51.344 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:51.371 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:51.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:51.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:51.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:51 smithi017 ceph-mon[27571]: pgmap v486: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 386 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 883 op/s; 17/18489 objects degraded (0.092%); 0 B/s, 50 objects/s recovering 2024-09-17T16:13:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:51 smithi017 ceph-mon[27571]: osdmap e197: 12 total, 12 up, 12 in 2024-09-17T16:13:51.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:51.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:51 smithi071 ceph-mon[30002]: pgmap v486: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 386 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 883 op/s; 17/18489 objects degraded (0.092%); 0 B/s, 50 objects/s recovering 2024-09-17T16:13:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:51 smithi071 ceph-mon[30002]: osdmap e197: 12 total, 12 up, 12 in 2024-09-17T16:13:51.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:51.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:51.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:51.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:51 smithi173 ceph-mon[30530]: pgmap v486: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 386 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 883 op/s; 17/18489 objects degraded (0.092%); 0 B/s, 50 objects/s recovering 2024-09-17T16:13:51.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:51 smithi173 ceph-mon[30530]: osdmap e197: 12 total, 12 up, 12 in 2024-09-17T16:13:51.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:51.742 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:51.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:51.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:51.873 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:51.874 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:51.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:52.001 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:52.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:52.103 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:52.128 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:52.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:52.231 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:52.259 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:52.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:52.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:52.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:52 smithi017 ceph-mon[27571]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:52.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:52 smithi017 ceph-mon[27571]: osdmap e198: 12 total, 12 up, 12 in 2024-09-17T16:13:52.361 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:52.386 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:52.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:52.391 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:13:52.488 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:52.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:52.511 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:52.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:52.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:52.636 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:52.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:52.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:52 smithi173 ceph-mon[30530]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:52.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:52 smithi173 ceph-mon[30530]: osdmap e198: 12 total, 12 up, 12 in 2024-09-17T16:13:52.736 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:52.762 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:52.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:52 smithi071 ceph-mon[30002]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:13:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:52 smithi071 ceph-mon[30002]: osdmap e198: 12 total, 12 up, 12 in 2024-09-17T16:13:52.865 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:52.884 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-17T16:13:50.819992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":322241080,"num_objects":4911,"num_object_clones":0,"num_object_copies":14733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4911,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":109045,"num_write_kb":2875175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6225,"num_bytes_recovered":290198048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156700,"ondisk_log_size":156700,"up":81,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":83,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":2206888,"kb_used_data":1236440,"kb_used_omap":58,"kb_used_meta":970117,"kb_avail":1122734936,"statfs":{"total":1151940427776,"available":1149680574464,"internally_reserved":0,"allocated":1266114560,"data_stored":1260057702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":993399971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-525074432,"num_objects":-8012,"num_object_clones":0,"num_object_copies":-24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8012,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.059568"},"pg_stats":[{"pgid":"2.19","version":"195'4918","reported_seq":6589,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:46.077093+0000","last_change":"2024-09-17T16:13:08.775898+0000","last_active":"2024-09-17T16:13:46.077093+0000","last_peered":"2024-09-17T16:13:46.077093+0000","last_clean":"2024-09-17T16:13:46.077093+0000","last_became_active":"2024-09-17T16:11:24.264470+0000","last_became_peered":"2024-09-17T16:11:24.264470+0000","last_unstale":"2024-09-17T16:13:46.077093+0000","last_undegraded":"2024-09-17T16:13:46.077093+0000","last_fullsized":"2024-09-17T16:13:46.077093+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"175'3870","last_scrub_stamp":"2024-09-17T16:13:08.775814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:08.775814+0000","objects_scrubbed":1265,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:30.625192+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13500416,"num_objects":206,"num_object_clones":0,"num_object_copies":618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3349,"num_write_kb":77952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"195'5108","reported_seq":9427,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:46.352732+0000","last_change":"2024-09-17T16:13:11.523551+0000","last_active":"2024-09-17T16:13:46.352732+0000","last_peered":"2024-09-17T16:13:46.352732+0000","last_clean":"2024-09-17T16:13:46.352732+0000","last_became_active":"2024-09-17T16:13:10.323458+0000","last_became_peered":"2024-09-17T16:13:10.323458+0000","last_unstale":"2024-09-17T16:13:46.352732+0000","last_undegraded":"2024-09-17T16:13:46.352732+0000","last_fullsized":"2024-09-17T16:13:46.352732+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"177'4073","last_scrub_stamp":"2024-09-17T16:13:11.523023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:11.523023+0000","objects_scrubbed":1267,"log_size":5108,"log_dups_size":0,"ondisk_log_size":5108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:25.569087+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14680064,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1043,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"196'5129","reported_seq":5383,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:50.447444+0000","last_change":"2024-09-17T16:12:16.704276+0000","last_active":"2024-09-17T16:13:50.447444+0000","last_peered":"2024-09-17T16:13:50.447444+0000","last_clean":"2024-09-17T16:13:50.447444+0000","last_became_active":"2024-09-17T16:11:13.358625+0000","last_became_peered":"2024-09-17T16:11:13.358625+0000","last_unstale":"2024-09-17T16:13:50.447444+0000","last_undegraded":"2024-09-17T16:13:50.447444+0000","last_fullsized":"2024-09-17T16:13:50.447444+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":5,"last_scrub":"145'2177","last_scrub_stamp":"2024-09-17T16:12:16.704167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:16.704167+0000","objects_scrubbed":1705,"log_size":5129,"log_dups_size":0,"ondisk_log_size":5129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:25.584490+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2964,"num_write_kb":43072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"196'4956","reported_seq":7039,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:50.396908+0000","last_change":"2024-09-17T16:11:55.554301+0000","last_active":"2024-09-17T16:13:50.396908+0000","last_peered":"2024-09-17T16:13:50.396908+0000","last_clean":"2024-09-17T16:13:50.396908+0000","last_became_active":"2024-09-17T16:11:25.266055+0000","last_became_peered":"2024-09-17T16:11:25.266055+0000","last_unstale":"2024-09-17T16:13:50.396908+0000","last_undegraded":"2024-09-17T16:13:50.396908+0000","last_fullsized":"2024-09-17T16:13:50.396908+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"132'1495","last_scrub_stamp":"2024-09-17T16:11:55.554231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:55.554231+0000","objects_scrubbed":1100,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:24.678149+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5439488,"num_objects":83,"num_object_clones":0,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3461,"num_write_kb":78208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"195'4994","reported_seq":7121,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:46.302594+0000","last_change":"2024-09-17T16:13:12.481602+0000","last_active":"2024-09-17T16:13:46.302594+0000","last_peered":"2024-09-17T16:13:46.302594+0000","last_clean":"2024-09-17T16:13:46.302594+0000","last_became_active":"2024-09-17T16:11:23.302908+0000","last_became_peered":"2024-09-17T16:11:23.302908+0000","last_unstale":"2024-09-17T16:13:46.302594+0000","last_undegraded":"2024-09-17T16:13:46.302594+0000","last_fullsized":"2024-09-17T16:13:46.302594+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"177'4029","last_scrub_stamp":"2024-09-17T16:13:12.481403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:12.481403+0000","objects_scrubbed":1189,"log_size":4994,"log_dups_size":0,"ondisk_log_size":4994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:39.732490+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":223,"num_object_clones":0,"num_object_copies":669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":83776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"196'5201","reported_seq":7479,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:50.482657+0000","last_change":"2024-09-17T16:13:12.834609+0000","last_active":"2024-09-17T16:13:50.482657+0000","last_peered":"2024-09-17T16:13:50.482657+0000","last_clean":"2024-09-17T16:13:50.482657+0000","last_became_active":"2024-09-17T16:11:27.290863+0000","last_became_peered":"2024-09-17T16:11:27.290863+0000","last_unstale":"2024-09-17T16:13:50.482657+0000","last_undegraded":"2024-09-17T16:13:50.482657+0000","last_fullsized":"2024-09-17T16:13:50.482657+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":5,"last_scrub":"178'4116","last_scrub_stamp":"2024-09-17T16:13:12.834492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:12.834492+0000","objects_scrubbed":1189,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:36.962652+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3540,"num_write_kb":77376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"195'4855","reported_seq":6872,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:46.326167+0000","last_change":"2024-09-17T16:11:53.132205+0000","last_active":"2024-09-17T16:13:46.326167+0000","last_peered":"2024-09-17T16:13:46.326167+0000","last_clean":"2024-09-17T16:13:46.326167+0000","last_became_active":"2024-09-17T16:11:22.310367+0000","last_became_peered":"2024-09-17T16:11:22.310367+0000","last_unstale":"2024-09-17T16:13:46.326167+0000","last_undegraded":"2024-09-17T16:13:46.326167+0000","last_fullsized":"2024-09-17T16:13:46.326167+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"132'1415","last_scrub_stamp":"2024-09-17T16:11:53.132100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:53.132100+0000","objects_scrubbed":973,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:24.400146+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3446,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"196'5196","reported_seq":12659,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:50.607319+0000","last_change":"2024-09-17T16:13:27.714585+0000","last_active":"2024-09-17T16:13:50.607319+0000","last_peered":"2024-09-17T16:13:50.607319+0000","last_clean":"2024-09-17T16:13:50.607319+0000","last_became_active":"2024-09-17T16:12:56.327408+0000","last_became_peered":"2024-09-17T16:12:56.327408+0000","last_unstale":"2024-09-17T16:13:50.607319+0000","last_undegraded":"2024-09-17T16:13:50.607319+0000","last_fullsized":"2024-09-17T16:13:50.607319+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'4560","last_scrub_stamp":"2024-09-17T16:13:27.714377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:27.714377+0000","objects_scrubbed":720,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:40.853484+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5373952,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,7],"acting":[10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"196'5129","reported_seq":7428,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:50.510527+0000","last_change":"2024-09-17T16:12:45.621594+0000","last_active":"2024-09-17T16:13:50.510527+0000","last_peered":"2024-09-17T16:13:50.510527+0000","last_clean":"2024-09-17T16:13:50.510527+0000","last_became_active":"2024-09-17T16:11:30.264463+0000","last_became_peered":"2024-09-17T16:11:30.264463+0000","last_unstale":"2024-09-17T16:13:50.510527+0000","last_undegraded":"2024-09-17T16:13:50.510527+0000","last_fullsized":"2024-09-17T16:13:50.510527+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":5,"last_scrub":"158'3284","last_scrub_stamp":"2024-09-17T16:12:45.621470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:45.621470+0000","objects_scrubbed":1937,"log_size":5129,"log_dups_size":0,"ondisk_log_size":5129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:00.545203+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":101440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"196'5046","reported_seq":6607,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:49.351648+0000","last_change":"2024-09-17T16:12:22.182841+0000","last_active":"2024-09-17T16:13:49.351648+0000","last_peered":"2024-09-17T16:13:49.351648+0000","last_clean":"2024-09-17T16:13:49.351648+0000","last_became_active":"2024-09-17T16:11:21.266149+0000","last_became_peered":"2024-09-17T16:11:21.266149+0000","last_unstale":"2024-09-17T16:13:49.351648+0000","last_undegraded":"2024-09-17T16:13:49.351648+0000","last_fullsized":"2024-09-17T16:13:49.351648+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'2377","last_scrub_stamp":"2024-09-17T16:12:22.182710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:22.182710+0000","objects_scrubbed":1944,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764864,"num_objects":149,"num_object_clones":0,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2675,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"196'9277","reported_seq":9493,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:49.401483+0000","last_change":"2024-09-17T16:10:54.283333+0000","last_active":"2024-09-17T16:13:49.401483+0000","last_peered":"2024-09-17T16:13:49.401483+0000","last_clean":"2024-09-17T16:13:49.401483+0000","last_became_active":"2024-09-17T16:10:54.282931+0000","last_became_peered":"2024-09-17T16:10:54.282931+0000","last_unstale":"2024-09-17T16:13:49.401483+0000","last_undegraded":"2024-09-17T16:13:49.401483+0000","last_fullsized":"2024-09-17T16:13:49.401483+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":9277,"log_dups_size":0,"ondisk_log_size":9277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:49.125421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17629184,"num_objects":269,"num_object_clones":0,"num_object_copies":807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":305472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"195'9116","reported_seq":9366,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:46.632551+0000","last_change":"2024-09-17T16:12:01.681365+0000","last_active":"2024-09-17T16:13:46.632551+0000","last_peered":"2024-09-17T16:13:46.632551+0000","last_clean":"2024-09-17T16:13:46.632551+0000","last_became_active":"2024-09-17T16:10:54.283208+0000","last_became_peered":"2024-09-17T16:10:54.283208+0000","last_unstale":"2024-09-17T16:13:46.632551+0000","last_undegraded":"2024-09-17T16:13:46.632551+0000","last_fullsized":"2024-09-17T16:13:46.632551+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'2642","last_scrub_stamp":"2024-09-17T16:12:01.681123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:01.681123+0000","objects_scrubbed":2614,"log_size":9116,"log_dups_size":0,"ondisk_log_size":9116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:17.223777+0000","scrub_duration":684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27000832,"num_objects":412,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9116,"num_write_kb":304896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"195'9123","reported_seq":9382,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:46.366435+0000","last_change":"2024-09-17T16:12:18.766646+0000","last_active":"2024-09-17T16:13:46.366435+0000","last_peered":"2024-09-17T16:13:46.366435+0000","last_clean":"2024-09-17T16:13:46.366435+0000","last_became_active":"2024-09-17T16:10:54.283016+0000","last_became_peered":"2024-09-17T16:10:54.283016+0000","last_unstale":"2024-09-17T16:13:46.366435+0000","last_undegraded":"2024-09-17T16:13:46.366435+0000","last_fullsized":"2024-09-17T16:13:46.366435+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3627","last_scrub_stamp":"2024-09-17T16:12:18.766492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:18.766492+0000","objects_scrubbed":3600,"log_size":9123,"log_dups_size":0,"ondisk_log_size":9123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:46.300018+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345496,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9123,"num_write_kb":304769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"196'9196","reported_seq":9412,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:49.348130+0000","last_change":"2024-09-17T16:10:54.282319+0000","last_active":"2024-09-17T16:13:49.348130+0000","last_peered":"2024-09-17T16:13:49.348130+0000","last_clean":"2024-09-17T16:13:49.348130+0000","last_became_active":"2024-09-17T16:10:54.282069+0000","last_became_peered":"2024-09-17T16:10:54.282069+0000","last_unstale":"2024-09-17T16:13:49.348130+0000","last_undegraded":"2024-09-17T16:13:49.348130+0000","last_fullsized":"2024-09-17T16:13:49.348130+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":9196,"log_dups_size":0,"ondisk_log_size":9196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:38.865424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9196,"num_write_kb":302016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":251,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:44.366936+0000","last_change":"2024-09-17T16:13:36.826049+0000","last_active":"2024-09-17T16:13:44.366936+0000","last_peered":"2024-09-17T16:13:44.366936+0000","last_clean":"2024-09-17T16:13:44.366936+0000","last_became_active":"2024-09-17T16:12:39.286388+0000","last_became_peered":"2024-09-17T16:12:39.286388+0000","last_unstale":"2024-09-17T16:13:44.366936+0000","last_undegraded":"2024-09-17T16:13:44.366936+0000","last_fullsized":"2024-09-17T16:13:44.366936+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:13:36.825983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:13:36.825983+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:54.394135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"196'5186","reported_seq":8434,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:50.426878+0000","last_change":"2024-09-17T16:13:22.680448+0000","last_active":"2024-09-17T16:13:50.426878+0000","last_peered":"2024-09-17T16:13:50.426878+0000","last_clean":"2024-09-17T16:13:50.426878+0000","last_became_active":"2024-09-17T16:12:52.302562+0000","last_became_peered":"2024-09-17T16:12:52.302562+0000","last_unstale":"2024-09-17T16:13:50.426878+0000","last_undegraded":"2024-09-17T16:13:50.426878+0000","last_fullsized":"2024-09-17T16:13:50.426878+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":5186,"log_dups_size":0,"ondisk_log_size":5186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:13:52.659598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5177344,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4765,"num_write_kb":155008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1497,"num_bytes_recovered":80609280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"195'4880","reported_seq":8023,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:47.850399+0000","last_change":"2024-09-17T16:13:33.904714+0000","last_active":"2024-09-17T16:13:47.850399+0000","last_peered":"2024-09-17T16:13:47.850399+0000","last_clean":"2024-09-17T16:13:47.850399+0000","last_became_active":"2024-09-17T16:13:03.359631+0000","last_became_peered":"2024-09-17T16:13:03.359631+0000","last_unstale":"2024-09-17T16:13:47.850399+0000","last_undegraded":"2024-09-17T16:13:47.850399+0000","last_fullsized":"2024-09-17T16:13:47.850399+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'4499","last_scrub_stamp":"2024-09-17T16:13:33.904633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:33.904633+0000","objects_scrubbed":560,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:35.002572+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11665408,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":382,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"195'5063","reported_seq":5319,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:46.639962+0000","last_change":"2024-09-17T16:12:16.414710+0000","last_active":"2024-09-17T16:13:46.639962+0000","last_peered":"2024-09-17T16:13:46.639962+0000","last_clean":"2024-09-17T16:13:46.639962+0000","last_became_active":"2024-09-17T16:11:13.356367+0000","last_became_peered":"2024-09-17T16:11:13.356367+0000","last_unstale":"2024-09-17T16:13:46.639962+0000","last_undegraded":"2024-09-17T16:13:46.639962+0000","last_fullsized":"2024-09-17T16:13:46.639962+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2190","last_scrub_stamp":"2024-09-17T16:12:16.414644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:16.414644+0000","objects_scrubbed":1711,"log_size":5063,"log_dups_size":0,"ondisk_log_size":5063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:20.196085+0000","scrub_duration":347,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14745600,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2882,"num_write_kb":44672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"196'5090","reported_seq":5361,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:50.390172+0000","last_change":"2024-09-17T16:13:32.993292+0000","last_active":"2024-09-17T16:13:50.390172+0000","last_peered":"2024-09-17T16:13:50.390172+0000","last_clean":"2024-09-17T16:13:50.390172+0000","last_became_active":"2024-09-17T16:11:13.352761+0000","last_became_peered":"2024-09-17T16:11:13.352761+0000","last_unstale":"2024-09-17T16:13:50.390172+0000","last_undegraded":"2024-09-17T16:13:50.390172+0000","last_fullsized":"2024-09-17T16:13:50.390172+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4605","last_scrub_stamp":"2024-09-17T16:13:32.993228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:32.993228+0000","objects_scrubbed":582,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:47.865099+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6225920,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":42368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"195'5063","reported_seq":10158,"reported_epoch":195,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-17T16:13:46.636514+0000","last_change":"2024-09-17T16:13:29.729752+0000","last_active":"2024-09-17T16:13:46.636514+0000","last_peered":"2024-09-17T16:13:46.636514+0000","last_clean":"2024-09-17T16:13:27.758735+0000","last_became_active":"2024-09-17T16:13:29.693012+0000","last_became_peered":"2024-09-17T16:13:29.693012+0000","last_unstale":"2024-09-17T16:13:46.636514+0000","last_undegraded":"2024-09-17T16:13:29.306887+0000","last_fullsized":"2024-09-17T16:13:29.296055+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'1233","last_scrub_stamp":"2024-09-17T16:11:30.184562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:30.184562+0000","objects_scrubbed":777,"log_size":5063,"log_dups_size":0,"ondisk_log_size":5063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13828096,"num_objects":211,"num_object_clones":0,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3830,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1790,"num_bytes_recovered":48365568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"0,1","objects":194}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"196'5173","reported_seq":5430,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:49.399834+0000","last_change":"2024-09-17T16:12:15.296063+0000","last_active":"2024-09-17T16:13:49.399834+0000","last_peered":"2024-09-17T16:13:49.399834+0000","last_clean":"2024-09-17T16:13:49.399834+0000","last_became_active":"2024-09-17T16:11:13.348959+0000","last_became_peered":"2024-09-17T16:11:13.348959+0000","last_unstale":"2024-09-17T16:13:49.399834+0000","last_undegraded":"2024-09-17T16:13:49.399834+0000","last_fullsized":"2024-09-17T16:13:49.399834+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2146","last_scrub_stamp":"2024-09-17T16:12:15.295952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:15.295952+0000","objects_scrubbed":1680,"log_size":5173,"log_dups_size":0,"ondisk_log_size":5173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":439,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9175040,"num_objects":140,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":48000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"195'5015","reported_seq":5267,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:46.549194+0000","last_change":"2024-09-17T16:12:18.628732+0000","last_active":"2024-09-17T16:13:46.549194+0000","last_peered":"2024-09-17T16:13:46.549194+0000","last_clean":"2024-09-17T16:13:46.549194+0000","last_became_active":"2024-09-17T16:11:13.354186+0000","last_became_peered":"2024-09-17T16:11:13.354186+0000","last_unstale":"2024-09-17T16:13:46.549194+0000","last_undegraded":"2024-09-17T16:13:46.549194+0000","last_fullsized":"2024-09-17T16:13:46.549194+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'2226","last_scrub_stamp":"2024-09-17T16:12:18.628468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:18.628468+0000","objects_scrubbed":1798,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2800,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"196'5131","reported_seq":5404,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:50.515909+0000","last_change":"2024-09-17T16:13:35.606144+0000","last_active":"2024-09-17T16:13:50.515909+0000","last_peered":"2024-09-17T16:13:50.515909+0000","last_clean":"2024-09-17T16:13:50.515909+0000","last_became_active":"2024-09-17T16:11:13.352397+0000","last_became_peered":"2024-09-17T16:11:13.352397+0000","last_unstale":"2024-09-17T16:13:50.515909+0000","last_undegraded":"2024-09-17T16:13:50.515909+0000","last_fullsized":"2024-09-17T16:13:50.515909+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'4691","last_scrub_stamp":"2024-09-17T16:13:35.606083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:35.606083+0000","objects_scrubbed":518,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:44.819021+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":78,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2909,"num_write_kb":37504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"195'5141","reported_seq":8372,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:47.833077+0000","last_change":"2024-09-17T16:13:08.898511+0000","last_active":"2024-09-17T16:13:47.833077+0000","last_peered":"2024-09-17T16:13:47.833077+0000","last_clean":"2024-09-17T16:13:47.833077+0000","last_became_active":"2024-09-17T16:13:08.416136+0000","last_became_peered":"2024-09-17T16:13:08.416136+0000","last_unstale":"2024-09-17T16:13:47.833077+0000","last_undegraded":"2024-09-17T16:13:47.833077+0000","last_fullsized":"2024-09-17T16:13:47.833077+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'4006","last_scrub_stamp":"2024-09-17T16:13:08.898311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:08.898311+0000","objects_scrubbed":1303,"log_size":5141,"log_dups_size":0,"ondisk_log_size":5141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:17.866499+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1137,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"195'5130","reported_seq":8504,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-17T16:13:46.630336+0000","last_change":"2024-09-17T16:13:28.524306+0000","last_active":"2024-09-17T16:13:46.630336+0000","last_peered":"2024-09-17T16:13:46.630336+0000","last_clean":"2024-09-17T16:13:46.630336+0000","last_became_active":"2024-09-17T16:12:57.338076+0000","last_became_peered":"2024-09-17T16:12:57.338076+0000","last_unstale":"2024-09-17T16:13:46.630336+0000","last_undegraded":"2024-09-17T16:13:46.630336+0000","last_fullsized":"2024-09-17T16:13:46.630336+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'4624","last_scrub_stamp":"2024-09-17T16:13:28.524255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:28.524255+0000","objects_scrubbed":707,"log_size":5130,"log_dups_size":0,"ondisk_log_size":5130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:33.220937+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13041664,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":508,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"196'9148","reported_seq":9398,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:49.378601+0000","last_change":"2024-09-17T16:11:58.924449+0000","last_active":"2024-09-17T16:13:49.378601+0000","last_peered":"2024-09-17T16:13:49.378601+0000","last_clean":"2024-09-17T16:13:49.378601+0000","last_became_active":"2024-09-17T16:10:54.283026+0000","last_became_peered":"2024-09-17T16:10:54.283026+0000","last_unstale":"2024-09-17T16:13:49.378601+0000","last_undegraded":"2024-09-17T16:13:49.378601+0000","last_fullsized":"2024-09-17T16:13:49.378601+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'2421","last_scrub_stamp":"2024-09-17T16:11:58.924214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:58.924214+0000","objects_scrubbed":2408,"log_size":9148,"log_dups_size":0,"ondisk_log_size":9148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:42.916375+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15990784,"num_objects":244,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9148,"num_write_kb":300544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"196'9344","reported_seq":15447,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-17T16:13:49.384462+0000","last_change":"2024-09-17T16:13:36.362764+0000","last_active":"2024-09-17T16:13:49.384462+0000","last_peered":"2024-09-17T16:13:49.384462+0000","last_clean":"2024-09-17T16:13:49.384462+0000","last_became_active":"2024-09-17T16:13:36.345378+0000","last_became_peered":"2024-09-17T16:13:36.345378+0000","last_unstale":"2024-09-17T16:13:49.384462+0000","last_undegraded":"2024-09-17T16:13:49.384462+0000","last_fullsized":"2024-09-17T16:13:49.384462+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":9344,"log_dups_size":0,"ondisk_log_size":9344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:45.123161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563648,"num_objects":268,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9344,"num_write_kb":307584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2936,"num_bytes_recovered":160694272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":321650712,"num_objects":4909,"num_object_clones":0,"num_object_copies":14727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108922,"num_write_kb":2873025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6223,"num_bytes_recovered":289669120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171599360,"data_stored":1171587144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38296,"internal_metadata":0},"log_size":156608,"ondisk_log_size":156608,"up":78,"acting":77,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285620,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140308,"kb_used_data":64708,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93604844,"statfs":{"total":95995035648,"available":95851360256,"internally_reserved":0,"allocated":66260992,"data_stored":65755712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":77396420},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547256,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95320,"kb_used_data":28040,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93649832,"statfs":{"total":95995035648,"available":95897427968,"internally_reserved":0,"allocated":28712960,"data_stored":28206762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743485,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320464,"kb_used_data":212928,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93424688,"statfs":{"total":95995035648,"available":95666880512,"internally_reserved":0,"allocated":218038272,"data_stored":217532492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005122,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":135692,"kb_used_data":68412,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93609460,"statfs":{"total":95995035648,"available":95856087040,"internally_reserved":0,"allocated":70053888,"data_stored":69552180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963599,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195156,"kb_used_data":106436,"kb_used_omap":41,"kb_used_meta":88662,"kb_avail":93549996,"statfs":{"total":95995035648,"available":95795195904,"internally_reserved":0,"allocated":108990464,"data_stored":108485204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":90790195},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108768,"kb_used_data":41488,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93636384,"statfs":{"total":95995035648,"available":95883657216,"internally_reserved":0,"allocated":42483712,"data_stored":41973962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724624,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102812,"kb_used_data":39688,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93642340,"statfs":{"total":95995035648,"available":95889756160,"internally_reserved":0,"allocated":40640512,"data_stored":40134314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986262,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":139984,"kb_used_data":72704,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93605168,"statfs":{"total":95995035648,"available":95851692032,"internally_reserved":0,"allocated":74448896,"data_stored":73943116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":332944,"kb_used_data":217088,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93412208,"statfs":{"total":95995035648,"available":95654100992,"internally_reserved":0,"allocated":222298112,"data_stored":221792332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":118618567},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204240,"kb_used_data":136956,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93540912,"statfs":{"total":95995035648,"available":95785893888,"internally_reserved":0,"allocated":140242944,"data_stored":139741236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":250064,"kb_used_data":138364,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93495088,"statfs":{"total":95995035648,"available":95738970112,"internally_reserved":0,"allocated":141684736,"data_stored":141183028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181136,"kb_used_data":109628,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93564016,"statfs":{"total":95995035648,"available":95809552384,"internally_reserved":0,"allocated":112259072,"data_stored":111757364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102498304,"data_stored":102498304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139526144,"data_stored":139526144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133562368,"data_stored":133562368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":201920512,"data_stored":201916440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":66129920,"data_stored":66125848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":31129600,"data_stored":31129600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40173568,"data_stored":40173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":95879168,"data_stored":95879168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38296,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":63438848,"data_stored":63438848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":213782528,"data_stored":213778456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":26869760,"data_stored":26869760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":56688640,"data_stored":56688640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:13:52.886 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-17T16:13:52.889 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:52.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:52.895 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:52.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:53.025 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:53.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:53.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:53.159 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:53.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:53.261 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:53.287 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:53.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:53.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:53.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:53 smithi017 ceph-mon[27571]: pgmap v488: 27 pgs: 1 peering, 26 active+clean; 204 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 0 B/s, 138 objects/s recovering 2024-09-17T16:13:53.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:53 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/18489 objects degraded (0.092%), 1 pg degraded) 2024-09-17T16:13:53.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:53 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:13:53.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:53 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:13:53.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:53.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:53.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:53.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:53.521 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:53.554 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:53.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:53.573 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:53.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:53 smithi173 ceph-mon[30530]: pgmap v488: 27 pgs: 1 peering, 26 active+clean; 204 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 0 B/s, 138 objects/s recovering 2024-09-17T16:13:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:53 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/18489 objects degraded (0.092%), 1 pg degraded) 2024-09-17T16:13:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:53 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:13:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:53 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:13:53.656 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:53.680 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:53.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:53.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:13:53.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:53.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:53.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:53.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:53 smithi071 ceph-mon[30002]: pgmap v488: 27 pgs: 1 peering, 26 active+clean; 204 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.31k op/s; 0 B/s, 138 objects/s recovering 2024-09-17T16:13:53.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:53 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/18489 objects degraded (0.092%), 1 pg degraded) 2024-09-17T16:13:53.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:53 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:13:53.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:53 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:13:53.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:53.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:53.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:54.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:54.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:54.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:54.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:54.176 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:54.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:54.269 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:13:54.269 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:13:54.269 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:13:54.269 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:13:54.269 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:13:54.278 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:54.300 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:54.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:54.307 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:13:54.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:54.425 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:54.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:54.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:54.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:54.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:54.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:54 smithi173 ceph-mon[30530]: 2.3 scrub starts 2024-09-17T16:13:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:54 smithi173 ceph-mon[30530]: osdmap e199: 12 total, 12 up, 12 in 2024-09-17T16:13:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4026115052' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:54 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:54 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:54 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:13:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:54 smithi173 ceph-mon[30530]: osdmap e200: 12 total, 12 up, 12 in 2024-09-17T16:13:54.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:54.669 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 76486 objects 2024-09-17T16:13:54.669 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :81.4451 2024-09-17T16:13:54.676 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:54.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:54 smithi017 ceph-mon[27571]: 2.3 scrub starts 2024-09-17T16:13:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:54 smithi017 ceph-mon[27571]: osdmap e199: 12 total, 12 up, 12 in 2024-09-17T16:13:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4026115052' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:54 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:54 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:54 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:13:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:54 smithi017 ceph-mon[27571]: osdmap e200: 12 total, 12 up, 12 in 2024-09-17T16:13:54.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:54.802 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:54.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:54 smithi071 ceph-mon[30002]: 2.3 scrub starts 2024-09-17T16:13:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:54 smithi071 ceph-mon[30002]: osdmap e199: 12 total, 12 up, 12 in 2024-09-17T16:13:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4026115052' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:54 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:13:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:54 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:13:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:54 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:13:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:54 smithi071 ceph-mon[30002]: osdmap e200: 12 total, 12 up, 12 in 2024-09-17T16:13:54.902 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:54.926 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:54.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:55.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:55.050 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:55.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:55.152 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:55.176 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:55.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:55.178 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:13:55.178 INFO:teuthology.orchestra.run.smithi017.stdout:{"fsid":"19341aec-750e-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":553,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":200,"num_osds":12,"num_up_osds":12,"osd_up_since":1726589558,"num_in_osds":12,"osd_in_since":1726589369,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":26},{"state_name":"peering","count":1}],"num_pgs":27,"num_pools":2,"num_objects":2962,"data_bytes":194511416,"bytes_used":1695920128,"bytes_avail":1150244507648,"bytes_total":1151940427776,"inactive_pgs_ratio":0.037037037312984467,"recovering_objects_per_sec":138,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":619,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1055},"fsmap":{"epoch":1,"btime":"2024-09-17T16:02:46:978559+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.17:8443/"}},"servicemap":{"epoch":20,"modified":"2024-09-17T16:13:39.759509+0000","services":{}},"progress_events":{}} 2024-09-17T16:13:55.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:55.302 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:55.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:55.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:55.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:55.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:55.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:55.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:55 smithi017 ceph-mon[27571]: pgmap v491: 27 pgs: 1 peering, 26 active+clean; 186 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 0 B/s, 138 objects/s recovering 2024-09-17T16:13:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/656380108' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:13:55.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:55.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:55.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:55.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:55 smithi173 ceph-mon[30530]: pgmap v491: 27 pgs: 1 peering, 26 active+clean; 186 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 0 B/s, 138 objects/s recovering 2024-09-17T16:13:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/656380108' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:13:55.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:55.684 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:55.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:55.746 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:13:55.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:55 smithi071 ceph-mon[30002]: pgmap v491: 27 pgs: 1 peering, 26 active+clean; 186 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s; 0 B/s, 138 objects/s recovering 2024-09-17T16:13:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/656380108' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:13:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:55.815 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:55.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:55.916 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:55.942 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:55.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:55.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:55.945 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:56.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:56.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:56.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:56.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:56.195 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:56.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:56.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:56.320 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-17T16:13:56.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:13:56.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:56.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:56.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:56.449 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:56.450 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:56.551 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:56.581 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:56.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:56.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:13:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:56.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:56.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:13:56.797 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:13:56.797 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:13:56.818 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:13:56.818 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:13:56.839 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:13:56.839 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:13:56.859 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:56.861 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:13:56.861 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:13:56.881 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:13:56.881 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:13:56.886 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:56.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:56.903 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:13:56.904 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:13:56.925 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:13:56.925 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:13:56.946 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:13:56.947 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:13:56.969 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:13:56.969 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:13:56.988 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:56.991 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:13:56.991 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:13:57.011 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:57.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:57.014 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:13:57.014 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:13:57.036 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:13:57.036 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:13:57.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:13:57.135 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:57.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:13:57.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:57.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:57.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:57.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:57.388 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:57.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:57.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:57 smithi017 ceph-mon[27571]: pgmap v493: 27 pgs: 1 peering, 26 active+clean; 60 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 0 B/s, 125 objects/s recovering 2024-09-17T16:13:57.469 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4194893975' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:57.489 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:57.512 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:57.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:57.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:57.637 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:57.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:57.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:57.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:57 smithi173 ceph-mon[30530]: pgmap v493: 27 pgs: 1 peering, 26 active+clean; 60 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 0 B/s, 125 objects/s recovering 2024-09-17T16:13:57.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4194893975' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:57.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:57.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:57.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:57 smithi071 ceph-mon[30002]: pgmap v493: 27 pgs: 1 peering, 26 active+clean; 60 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s; 0 B/s, 125 objects/s recovering 2024-09-17T16:13:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4194893975' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:13:57.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:13:57.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:57.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:57.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:13:57.919 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:57.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:58.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:58.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:58.121 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:13:58.146 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:58.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:13:58.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:58.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:58.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:58.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:58.401 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:58.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:58.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:58.425 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:13:58.425 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:13:58.503 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:58.527 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:58.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:58.629 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:58.651 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:58.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:58.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:13:58.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:58.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:13:58.881 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:58.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:58.905 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:58.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:59.006 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:13:59.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:13:59.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:59.033 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:59.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:13:59.067 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-17T16:13:59.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-17T16:13:59.071 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:59.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:13:59.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:13:59.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:59.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:13:59.245 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:13:59.262 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:13:59.311 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:59.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:13:59.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:59.436 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:59.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:59.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:59.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:13:59.564 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:13:59.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:59.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:13:59.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:13:59 smithi017 ceph-mon[27571]: pgmap v494: 27 pgs: 27 active+clean; 4.1 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s 2024-09-17T16:13:59.605 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:13:59.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:13:59 smithi173 ceph-mon[30530]: pgmap v494: 27 pgs: 27 active+clean; 4.1 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s 2024-09-17T16:13:59.666 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:59.688 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:59.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:59.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:59.785 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:13:59.786 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:13:59.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:13:59.813 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:13:59 smithi071 ceph-mon[30002]: pgmap v494: 27 pgs: 27 active+clean; 4.1 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s 2024-09-17T16:13:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:59.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:13:59.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:13:59.938 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:13:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:13:59.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:13:59.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:14:00.040 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:00.066 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:00.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:00.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:00.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:00.195 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:00.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:00.321 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:00.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:00.423 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:00.448 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:00.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:00.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:00.550 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:00.576 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:00.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:00 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:00 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:00 smithi173 ceph-mon[30530]: osdmap e201: 12 total, 12 up, 12 in 2024-09-17T16:14:00.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:00.703 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:00.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:00.723 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:00 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:00.723 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:00 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:00.723 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:00 smithi017 ceph-mon[27571]: osdmap e201: 12 total, 12 up, 12 in 2024-09-17T16:14:00.750 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-17T16:13:59.763984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":113969,"num_write_kb":2875175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6844,"num_bytes_recovered":290198048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161624,"ondisk_log_size":161624,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1005528,"kb_used_data":43384,"kb_used_omap":58,"kb_used_meta":961797,"kb_avail":1123936296,"statfs":{"total":1151940427776,"available":1150910767104,"internally_reserved":0,"allocated":44425216,"data_stored":38124296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":984880291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,1,0,1,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-193921048,"num_objects":-2960,"num_object_clones":0,"num_object_copies":-8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.441156"},"pg_stats":[{"pgid":"2.19","version":"199'5124","reported_seq":6805,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.303764+0000","last_change":"2024-09-17T16:13:08.775898+0000","last_active":"2024-09-17T16:13:54.303764+0000","last_peered":"2024-09-17T16:13:54.303764+0000","last_clean":"2024-09-17T16:13:54.303764+0000","last_became_active":"2024-09-17T16:11:24.264470+0000","last_became_peered":"2024-09-17T16:11:24.264470+0000","last_unstale":"2024-09-17T16:13:54.303764+0000","last_undegraded":"2024-09-17T16:13:54.303764+0000","last_fullsized":"2024-09-17T16:13:54.303764+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":5,"last_scrub":"175'3870","last_scrub_stamp":"2024-09-17T16:13:08.775814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:08.775814+0000","objects_scrubbed":1265,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:30.625192+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3555,"num_write_kb":77952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"199'5332","reported_seq":9661,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.373015+0000","last_change":"2024-09-17T16:13:11.523551+0000","last_active":"2024-09-17T16:13:54.373015+0000","last_peered":"2024-09-17T16:13:54.373015+0000","last_clean":"2024-09-17T16:13:54.373015+0000","last_became_active":"2024-09-17T16:13:10.323458+0000","last_became_peered":"2024-09-17T16:13:10.323458+0000","last_unstale":"2024-09-17T16:13:54.373015+0000","last_undegraded":"2024-09-17T16:13:54.373015+0000","last_fullsized":"2024-09-17T16:13:54.373015+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"177'4073","last_scrub_stamp":"2024-09-17T16:13:11.523023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:11.523023+0000","objects_scrubbed":1267,"log_size":5332,"log_dups_size":0,"ondisk_log_size":5332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:25.569087+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1267,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,7],"acting":[9,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"199'5216","reported_seq":5478,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463827+0000","last_change":"2024-09-17T16:12:16.704276+0000","last_active":"2024-09-17T16:13:54.463827+0000","last_peered":"2024-09-17T16:13:54.463827+0000","last_clean":"2024-09-17T16:13:54.463827+0000","last_became_active":"2024-09-17T16:11:13.358625+0000","last_became_peered":"2024-09-17T16:11:13.358625+0000","last_unstale":"2024-09-17T16:13:54.463827+0000","last_undegraded":"2024-09-17T16:13:54.463827+0000","last_fullsized":"2024-09-17T16:13:54.463827+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":5,"last_scrub":"145'2177","last_scrub_stamp":"2024-09-17T16:12:16.704167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:16.704167+0000","objects_scrubbed":1705,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:25.584490+0000","scrub_duration":445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3051,"num_write_kb":43072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"199'5039","reported_seq":7131,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463460+0000","last_change":"2024-09-17T16:11:55.554301+0000","last_active":"2024-09-17T16:13:54.463460+0000","last_peered":"2024-09-17T16:13:54.463460+0000","last_clean":"2024-09-17T16:13:54.463460+0000","last_became_active":"2024-09-17T16:11:25.266055+0000","last_became_peered":"2024-09-17T16:11:25.266055+0000","last_unstale":"2024-09-17T16:13:54.463460+0000","last_undegraded":"2024-09-17T16:13:54.463460+0000","last_fullsized":"2024-09-17T16:13:54.463460+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"132'1495","last_scrub_stamp":"2024-09-17T16:11:55.554231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:55.554231+0000","objects_scrubbed":1100,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:24.678149+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3544,"num_write_kb":78208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,11],"acting":[5,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"199'5217","reported_seq":7355,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.373149+0000","last_change":"2024-09-17T16:13:12.481602+0000","last_active":"2024-09-17T16:13:54.373149+0000","last_peered":"2024-09-17T16:13:54.373149+0000","last_clean":"2024-09-17T16:13:54.373149+0000","last_became_active":"2024-09-17T16:11:23.302908+0000","last_became_peered":"2024-09-17T16:11:23.302908+0000","last_unstale":"2024-09-17T16:13:54.373149+0000","last_undegraded":"2024-09-17T16:13:54.373149+0000","last_fullsized":"2024-09-17T16:13:54.373149+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"177'4029","last_scrub_stamp":"2024-09-17T16:13:12.481403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:12.481403+0000","objects_scrubbed":1189,"log_size":5217,"log_dups_size":0,"ondisk_log_size":5217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:39.732490+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3726,"num_write_kb":83776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"199'5302","reported_seq":7589,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463779+0000","last_change":"2024-09-17T16:13:12.834609+0000","last_active":"2024-09-17T16:13:54.463779+0000","last_peered":"2024-09-17T16:13:54.463779+0000","last_clean":"2024-09-17T16:13:54.463779+0000","last_became_active":"2024-09-17T16:11:27.290863+0000","last_became_peered":"2024-09-17T16:11:27.290863+0000","last_unstale":"2024-09-17T16:13:54.463779+0000","last_undegraded":"2024-09-17T16:13:54.463779+0000","last_fullsized":"2024-09-17T16:13:54.463779+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":5,"last_scrub":"178'4116","last_scrub_stamp":"2024-09-17T16:13:12.834492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:12.834492+0000","objects_scrubbed":1189,"log_size":5302,"log_dups_size":0,"ondisk_log_size":5302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:36.962652+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3641,"num_write_kb":77376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"199'5070","reported_seq":7097,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.372975+0000","last_change":"2024-09-17T16:11:53.132205+0000","last_active":"2024-09-17T16:13:54.372975+0000","last_peered":"2024-09-17T16:13:54.372975+0000","last_clean":"2024-09-17T16:13:54.372975+0000","last_became_active":"2024-09-17T16:11:22.310367+0000","last_became_peered":"2024-09-17T16:11:22.310367+0000","last_unstale":"2024-09-17T16:13:54.372975+0000","last_undegraded":"2024-09-17T16:13:54.372975+0000","last_fullsized":"2024-09-17T16:13:54.372975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"132'1415","last_scrub_stamp":"2024-09-17T16:11:53.132100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:53.132100+0000","objects_scrubbed":973,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:24.400146+0000","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3661,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.12","version":"199'5278","reported_seq":12749,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463831+0000","last_change":"2024-09-17T16:13:27.714585+0000","last_active":"2024-09-17T16:13:54.463831+0000","last_peered":"2024-09-17T16:13:54.463831+0000","last_clean":"2024-09-17T16:13:54.463831+0000","last_became_active":"2024-09-17T16:12:56.327408+0000","last_became_peered":"2024-09-17T16:12:56.327408+0000","last_unstale":"2024-09-17T16:13:54.463831+0000","last_undegraded":"2024-09-17T16:13:54.463831+0000","last_fullsized":"2024-09-17T16:13:54.463831+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"183'4560","last_scrub_stamp":"2024-09-17T16:13:27.714377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:27.714377+0000","objects_scrubbed":720,"log_size":5278,"log_dups_size":0,"ondisk_log_size":5278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:40.853484+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":720,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,7],"acting":[10,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"199'5219","reported_seq":7527,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463830+0000","last_change":"2024-09-17T16:12:45.621594+0000","last_active":"2024-09-17T16:13:54.463830+0000","last_peered":"2024-09-17T16:13:54.463830+0000","last_clean":"2024-09-17T16:13:54.463830+0000","last_became_active":"2024-09-17T16:11:30.264463+0000","last_became_peered":"2024-09-17T16:11:30.264463+0000","last_unstale":"2024-09-17T16:13:54.463830+0000","last_undegraded":"2024-09-17T16:13:54.463830+0000","last_fullsized":"2024-09-17T16:13:54.463830+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":5,"last_scrub":"158'3284","last_scrub_stamp":"2024-09-17T16:12:45.621470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:45.621470+0000","objects_scrubbed":1937,"log_size":5219,"log_dups_size":0,"ondisk_log_size":5219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:00.545203+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":101440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"199'5195","reported_seq":6766,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T16:13:59.340071+0000","last_change":"2024-09-17T16:12:22.182841+0000","last_active":"2024-09-17T16:13:59.340071+0000","last_peered":"2024-09-17T16:13:59.340071+0000","last_clean":"2024-09-17T16:13:59.340071+0000","last_became_active":"2024-09-17T16:11:21.266149+0000","last_became_peered":"2024-09-17T16:11:21.266149+0000","last_unstale":"2024-09-17T16:13:59.340071+0000","last_undegraded":"2024-09-17T16:13:59.340071+0000","last_fullsized":"2024-09-17T16:13:59.340071+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'2377","last_scrub_stamp":"2024-09-17T16:12:22.182710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:22.182710+0000","objects_scrubbed":1944,"log_size":5195,"log_dups_size":0,"ondisk_log_size":5195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:53.037965+0000","scrub_duration":332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":28160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,9],"acting":[8,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.f","version":"199'9546","reported_seq":9773,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T16:13:59.340144+0000","last_change":"2024-09-17T16:10:54.283333+0000","last_active":"2024-09-17T16:13:59.340144+0000","last_peered":"2024-09-17T16:13:59.340144+0000","last_clean":"2024-09-17T16:13:59.340144+0000","last_became_active":"2024-09-17T16:10:54.282931+0000","last_became_peered":"2024-09-17T16:10:54.282931+0000","last_unstale":"2024-09-17T16:13:59.340144+0000","last_undegraded":"2024-09-17T16:13:59.340144+0000","last_fullsized":"2024-09-17T16:13:59.340144+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":9546,"log_dups_size":0,"ondisk_log_size":9546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:49.125421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9546,"num_write_kb":305472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"199'9528","reported_seq":9788,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463613+0000","last_change":"2024-09-17T16:12:01.681365+0000","last_active":"2024-09-17T16:13:54.463613+0000","last_peered":"2024-09-17T16:13:54.463613+0000","last_clean":"2024-09-17T16:13:54.463613+0000","last_became_active":"2024-09-17T16:10:54.283208+0000","last_became_peered":"2024-09-17T16:10:54.283208+0000","last_unstale":"2024-09-17T16:13:54.463613+0000","last_undegraded":"2024-09-17T16:13:54.463613+0000","last_fullsized":"2024-09-17T16:13:54.463613+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'2642","last_scrub_stamp":"2024-09-17T16:12:01.681123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:01.681123+0000","objects_scrubbed":2614,"log_size":9528,"log_dups_size":0,"ondisk_log_size":9528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:17.223777+0000","scrub_duration":684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9528,"num_write_kb":304896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"200'9526","reported_seq":9796,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.373191+0000","last_change":"2024-09-17T16:12:18.766646+0000","last_active":"2024-09-17T16:13:54.373191+0000","last_peered":"2024-09-17T16:13:54.373191+0000","last_clean":"2024-09-17T16:13:54.373191+0000","last_became_active":"2024-09-17T16:10:54.283016+0000","last_became_peered":"2024-09-17T16:10:54.283016+0000","last_unstale":"2024-09-17T16:13:54.373191+0000","last_undegraded":"2024-09-17T16:13:54.373191+0000","last_fullsized":"2024-09-17T16:13:54.373191+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'3627","last_scrub_stamp":"2024-09-17T16:12:18.766492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:18.766492+0000","objects_scrubbed":3600,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:46.300018+0000","scrub_duration":754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9526,"num_write_kb":304769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"199'9438","reported_seq":9662,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.522287+0000","last_change":"2024-09-17T16:10:54.282319+0000","last_active":"2024-09-17T16:13:54.522287+0000","last_peered":"2024-09-17T16:13:54.522287+0000","last_clean":"2024-09-17T16:13:54.522287+0000","last_became_active":"2024-09-17T16:10:54.282069+0000","last_became_peered":"2024-09-17T16:10:54.282069+0000","last_unstale":"2024-09-17T16:13:54.522287+0000","last_undegraded":"2024-09-17T16:13:54.522287+0000","last_fullsized":"2024-09-17T16:13:54.522287+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:38.865424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9438,"num_write_kb":302016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":261,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.668397+0000","last_change":"2024-09-17T16:13:36.826049+0000","last_active":"2024-09-17T16:13:54.668397+0000","last_peered":"2024-09-17T16:13:54.668397+0000","last_clean":"2024-09-17T16:13:54.668397+0000","last_became_active":"2024-09-17T16:12:39.286388+0000","last_became_peered":"2024-09-17T16:12:39.286388+0000","last_unstale":"2024-09-17T16:13:54.668397+0000","last_undegraded":"2024-09-17T16:13:54.668397+0000","last_fullsized":"2024-09-17T16:13:54.668397+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:13:36.825983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:13:36.825983+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:54.394135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"199'5280","reported_seq":8539,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463535+0000","last_change":"2024-09-17T16:13:52.771474+0000","last_active":"2024-09-17T16:13:54.463535+0000","last_peered":"2024-09-17T16:13:54.463535+0000","last_clean":"2024-09-17T16:13:54.463535+0000","last_became_active":"2024-09-17T16:12:52.302562+0000","last_became_peered":"2024-09-17T16:12:52.302562+0000","last_unstale":"2024-09-17T16:13:54.463535+0000","last_undegraded":"2024-09-17T16:13:54.463535+0000","last_fullsized":"2024-09-17T16:13:54.463535+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:14:22.728846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4859,"num_write_kb":155008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1497,"num_bytes_recovered":80609280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"199'5058","reported_seq":8211,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.668528+0000","last_change":"2024-09-17T16:13:33.904714+0000","last_active":"2024-09-17T16:13:54.668528+0000","last_peered":"2024-09-17T16:13:54.668528+0000","last_clean":"2024-09-17T16:13:54.668528+0000","last_became_active":"2024-09-17T16:13:03.359631+0000","last_became_peered":"2024-09-17T16:13:03.359631+0000","last_unstale":"2024-09-17T16:13:54.668528+0000","last_undegraded":"2024-09-17T16:13:54.668528+0000","last_fullsized":"2024-09-17T16:13:54.668528+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'4499","last_scrub_stamp":"2024-09-17T16:13:33.904633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:33.904633+0000","objects_scrubbed":560,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:35.002572+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"199'5288","reported_seq":5554,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463586+0000","last_change":"2024-09-17T16:12:16.414710+0000","last_active":"2024-09-17T16:13:54.463586+0000","last_peered":"2024-09-17T16:13:54.463586+0000","last_clean":"2024-09-17T16:13:54.463586+0000","last_became_active":"2024-09-17T16:11:13.356367+0000","last_became_peered":"2024-09-17T16:11:13.356367+0000","last_unstale":"2024-09-17T16:13:54.463586+0000","last_undegraded":"2024-09-17T16:13:54.463586+0000","last_fullsized":"2024-09-17T16:13:54.463586+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2190","last_scrub_stamp":"2024-09-17T16:12:16.414644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:16.414644+0000","objects_scrubbed":1711,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:20.196085+0000","scrub_duration":347,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3107,"num_write_kb":44672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"199'5185","reported_seq":5464,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463430+0000","last_change":"2024-09-17T16:13:32.993292+0000","last_active":"2024-09-17T16:13:54.463430+0000","last_peered":"2024-09-17T16:13:54.463430+0000","last_clean":"2024-09-17T16:13:54.463430+0000","last_became_active":"2024-09-17T16:11:13.352761+0000","last_became_peered":"2024-09-17T16:11:13.352761+0000","last_unstale":"2024-09-17T16:13:54.463430+0000","last_undegraded":"2024-09-17T16:13:54.463430+0000","last_fullsized":"2024-09-17T16:13:54.463430+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4605","last_scrub_stamp":"2024-09-17T16:13:32.993228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:32.993228+0000","objects_scrubbed":582,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:47.865099+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3045,"num_write_kb":42368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"199'5274","reported_seq":11624,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463593+0000","last_change":"2024-09-17T16:13:51.833000+0000","last_active":"2024-09-17T16:13:54.463593+0000","last_peered":"2024-09-17T16:13:54.463593+0000","last_clean":"2024-09-17T16:13:54.463593+0000","last_became_active":"2024-09-17T16:13:51.822246+0000","last_became_peered":"2024-09-17T16:13:51.822246+0000","last_unstale":"2024-09-17T16:13:54.463593+0000","last_undegraded":"2024-09-17T16:13:54.463593+0000","last_fullsized":"2024-09-17T16:13:54.463593+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'1233","last_scrub_stamp":"2024-09-17T16:11:30.184562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:30.184562+0000","objects_scrubbed":777,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:53.665233+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":104448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2409,"num_bytes_recovered":48365568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"199'5313","reported_seq":5580,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T16:13:59.340072+0000","last_change":"2024-09-17T16:12:15.296063+0000","last_active":"2024-09-17T16:13:59.340072+0000","last_peered":"2024-09-17T16:13:59.340072+0000","last_clean":"2024-09-17T16:13:59.340072+0000","last_became_active":"2024-09-17T16:11:13.348959+0000","last_became_peered":"2024-09-17T16:11:13.348959+0000","last_unstale":"2024-09-17T16:13:59.340072+0000","last_undegraded":"2024-09-17T16:13:59.340072+0000","last_fullsized":"2024-09-17T16:13:59.340072+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'2146","last_scrub_stamp":"2024-09-17T16:12:15.295952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:15.295952+0000","objects_scrubbed":1680,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:50.081438+0000","scrub_duration":439,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":48000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"199'5255","reported_seq":5517,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463530+0000","last_change":"2024-09-17T16:12:18.628732+0000","last_active":"2024-09-17T16:13:54.463530+0000","last_peered":"2024-09-17T16:13:54.463530+0000","last_clean":"2024-09-17T16:13:54.463530+0000","last_became_active":"2024-09-17T16:11:13.354186+0000","last_became_peered":"2024-09-17T16:11:13.354186+0000","last_unstale":"2024-09-17T16:13:54.463530+0000","last_undegraded":"2024-09-17T16:13:54.463530+0000","last_fullsized":"2024-09-17T16:13:54.463530+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'2226","last_scrub_stamp":"2024-09-17T16:12:18.628468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:12:18.628468+0000","objects_scrubbed":1798,"log_size":5255,"log_dups_size":0,"ondisk_log_size":5255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:51.660242+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"199'5209","reported_seq":5490,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463833+0000","last_change":"2024-09-17T16:13:35.606144+0000","last_active":"2024-09-17T16:13:54.463833+0000","last_peered":"2024-09-17T16:13:54.463833+0000","last_clean":"2024-09-17T16:13:54.463833+0000","last_became_active":"2024-09-17T16:11:13.352397+0000","last_became_peered":"2024-09-17T16:11:13.352397+0000","last_unstale":"2024-09-17T16:13:54.463833+0000","last_undegraded":"2024-09-17T16:13:54.463833+0000","last_fullsized":"2024-09-17T16:13:54.463833+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'4691","last_scrub_stamp":"2024-09-17T16:13:35.606083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:35.606083+0000","objects_scrubbed":518,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:44.819021+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":37504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"199'5307","reported_seq":8548,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.668409+0000","last_change":"2024-09-17T16:13:08.898511+0000","last_active":"2024-09-17T16:13:54.668409+0000","last_peered":"2024-09-17T16:13:54.668409+0000","last_clean":"2024-09-17T16:13:54.668409+0000","last_became_active":"2024-09-17T16:13:08.416136+0000","last_became_peered":"2024-09-17T16:13:08.416136+0000","last_unstale":"2024-09-17T16:13:54.668409+0000","last_undegraded":"2024-09-17T16:13:54.668409+0000","last_fullsized":"2024-09-17T16:13:54.668409+0000","mapping_epoch":174,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'4006","last_scrub_stamp":"2024-09-17T16:13:08.898311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:08.898311+0000","objects_scrubbed":1303,"log_size":5307,"log_dups_size":0,"ondisk_log_size":5307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:17.866499+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1303,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"199'5329","reported_seq":8713,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.463414+0000","last_change":"2024-09-17T16:13:28.524306+0000","last_active":"2024-09-17T16:13:54.463414+0000","last_peered":"2024-09-17T16:13:54.463414+0000","last_clean":"2024-09-17T16:13:54.463414+0000","last_became_active":"2024-09-17T16:12:57.338076+0000","last_became_peered":"2024-09-17T16:12:57.338076+0000","last_unstale":"2024-09-17T16:13:54.463414+0000","last_undegraded":"2024-09-17T16:13:54.463414+0000","last_fullsized":"2024-09-17T16:13:54.463414+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'4624","last_scrub_stamp":"2024-09-17T16:13:28.524255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:13:28.524255+0000","objects_scrubbed":707,"log_size":5329,"log_dups_size":0,"ondisk_log_size":5329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:33.220937+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":707,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"199'9392","reported_seq":9650,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-17T16:13:54.522256+0000","last_change":"2024-09-17T16:11:58.924449+0000","last_active":"2024-09-17T16:13:54.522256+0000","last_peered":"2024-09-17T16:13:54.522256+0000","last_clean":"2024-09-17T16:13:54.522256+0000","last_became_active":"2024-09-17T16:10:54.283026+0000","last_became_peered":"2024-09-17T16:10:54.283026+0000","last_unstale":"2024-09-17T16:13:54.522256+0000","last_undegraded":"2024-09-17T16:13:54.522256+0000","last_fullsized":"2024-09-17T16:13:54.522256+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'2421","last_scrub_stamp":"2024-09-17T16:11:58.924214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:11:58.924214+0000","objects_scrubbed":2408,"log_size":9392,"log_dups_size":0,"ondisk_log_size":9392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:42.916375+0000","scrub_duration":471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9392,"num_write_kb":300544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"199'9612","reported_seq":15725,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T16:13:59.340074+0000","last_change":"2024-09-17T16:13:36.362764+0000","last_active":"2024-09-17T16:13:59.340074+0000","last_peered":"2024-09-17T16:13:59.340074+0000","last_clean":"2024-09-17T16:13:59.340074+0000","last_became_active":"2024-09-17T16:13:36.345378+0000","last_became_peered":"2024-09-17T16:13:36.345378+0000","last_unstale":"2024-09-17T16:13:59.340074+0000","last_undegraded":"2024-09-17T16:13:59.340074+0000","last_fullsized":"2024-09-17T16:13:59.340074+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:10:53.263777+0000","last_clean_scrub_stamp":"2024-09-17T16:10:53.263777+0000","objects_scrubbed":0,"log_size":9612,"log_dups_size":0,"ondisk_log_size":9612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:45.123161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":307584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2936,"num_bytes_recovered":160694272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":26,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113846,"num_write_kb":2873025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6842,"num_bytes_recovered":289669120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23658496,"data_stored":23658496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38296,"internal_metadata":0},"log_size":161532,"ondisk_log_size":161532,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285622,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65044,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93680108,"statfs":{"total":95995035648,"available":95928430592,"internally_reserved":0,"allocated":1966080,"data_stored":1438321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547257,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":68688,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93676464,"statfs":{"total":95995035648,"available":95924699136,"internally_reserved":0,"allocated":1433600,"data_stored":909413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743487,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104788,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93640364,"statfs":{"total":95995035648,"available":95887732736,"internally_reserved":0,"allocated":1433600,"data_stored":909413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":105839040},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005124,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72848,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672304,"statfs":{"total":95995035648,"available":95920439296,"internally_reserved":0,"allocated":1433600,"data_stored":909413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963601,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94808,"kb_used_data":1920,"kb_used_omap":41,"kb_used_meta":92822,"kb_avail":93650344,"statfs":{"total":95995035648,"available":95897952256,"internally_reserved":0,"allocated":1966080,"data_stored":1438341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":95050035},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69208,"kb_used_data":1920,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93675944,"statfs":{"total":95995035648,"available":95924166656,"internally_reserved":0,"allocated":1966080,"data_stored":1438341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724625,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64532,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93680620,"statfs":{"total":95995035648,"available":95928954880,"internally_reserved":0,"allocated":1433600,"data_stored":909413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986264,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72848,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672304,"statfs":{"total":95995035648,"available":95920439296,"internally_reserved":0,"allocated":1433600,"data_stored":909413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":133964,"kb_used_data":26424,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93611188,"statfs":{"total":95995035648,"available":95857856512,"internally_reserved":0,"allocated":27058176,"data_stored":26533989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411426,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":72848,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672304,"statfs":{"total":95995035648,"available":95920439296,"internally_reserved":0,"allocated":1433600,"data_stored":909413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108944,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636208,"statfs":{"total":95995035648,"available":95883476992,"internally_reserved":0,"allocated":1433600,"data_stored":909413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77008,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93668144,"statfs":{"total":95995035648,"available":95916179456,"internally_reserved":0,"allocated":1433600,"data_stored":909413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":23658496,"data_stored":23658496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38296,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:14:00.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:00.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:00 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:00.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:00 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:00.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:00 smithi071 ceph-mon[30002]: osdmap e201: 12 total, 12 up, 12 in 2024-09-17T16:14:00.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:00.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:00.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:00.958 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:00.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:00.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:01.060 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:01.083 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:01.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:01.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:01.211 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:01.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:01.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:01.335 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:01.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:01.356 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_0' removed 2024-09-17T16:14:01.438 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:01.461 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:01.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:01.563 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:01.587 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:01.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:01.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:01 smithi173 ceph-mon[30530]: pgmap v496: 27 pgs: 27 active+clean; 577 KiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 461 op/s 2024-09-17T16:14:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:01 smithi173 ceph-mon[30530]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:14:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:01 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:01 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:01 smithi173 ceph-mon[30530]: osdmap e202: 12 total, 12 up, 12 in 2024-09-17T16:14:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2815874482' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:14:01.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:01.707 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:01.707 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:01.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:01 smithi017 ceph-mon[27571]: pgmap v496: 27 pgs: 27 active+clean; 577 KiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 461 op/s 2024-09-17T16:14:01.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:01 smithi017 ceph-mon[27571]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:14:01.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:01 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:01.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:01 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:01.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:01 smithi017 ceph-mon[27571]: osdmap e202: 12 total, 12 up, 12 in 2024-09-17T16:14:01.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2815874482' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:14:01.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:01.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:01.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:01 smithi071 ceph-mon[30002]: pgmap v496: 27 pgs: 27 active+clean; 577 KiB data, 982 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 461 op/s 2024-09-17T16:14:01.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:01 smithi071 ceph-mon[30002]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:14:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:01 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:01 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:01 smithi071 ceph-mon[30002]: osdmap e202: 12 total, 12 up, 12 in 2024-09-17T16:14:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2815874482' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:14:01.814 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:01.839 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:01.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:01.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:01.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:01.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:02.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:02.091 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:02.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:02.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:02.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:02.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:02.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:02.348 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:02.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:02.358 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:14:02.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:02.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:02.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:02.574 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:02.597 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:02.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:02.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:02.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2815874482' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:14:02.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:02 smithi173 ceph-mon[30530]: osdmap e203: 12 total, 12 up, 12 in 2024-09-17T16:14:02.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:02.721 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:02.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:02.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2815874482' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:14:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:02 smithi017 ceph-mon[27571]: osdmap e203: 12 total, 12 up, 12 in 2024-09-17T16:14:02.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2815874482' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:14:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:02 smithi071 ceph-mon[30002]: osdmap e203: 12 total, 12 up, 12 in 2024-09-17T16:14:02.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:02.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:02.846 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:02.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:02.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:02.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:02.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:03.073 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:03.076 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T16:14:03.077 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T16:14:03.078 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-17T16:14:03.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-17T16:14:03.097 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:03.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:03.135 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:03.139 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:03.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:03.223 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:03.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:03.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:03.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:03.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:03.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:03.448 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:03.474 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:03.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:03.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:03.576 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:03.599 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:03.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: osdmap e204: 12 total, 12 up, 12 in 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4019650491' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:14:03.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:14:03.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: osdmap e204: 12 total, 12 up, 12 in 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4019650491' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:14:03.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:14:03.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:03.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:03.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:03.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:03.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: osdmap e204: 12 total, 12 up, 12 in 2024-09-17T16:14:03.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:14:03.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:14:03.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:14:03.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:14:03.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:03.716 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:03.716 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4019650491' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:03.716 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:14:03.716 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:14:03.716 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:03.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:03.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:03.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:03.754 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:14:03.761 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:14:03.820 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:14:03.820 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:14:03.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:03.842 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:14:03.842 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:14:03.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:03.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:03.863 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:14:03.863 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:14:03.900 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:14:03.900 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:14:03.926 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:14:03.926 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:14:03.947 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:14:03.947 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:14:03.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:03.969 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:14:03.969 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:14:03.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:03.974 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:03.990 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:14:03.990 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:14:04.012 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:14:04.013 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:14:04.034 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:14:04.034 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:14:04.056 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:14:04.056 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:14:04.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:04.077 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:14:04.077 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:14:04.096 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:04.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:04.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:04.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:04.223 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:04.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:04.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:04.349 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:04.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:04.474 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:04.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:04.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:04.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:04.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:04.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:04 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:14:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:04 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:14:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4019650491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:04 smithi173 ceph-mon[30530]: osdmap e205: 12 total, 12 up, 12 in 2024-09-17T16:14:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:04 smithi173 ceph-mon[30530]: pgmap v502: 1 pgs: 1 active+clean; 577 KiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1970818538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:04 smithi173 ceph-mon[30530]: osdmap e206: 12 total, 12 up, 12 in 2024-09-17T16:14:04.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:04.722 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:04.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:04.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:04 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:14:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:04 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:14:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4019650491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:04 smithi017 ceph-mon[27571]: osdmap e205: 12 total, 12 up, 12 in 2024-09-17T16:14:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:04 smithi017 ceph-mon[27571]: pgmap v502: 1 pgs: 1 active+clean; 577 KiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1970818538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:04 smithi017 ceph-mon[27571]: osdmap e206: 12 total, 12 up, 12 in 2024-09-17T16:14:04.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:04 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:14:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:04 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:14:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4019650491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:04 smithi071 ceph-mon[30002]: osdmap e205: 12 total, 12 up, 12 in 2024-09-17T16:14:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:04 smithi071 ceph-mon[30002]: pgmap v502: 1 pgs: 1 active+clean; 577 KiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1970818538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:04 smithi071 ceph-mon[30002]: osdmap e206: 12 total, 12 up, 12 in 2024-09-17T16:14:04.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:04.844 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:04.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:04.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:04.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:04.967 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:04.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:04.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:05.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:05.094 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:05.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:05.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:05.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:05.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:05.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:05.247 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:14:05.248 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:14:05.248 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:14:05.248 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:14:05.248 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:14:05.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:05.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:05.345 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:05.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:05.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:05.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:05.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:05.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:05.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:05.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:05.702 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:05.726 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:05.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:05.828 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:05.853 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:05.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:05.950 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:05.951 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:05.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:05.980 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:05.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:05.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:14:06.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:06.108 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:06.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:06.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:06.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:06.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:06.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:06.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:06.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:06.465 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:06.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:06 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:06.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:06 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:06.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:06 smithi017 ceph-mon[27571]: osdmap e207: 12 total, 12 up, 12 in 2024-09-17T16:14:06.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:06 smithi017 ceph-mon[27571]: pgmap v505: 1 pgs: 1 active+clean; 577 KiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:06.490 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:06.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:06.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:06.497 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:14:06.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:06.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:06.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:06.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:06 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:06 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:06 smithi173 ceph-mon[30530]: osdmap e207: 12 total, 12 up, 12 in 2024-09-17T16:14:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:06 smithi173 ceph-mon[30530]: pgmap v505: 1 pgs: 1 active+clean; 577 KiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:06.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:06.742 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:06.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:06.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:06 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:06 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:06 smithi071 ceph-mon[30002]: osdmap e207: 12 total, 12 up, 12 in 2024-09-17T16:14:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:06 smithi071 ceph-mon[30002]: pgmap v505: 1 pgs: 1 active+clean; 577 KiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:06.844 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:06.866 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:06.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:06.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:06.994 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:06.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:07.071 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:07.076 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:07.085 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:14:07.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:07.151 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:07.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:07.243 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:14:07.243 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:14:07.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:07.276 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:07.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:07.377 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_1' created 2024-09-17T16:14:07.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:07.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:07.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:07.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:07.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2197326054' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-17T16:14:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:07 smithi017 ceph-mon[27571]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:14:07.502 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:07.528 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:07.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:07.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:07.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2197326054' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-17T16:14:07.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:07 smithi173 ceph-mon[30530]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:14:07.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:07.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:07.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:07.778 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:07.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:07.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2197326054' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-17T16:14:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:07 smithi071 ceph-mon[30002]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:14:07.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:07.906 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:07.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:07.980 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-17T16:14:05.764992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":6,"kb":1124941824,"kb_used":1000548,"kb_used_data":34268,"kb_used_omap":58,"kb_used_meta":965957,"kb_avail":1123941276,"statfs":{"total":1151940427776,"available":1150915866624,"internally_reserved":0,"allocated":35090432,"data_stored":28585956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":989140131},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000632"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":266,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T16:14:02.373081+0000","last_change":"2024-09-17T16:13:36.826049+0000","last_active":"2024-09-17T16:14:02.373081+0000","last_peered":"2024-09-17T16:14:02.373081+0000","last_clean":"2024-09-17T16:14:02.373081+0000","last_became_active":"2024-09-17T16:12:39.286388+0000","last_became_peered":"2024-09-17T16:12:39.286388+0000","last_unstale":"2024-09-17T16:14:02.373081+0000","last_undegraded":"2024-09-17T16:14:02.373081+0000","last_fullsized":"2024-09-17T16:14:02.373081+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:13:36.825983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:13:36.825983+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:14:54.394135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285623,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65088,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93680064,"statfs":{"total":95995035648,"available":95928385536,"internally_reserved":0,"allocated":2007040,"data_stored":1461724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":68764,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93676388,"statfs":{"total":95995035648,"available":95924621312,"internally_reserved":0,"allocated":1511424,"data_stored":954991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743488,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104804,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93640348,"statfs":{"total":95995035648,"available":95887716352,"internally_reserved":0,"allocated":1458176,"data_stored":925051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":105839040},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005125,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":72912,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672240,"statfs":{"total":95995035648,"available":95920373760,"internally_reserved":0,"allocated":1499136,"data_stored":947616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963602,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94844,"kb_used_data":1960,"kb_used_omap":41,"kb_used_meta":92822,"kb_avail":93650308,"statfs":{"total":95995035648,"available":95897915392,"internally_reserved":0,"allocated":2007040,"data_stored":1461744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":95050035},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495696,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69284,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93675868,"statfs":{"total":95995035648,"available":95924088832,"internally_reserved":0,"allocated":2043904,"data_stored":1483919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724627,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":64604,"kb_used_data":1476,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93680548,"statfs":{"total":95995035648,"available":95928881152,"internally_reserved":0,"allocated":1511424,"data_stored":954991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":77044,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93668108,"statfs":{"total":95995035648,"available":95916142592,"internally_reserved":0,"allocated":1474560,"data_stored":932816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124360,"kb_used_data":16824,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93620792,"statfs":{"total":95995035648,"available":95867691008,"internally_reserved":0,"allocated":17227776,"data_stored":16703589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411427,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":72868,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93672284,"statfs":{"total":95995035648,"available":95920418816,"internally_reserved":0,"allocated":1458176,"data_stored":925051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":108968,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636184,"statfs":{"total":95995035648,"available":95883452416,"internally_reserved":0,"allocated":1458176,"data_stored":925051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77008,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93668144,"statfs":{"total":95995035648,"available":95916179456,"internally_reserved":0,"allocated":1433600,"data_stored":909413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:14:07.981 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T16:14:07.981 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:14:07.982 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T16:14:07.982 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:14:08.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:08.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:08.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:08.132 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:08.157 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:08.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:08.173 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:14:08.173 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:14:08.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:14:08.173 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:14:08.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.173 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-17T16:14:08.173 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:08.079913+0000", 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 480.09026068100002, 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86337366999999998, 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.14158", 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.17:0/2484390953", 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 1 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079913+0000", 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079913+0000", 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079915+0000", 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0330000000000001e-06 2024-09-17T16:14:08.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079920+0000", 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5210000000000002e-06 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079929+0000", 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4490000000000005e-06 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079939+0000", 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3519999999999996e-06 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.080004+0000", 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5324e-05 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.080011+0000", 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5300000000000002e-06 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.942061+0000", 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86205003199999997 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.942214+0000", 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000153287 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.942381+0000", 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016673 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943022+0000", 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00064124199999999998 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943096+0000", 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3516999999999994e-05 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943190+0000", 2024-09-17T16:14:08.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4339000000000006e-05 2024-09-17T16:14:08.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943247+0000", 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6779e-05 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943287+0000", 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0534999999999997e-05 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:13527 2.19 2:9902b078:::benchmark_data_smithi173_55121_object13526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.335470+0000", 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 175.83470375799999, 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0334308889999999, 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:08.182 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 13527 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335470+0000", 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335470+0000", 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335471+0000", 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.89e-07 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335500+0000", 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8971000000000001e-05 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335507+0000", 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.401e-06 2024-09-17T16:14:08.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335513+0000", 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.366e-06 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.362278+0000", 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.026764929999999999 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.362286+0000", 2024-09-17T16:14:08.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.038e-06 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.368810+0000", 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.006524102 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.368901+0000", 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1592e-05 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:13539 2.14 2:295cc8f1:::benchmark_data_smithi173_55121_object13538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.759315+0000", 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 175.410858701, 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.60567000500000001, 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 13539 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.187 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759315+0000", 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759315+0000", 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759324+0000", 2024-09-17T16:14:08.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.14e-06 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759364+0000", 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0513999999999999e-05 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759390+0000", 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6265e-05 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759403+0000", 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3088000000000001e-05 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759483+0000", 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9049999999999997e-05 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759492+0000", 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7119999999999995e-06 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.360226+0000", 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.60073335000000005 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.360357+0000", 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013103999999999999 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.360500+0000", 2024-09-17T16:14:08.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014305899999999999 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364416+0000", 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039165040000000003 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364455+0000", 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9304000000000002e-05 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364915+0000", 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045950800000000002 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364954+0000", 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8639000000000001e-05 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364985+0000", 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1832000000000003e-05 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:14851 2.19 2:9f3f4fbe:::benchmark_data_smithi173_55121_object14850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:13.952437+0000", 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 174.21773684999999, 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.272072806, 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.195 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 14851 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.952437+0000", 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.952437+0000", 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.952438+0000", 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8800000000000001e-07 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.952519+0000", 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1228000000000002e-05 2024-09-17T16:14:08.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.952530+0000", 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0601000000000001e-05 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.952535+0000", 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.181e-06 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.952578+0000", 2024-09-17T16:14:08.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2299999999999998e-05 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.952657+0000", 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9357999999999995e-05 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.952739+0000", 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1910999999999996e-05 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:08.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.953225+0000", 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048575400000000003 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.960574+0000", 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0073497700000000003 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.224431+0000", 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.263856698 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.224476+0000", 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5249e-05 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.224510+0000", 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3868000000000002e-05 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(2.11 116/107 [PushOp(2:88e52e2e:::benchmark_data_smithi173_55121_object4621:head, version: 100'307, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:88e52e2e:::benchmark_data_smithi173_55121_object4621:head@100'307, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:14:08.201 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:25.435893+0000", 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 162.73428078000001, 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31609990900000001, 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.435893+0000", 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.435893+0000", 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.435894+0000", 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2999999999999999e-07 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.435925+0000", 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0345000000000002e-05 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.751343+0000", 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31541794899999998 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.751359+0000", 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6339000000000001e-05 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.751462+0000", 2024-09-17T16:14:08.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000102836 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.751477+0000", 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5628e-05 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.751993+0000", 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051588199999999995 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25184 2.9 2:915f94f2:::benchmark_data_smithi173_55121_object25183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.745700+0000", 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 157.42447377600001, 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31523103000000002, 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25184 2024-09-17T16:14:08.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745700+0000", 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745700+0000", 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745702+0000", 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.64e-06 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745748+0000", 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6217000000000002e-05 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745760+0000", 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1348999999999999e-05 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745765+0000", 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100000000000003e-06 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745808+0000", 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2104000000000002e-05 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745895+0000", 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7670000000000006e-05 2024-09-17T16:14:08.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745986+0000", 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1096999999999999e-05 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.746541+0000", 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00055457599999999998 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.747059+0000", 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051780499999999998 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060862+0000", 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31380370299999999 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060899+0000", 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6366999999999999e-05 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060931+0000", 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2691999999999998e-05 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25215 2.10 e121/111 2:0e4d5290:::benchmark_data_smithi173_55121_object25214:head v 121'1162, mlcod=121'1158)", 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.775819+0000", 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 157.394354854, 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44938945499999999, 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775819+0000", 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775819+0000", 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775820+0000", 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775853+0000", 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3364000000000001e-05 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224477+0000", 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44862400299999999 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224483+0000", 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9730000000000002e-06 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224548+0000", 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4199e-05 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224566+0000", 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8686999999999999e-05 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.225168+0000", 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060204500000000005 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.225209+0000", 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0392999999999998e-05 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.216 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25240 2.9 2:9323c63b:::benchmark_data_smithi173_55121_object25239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.803581+0000", 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 157.36659281799999, 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.437608569, 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25240 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803581+0000", 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803581+0000", 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803583+0000", 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.265e-06 2024-09-17T16:14:08.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803637+0000", 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4765999999999997e-05 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803649+0000", 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1304999999999999e-05 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803655+0000", 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2899999999999999e-06 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803708+0000", 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2627999999999998e-05 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803793+0000", 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5376000000000004e-05 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803906+0000", 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011330100000000001 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804549+0000", 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00064244400000000004 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804821+0000", 2024-09-17T16:14:08.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027272499999999997 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.241122+0000", 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43630034000000001 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.241159+0000", 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7251e-05 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.241190+0000", 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0877999999999998e-05 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:28853 2.2 e128/104 2:4387b46d:::benchmark_data_smithi173_55121_object28852:head v 128'1332, mlcod=128'1329)", 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:48.540329+0000", 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 139.629844827, 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.364975192, 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.223 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540329+0000", 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540329+0000", 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540330+0000", 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3300000000000002e-07 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540380+0000", 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.02e-05 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904638+0000", 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36425802000000002 2024-09-17T16:14:08.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904645+0000", 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6579999999999996e-06 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904700+0000", 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4855999999999998e-05 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904714+0000", 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3902e-05 2024-09-17T16:14:08.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.905261+0000", 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054778599999999998 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.905305+0000", 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3136999999999997e-05 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.227 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.228 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:29877 2.2 e129/104 2:40729eb6:::benchmark_data_smithi173_55121_object29876:head v 129'1356, mlcod=129'1354)", 2024-09-17T16:14:08.228 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:14:08.228 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 137.96598337, 2024-09-17T16:14:08.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.87425371299999999, 2024-09-17T16:14:08.228 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.228 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.228 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.228 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5700000000000002e-07 2024-09-17T16:14:08.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204236+0000", 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4901999999999998e-05 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077424+0000", 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.87318755800000003 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077440+0000", 2024-09-17T16:14:08.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5753999999999999e-05 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077544+0000", 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104652 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077590+0000", 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5359999999999999e-05 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.078411+0000", 2024-09-17T16:14:08.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082190400000000004 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.078445+0000", 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3025999999999999e-05 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:88652 2.2 e157/104 2:45ffdfe0:::benchmark_data_smithi173_55121_object12163:head v 157'3206, mlcod=157'3203)", 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:43.957077+0000", 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 84.213097001999998, 2024-09-17T16:14:08.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.292041826, 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.957077+0000", 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.957077+0000", 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.957078+0000", 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.957084+0000", 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9970000000000004e-06 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.247707+0000", 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29062289299999999 2024-09-17T16:14:08.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.247724+0000", 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7059999999999999e-05 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.247813+0000", 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8834999999999999e-05 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.247834+0000", 2024-09-17T16:14:08.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1549e-05 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.249086+0000", 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001251956 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.249119+0000", 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2759000000000003e-05 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(2.9 158/155 [PushReplyOp(2:9441bd8f:::benchmark_data_smithi173_55121_object49095:head)])", 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:45.842362+0000", 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 82.327811677, 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32133242099999998, 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.842362+0000", 2024-09-17T16:14:08.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.842362+0000", 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.842363+0000", 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5099999999999999e-07 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.842368+0000", 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1229999999999999e-06 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.162585+0000", 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32021638200000002 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.162593+0000", 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2369999999999992e-06 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.162702+0000", 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108522 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.163695+0000", 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099340600000000006 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:90245 2.9 2:947e1be4:::benchmark_data_smithi173_55121_object13756:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:14:08.240 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:45.848474+0000", 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 82.321699918999997, 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31939979400000001, 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90245 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848474+0000", 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848474+0000", 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848476+0000", 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9719999999999999e-06 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848484+0000", 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3429999999999998e-06 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848503+0000", 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9905999999999999e-05 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848513+0000", 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0023999999999999e-05 2024-09-17T16:14:08.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848600+0000", 2024-09-17T16:14:08.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6085e-05 2024-09-17T16:14:08.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848753+0000", 2024-09-17T16:14:08.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000153444 2024-09-17T16:14:08.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848909+0000", 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000155608 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.851197+0000", 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022886439999999998 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.865002+0000", 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013805199000000001 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167298+0000", 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30229515499999998 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167350+0000", 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2330000000000002e-05 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167874+0000", 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00052408399999999996 2024-09-17T16:14:08.247 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:90245 2.9 e158/155 ondisk, result = 0)", 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:45.851236+0000", 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 82.318938552999995, 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31664304700000001, 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.851236+0000", 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.851236+0000", 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.851236+0000", 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8299999999999996e-07 2024-09-17T16:14:08.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.851243+0000", 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1019999999999999e-06 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167214+0000", 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31597043200000002 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167221+0000", 2024-09-17T16:14:08.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5340000000000002e-06 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167286+0000", 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4920999999999997e-05 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167293+0000", 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8889999999999996e-06 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167879+0000", 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00058548599999999997 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(2.9 158/155 [PushReplyOp(2:9287a5b2:::benchmark_data_smithi173_55121_object60862:head)])", 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:48.880360+0000", 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 79.289814148999994, 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38735041999999997, 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:14:08.252 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880360+0000", 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880360+0000", 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880361+0000", 2024-09-17T16:14:08.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3200000000000005e-07 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880365+0000", 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3189999999999996e-06 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267083+0000", 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38671788000000001 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267091+0000", 2024-09-17T16:14:08.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8050000000000007e-06 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267182+0000", 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1217000000000004e-05 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267710+0000", 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00052856699999999997 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:95505 2.4 2:25f242c1:::benchmark_data_smithi173_55121_object19016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:51.639132+0000", 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 76.531041669000004, 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33947701800000002, 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 95505 2024-09-17T16:14:08.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639132+0000", 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639132+0000", 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639134+0000", 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2589999999999999e-06 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639140+0000", 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1999999999999999e-06 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639152+0000", 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2275e-05 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639159+0000", 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3759999999999996e-06 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639206+0000", 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7608000000000002e-05 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639295+0000", 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8560000000000006e-05 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639383+0000", 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8293000000000003e-05 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.642147+0000", 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002763723 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.642168+0000", 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0795e-05 2024-09-17T16:14:08.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978532+0000", 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33636466399999998 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978575+0000", 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2593999999999999e-05 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978610+0000", 2024-09-17T16:14:08.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4671000000000001e-05 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:95505 2.4 e162/119 ondisk, result = 0)", 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:51.641024+0000", 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 76.529150005999995, 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.337590153, 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:14:08.262 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.641024+0000", 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.641024+0000", 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.641025+0000", 2024-09-17T16:14:08.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8400000000000004e-07 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.641029+0000", 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3159999999999998e-06 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978425+0000", 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33739633299999999 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978439+0000", 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3492999999999999e-05 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978519+0000", 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9736999999999999e-05 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978528+0000", 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0669999999999998e-06 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978614+0000", 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6522999999999998e-05 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(2.9 162/155 [PushReplyOp(2:96325bd0:::benchmark_data_smithi173_55121_object68334:head)])", 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:51.688909+0000", 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 76.481265440000001, 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33995969700000001, 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.266 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.688909+0000", 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.688909+0000", 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.688909+0000", 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7300000000000001e-07 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.688913+0000", 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.563e-06 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:52.028466+0000", 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33955346800000002 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:52.028475+0000", 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5299999999999996e-06 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:52.028513+0000", 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8674999999999998e-05 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:52.028868+0000", 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00035498799999999998 2024-09-17T16:14:08.269 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:135561 2.6 2:612899dc:::benchmark_data_smithi173_55121_object59072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 32.248797699999997, 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36676542899999998, 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:08.270 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 135561 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921378+0000", 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.897e-06 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921386+0000", 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.17e-06 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921398+0000", 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2408999999999999e-05 2024-09-17T16:14:08.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921406+0000", 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6179999999999997e-06 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921457+0000", 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1743999999999998e-05 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921550+0000", 2024-09-17T16:14:08.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2318000000000006e-05 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,8", 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921623+0000", 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3337999999999995e-05 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.923435+0000", 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001811654 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.923947+0000", 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051277600000000005 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288065+0000", 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36411789 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288107+0000", 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1479e-05 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288142+0000", 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5135999999999997e-05 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:152966 2.c e199/95 2:3484f1c7:::benchmark_data_smithi173_55121_object76477:head v 199'9438, mlcod=199'9436)", 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.314047874, 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58877864599999996, 2024-09-17T16:14:08.276 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856128+0000", 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3740000000000001e-06 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856136+0000", 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6950000000000006e-06 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441396+0000", 2024-09-17T16:14:08.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58525992800000004 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441405+0000", 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9770000000000006e-06 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441463+0000", 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8025000000000001e-05 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441483+0000", 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9678999999999999e-05 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.444869+0000", 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033864070000000001 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.444905+0000", 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5561000000000001e-05 2024-09-17T16:14:08.280 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.281 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.281 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.281 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:08.281 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:08.281 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:14:08.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:08.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:08.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:08.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:08.406 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:08.431 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:08.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:08.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2197326054' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-17T16:14:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:08 smithi017 ceph-mon[27571]: osdmap e208: 12 total, 12 up, 12 in 2024-09-17T16:14:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:08 smithi017 ceph-mon[27571]: pgmap v507: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:08.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:08.558 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:08.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:08.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2197326054' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-17T16:14:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:08 smithi173 ceph-mon[30530]: osdmap e208: 12 total, 12 up, 12 in 2024-09-17T16:14:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:08 smithi173 ceph-mon[30530]: pgmap v507: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:08.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:08.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:08.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:08.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:08.808 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:08.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2197326054' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-17T16:14:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:08 smithi071 ceph-mon[30002]: osdmap e208: 12 total, 12 up, 12 in 2024-09-17T16:14:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:08 smithi071 ceph-mon[30002]: pgmap v507: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:08.909 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:08.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:08.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:09.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:09.055 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:09.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:09.157 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:09.183 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:09.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:09.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:09.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:09.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:09.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:09.436 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:09.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:09.538 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:09.561 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:09.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:09 smithi173 ceph-mon[30530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:14:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:09 smithi173 ceph-mon[30530]: osdmap e209: 12 total, 12 up, 12 in 2024-09-17T16:14:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:09 smithi173 ceph-mon[30530]: osdmap e210: 12 total, 12 up, 12 in 2024-09-17T16:14:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2082569926' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:14:09.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:09.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:09.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:09 smithi017 ceph-mon[27571]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:14:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:09 smithi017 ceph-mon[27571]: osdmap e209: 12 total, 12 up, 12 in 2024-09-17T16:14:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:09 smithi017 ceph-mon[27571]: osdmap e210: 12 total, 12 up, 12 in 2024-09-17T16:14:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2082569926' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:14:09.786 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-17T16:14:09.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:09 smithi071 ceph-mon[30002]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:14:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:09 smithi071 ceph-mon[30002]: osdmap e209: 12 total, 12 up, 12 in 2024-09-17T16:14:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:09 smithi071 ceph-mon[30002]: osdmap e210: 12 total, 12 up, 12 in 2024-09-17T16:14:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2082569926' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:14:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:09.815 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:09.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:09.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:09.938 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:09.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:10.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:10.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:10.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:10.167 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:10.190 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:10.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:10.213 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:10.249 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:14:10.291 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:10.312 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:10.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:10.314 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:14:10.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:10.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:10.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:10.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:10.477 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:10 smithi017 ceph-mon[27571]: pgmap v510: 17 pgs: 2 creating+peering, 7 active+clean, 8 unknown; 577 KiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:10.477 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:10 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:10.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:10 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:10.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:10 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:10.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2082569926' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:14:10.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:10 smithi017 ceph-mon[27571]: osdmap e211: 12 total, 12 up, 12 in 2024-09-17T16:14:10.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:10.566 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:10.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:10.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:10 smithi173 ceph-mon[30530]: pgmap v510: 17 pgs: 2 creating+peering, 7 active+clean, 8 unknown; 577 KiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:10 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:10 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:10 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2082569926' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:14:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:10 smithi173 ceph-mon[30530]: osdmap e211: 12 total, 12 up, 12 in 2024-09-17T16:14:10.667 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:10.693 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:10.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:10.795 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:10 smithi071 ceph-mon[30002]: pgmap v510: 17 pgs: 2 creating+peering, 7 active+clean, 8 unknown; 577 KiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:10 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:10 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:10 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2082569926' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:14:10.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:10 smithi071 ceph-mon[30002]: osdmap e211: 12 total, 12 up, 12 in 2024-09-17T16:14:10.821 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:10.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:10.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:10.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:10.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:10.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:11.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:11.074 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:14:11.075 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:14:11.075 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 211, 2024-09-17T16:14:11.075 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:14:11.075 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:14:11.075 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:14:10.311264+0000", 2024-09-17T16:14:11.075 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:14:11.075 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:09:29.753083+0000", 2024-09-17T16:14:11.075 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:14:11.075 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5801984, 2024-09-17T16:14:11.075 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "noscrub", 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:14:11.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 3, 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "jewel", 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:14:11.077 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:14:11.078 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "21", 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:14:11.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:14:11.080 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:14:11.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:14:11.082 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:14:11.083 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 3, 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_1", 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:14:06.897515+0000", 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:14:11.084 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 16, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 16, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 16, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 16, 2024-09-17T16:14:11.085 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 16, 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "210", 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:14:11.086 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:14:11.087 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:14:11.088 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": {}, 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:14:11.089 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 2.059999942779541, 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.059999942779541, 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1.0900000333786011, 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 2.25, 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 2.25, 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 173, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.091 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.092 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.093 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.093 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:11.094 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:14:11.094 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:14:11.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.094 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:14:11.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.097 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.098 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.099 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.100 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.101 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.103 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 111, 2024-09-17T16:14:11.104 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.105 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.106 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:14:11.109 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.110 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.111 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:14:11.114 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.115 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:14:11.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:14:11.120 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.121 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.122 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.123 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:14:11.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.126 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:14:11.127 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:14:11.128 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:14:11.129 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.131 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.132 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:14:11.133 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:14:11.134 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:14:11.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.138 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:14:11.139 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.140 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.141 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.142 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:14:11.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.144 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.145 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 176, 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:14:11.146 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.147 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.151 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:14:11.151 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:14:11.151 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.151 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:11.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:11.152 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.152 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.152 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.152 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.152 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 165, 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.153 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:14:11.154 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:14:11.154 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.154 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.154 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.154 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.154 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.154 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.155 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.157 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:14:11.158 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:14:11.159 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:14:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:14:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:14:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:14:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.161 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.162 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.163 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:14:11.163 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:14:11.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.163 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:14:11.163 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:14:11.163 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:14:11.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:14:11.164 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:13:52.730368+0000", 2024-09-17T16:14:11.165 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:12:49.301926+0000", 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.166 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:13:33.219397+0000", 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.167 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:13:04.914567+0000", 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.168 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:14:10.073643+0000", 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:14:00.116712+0000", 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.169 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:13:02.216163+0000", 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.170 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:14:02.223339+0000", 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:13:24.141986+0000", 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:13:08.675519+0000", 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:14:11.172 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:14:04.758968+0000", 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:14:11.173 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:13:58.697151+0000", 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [], 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:14:11.174 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:14:11.175 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:14:11.176 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:14:11.177 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:14:11.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:11.177 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:14:11.221 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:11.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:11.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:11.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:11.380 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:11.403 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:11.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:11.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:11.530 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:11.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:11.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:11.632 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:11.654 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:11.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:11.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2136248049' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:14:11.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:11.778 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:11.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:11.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2136248049' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:14:11.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:11.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2136248049' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:14:11.906 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:11.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:11.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:12.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:12.033 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:12.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:12.053 INFO:tasks.thrashosds.thrasher:j is {'epoch': 211, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:14:10.311264+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:09:29.753083+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-17T16:14:06.897515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '210', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.059999942779541, 'score_stable': 2.059999942779541, 'optimal_score': 1.090000033378601, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 208, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 176, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 165, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:13:52.730368+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:12:49.301926+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:13:33.219397+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:13:04.914567+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:14:10.073643+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:14:00.116712+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:13:02.216163+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:14:02.223339+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:13:24.141986+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:13:08.675519+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:14:04.758968+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:13:58.697151+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:14:12.053 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-17T16:14:12.135 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:12.161 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:12.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:12.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:12.287 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:12.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:12.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:12.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:12.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:12.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:12.543 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:12.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:12.544 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:12.551 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:14:12.552 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:14:12.552 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:12 smithi017 ceph-mon[27571]: pgmap v512: 17 pgs: 2 creating+peering, 15 active+clean; 577 KiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:12.572 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:12.577 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:14:12.578 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:14:12.604 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:14:12.604 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:14:12.625 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:14:12.625 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:14:12.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:12.647 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:14:12.647 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:14:12.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:14:12.669 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:14:12.670 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:12.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:12.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:12.691 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:14:12.691 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:14:12.713 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:14:12.714 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:14:12.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:14:12.736 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:14:12.758 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:14:12.758 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:14:12.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:12.780 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:14:12.780 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:14:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:12.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:12.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:12.805 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:14:12.805 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:14:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:12 smithi071 ceph-mon[30002]: pgmap v512: 17 pgs: 2 creating+peering, 15 active+clean; 577 KiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:12.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:12 smithi173 ceph-mon[30530]: pgmap v512: 17 pgs: 2 creating+peering, 15 active+clean; 577 KiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:12.900 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:12.926 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:12.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:13.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:13.053 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:13.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:13.155 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:13.178 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:13.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:13.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:13.249 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:14:13.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:13.313 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:13.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:13.405 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-17T16:14:13.415 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:13.440 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:13.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:13.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:13.567 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:13.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:13.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:13.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:13.693 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:13.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4247670692' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:13 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:13 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2642625739' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:14:13.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4247670692' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:13 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:13 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2642625739' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:14:13.817 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:13.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4247670692' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:13 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:13 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2642625739' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:14:13.918 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:13.944 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:13.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:14.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:14.071 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:14.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:14.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:14.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:14.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:14.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:14.301 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:14.301 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:14.302 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:14.325 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:14.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:14.427 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:14.450 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:14.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:14.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:14.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:14.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:14.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:14.679 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:14.702 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:14.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:14.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:14.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:14.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2642625739' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:14:14.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:14 smithi017 ceph-mon[27571]: osdmap e212: 12 total, 12 up, 12 in 2024-09-17T16:14:14.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:14 smithi017 ceph-mon[27571]: pgmap v514: 17 pgs: 17 active+clean; 577 KiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:14.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:14.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2642625739' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:14:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:14 smithi071 ceph-mon[30002]: osdmap e212: 12 total, 12 up, 12 in 2024-09-17T16:14:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:14 smithi071 ceph-mon[30002]: pgmap v514: 17 pgs: 17 active+clean; 577 KiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:14.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:14.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:14.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2642625739' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:14:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:14 smithi173 ceph-mon[30530]: osdmap e212: 12 total, 12 up, 12 in 2024-09-17T16:14:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:14 smithi173 ceph-mon[30530]: pgmap v514: 17 pgs: 17 active+clean; 577 KiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:14:14.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:14.956 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:14.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:14.966 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:14:15.058 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:15.081 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:15.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:15.116 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-17T16:14:15.119 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T16:14:15.119 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T16:14:15.181 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-09-17T16:14:15.182 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T16:14:15.182 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_66272 2024-09-17T16:14:15.182 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:14:15.182 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-09-17T16:14:15.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:15.216 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:15.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:15.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:15.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:15.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:15.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:15.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:15.484 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:15.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:15.503 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:15.544 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:15.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:15.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:15.608 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:15.663 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:15 smithi017 ceph-mon[27571]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:14:15.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:15.732 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:15.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:15 smithi071 ceph-mon[30002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:14:15.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:15.857 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:15.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:15.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:15 smithi173 ceph-mon[30530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:14:15.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:15.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:15.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:16.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:16.108 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:16.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:16.182 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 1706 1690 105.615 105.625 0.00642813 0.00894787 2024-09-17T16:14:16.210 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:16.231 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:16.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:16.333 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:16.359 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:16.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:16.461 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:16.487 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:16.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:16.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:16.612 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:16.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:16.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:16 smithi017 ceph-mon[27571]: pgmap v515: 17 pgs: 17 active+clean; 12 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s wr, 27 op/s 2024-09-17T16:14:16.714 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:16.734 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:16.734 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:16.745 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:16.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:16.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:16 smithi071 ceph-mon[30002]: pgmap v515: 17 pgs: 17 active+clean; 12 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s wr, 27 op/s 2024-09-17T16:14:16.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:16.872 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:16.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:16.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:16 smithi173 ceph-mon[30530]: pgmap v515: 17 pgs: 17 active+clean; 12 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s wr, 27 op/s 2024-09-17T16:14:16.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:16.996 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:16.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:17.054 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:14:17.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T16:14:17.098 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:17.121 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:17.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:17.182 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 3187 3171 99.0842 92.5625 0.00153843 0.00981311 2024-09-17T16:14:17.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:17.251 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:17.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:17.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:17.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:17.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:17.426 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:14:17.481 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:17.482 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:14:17.490 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:14:17.512 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:17.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:17.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:17.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:17.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:17.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/247806573' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:17.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:17.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/247806573' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:17.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:17.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:17.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/247806573' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:17.863 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:17.886 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:17.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:17.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:18.017 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:18.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:18.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:18.133 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:18.145 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:18.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:18.182 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 4772 4756 99.0739 99.0625 0.0162693 0.00999107 2024-09-17T16:14:18.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:18.273 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:18.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:18.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:18.401 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:18.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:18.503 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:18.530 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:18.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:18.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:18.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:18.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:18.655 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:18.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:18 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:18.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/247806573' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:18.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:18 smithi017 ceph-mon[27571]: osdmap e213: 12 total, 12 up, 12 in 2024-09-17T16:14:18.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:18 smithi017 ceph-mon[27571]: pgmap v517: 17 pgs: 17 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s wr, 107 op/s 2024-09-17T16:14:18.715 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:18.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:18.777 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:18.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:18 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/247806573' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:18 smithi071 ceph-mon[30002]: osdmap e213: 12 total, 12 up, 12 in 2024-09-17T16:14:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:18 smithi071 ceph-mon[30002]: pgmap v517: 17 pgs: 17 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s wr, 107 op/s 2024-09-17T16:14:18.879 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:18.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:18 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/247806573' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:18 smithi173 ceph-mon[30530]: osdmap e213: 12 total, 12 up, 12 in 2024-09-17T16:14:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:18 smithi173 ceph-mon[30530]: pgmap v517: 17 pgs: 17 active+clean; 51 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s wr, 107 op/s 2024-09-17T16:14:18.903 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:18.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:19.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:19.033 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:19.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:19.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:19.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:19.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:19.182 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 6313 6297 98.3818 96.3125 0.0162099 0.0101368 2024-09-17T16:14:19.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:19.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:19.286 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:19.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:19.312 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T16:14:19.388 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:19.413 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:19.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:19.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:19.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:19.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:19.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:19.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:19.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:19.664 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:19.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2783039502' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:14:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2202322464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2783039502' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:14:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:19 smithi017 ceph-mon[27571]: osdmap e214: 12 total, 12 up, 12 in 2024-09-17T16:14:19.765 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:19.787 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:19.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:19.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2783039502' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:14:19.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2202322464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2783039502' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:14:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:19 smithi071 ceph-mon[30002]: osdmap e214: 12 total, 12 up, 12 in 2024-09-17T16:14:19.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:19.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2783039502' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:14:19.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2202322464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:19.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2783039502' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:14:19.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:19 smithi173 ceph-mon[30530]: osdmap e214: 12 total, 12 up, 12 in 2024-09-17T16:14:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:19.911 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:19.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:20.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:20.034 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:20.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:20.099 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:14:20.100 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:14:20.121 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:14:20.121 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:14:20.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:14:20.135 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:20.142 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:14:20.142 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:14:20.165 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:20.166 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:14:20.166 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:14:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:20.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:20.182 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 7279 7263 90.7781 60.375 0.00113061 0.0102908 2024-09-17T16:14:20.186 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:14:20.186 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:14:20.208 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:14:20.208 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:14:20.230 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:14:20.230 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:14:20.251 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:14:20.251 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:14:20.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:20.274 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:14:20.274 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:14:20.289 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:20.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:20.312 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:14:20.312 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:14:20.333 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:14:20.333 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:14:20.359 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:14:20.359 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:14:20.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:20.411 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:20.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:20.513 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:20.534 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:20.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:20.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:20.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:20.659 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:20.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:20 smithi017 ceph-mon[27571]: pgmap v519: 17 pgs: 17 active+clean; 244 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 612 op/s 2024-09-17T16:14:20.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:20.782 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:20.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:20 smithi071 ceph-mon[30002]: pgmap v519: 17 pgs: 17 active+clean; 244 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 612 op/s 2024-09-17T16:14:20.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:20.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:20 smithi173 ceph-mon[30530]: pgmap v519: 17 pgs: 17 active+clean; 244 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 612 op/s 2024-09-17T16:14:20.905 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:20.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:21.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:21.029 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:21.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:21.131 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:21.153 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:21.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:21.183 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 7343 7327 76.3143 4 0.00801691 0.0103553 2024-09-17T16:14:21.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:21.280 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:21.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:21.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:21.403 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:21.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:21.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:21.506 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:21.532 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:21.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:21.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:21.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:21.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:21.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T16:14:21.763 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:21.784 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:21.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:21.887 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:21.904 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:13535 2.11 2:8808aac6:::benchmark_data_smithi173_55121_object13534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 189.566148857, 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314941879999999, 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.905 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 13535 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:14:21.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335349+0000", 2024-09-17T16:14:21.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3599999999999999e-06 2024-09-17T16:14:21.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335378+0000", 2024-09-17T16:14:21.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8965000000000001e-05 2024-09-17T16:14:21.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335391+0000", 2024-09-17T16:14:21.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.257e-05 2024-09-17T16:14:21.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335398+0000", 2024-09-17T16:14:21.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3499999999999999e-06 2024-09-17T16:14:21.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.367429+0000", 2024-09-17T16:14:21.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.032030422000000003 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.367439+0000", 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0451e-05 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.366280+0000", 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.99884132299999995 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.366842+0000", 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00056174700000000003 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:14925 2.e 2:743cbf17:::benchmark_data_smithi173_55121_object14924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.264388+0000", 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 187.63710881, 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43099893299999997, 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.911 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 14925 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264388+0000", 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264388+0000", 2024-09-17T16:14:21.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264390+0000", 2024-09-17T16:14:21.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.677e-06 2024-09-17T16:14:21.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264432+0000", 2024-09-17T16:14:21.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2292000000000001e-05 2024-09-17T16:14:21.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264447+0000", 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.518e-05 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264455+0000", 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2490000000000001e-06 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264541+0000", 2024-09-17T16:14:21.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5760000000000006e-05 2024-09-17T16:14:21.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264676+0000", 2024-09-17T16:14:21.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134487 2024-09-17T16:14:21.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:14:21.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264784+0000", 2024-09-17T16:14:21.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108402 2024-09-17T16:14:21.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:21.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265461+0000", 2024-09-17T16:14:21.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067649699999999997 2024-09-17T16:14:21.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265775+0000", 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031408899999999998 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695211+0000", 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42943639900000002 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695308+0000", 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7176000000000006e-05 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695387+0000", 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8725000000000003e-05 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:14925 2.e e106/95 ondisk, result = 0)", 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 187.63566133500001, 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42955775600000001, 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.919 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265844+0000", 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2450000000000001e-06 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695035+0000", 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42919093699999999 2024-09-17T16:14:21.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:21.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695052+0000", 2024-09-17T16:14:21.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.772e-05 2024-09-17T16:14:21.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695186+0000", 2024-09-17T16:14:21.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013379200000000001 2024-09-17T16:14:21.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695202+0000", 2024-09-17T16:14:21.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5519000000000002e-05 2024-09-17T16:14:21.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695393+0000", 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019176600000000001 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:14944 2.e 2:79e7edec:::benchmark_data_smithi173_55121_object14943:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.358500+0000", 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 187.54299672100001, 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34235612100000001, 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.926 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 14944 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358500+0000", 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358500+0000", 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358502+0000", 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8899999999999999e-06 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358530+0000", 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8248000000000001e-05 2024-09-17T16:14:21.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358548+0000", 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8068e-05 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358555+0000", 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.934e-06 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358614+0000", 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8488e-05 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358718+0000", 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103991 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358818+0000", 2024-09-17T16:14:21.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100374 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359536+0000", 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00071760900000000002 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359808+0000", 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027181699999999999 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700797+0000", 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.340989443 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700830+0000", 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2543999999999999e-05 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700856+0000", 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6715000000000001e-05 2024-09-17T16:14:21.932 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:14944 2.e e106/95 ondisk, result = 0)", 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.359723+0000", 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 187.54177402600001, 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.341137735, 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:14:21.933 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359723+0000", 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359723+0000", 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359724+0000", 2024-09-17T16:14:21.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9299999999999996e-07 2024-09-17T16:14:21.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359729+0000", 2024-09-17T16:14:21.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5770000000000001e-06 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700733+0000", 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34100362499999998 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700739+0000", 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3199999999999996e-06 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700789+0000", 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9845000000000001e-05 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700794+0000", 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6299999999999997e-06 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700861+0000", 2024-09-17T16:14:21.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6845000000000001e-05 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(2.4 116/112 [PushOp(2:23763086:::benchmark_data_smithi173_55121_object9285:head, version: 101'619, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:23763086:::benchmark_data_smithi173_55121_object9285:head@101'619, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:25.348190+0000", 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 176.553306491, 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61457114000000002, 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:14:21.938 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348190+0000", 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348190+0000", 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348191+0000", 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3000000000000001e-07 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348211+0000", 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9811999999999999e-05 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.961507+0000", 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61329585900000005 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.961527+0000", 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0744999999999999e-05 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.962424+0000", 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089668899999999999 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.962438+0000", 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3458000000000001e-05 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.962762+0000", 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000324047 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.942 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(2.4 117/112 [PushOp(2:26e71621:::benchmark_data_smithi173_55121_object17115:head, version: 108'1004, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:26e71621:::benchmark_data_smithi173_55121_object17115:head@108'1004, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:26.624251+0000", 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 175.27724596100001, 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.252196103, 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624251+0000", 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624251+0000", 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624252+0000", 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.029e-06 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624283+0000", 2024-09-17T16:14:21.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0960000000000002e-05 2024-09-17T16:14:21.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875818+0000", 2024-09-17T16:14:21.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25153555500000002 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875841+0000", 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2894000000000001e-05 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875950+0000", 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108758 2024-09-17T16:14:21.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875961+0000", 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0784e-05 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.876447+0000", 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048612299999999998 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:24327 2.17 e120/104 2:ecf8c341:::benchmark_data_smithi173_55121_object24326:head v 120'1226, mlcod=120'1223)", 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 172.48359737199999, 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.245189986, 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:14:21.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8199999999999999e-07 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417946+0000", 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6263000000000001e-05 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662362+0000", 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24441564700000001 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662376+0000", 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3573e-05 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662484+0000", 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108559 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662517+0000", 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2755000000000001e-05 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.663062+0000", 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054487399999999999 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.663090+0000", 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7633000000000001e-05 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25228 2.1 2:8352dd7d:::benchmark_data_smithi173_55121_object25227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.786229+0000", 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 171.11526743300001, 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28014749900000002, 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:21.953 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25228 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786229+0000", 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786229+0000", 2024-09-17T16:14:21.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786230+0000", 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9199999999999999e-07 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786266+0000", 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5939999999999998e-05 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786275+0000", 2024-09-17T16:14:21.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0140000000000004e-06 2024-09-17T16:14:21.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786280+0000", 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4710000000000001e-06 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786324+0000", 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3829999999999999e-05 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786410+0000", 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6013000000000005e-05 2024-09-17T16:14:21.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786496+0000", 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5784000000000004e-05 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.787047+0000", 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00055157999999999997 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.788017+0000", 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097012900000000002 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066298+0000", 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27828037 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066347+0000", 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9508000000000001e-05 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066377+0000", 2024-09-17T16:14:21.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9867999999999999e-05 2024-09-17T16:14:21.960 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.960 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.960 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.960 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25249 2.e 2:7b694fa0:::benchmark_data_smithi173_55121_object25248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:14:21.960 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.912148+0000", 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 170.98934919199999, 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33397465900000001, 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25249 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912148+0000", 2024-09-17T16:14:21.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912148+0000", 2024-09-17T16:14:21.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912149+0000", 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912174+0000", 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5080999999999999e-05 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912187+0000", 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3311e-05 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912193+0000", 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6110000000000001e-06 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912270+0000", 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6598000000000001e-05 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912380+0000", 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110389 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912476+0000", 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5493000000000002e-05 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913408+0000", 2024-09-17T16:14:21.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093208600000000005 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.965233+0000", 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.051825489000000002 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246005+0000", 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28077230800000003 2024-09-17T16:14:21.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:21.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246071+0000", 2024-09-17T16:14:21.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6000999999999994e-05 2024-09-17T16:14:21.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246122+0000", 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0955999999999998e-05 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:25249 2.e e121/95 ondisk, result = 0)", 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.913496+0000", 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 170.98800097700001, 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.332636237, 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.968 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913496+0000", 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913496+0000", 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913497+0000", 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913500+0000", 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5449999999999999e-06 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245930+0000", 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33242972799999998 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245935+0000", 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9479999999999999e-06 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245989+0000", 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4187999999999997e-05 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245999+0000", 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9729999999999992e-06 2024-09-17T16:14:21.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246132+0000", 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000133118 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:26909 2.e 2:76c214cd:::benchmark_data_smithi173_55121_object26908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.325897+0000", 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 169.575599635, 2024-09-17T16:14:21.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23800827699999999, 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26909 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325897+0000", 2024-09-17T16:14:21.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325897+0000", 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325898+0000", 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1379999999999999e-06 2024-09-17T16:14:21.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325939+0000", 2024-09-17T16:14:21.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0466000000000002e-05 2024-09-17T16:14:21.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325950+0000", 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1287000000000001e-05 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325955+0000", 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1730000000000003e-06 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326000+0000", 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4674e-05 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326082+0000", 2024-09-17T16:14:21.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1706999999999995e-05 2024-09-17T16:14:21.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:14:21.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326170+0000", 2024-09-17T16:14:21.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7903000000000002e-05 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326971+0000", 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080154300000000005 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327360+0000", 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00038908699999999998 2024-09-17T16:14:21.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563817+0000", 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23645692900000001 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563856+0000", 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9150000000000003e-05 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563906+0000", 2024-09-17T16:14:21.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.922e-05 2024-09-17T16:14:21.980 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.980 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.980 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.980 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:26910 2.1 2:838cca45:::benchmark_data_smithi173_55121_object26909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:14:21.980 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:14:21.980 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 169.57402437799999, 2024-09-17T16:14:21.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26261527099999998, 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26910 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.23e-07 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327529+0000", 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5284000000000001e-05 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327538+0000", 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7790000000000007e-06 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327543+0000", 2024-09-17T16:14:21.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4700000000000004e-06 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327599+0000", 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6100999999999997e-05 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327722+0000", 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012263399999999999 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:14:21.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327808+0000", 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6803000000000002e-05 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.328412+0000", 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060394300000000001 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.328847+0000", 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00043514399999999999 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.590005+0000", 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26115713200000001 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.590043+0000", 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.824e-05 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.590088+0000", 2024-09-17T16:14:21.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5018000000000002e-05 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:26911 2.e 2:7501b6f7:::benchmark_data_smithi173_55121_object26910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.329019+0000", 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 169.57247812, 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26290611800000002, 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.987 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.988 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:21.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:21.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:21.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:21.988 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26911 2024-09-17T16:14:21.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.988 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329019+0000", 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329019+0000", 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329020+0000", 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329082+0000", 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2477000000000005e-05 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329093+0000", 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0814e-05 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329100+0000", 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1339999999999998e-06 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329141+0000", 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0915999999999999e-05 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329219+0000", 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7902000000000001e-05 2024-09-17T16:14:21.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329304+0000", 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4820000000000004e-05 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330001+0000", 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00069732400000000003 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330213+0000", 2024-09-17T16:14:21.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000212563 2024-09-17T16:14:21.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:21.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.591852+0000", 2024-09-17T16:14:21.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26163829100000002 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.591886+0000", 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4662000000000001e-05 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.591925+0000", 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8470000000000003e-05 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.994 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:26909 2.e e121/95 ondisk, result = 0)", 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.330196+0000", 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 169.57130073100001, 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.233715383, 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:14:21.995 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330196+0000", 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330196+0000", 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:21.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330197+0000", 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5299999999999999e-07 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330200+0000", 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.033e-06 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563761+0000", 2024-09-17T16:14:21.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23356138900000001 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563767+0000", 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4569999999999996e-06 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563808+0000", 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1121999999999997e-05 2024-09-17T16:14:21.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:21.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563813+0000", 2024-09-17T16:14:21.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4249999999999997e-06 2024-09-17T16:14:21.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:21.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:21.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:21.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563912+0000", 2024-09-17T16:14:21.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8504000000000004e-05 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:27009 2.e 2:7c49ab26:::benchmark_data_smithi173_55121_object27008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:33.169267+0000", 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 168.73222977500001, 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33189007700000001, 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:22.000 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 27009 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169267+0000", 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:22.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169267+0000", 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169270+0000", 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9469999999999999e-06 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169342+0000", 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1785999999999993e-05 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:22.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169372+0000", 2024-09-17T16:14:22.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0399000000000002e-05 2024-09-17T16:14:22.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.003 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:22.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:22.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:22.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169387+0000", 2024-09-17T16:14:22.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4323e-05 2024-09-17T16:14:22.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:22.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169516+0000", 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012973600000000001 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169614+0000", 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8103000000000006e-05 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169705+0000", 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0989000000000006e-05 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.170561+0000", 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085536800000000001 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.170823+0000", 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026205999999999999 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.501034+0000", 2024-09-17T16:14:22.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33021098799999998 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.501104+0000", 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0321000000000007e-05 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.501157+0000", 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3056999999999997e-05 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:22.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.008 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:29460 2.1 2:83aee601:::benchmark_data_smithi173_55121_object29459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-17T16:14:22.008 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:14:22.008 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 152.70383442799999, 2024-09-17T16:14:22.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31850449400000003, 2024-09-17T16:14:22.008 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:22.008 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:22.008 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:22.009 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:22.009 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:22.009 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:22.009 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 29460 2024-09-17T16:14:22.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.009 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:22.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:22.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:14:22.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:22.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:22.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:14:22.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:22.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:22.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197664+0000", 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1289999999999999e-06 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197740+0000", 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6361999999999998e-05 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197753+0000", 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3453e-05 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197759+0000", 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7610000000000004e-06 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197820+0000", 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0735e-05 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:22.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197960+0000", 2024-09-17T16:14:22.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013987799999999999 2024-09-17T16:14:22.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:14:22.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198120+0000", 2024-09-17T16:14:22.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016034 2024-09-17T16:14:22.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198880+0000", 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075943099999999997 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.322818+0000", 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.123938883 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516108+0000", 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.193289187 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516141+0000", 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3691e-05 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516167+0000", 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5644e-05 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:129847 2.1 2:86a30a1b:::benchmark_data_smithi173_55121_object53358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.102454457999997, 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29781534999999998, 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:22.016 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 129847 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:22.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:22.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799048+0000", 2024-09-17T16:14:22.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6289999999999998e-06 2024-09-17T16:14:22.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:22.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799056+0000", 2024-09-17T16:14:22.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3809999999999999e-06 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799074+0000", 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7864e-05 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799085+0000", 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1076e-05 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:22.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799177+0000", 2024-09-17T16:14:22.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1402000000000003e-05 2024-09-17T16:14:22.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:22.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799338+0000", 2024-09-17T16:14:22.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000161263 2024-09-17T16:14:22.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:14:22.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799482+0000", 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000143638 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801720+0000", 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022381739999999999 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.901264+0000", 2024-09-17T16:14:22.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.099544493999999997 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096745+0000", 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.195480396 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096804+0000", 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8990000000000003e-05 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:22.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096858+0000", 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4042999999999997e-05 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:129847 2.1 e184/104 ondisk, result = 0)", 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.099497630000002, 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29486439199999998, 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:22.023 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.802000+0000", 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1600000000000001e-07 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.802005+0000", 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9060000000000001e-06 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:22.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096600+0000", 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.294595249 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096606+0000", 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.733e-06 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096731+0000", 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000124735 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096739+0000", 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096864+0000", 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012515399999999999 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.027 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:135542 2.e 2:76ccd15d:::benchmark_data_smithi173_55121_object59053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:35.745945+0000", 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 46.155551574, 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.222641539, 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 135542 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:14:22.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.745945+0000", 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.745945+0000", 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:14:22.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.745947+0000", 2024-09-17T16:14:22.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4950000000000001e-06 2024-09-17T16:14:22.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:14:22.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.745953+0000", 2024-09-17T16:14:22.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6239999999999996e-06 2024-09-17T16:14:22.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:14:22.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.745969+0000", 2024-09-17T16:14:22.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5255e-05 2024-09-17T16:14:22.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:14:22.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.745976+0000", 2024-09-17T16:14:22.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0480000000000001e-06 2024-09-17T16:14:22.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.746037+0000", 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0778e-05 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.746127+0000", 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0927999999999994e-05 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.746191+0000", 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3060999999999998e-05 2024-09-17T16:14:22.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.749357+0000", 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031665780000000002 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.749958+0000", 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060109499999999995 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.968520+0000", 2024-09-17T16:14:22.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.218562066 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.968558+0000", 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7391000000000002e-05 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.968587+0000", 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9220000000000001e-05 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:14:22.036 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:14:22.036 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:14:22.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:22.128 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:22.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:22.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:22.183 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 7343 7327 65.4121 0 - 0.0103553 2024-09-17T16:14:22.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:22.255 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:22.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:22.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:22.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:22.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:22.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:22.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:22.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:22.511 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:22.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:22.613 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:22.638 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:22.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:22.740 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:22.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:22.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:22.895 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:22.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:22.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:22.999 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:14:23.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:23.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:23.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:14:23.081 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:23.082 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:23.092 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:22 smithi017 ceph-mon[27571]: pgmap v520: 17 pgs: 17 active+clean; 426 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 850 op/s 2024-09-17T16:14:23.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:23.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:22 smithi173 ceph-mon[30530]: pgmap v520: 17 pgs: 17 active+clean; 426 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 850 op/s 2024-09-17T16:14:23.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:23.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:23.183 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 7343 7327 57.2358 0 - 0.0103553 2024-09-17T16:14:23.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:23.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:23.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:23.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:22 smithi071 ceph-mon[30002]: pgmap v520: 17 pgs: 17 active+clean; 426 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 850 op/s 2024-09-17T16:14:23.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:23.401 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:23.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:23.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:23.502 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:23.528 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:23.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:23.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:23.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:23.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:23.758 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:23.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:23.882 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:23.904 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:23.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:24.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:24.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:24.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:24.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:24.033 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:24.039 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:24.134 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:24.159 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:24.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:24.183 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 7343 7327 50.8761 0 - 0.0103553 2024-09-17T16:14:24.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:24.285 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:24.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:24.388 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:24.410 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:24.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:24.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:24.512 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:24.535 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:24.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:24.636 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:24.658 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:24.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:24.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:24.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:24.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:24.829 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:14:24.830 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T16:14:24.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:24.914 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:24.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:24.971 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:14:24.971 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:14:24.971 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:14:24.971 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:14:25.016 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:25.038 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:25.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:25.140 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:25.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:24 smithi173 ceph-mon[30530]: pgmap v521: 17 pgs: 17 active+clean; 449 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s 2024-09-17T16:14:25.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2151407987' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:14:25.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/697845896' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:14:25.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2905830037' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:25.163 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:25.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:25.183 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 7343 7327 45.7884 0 - 0.0103553 2024-09-17T16:14:25.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:24 smithi017 ceph-mon[27571]: pgmap v521: 17 pgs: 17 active+clean; 449 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s 2024-09-17T16:14:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2151407987' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:14:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/697845896' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:14:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2905830037' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:25.264 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:25.290 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:25.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:25.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:24 smithi071 ceph-mon[30002]: pgmap v521: 17 pgs: 17 active+clean; 449 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s 2024-09-17T16:14:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2151407987' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:14:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/697845896' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:14:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2905830037' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:25.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:25.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:25.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:25.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:25.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:25.544 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:25.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:25.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:25.629 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:14:25.629 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:14:25.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:25.650 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:14:25.650 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:14:25.671 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:14:25.672 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:14:25.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:25.672 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:25.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:25.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:14:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:14:25.715 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:14:25.715 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:14:25.737 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:14:25.737 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:14:25.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:14:25.759 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:14:25.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:25.781 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:14:25.781 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:14:25.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:25.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:25.800 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:25.803 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:14:25.803 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:14:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:14:25.825 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:14:25.847 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:14:25.847 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:14:25.868 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:14:25.869 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:14:25.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:25 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:25.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2151407987' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:14:25.878 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/697845896' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:14:25.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:25 smithi017 ceph-mon[27571]: osdmap e215: 12 total, 12 up, 12 in 2024-09-17T16:14:25.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:25 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:25.879 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:25 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:25.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:25.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:25.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:25.927 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:26.028 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:26.050 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:26.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:25 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2151407987' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:14:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/697845896' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:14:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:25 smithi173 ceph-mon[30530]: osdmap e215: 12 total, 12 up, 12 in 2024-09-17T16:14:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:25 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:25 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:26.152 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:26.177 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:26.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:26.183 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 7343 7327 41.6258 0 - 0.0103553 2024-09-17T16:14:26.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:26.304 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:26.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:26.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:25 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:26.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2151407987' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:14:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/697845896' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:14:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:25 smithi071 ceph-mon[30002]: osdmap e215: 12 total, 12 up, 12 in 2024-09-17T16:14:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:25 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:25 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:26.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:26.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:26.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:26.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T16:14:26.534 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:26.582 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:26.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:26.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:26.708 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:26.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:26.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:26.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:26.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:26.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:26.960 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:26.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:27.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:27.084 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:27.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:27.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:26 smithi173 ceph-mon[30530]: pgmap v523: 17 pgs: 17 active+clean; 459 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 815 op/s 2024-09-17T16:14:27.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:26 smithi173 ceph-mon[30530]: osdmap e216: 12 total, 12 up, 12 in 2024-09-17T16:14:27.184 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 7343 7327 38.1569 0 - 0.0103553 2024-09-17T16:14:27.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:27.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:27.213 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:27.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:26 smithi017 ceph-mon[27571]: pgmap v523: 17 pgs: 17 active+clean; 459 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 815 op/s 2024-09-17T16:14:27.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:26 smithi017 ceph-mon[27571]: osdmap e216: 12 total, 12 up, 12 in 2024-09-17T16:14:27.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:26 smithi071 ceph-mon[30002]: pgmap v523: 17 pgs: 17 active+clean; 459 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 815 op/s 2024-09-17T16:14:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:26 smithi071 ceph-mon[30002]: osdmap e216: 12 total, 12 up, 12 in 2024-09-17T16:14:27.340 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:27.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:27.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:27.464 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:27.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:27.488 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:27.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:14:27.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:27.591 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:27.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:27.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:27.702 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:14:27.718 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:27.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:27.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:27.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:27.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:27.948 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:27.974 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:27.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:28.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:28.100 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:28.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:28.184 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 7343 7327 35.2217 0 - 0.0103553 2024-09-17T16:14:28.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:28.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:28.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:28.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:28.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:28.355 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:28.453 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:28.457 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:28.482 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:28.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:28.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:28.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:28.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:28.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:28.738 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:28.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:28.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:28.843 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T16:14:28.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:28.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:28.967 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:28.979 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:28.980 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:28.980 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:28 smithi017 ceph-mon[27571]: pgmap v525: 17 pgs: 17 active+clean; 459 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 428 op/s 2024-09-17T16:14:28.980 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2483351652' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:14:28.989 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:28.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:29.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:29.115 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:29.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:29.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:28 smithi173 ceph-mon[30530]: pgmap v525: 17 pgs: 17 active+clean; 459 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 428 op/s 2024-09-17T16:14:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2483351652' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:14:29.184 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 7343 7327 32.7058 0 - 0.0103553 2024-09-17T16:14:29.216 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:29.239 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:29.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:29.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:28 smithi071 ceph-mon[30002]: pgmap v525: 17 pgs: 17 active+clean; 459 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 428 op/s 2024-09-17T16:14:29.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2483351652' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:14:29.341 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:29.363 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:29.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:29.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:29.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:29.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:29.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-17T16:14:29.593 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:29.614 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:29.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:29.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:29.716 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:29.740 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:29.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:29.826 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:29.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:29.867 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:29.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:29.969 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:29.992 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:29.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:30.094 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:30.120 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:30.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:29 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:29 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:29 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:29 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2483351652' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:14:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:29 smithi173 ceph-mon[30530]: osdmap e217: 12 total, 12 up, 12 in 2024-09-17T16:14:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:29 smithi173 ceph-mon[30530]: osdmap e218: 12 total, 12 up, 12 in 2024-09-17T16:14:30.184 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 7343 7327 30.5253 0 - 0.0103553 2024-09-17T16:14:30.191 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:14:30.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:29 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:30.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:29 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:30.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:29 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:30.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:29 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:30.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2483351652' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:14:30.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:29 smithi017 ceph-mon[27571]: osdmap e217: 12 total, 12 up, 12 in 2024-09-17T16:14:30.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:29 smithi017 ceph-mon[27571]: osdmap e218: 12 total, 12 up, 12 in 2024-09-17T16:14:30.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:30.249 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:30.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:30.283 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:14:30.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:29 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:29 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:29 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:29 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2483351652' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:14:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:29 smithi071 ceph-mon[30002]: osdmap e217: 12 total, 12 up, 12 in 2024-09-17T16:14:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:29 smithi071 ceph-mon[30002]: osdmap e218: 12 total, 12 up, 12 in 2024-09-17T16:14:30.351 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:30.390 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:30.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:30.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:30.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:30.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:30.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:30.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:30.642 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:30.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:30.744 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:30.766 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:30.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:30.810 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:30.830 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:30.868 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:30.891 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:30.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:30.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:31.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:31.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:31.121 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:31.148 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:31.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:31.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:31.184 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 7343 7327 28.6175 0 - 0.0103553 2024-09-17T16:14:31.250 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:31.276 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:31.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:31.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:31.329 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:14:31.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:31.382 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:31.382 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:31.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:31.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:31.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:31.531 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:31.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:31.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:31.570 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:31 smithi017 ceph-mon[27571]: pgmap v528: 17 pgs: 17 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s wr, 3 op/s 2024-09-17T16:14:31.570 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4214712692' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:31.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:31 smithi173 ceph-mon[30530]: pgmap v528: 17 pgs: 17 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s wr, 3 op/s 2024-09-17T16:14:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4214712692' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:31.656 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:31.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:31.759 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:31.785 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:31.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:31 smithi071 ceph-mon[30002]: pgmap v528: 17 pgs: 17 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s wr, 3 op/s 2024-09-17T16:14:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4214712692' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:31.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:31.914 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:32.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:32.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:32.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:32.142 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:32.166 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:32.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:32.184 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 7343 7327 26.9341 0 - 0.0103553 2024-09-17T16:14:32.267 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:32.289 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:32.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:32.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:32.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:32.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:32.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:32 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:14:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:32 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:14:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4214712692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:32 smithi017 ceph-mon[27571]: osdmap e219: 12 total, 12 up, 12 in 2024-09-17T16:14:32.519 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:32.545 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:32.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:32 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:14:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:32 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:14:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4214712692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:32 smithi173 ceph-mon[30530]: osdmap e219: 12 total, 12 up, 12 in 2024-09-17T16:14:32.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:32.670 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:32.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:32.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:32.677 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:14:32.677 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:14:32.698 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:14:32.698 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:14:32.718 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:14:32.719 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:14:32.739 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:14:32.739 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:14:32.760 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:14:32.761 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:14:32.772 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:32.783 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:14:32.784 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:14:32.797 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:32.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:32.805 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:14:32.805 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:14:32.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:32 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:14:32.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:32 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:14:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4214712692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:32 smithi071 ceph-mon[30002]: osdmap e219: 12 total, 12 up, 12 in 2024-09-17T16:14:32.826 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:14:32.826 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:14:32.848 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:14:32.848 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:14:32.870 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:14:32.870 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:14:32.892 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:14:32.892 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:14:32.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:32.914 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:14:32.914 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:14:32.923 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:32.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:33.027 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:33.053 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:33.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:33.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:33.155 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:33.177 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:33.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:33.184 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 7343 7327 25.4378 0 - 0.0103553 2024-09-17T16:14:33.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:33.304 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:33.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:33.332 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T16:14:33.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:33.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:33.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:33.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:33.534 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:33.556 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:33.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:33 smithi173 ceph-mon[30530]: pgmap v530: 17 pgs: 17 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s wr, 2 op/s 2024-09-17T16:14:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/730562062' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:14:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2803787891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:33.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:33.675 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:33 smithi017 ceph-mon[27571]: pgmap v530: 17 pgs: 17 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s wr, 2 op/s 2024-09-17T16:14:33.675 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/730562062' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:14:33.675 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2803787891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:33.684 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:33.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:33.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:33 smithi071 ceph-mon[30002]: pgmap v530: 17 pgs: 17 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s wr, 2 op/s 2024-09-17T16:14:33.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/730562062' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:14:33.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2803787891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:33.786 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:33.809 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:33.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:33.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:33.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:33.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:34.039 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:34.060 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:34.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:34.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:34.184 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 7343 7327 24.0989 0 - 0.0103553 2024-09-17T16:14:34.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:34.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:34.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T16:14:34.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:34.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:34.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:34.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:34.442 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:34.444 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:34.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:14:34.467 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/730562062' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:14:34.467 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:34 smithi071 ceph-mon[30002]: osdmap e220: 12 total, 12 up, 12 in 2024-09-17T16:14:34.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/730562062' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:14:34.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:34 smithi017 ceph-mon[27571]: osdmap e220: 12 total, 12 up, 12 in 2024-09-17T16:14:34.544 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:34.567 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:34.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:34.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:34.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/730562062' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:14:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:34 smithi173 ceph-mon[30530]: osdmap e220: 12 total, 12 up, 12 in 2024-09-17T16:14:34.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:34.702 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:34.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:34.767 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:14:34.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:34.828 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:34.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:34.910 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:34.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:34.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:34.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:35.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:35.078 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:35.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:35.180 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:35.184 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:14:35.183826+0000 min lat: 0.00101616 max lat: 15.0111 avg lat: 0.0389259 2024-09-17T16:14:35.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:14:35.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 8164 8148 25.4593 3.66518 0.114484 0.0389259 2024-09-17T16:14:35.202 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:35.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:35.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:35.320 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:14:35.320 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:14:35.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.320 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:8353 3.1 3:8f54f2af:::benchmark_data_smithi173_66272_object8352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-17T16:14:35.320 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:14:35.312120+0000", 2024-09-17T16:14:35.321 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0058759809999999997, 2024-09-17T16:14:35.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0058968700000000002, 2024-09-17T16:14:35.321 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:35.321 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:35.321 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:14:35.321 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:35.321 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:14:35.321 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:14:35.321 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8353 2024-09-17T16:14:35.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.312120+0000", 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.312120+0000", 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:35.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.312121+0000", 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5400000000000001e-07 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.312194+0000", 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2909000000000003e-05 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:35.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.312201+0000", 2024-09-17T16:14:35.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4410000000000004e-06 2024-09-17T16:14:35.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:35.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.312207+0000", 2024-09-17T16:14:35.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9309999999999996e-06 2024-09-17T16:14:35.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:35.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.312243+0000", 2024-09-17T16:14:35.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5605000000000002e-05 2024-09-17T16:14:35.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:35.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.312321+0000", 2024-09-17T16:14:35.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7841000000000002e-05 2024-09-17T16:14:35.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:14:35.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.312400+0000", 2024-09-17T16:14:35.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9565000000000001e-05 2024-09-17T16:14:35.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:35.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.312885+0000", 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00048492099999999997 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.313300+0000", 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00041437499999999998 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 1 2024-09-17T16:14:35.327 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:14:35.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:35.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:35.430 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:35.461 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:35.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:35.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:35 smithi017 ceph-mon[27571]: pgmap v532: 17 pgs: 17 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s wr, 3 op/s 2024-09-17T16:14:35.563 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:35 smithi071 ceph-mon[30002]: pgmap v532: 17 pgs: 17 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s wr, 3 op/s 2024-09-17T16:14:35.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:35.587 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:35.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:35.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:35 smithi173 ceph-mon[30530]: pgmap v532: 17 pgs: 17 active+clean; 460 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s wr, 3 op/s 2024-09-17T16:14:35.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:35.703 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:14:35.718 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:35.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:35.821 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:35.849 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:35.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:35.951 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:35.983 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:35.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:36.042 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:36.085 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:36.110 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:36.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:36.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:14:36.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 9947 9931 29.5528 111.438 0.00260439 0.0338163 2024-09-17T16:14:36.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:36.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:36.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:36.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:36.352 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T16:14:36.361 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:36.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:36.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:36.445 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:14:36.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:36.474 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1058268475' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:14:36.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:36.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:36.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1058268475' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:14:36.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:36.622 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:36.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:36.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1058268475' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:14:36.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:36.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:36.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:36.831 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:14:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:14:36.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:36.859 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:14:36.859 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:14:36.877 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:36.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:36.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:36.885 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:14:36.885 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:14:36.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:14:36.909 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:14:36.909 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:14:36.931 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:14:36.931 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:14:36.952 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:14:36.952 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:14:36.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:37.002 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:37.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:37.003 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:37.003 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:37.103 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:37.131 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:37.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:37.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 11754 11738 33.3425 112.938 0.00429756 0.0299825 2024-09-17T16:14:37.233 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:37.258 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:37.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:37.292 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T16:14:37.326 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:37.360 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:37.386 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:37.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:37.488 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:37.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:37 smithi017 ceph-mon[27571]: pgmap v533: 17 pgs: 17 active+clean; 475 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s wr, 40 op/s 2024-09-17T16:14:37.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1058268475' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:14:37.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:37 smithi017 ceph-mon[27571]: osdmap e221: 12 total, 12 up, 12 in 2024-09-17T16:14:37.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1288806068' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:37.517 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:37.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:37.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:37.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:37 smithi173 ceph-mon[30530]: pgmap v533: 17 pgs: 17 active+clean; 475 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s wr, 40 op/s 2024-09-17T16:14:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1058268475' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:14:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:37 smithi173 ceph-mon[30530]: osdmap e221: 12 total, 12 up, 12 in 2024-09-17T16:14:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1288806068' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:37.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:37.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:37.650 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:37 smithi071 ceph-mon[30002]: pgmap v533: 17 pgs: 17 active+clean; 475 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s wr, 40 op/s 2024-09-17T16:14:37.650 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1058268475' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:14:37.650 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:37 smithi071 ceph-mon[30002]: osdmap e221: 12 total, 12 up, 12 in 2024-09-17T16:14:37.650 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1288806068' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:37.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:14:37.749 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:37.777 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:14:37.778 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:14:37.781 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:37.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:37.808 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:14:37.808 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:14:37.833 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:14:37.833 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:14:37.856 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:14:37.857 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:14:37.882 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:14:37.882 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:14:37.883 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:37.909 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:14:37.909 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:14:37.910 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:37.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:38.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:38.036 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:14:38.038 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:38.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:38.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:38.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:38.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:38.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:38.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:38.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 13385 13369 36.3244 101.938 0.00540267 0.0275131 2024-09-17T16:14:38.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:38.291 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:38.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:38.355 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:14:38.393 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:38.424 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:38.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:38.445 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1671855462' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:14:38.526 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:38.550 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:38.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:38.575 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:38.575 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1671855462' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:14:38.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:38.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:38.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:38.683 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1671855462' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:14:38.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:38.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:38.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:38.902 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:38.924 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:38.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:39.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:39.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:39.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:39.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:39.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:39.174 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:39.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:39.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 14706 14690 38.2506 82.5625 0.00434809 0.0261289 2024-09-17T16:14:39.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:14:39.276 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:39.298 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:39.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:39.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:39.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:39.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:39.431 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:39.465 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:39 smithi017 ceph-mon[27571]: pgmap v535: 17 pgs: 17 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 180 op/s 2024-09-17T16:14:39.465 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:39 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:39.465 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1671855462' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:14:39.465 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:39 smithi017 ceph-mon[27571]: osdmap e222: 12 total, 12 up, 12 in 2024-09-17T16:14:39.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:39.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:39.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:39 smithi071 ceph-mon[30002]: pgmap v535: 17 pgs: 17 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 180 op/s 2024-09-17T16:14:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:39 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1671855462' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:14:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:39 smithi071 ceph-mon[30002]: osdmap e222: 12 total, 12 up, 12 in 2024-09-17T16:14:39.607 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:14:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:39 smithi173 ceph-mon[30530]: pgmap v535: 17 pgs: 17 active+clean; 532 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s wr, 180 op/s 2024-09-17T16:14:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:39 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1671855462' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:14:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:39 smithi173 ceph-mon[30530]: osdmap e222: 12 total, 12 up, 12 in 2024-09-17T16:14:39.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:39.676 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:39.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:39.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:39.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:39.802 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:39.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:39.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:39.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:39.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:40.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:40.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:40.056 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:40.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:40.141 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:40.141 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:40.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:40.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:40.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:40.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 16022 16006 40.0103 82.25 0.00187429 0.0249784 2024-09-17T16:14:40.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:40.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:40.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:40.369 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T16:14:40.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:40.428 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:40.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:40.530 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:40.551 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:40.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:40.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1935749928' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:14:40.653 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:40.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:40.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:40.714 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:14:40.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1935749928' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:14:40.778 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1935749928' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:14:40.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:40.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:40.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:40.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:14:40.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:40.926 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:40.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:41.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:41.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:41.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:41.103 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T16:14:41.107 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:14:41.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:41.176 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:41.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:41.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 17229 17213 41.3726 75.4375 0.0221222 0.0241471 2024-09-17T16:14:41.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:41.301 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:41.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:41.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:41.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:41.432 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:41.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:41.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:14:41.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:41 smithi071 ceph-mon[30002]: pgmap v537: 17 pgs: 17 active+clean; 751 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 723 op/s 2024-09-17T16:14:41.486 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1935749928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:14:41.486 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:41 smithi071 ceph-mon[30002]: osdmap e223: 12 total, 12 up, 12 in 2024-09-17T16:14:41.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:41.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:41.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:41.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:41.630 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:41 smithi017 ceph-mon[27571]: pgmap v537: 17 pgs: 17 active+clean; 751 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 723 op/s 2024-09-17T16:14:41.630 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1935749928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:14:41.630 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:41 smithi017 ceph-mon[27571]: osdmap e223: 12 total, 12 up, 12 in 2024-09-17T16:14:41.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:41 smithi173 ceph-mon[30530]: pgmap v537: 17 pgs: 17 active+clean; 751 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 723 op/s 2024-09-17T16:14:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1935749928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:14:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:41 smithi173 ceph-mon[30530]: osdmap e223: 12 total, 12 up, 12 in 2024-09-17T16:14:41.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:41.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:41.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:41.726 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:41.726 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:41.769 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:14:41.769 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:14:41.784 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:41.797 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:41.801 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:14:41.802 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:14:41.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:41.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:41.832 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:14:41.832 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:14:41.851 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:41.856 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:14:41.856 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:14:41.878 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:14:41.879 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:14:41.900 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:14:41.900 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:14:41.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:41.921 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:14:41.921 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:14:41.939 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:41.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:41.943 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:14:41.944 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:14:41.965 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:14:41.965 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:14:41.987 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:14:41.987 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:14:42.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:14:42.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:14:42.031 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:14:42.031 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:14:42.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:42.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:42.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:42.169 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:42.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 18146 18130 41.9628 57.3125 0.0164605 0.0238091 2024-09-17T16:14:42.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:42.196 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:42.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:42.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:42.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:42.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:42.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:42.440 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:14:42.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:42.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:42.553 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:42.585 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:42.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:42.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2304981597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:42.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3714397658,config=1059521453,health=3340005642,kv=395782027,logm=2579506797}) 2024-09-17T16:14:42.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630130691}) 2024-09-17T16:14:42.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=434595088}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2376860205}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3466049514}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607080007}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082393219}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531238976}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439919739}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2959850593}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235822878}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1038502643}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1607339798}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2216288933}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2509927154}) 2024-09-17T16:14:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=935181607}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=683641131}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1665612004}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3293493472}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108039187}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3289490118}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2602227084}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1999683435}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167112881}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323154862}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415238977}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539971106}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=83,mdsmap=3,mgr=14} crc {logm=1723335505,mdsmap=376187277,mgr=944571422}) 2024-09-17T16:14:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=79} crc {mgr=3629746594,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=3725484557,mon_config_key=1869512560}) 2024-09-17T16:14:42.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=970271318}) 2024-09-17T16:14:42.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2653745036}) 2024-09-17T16:14:42.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=6,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=73} crc {mon_config_key=859395853,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=453731189,osdmap=802278396}) 2024-09-17T16:14:42.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3986557968}) 2024-09-17T16:14:42.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=66} crc {osdmap=3404529960}) 2024-09-17T16:14:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2304981597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3714397658,config=1059521453,health=3340005642,kv=395782027,logm=2579506797}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630130691}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=434595088}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2376860205}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3466049514}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607080007}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082393219}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531238976}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439919739}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2959850593}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235822878}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1038502643}) 2024-09-17T16:14:42.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1607339798}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2216288933}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2509927154}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=935181607}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=683641131}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1665612004}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3293493472}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108039187}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3289490118}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2602227084}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1999683435}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167112881}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323154862}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415238977}) 2024-09-17T16:14:42.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539971106}) 2024-09-17T16:14:42.648 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=83,mdsmap=3,mgr=14} crc {logm=1723335505,mdsmap=376187277,mgr=944571422}) 2024-09-17T16:14:42.648 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=79} crc {mgr=3629746594,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=3725484557,mon_config_key=1869512560}) 2024-09-17T16:14:42.648 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=970271318}) 2024-09-17T16:14:42.648 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2653745036}) 2024-09-17T16:14:42.648 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=6,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=73} crc {mon_config_key=859395853,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=453731189,osdmap=802278396}) 2024-09-17T16:14:42.648 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3986557968}) 2024-09-17T16:14:42.648 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=66} crc {osdmap=3404529960}) 2024-09-17T16:14:42.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:42.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:42.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:42.794 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:14:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2304981597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3714397658,config=1059521453,health=3340005642,kv=395782027,logm=2579506797}) 2024-09-17T16:14:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630130691}) 2024-09-17T16:14:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=434595088}) 2024-09-17T16:14:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2376860205}) 2024-09-17T16:14:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3466049514}) 2024-09-17T16:14:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607080007}) 2024-09-17T16:14:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082393219}) 2024-09-17T16:14:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2531238976}) 2024-09-17T16:14:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=439919739}) 2024-09-17T16:14:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2959850593}) 2024-09-17T16:14:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235822878}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1038502643}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1607339798}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2216288933}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2509927154}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=935181607}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=683641131}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1665612004}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3293493472}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108039187}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3289490118}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2602227084}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1999683435}) 2024-09-17T16:14:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4167112881}) 2024-09-17T16:14:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323154862}) 2024-09-17T16:14:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415238977}) 2024-09-17T16:14:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539971106}) 2024-09-17T16:14:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=83,mdsmap=3,mgr=14} crc {logm=1723335505,mdsmap=376187277,mgr=944571422}) 2024-09-17T16:14:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=79} crc {mgr=3629746594,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=3725484557,mon_config_key=1869512560}) 2024-09-17T16:14:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=970271318}) 2024-09-17T16:14:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2653745036}) 2024-09-17T16:14:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=6,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=73} crc {mon_config_key=859395853,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=453731189,osdmap=802278396}) 2024-09-17T16:14:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3986557968}) 2024-09-17T16:14:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=66} crc {osdmap=3404529960}) 2024-09-17T16:14:42.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:42.840 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:42.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:42.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:42.965 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:42.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:43.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:43.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:43.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:43.090 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:43.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 19601 19585 43.7114 90.9375 0.00206856 0.0227794 2024-09-17T16:14:43.191 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:43.213 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:43.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:43.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:43.349 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:43.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:43.451 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:43.481 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:43.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:43.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:43.500 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:43 smithi071 ceph-mon[30002]: pgmap v539: 17 pgs: 17 active+clean; 983 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-17T16:14:43.501 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:14:43.501 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:14:43.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:21741 2.3 2:c4bdb6e7:::benchmark_data_smithi173_55121_object21740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 196.833280997, 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55429691999999997, 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.502 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 21741 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662972+0000", 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.722e-06 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663024+0000", 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2175000000000001e-05 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663036+0000", 2024-09-17T16:14:43.504 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2131e-05 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663042+0000", 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9649999999999996e-06 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663084+0000", 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2354999999999997e-05 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663166+0000", 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1699999999999994e-05 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663249+0000", 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2714999999999997e-05 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663804+0000", 2024-09-17T16:14:43.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055499599999999998 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.664173+0000", 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000369181 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217193+0000", 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55302014600000005 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.507 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217234+0000", 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1149000000000002e-05 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217267+0000", 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2685000000000003e-05 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:23782 2.2 2:41257b37:::benchmark_data_smithi173_55121_object23781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-17T16:14:43.508 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 194.807962718, 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51828190799999996, 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 23782 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.509 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688289+0000", 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7400000000000002e-07 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688376+0000", 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6744999999999997e-05 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688387+0000", 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0499000000000001e-05 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688393+0000", 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9689999999999997e-06 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688434+0000", 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0694e-05 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688511+0000", 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6656000000000006e-05 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688594+0000", 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2682999999999994e-05 2024-09-17T16:14:43.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689092+0000", 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049850100000000004 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689369+0000", 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027721100000000003 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206489+0000", 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51712006899999996 2024-09-17T16:14:43.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206524+0000", 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4516e-05 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206570+0000", 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6490999999999999e-05 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:23782 2.2 e119/104 ondisk, result = 0)", 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 194.806869933, 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51719464199999998, 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689382+0000", 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689385+0000", 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3569999999999998e-06 2024-09-17T16:14:43.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206448+0000", 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51706313699999995 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206453+0000", 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4360000000000004e-06 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206480+0000", 2024-09-17T16:14:43.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7770999999999999e-05 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206486+0000", 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6779999999999997e-06 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206576+0000", 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9701000000000001e-05 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24369 2.3 2:c67d9130:::benchmark_data_smithi173_55121_object24368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.808743397, 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55710953100000005, 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24369 2024-09-17T16:14:43.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687509+0000", 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1399999999999995e-07 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687596+0000", 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7288999999999997e-05 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687607+0000", 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1505000000000001e-05 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687613+0000", 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1440000000000002e-06 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687655+0000", 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2240000000000002e-05 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687782+0000", 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000127043 2024-09-17T16:14:43.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687864+0000", 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2064000000000006e-05 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688444+0000", 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058032600000000004 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688721+0000", 2024-09-17T16:14:43.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000276734 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244551+0000", 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55582960199999998 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244588+0000", 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7004e-05 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244617+0000", 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9666e-05 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:24369 2.3 e120/104 ondisk, result = 0)", 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.80753620600001, 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55590799999999996, 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.525 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688716+0000", 2024-09-17T16:14:43.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8999999999999996e-07 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688719+0000", 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5980000000000001e-06 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244478+0000", 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55575852299999995 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244483+0000", 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244541+0000", 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8377e-05 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244547+0000", 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8980000000000001e-06 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244623+0000", 2024-09-17T16:14:43.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5780999999999998e-05 2024-09-17T16:14:43.529 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.529 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.529 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.529 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:29877 2.2 2:40729eb6:::benchmark_data_smithi173_55121_object29876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T16:14:43.530 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:43 smithi017 ceph-mon[27571]: pgmap v539: 17 pgs: 17 active+clean; 983 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-17T16:14:43.530 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:14:43.530 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 173.29238182, 2024-09-17T16:14:43.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.874997425, 2024-09-17T16:14:43.530 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 29877 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203870+0000", 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203910+0000", 2024-09-17T16:14:43.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9477999999999997e-05 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203918+0000", 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203922+0000", 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6369999999999998e-06 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203958+0000", 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5045000000000001e-05 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204026+0000", 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8693000000000001e-05 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204105+0000", 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8458e-05 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204758+0000", 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00065323300000000003 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.205073+0000", 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031491100000000002 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078782+0000", 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87370871400000005 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078827+0000", 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5219999999999997e-05 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078867+0000", 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9953000000000002e-05 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:79524 2.15 e150/113 2:a890cf23:::benchmark_data_smithi173_55121_object3035:head v 150'2897, mlcod=150'2893)", 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:35.019124+0000", 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 128.47712755699999, 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24077440899999999, 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.019124+0000", 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.019124+0000", 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.019124+0000", 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0200000000000002e-07 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.019129+0000", 2024-09-17T16:14:43.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5260000000000004e-06 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.258791+0000", 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.239661979 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.258795+0000", 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7929999999999997e-06 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.258866+0000", 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.046e-05 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.258888+0000", 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2257000000000001e-05 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.259861+0000", 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00097262499999999997 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:35.259898+0000", 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7166999999999998e-05 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:88652 2.2 2:45ffdfe0:::benchmark_data_smithi173_55121_object12163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 119.539496226, 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29260301500000002, 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.541 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 88652 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956757+0000", 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7859999999999999e-06 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956762+0000", 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.31e-06 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956773+0000", 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0832999999999999e-05 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956778+0000", 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2320000000000001e-06 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956829+0000", 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1124e-05 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956915+0000", 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5541e-05 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.957003+0000", 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8332999999999996e-05 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.958148+0000", 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001144937 2024-09-17T16:14:43.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.961994+0000", 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038457320000000001 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249267+0000", 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28727286400000002 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249320+0000", 2024-09-17T16:14:43.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3016000000000002e-05 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249358+0000", 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8306999999999998e-05 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(2.3 158/155 [PushReplyOp(2:c6916317:::benchmark_data_smithi173_55121_object76243:head)])", 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:14:43.547 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 114.77320397699999, 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45253040700000002, 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:14:43.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723048+0000", 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4300000000000003e-07 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723052+0000", 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9310000000000001e-06 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175226+0000", 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45217462899999999 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175233+0000", 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4099999999999996e-06 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175322+0000", 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9011000000000005e-05 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175578+0000", 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025588299999999999 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:94077 2.16 2:6e055a89:::benchmark_data_smithi173_55121_object17588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 113.41777835400001, 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28469862699999998, 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.551 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 94077 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078474+0000", 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.274e-06 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078479+0000", 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6310000000000002e-06 2024-09-17T16:14:43.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078490+0000", 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1799e-05 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078496+0000", 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7579999999999998e-06 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078540+0000", 2024-09-17T16:14:43.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3686999999999997e-05 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078624+0000", 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4153000000000006e-05 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078694+0000", 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9533e-05 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.079805+0000", 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011117550000000001 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.085499+0000", 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0056938969999999998 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363107+0000", 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27760825700000002 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363140+0000", 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2055999999999999e-05 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363171+0000", 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1826999999999999e-05 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97425 2.3 2:c4562d4f:::benchmark_data_smithi173_55121_object20936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:14:43.557 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 109.64918, 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29697053200000001, 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97425 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847072+0000", 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1399999999999997e-07 2024-09-17T16:14:43.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847074+0000", 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7329999999999999e-06 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847082+0000", 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6830000000000008e-06 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847088+0000", 2024-09-17T16:14:43.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0499999999999997e-06 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848028+0000", 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093989799999999999 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848133+0000", 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000104874 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848231+0000", 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8121999999999993e-05 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.849551+0000", 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00131985 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143633+0000", 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29408256900000002 2024-09-17T16:14:43.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143940+0000", 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030690399999999999 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143994+0000", 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3161999999999997e-05 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.144042+0000", 2024-09-17T16:14:43.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8072999999999999e-05 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:105495 2.2 2:47e62fd2:::benchmark_data_smithi173_55121_object29006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:02.950903+0000", 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 100.54534796900001, 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.236890983, 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.564 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.565 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.565 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.565 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.565 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.565 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 105495 2024-09-17T16:14:43.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.565 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950903+0000", 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950903+0000", 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950905+0000", 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1330000000000002e-06 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950912+0000", 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8560000000000001e-06 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950929+0000", 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7039000000000001e-05 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950936+0000", 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0999999999999998e-06 2024-09-17T16:14:43.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.950999+0000", 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2403999999999994e-05 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.951087+0000", 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7913999999999999e-05 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:14:43.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.951156+0000", 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9014e-05 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.952531+0000", 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013754959999999999 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:02.953080+0000", 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054879500000000003 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:03.187722+0000", 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23464195500000001 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:03.187764+0000", 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2058999999999998e-05 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:03.187794+0000", 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0218000000000001e-05 2024-09-17T16:14:43.570 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:129838 2.2 2:4555d77c:::benchmark_data_smithi173_55121_object53349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 75.709719488999994, 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.288916852, 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.571 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 129838 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786533+0000", 2024-09-17T16:14:43.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.728e-06 2024-09-17T16:14:43.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786540+0000", 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4320000000000004e-06 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786556+0000", 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6302000000000001e-05 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786563+0000", 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.934e-06 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786626+0000", 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2988000000000001e-05 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786633+0000", 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6150000000000002e-06 2024-09-17T16:14:43.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069502+0000", 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.282869606 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069572+0000", 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0201000000000001e-05 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069639+0000", 2024-09-17T16:14:43.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6323999999999997e-05 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075296+0000", 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0056576639999999997 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075309+0000", 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2621e-05 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075356+0000", 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6653000000000002e-05 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075405+0000", 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9438000000000003e-05 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075448+0000", 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3346000000000001e-05 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.578 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:129850 2.16 2:6f12710d:::benchmark_data_smithi173_55121_object53361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 75.693904680000003, 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27163233199999998, 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.579 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 129850 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802348+0000", 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0549999999999998e-06 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802356+0000", 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6539999999999999e-06 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802372+0000", 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6314999999999999e-05 2024-09-17T16:14:43.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802379+0000", 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0419999999999997e-06 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802428+0000", 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8367e-05 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802433+0000", 2024-09-17T16:14:43.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6519999999999997e-06 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069491+0000", 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26705805599999999 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069572+0000", 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1174999999999994e-05 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:14:43.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069639+0000", 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6946000000000006e-05 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073713+0000", 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040741709999999997 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073743+0000", 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9740999999999999e-05 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073843+0000", 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100497 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073930+0000", 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6835999999999993e-05 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073979+0000", 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8825000000000001e-05 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.585 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:150176 2.16 2:6f747f77:::benchmark_data_smithi173_55121_object73687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:50.480487+0000", 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 53.015764171000001, 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25565104700000002, 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 150176 2024-09-17T16:14:43.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480487+0000", 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480487+0000", 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480488+0000", 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.06e-06 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480493+0000", 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6700000000000002e-06 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480503+0000", 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0052e-05 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480510+0000", 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8820000000000003e-06 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480554+0000", 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4697000000000003e-05 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480627+0000", 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2498999999999998e-05 2024-09-17T16:14:43.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480688+0000", 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1081999999999996e-05 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.482798+0000", 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021097479999999998 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.483396+0000", 2024-09-17T16:14:43.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059859300000000002 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.736057+0000", 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25266103699999998 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.736109+0000", 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.168e-05 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.736138+0000", 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9047000000000002e-05 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:150269 2.16 2:6d6b4de7:::benchmark_data_smithi173_55121_object73780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 52.907717400999999, 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29341146400000001, 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.592 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 150269 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588535+0000", 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.751e-06 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588542+0000", 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7479999999999996e-06 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588558+0000", 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5885000000000001e-05 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588582+0000", 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4042000000000001e-05 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588646+0000", 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4103999999999995e-05 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588789+0000", 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000143094 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588910+0000", 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120751 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.590898+0000", 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001988058 2024-09-17T16:14:43.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.590948+0000", 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9959000000000001e-05 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881725+0000", 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29077654800000002 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881900+0000", 2024-09-17T16:14:43.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017578599999999999 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881945+0000", 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4737999999999998e-05 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:150365 2.16 2:68fc6af2:::benchmark_data_smithi173_55121_object73876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:50.878053+0000", 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 52.618198161000002, 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23419889099999999, 2024-09-17T16:14:43.598 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 150365 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.878053+0000", 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.878053+0000", 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.878055+0000", 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.463e-06 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.878066+0000", 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0963e-05 2024-09-17T16:14:43.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.878096+0000", 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9989999999999999e-05 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.878111+0000", 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4676e-05 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.878243+0000", 2024-09-17T16:14:43.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000131846 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.878367+0000", 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000123898 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.878502+0000", 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000134771 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.880366+0000", 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001864196 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.880533+0000", 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016694400000000001 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.112176+0000", 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23164363900000001 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.112210+0000", 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3460000000000002e-05 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.112252+0000", 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2045000000000001e-05 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:152907 2.3 2:c65c8e4e:::benchmark_data_smithi173_55121_object76418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:14:43.604 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 49.698831335999998, 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87037030500000001, 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152907 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4099999999999999e-07 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797423+0000", 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797430+0000", 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1119999999999998e-06 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797436+0000", 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6189999999999999e-06 2024-09-17T16:14:43.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797479+0000", 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3050999999999998e-05 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797597+0000", 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000117637 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797696+0000", 2024-09-17T16:14:43.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9079999999999993e-05 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.799730+0000", 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020345239999999998 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.801020+0000", 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012897270000000001 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:43.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667698+0000", 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.866677794 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667733+0000", 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4953999999999998e-05 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667790+0000", 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7265999999999997e-05 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.610 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25036.0:17923 3.c e223/208 ondisk, result = 0)", 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:14:41.811355+0000", 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.6848962379999999, 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25393519799999997, 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.811355+0000", 2024-09-17T16:14:43.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.811355+0000", 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.811355+0000", 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.811358+0000", 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.027e-06 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.065167+0000", 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25380819199999999 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.065172+0000", 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.677e-06 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.065217+0000", 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4876999999999997e-05 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.065223+0000", 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4410000000000001e-06 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.065290+0000", 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7422000000000006e-05 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.614 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25036.0:17943 3.5 e223/208 ondisk, result = 0)", 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:14:41.833608+0000", 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.662643471, 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22793179599999999, 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:14:43.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.833608+0000", 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.833608+0000", 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.833609+0000", 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.55e-06 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.833618+0000", 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5529999999999993e-06 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.061345+0000", 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22772756699999999 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.061352+0000", 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2489999999999998e-06 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.061401+0000", 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9889000000000003e-05 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.061414+0000", 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2323999999999999e-05 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.061539+0000", 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012566399999999999 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.618 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.619 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.619 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:14:43.619 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:14:43.619 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:14:43.619 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:43.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:43 smithi173 ceph-mon[30530]: pgmap v539: 17 pgs: 17 active+clean; 983 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-17T16:14:43.645 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:43.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:43.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:43.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:43.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:43.875 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:43.903 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:43.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:44.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:44.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:44.029 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:44.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:44.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:44.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:44.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 21171 21155 45.5874 98.125 0.00159098 0.0218972 2024-09-17T16:14:44.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:44.277 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:44.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:44.317 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T16:14:44.318 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:14:44.378 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:44.405 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:44.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:44.507 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:44.530 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:44.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:44.598 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/878987872' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:14:44.598 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:44 smithi071 ceph-mon[30002]: pgmap v540: 17 pgs: 17 active+clean; 1.0 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:14:44.598 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3419154466' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:44.598 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:44 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:44.598 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/878987872' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:14:44.598 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3419154466' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:44.598 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:44 smithi071 ceph-mon[30002]: osdmap e224: 12 total, 12 up, 12 in 2024-09-17T16:14:44.632 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:44.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:14:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/878987872' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:14:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:44 smithi173 ceph-mon[30530]: pgmap v540: 17 pgs: 17 active+clean; 1.0 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:14:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3419154466' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:44 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/878987872' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:14:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3419154466' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:44 smithi173 ceph-mon[30530]: osdmap e224: 12 total, 12 up, 12 in 2024-09-17T16:14:44.656 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:44.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:44.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/878987872' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:14:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:44 smithi017 ceph-mon[27571]: pgmap v540: 17 pgs: 17 active+clean; 1.0 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:14:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3419154466' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:44 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/878987872' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:14:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3419154466' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:44 smithi017 ceph-mon[27571]: osdmap e224: 12 total, 12 up, 12 in 2024-09-17T16:14:44.758 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:44.788 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:44.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:44.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:14:44.889 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:44.915 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:44.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:44.943 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:14:45.016 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:45.047 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:45.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:45.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:45.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:45.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:45.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 22527 22511 46.8925 84.75 0.0016982 0.0213106 2024-09-17T16:14:45.228 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T16:14:45.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:45.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:45.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:45.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:45.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:45.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:45.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:45.527 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:45.557 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:45.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:45.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:45.672 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:14:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:14:45.672 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:14:45.673 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:14:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:45.682 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:45.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:45.732 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:45.783 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:45.805 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:45.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:45.907 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:45.932 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:45.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:46.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:46.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:46.059 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:46.160 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:46.185 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 23591 23575 47.5249 66.5 0.00844876 0.0210336 2024-09-17T16:14:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:46.190 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:46.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:46.291 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:46.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:46.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:46.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:46.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:14:46.423 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:46.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:46.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:46.551 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:46.581 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:46.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:46.683 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:46.686 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:14:46.705 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:46.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:46.714 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:14:46.714 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:14:46.714 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:14:46.714 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:14:46.725 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:14:46.725 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:14:46.752 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:14:46.752 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:14:46.779 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:14:46.779 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:14:46.781 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:46.802 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:14:46.802 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:14:46.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:46.823 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:14:46.824 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:14:46.830 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:46.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:46.860 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:14:46.860 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:14:46.890 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:14:46.890 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:14:46.915 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:14:46.916 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:14:46.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:46.939 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:14:46.939 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:14:46.955 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:46.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:46.964 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:14:46.965 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:14:47.057 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:46 smithi017 ceph-mon[27571]: pgmap v542: 17 pgs: 17 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-17T16:14:47.057 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/490851838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:47.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:47.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:46 smithi071 ceph-mon[30002]: pgmap v542: 17 pgs: 17 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-17T16:14:47.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/490851838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:47.083 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:47.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:47.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:46 smithi173 ceph-mon[30530]: pgmap v542: 17 pgs: 17 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-17T16:14:47.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/490851838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:47.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:47.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 15 24951 24936 48.6977 85.0625 0.0115834 0.0205205 2024-09-17T16:14:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:47.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:47.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:47.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:47.311 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:47.311 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:47.331 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:47.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:47.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:47.458 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:47.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:47.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:47.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:47.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:47.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:47.710 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:47.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:47.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:14:47.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:47.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:47.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:47.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:47.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:47.969 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:47.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:48.041 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:14:48.071 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:48.099 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:48.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:48.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 26316 26300 49.8051 85.25 0.00816118 0.0200718 2024-09-17T16:14:48.201 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:48.230 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:48.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:48.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:48.357 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:48.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:48.388 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:14:48.459 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:48.491 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:48.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:48.531 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:48.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:48.619 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:48.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:48.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:48.749 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:48.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:48.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:48.837 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T16:14:48.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:48.877 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:48.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:48.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:48 smithi017 ceph-mon[27571]: pgmap v543: 17 pgs: 17 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-17T16:14:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/578898230' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:14:48.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:49.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:49.085 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:48 smithi071 ceph-mon[30002]: pgmap v543: 17 pgs: 17 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-17T16:14:49.086 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/578898230' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:14:49.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:49.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:49.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:48 smithi173 ceph-mon[30530]: pgmap v543: 17 pgs: 17 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-17T16:14:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/578898230' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:14:49.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 27661 27645 50.8125 84.0625 0.0363906 0.0196532 2024-09-17T16:14:49.225 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:49.249 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:49.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:49.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:49.320 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:14:49.327 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:49.327 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:49.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:49.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:49.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:49.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:49.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:49.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:49.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:49.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:49.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:49.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:49.755 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:49.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:49.765 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-17T16:14:49.857 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:49.880 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:49.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:49.982 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:49.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/578898230' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:14:49.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:49 smithi017 ceph-mon[27571]: osdmap e225: 12 total, 12 up, 12 in 2024-09-17T16:14:49.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4177372163' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:14:49.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:49 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:49.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4177372163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:14:49.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:49 smithi017 ceph-mon[27571]: osdmap e226: 12 total, 12 up, 12 in 2024-09-17T16:14:50.006 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:50.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:50.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:50.108 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:50.131 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:50.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:50.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/578898230' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:14:50.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:49 smithi173 ceph-mon[30530]: osdmap e225: 12 total, 12 up, 12 in 2024-09-17T16:14:50.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4177372163' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:14:50.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:49 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:50.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4177372163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:14:50.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:49 smithi173 ceph-mon[30530]: osdmap e226: 12 total, 12 up, 12 in 2024-09-17T16:14:50.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 28402 28386 50.6838 46.3125 0.00185118 0.0197044 2024-09-17T16:14:50.234 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:50.256 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:50.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:50.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/578898230' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:14:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:49 smithi071 ceph-mon[30002]: osdmap e225: 12 total, 12 up, 12 in 2024-09-17T16:14:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4177372163' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:14:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:49 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:14:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4177372163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:14:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:49 smithi071 ceph-mon[30002]: osdmap e226: 12 total, 12 up, 12 in 2024-09-17T16:14:50.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:50.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:50.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:50.465 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:14:50.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:50.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:50.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:50.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:50.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:50.627 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:50.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:50.719 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:50.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:50.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:50.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:50.755 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:50.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:50.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:50.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:50.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:51.009 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:51.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:51.110 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:51.135 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:51.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:51.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 29472 29456 51.1334 66.875 0.0172967 0.0195466 2024-09-17T16:14:51.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:51.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:51.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:51.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:51 smithi017 ceph-mon[27571]: pgmap v546: 17 pgs: 17 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-17T16:14:51.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:51.396 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:51.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:51.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:51.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:51.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:51.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:51 smithi173 ceph-mon[30530]: pgmap v546: 17 pgs: 17 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-17T16:14:51.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:51.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:51.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:51.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:14:51.768 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:14:51.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:51.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:51.799 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:14:51.799 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:14:51.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:51 smithi071 ceph-mon[30002]: pgmap v546: 17 pgs: 17 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-17T16:14:51.827 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:14:51.827 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:14:51.854 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:14:51.854 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:14:51.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:51.886 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:14:51.886 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:14:51.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:51.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:51.917 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:14:51.917 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:14:51.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:14:51.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:14:51.945 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:14:51.970 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:14:51.970 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:14:51.998 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:14:51.998 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:14:52.011 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:52.028 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:14:52.028 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:14:52.038 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:52.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:52.055 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:14:52.055 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:14:52.082 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:14:52.083 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:14:52.140 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:52.165 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:52.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:52.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 31169 31153 52.6177 106.062 0.00202381 0.0189987 2024-09-17T16:14:52.256 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:14:52.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:52.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:52.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:52.331 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.8 to 1 (10000) 2024-09-17T16:14:52.398 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:52.427 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:52.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:52.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:52.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:52.556 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:52.571 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:14:52.593 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2534579429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:52.593 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1878727451' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:14:52.593 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:52 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:14:52.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2534579429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1878727451' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:14:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:52 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:14:52.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:52.685 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:52.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:52.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:52.787 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:52.813 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2534579429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1878727451' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:14:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:52 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:14:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:52.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:52.833 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:14:52.833 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:14:52.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.833 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:32148 3.9 3:90fd8dca:::benchmark_data_smithi173_66272_object32147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-09-17T16:14:52.833 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:14:52.825799+0000", 2024-09-17T16:14:52.833 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0046766120000000001, 2024-09-17T16:14:52.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046993529999999999, 2024-09-17T16:14:52.833 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 32148 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.825799+0000", 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:14:52.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.825799+0000", 2024-09-17T16:14:52.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:14:52.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:14:52.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.825801+0000", 2024-09-17T16:14:52.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9709999999999998e-06 2024-09-17T16:14:52.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.825838+0000", 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6451e-05 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.825853+0000", 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5637e-05 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:14:52.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.825861+0000", 2024-09-17T16:14:52.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8099999999999998e-06 2024-09-17T16:14:52.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:14:52.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.825916+0000", 2024-09-17T16:14:52.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5312000000000002e-05 2024-09-17T16:14:52.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.826003+0000", 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6178000000000001e-05 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.826099+0000", 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6268999999999996e-05 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.827277+0000", 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001178232 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:14:52.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:14:52.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:52.827595+0000", 2024-09-17T16:14:52.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000317602 2024-09-17T16:14:52.839 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:14:52.839 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:14:52.839 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:14:52.839 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:14:52.839 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:14:52.839 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-09-17T16:14:52.839 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:14:52.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:52.941 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:52.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:53.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:53.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:53.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:53.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:53.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:53.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 32590 32574 53.57 88.8125 0.0031204 0.0186454 2024-09-17T16:14:53.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:53.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:53.297 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:53.322 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:53.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:53.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:53.451 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:53.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:53 smithi017 ceph-mon[27571]: pgmap v547: 17 pgs: 17 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-17T16:14:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:53 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:14:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:53 smithi017 ceph-mon[27571]: osdmap e227: 12 total, 12 up, 12 in 2024-09-17T16:14:53.553 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:53.577 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:53.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:53.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:53 smithi173 ceph-mon[30530]: pgmap v547: 17 pgs: 17 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-17T16:14:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:53 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:14:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:53 smithi173 ceph-mon[30530]: osdmap e227: 12 total, 12 up, 12 in 2024-09-17T16:14:53.679 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:53.708 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:53.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:53.810 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:53.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:53 smithi071 ceph-mon[30002]: pgmap v547: 17 pgs: 17 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-17T16:14:53.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:53 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:14:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:53 smithi071 ceph-mon[30002]: osdmap e227: 12 total, 12 up, 12 in 2024-09-17T16:14:53.838 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:53.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:53.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:53.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:53.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:53.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:54.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:54.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:54.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:54.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:54.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 33806 33790 54.1449 76 0.0107659 0.0184559 2024-09-17T16:14:54.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:54.216 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:54.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:54.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:54.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:54.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:54.444 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:54.471 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:54.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:54.574 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:54.601 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:54.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:54.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:54.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:54.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:54.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:54.852 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:54.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:54.954 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:54.982 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:54.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:54.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:55.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:14:55.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-17T16:14:55.078 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:55.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:55.105 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:55.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:55.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:55.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:14:55.185493+0000 min lat: 0.00101616 max lat: 15.0111 avg lat: 0.0183837 2024-09-17T16:14:55.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:14:55.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 34820 34804 54.3755 63.375 0.00304175 0.0183837 2024-09-17T16:14:55.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:55.219 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:14:55.233 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:55.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:55.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:55.357 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:14:55.367 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:55.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:55.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:55.494 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:55.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:55.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:55.620 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:55.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:55.623 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:55 smithi173 ceph-mon[30530]: pgmap v549: 17 pgs: 17 active+clean; 1.8 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-17T16:14:55.623 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:55 smithi173 ceph-mon[30530]: osdmap e228: 12 total, 12 up, 12 in 2024-09-17T16:14:55.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:55 smithi017 ceph-mon[27571]: pgmap v549: 17 pgs: 17 active+clean; 1.8 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-17T16:14:55.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:55 smithi017 ceph-mon[27571]: osdmap e228: 12 total, 12 up, 12 in 2024-09-17T16:14:55.662 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:55.722 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:55.744 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:55.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:55.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:55 smithi071 ceph-mon[30002]: pgmap v549: 17 pgs: 17 active+clean; 1.8 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-17T16:14:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:55 smithi071 ceph-mon[30002]: osdmap e228: 12 total, 12 up, 12 in 2024-09-17T16:14:55.845 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:55.875 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:55.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:55.926 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:14:55.927 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:14:55.927 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:14:55.927 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:14:55.927 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:14:55.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:56.007 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:56.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:56.108 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:56.137 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:56.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:56.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:56.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 36074 36058 54.9607 78.375 0.00284249 0.0181862 2024-09-17T16:14:56.188 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:56.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:56.266 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:56.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:56.329 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:14:56.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:56.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:56.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:56.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:56.521 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:56.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:56.623 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1482242618' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:14:56.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:14:56.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:56.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1482242618' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:14:56.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:14:56.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:14:56.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:56.770 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:56.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:56.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1482242618' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:14:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:14:56.872 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:56.898 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:56.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:57.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:57.026 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:57.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:57.028 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:14:57.074 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:14:57.074 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:14:57.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:14:57.100 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:14:57.125 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:14:57.125 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:14:57.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:57.150 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:14:57.151 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:14:57.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:57.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:57.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:57.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:14:57.177 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:14:57.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 37186 37170 55.3067 69.5 0.0192582 0.0180733 2024-09-17T16:14:57.201 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:14:57.202 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:14:57.229 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:14:57.229 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:14:57.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:57.258 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:14:57.258 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:14:57.284 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:14:57.285 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:14:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:57.286 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:57.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:57.312 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:14:57.312 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:14:57.330 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.9 to 1 (10000) 2024-09-17T16:14:57.341 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:14:57.342 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:14:57.377 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:14:57.377 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:14:57.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:57.417 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:57.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:57.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:57 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:14:57.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:57 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:14:57.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1482242618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:57.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:57 smithi173 ceph-mon[30530]: osdmap e229: 12 total, 12 up, 12 in 2024-09-17T16:14:57.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2021265218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:14:57.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1726930273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:57.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:57 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:14:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:57 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:14:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1482242618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:57 smithi017 ceph-mon[27571]: osdmap e229: 12 total, 12 up, 12 in 2024-09-17T16:14:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2021265218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:14:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1726930273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:57.518 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:57.547 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:57.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:57.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:57.578 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:57.578 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:57.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:57.678 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:57.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:57.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:14:57.808 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:57.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:57.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:14:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:57 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:14:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:57 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:14:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1482242618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:14:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:57 smithi071 ceph-mon[30002]: osdmap e229: 12 total, 12 up, 12 in 2024-09-17T16:14:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2021265218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:14:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1726930273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:14:57.910 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:57.934 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:57.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:57.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:58.036 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:58.059 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:58.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:14:58.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:58.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:58.187 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 38889 38873 56.4956 106.438 0.00282072 0.0176924 2024-09-17T16:14:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:58.187 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:58.288 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:14:58.313 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:58.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:58.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:14:58.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:14:58.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:14:58.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:58.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:14:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2021265218' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:14:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:58 smithi017 ceph-mon[27571]: osdmap e230: 12 total, 12 up, 12 in 2024-09-17T16:14:58.540 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:14:58.567 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:58.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:58.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:14:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2021265218' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:14:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:58 smithi173 ceph-mon[30530]: osdmap e230: 12 total, 12 up, 12 in 2024-09-17T16:14:58.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:58.698 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:58.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:58.745 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:14:58.800 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:58.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2021265218' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:14:58.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:58 smithi071 ceph-mon[30002]: osdmap e230: 12 total, 12 up, 12 in 2024-09-17T16:14:58.830 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:58.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:58.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-17T16:14:58.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:14:58.958 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:58.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:14:59.060 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:14:59.081 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:59.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:14:59.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:14:59.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 40175 40159 57.0381 80.375 0.118168 0.0175143 2024-09-17T16:14:59.210 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:59.211 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:14:59.301 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:14:59.301 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:14:59.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:59.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:59.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:59.420 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:14:59.438 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:14:59.466 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:59.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:14:59.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:14:59.594 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:59.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:14:59.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:59 smithi173 ceph-mon[30530]: pgmap v554: 17 pgs: 17 active+clean; 2.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-17T16:14:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:59 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:14:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:14:59 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:14:59.695 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:14:59.721 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:14:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:59.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:14:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:59 smithi017 ceph-mon[27571]: pgmap v554: 17 pgs: 17 active+clean; 2.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-17T16:14:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:59 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:14:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:14:59 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:14:59.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:59 smithi071 ceph-mon[30002]: pgmap v554: 17 pgs: 17 active+clean; 2.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-17T16:14:59.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:59 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:14:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:14:59 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:14:59.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:14:59.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:14:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:59.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:14:59.907 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:14:59.950 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:14:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:14:59.979 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:14:59.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:00.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:00.103 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:00.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:00.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:00.187 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 41242 41226 57.2524 66.6875 0.0229827 0.0174577 2024-09-17T16:15:00.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:00.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:00.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:00.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:15:00.325 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:00.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:00.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:00.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:00.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:00.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:00 smithi017 ceph-mon[27571]: osdmap e231: 12 total, 12 up, 12 in 2024-09-17T16:15:00.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:00.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:00.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:00.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:00.613 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:15:00.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:00.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:00.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:00 smithi173 ceph-mon[30530]: osdmap e231: 12 total, 12 up, 12 in 2024-09-17T16:15:00.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:00.749 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:00.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:00.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:00 smithi071 ceph-mon[30002]: osdmap e231: 12 total, 12 up, 12 in 2024-09-17T16:15:00.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:00.875 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:00.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:00.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:01.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:01.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:01.102 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:01.124 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:15:01.124 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:15:01.125 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:01.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:01.186 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 42193 42177 57.2998 59.4375 0.00209663 0.0173679 2024-09-17T16:15:01.227 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:01.248 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:01.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:01.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:01.340 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:15:01.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:01.362 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:15:01.362 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.10 to 1 (10000) 2024-09-17T16:15:01.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:01.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:01.373 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:01.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:01.496 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:01.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:01.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:01.598 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:01.620 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:01.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:01.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:01 smithi173 ceph-mon[30530]: pgmap v556: 17 pgs: 17 active+clean; 2.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-17T16:15:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1515769642' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:15:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3047460959' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:15:01.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:01.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:15:01.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:01 smithi017 ceph-mon[27571]: pgmap v556: 17 pgs: 17 active+clean; 2.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-17T16:15:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1515769642' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:15:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3047460959' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:15:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:01.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:01.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:01.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:01 smithi071 ceph-mon[30002]: pgmap v556: 17 pgs: 17 active+clean; 2.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-17T16:15:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1515769642' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:15:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3047460959' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:15:01.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:01.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:01.869 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:01.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:01.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:01.982 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:15:01.993 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:01.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:02.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:02.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:02.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:02.187 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 42235 42219 56.1365 2.625 0.00256072 0.0173965 2024-09-17T16:15:02.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:02.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:02.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:02.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:02.373 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:02.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:02.473 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:15:02.473 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:15:02.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:02.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:02.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:02.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:02.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:02.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:02.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:02.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:02 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:02.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1515769642' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:15:02.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3047460959' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:15:02.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:02 smithi173 ceph-mon[30530]: osdmap e232: 12 total, 12 up, 12 in 2024-09-17T16:15:02.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:02 smithi173 ceph-mon[30530]: pgmap v558: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-09-17T16:15:02.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:02 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:02.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1515769642' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:15:02.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3047460959' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:15:02.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:02 smithi017 ceph-mon[27571]: osdmap e232: 12 total, 12 up, 12 in 2024-09-17T16:15:02.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:02 smithi017 ceph-mon[27571]: pgmap v558: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-09-17T16:15:02.724 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:02.750 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:02.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:02.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:02 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1515769642' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:15:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3047460959' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:15:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:02 smithi071 ceph-mon[30002]: osdmap e232: 12 total, 12 up, 12 in 2024-09-17T16:15:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:02 smithi071 ceph-mon[30002]: pgmap v558: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-09-17T16:15:02.852 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:02.877 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:02.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:02.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:02.992 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-17T16:15:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:03.005 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:03.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:03.067 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:03.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:03.128 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:03.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:03.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:15:03.187 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 42236 42220 54.9684 0.0625 0.493103 0.0174077 2024-09-17T16:15:03.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:03.252 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:03.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:03.354 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:03.380 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:03.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:03.426 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:15:03.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:03.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:03.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:03.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:03.625 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:03.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:03.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:03.635 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:15:03.636 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:15:03.658 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:15:03.658 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:15:03.679 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:15:03.679 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:15:03.699 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:15:03.699 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:15:03.721 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:15:03.721 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:15:03.728 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:03.742 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:15:03.743 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:15:03.753 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:03.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:03.764 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:15:03.764 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:15:03.787 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:15:03.787 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:15:03.808 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:15:03.809 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:15:03.831 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:15:03.831 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:15:03.852 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:15:03.853 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:15:03.853 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/628841338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:03.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:03.875 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:15:03.875 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:15:03.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:03.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:03.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:03.880 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/628841338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:03.885 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:15:03.885 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:15:03.885 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:15:03.885 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:15:03.968 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/628841338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:03.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:04.001 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:04.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:04.102 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:04.123 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:04.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:04.187 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 42236 42220 53.8465 0 - 0.0174077 2024-09-17T16:15:04.225 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:04.247 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:04.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:04.350 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:04.372 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:04.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:04.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:04.495 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:04.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:04.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:15:04.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:15:04.597 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:04.619 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:04.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:04.665 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.11 to 1 (10000) 2024-09-17T16:15:04.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:04.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:04.744 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:04.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:04.785 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:15:04.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:04 smithi017 ceph-mon[27571]: pgmap v559: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-17T16:15:04.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3785169833' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:15:04.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:04.867 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:04.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:04 smithi173 ceph-mon[30530]: pgmap v559: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-17T16:15:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3785169833' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:15:04.925 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:04 smithi071 ceph-mon[30002]: pgmap v559: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-17T16:15:04.925 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3785169833' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:15:04.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:04.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:04.990 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:04.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:05.092 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:05.112 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:05.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:05.133 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:05.188 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 42236 42220 52.7696 0 - 0.0174077 2024-09-17T16:15:05.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:05.235 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:05.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:05.243 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:15:05.244 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:15:05.244 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:15:05.244 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:15:05.244 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:15:05.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:05.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:05.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:05.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:05.481 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:05.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:05.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:05.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:05.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:05.674 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T16:15:05.675 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:15:05.708 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:05.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:05.731 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:05.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:05.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:05.854 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:05.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:05.885 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3785169833' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:15:05.885 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:05 smithi173 ceph-mon[30530]: osdmap e233: 12 total, 12 up, 12 in 2024-09-17T16:15:05.885 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:05 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:05.885 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:05 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:05.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:15:05.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3785169833' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:15:05.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:05 smithi017 ceph-mon[27571]: osdmap e233: 12 total, 12 up, 12 in 2024-09-17T16:15:05.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:05 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:05.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:05 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:05.956 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:05.978 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:05.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:06.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3785169833' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:15:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:05 smithi071 ceph-mon[30002]: osdmap e233: 12 total, 12 up, 12 in 2024-09-17T16:15:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:05 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:06.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:05 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:06.080 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:06.104 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:06.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:06.153 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:15:06.188 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 42236 42220 51.7345 0 - 0.0174077 2024-09-17T16:15:06.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:06.226 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:06.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:06.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:06.241 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:06.328 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:06.351 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:06.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:06.453 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:06.475 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:06.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:06.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:06.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:06.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:06.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:06.646 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:15:06.647 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:15:06.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:15:06.647 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:15:06.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.647 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:11693 2.7 e102/95 2:eb9c71a1:::benchmark_data_smithi173_55121_object11692:head v 102'755, mlcod=102'752)", 2024-09-17T16:15:06.647 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:10.832712+0000", 2024-09-17T16:15:06.647 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 235.810806267, 2024-09-17T16:15:06.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.171520639, 2024-09-17T16:15:06.647 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.647 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.832712+0000", 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.832712+0000", 2024-09-17T16:15:06.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.832713+0000", 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8999999999999996e-07 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:10.832794+0000", 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1312999999999999e-05 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.003701+0000", 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17090686399999999 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.003706+0000", 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2209999999999996e-06 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.003748+0000", 2024-09-17T16:15:06.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1372000000000003e-05 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.003762+0000", 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.453e-05 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.004207+0000", 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044488900000000002 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:11.004233+0000", 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.586e-05 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:21738 2.7 e117/104 2:e3b5b582:::benchmark_data_smithi173_55121_object21737:head v 117'1138, mlcod=117'1136)", 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.656805+0000", 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 219.98671319900001, 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.168970552, 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.652 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.653 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.653 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.656805+0000", 2024-09-17T16:15:06.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.656805+0000", 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.656807+0000", 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1340000000000001e-06 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.656851+0000", 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4484999999999998e-05 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.824917+0000", 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.168065465 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.824939+0000", 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2339000000000001e-05 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.825069+0000", 2024-09-17T16:15:06.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000130307 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.825111+0000", 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1946000000000001e-05 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.825745+0000", 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00063373699999999999 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.825776+0000", 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1139000000000001e-05 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:21834 2.12 2:4e087699:::benchmark_data_smithi173_55121_object21833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 219.65602664299999, 2024-09-17T16:15:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23867154600000001, 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21834 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:15:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:15:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987494+0000", 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2919999999999998e-06 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987518+0000", 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3598e-05 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987534+0000", 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5896000000000002e-05 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987542+0000", 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1829999999999994e-06 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987592+0000", 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0033e-05 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987671+0000", 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8993999999999996e-05 2024-09-17T16:15:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987759+0000", 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7751999999999996e-05 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988272+0000", 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051340700000000001 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988663+0000", 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039071199999999998 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226103+0000", 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23744021000000001 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226137+0000", 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3929e-05 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226164+0000", 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.654e-05 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:21834 2.12 e117/114 ondisk, result = 0)", 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:15:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 219.65491708299999, 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23756740700000001, 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:15:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988605+0000", 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0079999999999998e-06 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226056+0000", 2024-09-17T16:15:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23745111999999999 2024-09-17T16:15:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226062+0000", 2024-09-17T16:15:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9739999999999999e-06 2024-09-17T16:15:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226095+0000", 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2503999999999999e-05 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226100+0000", 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9049999999999996e-06 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226169+0000", 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9303999999999999e-05 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:25268 2.7 e121/104 2:e432f6fb:::benchmark_data_smithi173_55121_object25267:head v 121'1249, mlcod=121'1247)", 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:31.069057+0000", 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 215.57446134099999, 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.163705394, 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.069057+0000", 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.069057+0000", 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.069058+0000", 2024-09-17T16:15:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5899999999999996e-07 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.069175+0000", 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011732599999999999 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232110+0000", 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16293511299999999 2024-09-17T16:15:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232117+0000", 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.46e-06 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232162+0000", 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5345e-05 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232181+0000", 2024-09-17T16:15:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.855e-05 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232732+0000", 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00055167900000000001 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.232763+0000", 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0261999999999999e-05 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:26989 2.12 2:48690473:::benchmark_data_smithi173_55121_object26988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 213.72230294600001, 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22401436399999999, 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 26989 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921217+0000", 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5e-06 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921237+0000", 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9777000000000001e-05 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921250+0000", 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2704999999999999e-05 2024-09-17T16:15:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921256+0000", 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4080000000000003e-06 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921298+0000", 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2169000000000002e-05 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921385+0000", 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6536e-05 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921467+0000", 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2657999999999994e-05 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922015+0000", 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054718699999999998 2024-09-17T16:15:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922501+0000", 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048639400000000002 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145060+0000", 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.222559429 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145161+0000", 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100058 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145230+0000", 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9542999999999995e-05 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:26989 2.12 e121/114 ondisk, result = 0)", 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:15:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 213.72104131200001, 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22276984499999999, 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:15:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922478+0000", 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9100000000000004e-07 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922484+0000", 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.144e-06 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.144878+0000", 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.222394432 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.144910+0000", 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1365999999999998e-05 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145033+0000", 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122717 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145050+0000", 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.753e-05 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145247+0000", 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019706500000000001 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 136/134 [PushReplyOp(2:482f57f9:::benchmark_data_smithi173_55121_object24323:head)])", 2024-09-17T16:15:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 184.479554979, 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.229086917, 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0200000000000002e-07 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163968+0000", 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9369999999999997e-06 2024-09-17T16:15:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392756+0000", 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22878752399999999 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392762+0000", 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0959999999999997e-06 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392828+0000", 2024-09-17T16:15:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5746000000000004e-05 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.393051+0000", 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000223012 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:47735 2.7 e142/104 2:e2cd41e4:::benchmark_data_smithi173_55121_object47734:head v 142'1954, mlcod=142'1952)", 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:10.497154+0000", 2024-09-17T16:15:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 176.14636454999999, 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16541420900000001, 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.497154+0000", 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.497154+0000", 2024-09-17T16:15:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.497156+0000", 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6670000000000001e-06 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.497194+0000", 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8078999999999999e-05 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.661482+0000", 2024-09-17T16:15:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16428769600000001 2024-09-17T16:15:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.661495+0000", 2024-09-17T16:15:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2989999999999999e-05 2024-09-17T16:15:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.661590+0000", 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5414000000000005e-05 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.661624+0000", 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4266e-05 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.662538+0000", 2024-09-17T16:15:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091425999999999996 2024-09-17T16:15:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:10.662568+0000", 2024-09-17T16:15:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9836999999999999e-05 2024-09-17T16:15:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:79589 2.0 2:00bb2590:::benchmark_data_smithi173_55121_object3100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:35.115215+0000", 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 151.52830367999999, 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20227213599999999, 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:15:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 79589 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115215+0000", 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115215+0000", 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115216+0000", 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.24e-06 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115221+0000", 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4989999999999996e-06 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115232+0000", 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.166e-05 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115239+0000", 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6069999999999996e-06 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115291+0000", 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2428e-05 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115411+0000", 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120098 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9", 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.115547+0000", 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000135368 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.116715+0000", 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011682369999999999 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.317399+0000", 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20068405 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.317433+0000", 2024-09-17T16:15:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4178999999999999e-05 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:35.317487+0000", 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3770000000000002e-05 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 157/155 [PushReplyOp(2:4c436b3d:::benchmark_data_smithi173_55121_object46284:head)])", 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:43.901089+0000", 2024-09-17T16:15:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 142.74242990900001, 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19615692600000001, 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:43.901089+0000", 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:43.901089+0000", 2024-09-17T16:15:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:43.901090+0000", 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5300000000000002e-07 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:43.901095+0000", 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4929999999999998e-06 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.096430+0000", 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.195335068 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.096434+0000", 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2169999999999996e-06 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.096463+0000", 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8495000000000001e-05 2024-09-17T16:15:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.097246+0000", 2024-09-17T16:15:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0007827 2024-09-17T16:15:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 158/155 [PushReplyOp(2:499d4037:::benchmark_data_smithi173_55121_object46772:head)])", 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:44.315863+0000", 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 142.32765549000001, 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16739532600000001, 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.315863+0000", 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.315863+0000", 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.315864+0000", 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9999999999999997e-07 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.315868+0000", 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6190000000000002e-06 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.482487+0000", 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16661815599999999 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.482491+0000", 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2749999999999997e-06 2024-09-17T16:15:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.482520+0000", 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9130999999999999e-05 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:44.483258+0000", 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073854499999999996 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:91009 2.7 e158/104 2:e0d66b6b:::benchmark_data_smithi173_55121_object14520:head v 158'3309, mlcod=158'3305)", 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:46.779887+0000", 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 139.863632037, 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20306453699999999, 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.779887+0000", 2024-09-17T16:15:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.779887+0000", 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.779887+0000", 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9599999999999999e-07 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.779892+0000", 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5249999999999999e-06 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.981252+0000", 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20136042900000001 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.981260+0000", 2024-09-17T16:15:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.824e-06 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.981325+0000", 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5195000000000002e-05 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.981351+0000", 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5935000000000002e-05 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.982913+0000", 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015619379999999999 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:46.982951+0000", 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8095e-05 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:93035 2.12 2:4eb811d6:::benchmark_data_smithi173_55121_object16546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.696790+0000", 2024-09-17T16:15:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 137.94672867099999, 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22142989699999999, 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93035 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696790+0000", 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696790+0000", 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696792+0000", 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.592e-06 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696796+0000", 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4859999999999999e-06 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696806+0000", 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0261999999999999e-05 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696812+0000", 2024-09-17T16:15:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4960000000000004e-06 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696851+0000", 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9588000000000001e-05 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696952+0000", 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100528 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.697033+0000", 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1314000000000001e-05 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.698646+0000", 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016126459999999999 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.698655+0000", 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6290000000000005e-06 2024-09-17T16:15:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917751+0000", 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21909500900000001 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917859+0000", 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108412 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.918220+0000", 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000360935 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:93035 2.12 e158/155 ondisk, result = 0)", 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.699652+0000", 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 137.943866361, 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21857501500000001, 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699652+0000", 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699652+0000", 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699653+0000", 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5600000000000005e-07 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699657+0000", 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.092e-06 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917601+0000", 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21794412199999999 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917615+0000", 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3397000000000001e-05 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917723+0000", 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108371 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917739+0000", 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6042999999999999e-05 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.918227+0000", 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048833399999999999 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 158/155 [PushReplyOp(2:4d5499b6:::benchmark_data_smithi173_55121_object58972:head)])", 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 137.881843452, 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46593107700000003, 2024-09-17T16:15:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761676+0000", 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1900000000000002e-07 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761680+0000", 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1710000000000004e-06 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226484+0000", 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46480413500000001 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226502+0000", 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7895999999999999e-05 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226671+0000", 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016863300000000001 2024-09-17T16:15:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.227606+0000", 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093562300000000001 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(2.12 e163,155 2:4c1d21f9:::benchmark_data_smithi173_55121_object3753:head,150'2969)", 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:53.886957+0000", 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 132.75656143800001, 2024-09-17T16:15:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16653584900000001, 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.886957+0000", 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.886957+0000", 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.886958+0000", 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2300000000000001e-07 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:53.886962+0000", 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1130000000000003e-06 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:54.053273+0000", 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16631127800000001 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:54.053282+0000", 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5930000000000006e-06 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:54.053330+0000", 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7741999999999998e-05 2024-09-17T16:15:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:54.053493+0000", 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016349999999999999 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:117822 2.12 2:4d80328a:::benchmark_data_smithi173_55121_object41333:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:15.355549+0000", 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 111.287969988, 2024-09-17T16:15:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17288575, 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 117822 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355549+0000", 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355549+0000", 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355550+0000", 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.691e-06 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355556+0000", 2024-09-17T16:15:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0519999999999998e-06 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355569+0000", 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2748000000000001e-05 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355577+0000", 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9189999999999999e-06 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355640+0000", 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3325999999999996e-05 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355767+0000", 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012693900000000001 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.355879+0000", 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011152100000000001 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.357392+0000", 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015128850000000001 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.357507+0000", 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115241 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.528162+0000", 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.170655316 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.528385+0000", 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000222463 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:15.528434+0000", 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9648999999999998e-05 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25036.0:19586 3.1 e223/208 3:8cb67056:::benchmark_data_smithi173_66272_object19585:head v 223'1267, mlcod=223'1263)", 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:14:43.072456+0000", 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.571062213000001, 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21023583000000001, 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:43.072456+0000", 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:43.072456+0000", 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:43.072457+0000", 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8700000000000004e-07 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:43.072501+0000", 2024-09-17T16:15:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4048999999999998e-05 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:43.281755+0000", 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20925382300000001 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:43.281764+0000", 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6750000000000008e-06 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:43.281823+0000", 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8604000000000003e-05 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:43.281845+0000", 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2504e-05 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:43.282661+0000", 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081572100000000005 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:43.282692+0000", 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1467000000000002e-05 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25036.0:36629 3.1 e229/208 3:8b324404:::benchmark_data_smithi173_66272_object36628:head v 229'2297, mlcod=229'2295)", 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:14:56.617771+0000", 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.025747642000001, 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17118349899999999, 2024-09-17T16:15:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:56.617771+0000", 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:56.617771+0000", 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:15:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:56.617773+0000", 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.584e-06 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:56.617835+0000", 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2141e-05 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:56.787427+0000", 2024-09-17T16:15:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.169592302 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:56.787440+0000", 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3284e-05 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:56.787514+0000", 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4072999999999994e-05 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:56.787540+0000", 2024-09-17T16:15:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5514e-05 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:56.788918+0000", 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001377908 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:56.788954+0000", 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6693000000000003e-05 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:15:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:15:06.749 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:15:06.749 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:06.750 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:15:06.771 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:06.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:06.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:15:06.873 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:06.894 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:06 smithi017 ceph-mon[27571]: osdmap e234: 12 total, 12 up, 12 in 2024-09-17T16:15:06.894 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:06 smithi017 ceph-mon[27571]: pgmap v562: 17 pgs: 17 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 726 op/s 2024-09-17T16:15:06.894 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3038529526' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:06.900 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:06.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:06.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:06.915 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:15:06.916 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:15:07.001 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:07.027 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:07.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:07.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:06 smithi071 ceph-mon[30002]: osdmap e234: 12 total, 12 up, 12 in 2024-09-17T16:15:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:06 smithi071 ceph-mon[30002]: pgmap v562: 17 pgs: 17 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 726 op/s 2024-09-17T16:15:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3038529526' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:07.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:07.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:06 smithi173 ceph-mon[30530]: osdmap e234: 12 total, 12 up, 12 in 2024-09-17T16:15:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:06 smithi173 ceph-mon[30530]: pgmap v562: 17 pgs: 17 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 726 op/s 2024-09-17T16:15:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3038529526' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:07.152 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:07.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:07.188 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 42236 42220 50.7396 0 - 0.0174077 2024-09-17T16:15:07.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:07.277 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:07.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:07.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:07.405 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:07.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:07.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:07.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:15:07.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:07.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:07.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:07.656 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:07.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:07.758 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:07.779 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:15:07.784 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:07.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:07.829 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:07 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:07.829 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3038529526' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:15:07.829 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:07 smithi017 ceph-mon[27571]: osdmap e235: 12 total, 12 up, 12 in 2024-09-17T16:15:07.829 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:07 smithi017 ceph-mon[27571]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:15:07.886 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:07.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:07 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:07.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3038529526' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:15:07.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:07 smithi173 ceph-mon[30530]: osdmap e235: 12 total, 12 up, 12 in 2024-09-17T16:15:07.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:07 smithi173 ceph-mon[30530]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:15:07.911 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:07.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:08.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:08.022 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":563,"stamp":"2024-09-17T16:15:06.679081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2768437792,"num_objects":42236,"num_object_clones":0,"num_object_copies":126708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42236,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":42357,"num_write_kb":2705126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42326,"ondisk_log_size":42326,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9264180,"kb_used_data":8131324,"kb_used_omap":58,"kb_used_meta":1132549,"kb_avail":1115677644,"statfs":{"total":1151940427776,"available":1142453907456,"internally_reserved":0,"allocated":8326475776,"data_stored":8318351362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":1159730339},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":130613248,"num_objects":1993,"num_object_clones":0,"num_object_copies":5979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1993,"num_write_kb":127552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.318550"},"pg_stats":[{"pgid":"3.f","version":"231'2620","reported_seq":2678,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.379320+0000","last_change":"2024-09-17T16:14:08.390556+0000","last_active":"2024-09-17T16:15:01.379320+0000","last_peered":"2024-09-17T16:15:01.379320+0000","last_clean":"2024-09-17T16:15:01.379320+0000","last_became_active":"2024-09-17T16:14:08.390112+0000","last_became_peered":"2024-09-17T16:14:08.390112+0000","last_unstale":"2024-09-17T16:15:01.379320+0000","last_undegraded":"2024-09-17T16:15:01.379320+0000","last_fullsized":"2024-09-17T16:15:01.379320+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2620,"log_dups_size":0,"ondisk_log_size":2620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:30.215439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171704320,"num_objects":2620,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2620,"num_write_kb":167680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"231'2655","reported_seq":2713,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.361303+0000","last_change":"2024-09-17T16:14:08.430036+0000","last_active":"2024-09-17T16:15:01.361303+0000","last_peered":"2024-09-17T16:15:01.361303+0000","last_clean":"2024-09-17T16:15:01.361303+0000","last_became_active":"2024-09-17T16:14:08.429780+0000","last_became_peered":"2024-09-17T16:14:08.429780+0000","last_unstale":"2024-09-17T16:15:01.361303+0000","last_undegraded":"2024-09-17T16:15:01.361303+0000","last_fullsized":"2024-09-17T16:15:01.361303+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2655,"log_dups_size":0,"ondisk_log_size":2655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:25.106933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173998080,"num_objects":2655,"num_object_clones":0,"num_object_copies":7965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":169920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"232'2613","reported_seq":2671,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.378632+0000","last_change":"2024-09-17T16:14:08.487685+0000","last_active":"2024-09-17T16:15:01.378632+0000","last_peered":"2024-09-17T16:15:01.378632+0000","last_clean":"2024-09-17T16:15:01.378632+0000","last_became_active":"2024-09-17T16:14:08.487146+0000","last_became_peered":"2024-09-17T16:14:08.487146+0000","last_unstale":"2024-09-17T16:15:01.378632+0000","last_undegraded":"2024-09-17T16:15:01.378632+0000","last_fullsized":"2024-09-17T16:15:01.378632+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2613,"log_dups_size":0,"ondisk_log_size":2613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:22.079514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171245568,"num_objects":2613,"num_object_clones":0,"num_object_copies":7839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2613,"num_write_kb":167232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"231'2588","reported_seq":2646,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.380740+0000","last_change":"2024-09-17T16:14:08.389726+0000","last_active":"2024-09-17T16:15:01.380740+0000","last_peered":"2024-09-17T16:15:01.380740+0000","last_clean":"2024-09-17T16:15:01.380740+0000","last_became_active":"2024-09-17T16:14:08.389315+0000","last_became_peered":"2024-09-17T16:14:08.389315+0000","last_unstale":"2024-09-17T16:15:01.380740+0000","last_undegraded":"2024-09-17T16:15:01.380740+0000","last_fullsized":"2024-09-17T16:15:01.380740+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2588,"log_dups_size":0,"ondisk_log_size":2588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:21.272899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":165632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"232'2638","reported_seq":2700,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-17T16:15:05.674850+0000","last_change":"2024-09-17T16:14:08.429967+0000","last_active":"2024-09-17T16:15:05.674850+0000","last_peered":"2024-09-17T16:15:05.674850+0000","last_clean":"2024-09-17T16:15:05.674850+0000","last_became_active":"2024-09-17T16:14:08.429641+0000","last_became_peered":"2024-09-17T16:14:08.429641+0000","last_unstale":"2024-09-17T16:15:05.674850+0000","last_undegraded":"2024-09-17T16:15:05.674850+0000","last_fullsized":"2024-09-17T16:15:05.674850+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2638,"log_dups_size":0,"ondisk_log_size":2638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:11.920378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172883968,"num_objects":2638,"num_object_clones":0,"num_object_copies":7914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2638,"num_write_kb":168832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"231'2683","reported_seq":2744,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T16:15:04.665349+0000","last_change":"2024-09-17T16:14:08.401229+0000","last_active":"2024-09-17T16:15:04.665349+0000","last_peered":"2024-09-17T16:15:04.665349+0000","last_clean":"2024-09-17T16:15:04.665349+0000","last_became_active":"2024-09-17T16:14:08.400841+0000","last_became_peered":"2024-09-17T16:14:08.400841+0000","last_unstale":"2024-09-17T16:15:04.665349+0000","last_undegraded":"2024-09-17T16:15:04.665349+0000","last_fullsized":"2024-09-17T16:15:04.665349+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2683,"log_dups_size":0,"ondisk_log_size":2683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:07.787994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175833088,"num_objects":2683,"num_object_clones":0,"num_object_copies":8049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2683,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"231'2619","reported_seq":2677,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.640530+0000","last_change":"2024-09-17T16:14:08.388882+0000","last_active":"2024-09-17T16:15:01.640530+0000","last_peered":"2024-09-17T16:15:01.640530+0000","last_clean":"2024-09-17T16:15:01.640530+0000","last_became_active":"2024-09-17T16:14:08.388540+0000","last_became_peered":"2024-09-17T16:14:08.388540+0000","last_unstale":"2024-09-17T16:15:01.640530+0000","last_undegraded":"2024-09-17T16:15:01.640530+0000","last_fullsized":"2024-09-17T16:15:01.640530+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2619,"log_dups_size":0,"ondisk_log_size":2619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:23.901375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171638784,"num_objects":2619,"num_object_clones":0,"num_object_copies":7857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2619,"num_write_kb":167616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"231'2590","reported_seq":2648,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.361164+0000","last_change":"2024-09-17T16:14:08.386368+0000","last_active":"2024-09-17T16:15:01.361164+0000","last_peered":"2024-09-17T16:15:01.361164+0000","last_clean":"2024-09-17T16:15:01.361164+0000","last_became_active":"2024-09-17T16:14:08.386099+0000","last_became_peered":"2024-09-17T16:14:08.386099+0000","last_unstale":"2024-09-17T16:15:01.361164+0000","last_undegraded":"2024-09-17T16:15:01.361164+0000","last_fullsized":"2024-09-17T16:15:01.361164+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:11.876057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":7770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":165760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"232'2673","reported_seq":2735,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-17T16:15:05.674895+0000","last_change":"2024-09-17T16:14:08.389198+0000","last_active":"2024-09-17T16:15:05.674895+0000","last_peered":"2024-09-17T16:15:05.674895+0000","last_clean":"2024-09-17T16:15:05.674895+0000","last_became_active":"2024-09-17T16:14:08.388835+0000","last_became_peered":"2024-09-17T16:14:08.388835+0000","last_unstale":"2024-09-17T16:15:05.674895+0000","last_undegraded":"2024-09-17T16:15:05.674895+0000","last_fullsized":"2024-09-17T16:15:05.674895+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2673,"log_dups_size":0,"ondisk_log_size":2673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:11.956946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175177728,"num_objects":2673,"num_object_clones":0,"num_object_copies":8019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2673,"num_write_kb":171072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"231'2629","reported_seq":2690,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-17T16:15:05.675074+0000","last_change":"2024-09-17T16:14:08.429867+0000","last_active":"2024-09-17T16:15:05.675074+0000","last_peered":"2024-09-17T16:15:05.675074+0000","last_clean":"2024-09-17T16:15:05.675074+0000","last_became_active":"2024-09-17T16:14:08.429516+0000","last_became_peered":"2024-09-17T16:14:08.429516+0000","last_unstale":"2024-09-17T16:15:05.675074+0000","last_undegraded":"2024-09-17T16:15:05.675074+0000","last_fullsized":"2024-09-17T16:15:05.675074+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:13.029924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172294144,"num_objects":2629,"num_object_clones":0,"num_object_copies":7887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":168256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"232'2633","reported_seq":2691,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:02.145333+0000","last_change":"2024-09-17T16:14:08.391300+0000","last_active":"2024-09-17T16:15:02.145333+0000","last_peered":"2024-09-17T16:15:02.145333+0000","last_clean":"2024-09-17T16:15:02.145333+0000","last_became_active":"2024-09-17T16:14:08.390988+0000","last_became_peered":"2024-09-17T16:14:08.390988+0000","last_unstale":"2024-09-17T16:15:02.145333+0000","last_undegraded":"2024-09-17T16:15:02.145333+0000","last_fullsized":"2024-09-17T16:15:02.145333+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2633,"log_dups_size":0,"ondisk_log_size":2633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:31.739010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172556288,"num_objects":2633,"num_object_clones":0,"num_object_copies":7899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":168512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"231'2662","reported_seq":2723,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-17T16:15:05.675045+0000","last_change":"2024-09-17T16:14:08.389363+0000","last_active":"2024-09-17T16:15:05.675045+0000","last_peered":"2024-09-17T16:15:05.675045+0000","last_clean":"2024-09-17T16:15:05.675045+0000","last_became_active":"2024-09-17T16:14:08.389008+0000","last_became_peered":"2024-09-17T16:14:08.389008+0000","last_unstale":"2024-09-17T16:15:05.675045+0000","last_undegraded":"2024-09-17T16:15:05.675045+0000","last_fullsized":"2024-09-17T16:15:05.675045+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2662,"log_dups_size":0,"ondisk_log_size":2662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:32.838518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174456832,"num_objects":2662,"num_object_clones":0,"num_object_copies":7986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"231'2674","reported_seq":2733,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.640514+0000","last_change":"2024-09-17T16:14:08.430014+0000","last_active":"2024-09-17T16:15:01.640514+0000","last_peered":"2024-09-17T16:15:01.640514+0000","last_clean":"2024-09-17T16:15:01.640514+0000","last_became_active":"2024-09-17T16:14:08.429652+0000","last_became_peered":"2024-09-17T16:14:08.429652+0000","last_unstale":"2024-09-17T16:15:01.640514+0000","last_undegraded":"2024-09-17T16:15:01.640514+0000","last_fullsized":"2024-09-17T16:15:01.640514+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2674,"log_dups_size":0,"ondisk_log_size":2674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:27.959451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175243264,"num_objects":2674,"num_object_clones":0,"num_object_copies":8022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2674,"num_write_kb":171136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"231'2654","reported_seq":2713,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.640470+0000","last_change":"2024-09-17T16:14:08.487461+0000","last_active":"2024-09-17T16:15:01.640470+0000","last_peered":"2024-09-17T16:15:01.640470+0000","last_clean":"2024-09-17T16:15:01.640470+0000","last_became_active":"2024-09-17T16:14:08.487062+0000","last_became_peered":"2024-09-17T16:14:08.487062+0000","last_unstale":"2024-09-17T16:15:01.640470+0000","last_undegraded":"2024-09-17T16:15:01.640470+0000","last_fullsized":"2024-09-17T16:15:01.640470+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2654,"log_dups_size":0,"ondisk_log_size":2654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:36.384153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173932544,"num_objects":2654,"num_object_clones":0,"num_object_copies":7962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":169856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,10],"acting":[4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"231'2653","reported_seq":2697,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.361193+0000","last_change":"2024-09-17T16:14:08.488081+0000","last_active":"2024-09-17T16:15:01.361193+0000","last_peered":"2024-09-17T16:15:01.361193+0000","last_clean":"2024-09-17T16:15:01.361193+0000","last_became_active":"2024-09-17T16:14:08.487747+0000","last_became_peered":"2024-09-17T16:14:08.487747+0000","last_unstale":"2024-09-17T16:15:01.361193+0000","last_undegraded":"2024-09-17T16:15:01.361193+0000","last_fullsized":"2024-09-17T16:15:01.361193+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2653,"log_dups_size":0,"ondisk_log_size":2653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:20.221194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173867008,"num_objects":2653,"num_object_clones":0,"num_object_copies":7959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2653,"num_write_kb":169792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"232'2650","reported_seq":2712,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-17T16:15:05.674850+0000","last_change":"2024-09-17T16:14:08.487206+0000","last_active":"2024-09-17T16:15:05.674850+0000","last_peered":"2024-09-17T16:15:05.674850+0000","last_clean":"2024-09-17T16:15:05.674850+0000","last_became_active":"2024-09-17T16:14:08.486914+0000","last_became_peered":"2024-09-17T16:14:08.486914+0000","last_unstale":"2024-09-17T16:15:05.674850+0000","last_undegraded":"2024-09-17T16:15:05.674850+0000","last_fullsized":"2024-09-17T16:15:05.674850+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2650,"log_dups_size":0,"ondisk_log_size":2650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:08.901512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173670400,"num_objects":2650,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2650,"num_write_kb":169600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":331,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.378575+0000","last_change":"2024-09-17T16:14:54.958180+0000","last_active":"2024-09-17T16:15:01.378575+0000","last_peered":"2024-09-17T16:15:01.378575+0000","last_clean":"2024-09-17T16:15:01.378575+0000","last_became_active":"2024-09-17T16:12:39.286388+0000","last_became_peered":"2024-09-17T16:12:39.286388+0000","last_unstale":"2024-09-17T16:15:01.378575+0000","last_undegraded":"2024-09-17T16:15:01.378575+0000","last_fullsized":"2024-09-17T16:15:01.378575+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:02.145235+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2767847424,"num_objects":42234,"num_object_clones":0,"num_object_copies":126702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42234,"num_write_kb":2702976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8302886912,"data_stored":8302886912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42234,"ondisk_log_size":42234,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285635,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1292184,"kb_used_data":1184644,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92452968,"statfs":{"total":95995035648,"available":94671839232,"internally_reserved":0,"allocated":1213075456,"data_stored":1212398377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547271,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":251412,"kb_used_data":171644,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93493740,"statfs":{"total":95995035648,"available":95737589760,"internally_reserved":0,"allocated":175763456,"data_stored":175089949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":81656256},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743501,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":629992,"kb_used_data":508564,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93115160,"statfs":{"total":95995035648,"available":95349923840,"internally_reserved":0,"allocated":520769536,"data_stored":520086991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":124320192},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005137,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412688,"kb_used_data":337084,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93332464,"statfs":{"total":95995035648,"available":95572443136,"internally_reserved":0,"allocated":345174016,"data_stored":344500509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963614,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":764572,"kb_used_data":675844,"kb_used_omap":41,"kb_used_meta":88662,"kb_avail":92980580,"statfs":{"total":95995035648,"available":95212113920,"internally_reserved":0,"allocated":692064256,"data_stored":691387197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":90790195},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495708,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938788,"kb_used_data":849296,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92806364,"statfs":{"total":95995035648,"available":95033716736,"internally_reserved":0,"allocated":869679104,"data_stored":868997505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724639,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749136,"kb_used_data":673532,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92996016,"statfs":{"total":95995035648,"available":95227920384,"internally_reserved":0,"allocated":689696768,"data_stored":689023261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986277,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286480,"kb_used_data":1187260,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92458672,"statfs":{"total":95995035648,"available":94677680128,"internally_reserved":0,"allocated":1215754240,"data_stored":1215080733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":975184,"kb_used_data":845436,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92769968,"statfs":{"total":95995035648,"available":94996447232,"internally_reserved":0,"allocated":865726464,"data_stored":865052957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132839879},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":237608,"kb_used_data":170324,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93507544,"statfs":{"total":95995035648,"available":95751725056,"internally_reserved":0,"allocated":174411776,"data_stored":173729231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":777128,"kb_used_data":677908,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92968024,"statfs":{"total":95995035648,"available":95199256576,"internally_reserved":0,"allocated":694177792,"data_stored":693495247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":949008,"kb_used_data":849788,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92796144,"statfs":{"total":95995035648,"available":95023251456,"internally_reserved":0,"allocated":870182912,"data_stored":869509405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":868352000,"data_stored":868352000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":692322304,"data_stored":692322304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172556288,"data_stored":172556288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":863895552,"data_stored":863895552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1213923328,"data_stored":1213923328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":687865856,"data_stored":687865856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":867303424,"data_stored":867303424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":689766400,"data_stored":689766400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":343343104,"data_stored":343343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":518914048,"data_stored":518914048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":173932544,"data_stored":173932544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1210712064,"data_stored":1210712064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:15:08.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:15:08.037 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:08.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:08.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:07 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:08.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3038529526' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:15:08.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:07 smithi071 ceph-mon[30002]: osdmap e235: 12 total, 12 up, 12 in 2024-09-17T16:15:08.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:07 smithi071 ceph-mon[30002]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:15:08.079 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:08.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:08.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:08.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:08.188 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 42236 42220 49.7823 0 - 0.0174077 2024-09-17T16:15:08.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:08.286 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:08.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:08.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:08.307 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:15:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:15:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:15:08.307 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:15:08.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:08.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:08.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:08.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:08.557 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:08.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:08.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:08.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:08.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:08.704 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:15:08.704 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:15:08.732 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:15:08.732 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:15:08.732 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:08 smithi017 ceph-mon[27571]: pgmap v564: 17 pgs: 17 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 214 op/s 2024-09-17T16:15:08.733 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:08.754 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:15:08.754 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:15:08.779 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:15:08.779 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:15:08.780 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:08.802 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:15:08.803 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:15:08.803 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:08.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:08.824 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:15:08.825 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:15:08.846 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:15:08.846 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:15:08.867 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:15:08.868 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:15:08.888 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:15:08.889 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:15:08.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:08 smithi173 ceph-mon[30530]: pgmap v564: 17 pgs: 17 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 214 op/s 2024-09-17T16:15:08.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:08.912 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:15:08.912 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:15:08.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:15:08.927 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:08.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:08.932 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:15:08.932 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:15:08.953 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:15:08.954 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:15:09.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:09.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:09.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:09.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:08 smithi071 ceph-mon[30002]: pgmap v564: 17 pgs: 17 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 214 op/s 2024-09-17T16:15:09.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:09.168 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:15:09.174 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:09.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:09.188 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 42236 42220 48.8604 0 - 0.0174077 2024-09-17T16:15:09.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:09.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:09.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:09.403 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:09.429 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:09.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:09.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:09.556 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:09.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:09.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:09.602 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:15:09.637 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:15:09.638 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:15:09.639 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:15:09.639 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:15:09.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:09.682 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:09.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:09.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:09.810 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:09.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:09.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:09.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:09 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:09 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/927836972' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:09 smithi173 ceph-mon[30530]: osdmap e236: 12 total, 12 up, 12 in 2024-09-17T16:15:09.897 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:09 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:09.898 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:09 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:09.898 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/927836972' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:09.898 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:09 smithi017 ceph-mon[27571]: osdmap e236: 12 total, 12 up, 12 in 2024-09-17T16:15:09.912 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:09.945 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:09.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:10.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:10.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:09 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:10.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:09 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:10.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/927836972' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:10.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:09 smithi071 ceph-mon[30002]: osdmap e236: 12 total, 12 up, 12 in 2024-09-17T16:15:10.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:10.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:10.169 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:10.188 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 42746 42730 48.5516 4.55357 0.00262804 0.0205911 2024-09-17T16:15:10.200 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:10.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:10.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:15:10.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:10.332 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:10.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:10.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:10.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:10.460 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:10.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:10.563 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:10.590 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:10.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:10.607 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:15:10.692 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:10.720 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:10.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:10.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:10.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:10.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:10.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:10.917 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:10 smithi017 ceph-mon[27571]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:15:10.918 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:10 smithi017 ceph-mon[27571]: pgmap v566: 17 pgs: 17 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s wr, 21 op/s 2024-09-17T16:15:10.939 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:10.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:10.952 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":565,"stamp":"2024-09-17T16:15:08.684700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2768437792,"num_objects":42236,"num_object_clones":0,"num_object_copies":126708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42236,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":42357,"num_write_kb":2705126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42326,"ondisk_log_size":42326,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9264264,"kb_used_data":8131412,"kb_used_omap":58,"kb_used_meta":1132549,"kb_avail":1115677560,"statfs":{"total":1151940427776,"available":1142453821440,"internally_reserved":0,"allocated":8326565888,"data_stored":8318432436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":1159730339},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":52428800,"num_objects":800,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":800,"num_write_kb":51200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.908558"},"pg_stats":[{"pgid":"3.f","version":"231'2620","reported_seq":2683,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-17T16:15:07.276124+0000","last_change":"2024-09-17T16:14:08.390556+0000","last_active":"2024-09-17T16:15:07.276124+0000","last_peered":"2024-09-17T16:15:07.276124+0000","last_clean":"2024-09-17T16:15:07.276124+0000","last_became_active":"2024-09-17T16:14:08.390112+0000","last_became_peered":"2024-09-17T16:14:08.390112+0000","last_unstale":"2024-09-17T16:15:07.276124+0000","last_undegraded":"2024-09-17T16:15:07.276124+0000","last_fullsized":"2024-09-17T16:15:07.276124+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2620,"log_dups_size":0,"ondisk_log_size":2620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:30.215439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171704320,"num_objects":2620,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2620,"num_write_kb":167680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"231'2655","reported_seq":2713,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.361303+0000","last_change":"2024-09-17T16:14:08.430036+0000","last_active":"2024-09-17T16:15:01.361303+0000","last_peered":"2024-09-17T16:15:01.361303+0000","last_clean":"2024-09-17T16:15:01.361303+0000","last_became_active":"2024-09-17T16:14:08.429780+0000","last_became_peered":"2024-09-17T16:14:08.429780+0000","last_unstale":"2024-09-17T16:15:01.361303+0000","last_undegraded":"2024-09-17T16:15:01.361303+0000","last_fullsized":"2024-09-17T16:15:01.361303+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2655,"log_dups_size":0,"ondisk_log_size":2655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:25.106933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173998080,"num_objects":2655,"num_object_clones":0,"num_object_copies":7965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":169920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"232'2613","reported_seq":2676,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-17T16:15:07.276102+0000","last_change":"2024-09-17T16:14:08.487685+0000","last_active":"2024-09-17T16:15:07.276102+0000","last_peered":"2024-09-17T16:15:07.276102+0000","last_clean":"2024-09-17T16:15:07.276102+0000","last_became_active":"2024-09-17T16:14:08.487146+0000","last_became_peered":"2024-09-17T16:14:08.487146+0000","last_unstale":"2024-09-17T16:15:07.276102+0000","last_undegraded":"2024-09-17T16:15:07.276102+0000","last_fullsized":"2024-09-17T16:15:07.276102+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2613,"log_dups_size":0,"ondisk_log_size":2613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:22.079514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171245568,"num_objects":2613,"num_object_clones":0,"num_object_copies":7839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2613,"num_write_kb":167232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"231'2588","reported_seq":2646,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.380740+0000","last_change":"2024-09-17T16:14:08.389726+0000","last_active":"2024-09-17T16:15:01.380740+0000","last_peered":"2024-09-17T16:15:01.380740+0000","last_clean":"2024-09-17T16:15:01.380740+0000","last_became_active":"2024-09-17T16:14:08.389315+0000","last_became_peered":"2024-09-17T16:14:08.389315+0000","last_unstale":"2024-09-17T16:15:01.380740+0000","last_undegraded":"2024-09-17T16:15:01.380740+0000","last_fullsized":"2024-09-17T16:15:01.380740+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2588,"log_dups_size":0,"ondisk_log_size":2588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:21.272899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":165632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"232'2638","reported_seq":2700,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-17T16:15:05.674850+0000","last_change":"2024-09-17T16:14:08.429967+0000","last_active":"2024-09-17T16:15:05.674850+0000","last_peered":"2024-09-17T16:15:05.674850+0000","last_clean":"2024-09-17T16:15:05.674850+0000","last_became_active":"2024-09-17T16:14:08.429641+0000","last_became_peered":"2024-09-17T16:14:08.429641+0000","last_unstale":"2024-09-17T16:15:05.674850+0000","last_undegraded":"2024-09-17T16:15:05.674850+0000","last_fullsized":"2024-09-17T16:15:05.674850+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2638,"log_dups_size":0,"ondisk_log_size":2638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:11.920378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172883968,"num_objects":2638,"num_object_clones":0,"num_object_copies":7914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2638,"num_write_kb":168832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"231'2683","reported_seq":2744,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T16:15:04.665349+0000","last_change":"2024-09-17T16:14:08.401229+0000","last_active":"2024-09-17T16:15:04.665349+0000","last_peered":"2024-09-17T16:15:04.665349+0000","last_clean":"2024-09-17T16:15:04.665349+0000","last_became_active":"2024-09-17T16:14:08.400841+0000","last_became_peered":"2024-09-17T16:14:08.400841+0000","last_unstale":"2024-09-17T16:15:04.665349+0000","last_undegraded":"2024-09-17T16:15:04.665349+0000","last_fullsized":"2024-09-17T16:15:04.665349+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2683,"log_dups_size":0,"ondisk_log_size":2683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:07.787994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175833088,"num_objects":2683,"num_object_clones":0,"num_object_copies":8049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2683,"num_write_kb":171712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"231'2619","reported_seq":2677,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.640530+0000","last_change":"2024-09-17T16:14:08.388882+0000","last_active":"2024-09-17T16:15:01.640530+0000","last_peered":"2024-09-17T16:15:01.640530+0000","last_clean":"2024-09-17T16:15:01.640530+0000","last_became_active":"2024-09-17T16:14:08.388540+0000","last_became_peered":"2024-09-17T16:14:08.388540+0000","last_unstale":"2024-09-17T16:15:01.640530+0000","last_undegraded":"2024-09-17T16:15:01.640530+0000","last_fullsized":"2024-09-17T16:15:01.640530+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2619,"log_dups_size":0,"ondisk_log_size":2619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:23.901375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171638784,"num_objects":2619,"num_object_clones":0,"num_object_copies":7857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2619,"num_write_kb":167616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"231'2590","reported_seq":2648,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.361164+0000","last_change":"2024-09-17T16:14:08.386368+0000","last_active":"2024-09-17T16:15:01.361164+0000","last_peered":"2024-09-17T16:15:01.361164+0000","last_clean":"2024-09-17T16:15:01.361164+0000","last_became_active":"2024-09-17T16:14:08.386099+0000","last_became_peered":"2024-09-17T16:14:08.386099+0000","last_unstale":"2024-09-17T16:15:01.361164+0000","last_undegraded":"2024-09-17T16:15:01.361164+0000","last_fullsized":"2024-09-17T16:15:01.361164+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:11.876057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169738240,"num_objects":2590,"num_object_clones":0,"num_object_copies":7770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":165760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"232'2673","reported_seq":2735,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-17T16:15:05.674895+0000","last_change":"2024-09-17T16:14:08.389198+0000","last_active":"2024-09-17T16:15:05.674895+0000","last_peered":"2024-09-17T16:15:05.674895+0000","last_clean":"2024-09-17T16:15:05.674895+0000","last_became_active":"2024-09-17T16:14:08.388835+0000","last_became_peered":"2024-09-17T16:14:08.388835+0000","last_unstale":"2024-09-17T16:15:05.674895+0000","last_undegraded":"2024-09-17T16:15:05.674895+0000","last_fullsized":"2024-09-17T16:15:05.674895+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2673,"log_dups_size":0,"ondisk_log_size":2673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:11.956946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175177728,"num_objects":2673,"num_object_clones":0,"num_object_copies":8019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2673,"num_write_kb":171072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"231'2629","reported_seq":2690,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-17T16:15:05.675074+0000","last_change":"2024-09-17T16:14:08.429867+0000","last_active":"2024-09-17T16:15:05.675074+0000","last_peered":"2024-09-17T16:15:05.675074+0000","last_clean":"2024-09-17T16:15:05.675074+0000","last_became_active":"2024-09-17T16:14:08.429516+0000","last_became_peered":"2024-09-17T16:14:08.429516+0000","last_unstale":"2024-09-17T16:15:05.675074+0000","last_undegraded":"2024-09-17T16:15:05.675074+0000","last_fullsized":"2024-09-17T16:15:05.675074+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:13.029924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172294144,"num_objects":2629,"num_object_clones":0,"num_object_copies":7887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":168256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"232'2633","reported_seq":2691,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:02.145333+0000","last_change":"2024-09-17T16:14:08.391300+0000","last_active":"2024-09-17T16:15:02.145333+0000","last_peered":"2024-09-17T16:15:02.145333+0000","last_clean":"2024-09-17T16:15:02.145333+0000","last_became_active":"2024-09-17T16:14:08.390988+0000","last_became_peered":"2024-09-17T16:14:08.390988+0000","last_unstale":"2024-09-17T16:15:02.145333+0000","last_undegraded":"2024-09-17T16:15:02.145333+0000","last_fullsized":"2024-09-17T16:15:02.145333+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2633,"log_dups_size":0,"ondisk_log_size":2633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:31.739010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172556288,"num_objects":2633,"num_object_clones":0,"num_object_copies":7899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":168512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"231'2662","reported_seq":2723,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-17T16:15:05.675045+0000","last_change":"2024-09-17T16:14:08.389363+0000","last_active":"2024-09-17T16:15:05.675045+0000","last_peered":"2024-09-17T16:15:05.675045+0000","last_clean":"2024-09-17T16:15:05.675045+0000","last_became_active":"2024-09-17T16:14:08.389008+0000","last_became_peered":"2024-09-17T16:14:08.389008+0000","last_unstale":"2024-09-17T16:15:05.675045+0000","last_undegraded":"2024-09-17T16:15:05.675045+0000","last_fullsized":"2024-09-17T16:15:05.675045+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2662,"log_dups_size":0,"ondisk_log_size":2662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:32.838518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174456832,"num_objects":2662,"num_object_clones":0,"num_object_copies":7986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"231'2674","reported_seq":2733,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.640514+0000","last_change":"2024-09-17T16:14:08.430014+0000","last_active":"2024-09-17T16:15:01.640514+0000","last_peered":"2024-09-17T16:15:01.640514+0000","last_clean":"2024-09-17T16:15:01.640514+0000","last_became_active":"2024-09-17T16:14:08.429652+0000","last_became_peered":"2024-09-17T16:14:08.429652+0000","last_unstale":"2024-09-17T16:15:01.640514+0000","last_undegraded":"2024-09-17T16:15:01.640514+0000","last_fullsized":"2024-09-17T16:15:01.640514+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2674,"log_dups_size":0,"ondisk_log_size":2674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:27.959451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175243264,"num_objects":2674,"num_object_clones":0,"num_object_copies":8022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2674,"num_write_kb":171136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"231'2654","reported_seq":2713,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.640470+0000","last_change":"2024-09-17T16:14:08.487461+0000","last_active":"2024-09-17T16:15:01.640470+0000","last_peered":"2024-09-17T16:15:01.640470+0000","last_clean":"2024-09-17T16:15:01.640470+0000","last_became_active":"2024-09-17T16:14:08.487062+0000","last_became_peered":"2024-09-17T16:14:08.487062+0000","last_unstale":"2024-09-17T16:15:01.640470+0000","last_undegraded":"2024-09-17T16:15:01.640470+0000","last_fullsized":"2024-09-17T16:15:01.640470+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2654,"log_dups_size":0,"ondisk_log_size":2654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:36.384153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173932544,"num_objects":2654,"num_object_clones":0,"num_object_copies":7962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2654,"num_write_kb":169856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,10],"acting":[4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"231'2653","reported_seq":2697,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T16:15:01.361193+0000","last_change":"2024-09-17T16:14:08.488081+0000","last_active":"2024-09-17T16:15:01.361193+0000","last_peered":"2024-09-17T16:15:01.361193+0000","last_clean":"2024-09-17T16:15:01.361193+0000","last_became_active":"2024-09-17T16:14:08.487747+0000","last_became_peered":"2024-09-17T16:14:08.487747+0000","last_unstale":"2024-09-17T16:15:01.361193+0000","last_undegraded":"2024-09-17T16:15:01.361193+0000","last_fullsized":"2024-09-17T16:15:01.361193+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2653,"log_dups_size":0,"ondisk_log_size":2653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:20.221194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173867008,"num_objects":2653,"num_object_clones":0,"num_object_copies":7959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2653,"num_write_kb":169792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"232'2650","reported_seq":2712,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-17T16:15:05.674850+0000","last_change":"2024-09-17T16:14:08.487206+0000","last_active":"2024-09-17T16:15:05.674850+0000","last_peered":"2024-09-17T16:15:05.674850+0000","last_clean":"2024-09-17T16:15:05.674850+0000","last_became_active":"2024-09-17T16:14:08.486914+0000","last_became_peered":"2024-09-17T16:14:08.486914+0000","last_unstale":"2024-09-17T16:15:05.674850+0000","last_undegraded":"2024-09-17T16:15:05.674850+0000","last_fullsized":"2024-09-17T16:15:05.674850+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":2650,"log_dups_size":0,"ondisk_log_size":2650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:15:08.901512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173670400,"num_objects":2650,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2650,"num_write_kb":169600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":336,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-17T16:15:07.276100+0000","last_change":"2024-09-17T16:14:54.958180+0000","last_active":"2024-09-17T16:15:07.276100+0000","last_peered":"2024-09-17T16:15:07.276100+0000","last_clean":"2024-09-17T16:15:07.276100+0000","last_became_active":"2024-09-17T16:12:39.286388+0000","last_became_peered":"2024-09-17T16:12:39.286388+0000","last_unstale":"2024-09-17T16:15:07.276100+0000","last_undegraded":"2024-09-17T16:15:07.276100+0000","last_fullsized":"2024-09-17T16:15:07.276100+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:02.145235+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2767847424,"num_objects":42234,"num_object_clones":0,"num_object_copies":126702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42234,"num_write_kb":2702976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8302886912,"data_stored":8302886912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42234,"ondisk_log_size":42234,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285635,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1292184,"kb_used_data":1184644,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92452968,"statfs":{"total":95995035648,"available":94671839232,"internally_reserved":0,"allocated":1213075456,"data_stored":1212398377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547271,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":251412,"kb_used_data":171644,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93493740,"statfs":{"total":95995035648,"available":95737589760,"internally_reserved":0,"allocated":175763456,"data_stored":175089949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":81656256},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743501,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":629992,"kb_used_data":508564,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93115160,"statfs":{"total":95995035648,"available":95349923840,"internally_reserved":0,"allocated":520769536,"data_stored":520086991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":124320192},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005137,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412688,"kb_used_data":337084,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93332464,"statfs":{"total":95995035648,"available":95572443136,"internally_reserved":0,"allocated":345174016,"data_stored":344500509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963615,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":764656,"kb_used_data":675932,"kb_used_omap":41,"kb_used_meta":88662,"kb_avail":92980496,"statfs":{"total":95995035648,"available":95212027904,"internally_reserved":0,"allocated":692154368,"data_stored":691468271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":90790195},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495708,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938788,"kb_used_data":849296,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92806364,"statfs":{"total":95995035648,"available":95033716736,"internally_reserved":0,"allocated":869679104,"data_stored":868997505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724639,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749136,"kb_used_data":673532,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92996016,"statfs":{"total":95995035648,"available":95227920384,"internally_reserved":0,"allocated":689696768,"data_stored":689023261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986277,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286480,"kb_used_data":1187260,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92458672,"statfs":{"total":95995035648,"available":94677680128,"internally_reserved":0,"allocated":1215754240,"data_stored":1215080733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":975184,"kb_used_data":845436,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92769968,"statfs":{"total":95995035648,"available":94996447232,"internally_reserved":0,"allocated":865726464,"data_stored":865052957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132839879},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":237608,"kb_used_data":170324,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93507544,"statfs":{"total":95995035648,"available":95751725056,"internally_reserved":0,"allocated":174411776,"data_stored":173729231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":777128,"kb_used_data":677908,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92968024,"statfs":{"total":95995035648,"available":95199256576,"internally_reserved":0,"allocated":694177792,"data_stored":693495247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":949008,"kb_used_data":849788,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92796144,"statfs":{"total":95995035648,"available":95023251456,"internally_reserved":0,"allocated":870182912,"data_stored":869509405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":868352000,"data_stored":868352000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":692322304,"data_stored":692322304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172556288,"data_stored":172556288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":863895552,"data_stored":863895552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1213923328,"data_stored":1213923328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":687865856,"data_stored":687865856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":867303424,"data_stored":867303424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":689766400,"data_stored":689766400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":343343104,"data_stored":343343104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":518914048,"data_stored":518914048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":173932544,"data_stored":173932544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1210712064,"data_stored":1210712064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:15:10.953 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T16:15:10.953 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:15:10.954 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-17T16:15:10.954 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd out 11 2024-09-17T16:15:10.974 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:10.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:11.032 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:10 smithi173 ceph-mon[30530]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:15:11.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:10 smithi173 ceph-mon[30530]: pgmap v566: 17 pgs: 17 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s wr, 21 op/s 2024-09-17T16:15:11.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:10 smithi071 ceph-mon[30002]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:15:11.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:10 smithi071 ceph-mon[30002]: pgmap v566: 17 pgs: 17 active+clean; 2.6 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s wr, 21 op/s 2024-09-17T16:15:11.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:11.100 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:11.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:11.152 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:15:11.153 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:15:11.188 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 44369 44353 49.4958 101.438 0.003929 0.0201937 2024-09-17T16:15:11.202 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:11.235 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:11.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:11.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:11.360 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:11.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:11.463 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:11.488 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:11.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:11.544 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:11.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:11.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:11.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:11.715 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:11.743 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:11.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:11.825 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:15:11.844 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:11.874 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:11.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:11.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:15:11.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:11.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/939527840' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:11.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:11 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:12.004 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:12.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:12.106 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:12.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:15:12.134 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:12.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:12.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/939527840' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:12.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:11 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:12.188 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 45575 45559 49.9497 75.375 0.00669277 0.0200152 2024-09-17T16:15:12.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:12.241 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:15:12.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:12.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:12.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/939527840' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:12.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:11 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:12.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:12.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:12.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:12.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:12.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:12.520 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:12.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:12.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:12.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:12.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:12.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:12.750 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:12.777 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:12.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:12.816 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:15:12.816 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:15:12.836 INFO:teuthology.orchestra.run.smithi017.stderr:marked out osd.11. 2024-09-17T16:15:12.850 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:12 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:15:12.597+0000 7f086a07d640 -1 mon.a@0(leader).osd e237 definitely_dead 0 2024-09-17T16:15:12.850 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:12 smithi017 ceph-mon[27571]: pgmap v567: 17 pgs: 17 active+clean; 2.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 154 op/s 2024-09-17T16:15:12.850 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:12 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:12.851 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:12 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:15:12.851 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:12 smithi017 ceph-mon[27571]: osdmap e237: 12 total, 12 up, 12 in 2024-09-17T16:15:12.851 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1463624067' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-09-17T16:15:12.851 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:12 smithi017 ceph-mon[27571]: Client client.admin marked osd.11 out, while it was still marked up 2024-09-17T16:15:12.879 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:12.902 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:12.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:13.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:13.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:13.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:13.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:12 smithi071 ceph-mon[30002]: pgmap v567: 17 pgs: 17 active+clean; 2.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 154 op/s 2024-09-17T16:15:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:12 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:12 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:15:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:12 smithi071 ceph-mon[30002]: osdmap e237: 12 total, 12 up, 12 in 2024-09-17T16:15:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1463624067' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-09-17T16:15:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:12 smithi071 ceph-mon[30002]: Client client.admin marked osd.11 out, while it was still marked up 2024-09-17T16:15:13.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:13.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:12 smithi173 ceph-mon[30530]: pgmap v567: 17 pgs: 17 active+clean; 2.6 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s wr, 154 op/s 2024-09-17T16:15:13.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:12 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:12 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:15:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:12 smithi173 ceph-mon[30530]: osdmap e237: 12 total, 12 up, 12 in 2024-09-17T16:15:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1463624067' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-09-17T16:15:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:12 smithi173 ceph-mon[30530]: Client client.admin marked osd.11 out, while it was still marked up 2024-09-17T16:15:13.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:13.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:13.151 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:13.188 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 46505 46489 50.0905 58.125 0.38799 0.0198655 2024-09-17T16:15:13.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:13.275 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:13.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:13.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:13.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:13.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:13.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:13.500 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:13.527 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:13.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:13.545 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:15:13.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:13.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:13.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:13.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:13.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:13.774 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:13.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:13.791 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:15:13.876 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:13.899 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:13.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:14.001 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:14.020 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:14.022 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:14.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:14.099 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1463624067' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-09-17T16:15:14.099 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:13 smithi071 ceph-mon[30002]: osdmap e238: 12 total, 12 up, 11 in 2024-09-17T16:15:14.099 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:13 smithi071 ceph-mon[30002]: 3.6 scrub starts 2024-09-17T16:15:14.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:14.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1463624067' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-09-17T16:15:14.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:13 smithi017 ceph-mon[27571]: osdmap e238: 12 total, 12 up, 11 in 2024-09-17T16:15:14.145 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:13 smithi017 ceph-mon[27571]: 3.6 scrub starts 2024-09-17T16:15:14.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1463624067' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-09-17T16:15:14.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:13 smithi173 ceph-mon[30530]: osdmap e238: 12 total, 12 up, 11 in 2024-09-17T16:15:14.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:13 smithi173 ceph-mon[30530]: 3.6 scrub starts 2024-09-17T16:15:14.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:14.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:14.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:14.188 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 46512 46496 49.2489 0.4375 0.0171011 0.0198955 2024-09-17T16:15:14.248 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:14.270 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:14.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:14.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:14.371 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:15:14.372 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:15:14.372 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:14.400 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:14.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:14.502 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:14.524 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:14.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:14.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:14.582 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:15:14.582 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:15:14.609 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:15:14.609 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:15:14.626 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:14.635 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:15:14.635 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:15:14.648 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:14.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:14.662 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:15:14.662 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:15:14.690 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:15:14.690 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:15:14.718 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:15:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:15:14.746 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:15:14.746 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:15:14.750 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:14.774 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:15:14.774 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:15:14.776 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:14.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:14.799 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:15:14.799 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:15:14.822 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:15:14.822 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:15:14.845 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:15:14.845 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:15:14.850 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:15:14.869 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:15:14.869 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:15:14.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:14.905 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:14.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:14.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:14.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:14 smithi017 ceph-mon[27571]: pgmap v570: 17 pgs: 2 peering, 15 active+clean; 2.7 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 15 MiB/s wr, 238 op/s 2024-09-17T16:15:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:14 smithi017 ceph-mon[27571]: osdmap e239: 12 total, 12 up, 11 in 2024-09-17T16:15:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:14 smithi017 ceph-mon[27571]: 3.6 scrub ok 2024-09-17T16:15:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:14 smithi017 ceph-mon[27571]: osdmap e240: 12 total, 12 up, 11 in 2024-09-17T16:15:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/455254356' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:15.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:15.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:15.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:15.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:15.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:15.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:14 smithi071 ceph-mon[30002]: pgmap v570: 17 pgs: 2 peering, 15 active+clean; 2.7 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 15 MiB/s wr, 238 op/s 2024-09-17T16:15:15.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:14 smithi071 ceph-mon[30002]: osdmap e239: 12 total, 12 up, 11 in 2024-09-17T16:15:15.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:14 smithi071 ceph-mon[30002]: 3.6 scrub ok 2024-09-17T16:15:15.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:14 smithi071 ceph-mon[30002]: osdmap e240: 12 total, 12 up, 11 in 2024-09-17T16:15:15.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/455254356' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:15.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:15:15.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:15.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:14 smithi173 ceph-mon[30530]: pgmap v570: 17 pgs: 2 peering, 15 active+clean; 2.7 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 15 MiB/s wr, 238 op/s 2024-09-17T16:15:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:14 smithi173 ceph-mon[30530]: osdmap e239: 12 total, 12 up, 11 in 2024-09-17T16:15:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:14 smithi173 ceph-mon[30530]: 3.6 scrub ok 2024-09-17T16:15:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:14 smithi173 ceph-mon[30530]: osdmap e240: 12 total, 12 up, 11 in 2024-09-17T16:15:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/455254356' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:15.157 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:15.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:15.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:15:15.187762+0000 min lat: 0.00101616 max lat: 15.0111 avg lat: 0.020276 2024-09-17T16:15:15.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:15:15.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 46615 46599 48.5354 6.4375 0.00295384 0.020276 2024-09-17T16:15:15.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:15.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:15.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:15.378 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:15:15.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:15.406 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:15.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:15.508 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:15.533 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:15.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:15.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:15.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:15.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:15.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:15.726 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:15.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:15.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:15.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:15.881 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:15.904 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:15.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:15.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:15 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:16.006 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:16.023 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:15 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:16.024 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:15:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:15:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:15:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:15:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDelete(2.3 e160,155 2:c2c889ad:::benchmark_data_smithi173_55121_object5047:head,151'3009)", 2024-09-17T16:15:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:49.878573+0000", 2024-09-17T16:15:16.024 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 146.14263395699999, 2024-09-17T16:15:16.025 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16635193100000001, 2024-09-17T16:15:16.025 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.025 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.025 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:15:16.025 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.025 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.878573+0000", 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.878573+0000", 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.026 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.878573+0000", 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.99e-07 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.878577+0000", 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7570000000000002e-06 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.044831+0000", 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.166254343 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.027 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.044837+0000", 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8100000000000003e-06 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.044885+0000", 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7420999999999999e-05 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.044890+0000", 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4229999999999996e-06 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.044925+0000", 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4578e-05 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(2.9 160/155 [PushOp(2:94005a9d:::benchmark_data_smithi173_55121_object63654:head, version: 146'2470, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:94005a9d:::benchmark_data_smithi173_55121_object63654:head@146'2470, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:15:16.029 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:49.897521+0000", 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 146.123685346, 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18302432599999999, 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.897521+0000", 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.030 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.897521+0000", 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.897522+0000", 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.75e-07 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.897589+0000", 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6979999999999999e-05 2024-09-17T16:15:16.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.079658+0000", 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18206876499999999 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.079671+0000", 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3885e-05 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.079736+0000", 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5001999999999999e-05 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.079755+0000", 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8620000000000001e-05 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.080546+0000", 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00079049899999999995 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.033 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDelete(2.3 e160,155 2:c46c76ef:::benchmark_data_smithi173_55121_object5059:head,151'3010)", 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:50.046445+0000", 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 145.97476110299999, 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25348822999999998, 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.046445+0000", 2024-09-17T16:15:16.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.046445+0000", 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.046446+0000", 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.06e-07 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.046451+0000", 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0139999999999998e-06 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.299807+0000", 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25335623000000002 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.299815+0000", 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5009999999999998e-06 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.299871+0000", 2024-09-17T16:15:16.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6076000000000003e-05 2024-09-17T16:15:16.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.299880+0000", 2024-09-17T16:15:16.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7779999999999996e-06 2024-09-17T16:15:16.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.299934+0000", 2024-09-17T16:15:16.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4024999999999999e-05 2024-09-17T16:15:16.039 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.039 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.039 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:16.041 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(2.12 160/155 [PushOp(2:48e3b754:::benchmark_data_smithi173_55121_object61434:head, version: 146'2378, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:48e3b754:::benchmark_data_smithi173_55121_object61434:head@146'2378, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:15:16.041 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:50.104141+0000", 2024-09-17T16:15:16.041 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 145.91706586699999, 2024-09-17T16:15:16.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.214399545, 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.104141+0000", 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.104141+0000", 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.104141+0000", 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6300000000000005e-07 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.104210+0000", 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8403000000000002e-05 2024-09-17T16:15:16.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.317511+0000", 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21330083799999999 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.317525+0000", 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4382000000000001e-05 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.317655+0000", 2024-09-17T16:15:16.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012977900000000001 2024-09-17T16:15:16.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.317668+0000", 2024-09-17T16:15:16.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3326000000000001e-05 2024-09-17T16:15:16.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.318540+0000", 2024-09-17T16:15:16.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00087215399999999998 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(2.12 162/155 [PushOp(2:4bbf9771:::benchmark_data_smithi173_55121_object70551:head, version: 148'2654, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4bbf9771:::benchmark_data_smithi173_55121_object70551:head@148'2654, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:51.719799+0000", 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 144.301407848, 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20531496299999999, 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.047 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:51.719799+0000", 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:51.719799+0000", 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:51.719800+0000", 2024-09-17T16:15:16.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2799999999999995e-07 2024-09-17T16:15:16.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:51.719826+0000", 2024-09-17T16:15:16.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6434000000000001e-05 2024-09-17T16:15:16.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.050 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:51.924044+0000", 2024-09-17T16:15:16.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.204218173 2024-09-17T16:15:16.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.050 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:51.924071+0000", 2024-09-17T16:15:16.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7066000000000001e-05 2024-09-17T16:15:16.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:51.924229+0000", 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000157904 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:51.924252+0000", 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2413e-05 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:51.925114+0000", 2024-09-17T16:15:16.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00086214500000000003 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:97425 2.3 e163/162 2:c4562d4f:::benchmark_data_smithi173_55121_object20936:head v 163'3539, mlcod=163'3535)", 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.848313+0000", 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 142.17289367199999, 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29519683400000002, 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:16.052 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848313+0000", 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848313+0000", 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848313+0000", 2024-09-17T16:15:16.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1799999999999995e-07 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848319+0000", 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4269999999999999e-06 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.142150+0000", 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29383070500000003 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.142156+0000", 2024-09-17T16:15:16.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4590000000000003e-06 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.142211+0000", 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4429999999999999e-05 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.142224+0000", 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3319000000000001e-05 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:15:16.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143466+0000", 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012424669999999999 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143510+0000", 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3408999999999997e-05 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(2.8 169/155 [PushOp(2:15367df9:::benchmark_data_smithi173_55121_object57692:head, version: 145'2302, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:15367df9:::benchmark_data_smithi173_55121_object57692:head@145'2302, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:15:16.056 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:01.411091+0000", 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 134.61011574700001, 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19032403000000001, 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:01.411091+0000", 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:01.411091+0000", 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:01.411091+0000", 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3700000000000001e-07 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:01.411150+0000", 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8499999999999999e-05 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:01.600191+0000", 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18904146599999999 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:01.600201+0000", 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9879999999999999e-06 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:01.600292+0000", 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0308000000000003e-05 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.060 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:01.600305+0000", 2024-09-17T16:15:16.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3071000000000001e-05 2024-09-17T16:15:16.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.060 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:01.601415+0000", 2024-09-17T16:15:16.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011102600000000001 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:117054 2.8 2:1421ce96:::benchmark_data_smithi173_55121_object40565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:14.525111+0000", 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 121.496095055, 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18235525399999999, 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:15:16.061 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 117054 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.525111+0000", 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.525111+0000", 2024-09-17T16:15:16.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.525113+0000", 2024-09-17T16:15:16.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.125e-06 2024-09-17T16:15:16.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.525117+0000", 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6380000000000003e-06 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.525126+0000", 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9879999999999993e-06 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.525133+0000", 2024-09-17T16:15:16.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.905e-06 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.525185+0000", 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1925999999999997e-05 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.525305+0000", 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120412 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:15:16.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.525428+0000", 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000122426 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.527027+0000", 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015994309999999999 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.610928+0000", 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.083900796999999999 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.707396+0000", 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.096468134999999997 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.707426+0000", 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9961e-05 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:14.707467+0000", 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0509999999999997e-05 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.067 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:150345 2.3 e196/162 2:c2afc64a:::benchmark_data_smithi173_55121_object73856:head v 196'5196, mlcod=196'5193)", 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:50.745648+0000", 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 85.275558700000005, 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18199242500000001, 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.745648+0000", 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.745648+0000", 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.745649+0000", 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1099999999999995e-07 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.745653+0000", 2024-09-17T16:15:16.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6859999999999997e-06 2024-09-17T16:15:16.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.925954+0000", 2024-09-17T16:15:16.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.180300504 2024-09-17T16:15:16.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.925959+0000", 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0499999999999999e-06 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.926034+0000", 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5329999999999999e-05 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.926055+0000", 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0990000000000001e-05 2024-09-17T16:15:16.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.927600+0000", 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015453660000000001 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.927640+0000", 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9787999999999999e-05 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:150371 2.8 2:12ea3975:::benchmark_data_smithi173_55121_object73882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:50.887617+0000", 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 85.133589944999997, 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.180404119, 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 150371 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.073 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.887617+0000", 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.887617+0000", 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.887619+0000", 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9709999999999998e-06 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.887626+0000", 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5970000000000003e-06 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.887646+0000", 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9902999999999999e-05 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.887655+0000", 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1560000000000001e-06 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.887718+0000", 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2317999999999995e-05 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.887805+0000", 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7050000000000002e-05 2024-09-17T16:15:16.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.887879+0000", 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3907999999999998e-05 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.889487+0000", 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001608906 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.890188+0000", 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00070065800000000005 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.067958+0000", 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17776982499999999 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.067991+0000", 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3423000000000002e-05 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.068021+0000", 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9403999999999998e-05 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:150420 2.8 2:176d9c95:::benchmark_data_smithi173_55121_object73931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:51.065422+0000", 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 84.955784854000001, 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22095720899999999, 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.079 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 150420 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.065422+0000", 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.065422+0000", 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.065423+0000", 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3340000000000001e-06 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.065428+0000", 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9599999999999999e-06 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.065440+0000", 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1815e-05 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.065446+0000", 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.117e-06 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.065501+0000", 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5019000000000003e-05 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.065623+0000", 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000122509 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.065738+0000", 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000114159 2024-09-17T16:15:16.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.067377+0000", 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00163922 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.068844+0000", 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014667669999999999 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.286321+0000", 2024-09-17T16:15:16.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21747712799999999 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.286352+0000", 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1099999999999997e-05 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.286379+0000", 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7081e-05 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:150422 2.18 e197/176 2:1f0d1d1f:::benchmark_data_smithi173_55121_object73933:head v 197'5252, mlcod=197'5250)", 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:51.068780+0000", 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 84.952426716999994, 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18489804000000001, 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.068780+0000", 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.068780+0000", 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.068780+0000", 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0200000000000002e-07 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.068785+0000", 2024-09-17T16:15:16.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.228e-06 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.251854+0000", 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18306962900000001 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.251859+0000", 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6140000000000002e-06 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.251906+0000", 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7617000000000002e-05 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.251920+0000", 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3431000000000001e-05 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:15:16.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.253648+0000", 2024-09-17T16:15:16.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017280310000000001 2024-09-17T16:15:16.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:51.253678+0000", 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9988000000000002e-05 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:152907 2.3 e199/162 2:c65c8e4e:::benchmark_data_smithi173_55121_object76418:head v 199'5280, mlcod=199'5277)", 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.797893+0000", 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 82.223313352999995, 2024-09-17T16:15:16.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86957243200000001, 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797893+0000", 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797893+0000", 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797902+0000", 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9670000000000007e-06 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797908+0000", 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8279999999999999e-06 2024-09-17T16:15:16.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.665615+0000", 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86770716699999995 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.665626+0000", 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1146e-05 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.665718+0000", 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1854999999999994e-05 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.665739+0000", 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0394000000000002e-05 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667418+0000", 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016790710000000001 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667466+0000", 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8004000000000003e-05 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:152967 2.8 2:178f648d:::benchmark_data_smithi173_55121_object76478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.857328+0000", 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 82.163878272999995, 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.592092548, 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.098 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152967 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857328+0000", 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857328+0000", 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857329+0000", 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0559999999999999e-06 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857334+0000", 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4900000000000002e-06 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857344+0000", 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6830000000000003e-06 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857349+0000", 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4110000000000004e-06 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857387+0000", 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8015000000000001e-05 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857468+0000", 2024-09-17T16:15:16.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1210000000000003e-05 2024-09-17T16:15:16.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:15:16.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857534+0000", 2024-09-17T16:15:16.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5752000000000003e-05 2024-09-17T16:15:16.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.103 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:16.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:16.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:15:16.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859065+0000", 2024-09-17T16:15:16.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001530942 2024-09-17T16:15:16.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859971+0000", 2024-09-17T16:15:16.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00090643500000000003 2024-09-17T16:15:16.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449347+0000", 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58937566600000002 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449387+0000", 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9771999999999998e-05 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449421+0000", 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4116000000000003e-05 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:152967 2.8 e199/174 ondisk, result = 0)", 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.859730+0000", 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 82.161476948000001, 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58969649800000001, 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.106 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859730+0000", 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859730+0000", 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859730+0000", 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859734+0000", 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7869999999999999e-06 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449259+0000", 2024-09-17T16:15:16.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58952536 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449283+0000", 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3918999999999999e-05 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449335+0000", 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1146000000000001e-05 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449343+0000", 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3240000000000003e-06 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449426+0000", 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3239999999999996e-05 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:8164 3.f 3:fe9aa252:::benchmark_data_smithi173_66272_object8163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:14:35.045522+0000", 2024-09-17T16:15:16.110 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 40.975684344999998, 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18094761600000001, 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8164 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.045522+0000", 2024-09-17T16:15:16.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.045522+0000", 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.045523+0000", 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.017e-06 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.045581+0000", 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8112999999999997e-05 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.045592+0000", 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0492999999999999e-05 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.045597+0000", 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5040000000000002e-06 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.045637+0000", 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0086000000000002e-05 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.045725+0000", 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7841000000000001e-05 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,11", 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.045818+0000", 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2842000000000003e-05 2024-09-17T16:15:16.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.046321+0000", 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00050303900000000005 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.220665+0000", 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.174343941 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.226368+0000", 2024-09-17T16:15:16.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0057029469999999999 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.226428+0000", 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9920000000000002e-05 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.226470+0000", 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1872999999999997e-05 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.116 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25036.0:8164 3.f e220/208 ondisk, result = 0)", 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:14:35.046580+0000", 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 40.974626374000003, 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17989587600000001, 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.046580+0000", 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.046580+0000", 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.046581+0000", 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.75e-07 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.046584+0000", 2024-09-17T16:15:16.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6160000000000002e-06 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.226300+0000", 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17971596300000001 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.226306+0000", 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6030000000000004e-06 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.226355+0000", 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8705000000000002e-05 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.226362+0000", 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4630000000000004e-06 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.226476+0000", 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000113851 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.120 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.121 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.121 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.121 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:11477 3.d 3:bb51aaf5:::benchmark_data_smithi173_66272_object11476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-17T16:15:16.121 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:14:36.909165+0000", 2024-09-17T16:15:16.121 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 39.112041570999999, 2024-09-17T16:15:16.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.184479069, 2024-09-17T16:15:16.121 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.121 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 11477 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.909165+0000", 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.909165+0000", 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.909167+0000", 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8300000000000001e-06 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.909198+0000", 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0760999999999999e-05 2024-09-17T16:15:16.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.909215+0000", 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7010000000000001e-05 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.909223+0000", 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8629999999999997e-06 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.909281+0000", 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7482999999999998e-05 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.909385+0000", 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010403099999999999 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 5,9", 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.909474+0000", 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9285999999999999e-05 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.910062+0000", 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058749200000000001 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:36.910523+0000", 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00046113300000000002 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:37.093533+0000", 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.183009798 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:37.093601+0000", 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.834e-05 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:37.093644+0000", 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3041999999999999e-05 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.127 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(3.f 240/239 [PushReplyOp(3:fa6b8f5a:::benchmark_data_smithi173_66272_object267:head)])", 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:14.634025+0000", 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.3871810600000001, 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27450650500000001, 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.634025+0000", 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.128 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.634025+0000", 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.634026+0000", 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3099999999999996e-07 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.634032+0000", 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4269999999999999e-06 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.906376+0000", 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27234407799999999 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.906381+0000", 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.322e-06 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.906421+0000", 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9978000000000002e-05 2024-09-17T16:15:16.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.908532+0000", 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021107690000000002 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(3.f 240/239 [PushReplyOp(3:f5f4cfa1:::benchmark_data_smithi173_66272_object320:head)])", 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:14.963777+0000", 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.0574294710000001, 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58111184800000004, 2024-09-17T16:15:16.131 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963777+0000", 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963777+0000", 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963778+0000", 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1500000000000003e-07 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963783+0000", 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9540000000000003e-06 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.542775+0000", 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57899182000000005 2024-09-17T16:15:16.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.542785+0000", 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7559999999999994e-06 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.542840+0000", 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5593000000000001e-05 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.544889+0000", 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020488099999999999 2024-09-17T16:15:16.134 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.135 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.135 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.135 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:15:16.135 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:15:16.135 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:15:16.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:15 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:16.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 46627 46611 47.752 0.75 0.0123318 0.0202981 2024-09-17T16:15:16.203 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:16.226 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:16.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:16.328 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:16.354 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:16.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:16.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:16.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:16.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:16.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:16.602 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:16.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:16.704 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:16.730 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:16.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:16.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:16.833 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:16.857 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:16.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:16.876 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:15:16.900 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:15:16.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:16.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:16.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:17.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:17.103 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:17.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:17.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:16 smithi173 ceph-mon[30530]: pgmap v573: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 peering, 13 active+clean; 2.8 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 35 MiB/s wr, 555 op/s; 5811/136170 objects misplaced (4.267%); 192 KiB/s, 2 objects/s recovering 2024-09-17T16:15:17.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:16 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:17.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:16 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:17.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/344054710' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:17.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:16 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:17.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 46627 46611 46.9819 0 - 0.0202981 2024-09-17T16:15:17.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:17.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:16 smithi017 ceph-mon[27571]: pgmap v573: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 peering, 13 active+clean; 2.8 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 35 MiB/s wr, 555 op/s; 5811/136170 objects misplaced (4.267%); 192 KiB/s, 2 objects/s recovering 2024-09-17T16:15:17.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:16 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:17.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:16 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:17.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/344054710' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:17.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:16 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:17.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:17.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:17.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:16 smithi071 ceph-mon[30002]: pgmap v573: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 peering, 13 active+clean; 2.8 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 35 MiB/s wr, 555 op/s; 5811/136170 objects misplaced (4.267%); 192 KiB/s, 2 objects/s recovering 2024-09-17T16:15:17.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:16 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:17.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:16 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:17.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/344054710' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:16 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:17.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:17.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:17.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:17.453 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:17.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:17.583 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:17.607 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:17.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:17.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:17.714 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:15:17.734 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:17.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:17.836 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:17.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:17.863 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:17.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:17.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:17.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:17.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:17.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:17.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:17 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:15:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:17 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:15:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:17 smithi017 ceph-mon[27571]: osdmap e241: 12 total, 12 up, 11 in 2024-09-17T16:15:18.089 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:18.110 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:18.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:18.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:18.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:17 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:15:18.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:17 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:15:18.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:17 smithi173 ceph-mon[30530]: osdmap e241: 12 total, 12 up, 11 in 2024-09-17T16:15:18.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 46627 46611 46.2361 0 - 0.0202981 2024-09-17T16:15:18.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:18.235 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:18.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:18.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:17 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:15:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:17 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:15:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:17 smithi071 ceph-mon[30002]: osdmap e241: 12 total, 12 up, 11 in 2024-09-17T16:15:18.337 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:18.379 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:18.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:18.469 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:15:18.469 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:15:18.469 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:15:18.469 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:15:18.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:18.504 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:18.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:18.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:18.639 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:18.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:18.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:18.741 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:18.769 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:18.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:18.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:18.871 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:18.896 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:18.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:18.971 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:15:18.971 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:15:18.971 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-17T16:15:18.971 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-17T16:15:18.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:19.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:19.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:19.025 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:19.126 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:19.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:18 smithi173 ceph-mon[30530]: pgmap v575: 17 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 peering, 9 active+clean; 2.8 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 43 MiB/s wr, 688 op/s; 2937/139716 objects degraded (2.102%); 13962/139716 objects misplaced (9.993%); 7.8 MiB/s, 125 objects/s recovering 2024-09-17T16:15:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:19.149 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:19.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:19.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 15 47414 47399 46.2832 16.4167 0.120284 0.0215986 2024-09-17T16:15:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:18 smithi017 ceph-mon[27571]: pgmap v575: 17 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 peering, 9 active+clean; 2.8 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 43 MiB/s wr, 688 op/s; 2937/139716 objects degraded (2.102%); 13962/139716 objects misplaced (9.993%); 7.8 MiB/s, 125 objects/s recovering 2024-09-17T16:15:19.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:19.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:19.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:19.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:18 smithi071 ceph-mon[30002]: pgmap v575: 17 pgs: 1 activating+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 peering, 9 active+clean; 2.8 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 43 MiB/s wr, 688 op/s; 2937/139716 objects degraded (2.102%); 13962/139716 objects misplaced (9.993%); 7.8 MiB/s, 125 objects/s recovering 2024-09-17T16:15:19.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:19.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:19.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:19.510 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:19.539 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:19.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:19.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:19.627 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:19.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:19.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:15:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:19.668 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:19.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:19.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:15:19.769 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:19.853 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:19.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:19.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:19.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:19.982 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:20.083 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:20.131 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:20.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:20.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 48432 48416 46.5489 63.5625 0.0152989 0.0214764 2024-09-17T16:15:20.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:20.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:20.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:20.267 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:20.301 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:15:20.301 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:15:20.336 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:20.351 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:15:20.352 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:15:20.360 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:20.387 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:20.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:20.397 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:15:20.397 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:15:20.429 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:15:20.429 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:15:20.463 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:15:20.463 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:15:20.464 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:20 smithi017 ceph-mon[27571]: osdmap e242: 12 total, 12 up, 11 in 2024-09-17T16:15:20.464 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1528480625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:20.488 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:20.501 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:15:20.502 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:15:20.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:20.524 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:20.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:20.541 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:15:20.541 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:15:20.575 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:15:20.575 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:15:20.613 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:15:20.613 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:15:20.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:20.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:20 smithi173 ceph-mon[30530]: osdmap e242: 12 total, 12 up, 11 in 2024-09-17T16:15:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1528480625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:20.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:20.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:20.651 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:15:20.652 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:15:20.690 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:15:20.690 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:15:20.725 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:15:20.725 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:15:20.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:20.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:20.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:20 smithi071 ceph-mon[30002]: osdmap e242: 12 total, 12 up, 11 in 2024-09-17T16:15:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1528480625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:20.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:20.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:20.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:20.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:21.005 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:21.032 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:21.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:21.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:21.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:21.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:21.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 49489 49473 46.8445 66.0625 0.0190655 0.0213401 2024-09-17T16:15:21.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:21.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:21.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:21.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:21.429 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:21.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:21.470 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:21 smithi017 ceph-mon[27571]: pgmap v577: 17 pgs: 1 activating+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.9 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 34 MiB/s wr, 547 op/s; 2937/140763 objects degraded (2.086%); 15793/140763 objects misplaced (11.220%); 18 MiB/s, 280 objects/s recovering 2024-09-17T16:15:21.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:21.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:21.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:21.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:21 smithi173 ceph-mon[30530]: pgmap v577: 17 pgs: 1 activating+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.9 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 34 MiB/s wr, 547 op/s; 2937/140763 objects degraded (2.086%); 15793/140763 objects misplaced (11.220%); 18 MiB/s, 280 objects/s recovering 2024-09-17T16:15:21.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:21.657 INFO:teuthology.orchestra.run.smithi017.stdout:moved 0 / 51 (0%) 2024-09-17T16:15:21.657 INFO:teuthology.orchestra.run.smithi017.stdout:avg 4.63636 2024-09-17T16:15:21.657 INFO:teuthology.orchestra.run.smithi017.stdout:stddev 1.96666 -> 1.96666 (expected baseline 2.05302) 2024-09-17T16:15:21.658 INFO:teuthology.orchestra.run.smithi017.stdout:min osd.10 with 1 -> 1 pgs (0.215686 -> 0.215686 * mean) 2024-09-17T16:15:21.658 INFO:teuthology.orchestra.run.smithi017.stdout:max osd.4 with 7 -> 7 pgs (1.5098 -> 1.5098 * mean) 2024-09-17T16:15:21.658 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:15:21.658 INFO:teuthology.orchestra.run.smithi017.stdout:oload 1000 2024-09-17T16:15:21.658 INFO:teuthology.orchestra.run.smithi017.stdout:max_change 3 2024-09-17T16:15:21.658 INFO:teuthology.orchestra.run.smithi017.stdout:max_change_osds 4 2024-09-17T16:15:21.658 INFO:teuthology.orchestra.run.smithi017.stdout:average_utilization 0.0087 2024-09-17T16:15:21.658 INFO:teuthology.orchestra.run.smithi017.stdout:overload_utilization 0.0872 2024-09-17T16:15:21.658 INFO:teuthology.orchestra.run.smithi017.stderr:no change 2024-09-17T16:15:21.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:21.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:21.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:21.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:21.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:21.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:21.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:21 smithi071 ceph-mon[30002]: pgmap v577: 17 pgs: 1 activating+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 10 active+clean; 2.9 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 34 MiB/s wr, 547 op/s; 2937/140763 objects degraded (2.086%); 15793/140763 objects misplaced (11.220%); 18 MiB/s, 280 objects/s recovering 2024-09-17T16:15:21.904 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:21.911 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:21.950 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:21.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:21.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:22.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:22.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:22.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:22.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:22.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:22.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 50444 50428 47.0361 59.6875 0.00743212 0.0212554 2024-09-17T16:15:22.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:22.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:22.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:22.312 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:22.342 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:22.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:22.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:22.474 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:22.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:22.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:22.616 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:22.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:22.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:22 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:22 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:22 smithi173 ceph-mon[30530]: osdmap e243: 12 total, 12 up, 11 in 2024-09-17T16:15:22.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:22 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:22.681 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:22 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:22.681 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:22 smithi017 ceph-mon[27571]: osdmap e243: 12 total, 12 up, 11 in 2024-09-17T16:15:22.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:22.737 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:15:22.737 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:15:22.744 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:22 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:22.744 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:22 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:22.744 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:22 smithi071 ceph-mon[30002]: osdmap e243: 12 total, 12 up, 11 in 2024-09-17T16:15:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:22.755 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:22.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:22.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:22.892 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:22.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:22.994 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:23.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:23.027 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:23.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:23.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:23.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:23.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:23.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 51614 51598 47.4196 73.125 0.00592601 0.0210818 2024-09-17T16:15:23.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:23.288 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:23.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:23.390 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:23.423 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:23.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:23.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:23.553 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:23.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:23.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:23 smithi017 ceph-mon[27571]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:23.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:23 smithi017 ceph-mon[27571]: pgmap v579: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 10 active+clean; 3.0 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 32 MiB/s wr, 507 op/s; 2937/145308 objects degraded (2.021%); 14636/145308 objects misplaced (10.072%); 29 MiB/s, 468 objects/s recovering 2024-09-17T16:15:23.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:23 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2937/145308 objects degraded (2.021%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:15:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:23 smithi173 ceph-mon[30530]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:23 smithi173 ceph-mon[30530]: pgmap v579: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 10 active+clean; 3.0 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 32 MiB/s wr, 507 op/s; 2937/145308 objects degraded (2.021%); 14636/145308 objects misplaced (10.072%); 29 MiB/s, 468 objects/s recovering 2024-09-17T16:15:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:23 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2937/145308 objects degraded (2.021%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:15:23.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:23.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:23.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:23.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:23.783 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:23.811 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:23.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:23.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:23 smithi071 ceph-mon[30002]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:23.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:23 smithi071 ceph-mon[30002]: pgmap v579: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 10 active+clean; 3.0 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 32 MiB/s wr, 507 op/s; 2937/145308 objects degraded (2.021%); 14636/145308 objects misplaced (10.072%); 29 MiB/s, 468 objects/s recovering 2024-09-17T16:15:23.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:23 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2937/145308 objects degraded (2.021%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:15:23.913 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:23.938 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:23.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:23.976 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-17T16:15:23.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-17T16:15:24.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:24.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:24.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:24.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:24.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:24.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 52539 52523 47.5702 57.8125 0.00635411 0.0210159 2024-09-17T16:15:24.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:24.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:24.297 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:24.322 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:24.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:24.332 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:15:24.424 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:24.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:24.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:24.482 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:15:24.551 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:24.585 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:24.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:24.640 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/403776625' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:15:24.640 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:24 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:15:24.640 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:24 smithi017 ceph-mon[27571]: pgmap v580: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.0 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 27 MiB/s wr, 432 op/s; 1/147813 objects degraded (0.001%); 17011/147813 objects misplaced (11.508%); 25 MiB/s, 401 objects/s recovering 2024-09-17T16:15:24.641 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:24 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:24.641 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:24 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:15:24.641 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:24 smithi017 ceph-mon[27571]: osdmap e244: 12 total, 12 up, 11 in 2024-09-17T16:15:24.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/403776625' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:15:24.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:24 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:15:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:24 smithi173 ceph-mon[30530]: pgmap v580: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.0 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 27 MiB/s wr, 432 op/s; 1/147813 objects degraded (0.001%); 17011/147813 objects misplaced (11.508%); 25 MiB/s, 401 objects/s recovering 2024-09-17T16:15:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:24 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:24 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:15:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:24 smithi173 ceph-mon[30530]: osdmap e244: 12 total, 12 up, 11 in 2024-09-17T16:15:24.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:24.690 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:15:24.710 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:24.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:24.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:24.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:24.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/403776625' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:15:24.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:24 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:15:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:24 smithi071 ceph-mon[30002]: pgmap v580: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.0 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 27 MiB/s wr, 432 op/s; 1/147813 objects degraded (0.001%); 17011/147813 objects misplaced (11.508%); 25 MiB/s, 401 objects/s recovering 2024-09-17T16:15:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:24 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:24 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:15:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:24 smithi071 ceph-mon[30002]: osdmap e244: 12 total, 12 up, 11 in 2024-09-17T16:15:24.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:24.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:24.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:24.960 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:24.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:25.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:25.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:25.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:25.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:25.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 53328 53312 47.595 49.3125 0.0116928 0.0210043 2024-09-17T16:15:25.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:25.221 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:25.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:25.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:25.323 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:25.358 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:25.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:25.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:25.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:25.486 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:25.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:25.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:25.615 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:25.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:25.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:25.744 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:25.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:25.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:25.876 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:25.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:25.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:25.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:26.004 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:26.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:26.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:26.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:26.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:26.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:26.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 54271 54255 47.7547 58.9375 0.0116795 0.0209297 2024-09-17T16:15:26.236 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:26.263 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:26.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:26.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:26.392 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:26.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:26.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:26.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:26.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:26.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:26.648 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:26.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:26.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:26.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:26.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:26.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:26.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:26.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:27.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:27.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:27.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:27.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:27.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:27.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:26 smithi173 ceph-mon[30530]: pgmap v582: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.1 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 47 MiB/s wr, 744 op/s; 1/154164 objects degraded (0.001%); 16119/154164 objects misplaced (10.456%); 33 MiB/s, 535 objects/s recovering 2024-09-17T16:15:27.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:27.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:27.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 55083 55067 47.7962 50.75 0.00287075 0.0209128 2024-09-17T16:15:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:26 smithi017 ceph-mon[27571]: pgmap v582: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.1 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 47 MiB/s wr, 744 op/s; 1/154164 objects degraded (0.001%); 16119/154164 objects misplaced (10.456%); 33 MiB/s, 535 objects/s recovering 2024-09-17T16:15:27.256 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:27.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:27.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:27.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:27.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:26 smithi071 ceph-mon[30002]: pgmap v582: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.1 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 47 MiB/s wr, 744 op/s; 1/154164 objects degraded (0.001%); 16119/154164 objects misplaced (10.456%); 33 MiB/s, 535 objects/s recovering 2024-09-17T16:15:27.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:27.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:27.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:27.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:27.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:27.553 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:27.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:27.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:27.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:27.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:27.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:27.804 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:27.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:27.906 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:27.935 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:27.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:28.036 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:28.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:28.167 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:28.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 56071 56055 47.9873 61.75 0.0203932 0.0208329 2024-09-17T16:15:28.200 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:28.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:28.235 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:27 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/160521 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:28.302 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:28.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:27 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/160521 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:28.329 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:28.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:28.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:27 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/160521 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:28.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:28.462 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:28.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:28.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:28.594 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:28.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:28.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:28.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:28.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:28.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:28.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:28.856 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:28.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:28.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:28.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:28.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:29.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:29.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:29.130 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:29.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:29.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 56967 56951 48.0955 56 0.00662834 0.0207853 2024-09-17T16:15:29.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:29.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:28 smithi017 ceph-mon[27571]: pgmap v583: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 51 MiB/s wr, 823 op/s; 1/160521 objects degraded (0.001%); 14861/160521 objects misplaced (9.258%); 30 MiB/s, 483 objects/s recovering 2024-09-17T16:15:29.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:29.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:29.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:28 smithi071 ceph-mon[30002]: pgmap v583: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 51 MiB/s wr, 823 op/s; 1/160521 objects degraded (0.001%); 14861/160521 objects misplaced (9.258%); 30 MiB/s, 483 objects/s recovering 2024-09-17T16:15:29.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:29.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:29.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:29.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:28 smithi173 ceph-mon[30530]: pgmap v583: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 51 MiB/s wr, 823 op/s; 1/160521 objects degraded (0.001%); 14861/160521 objects misplaced (9.258%); 30 MiB/s, 483 objects/s recovering 2024-09-17T16:15:29.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:29.516 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:29.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:29.617 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:29.651 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:29.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:29.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:29.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:29.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:29.881 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:29.904 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:29.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:30.006 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:30.043 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:30.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:30.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:30.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:30.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:30.189 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 57887 57871 48.2208 57.5 0.0128014 0.0207319 2024-09-17T16:15:30.272 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:30.302 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:30.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:30.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:30.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:30.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:30.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:30.559 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:30.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:30.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:30.689 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:30.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:30.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:30.818 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:30.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:30.921 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:30.943 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:30.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:31.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:31.072 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:31.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:31.173 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:31.190 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 58718 58702 48.2697 51.9375 0.0117635 0.020705 2024-09-17T16:15:31.194 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:31.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:31.296 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:31.326 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:31.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:31.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:31.428 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:31.459 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:31.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:31.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:31.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:31.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:31 smithi173 ceph-mon[30530]: pgmap v584: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 54 MiB/s wr, 870 op/s; 1/165627 objects degraded (0.001%); 13689/165627 objects misplaced (8.265%); 34 MiB/s, 545 objects/s recovering 2024-09-17T16:15:31.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:31.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:31.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:31.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:31 smithi017 ceph-mon[27571]: pgmap v584: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 54 MiB/s wr, 870 op/s; 1/165627 objects degraded (0.001%); 13689/165627 objects misplaced (8.265%); 34 MiB/s, 545 objects/s recovering 2024-09-17T16:15:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:31 smithi071 ceph-mon[30002]: pgmap v584: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 54 MiB/s wr, 870 op/s; 1/165627 objects degraded (0.001%); 13689/165627 objects misplaced (8.265%); 34 MiB/s, 545 objects/s recovering 2024-09-17T16:15:31.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:31.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:31.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:31.947 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:31.979 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:31.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:32.081 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:32.103 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:32.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:32.190 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 59613 59597 48.3692 55.9375 0.211752 0.0206506 2024-09-17T16:15:32.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:32.239 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:32.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:32.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:32.341 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:32.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:32.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:32.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:32.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:32.494 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:32.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:32.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:32.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:32.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:32.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:32.759 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:32.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:32.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:32.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:32.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:32.989 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:33.014 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:33.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:33.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:33.139 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:33.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:33.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:33.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:33.190 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 60597 60581 48.5375 61.5 0.00321933 0.0205956 2024-09-17T16:15:33.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:33.267 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:33.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:33.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:33.370 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:33.394 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:33.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:33.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:33.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:33.523 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:33.616 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:33 smithi071 ceph-mon[30002]: pgmap v585: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 58 MiB/s wr, 927 op/s; 1/173145 objects degraded (0.001%); 12802/173145 objects misplaced (7.394%); 30 MiB/s, 476 objects/s recovering 2024-09-17T16:15:33.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:33.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:33 smithi173 ceph-mon[30530]: pgmap v585: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 58 MiB/s wr, 927 op/s; 1/173145 objects degraded (0.001%); 12802/173145 objects misplaced (7.394%); 30 MiB/s, 476 objects/s recovering 2024-09-17T16:15:33.655 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:33.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:33.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:33 smithi017 ceph-mon[27571]: pgmap v585: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 58 MiB/s wr, 927 op/s; 1/173145 objects degraded (0.001%); 12802/173145 objects misplaced (7.394%); 30 MiB/s, 476 objects/s recovering 2024-09-17T16:15:33.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:33.788 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:33.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:33.890 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:33.923 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:33.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:34.025 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:34.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:34.074 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:34.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:34.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:34.195 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 61488 61472 48.6279 55.6875 0.0201184 0.0205585 2024-09-17T16:15:34.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:34.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:34.309 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:34.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:34.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:34.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:34.464 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:34.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:34 smithi071 ceph-mon[30002]: osdmap e245: 12 total, 12 up, 11 in 2024-09-17T16:15:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:34 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/173145 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:34.566 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:34.594 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:34.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:34.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:34.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:34.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:34 smithi173 ceph-mon[30530]: osdmap e245: 12 total, 12 up, 11 in 2024-09-17T16:15:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:34 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/173145 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:34.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:34.726 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:34.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:34.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:34 smithi017 ceph-mon[27571]: osdmap e245: 12 total, 12 up, 11 in 2024-09-17T16:15:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:34 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/173145 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:34.793 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:34.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:34.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:34.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:34.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:34.991 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:34.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:35.093 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:35.115 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:35.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:35.118 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:15:35.190 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:15:35.189458+0000 min lat: 0.00101616 max lat: 15.0111 avg lat: 0.0205145 2024-09-17T16:15:35.190 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:15:35.190 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 62377 62361 48.7145 55.5625 0.122877 0.0205145 2024-09-17T16:15:35.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:35.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:35.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:35.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:35.375 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:35.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:35.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:35.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:35.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:35 smithi017 ceph-mon[27571]: pgmap v587: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 61 MiB/s wr, 981 op/s; 1/175653 objects degraded (0.001%); 11969/175653 objects misplaced (6.814%); 33 MiB/s, 533 objects/s recovering 2024-09-17T16:15:35.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:35.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:35 smithi071 ceph-mon[30002]: pgmap v587: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 61 MiB/s wr, 981 op/s; 1/175653 objects degraded (0.001%); 11969/175653 objects misplaced (6.814%); 33 MiB/s, 533 objects/s recovering 2024-09-17T16:15:35.603 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:35.633 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:35.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:35.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:35 smithi173 ceph-mon[30530]: pgmap v587: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 61 MiB/s wr, 981 op/s; 1/175653 objects degraded (0.001%); 11969/175653 objects misplaced (6.814%); 33 MiB/s, 533 objects/s recovering 2024-09-17T16:15:35.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:35.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:35.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:35.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:35.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:35.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:35.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:36.018 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:36.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:36.120 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:36.142 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:36.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:36.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:36.190 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 63449 63433 48.9402 67 0.00334705 0.0204112 2024-09-17T16:15:36.244 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:36.278 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:36.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:36.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:36.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:36.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:36.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:36.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:36.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:36.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:36.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:36.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:36.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:36.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:36.788 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:36.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:36.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:36.920 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:36.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:37.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:37.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:37.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:37.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:37.182 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:37.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:37.190 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 64416 64400 49.0804 60.4375 0.0429843 0.0203675 2024-09-17T16:15:37.283 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:37.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:37.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:37.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:37.465 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:37.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:37.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:37.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:37 smithi017 ceph-mon[27571]: pgmap v588: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 58 MiB/s wr, 920 op/s; 1/181800 objects degraded (0.001%); 11105/181800 objects misplaced (6.108%); 31 MiB/s, 501 objects/s recovering 2024-09-17T16:15:37.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4016785231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:37.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:37.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:37.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:37 smithi173 ceph-mon[30530]: pgmap v588: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 58 MiB/s wr, 920 op/s; 1/181800 objects degraded (0.001%); 11105/181800 objects misplaced (6.108%); 31 MiB/s, 501 objects/s recovering 2024-09-17T16:15:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4016785231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:37.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:37.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:37.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:37.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:37 smithi071 ceph-mon[30002]: pgmap v588: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 58 MiB/s wr, 920 op/s; 1/181800 objects degraded (0.001%); 11105/181800 objects misplaced (6.108%); 31 MiB/s, 501 objects/s recovering 2024-09-17T16:15:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4016785231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:37.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:37.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:37.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:37.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:37.981 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:37.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:38.079 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:15:38.079 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:15:38.079 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:15:38.079 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:15:38.079 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:15:38.079 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:15:38.080 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:15:38.081 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:15:38.081 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:15:38.081 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:15:38.081 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:15:38.081 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:15:38.081 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:15:38.081 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:15:38.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:38.116 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:38.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:38.190 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 65381 65365 49.2156 60.3125 0.00542961 0.0203137 2024-09-17T16:15:38.218 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:38.256 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:38.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:38.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:38.389 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:38.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:38.445 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:38.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:38.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:38.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:38.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:38.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:38 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:38.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:38 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:38.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:38.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:38.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:38 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:38 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:38.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:38.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:38.778 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:38 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:38 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:38.879 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:38.908 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:38.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:39.009 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:39.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:39.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:39.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:39.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:39.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:39.190 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 66413 66397 49.3975 64.5 0.00910465 0.0202388 2024-09-17T16:15:39.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:39.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:39.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:39.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:39.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:39 smithi017 ceph-mon[27571]: pgmap v589: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 57 MiB/s wr, 915 op/s; 1/188001 objects degraded (0.001%); 9820/188001 objects misplaced (5.223%); 31 MiB/s, 503 objects/s recovering 2024-09-17T16:15:39.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:39 smithi017 ceph-mon[27571]: osdmap e246: 12 total, 12 up, 11 in 2024-09-17T16:15:39.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:39 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/188001 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:39.436 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:39.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:39.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:39.570 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:39.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:39 smithi173 ceph-mon[30530]: pgmap v589: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 57 MiB/s wr, 915 op/s; 1/188001 objects degraded (0.001%); 9820/188001 objects misplaced (5.223%); 31 MiB/s, 503 objects/s recovering 2024-09-17T16:15:39.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:39 smithi173 ceph-mon[30530]: osdmap e246: 12 total, 12 up, 11 in 2024-09-17T16:15:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:39 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/188001 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:39.671 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:39.710 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:39.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:39.812 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:39.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:39 smithi071 ceph-mon[30002]: pgmap v589: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 57 MiB/s wr, 915 op/s; 1/188001 objects degraded (0.001%); 9820/188001 objects misplaced (5.223%); 31 MiB/s, 503 objects/s recovering 2024-09-17T16:15:39.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:39 smithi071 ceph-mon[30002]: osdmap e246: 12 total, 12 up, 11 in 2024-09-17T16:15:39.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:39 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/188001 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:39.841 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:39.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:39.943 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:39.972 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:39.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:40.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:40.100 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:40.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:40.101 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:40.191 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 67569 67553 49.6663 72.25 0.00837502 0.0201287 2024-09-17T16:15:40.202 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:40.258 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:40.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:40.263 INFO:teuthology.orchestra.run.smithi017.stdout:moved 0 / 51 (0%) 2024-09-17T16:15:40.263 INFO:teuthology.orchestra.run.smithi017.stdout:avg 4.63636 2024-09-17T16:15:40.263 INFO:teuthology.orchestra.run.smithi017.stdout:stddev 1.96666 -> 1.96666 (expected baseline 2.05302) 2024-09-17T16:15:40.263 INFO:teuthology.orchestra.run.smithi017.stdout:min osd.10 with 1 -> 1 pgs (0.215686 -> 0.215686 * mean) 2024-09-17T16:15:40.263 INFO:teuthology.orchestra.run.smithi017.stdout:max osd.4 with 7 -> 7 pgs (1.5098 -> 1.5098 * mean) 2024-09-17T16:15:40.263 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:15:40.264 INFO:teuthology.orchestra.run.smithi017.stdout:oload 1000 2024-09-17T16:15:40.264 INFO:teuthology.orchestra.run.smithi017.stdout:max_change 1 2024-09-17T16:15:40.264 INFO:teuthology.orchestra.run.smithi017.stdout:max_change_osds 4 2024-09-17T16:15:40.264 INFO:teuthology.orchestra.run.smithi017.stdout:average_utilization 0.0125 2024-09-17T16:15:40.264 INFO:teuthology.orchestra.run.smithi017.stdout:overload_utilization 0.1246 2024-09-17T16:15:40.264 INFO:teuthology.orchestra.run.smithi017.stderr:no change 2024-09-17T16:15:40.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:40.396 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:40.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:40.498 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:40.540 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:40.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:40.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:40.674 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:40.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:40.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:40.776 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:40.809 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:15:40.809 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:15:40.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:15:40.809 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:15:40.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.809 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25217 2.19 e121/114 2:9a6d6ce1:::benchmark_data_smithi173_55121_object25216:head v 121'1238, mlcod=121'1235)", 2024-09-17T16:15:40.809 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.777156+0000", 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 249.69814928700001, 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25947270300000003, 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777156+0000", 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777156+0000", 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777157+0000", 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7300000000000005e-07 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777182+0000", 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5307e-05 2024-09-17T16:15:40.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035767+0000", 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25858533500000003 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035781+0000", 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3567000000000001e-05 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035903+0000", 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012237899999999999 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035938+0000", 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4215999999999999e-05 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.036602+0000", 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00066465100000000004 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.036629+0000", 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6475e-05 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25228 2.1 e121/104 2:8352dd7d:::benchmark_data_smithi173_55121_object25227:head v 121'1242, mlcod=121'1239)", 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.786569+0000", 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 249.688736019, 2024-09-17T16:15:40.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27958702099999999, 2024-09-17T16:15:40.815 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.815 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.815 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.815 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786569+0000", 2024-09-17T16:15:40.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786569+0000", 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786570+0000", 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1399999999999997e-07 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786589+0000", 2024-09-17T16:15:40.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8754999999999999e-05 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065420+0000", 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.278831679 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065429+0000", 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4149999999999999e-06 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065499+0000", 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9944000000000007e-05 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065517+0000", 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8499e-05 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066120+0000", 2024-09-17T16:15:40.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060275699999999999 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066156+0000", 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6358e-05 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:29460 2.1 e128/104 2:83aee601:::benchmark_data_smithi173_55121_object29459:head v 128'1369, mlcod=128'1366)", 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.198210+0000", 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 231.27709498600001, 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31777874900000003, 2024-09-17T16:15:40.819 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198210+0000", 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198210+0000", 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198211+0000", 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198235+0000", 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3884999999999999e-05 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515223+0000", 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31698817499999998 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515231+0000", 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9189999999999999e-06 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515321+0000", 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9240000000000006e-05 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515344+0000", 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3362000000000001e-05 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515954+0000", 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000610291 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515989+0000", 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4758000000000002e-05 2024-09-17T16:15:40.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:29625 2.5 e129/104 2:a21e4127:::benchmark_data_smithi173_55121_object29624:head v 129'1363, mlcod=129'1360)", 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.564207+0000", 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 230.91109822499999, 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29265457499999997, 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564207+0000", 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564207+0000", 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564208+0000", 2024-09-17T16:15:40.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2399999999999998e-07 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564411+0000", 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020328800000000001 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856209+0000", 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29179771999999998 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856213+0000", 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0579999999999999e-06 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856281+0000", 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7911999999999996e-05 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856295+0000", 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4559000000000001e-05 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856830+0000", 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00053516399999999997 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856862+0000", 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1350000000000003e-05 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:76477 2.1 e149/104 2:8614c6c1:::benchmark_data_smithi173_55121_object76476:head v 149'2879, mlcod=149'2876)", 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:32.192527+0000", 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 188.28277836699999, 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23702707100000001, 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:32.192527+0000", 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:32.192527+0000", 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:32.192528+0000", 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.469e-06 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:32.192556+0000", 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7412000000000001e-05 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:32.428279+0000", 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23572337400000001 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:32.428287+0000", 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0849999999999997e-06 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:32.428341+0000", 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4055000000000003e-05 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:32.428361+0000", 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9828999999999999e-05 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:32.429516+0000", 2024-09-17T16:15:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011544979999999999 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:32.429554+0000", 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8349000000000002e-05 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:88644 2.8 2:10ace813:::benchmark_data_smithi173_55121_object12155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:43.950796+0000", 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 176.52450963300001, 2024-09-17T16:15:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31997915599999999, 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 88644 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950796+0000", 2024-09-17T16:15:40.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950796+0000", 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950797+0000", 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4079999999999999e-06 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950804+0000", 2024-09-17T16:15:40.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2490000000000004e-06 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950817+0000", 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.234e-05 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950823+0000", 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2199999999999997e-06 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950884+0000", 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0770000000000003e-05 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.951019+0000", 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134907 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.951145+0000", 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126663 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.952746+0000", 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016012859999999999 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.953782+0000", 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010357210000000001 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270185+0000", 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31640305000000002 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270223+0000", 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7607000000000001e-05 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270775+0000", 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00055193500000000003 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.839 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:88644 2.8 e157/155 ondisk, result = 0)", 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:43.952810+0000", 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 176.52249536100001, 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31796888800000001, 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.952810+0000", 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.952810+0000", 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.952811+0000", 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.85e-07 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.952815+0000", 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0740000000000003e-06 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270131+0000", 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31731594699999999 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270135+0000", 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.374e-06 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270176+0000", 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0914000000000002e-05 2024-09-17T16:15:40.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270182+0000", 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6659999999999996e-06 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270779+0000", 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00059712800000000005 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:88712 2.8 2:12896178:::benchmark_data_smithi173_55121_object12223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:44.153008+0000", 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 176.322297572, 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24794143799999999, 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 88712 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.153008+0000", 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.153008+0000", 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.153009+0000", 2024-09-17T16:15:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2750000000000001e-06 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.153014+0000", 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3530000000000002e-06 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.153025+0000", 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0983e-05 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.153030+0000", 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0209999999999999e-06 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.153091+0000", 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0887000000000002e-05 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.153192+0000", 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010060600000000001 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.153267+0000", 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4752000000000005e-05 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.154430+0000", 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011638149999999999 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.154823+0000", 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000392545 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.400084+0000", 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.245261326 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.400146+0000", 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2152999999999999e-05 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.400949+0000", 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080272199999999996 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:88712 2.8 e157/155 ondisk, result = 0)", 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:44.155114+0000", 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 176.320191626, 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24584331100000001, 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.155114+0000", 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.155114+0000", 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.155114+0000", 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6199999999999997e-07 2024-09-17T16:15:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.155118+0000", 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.721e-06 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.400009+0000", 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.244890733 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.400015+0000", 2024-09-17T16:15:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3110000000000002e-06 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.400067+0000", 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1724000000000001e-05 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.400077+0000", 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0608000000000001e-05 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.400957+0000", 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087955200000000002 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:88801 2.8 2:11218576:::benchmark_data_smithi173_55121_object12312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:44.267198+0000", 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 176.20810753800001, 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.237334829, 2024-09-17T16:15:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 88801 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.267198+0000", 2024-09-17T16:15:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.267198+0000", 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.267199+0000", 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2669999999999999e-06 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.267208+0000", 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.85e-06 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.267223+0000", 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4644000000000001e-05 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.267230+0000", 2024-09-17T16:15:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6890000000000004e-06 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.267283+0000", 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3108e-05 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.267437+0000", 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015375499999999999 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:15:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.267572+0000", 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013542799999999999 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.269052+0000", 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0014794369999999999 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.269345+0000", 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029278000000000003 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.503820+0000", 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23447547499999999 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.503881+0000", 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1038999999999996e-05 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.504533+0000", 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065135699999999998 2024-09-17T16:15:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:90287 2.8 2:13eb058f:::benchmark_data_smithi173_55121_object13798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:45.941984+0000", 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 174.53332153400001, 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25228158000000001, 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:40.862 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90287 2024-09-17T16:15:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.941984+0000", 2024-09-17T16:15:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.941984+0000", 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.941987+0000", 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4230000000000001e-06 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942001+0000", 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3647e-05 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942041+0000", 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9731999999999998e-05 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942064+0000", 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3604e-05 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942178+0000", 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113508 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942270+0000", 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2823000000000002e-05 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942366+0000", 2024-09-17T16:15:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5445999999999994e-05 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.943598+0000", 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001232089 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.944062+0000", 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046364399999999998 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.193517+0000", 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.249455344 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.193584+0000", 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7328000000000003e-05 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.194265+0000", 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00068099199999999995 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:90287 2.8 e158/155 ondisk, result = 0)", 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:45.945313+0000", 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 174.52999222, 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24895779500000001, 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.945313+0000", 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.945313+0000", 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.945314+0000", 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9299999999999997e-07 2024-09-17T16:15:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.945325+0000", 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1157e-05 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.193446+0000", 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.248121074 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.193453+0000", 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0779999999999998e-06 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.193497+0000", 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3810000000000002e-05 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.193508+0000", 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1032999999999999e-05 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.194271+0000", 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076294999999999998 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:94044 2.8 2:165f9939:::benchmark_data_smithi173_55121_object17555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:49.924944+0000", 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 170.55036154999999, 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35947178600000002, 2024-09-17T16:15:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 94044 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924944+0000", 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924944+0000", 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924946+0000", 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.21e-06 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924966+0000", 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9626000000000001e-05 2024-09-17T16:15:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924978+0000", 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.29e-05 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924987+0000", 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2930000000000001e-06 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.925037+0000", 2024-09-17T16:15:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0133999999999998e-05 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.925123+0000", 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6194999999999997e-05 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.925201+0000", 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8362999999999995e-05 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.927427+0000", 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022257879999999998 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.927457+0000", 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9961999999999999e-05 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.283840+0000", 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35638287000000002 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.283883+0000", 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2812999999999998e-05 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.284416+0000", 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00053263200000000001 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:94100 2.4 e160/119 2:21621a6c:::benchmark_data_smithi173_55121_object17611:head v 160'3435, mlcod=160'3433)", 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:50.106901+0000", 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 170.368404344, 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.250786487, 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.106901+0000", 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.106901+0000", 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.106902+0000", 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.362e-06 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.106909+0000", 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2400000000000004e-06 2024-09-17T16:15:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.355413+0000", 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24850399200000001 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.355419+0000", 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0650000000000004e-06 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.355486+0000", 2024-09-17T16:15:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7051000000000006e-05 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.355506+0000", 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0679e-05 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.357648+0000", 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021419719999999998 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.357687+0000", 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9125999999999998e-05 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:104873 2.8 2:108401c3:::benchmark_data_smithi173_55121_object28384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:01.984001+0000", 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 158.49130413099999, 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.247052939, 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 104873 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984001+0000", 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984001+0000", 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984004+0000", 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5009999999999999e-06 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984010+0000", 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9179999999999999e-06 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984026+0000", 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6096999999999999e-05 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984034+0000", 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5869999999999993e-06 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984086+0000", 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1382999999999999e-05 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984183+0000", 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7409000000000002e-05 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984265+0000", 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1751999999999999e-05 2024-09-17T16:15:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.987058+0000", 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027928660000000002 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.988233+0000", 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011750160000000001 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.230812+0000", 2024-09-17T16:15:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.242579195 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.230865+0000", 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2779999999999999e-05 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.231054+0000", 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000189435 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:104874 2.8 2:13fbf047:::benchmark_data_smithi173_55121_object28385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:01.984418+0000", 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 158.49088700499999, 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.254036761, 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 104874 2024-09-17T16:15:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984418+0000", 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984418+0000", 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984419+0000", 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7900000000000007e-07 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984427+0000", 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2029999999999992e-06 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984439+0000", 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1938e-05 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984446+0000", 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8979999999999999e-06 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.985736+0000", 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001289622 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.985889+0000", 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015309500000000001 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.986033+0000", 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014426799999999999 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.987137+0000", 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011038039999999999 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.991811+0000", 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046738090000000001 2024-09-17T16:15:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.237979+0000", 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.246168528 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.238037+0000", 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7225000000000002e-05 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.238455+0000", 2024-09-17T16:15:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00041839200000000002 2024-09-17T16:15:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(2.8 169/155 [PushReplyOp(2:121c2625:::benchmark_data_smithi173_55121_object58122:head)])", 2024-09-17T16:15:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:01.991708+0000", 2024-09-17T16:15:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 158.48359739599999, 2024-09-17T16:15:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.239050865, 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.991708+0000", 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.991708+0000", 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.991709+0000", 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.421e-06 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.991718+0000", 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0780000000000002e-06 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.229841+0000", 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23812249599999999 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.229849+0000", 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7039999999999994e-06 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.230210+0000", 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00036139400000000002 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.230759+0000", 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000548772 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:152973 2.19 e199/114 2:9e1f786b:::benchmark_data_smithi173_55121_object76484:head v 199'5124, mlcod=199'5121)", 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.867324+0000", 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 106.607981739, 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59753164299999995, 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867324+0000", 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867324+0000", 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867325+0000", 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.78e-07 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867331+0000", 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6239999999999996e-06 2024-09-17T16:15:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462706+0000", 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59537525499999999 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462713+0000", 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4019999999999999e-06 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462789+0000", 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6006000000000004e-05 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462816+0000", 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7087e-05 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.464821+0000", 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002004731 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.464855+0000", 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4560000000000001e-05 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:8110 3.b e220/208 3:de8160e2:::benchmark_data_smithi173_66272_object8109:head v 220'485, mlcod=220'481)", 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:14:34.926435+0000", 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 65.548870530000002, 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.293198399, 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926435+0000", 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926435+0000", 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926436+0000", 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.049e-06 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926456+0000", 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0636e-05 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219071+0000", 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29261461100000002 2024-09-17T16:15:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219077+0000", 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9880000000000001e-06 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219126+0000", 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8807000000000002e-05 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219142+0000", 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5846e-05 2024-09-17T16:15:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219593+0000", 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045157699999999999 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219633+0000", 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9885000000000002e-05 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:17931 3.6 e223/208 3:644b4ff5:::benchmark_data_smithi173_66272_object17930:head v 223'1147, mlcod=223'1144)", 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:14:41.818353+0000", 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.656952586000003, 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23997908900000001, 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:41.818353+0000", 2024-09-17T16:15:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:41.818353+0000", 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:41.818354+0000", 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6500000000000008e-07 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:41.818377+0000", 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3609e-05 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:42.057187+0000", 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23880932099999999 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:42.057190+0000", 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5039999999999998e-06 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:42.057297+0000", 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107329 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:42.057327+0000", 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9887e-05 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:42.058252+0000", 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092493000000000002 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:42.058332+0000", 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9543999999999996e-05 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:40.916 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:15:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:40.918 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:40.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:41.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:41.046 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:41.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:41.134 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:41.148 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:41.191 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 68761 68745 49.955 74.5 0.00576934 0.0200133 2024-09-17T16:15:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:41.196 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:41.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:41.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:41.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:41.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:41.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:41.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:41.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:41.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:41.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:41.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:41.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:15:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:41 smithi173 ceph-mon[30530]: pgmap v591: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 53 MiB/s wr, 850 op/s; 1/193560 objects degraded (0.001%); 8274/193560 objects misplaced (4.275%); 35 MiB/s, 565 objects/s recovering 2024-09-17T16:15:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:41 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:41 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:41 smithi173 ceph-mon[30530]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:41 smithi173 ceph-mon[30530]: osdmap e247: 12 total, 12 up, 11 in 2024-09-17T16:15:41.672 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:41 smithi017 ceph-mon[27571]: pgmap v591: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 53 MiB/s wr, 850 op/s; 1/193560 objects degraded (0.001%); 8274/193560 objects misplaced (4.275%); 35 MiB/s, 565 objects/s recovering 2024-09-17T16:15:41.673 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:41 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:41.673 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:41 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:41.673 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:41 smithi017 ceph-mon[27571]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:41.673 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:41 smithi017 ceph-mon[27571]: osdmap e247: 12 total, 12 up, 11 in 2024-09-17T16:15:41.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:41.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:41.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:41.809 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:41 smithi071 ceph-mon[30002]: pgmap v591: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 53 MiB/s wr, 850 op/s; 1/193560 objects degraded (0.001%); 8274/193560 objects misplaced (4.275%); 35 MiB/s, 565 objects/s recovering 2024-09-17T16:15:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:41 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:41 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:41 smithi071 ceph-mon[30002]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:41 smithi071 ceph-mon[30002]: osdmap e247: 12 total, 12 up, 11 in 2024-09-17T16:15:41.837 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:41.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:41.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:41.961 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:41.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:42.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:15:42.063 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:42.123 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:42.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:42.191 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 69800 69784 50.1271 64.9375 0.00462074 0.0199446 2024-09-17T16:15:42.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:42.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:42.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:42.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:42.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:42.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:42.479 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:42.505 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:42.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:42.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:42.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:42.640 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:42.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:42.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:42.770 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:42.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:42.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:42.896 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:42.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:42.904 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-17T16:15:42.904 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-17T16:15:42.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:43.027 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:43.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:43.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:43.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:43.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:43.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:43.155 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:43.191 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 70806 70790 50.2719 62.875 0.0095199 0.0198843 2024-09-17T16:15:43.225 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:43.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:43.277 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:43.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:43.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:43.410 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:43.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:43.513 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:43.542 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:43.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:43.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:43 smithi173 ceph-mon[30530]: pgmap v593: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s; 1/202440 objects degraded (0.000%); 7099/202440 objects misplaced (3.507%); 38 MiB/s, 608 objects/s recovering 2024-09-17T16:15:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:43 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:43 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:43.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:43.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:43.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:43 smithi017 ceph-mon[27571]: pgmap v593: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s; 1/202440 objects degraded (0.000%); 7099/202440 objects misplaced (3.507%); 38 MiB/s, 608 objects/s recovering 2024-09-17T16:15:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:43 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:43 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:43.772 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:43.802 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:43.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:43.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:43 smithi071 ceph-mon[30002]: pgmap v593: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 70 MiB/s wr, 1.11k op/s; 1/202440 objects degraded (0.000%); 7099/202440 objects misplaced (3.507%); 38 MiB/s, 608 objects/s recovering 2024-09-17T16:15:43.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:43 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:43.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:43 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:43.902 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:43.935 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:43.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:44.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:44.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:44.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:44.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:44.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:44.191 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 71539 71523 50.2218 45.8125 0.00911363 0.0199068 2024-09-17T16:15:44.200 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:44.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:44.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:44.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:44.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:44.343 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:15:44.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:44.460 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:44.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:44.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:44.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:44.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:44 smithi173 ceph-mon[30530]: osdmap e248: 12 total, 12 up, 11 in 2024-09-17T16:15:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/135111634' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:44 smithi173 ceph-mon[30530]: pgmap v595: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 61 MiB/s wr, 971 op/s; 1/205488 objects degraded (0.000%); 6243/205488 objects misplaced (3.038%); 37 MiB/s, 596 objects/s recovering 2024-09-17T16:15:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:44 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/202440 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:44 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:15:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:44 smithi173 ceph-mon[30530]: osdmap e249: 12 total, 12 up, 11 in 2024-09-17T16:15:44.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:44.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:44.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:44.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:44.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:44 smithi017 ceph-mon[27571]: osdmap e248: 12 total, 12 up, 11 in 2024-09-17T16:15:44.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/135111634' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:44 smithi017 ceph-mon[27571]: pgmap v595: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 61 MiB/s wr, 971 op/s; 1/205488 objects degraded (0.000%); 6243/205488 objects misplaced (3.038%); 37 MiB/s, 596 objects/s recovering 2024-09-17T16:15:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:44 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/202440 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:44 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:15:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:44 smithi017 ceph-mon[27571]: osdmap e249: 12 total, 12 up, 11 in 2024-09-17T16:15:44.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:44.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:44 smithi071 ceph-mon[30002]: osdmap e248: 12 total, 12 up, 11 in 2024-09-17T16:15:44.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/135111634' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:15:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:44 smithi071 ceph-mon[30002]: pgmap v595: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 10 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 61 MiB/s wr, 971 op/s; 1/205488 objects degraded (0.000%); 6243/205488 objects misplaced (3.038%); 37 MiB/s, 596 objects/s recovering 2024-09-17T16:15:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:44 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/202440 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:44 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:15:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:44 smithi071 ceph-mon[30002]: osdmap e249: 12 total, 12 up, 11 in 2024-09-17T16:15:44.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:44.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:44.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:44.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:44.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:44.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:45.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:45.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:45.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:45.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:45.191 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 16 71953 71937 49.9513 25.875 0.00253987 0.0198959 2024-09-17T16:15:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:45.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:45.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:45.304 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:45.328 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:45.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:45.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:45.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:45.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:45.554 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:45.584 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:45.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:45.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:45.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:45.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:45.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:45.837 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:45.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:45.911 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:15:45.911 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:15:45.939 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:45.963 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:45.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:45.984 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:15:45.985 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:15:45.985 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:15:45.985 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:15:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:15:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:15:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:15:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:15:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:15:45.986 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:15:45.986 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:15:45.986 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:15:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:15:45.986 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:15:45.987 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:15:45.987 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:15:46.011 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:15:46.011 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:15:46.033 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:15:46.034 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:15:46.058 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:15:46.059 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:15:46.065 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:46.082 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:15:46.083 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:15:46.089 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:46.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:46.191 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.1934 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 71953 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 49.8602 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 37.105 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 112.938 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 797 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 593.707 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 1807 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 0 2024-09-17T16:15:46.192 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0200493 2024-09-17T16:15:46.193 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.262948 2024-09-17T16:15:46.193 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 15.0111 2024-09-17T16:15:46.193 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00101616 2024-09-17T16:15:46.202 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T16:15:46.222 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:46.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:46.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:46.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:46.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:46.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:46.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:46.478 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:46.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:46.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:46.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:46.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:46.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:46 smithi173 ceph-mon[30530]: osdmap e250: 12 total, 12 up, 11 in 2024-09-17T16:15:46.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2011808295' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:46.704 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:46.727 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:46.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:46.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:46 smithi017 ceph-mon[27571]: osdmap e250: 12 total, 12 up, 11 in 2024-09-17T16:15:46.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2011808295' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:46 smithi071 ceph-mon[30002]: osdmap e250: 12 total, 12 up, 11 in 2024-09-17T16:15:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2011808295' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:46.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:46.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:46.854 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:46.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:46.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:46.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:47.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:15:47.079 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:47.144 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:47.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:47.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:47.279 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:47.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:47.381 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:47.416 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:47.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:47.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:47.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:47.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:47.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:47 smithi173 ceph-mon[30530]: pgmap v598: 17 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 10 active+clean; 4.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 1/211569 objects degraded (0.000%); 4901/211569 objects misplaced (2.317%); 39 MiB/s, 623 objects/s recovering 2024-09-17T16:15:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:47 smithi173 ceph-mon[30530]: osdmap e251: 12 total, 12 up, 11 in 2024-09-17T16:15:47.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:47.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:47.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:47 smithi017 ceph-mon[27571]: pgmap v598: 17 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 10 active+clean; 4.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 1/211569 objects degraded (0.000%); 4901/211569 objects misplaced (2.317%); 39 MiB/s, 623 objects/s recovering 2024-09-17T16:15:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:47 smithi017 ceph-mon[27571]: osdmap e251: 12 total, 12 up, 11 in 2024-09-17T16:15:47.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:47.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:47.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:47.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:47 smithi071 ceph-mon[30002]: pgmap v598: 17 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 10 active+clean; 4.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 1/211569 objects degraded (0.000%); 4901/211569 objects misplaced (2.317%); 39 MiB/s, 623 objects/s recovering 2024-09-17T16:15:47.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:47 smithi071 ceph-mon[30002]: osdmap e251: 12 total, 12 up, 11 in 2024-09-17T16:15:47.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:47.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:47.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:48.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:48.065 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:48.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:48.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:48.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:48.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:48.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:48.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:48.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:48.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:48.391 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:48.398 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:48.416 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:48.441 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:48.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:48.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:48.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:48.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:48.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:48.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:48.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:48.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:48.817 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:48.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:48.919 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:48.953 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:48.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:49.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:49.080 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:49.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:49.183 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:49.208 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:49.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:49.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:49.326 INFO:teuthology.orchestra.run.smithi017.stdout:moved 0 / 51 (0%) 2024-09-17T16:15:49.326 INFO:teuthology.orchestra.run.smithi017.stdout:avg 4.63636 2024-09-17T16:15:49.326 INFO:teuthology.orchestra.run.smithi017.stdout:stddev 1.96666 -> 1.96666 (expected baseline 2.05302) 2024-09-17T16:15:49.326 INFO:teuthology.orchestra.run.smithi017.stdout:min osd.10 with 1 -> 1 pgs (0.215686 -> 0.215686 * mean) 2024-09-17T16:15:49.326 INFO:teuthology.orchestra.run.smithi017.stdout:max osd.4 with 7 -> 7 pgs (1.5098 -> 1.5098 * mean) 2024-09-17T16:15:49.326 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:15:49.326 INFO:teuthology.orchestra.run.smithi017.stdout:oload 1000 2024-09-17T16:15:49.326 INFO:teuthology.orchestra.run.smithi017.stdout:max_change 3 2024-09-17T16:15:49.326 INFO:teuthology.orchestra.run.smithi017.stdout:max_change_osds 4 2024-09-17T16:15:49.327 INFO:teuthology.orchestra.run.smithi017.stdout:average_utilization 0.0145 2024-09-17T16:15:49.327 INFO:teuthology.orchestra.run.smithi017.stdout:overload_utilization 0.1446 2024-09-17T16:15:49.327 INFO:teuthology.orchestra.run.smithi017.stderr:no change 2024-09-17T16:15:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:49.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:49.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:49.338 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:15:49.339 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:15:49.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:49.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:49.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:49 smithi017 ceph-mon[27571]: pgmap v600: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 10 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 46 MiB/s wr, 777 op/s; 1/214710 objects degraded (0.000%); 2530/214710 objects misplaced (1.178%); 53 MiB/s, 842 objects/s recovering 2024-09-17T16:15:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:49 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/214710 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:49.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:49 smithi017 ceph-mon[27571]: osdmap e252: 12 total, 12 up, 11 in 2024-09-17T16:15:49.561 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:49.586 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:49.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:49.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:49.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:49 smithi173 ceph-mon[30530]: pgmap v600: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 10 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 46 MiB/s wr, 777 op/s; 1/214710 objects degraded (0.000%); 2530/214710 objects misplaced (1.178%); 53 MiB/s, 842 objects/s recovering 2024-09-17T16:15:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:49 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/214710 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:49 smithi173 ceph-mon[30530]: osdmap e252: 12 total, 12 up, 11 in 2024-09-17T16:15:49.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:49.709 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:49.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:49.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:49.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:49 smithi071 ceph-mon[30002]: pgmap v600: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 10 active+clean; 4.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 46 MiB/s wr, 777 op/s; 1/214710 objects degraded (0.000%); 2530/214710 objects misplaced (1.178%); 53 MiB/s, 842 objects/s recovering 2024-09-17T16:15:49.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:49 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/214710 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:15:49.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:49 smithi071 ceph-mon[30002]: osdmap e252: 12 total, 12 up, 11 in 2024-09-17T16:15:49.832 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:49.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:49.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:15:49.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:49.955 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:49.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:50.057 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:50.079 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:50.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:50.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:50.204 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:50.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:50.306 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:50.328 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:50.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:50.360 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-17T16:15:50.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-09-17T16:15:50.386 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:15:50.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:50.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:50.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:50.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:50.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:50 smithi173 ceph-mon[30530]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:50 smithi173 ceph-mon[30530]: osdmap e253: 12 total, 12 up, 11 in 2024-09-17T16:15:50.699 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:50.721 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:50.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:50 smithi017 ceph-mon[27571]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:50 smithi017 ceph-mon[27571]: osdmap e253: 12 total, 12 up, 11 in 2024-09-17T16:15:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:50 smithi071 ceph-mon[30002]: from='client.25264 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:50 smithi071 ceph-mon[30002]: osdmap e253: 12 total, 12 up, 11 in 2024-09-17T16:15:50.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:50.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:50.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:50.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:50.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:50.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:51.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:51.093 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:51.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:51.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:51.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:51.216 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:51.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:51.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:51.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:51.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:51.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:51.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:51.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:51.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:51.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:51.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:51.596 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:51.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:51.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:51 smithi017 ceph-mon[27571]: pgmap v602: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 375 B/s rd, 40 MiB/s wr, 859 op/s; 1/212205 objects degraded (0.000%); 2213/212205 objects misplaced (1.043%); 46 MiB/s, 739 objects/s recovering 2024-09-17T16:15:51.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:51 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:51.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:51 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:51.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:51 smithi017 ceph-mon[27571]: osdmap e254: 12 total, 12 up, 11 in 2024-09-17T16:15:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:51 smithi173 ceph-mon[30530]: pgmap v602: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 375 B/s rd, 40 MiB/s wr, 859 op/s; 1/212205 objects degraded (0.000%); 2213/212205 objects misplaced (1.043%); 46 MiB/s, 739 objects/s recovering 2024-09-17T16:15:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:51 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:51 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:51 smithi173 ceph-mon[30530]: osdmap e254: 12 total, 12 up, 11 in 2024-09-17T16:15:51.654 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:15:51.697 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:51.730 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:51.748 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:51.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:51.763 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:51.790 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:51.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:51 smithi071 ceph-mon[30002]: pgmap v602: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 375 B/s rd, 40 MiB/s wr, 859 op/s; 1/212205 objects degraded (0.000%); 2213/212205 objects misplaced (1.043%); 46 MiB/s, 739 objects/s recovering 2024-09-17T16:15:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:51 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:51 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:51 smithi071 ceph-mon[30002]: osdmap e254: 12 total, 12 up, 11 in 2024-09-17T16:15:51.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:51.876 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:51.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:51.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:52.004 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:52.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:52.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:52.131 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:52.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:52.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:52.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:52.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:52.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:52.387 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:52.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:52.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:52 smithi017 ceph-mon[27571]: pgmap v605: 17 pgs: 1 peering, 1 active+recovering+undersized+remapped, 15 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 375 B/s rd, 10 MiB/s wr, 740 op/s; 1162/206355 objects misplaced (0.563%); 36 MiB/s, 577 objects/s recovering 2024-09-17T16:15:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:52 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/212205 objects degraded (0.000%), 1 pg degraded) 2024-09-17T16:15:52.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:52.516 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:52.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:52.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:52.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:52.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:52.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:52.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:52.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:52.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:52.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:52 smithi071 ceph-mon[30002]: pgmap v605: 17 pgs: 1 peering, 1 active+recovering+undersized+remapped, 15 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 375 B/s rd, 10 MiB/s wr, 740 op/s; 1162/206355 objects misplaced (0.563%); 36 MiB/s, 577 objects/s recovering 2024-09-17T16:15:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:52 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/212205 objects degraded (0.000%), 1 pg degraded) 2024-09-17T16:15:52.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:52.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:52.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:52.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:52 smithi173 ceph-mon[30530]: pgmap v605: 17 pgs: 1 peering, 1 active+recovering+undersized+remapped, 15 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 375 B/s rd, 10 MiB/s wr, 740 op/s; 1162/206355 objects misplaced (0.563%); 36 MiB/s, 577 objects/s recovering 2024-09-17T16:15:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:52 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/212205 objects degraded (0.000%), 1 pg degraded) 2024-09-17T16:15:52.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:53.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:53.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:53.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:53.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:53.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:53.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:53.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:53.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:53.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:53.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:53.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:53.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:53.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:53.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:53.668 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:53.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:53.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:53.798 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:53.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:53.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:53.930 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:53.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:53.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:54.032 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:54.059 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:54.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:54.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:54.161 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:54.170 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-17T16:15:54.167+0000 7f0e62ffd640 0 --2- 172.21.15.17:0/655641074 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f0e64103b00 0x7f0e64103ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T16:15:54.191 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:54.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:54.217 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:15:54.217 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:15:54.252 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:15:54.252 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:15:54.282 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:15:54.282 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:15:54.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:54.312 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:15:54.312 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:15:54.321 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:54.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:54.336 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:15:54.336 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:15:54.371 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:15:54.372 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:15:54.405 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:15:54.406 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:15:54.423 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:54.439 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:15:54.440 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:15:54.452 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:54.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:54.471 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:15:54.472 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:15:54.503 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:15:54.503 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:15:54.537 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:15:54.537 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:15:54.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:54.561 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:15:54.562 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:15:54.582 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:54.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:54.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:54.709 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:54.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:54.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:54.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:54.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:54.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:54.970 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:54.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:54 smithi017 ceph-mon[27571]: pgmap v606: 17 pgs: 1 active+recovering+undersized+remapped, 1 peering, 15 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 341 B/s rd, 2.2 MiB/s wr, 651 op/s; 708/204240 objects misplaced (0.347%); 19 MiB/s, 303 objects/s recovering 2024-09-17T16:15:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/828054903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:55.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:55.103 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:55.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:55.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:54 smithi173 ceph-mon[30530]: pgmap v606: 17 pgs: 1 active+recovering+undersized+remapped, 1 peering, 15 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 341 B/s rd, 2.2 MiB/s wr, 651 op/s; 708/204240 objects misplaced (0.347%); 19 MiB/s, 303 objects/s recovering 2024-09-17T16:15:55.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/828054903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:55.205 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:55.237 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:55.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:55.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:54 smithi071 ceph-mon[30002]: pgmap v606: 17 pgs: 1 active+recovering+undersized+remapped, 1 peering, 15 active+clean; 4.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 341 B/s rd, 2.2 MiB/s wr, 651 op/s; 708/204240 objects misplaced (0.347%); 19 MiB/s, 303 objects/s recovering 2024-09-17T16:15:55.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/828054903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:15:55.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:55.366 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:55.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:55.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:55.495 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:55.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:55.597 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:55.628 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:55.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:55.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:55.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:55.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:55.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:55.850 INFO:teuthology.orchestra.run.smithi017.stdout:moved 2 / 51 (3.92157%) 2024-09-17T16:15:55.850 INFO:teuthology.orchestra.run.smithi017.stdout:avg 4.63636 2024-09-17T16:15:55.850 INFO:teuthology.orchestra.run.smithi017.stdout:stddev 1.96666 -> 1.91988 (expected baseline 2.05302) 2024-09-17T16:15:55.850 INFO:teuthology.orchestra.run.smithi017.stdout:min osd.10 with 1 -> 1 pgs (0.215686 -> 0.215686 * mean) 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout:max osd.4 with 7 -> 6 pgs (1.5098 -> 1.29412 * mean) 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout:oload 110 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout:max_change 0.05 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout:max_change_osds 4 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout:average_utilization 0.0140 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout:overload_utilization 0.0155 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 weight 1.0000 -> 0.9500 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9 weight 1.0000 -> 0.9500 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 weight 1.0000 -> 0.9500 2024-09-17T16:15:55.851 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-17T16:15:55.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:55.859 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:15:55.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:55.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:55.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:15:56.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:56.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:15:56.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:15:56.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:56.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:15:56.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:55 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:15:56.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:55 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:15:56.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3017156045' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:56.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:55 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:56.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260','4':'62260','9':'62260'}"} : dispatch 2024-09-17T16:15:56.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:15:56.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:55 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:15:56.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:55 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:15:56.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3017156045' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:56.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:55 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:56.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260','4':'62260','9':'62260'}"} : dispatch 2024-09-17T16:15:56.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:56.250 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:15:56.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:55 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:15:56.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:55 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:15:56.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3017156045' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:56.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:55 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:15:56.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260','4':'62260','9':'62260'}"} : dispatch 2024-09-17T16:15:56.350 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:56.372 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:56.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:56.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:56.497 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:56.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:56.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:56.621 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:56.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:56.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:56.745 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:56.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:56.762 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-17T16:15:56.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-17T16:15:56.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:56.867 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:56.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:56.970 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:56.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:56 smithi017 ceph-mon[27571]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:56 smithi017 ceph-mon[27571]: pgmap v607: 17 pgs: 1 active+recovering+undersized+remapped, 1 peering, 15 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 486 KiB/s wr, 885 op/s; 708/197655 objects misplaced (0.358%); 15 MiB/s, 233 objects/s recovering 2024-09-17T16:15:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:56 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:56 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:15:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260','4':'62260','9':'62260'}"}]': finished 2024-09-17T16:15:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:56 smithi017 ceph-mon[27571]: osdmap e255: 12 total, 12 up, 11 in 2024-09-17T16:15:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:56 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:15:56.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:56.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:57.014 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:57.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:57.117 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:57.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:57.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:56 smithi173 ceph-mon[30530]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:57.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:56 smithi173 ceph-mon[30530]: pgmap v607: 17 pgs: 1 active+recovering+undersized+remapped, 1 peering, 15 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 486 KiB/s wr, 885 op/s; 708/197655 objects misplaced (0.358%); 15 MiB/s, 233 objects/s recovering 2024-09-17T16:15:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:56 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:56 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:15:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260','4':'62260','9':'62260'}"}]': finished 2024-09-17T16:15:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:56 smithi173 ceph-mon[30530]: osdmap e255: 12 total, 12 up, 11 in 2024-09-17T16:15:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:56 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:15:57.218 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:57.240 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:57.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:57.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:57.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:56 smithi071 ceph-mon[30002]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:15:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:56 smithi071 ceph-mon[30002]: pgmap v607: 17 pgs: 1 active+recovering+undersized+remapped, 1 peering, 15 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 486 KiB/s wr, 885 op/s; 708/197655 objects misplaced (0.358%); 15 MiB/s, 233 objects/s recovering 2024-09-17T16:15:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:56 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:15:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:56 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:15:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','3':'62260','4':'62260','9':'62260'}"}]': finished 2024-09-17T16:15:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:56 smithi071 ceph-mon[30002]: osdmap e255: 12 total, 12 up, 11 in 2024-09-17T16:15:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:56 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:15:57.343 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:57.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:57.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:57.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:57.488 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:57.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:57.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:57.590 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:15:57.611 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:57.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:57.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:15:57.713 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:57.734 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:57.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:57.824 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:57.836 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:57.857 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:57.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:57.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:57.984 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:57.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:58.020 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:15:58.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:58.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:57 smithi017 ceph-mon[27571]: osdmap e256: 12 total, 12 up, 11 in 2024-09-17T16:15:58.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:15:58.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:15:58.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:15:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:58.120 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:58.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:58.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:57 smithi173 ceph-mon[30530]: osdmap e256: 12 total, 12 up, 11 in 2024-09-17T16:15:58.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:57 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:15:58.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:57 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:15:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:57 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:15:58.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:58.258 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:58.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:58.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:57 smithi071 ceph-mon[30002]: osdmap e256: 12 total, 12 up, 11 in 2024-09-17T16:15:58.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:57 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:15:58.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:57 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:15:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:57 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:15:58.360 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:58.420 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:58.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:58.522 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:15:58.555 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:58.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:15:58.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:58.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:58.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:58.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:58.822 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:58.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:58.892 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:15:58.892 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:15:58.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:15:58.892 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:06:08.079913+0000", 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 590.80825149600003, 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86337366999999998, 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.14158", 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.17:0/2484390953", 2024-09-17T16:15:58.893 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 1 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079913+0000", 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079913+0000", 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079915+0000", 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0330000000000001e-06 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079920+0000", 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5210000000000002e-06 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079929+0000", 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4490000000000005e-06 2024-09-17T16:15:58.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.079939+0000", 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3519999999999996e-06 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.080004+0000", 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5324e-05 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:15:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.080011+0000", 2024-09-17T16:15:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5300000000000002e-06 2024-09-17T16:15:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.942061+0000", 2024-09-17T16:15:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86205003199999997 2024-09-17T16:15:58.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.942214+0000", 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000153287 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.942381+0000", 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016673 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:58.898 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943022+0000", 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00064124199999999998 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943096+0000", 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3516999999999994e-05 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943190+0000", 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4339000000000006e-05 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943247+0000", 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6779e-05 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:06:08.943287+0000", 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0534999999999997e-05 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:13527 2.19 2:9902b078:::benchmark_data_smithi173_55121_object13526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.335470+0000", 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 286.552694573, 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0334308889999999, 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 13527 2024-09-17T16:15:58.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335470+0000", 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335470+0000", 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335471+0000", 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.89e-07 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335500+0000", 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8971000000000001e-05 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335507+0000", 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.401e-06 2024-09-17T16:15:58.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335513+0000", 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.366e-06 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.362278+0000", 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.026764929999999999 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.362286+0000", 2024-09-17T16:15:58.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.038e-06 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.368810+0000", 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.006524102 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.368901+0000", 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1592e-05 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.905 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:13539 2.14 2:295cc8f1:::benchmark_data_smithi173_55121_object13538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.759315+0000", 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 286.128849516, 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.60567000500000001, 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 13539 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.906 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759315+0000", 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759315+0000", 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759324+0000", 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.14e-06 2024-09-17T16:15:58.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759364+0000", 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0513999999999999e-05 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759390+0000", 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6265e-05 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759403+0000", 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3088000000000001e-05 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759483+0000", 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9049999999999997e-05 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759492+0000", 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7119999999999995e-06 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.360226+0000", 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.60073335000000005 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.360357+0000", 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013103999999999999 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.360500+0000", 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014305899999999999 2024-09-17T16:15:58.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364416+0000", 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039165040000000003 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364455+0000", 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9304000000000002e-05 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364915+0000", 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045950800000000002 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364954+0000", 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8639000000000001e-05 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364985+0000", 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1832000000000003e-05 2024-09-17T16:15:58.913 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(2.11 116/107 [PushOp(2:88e52e2e:::benchmark_data_smithi173_55121_object4621:head, version: 100'307, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:88e52e2e:::benchmark_data_smithi173_55121_object4621:head@100'307, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:25.435893+0000", 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 273.45227159500001, 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31609990900000001, 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.435893+0000", 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.435893+0000", 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.435894+0000", 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2999999999999999e-07 2024-09-17T16:15:58.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.435925+0000", 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0345000000000002e-05 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.751343+0000", 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31541794899999998 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.751359+0000", 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6339000000000001e-05 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.751462+0000", 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000102836 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.751477+0000", 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5628e-05 2024-09-17T16:15:58.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.751993+0000", 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051588199999999995 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25184 2.9 2:915f94f2:::benchmark_data_smithi173_55121_object25183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.745700+0000", 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 268.14246459100002, 2024-09-17T16:15:58.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31523103000000002, 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25184 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745700+0000", 2024-09-17T16:15:58.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745700+0000", 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745702+0000", 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.64e-06 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745748+0000", 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6217000000000002e-05 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745760+0000", 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1348999999999999e-05 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745765+0000", 2024-09-17T16:15:58.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100000000000003e-06 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745808+0000", 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2104000000000002e-05 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745895+0000", 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7670000000000006e-05 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.745986+0000", 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1096999999999999e-05 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.746541+0000", 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00055457599999999998 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.747059+0000", 2024-09-17T16:15:58.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051780499999999998 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060862+0000", 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31380370299999999 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060899+0000", 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6366999999999999e-05 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060931+0000", 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2691999999999998e-05 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25215 2.10 e121/111 2:0e4d5290:::benchmark_data_smithi173_55121_object25214:head v 121'1162, mlcod=121'1158)", 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.775819+0000", 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 268.11234566899998, 2024-09-17T16:15:58.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44938945499999999, 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775819+0000", 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775819+0000", 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775820+0000", 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775853+0000", 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3364000000000001e-05 2024-09-17T16:15:58.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224477+0000", 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44862400299999999 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224483+0000", 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9730000000000002e-06 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224548+0000", 2024-09-17T16:15:58.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4199e-05 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224566+0000", 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8686999999999999e-05 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.225168+0000", 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060204500000000005 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.225209+0000", 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0392999999999998e-05 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25240 2.9 2:9323c63b:::benchmark_data_smithi173_55121_object25239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.803581+0000", 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 268.08458363300002, 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.437608569, 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.930 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25240 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803581+0000", 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803581+0000", 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803583+0000", 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.265e-06 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803637+0000", 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4765999999999997e-05 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803649+0000", 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1304999999999999e-05 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803655+0000", 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2899999999999999e-06 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803708+0000", 2024-09-17T16:15:58.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2627999999999998e-05 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803793+0000", 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5376000000000004e-05 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803906+0000", 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011330100000000001 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804549+0000", 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00064244400000000004 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804821+0000", 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027272499999999997 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.241122+0000", 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43630034000000001 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.241159+0000", 2024-09-17T16:15:58.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7251e-05 2024-09-17T16:15:58.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.936 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:15:58.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.241190+0000", 2024-09-17T16:15:58.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0877999999999998e-05 2024-09-17T16:15:58.937 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.937 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.937 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.937 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:28853 2.2 e128/104 2:4387b46d:::benchmark_data_smithi173_55121_object28852:head v 128'1332, mlcod=128'1329)", 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:48.540329+0000", 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 250.34783564200001, 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.364975192, 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540329+0000", 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540329+0000", 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540330+0000", 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3300000000000002e-07 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540380+0000", 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.02e-05 2024-09-17T16:15:58.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904638+0000", 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36425802000000002 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904645+0000", 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6579999999999996e-06 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904700+0000", 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4855999999999998e-05 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904714+0000", 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3902e-05 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.905261+0000", 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054778599999999998 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.905305+0000", 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3136999999999997e-05 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:29877 2.2 e129/104 2:40729eb6:::benchmark_data_smithi173_55121_object29876:head v 129'1356, mlcod=129'1354)", 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 248.68397418500001, 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.87425371299999999, 2024-09-17T16:15:58.942 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5700000000000002e-07 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204236+0000", 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4901999999999998e-05 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077424+0000", 2024-09-17T16:15:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.87318755800000003 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077440+0000", 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5753999999999999e-05 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077544+0000", 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104652 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077590+0000", 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5359999999999999e-05 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.078411+0000", 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082190400000000004 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.078445+0000", 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3025999999999999e-05 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.946 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(2.9 158/155 [PushReplyOp(2:9441bd8f:::benchmark_data_smithi173_55121_object49095:head)])", 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:45.842362+0000", 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 193.04580249200001, 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32133242099999998, 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.842362+0000", 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.842362+0000", 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.842363+0000", 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5099999999999999e-07 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.842368+0000", 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1229999999999999e-06 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.162585+0000", 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32021638200000002 2024-09-17T16:15:58.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.162593+0000", 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2369999999999992e-06 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.162702+0000", 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108522 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.163695+0000", 2024-09-17T16:15:58.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099340600000000006 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:90245 2.9 2:947e1be4:::benchmark_data_smithi173_55121_object13756:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:45.848474+0000", 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 193.039690734, 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31939979400000001, 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:58.951 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90245 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848474+0000", 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848474+0000", 2024-09-17T16:15:58.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848476+0000", 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9719999999999999e-06 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848484+0000", 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3429999999999998e-06 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848503+0000", 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9905999999999999e-05 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848513+0000", 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0023999999999999e-05 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848600+0000", 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6085e-05 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848753+0000", 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000153444 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.848909+0000", 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000155608 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.851197+0000", 2024-09-17T16:15:58.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022886439999999998 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.865002+0000", 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013805199000000001 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167298+0000", 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30229515499999998 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167350+0000", 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2330000000000002e-05 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167874+0000", 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00052408399999999996 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:90245 2.9 e158/155 ondisk, result = 0)", 2024-09-17T16:15:58.957 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:45.851236+0000", 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 193.03692936799999, 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31664304700000001, 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.851236+0000", 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.851236+0000", 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.851236+0000", 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8299999999999996e-07 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.851243+0000", 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1019999999999999e-06 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167214+0000", 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31597043200000002 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167221+0000", 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5340000000000002e-06 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167286+0000", 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4920999999999997e-05 2024-09-17T16:15:58.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167293+0000", 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8889999999999996e-06 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.167879+0000", 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00058548599999999997 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(2.9 158/155 [PushReplyOp(2:9287a5b2:::benchmark_data_smithi173_55121_object60862:head)])", 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:48.880360+0000", 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 190.007804964, 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38735041999999997, 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880360+0000", 2024-09-17T16:15:58.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880360+0000", 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880361+0000", 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3200000000000005e-07 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880365+0000", 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3189999999999996e-06 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267083+0000", 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38671788000000001 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267091+0000", 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8050000000000007e-06 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267182+0000", 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1217000000000004e-05 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267710+0000", 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00052856699999999997 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.965 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:95505 2.4 2:25f242c1:::benchmark_data_smithi173_55121_object19016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:51.639132+0000", 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 187.249032484, 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33947701800000002, 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 95505 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.966 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639132+0000", 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639132+0000", 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639134+0000", 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2589999999999999e-06 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639140+0000", 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1999999999999999e-06 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639152+0000", 2024-09-17T16:15:58.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2275e-05 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639159+0000", 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3759999999999996e-06 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639206+0000", 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7608000000000002e-05 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639295+0000", 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8560000000000006e-05 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.639383+0000", 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8293000000000003e-05 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.642147+0000", 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002763723 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.642168+0000", 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0795e-05 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978532+0000", 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33636466399999998 2024-09-17T16:15:58.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978575+0000", 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2593999999999999e-05 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978610+0000", 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4671000000000001e-05 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:95505 2.4 e162/119 ondisk, result = 0)", 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:51.641024+0000", 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 187.24714082099999, 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.337590153, 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.641024+0000", 2024-09-17T16:15:58.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.641024+0000", 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.641025+0000", 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8400000000000004e-07 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.641029+0000", 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3159999999999998e-06 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978425+0000", 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33739633299999999 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978439+0000", 2024-09-17T16:15:58.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3492999999999999e-05 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978519+0000", 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9736999999999999e-05 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978528+0000", 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0669999999999998e-06 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.978614+0000", 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6522999999999998e-05 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(2.9 162/155 [PushReplyOp(2:96325bd0:::benchmark_data_smithi173_55121_object68334:head)])", 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:51.688909+0000", 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 187.19925625499999, 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33995969700000001, 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.977 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.688909+0000", 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.688909+0000", 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.688909+0000", 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7300000000000001e-07 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:51.688913+0000", 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.563e-06 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:52.028466+0000", 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33955346800000002 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:52.028475+0000", 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5299999999999996e-06 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:52.028513+0000", 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8674999999999998e-05 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:52.028868+0000", 2024-09-17T16:15:58.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00035498799999999998 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:135561 2.6 2:612899dc:::benchmark_data_smithi173_55121_object59072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 142.96678851499999, 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36676542899999998, 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.981 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:15:58.982 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 135561 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921378+0000", 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.897e-06 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921386+0000", 2024-09-17T16:15:58.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.17e-06 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921398+0000", 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2408999999999999e-05 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921406+0000", 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6179999999999997e-06 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921457+0000", 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1743999999999998e-05 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921550+0000", 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2318000000000006e-05 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,8", 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921623+0000", 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3337999999999995e-05 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.923435+0000", 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001811654 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.923947+0000", 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051277600000000005 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288065+0000", 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36411789 2024-09-17T16:15:58.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288107+0000", 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1479e-05 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288142+0000", 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5135999999999997e-05 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:152966 2.c e199/95 2:3484f1c7:::benchmark_data_smithi173_55121_object76477:head v 199'9438, mlcod=199'9436)", 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 125.032038689, 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58877864599999996, 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:15:58.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856128+0000", 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3740000000000001e-06 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856136+0000", 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6950000000000006e-06 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441396+0000", 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58525992800000004 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441405+0000", 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9770000000000006e-06 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441463+0000", 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8025000000000001e-05 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441483+0000", 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9678999999999999e-05 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.444869+0000", 2024-09-17T16:15:58.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033864070000000001 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.444905+0000", 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5561000000000001e-05 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(3.0 255/239 [PushOp(3:0a910604:::benchmark_data_smithi173_66272_object41747:head, version: 231'2622, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0a910604:::benchmark_data_smithi173_66272_object41747:head@231'2622, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:56.548796+0000", 2024-09-17T16:15:58.993 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 2.3393688159999999, 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31237601100000001, 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:56.548796+0000", 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:56.548796+0000", 2024-09-17T16:15:58.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:56.548797+0000", 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3099999999999997e-07 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:56.548857+0000", 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0498999999999998e-05 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:58.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:56.859411+0000", 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31055336900000002 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:56.859436+0000", 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5332000000000001e-05 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:56.859542+0000", 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010626 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:56.859553+0000", 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0301e-05 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:56.861172+0000", 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016196190000000001 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(3.0 256/239 [PushOp(3:0ca39f28:::benchmark_data_smithi173_66272_object41987:head, version: 231'2635, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0ca39f28:::benchmark_data_smithi173_66272_object41987:head@231'2635, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.8475013260000002, 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59916758699999995, 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040665+0000", 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0100000000000003e-07 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040719+0000", 2024-09-17T16:15:58.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4487000000000002e-05 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638102+0000", 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.597383255 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638121+0000", 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8595999999999999e-05 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:59.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638205+0000", 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4042999999999995e-05 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638216+0000", 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1299e-05 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.639831+0000", 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001615006 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:59.001 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:59.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:59.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:15:59.002 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:15:59.002 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:15:59.002 INFO:teuthology.orchestra.run.smithi017.stdout:moved 0 / 51 (0%) 2024-09-17T16:15:59.002 INFO:teuthology.orchestra.run.smithi017.stdout:avg 4.63636 2024-09-17T16:15:59.002 INFO:teuthology.orchestra.run.smithi017.stdout:stddev 1.91988 -> 1.91988 (expected baseline 2.05302) 2024-09-17T16:15:59.002 INFO:teuthology.orchestra.run.smithi017.stdout:min osd.10 with 1 -> 1 pgs (0.215686 -> 0.215686 * mean) 2024-09-17T16:15:59.002 INFO:teuthology.orchestra.run.smithi017.stdout:max osd.5 with 7 -> 7 pgs (1.5098 -> 1.5098 * mean) 2024-09-17T16:15:59.002 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:15:59.003 INFO:teuthology.orchestra.run.smithi017.stdout:oload 1000 2024-09-17T16:15:59.003 INFO:teuthology.orchestra.run.smithi017.stdout:max_change 3 2024-09-17T16:15:59.003 INFO:teuthology.orchestra.run.smithi017.stdout:max_change_osds 4 2024-09-17T16:15:59.003 INFO:teuthology.orchestra.run.smithi017.stdout:average_utilization 0.0132 2024-09-17T16:15:59.003 INFO:teuthology.orchestra.run.smithi017.stdout:overload_utilization 0.1318 2024-09-17T16:15:59.003 INFO:teuthology.orchestra.run.smithi017.stderr:no change 2024-09-17T16:15:59.031 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:59.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:15:59.039 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:58 smithi017 ceph-mon[27571]: pgmap v610: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 941 op/s; 301/190179 objects misplaced (0.158%); 14 MiB/s, 218 objects/s recovering 2024-09-17T16:15:59.039 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:58 smithi017 ceph-mon[27571]: osdmap e257: 12 total, 12 up, 11 in 2024-09-17T16:15:59.039 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:58 smithi017 ceph-mon[27571]: 3.d scrub starts 2024-09-17T16:15:59.039 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:58 smithi017 ceph-mon[27571]: 3.d scrub ok 2024-09-17T16:15:59.133 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:15:59.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:58 smithi173 ceph-mon[30530]: pgmap v610: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 941 op/s; 301/190179 objects misplaced (0.158%); 14 MiB/s, 218 objects/s recovering 2024-09-17T16:15:59.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:58 smithi173 ceph-mon[30530]: osdmap e257: 12 total, 12 up, 11 in 2024-09-17T16:15:59.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:58 smithi173 ceph-mon[30530]: 3.d scrub starts 2024-09-17T16:15:59.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:58 smithi173 ceph-mon[30530]: 3.d scrub ok 2024-09-17T16:15:59.157 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:59.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:15:59.259 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:59.285 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:59.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:58 smithi071 ceph-mon[30002]: pgmap v610: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 941 op/s; 301/190179 objects misplaced (0.158%); 14 MiB/s, 218 objects/s recovering 2024-09-17T16:15:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:58 smithi071 ceph-mon[30002]: osdmap e257: 12 total, 12 up, 11 in 2024-09-17T16:15:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:58 smithi071 ceph-mon[30002]: 3.d scrub starts 2024-09-17T16:15:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:58 smithi071 ceph-mon[30002]: 3.d scrub ok 2024-09-17T16:15:59.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:15:59.408 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:59.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:15:59.510 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:15:59.531 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:59.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:59.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:15:59.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:15:59.666 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:15:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:59.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:15:59.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:15:59.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:15:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:59.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:15:59.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:15:59.896 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:15:59.921 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:15:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:15:59.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:15:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:15:59.960 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:15:59.982 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:15:59.982 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:16:00.004 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:16:00.005 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:16:00.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:00.027 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:16:00.027 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:16:00.044 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:00.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:00.049 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:16:00.049 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:16:00.049 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:15:59 smithi017 ceph-mon[27571]: from='client.25313 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:16:00.070 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:16:00.070 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:16:00.095 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:16:00.095 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:16:00.119 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:16:00.119 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:16:00.141 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:16:00.141 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:16:00.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:15:59 smithi173 ceph-mon[30530]: from='client.25313 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:16:00.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:00.162 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:16:00.162 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:16:00.167 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:00.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:00.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:00.183 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:16:00.183 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:16:00.205 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:16:00.205 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:16:00.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:00.291 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:00.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:00.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:15:59 smithi071 ceph-mon[30002]: from='client.25313 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:16:00.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:00.416 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:00.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:00.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:00.541 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:00.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:00.643 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:00.665 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:00.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:00.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:00.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:00.790 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:00.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:00.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:00.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:00.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:00 smithi017 ceph-mon[27571]: pgmap v612: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 951 op/s; 20/187104 objects degraded (0.011%); 4033/187104 objects misplaced (2.155%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:16:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:00 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 20/187104 objects degraded (0.011%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:16:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:00 smithi017 ceph-mon[27571]: osdmap e258: 12 total, 12 up, 11 in 2024-09-17T16:16:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2588981739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:01.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:01.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:01.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:01.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:00 smithi173 ceph-mon[30530]: pgmap v612: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 951 op/s; 20/187104 objects degraded (0.011%); 4033/187104 objects misplaced (2.155%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:16:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:00 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 20/187104 objects degraded (0.011%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:16:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:00 smithi173 ceph-mon[30530]: osdmap e258: 12 total, 12 up, 11 in 2024-09-17T16:16:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2588981739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:01.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:01.175 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:01.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:01.277 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:01.305 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:01.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:01.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:00 smithi071 ceph-mon[30002]: pgmap v612: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 15 active+clean; 3.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 951 op/s; 20/187104 objects degraded (0.011%); 4033/187104 objects misplaced (2.155%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:16:01.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:00 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 20/187104 objects degraded (0.011%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:16:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:00 smithi071 ceph-mon[30002]: osdmap e258: 12 total, 12 up, 11 in 2024-09-17T16:16:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2588981739' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:01.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:01.431 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:01.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:01.507 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:16:01.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:01.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:01.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:01.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:01.662 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:01.694 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:01.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:01.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:01.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:01.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:01.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:01.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:01.948 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:01.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:01.995 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:02.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:02.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:02.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:02.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:01 smithi173 ceph-mon[30530]: 3.4 scrub starts 2024-09-17T16:16:02.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:01 smithi173 ceph-mon[30530]: osdmap e259: 12 total, 12 up, 11 in 2024-09-17T16:16:02.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:01 smithi173 ceph-mon[30530]: 3.0 scrub starts 2024-09-17T16:16:02.156 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:01 smithi017 ceph-mon[27571]: 3.4 scrub starts 2024-09-17T16:16:02.156 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:01 smithi017 ceph-mon[27571]: osdmap e259: 12 total, 12 up, 11 in 2024-09-17T16:16:02.156 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:01 smithi017 ceph-mon[27571]: 3.0 scrub starts 2024-09-17T16:16:02.179 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:02.206 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:02.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:02.308 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:02.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:01 smithi071 ceph-mon[30002]: 3.4 scrub starts 2024-09-17T16:16:02.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:01 smithi071 ceph-mon[30002]: osdmap e259: 12 total, 12 up, 11 in 2024-09-17T16:16:02.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:01 smithi071 ceph-mon[30002]: 3.0 scrub starts 2024-09-17T16:16:02.372 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:02.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:02.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:02.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:02.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:02.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:02.610 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:02.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:02.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:02.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:02.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:02.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:02.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:02.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:02.852 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:02.875 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:02.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:02.897 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:16:02.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:03.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:03.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:03.102 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:03.131 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:03.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:02 smithi173 ceph-mon[30530]: 3.4 scrub ok 2024-09-17T16:16:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:02 smithi173 ceph-mon[30530]: pgmap v615: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 14 active+clean; 3.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 45/181818 objects degraded (0.025%); 7776/181818 objects misplaced (4.277%); 8.3 MiB/s, 132 objects/s recovering 2024-09-17T16:16:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:02 smithi173 ceph-mon[30530]: 3.0 scrub ok 2024-09-17T16:16:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3002578638' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:02 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:03.214 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:02 smithi071 ceph-mon[30002]: 3.4 scrub ok 2024-09-17T16:16:03.214 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:02 smithi071 ceph-mon[30002]: pgmap v615: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 14 active+clean; 3.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 45/181818 objects degraded (0.025%); 7776/181818 objects misplaced (4.277%); 8.3 MiB/s, 132 objects/s recovering 2024-09-17T16:16:03.214 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:02 smithi071 ceph-mon[30002]: 3.0 scrub ok 2024-09-17T16:16:03.214 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3002578638' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:03.214 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:02 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:03.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:03.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:02 smithi017 ceph-mon[27571]: 3.4 scrub ok 2024-09-17T16:16:03.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:02 smithi017 ceph-mon[27571]: pgmap v615: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 14 active+clean; 3.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 631 op/s; 45/181818 objects degraded (0.025%); 7776/181818 objects misplaced (4.277%); 8.3 MiB/s, 132 objects/s recovering 2024-09-17T16:16:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:02 smithi017 ceph-mon[27571]: 3.0 scrub ok 2024-09-17T16:16:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3002578638' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:02 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:03.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:03.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:03.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:03.385 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:03.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:03.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:03.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:03.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:03.617 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:03.643 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:03.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:03.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:03.778 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:03.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:03.880 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:03.907 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:03.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:03.997 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:03 smithi071 ceph-mon[30002]: 3.8 scrub starts 2024-09-17T16:16:03.998 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:03 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:16:03.998 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:03 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:03.998 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:03 smithi071 ceph-mon[30002]: osdmap e260: 12 total, 12 up, 11 in 2024-09-17T16:16:04.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:04.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:04.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:04.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:04.034 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:16:04.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:04.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:03 smithi173 ceph-mon[30530]: 3.8 scrub starts 2024-09-17T16:16:04.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:03 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:16:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:03 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:03 smithi173 ceph-mon[30530]: osdmap e260: 12 total, 12 up, 11 in 2024-09-17T16:16:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:04.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:04.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:04.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:03 smithi017 ceph-mon[27571]: 3.8 scrub starts 2024-09-17T16:16:04.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:03 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:16:04.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:03 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:04.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:03 smithi017 ceph-mon[27571]: osdmap e260: 12 total, 12 up, 11 in 2024-09-17T16:16:04.261 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:04.290 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:04.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:04.327 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:04.327 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:04.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:04.418 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:04.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:04.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:04.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:04.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:04.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:04.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:04.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:04.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:04.805 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:04.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:04.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:04.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:04.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:04 smithi017 ceph-mon[27571]: 3.8 scrub ok 2024-09-17T16:16:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:04 smithi017 ceph-mon[27571]: pgmap v617: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 585 op/s; 45/179754 objects degraded (0.025%); 7475/179754 objects misplaced (4.158%); 5.8 MiB/s, 92 objects/s recovering 2024-09-17T16:16:05.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:05.062 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:05.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:04 smithi173 ceph-mon[30530]: 3.8 scrub ok 2024-09-17T16:16:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:04 smithi173 ceph-mon[30530]: pgmap v617: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 585 op/s; 45/179754 objects degraded (0.025%); 7475/179754 objects misplaced (4.158%); 5.8 MiB/s, 92 objects/s recovering 2024-09-17T16:16:05.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:16:05.162 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:05.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:05.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:05.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:05.288 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:05.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:04 smithi071 ceph-mon[30002]: 3.8 scrub ok 2024-09-17T16:16:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:04 smithi071 ceph-mon[30002]: pgmap v617: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 585 op/s; 45/179754 objects degraded (0.025%); 7475/179754 objects misplaced (4.158%); 5.8 MiB/s, 92 objects/s recovering 2024-09-17T16:16:05.315 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:05.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:05.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:05.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:05.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:05.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:05.543 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:05.568 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:05.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:05.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:05.684 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:16:05.684 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:16:05.684 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T16:16:05.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:16:05.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:05.696 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:05.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:05.793 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:16:05.793 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:16:05.794 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:05.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:05.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:05.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:05.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:16:05.837 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:16:05.868 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:16:05.868 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:16:05.896 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:16:05.897 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:16:05.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:05.935 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:16:05.935 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:16:05.946 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:05.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:05.961 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:16:05.961 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:16:05.989 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:16:05.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:16:06.018 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:16:06.018 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:16:06.019 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2131315090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:06.024 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:16:06.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:16:06.042 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:16:06.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:06.066 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:16:06.066 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:16:06.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:06.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:06.095 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:16:06.095 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:16:06.129 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:16:06.129 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:16:06.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2131315090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:06.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:06.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:06.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:06.301 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:06.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2131315090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:06.328 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:06.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:06.392 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:16:06.392 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:16:06.392 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:16:06.392 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:16:06.392 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:16:06.430 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:06.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:06.461 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:06.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:06.563 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:06.592 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:06.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:06.694 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:06.725 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:06.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:06.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:06.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:06.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:06.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:06.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:06.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:06.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:06.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:07.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:07.109 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:07.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:07.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:07.229 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:06 smithi173 ceph-mon[30530]: pgmap v618: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 48/174138 objects degraded (0.028%); 6950/174138 objects misplaced (3.991%); 10 MiB/s, 166 objects/s recovering 2024-09-17T16:16:07.229 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:06 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 48/174138 objects degraded (0.028%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:07.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:07.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:07.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:06 smithi017 ceph-mon[27571]: pgmap v618: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 48/174138 objects degraded (0.028%); 6950/174138 objects misplaced (3.991%); 10 MiB/s, 166 objects/s recovering 2024-09-17T16:16:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:06 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 48/174138 objects degraded (0.028%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:07.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:06 smithi071 ceph-mon[30002]: pgmap v618: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 3.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 48/174138 objects degraded (0.028%); 6950/174138 objects misplaced (3.991%); 10 MiB/s, 166 objects/s recovering 2024-09-17T16:16:07.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:06 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 48/174138 objects degraded (0.028%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:07.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:07.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:07.361 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:07.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:07.462 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:07.468 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:07.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:16:07.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:07.485 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:07.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:07.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:07.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:07.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:07.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:07.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:07.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:07.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:07.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:07.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:07.947 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:07.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:07.992 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:07.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:08.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:08.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:08.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:08.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:08.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:07 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:16:08.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:07 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:16:08.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:07 smithi173 ceph-mon[30530]: osdmap e261: 12 total, 12 up, 11 in 2024-09-17T16:16:08.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:08.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:07 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:16:08.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:07 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:16:08.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:07 smithi017 ceph-mon[27571]: osdmap e261: 12 total, 12 up, 11 in 2024-09-17T16:16:08.244 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:08.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:08.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:07 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:16:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:07 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:16:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:07 smithi071 ceph-mon[30002]: osdmap e261: 12 total, 12 up, 11 in 2024-09-17T16:16:08.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:08.369 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:08.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:08.471 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:08.495 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:08.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:08.597 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:08.622 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:08.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:08.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:08.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:08.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:08.851 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:08.877 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:08.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:08.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:09.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:09.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:09.062 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:16:09.064 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:16:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 261, 2024-09-17T16:16:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:16:09.064 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:16:06.897605+0000", 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:15:12.833331+0000", 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5799936, 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:16:09.065 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:16:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:16:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:16:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:16:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 3, 2024-09-17T16:16:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:16:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:16:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "jewel", 2024-09-17T16:16:09.066 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:16:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:16:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:16:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:16:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:16:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:16:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:16:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:16:09.067 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:16:09.068 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:16:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:16:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:16:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:16:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:16:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:16:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:16:09.069 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:16:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:16:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:16:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:16:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:16:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "21", 2024-09-17T16:16:09.070 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:16:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:16:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:16:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:16:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:16:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:16:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:16:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:16:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:16:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:16:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:16:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:16:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:16:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:16:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:16:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:16:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:16:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:16:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:16:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:16:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-17T16:16:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 3, 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_1", 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:14:06.897515+0000", 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:16:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:16:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:16:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:16:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:16:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:16:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:16:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:16:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:16:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:16:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 16, 2024-09-17T16:16:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 16, 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 16, 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 16, 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 16, 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "212", 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:16:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:16:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:16:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:16:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:16:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:16:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:16:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:16:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:16:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:16:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:16:09.082 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 2.059999942779541, 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.059999942779541, 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1, 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 2.059999942779541, 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 2.059999942779541, 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.083 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 173, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.084 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:16:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:16:09.085 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:09.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.086 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.088 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:16:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:09.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:16:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.090 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.092 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:16:09.093 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 252, 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.094 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:09.096 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:16:09.097 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.098 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 111, 2024-09-17T16:16:09.100 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.101 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:16:09.103 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.104 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:16:09.105 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:16:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:16:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 253, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.107 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.108 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.109 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.110 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:16:09.110 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:09.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:16:09.112 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:16:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:16:09.113 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:16:09.114 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:16:09.114 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:16:09.114 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.114 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.114 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:16:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:16:09.115 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.116 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.116 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:16:09.116 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.116 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.116 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.116 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:16:09.117 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 249, 2024-09-17T16:16:09.117 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.117 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.117 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.117 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.117 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.118 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:16:09.118 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:09.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.118 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:16:09.118 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:09.119 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.119 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.119 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.119 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.120 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.120 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:16:09.120 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:09.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.121 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.121 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:16:09.121 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:09.121 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.121 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.122 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.123 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.123 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.123 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.123 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:16:09.123 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:09.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.123 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.124 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:16:09.125 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:09.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:16:09.129 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:16:09.130 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:16:09.131 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.131 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.131 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.131 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.131 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:16:09.132 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:16:09.132 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.132 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.132 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:09.132 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.132 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.133 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.133 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:16:09.133 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:16:09.133 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.133 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.133 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.134 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:16:09.134 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:09.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.135 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:16:09.135 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:09.135 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.135 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.136 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.136 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.136 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:16:09.137 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:09.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.137 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:16:09.138 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:09.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.138 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.139 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.139 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.139 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.139 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:16:09.139 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:09.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.140 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.140 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:16:09.140 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:09.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.141 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.141 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.141 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.141 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.142 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:16:09.142 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:09.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.142 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:16:09.143 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:09.143 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.143 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.144 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:16:09.144 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:16:09.144 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:16:09.144 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:16:09.144 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.144 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.145 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.145 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.145 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:16:09.145 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:16:09.146 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.146 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.146 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:09.146 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.146 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:16:09.146 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:16:09.147 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:16:09.147 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 250, 2024-09-17T16:16:09.147 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:16:09.147 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.147 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.148 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:16:09.148 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:09.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:16:09.149 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:09.149 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.149 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.150 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.150 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.150 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.150 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:16:09.151 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:09.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.151 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.152 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:16:09.152 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:09.152 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.152 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.153 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.153 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.153 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.153 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:16:09.153 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:09.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.154 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.154 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:16:09.154 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:09.155 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.155 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.155 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.155 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.156 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.156 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:16:09.156 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:09.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.157 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.157 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:16:09.157 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:09.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.157 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.157 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:16:09.158 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:16:09.158 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:16:09.158 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:16:09.158 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.158 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.158 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.159 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.159 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:16:09.159 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:16:09.159 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.159 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.159 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:09.159 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 256, 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.163 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.163 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:16:09.163 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:09.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.163 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:09.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:16:09.165 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.166 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 258, 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.167 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.168 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:09.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:16:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:16:09.171 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:09.172 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 256, 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.173 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:16:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:16:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:09.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:16:09.175 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.176 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:09.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:16:09.178 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 0, 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:09.179 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:16:09.180 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:09.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:16:09.181 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:09.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.181 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.183 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.184 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:09.184 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:09.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.184 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:09.184 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:16:09.184 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:09.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.184 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:09.185 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.186 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:13.639996+0000", 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:42.635550+0000", 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:16:06.612720+0000", 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.188 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:16.501057+0000", 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.189 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.190 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:37.628130+0000", 2024-09-17T16:16:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:16:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:21.088656+0000", 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:54.920476+0000", 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:07.879743+0000", 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.193 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:16:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:50.115461+0000", 2024-09-17T16:16:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:16:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.195 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.196 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:40.160542+0000", 2024-09-17T16:16:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:32.545695+0000", 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:09.197 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 65536, 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:05.015584+0000", 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [], 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:16:09.198 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [ 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "3.1", 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: 10, 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: 0 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "3.9", 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: 8, 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: 7 2024-09-17T16:16:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:16:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:16:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:16:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:16:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:16:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:16:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:16:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:16:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:16:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:16:09.201 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:09.202 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:16:09.203 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:08 smithi173 ceph-mon[30530]: pgmap v620: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 801 op/s; 48/168159 objects degraded (0.029%); 6950/168159 objects misplaced (4.133%); 7.4 MiB/s, 118 objects/s recovering 2024-09-17T16:16:09.224 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:09.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:09.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:09.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:08 smithi017 ceph-mon[27571]: pgmap v620: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 801 op/s; 48/168159 objects degraded (0.029%); 6950/168159 objects misplaced (4.133%); 7.4 MiB/s, 118 objects/s recovering 2024-09-17T16:16:09.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:08 smithi071 ceph-mon[30002]: pgmap v620: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 801 op/s; 48/168159 objects degraded (0.029%); 6950/168159 objects misplaced (4.133%); 7.4 MiB/s, 118 objects/s recovering 2024-09-17T16:16:09.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:09.355 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:09.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:09.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:09.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:09.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:09.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:09.583 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:09.604 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:09.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:09.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:09.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:09.734 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:09.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:09.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:09.864 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:09.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:09.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:09.913 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:16:09.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:09.989 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:16:09.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:09.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:10.092 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:10.114 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:10.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:10.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:10.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:10.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/984497281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:16:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3299855338' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:16:10.239 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:10.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:10.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/984497281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:16:10.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3299855338' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:16:10.341 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:10.368 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:10.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:10.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:10.387 INFO:tasks.thrashosds.thrasher:j is {'epoch': 261, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:16:06.897605+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:15:12.833331+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-17T16:14:06.897515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.059999942779541, 'score_stable': 2.059999942779541, 'optimal_score': 1, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 252, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 253, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 249, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 250, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 256, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 258, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 256, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:13.639996+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:42.635550+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:16:06.612720+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:16.501057+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:37.628130+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:21.088656+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:54.920476+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:07.879743+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:50.115461+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:40.160542+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:32.545695+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-17T16:15:05.015584+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.1', 'osds': [10, 0]}, {'pgid': '3.9', 'osds': [8, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:16:10.387 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-17T16:16:10.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/984497281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:16:10.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3299855338' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:16:10.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:10.491 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:10.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:10.593 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:10.616 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:10.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:10.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:10.743 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:10.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:10.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:10.867 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:10.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:10.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:10.943 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:10.968 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:10.998 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:10.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:11.052 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:10 smithi017 ceph-mon[27571]: pgmap v621: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 754 op/s; 130/163704 objects degraded (0.079%); 6691/163704 objects misplaced (4.087%); 7.8 MiB/s, 124 objects/s recovering 2024-09-17T16:16:11.052 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:10 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:11.052 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3299855338' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:16:11.052 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:10 smithi017 ceph-mon[27571]: osdmap e262: 12 total, 12 up, 11 in 2024-09-17T16:16:11.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:11.124 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:11.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:11.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:11.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:11.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:11.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:11.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:10 smithi071 ceph-mon[30002]: pgmap v621: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 754 op/s; 130/163704 objects degraded (0.079%); 6691/163704 objects misplaced (4.087%); 7.8 MiB/s, 124 objects/s recovering 2024-09-17T16:16:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:10 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3299855338' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:16:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:10 smithi071 ceph-mon[30002]: osdmap e262: 12 total, 12 up, 11 in 2024-09-17T16:16:11.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:11.380 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:11.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:11.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:10 smithi173 ceph-mon[30530]: pgmap v621: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 754 op/s; 130/163704 objects degraded (0.079%); 6691/163704 objects misplaced (4.087%); 7.8 MiB/s, 124 objects/s recovering 2024-09-17T16:16:11.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:10 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:11.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3299855338' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:16:11.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:10 smithi173 ceph-mon[30530]: osdmap e262: 12 total, 12 up, 11 in 2024-09-17T16:16:11.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:11.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:11.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:11.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:11.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:11.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:11.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:11.761 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:11.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:11.862 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:11.890 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:11.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:11.992 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:12.021 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:12.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:12.063 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:11 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 150/155274 objects degraded (0.097%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:12.065 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:16:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:16:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:16:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:16:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:13535 2.11 2:8808aac6:::benchmark_data_smithi173_55121_object13534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:16:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:16:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 299.72488254799998, 2024-09-17T16:16:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314941879999999, 2024-09-17T16:16:12.065 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 13535 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:16:12.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335349+0000", 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3599999999999999e-06 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335378+0000", 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8965000000000001e-05 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335391+0000", 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.257e-05 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335398+0000", 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3499999999999999e-06 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.367429+0000", 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.032030422000000003 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.367439+0000", 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0451e-05 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.366280+0000", 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.99884132299999995 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.366842+0000", 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00056174700000000003 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:14925 2.e 2:743cbf17:::benchmark_data_smithi173_55121_object14924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.264388+0000", 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 297.79584250099998, 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43099893299999997, 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.071 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 14925 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264388+0000", 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264388+0000", 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264390+0000", 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.677e-06 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264432+0000", 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2292000000000001e-05 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264447+0000", 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.518e-05 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264455+0000", 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2490000000000001e-06 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264541+0000", 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5760000000000006e-05 2024-09-17T16:16:12.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264676+0000", 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134487 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264784+0000", 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108402 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:12.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265461+0000", 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067649699999999997 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265775+0000", 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031408899999999998 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695211+0000", 2024-09-17T16:16:12.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42943639900000002 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695308+0000", 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7176000000000006e-05 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695387+0000", 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8725000000000003e-05 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:14925 2.e e106/95 ondisk, result = 0)", 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 297.79439502600002, 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42955775600000001, 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265844+0000", 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2450000000000001e-06 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695035+0000", 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42919093699999999 2024-09-17T16:16:12.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695052+0000", 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.772e-05 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695186+0000", 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013379200000000001 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695202+0000", 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5519000000000002e-05 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695393+0000", 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019176600000000001 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.082 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:14944 2.e 2:79e7edec:::benchmark_data_smithi173_55121_object14943:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.358500+0000", 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 297.70173041200002, 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34235612100000001, 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 14944 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358500+0000", 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358500+0000", 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358502+0000", 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8899999999999999e-06 2024-09-17T16:16:12.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358530+0000", 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8248000000000001e-05 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358548+0000", 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8068e-05 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358555+0000", 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.934e-06 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358614+0000", 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8488e-05 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358718+0000", 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103991 2024-09-17T16:16:12.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358818+0000", 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100374 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359536+0000", 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00071760900000000002 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359808+0000", 2024-09-17T16:16:12.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027181699999999999 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700797+0000", 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.340989443 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700830+0000", 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2543999999999999e-05 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700856+0000", 2024-09-17T16:16:12.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6715000000000001e-05 2024-09-17T16:16:12.089 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.089 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.089 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.089 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:14944 2.e e106/95 ondisk, result = 0)", 2024-09-17T16:16:12.089 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.359723+0000", 2024-09-17T16:16:12.089 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 297.70050771699999, 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.341137735, 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359723+0000", 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359723+0000", 2024-09-17T16:16:12.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359724+0000", 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9299999999999996e-07 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359729+0000", 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5770000000000001e-06 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700733+0000", 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34100362499999998 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700739+0000", 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3199999999999996e-06 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700789+0000", 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9845000000000001e-05 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700794+0000", 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6299999999999997e-06 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700861+0000", 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6845000000000001e-05 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(2.4 116/112 [PushOp(2:23763086:::benchmark_data_smithi173_55121_object9285:head, version: 101'619, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:23763086:::benchmark_data_smithi173_55121_object9285:head@101'619, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:25.348190+0000", 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 286.71204018200001, 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61457114000000002, 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348190+0000", 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348190+0000", 2024-09-17T16:16:12.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348191+0000", 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3000000000000001e-07 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348211+0000", 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9811999999999999e-05 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.961507+0000", 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61329585900000005 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.961527+0000", 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0744999999999999e-05 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.962424+0000", 2024-09-17T16:16:12.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089668899999999999 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.962438+0000", 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3458000000000001e-05 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.962762+0000", 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000324047 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(2.4 117/112 [PushOp(2:26e71621:::benchmark_data_smithi173_55121_object17115:head, version: 108'1004, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:26e71621:::benchmark_data_smithi173_55121_object17115:head@108'1004, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:26.624251+0000", 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 285.43597965200001, 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.252196103, 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624251+0000", 2024-09-17T16:16:12.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624251+0000", 2024-09-17T16:16:12.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624252+0000", 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.029e-06 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624283+0000", 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0960000000000002e-05 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875818+0000", 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25153555500000002 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875841+0000", 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2894000000000001e-05 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875950+0000", 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108758 2024-09-17T16:16:12.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875961+0000", 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0784e-05 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.876447+0000", 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048612299999999998 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.103 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:24327 2.17 e120/104 2:ecf8c341:::benchmark_data_smithi173_55121_object24326:head v 120'1226, mlcod=120'1223)", 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 282.64233106299997, 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.245189986, 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8199999999999999e-07 2024-09-17T16:16:12.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417946+0000", 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6263000000000001e-05 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662362+0000", 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24441564700000001 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662376+0000", 2024-09-17T16:16:12.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3573e-05 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662484+0000", 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108559 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662517+0000", 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2755000000000001e-05 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.663062+0000", 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054487399999999999 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.663090+0000", 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7633000000000001e-05 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25228 2.1 2:8352dd7d:::benchmark_data_smithi173_55121_object25227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:12.108 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.786229+0000", 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 281.27400112399999, 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28014749900000002, 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25228 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786229+0000", 2024-09-17T16:16:12.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786229+0000", 2024-09-17T16:16:12.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786230+0000", 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9199999999999999e-07 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786266+0000", 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5939999999999998e-05 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786275+0000", 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0140000000000004e-06 2024-09-17T16:16:12.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786280+0000", 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4710000000000001e-06 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786324+0000", 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3829999999999999e-05 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786410+0000", 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6013000000000005e-05 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786496+0000", 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5784000000000004e-05 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.787047+0000", 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00055157999999999997 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.788017+0000", 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097012900000000002 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066298+0000", 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27828037 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066347+0000", 2024-09-17T16:16:12.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9508000000000001e-05 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066377+0000", 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9867999999999999e-05 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25249 2.e 2:7b694fa0:::benchmark_data_smithi173_55121_object25248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.912148+0000", 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 281.14808288299997, 2024-09-17T16:16:12.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33397465900000001, 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25249 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912148+0000", 2024-09-17T16:16:12.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912148+0000", 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912149+0000", 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912174+0000", 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5080999999999999e-05 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912187+0000", 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3311e-05 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912193+0000", 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6110000000000001e-06 2024-09-17T16:16:12.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912270+0000", 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6598000000000001e-05 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912380+0000", 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110389 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912476+0000", 2024-09-17T16:16:12.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5493000000000002e-05 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913408+0000", 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093208600000000005 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.965233+0000", 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.051825489000000002 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246005+0000", 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28077230800000003 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246071+0000", 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6000999999999994e-05 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246122+0000", 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0955999999999998e-05 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.121 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:25249 2.e e121/95 ondisk, result = 0)", 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.913496+0000", 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 281.14673466800002, 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.332636237, 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913496+0000", 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913496+0000", 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913497+0000", 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913500+0000", 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5449999999999999e-06 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245930+0000", 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33242972799999998 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245935+0000", 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9479999999999999e-06 2024-09-17T16:16:12.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245989+0000", 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4187999999999997e-05 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245999+0000", 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9729999999999992e-06 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246132+0000", 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000133118 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:26909 2.e 2:76c214cd:::benchmark_data_smithi173_55121_object26908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.325897+0000", 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 279.73433332600001, 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23800827699999999, 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.126 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26909 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325897+0000", 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325897+0000", 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325898+0000", 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1379999999999999e-06 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325939+0000", 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0466000000000002e-05 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325950+0000", 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1287000000000001e-05 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325955+0000", 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1730000000000003e-06 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326000+0000", 2024-09-17T16:16:12.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4674e-05 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326082+0000", 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1706999999999995e-05 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326170+0000", 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7903000000000002e-05 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326971+0000", 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080154300000000005 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327360+0000", 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00038908699999999998 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563817+0000", 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23645692900000001 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563856+0000", 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9150000000000003e-05 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563906+0000", 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.922e-05 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:26910 2.1 2:838cca45:::benchmark_data_smithi173_55121_object26909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 279.732758069, 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26261527099999998, 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26910 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.133 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.23e-07 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327529+0000", 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5284000000000001e-05 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327538+0000", 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7790000000000007e-06 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327543+0000", 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4700000000000004e-06 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327599+0000", 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6100999999999997e-05 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327722+0000", 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012263399999999999 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327808+0000", 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6803000000000002e-05 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.328412+0000", 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060394300000000001 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.328847+0000", 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00043514399999999999 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.590005+0000", 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26115713200000001 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.590043+0000", 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.824e-05 2024-09-17T16:16:12.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.590088+0000", 2024-09-17T16:16:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5018000000000002e-05 2024-09-17T16:16:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.140 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:26911 2.e 2:7501b6f7:::benchmark_data_smithi173_55121_object26910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.329019+0000", 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 279.73121181099998, 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26290611800000002, 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.141 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26911 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329019+0000", 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329019+0000", 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329020+0000", 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329082+0000", 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2477000000000005e-05 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329093+0000", 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0814e-05 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329100+0000", 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1339999999999998e-06 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329141+0000", 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0915999999999999e-05 2024-09-17T16:16:12.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329219+0000", 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7902000000000001e-05 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329304+0000", 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4820000000000004e-05 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:12.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330001+0000", 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00069732400000000003 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330213+0000", 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000212563 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.591852+0000", 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26163829100000002 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.591886+0000", 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4662000000000001e-05 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.591925+0000", 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8470000000000003e-05 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.147 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:26909 2.e e121/95 ondisk, result = 0)", 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.330196+0000", 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 279.73003442200002, 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.233715383, 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330196+0000", 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330196+0000", 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330197+0000", 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5299999999999999e-07 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330200+0000", 2024-09-17T16:16:12.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.033e-06 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563761+0000", 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23356138900000001 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563767+0000", 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4569999999999996e-06 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563808+0000", 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1121999999999997e-05 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563813+0000", 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4249999999999997e-06 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563912+0000", 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8504000000000004e-05 2024-09-17T16:16:12.151 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:27009 2.e 2:7c49ab26:::benchmark_data_smithi173_55121_object27008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:33.169267+0000", 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 278.89096346600002, 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33189007700000001, 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.152 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 27009 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169267+0000", 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169267+0000", 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169270+0000", 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9469999999999999e-06 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169342+0000", 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1785999999999993e-05 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169372+0000", 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0399000000000002e-05 2024-09-17T16:16:12.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169387+0000", 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4323e-05 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169516+0000", 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012973600000000001 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169614+0000", 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8103000000000006e-05 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169705+0000", 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0989000000000006e-05 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.170561+0000", 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085536800000000001 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.170823+0000", 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026205999999999999 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.501034+0000", 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33021098799999998 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.501104+0000", 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0321000000000007e-05 2024-09-17T16:16:12.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.501157+0000", 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3056999999999997e-05 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:29460 2.1 2:83aee601:::benchmark_data_smithi173_55121_object29459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:16:12.158 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 262.862568119, 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31850449400000003, 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 29460 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197664+0000", 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1289999999999999e-06 2024-09-17T16:16:12.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197740+0000", 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6361999999999998e-05 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197753+0000", 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3453e-05 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197759+0000", 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7610000000000004e-06 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197820+0000", 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0735e-05 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197960+0000", 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013987799999999999 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198120+0000", 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016034 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198880+0000", 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075943099999999997 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.322818+0000", 2024-09-17T16:16:12.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.123938883 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516108+0000", 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.193289187 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516141+0000", 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3691e-05 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516167+0000", 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5644e-05 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:129847 2.1 2:86a30a1b:::benchmark_data_smithi173_55121_object53358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 164.26118814899999, 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29781534999999998, 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 129847 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799048+0000", 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6289999999999998e-06 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799056+0000", 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3809999999999999e-06 2024-09-17T16:16:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799074+0000", 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7864e-05 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799085+0000", 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1076e-05 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799177+0000", 2024-09-17T16:16:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1402000000000003e-05 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799338+0000", 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000161263 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799482+0000", 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000143638 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801720+0000", 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022381739999999999 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.901264+0000", 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.099544493999999997 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096745+0000", 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.195480396 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096804+0000", 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8990000000000003e-05 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096858+0000", 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4042999999999997e-05 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:129847 2.1 e184/104 ondisk, result = 0)", 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 164.25823132100001, 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29486439199999998, 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.802000+0000", 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1600000000000001e-07 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.802005+0000", 2024-09-17T16:16:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9060000000000001e-06 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096600+0000", 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.294595249 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096606+0000", 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.733e-06 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096731+0000", 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000124735 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096739+0000", 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096864+0000", 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012515399999999999 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(3.4 241/239 [PushOp(3:20540438:::benchmark_data_smithi173_66272_object5569:head, version: 213'343, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:20540438:::benchmark_data_smithi173_66272_object5569:head@213'343, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.507436634999998, 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56341482399999998, 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:16:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552796+0000", 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.888e-06 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552825+0000", 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9417999999999999e-05 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115364+0000", 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56253852800000004 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115383+0000", 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9055999999999998e-05 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115470+0000", 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7520000000000002e-05 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115481+0000", 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0329000000000001e-05 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.116209+0000", 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072808499999999995 2024-09-17T16:16:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:12.180 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:16:12.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:12.181 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:12.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:12.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:12.311 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:12.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:12.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:12.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:11 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 150/155274 objects degraded (0.097%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:12.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:11 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 150/155274 objects degraded (0.097%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:12.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:12.441 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:12.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:12.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:12.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:12.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:12.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:12.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:12.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:12.700 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:16:12.700 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:16:12.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:12.702 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:12.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:12.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:16:12.736 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:16:12.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:16:12.768 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:16:12.800 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:16:12.801 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:16:12.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:12.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:12.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:12.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:12.834 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:16:12.835 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:16:12.868 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:16:12.868 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:16:12.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:16:12.894 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:16:12.920 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:16:12.921 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:16:12.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:12.946 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:16:12.946 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:16:12.959 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:12.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:12.975 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:16:12.976 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:16:13.006 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:16:13.006 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:16:13.038 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:16:13.038 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:16:13.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:13.068 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:12 smithi017 ceph-mon[27571]: pgmap v623: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 150/155274 objects degraded (0.097%); 6449/155274 objects misplaced (4.153%); 7.2 MiB/s, 115 objects/s recovering 2024-09-17T16:16:13.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/755524456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:13.092 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:13.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:13.194 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:13.219 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:13.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:13.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:13.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:12 smithi071 ceph-mon[30002]: pgmap v623: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 150/155274 objects degraded (0.097%); 6449/155274 objects misplaced (4.153%); 7.2 MiB/s, 115 objects/s recovering 2024-09-17T16:16:13.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/755524456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:13.321 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:13.346 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:13.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:13.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:12 smithi173 ceph-mon[30530]: pgmap v623: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 150/155274 objects degraded (0.097%); 6449/155274 objects misplaced (4.153%); 7.2 MiB/s, 115 objects/s recovering 2024-09-17T16:16:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/755524456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:13.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:13.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:13.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:13.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:16:13.574 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:13.596 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:13.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:13.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:13.720 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:13.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:13.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:13.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:13.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:13.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:13.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:13.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:13.979 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:13.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:14.080 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:14.109 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:14.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:14.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:16:14.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:14.270 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:14.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:14.371 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:14.399 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:14.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:14.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:14.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:14.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:14.628 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:14.656 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:14.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:14.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:14.758 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:14.765 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:14.789 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:14.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:14.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:14.915 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:14.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:15.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:15.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:15.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:15.142 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:15.166 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:15.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:15.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:14 smithi017 ceph-mon[27571]: pgmap v624: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 880 op/s; 150/153006 objects degraded (0.098%); 6449/153006 objects misplaced (4.215%); 3.1 MiB/s, 49 objects/s recovering 2024-09-17T16:16:15.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:15.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:15.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:15.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:14 smithi071 ceph-mon[30002]: pgmap v624: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 880 op/s; 150/153006 objects degraded (0.098%); 6449/153006 objects misplaced (4.215%); 3.1 MiB/s, 49 objects/s recovering 2024-09-17T16:16:15.388 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:16:15.388 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:16:15.388 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T16:16:15.389 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:16:15.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:15.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:14 smithi173 ceph-mon[30530]: pgmap v624: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 880 op/s; 150/153006 objects degraded (0.098%); 6449/153006 objects misplaced (4.215%); 3.1 MiB/s, 49 objects/s recovering 2024-09-17T16:16:15.421 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:15.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:15.459 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:15.484 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:15.485 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:15.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:15.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:15.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:15.648 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:15.675 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:15.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:15.777 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:15.804 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:15.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:15.906 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:15.934 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:15.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:15.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:16.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:16.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:16.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:16.162 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:16.184 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:16.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:16.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:16.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:16.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:16.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:16.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:16.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:16.484 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:16:16.535 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:16.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:16.671 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:16.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:16:16.711 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:16.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:16.814 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:16.845 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:16.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:16.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:16.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:16.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:16.973 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:17.033 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:16 smithi017 ceph-mon[27571]: pgmap v625: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 176/147042 objects degraded (0.120%); 5801/147042 objects misplaced (3.945%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T16:16:17.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:17.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:17.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:17.178 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:17.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:17.224 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:17.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:17.249 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:16 smithi071 ceph-mon[30002]: pgmap v625: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 176/147042 objects degraded (0.120%); 5801/147042 objects misplaced (3.945%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T16:16:17.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:17.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:17.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:17.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:16 smithi173 ceph-mon[30530]: pgmap v625: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 176/147042 objects degraded (0.120%); 5801/147042 objects misplaced (3.945%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T16:16:17.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:17.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:17.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:17.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:17.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:17.610 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:17.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:17.712 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:17.751 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:17.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:17.852 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:17.882 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:17.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:17.975 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:16:17.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:18.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:18.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:18.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:18.067 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:16:18.067 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:16:18.067 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 263, 2024-09-17T16:16:18.068 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:16:18.068 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:16:18.068 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:16:17.965628+0000", 2024-09-17T16:16:18.068 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:16:18.068 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:15:12.833331+0000", 2024-09-17T16:16:18.068 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:16:18.069 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5806080, 2024-09-17T16:16:18.069 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:16:18.069 INFO:teuthology.orchestra.run.smithi017.stdout: "nodeep-scrub", 2024-09-17T16:16:18.069 INFO:teuthology.orchestra.run.smithi017.stdout: "noscrub", 2024-09-17T16:16:18.070 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:16:18.070 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:16:18.070 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:16:18.070 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:16:18.070 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:16:18.071 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:16:18.071 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:16:18.071 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:16:18.071 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:16:18.071 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:16:18.071 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 3, 2024-09-17T16:16:18.072 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:16:18.072 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:16:18.072 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "jewel", 2024-09-17T16:16:18.072 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:16:18.072 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:16:18.073 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:16:18.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.073 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:16:18.073 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:16:18.073 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:16:18.073 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:16:18.074 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:16:18.074 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:16:18.074 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:16:18.074 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:16:18.075 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:16:18.075 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:16:18.075 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:16:18.075 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:16:18.075 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:16:18.075 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:16:18.076 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:16:18.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:16:18.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:16:18.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:16:18.076 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:16:18.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:16:18.077 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:16:18.077 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:16:18.077 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:16:18.078 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:16:18.078 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:16:18.078 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:16:18.078 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:16:18.078 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:16:18.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "21", 2024-09-17T16:16:18.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:16:18.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:16:18.079 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:16:18.079 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:16:18.080 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:16:18.080 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:16:18.080 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:16:18.080 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:16:18.081 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:16:18.081 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:16:18.081 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:16:18.081 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:16:18.081 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:16:18.081 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:16:18.082 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:16:18.082 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:16:18.082 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:16:18.082 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:16:18.082 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:16:18.083 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:16:18.083 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:16:18.083 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:16:18.083 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:16:18.083 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:16:18.084 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:16:18.084 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:16:18.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.084 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:16:18.084 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:16:18.084 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:16:18.085 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:16:18.085 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:16:18.085 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:16:18.085 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:16:18.085 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:16:18.086 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:16:18.086 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:16:18.086 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:16:18.086 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:16:18.086 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:16:18.086 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:16:18.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.087 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:16:18.087 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:16:18.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.087 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:16:18.087 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:16:18.088 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:16:18.088 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:16:18.088 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:16:18.088 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:16:18.088 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:16:18.089 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-17T16:16:18.089 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-17T16:16:18.089 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-17T16:16:18.089 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.090 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 3, 2024-09-17T16:16:18.090 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_1", 2024-09-17T16:16:18.090 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:14:06.897515+0000", 2024-09-17T16:16:18.090 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:16:18.090 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:16:18.091 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:16:18.091 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:16:18.091 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:16:18.091 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:16:18.091 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:16:18.091 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:16:18.092 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:16:18.092 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:16:18.092 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:16:18.092 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:16:18.092 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:16:18.092 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 16, 2024-09-17T16:16:18.092 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 16, 2024-09-17T16:16:18.092 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 16, 2024-09-17T16:16:18.093 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 16, 2024-09-17T16:16:18.093 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 16, 2024-09-17T16:16:18.093 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:16:18.093 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:16:18.093 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:16:18.093 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:16:18.094 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:16:18.094 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:16:18.094 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:16:18.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.094 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "212", 2024-09-17T16:16:18.094 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:16:18.094 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:16:18.094 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:16:18.095 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:16:18.095 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:16:18.095 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:16:18.095 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:16:18.095 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:16:18.095 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:16:18.096 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:16:18.096 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:16:18.096 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:16:18.096 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:16:18.096 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:16:18.096 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:16:18.096 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:16:18.096 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:16:18.097 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:16:18.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 2.059999942779541, 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.059999942779541, 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1, 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 2.059999942779541, 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 2.059999942779541, 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.099 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 173, 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.100 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.101 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.102 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.102 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.102 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.102 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:16:18.102 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:18.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.102 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:16:18.104 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:18.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.105 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.105 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:16:18.105 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:16:18.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.105 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 252, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.107 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:18.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.110 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.111 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.112 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:16:18.112 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:16:18.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.112 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.112 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:16:18.113 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:16:18.113 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:16:18.113 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:16:18.113 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.113 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.113 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.114 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.114 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:16:18.114 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:16:18.114 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.115 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.115 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:18.115 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.115 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.115 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.115 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:16:18.115 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 111, 2024-09-17T16:16:18.116 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.116 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.116 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.116 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.116 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.116 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:16:18.117 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:18.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.117 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.117 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:16:18.118 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:18.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.118 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.118 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.118 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.119 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.119 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:16:18.119 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:18.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.120 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:16:18.121 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:18.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.121 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.121 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:16:18.121 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:18.121 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.121 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.122 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.122 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.122 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.122 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:16:18.122 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:18.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.123 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.123 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:16:18.123 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:16:18.123 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.123 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.124 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:16:18.124 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:16:18.124 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:16:18.124 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:16:18.124 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.124 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.124 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.124 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 253, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.125 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.127 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.128 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.128 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.128 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:18.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:16:18.130 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:18.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:16:18.130 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:18.130 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.130 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.131 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.132 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:16:18.133 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:16:18.133 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.133 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.133 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:16:18.133 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.133 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.133 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.133 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:16:18.133 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 249, 2024-09-17T16:16:18.133 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:18.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:16:18.135 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.136 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.138 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.139 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.140 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.142 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.142 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:16:18.142 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:18.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.142 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.142 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:18.143 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.144 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.144 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.144 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.144 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.144 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:16:18.144 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:18.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:16:18.145 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.146 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:16:18.148 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.151 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:16:18.152 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:16:18.152 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:16:18.152 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:16:18.152 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.152 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.152 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.152 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.152 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 250, 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:16:18.153 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:16:18.154 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:16:18.156 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:16:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 256, 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:16:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:16:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:16:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 258, 2024-09-17T16:16:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:16:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:16:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:16:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:16:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:16:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:16:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:16:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:16:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:16:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:16:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:16:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:16:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:16:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:16:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:16:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:16:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:16:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:16:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 256, 2024-09-17T16:16:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:16:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:16:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:16:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:16:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:16:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:16:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:16:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:16:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:16:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:16:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:16:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:16:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:16:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:16:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:16:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:16:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 0, 2024-09-17T16:16:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-17T16:16:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:16:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:16:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:16:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:16:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 208, 2024-09-17T16:16:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:16:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:16:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:16:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:16:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:13.639996+0000", 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:42.635550+0000", 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:16:06.612720+0000", 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:16.501057+0000", 2024-09-17T16:16:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:37.628130+0000", 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:16:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:21.088656+0000", 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:54.920476+0000", 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:07.879743+0000", 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:16:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:50.115461+0000", 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:40.160542+0000", 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:16:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:32.545695+0000", 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 65536, 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:15:05.015584+0000", 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [], 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [ 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "3.1", 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: 10, 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: 0 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "3.9", 2024-09-17T16:16:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: 8, 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: 7 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:16:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:16:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:18.207 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:16:18.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3062093859' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:18.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:18.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:18.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:18.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3062093859' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:18.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:18.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:18.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3062093859' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:18.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:18.463 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:18.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:18.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:18.588 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:18.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:18.690 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:18.721 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:18.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:18.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:18.852 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:18.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:18.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:18.954 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:18.984 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:18.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:19.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:19.108 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:19.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:19.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:18 smithi173 ceph-mon[30530]: pgmap v626: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 920 op/s; 176/140547 objects degraded (0.125%); 5801/140547 objects misplaced (4.127%); 6.4 MiB/s, 102 objects/s recovering 2024-09-17T16:16:19.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:18 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 176/140547 objects degraded (0.125%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:19.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:18 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:19.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3062093859' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:19.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:18 smithi173 ceph-mon[30530]: osdmap e263: 12 total, 12 up, 11 in 2024-09-17T16:16:19.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1158897793' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:16:19.210 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:19.221 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:18 smithi017 ceph-mon[27571]: pgmap v626: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 920 op/s; 176/140547 objects degraded (0.125%); 5801/140547 objects misplaced (4.127%); 6.4 MiB/s, 102 objects/s recovering 2024-09-17T16:16:19.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:18 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 176/140547 objects degraded (0.125%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:19.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:18 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:19.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3062093859' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:19.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:18 smithi017 ceph-mon[27571]: osdmap e263: 12 total, 12 up, 11 in 2024-09-17T16:16:19.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1158897793' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:16:19.233 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:19.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:18 smithi071 ceph-mon[30002]: pgmap v626: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 920 op/s; 176/140547 objects degraded (0.125%); 5801/140547 objects misplaced (4.127%); 6.4 MiB/s, 102 objects/s recovering 2024-09-17T16:16:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:18 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 176/140547 objects degraded (0.125%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:18 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3062093859' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:18 smithi071 ceph-mon[30002]: osdmap e263: 12 total, 12 up, 11 in 2024-09-17T16:16:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1158897793' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:16:19.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:19.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:19.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:19.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:19.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:19.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:19.596 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:19.619 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:19.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:19.720 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:19.748 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:19.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:19.849 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:19.874 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:19.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:19.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:19.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:20.004 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:20.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:20.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:20.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:20.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:20.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:20.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:20.265 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:20.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:20.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:20.330 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:20.331 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:20.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:20.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:20.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:20.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:20.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:20.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:20.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:20.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:20.648 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:20.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:20.750 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:20.781 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:20.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:20.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:20.912 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:20.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:21.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:21.041 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:21.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:21.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:21.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:21.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:21.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:21.296 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:21.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:21.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:21.420 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:21.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:21.522 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:21.550 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:21.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:21.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:21 smithi173 ceph-mon[30530]: pgmap v628: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.8 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 935 op/s; 176/135966 objects degraded (0.129%); 5512/135966 objects misplaced (4.054%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T16:16:21.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:21.681 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:21.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:21.721 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:21 smithi017 ceph-mon[27571]: pgmap v628: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.8 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 935 op/s; 176/135966 objects degraded (0.129%); 5512/135966 objects misplaced (4.054%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T16:16:21.783 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:21.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:21 smithi071 ceph-mon[30002]: pgmap v628: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.8 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 935 op/s; 176/135966 objects degraded (0.129%); 5512/135966 objects misplaced (4.054%); 7.2 MiB/s, 114 objects/s recovering 2024-09-17T16:16:21.814 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:21.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:21.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:21.944 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:21.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:22.046 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:22.076 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:22.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:22.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:22.206 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:22.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:22.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:22.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:22.335 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:22.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:22.340 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:16:22.436 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:22.474 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:22.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:22.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:22.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:22.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:22 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:16:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:22 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:16:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:22 smithi173 ceph-mon[30530]: osdmap e264: 12 total, 12 up, 11 in 2024-09-17T16:16:22.702 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:22.728 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:22.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:22.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:22.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:22 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:16:22.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:22 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:16:22.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:22 smithi017 ceph-mon[27571]: osdmap e264: 12 total, 12 up, 11 in 2024-09-17T16:16:22.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:22 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:16:22.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:22 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:16:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:22 smithi071 ceph-mon[30002]: osdmap e264: 12 total, 12 up, 11 in 2024-09-17T16:16:22.830 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:22.854 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:22.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:22.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:22.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:22.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:22.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:22.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:23.092 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:23.119 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:23.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:23.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:23.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:23.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:23.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:23.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:23.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:23.370 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:23.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:23.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:23.389 INFO:tasks.thrashosds.thrasher:j is {'epoch': 263, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:16:17.965628+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:15:12.833331+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-17T16:14:06.897515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '212', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.059999942779541, 'score_stable': 2.059999942779541, 'optimal_score': 1, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 252, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 253, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 249, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 250, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 256, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 258, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 256, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 208, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:13.639996+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:42.635550+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:16:06.612720+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:16.501057+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:37.628130+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:21.088656+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:54.920476+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:07.879743+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:50.115461+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:40.160542+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:15:32.545695+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-17T16:15:05.015584+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.1', 'osds': [10, 0]}, {'pgid': '3.9', 'osds': [8, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:16:23.389 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:16:23.472 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:23.497 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:23.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:23.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:23.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:23 smithi017 ceph-mon[27571]: pgmap v630: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.6 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s; 184/126822 objects degraded (0.145%); 5236/126822 objects misplaced (4.129%); 9.2 MiB/s, 147 objects/s recovering 2024-09-17T16:16:23.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:23.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:23.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:23 smithi173 ceph-mon[30530]: pgmap v630: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.6 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s; 184/126822 objects degraded (0.145%); 5236/126822 objects misplaced (4.129%); 9.2 MiB/s, 147 objects/s recovering 2024-09-17T16:16:23.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:23.754 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:23.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:23.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:23.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:23 smithi071 ceph-mon[30002]: pgmap v630: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.6 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s; 184/126822 objects degraded (0.145%); 5236/126822 objects misplaced (4.129%); 9.2 MiB/s, 147 objects/s recovering 2024-09-17T16:16:23.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:23.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:23.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:23.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:23.901 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:16:23.902 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:16:23.912 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:23.932 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:16:23.933 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:16:23.968 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:16:23.968 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:16:23.968 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:23.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:23.995 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:16:23.996 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:16:24.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:24.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:24.023 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:16:24.024 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:16:24.050 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:16:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:16:24.075 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:16:24.075 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:16:24.103 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:16:24.103 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:16:24.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:24.126 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:16:24.126 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:16:24.129 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:24.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:24.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:24.154 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:16:24.155 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:16:24.183 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:16:24.184 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:16:24.210 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:16:24.211 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:16:24.231 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:24.261 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:24.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:24.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:24.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:24.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:24.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:24.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2160232432' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:24 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 184/126822 objects degraded (0.145%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:24.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:24.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:24.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:24.516 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:16:24.516 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:16:24.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:16:24.516 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:16:24.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.516 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:13526 2.d e103/95 2:b276e7f5:::benchmark_data_smithi173_55121_object13525:head v 103'845, mlcod=103'842)", 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.327883+0000", 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 312.18444128599998, 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41382291199999999, 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.327883+0000", 2024-09-17T16:16:24.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.327883+0000", 2024-09-17T16:16:24.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.327884+0000", 2024-09-17T16:16:24.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5899999999999996e-07 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.328029+0000", 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014476699999999999 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.740919+0000", 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41289076499999999 2024-09-17T16:16:24.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.740932+0000", 2024-09-17T16:16:24.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2248e-05 2024-09-17T16:16:24.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741023+0000", 2024-09-17T16:16:24.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1221e-05 2024-09-17T16:16:24.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741050+0000", 2024-09-17T16:16:24.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7132e-05 2024-09-17T16:16:24.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741673+0000", 2024-09-17T16:16:24.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062324600000000004 2024-09-17T16:16:24.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741706+0000", 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2974e-05 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:13531 2.a e103/95 2:59aa3d24:::benchmark_data_smithi173_55121_object13530:head v 103'829, mlcod=103'826)", 2024-09-17T16:16:24.522 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.331447+0000", 2024-09-17T16:16:24.524 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 312.18087712300002, 2024-09-17T16:16:24.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.507364445, 2024-09-17T16:16:24.524 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.524 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.524 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.524 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331447+0000", 2024-09-17T16:16:24.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331447+0000", 2024-09-17T16:16:24.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331448+0000", 2024-09-17T16:16:24.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8899999999999999e-07 2024-09-17T16:16:24.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331493+0000", 2024-09-17T16:16:24.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4728999999999998e-05 2024-09-17T16:16:24.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838043+0000", 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.50655012099999996 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838053+0000", 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7850000000000003e-06 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838135+0000", 2024-09-17T16:16:24.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2113000000000005e-05 2024-09-17T16:16:24.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838163+0000", 2024-09-17T16:16:24.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8181999999999999e-05 2024-09-17T16:16:24.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838781+0000", 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061825400000000005 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838812+0000", 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0571999999999998e-05 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.529 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:14940 2.a e106/95 2:57eacb6e:::benchmark_data_smithi173_55121_object14939:head v 106'904, mlcod=105'901)", 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.352719+0000", 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 310.15960599099998, 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25501401899999998, 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.352719+0000", 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.352719+0000", 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.352719+0000", 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0900000000000001e-07 2024-09-17T16:16:24.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.352797+0000", 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8271999999999999e-05 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606767+0000", 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25397041100000001 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606776+0000", 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5609999999999999e-06 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606852+0000", 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6020999999999996e-05 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606884+0000", 2024-09-17T16:16:24.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2354000000000002e-05 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.607694+0000", 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080928700000000001 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.607733+0000", 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8903999999999999e-05 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.534 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.535 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:14949 2.a e106/95 2:58bfb129:::benchmark_data_smithi173_55121_object14948:head v 106'905, mlcod=105'902)", 2024-09-17T16:16:24.535 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.606700+0000", 2024-09-17T16:16:24.535 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 309.90562425899998, 2024-09-17T16:16:24.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25693972399999998, 2024-09-17T16:16:24.536 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.536 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.536 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.536 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606700+0000", 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606700+0000", 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606702+0000", 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8810000000000001e-06 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.606738+0000", 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5660000000000001e-05 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.862886+0000", 2024-09-17T16:16:24.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25614775400000001 2024-09-17T16:16:24.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.862891+0000", 2024-09-17T16:16:24.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.057e-06 2024-09-17T16:16:24.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.862948+0000", 2024-09-17T16:16:24.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7484000000000001e-05 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.862968+0000", 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9629000000000001e-05 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.863600+0000", 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000632629 2024-09-17T16:16:24.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.863640+0000", 2024-09-17T16:16:24.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9629999999999998e-05 2024-09-17T16:16:24.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:20838 2.e e116/95 2:7e96b7fd:::benchmark_data_smithi173_55121_object20837:head v 116'1252, mlcod=116'1249)", 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:25.455525+0000", 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 299.05679968800001, 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31922061699999998, 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455525+0000", 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455525+0000", 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455526+0000", 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.33e-07 2024-09-17T16:16:24.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455549+0000", 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3686999999999999e-05 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773668+0000", 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31811887700000002 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773674+0000", 2024-09-17T16:16:24.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.037e-06 2024-09-17T16:16:24.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773734+0000", 2024-09-17T16:16:24.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9619e-05 2024-09-17T16:16:24.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773758+0000", 2024-09-17T16:16:24.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4097999999999999e-05 2024-09-17T16:16:24.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.774701+0000", 2024-09-17T16:16:24.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00094260999999999998 2024-09-17T16:16:24.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.774745+0000", 2024-09-17T16:16:24.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4756000000000003e-05 2024-09-17T16:16:24.548 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.548 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.548 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.548 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25217 2.19 2:9a6d6ce1:::benchmark_data_smithi173_55121_object25216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:24.548 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.776796+0000", 2024-09-17T16:16:24.548 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 293.735528332, 2024-09-17T16:16:24.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47785892200000002, 2024-09-17T16:16:24.548 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25217 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776796+0000", 2024-09-17T16:16:24.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776796+0000", 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776798+0000", 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8899999999999999e-06 2024-09-17T16:16:24.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776843+0000", 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5306999999999998e-05 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776856+0000", 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2858e-05 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776862+0000", 2024-09-17T16:16:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3120000000000001e-06 2024-09-17T16:16:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776906+0000", 2024-09-17T16:16:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4311000000000003e-05 2024-09-17T16:16:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777001+0000", 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4927999999999996e-05 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777093+0000", 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1878000000000004e-05 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777671+0000", 2024-09-17T16:16:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00057851599999999995 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.036749+0000", 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.259078209 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254584+0000", 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21783461500000001 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254624+0000", 2024-09-17T16:16:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9915e-05 2024-09-17T16:16:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254655+0000", 2024-09-17T16:16:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1183000000000003e-05 2024-09-17T16:16:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:25217 2.19 e121/114 ondisk, result = 0)", 2024-09-17T16:16:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.781044+0000", 2024-09-17T16:16:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 293.731280626, 2024-09-17T16:16:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47361829999999999, 2024-09-17T16:16:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781044+0000", 2024-09-17T16:16:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781044+0000", 2024-09-17T16:16:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781045+0000", 2024-09-17T16:16:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4099999999999998e-07 2024-09-17T16:16:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781049+0000", 2024-09-17T16:16:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4859999999999999e-06 2024-09-17T16:16:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254521+0000", 2024-09-17T16:16:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47347151599999998 2024-09-17T16:16:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254527+0000", 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5239999999999997e-06 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254576+0000", 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8445000000000001e-05 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254581+0000", 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:16:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254662+0000", 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1454999999999999e-05 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25240 2.9 e121/104 2:9323c63b:::benchmark_data_smithi173_55121_object25239:head v 121'1240, mlcod=121'1236)", 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.804013+0000", 2024-09-17T16:16:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 293.70831178399999, 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25673434699999997, 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804013+0000", 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804013+0000", 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804014+0000", 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4499999999999996e-07 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804036+0000", 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2107e-05 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.059951+0000", 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25591505199999998 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.059958+0000", 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8580000000000002e-06 2024-09-17T16:16:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060030+0000", 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1942999999999999e-05 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060054+0000", 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4596e-05 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060719+0000", 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00066500100000000003 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.060747+0000", 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7945000000000001e-05 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:26910 2.1 e121/104 2:838cca45:::benchmark_data_smithi173_55121_object26909:head v 121'1292, mlcod=121'1289)", 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.327897+0000", 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 292.18442713500002, 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23580620599999999, 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327897+0000", 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327897+0000", 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327898+0000", 2024-09-17T16:16:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4900000000000005e-07 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327921+0000", 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3317000000000001e-05 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.562770+0000", 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23484860900000001 2024-09-17T16:16:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.562785+0000", 2024-09-17T16:16:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.484e-05 2024-09-17T16:16:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.562965+0000", 2024-09-17T16:16:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018036500000000001 2024-09-17T16:16:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563003+0000", 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7696e-05 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563671+0000", 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00066814399999999996 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563704+0000", 2024-09-17T16:16:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2486e-05 2024-09-17T16:16:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:26911 2.e e121/95 2:7501b6f7:::benchmark_data_smithi173_55121_object26910:head v 121'1608, mlcod=121'1605)", 2024-09-17T16:16:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.329368+0000", 2024-09-17T16:16:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 292.18295647399998, 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.260516003, 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329368+0000", 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329368+0000", 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329369+0000", 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5400000000000001e-07 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329387+0000", 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8306000000000001e-05 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.588701+0000", 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.259314139 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.588719+0000", 2024-09-17T16:16:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8053000000000001e-05 2024-09-17T16:16:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.588863+0000", 2024-09-17T16:16:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014388899999999999 2024-09-17T16:16:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.588903+0000", 2024-09-17T16:16:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0469000000000002e-05 2024-09-17T16:16:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.589849+0000", 2024-09-17T16:16:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00094574899999999996 2024-09-17T16:16:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.589884+0000", 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4844e-05 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:47778 2.13 e142/112 2:cf64c7ca:::benchmark_data_smithi173_55121_object47777:head v 142'1859, mlcod=142'1857)", 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:10.556687+0000", 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 253.95563767300001, 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22771733999999999, 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.556687+0000", 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.556687+0000", 2024-09-17T16:16:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.556688+0000", 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8000000000000005e-07 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.556761+0000", 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3238000000000006e-05 2024-09-17T16:16:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.783341+0000", 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22657978500000001 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.783353+0000", 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2427e-05 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.783472+0000", 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011927200000000001 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.783506+0000", 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3176000000000002e-05 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.784367+0000", 2024-09-17T16:16:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086174799999999996 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:10.784404+0000", 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6913999999999999e-05 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:88568 2.19 2:9815a9f0:::benchmark_data_smithi173_55121_object12079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:16:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:43.869911+0000", 2024-09-17T16:16:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 220.64241315300001, 2024-09-17T16:16:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24823031700000001, 2024-09-17T16:16:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 88568 2024-09-17T16:16:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869911+0000", 2024-09-17T16:16:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869911+0000", 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869913+0000", 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.57e-06 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869918+0000", 2024-09-17T16:16:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7400000000000004e-06 2024-09-17T16:16:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869927+0000", 2024-09-17T16:16:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7710000000000001e-06 2024-09-17T16:16:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869933+0000", 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3129999999999998e-06 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.869973+0000", 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0599999999999998e-05 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.870044+0000", 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0654999999999994e-05 2024-09-17T16:16:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.870105+0000", 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0659999999999999e-05 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.871249+0000", 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001144064 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.872222+0000", 2024-09-17T16:16:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097360800000000005 2024-09-17T16:16:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.118065+0000", 2024-09-17T16:16:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24584251600000001 2024-09-17T16:16:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.118106+0000", 2024-09-17T16:16:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1556e-05 2024-09-17T16:16:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.118142+0000", 2024-09-17T16:16:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5264e-05 2024-09-17T16:16:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:94007 2.19 2:9e42b13d:::benchmark_data_smithi173_55121_object17518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:16:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:16:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 214.62463221499999, 2024-09-17T16:16:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.467128445, 2024-09-17T16:16:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 94007 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887693+0000", 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887700+0000", 2024-09-17T16:16:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-17T16:16:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887711+0000", 2024-09-17T16:16:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0739e-05 2024-09-17T16:16:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887716+0000", 2024-09-17T16:16:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2000000000000002e-06 2024-09-17T16:16:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887759+0000", 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2898999999999997e-05 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887839+0000", 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9500999999999997e-05 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887937+0000", 2024-09-17T16:16:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8531999999999998e-05 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.889187+0000", 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012492580000000001 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.889505+0000", 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031885699999999997 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354736+0000", 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46523019100000002 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354784+0000", 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8776999999999997e-05 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354821+0000", 2024-09-17T16:16:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6321e-05 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:105470 2.19 2:9b81cd1a:::benchmark_data_smithi173_55121_object28981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:02.872711+0000", 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 201.63961335400001, 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43773533999999997, 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 105470 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872711+0000", 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872711+0000", 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872713+0000", 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3689999999999999e-06 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872717+0000", 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9400000000000001e-06 2024-09-17T16:16:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872729+0000", 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1994000000000001e-05 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872735+0000", 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.711e-06 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872777+0000", 2024-09-17T16:16:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1894000000000002e-05 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872864+0000", 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7322000000000002e-05 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872942+0000", 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7465000000000005e-05 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874265+0000", 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0013235009999999999 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874550+0000", 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000285059 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310369+0000", 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43581851399999999 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310403+0000", 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3594999999999999e-05 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310447+0000", 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3976e-05 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:105470 2.19 e170/114 ondisk, result = 0)", 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:02.874849+0000", 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 201.63747581000001, 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43560358700000001, 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874849+0000", 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874849+0000", 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874850+0000", 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2130000000000001e-06 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874857+0000", 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9970000000000001e-06 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310301+0000", 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43544382999999998 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310308+0000", 2024-09-17T16:16:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9210000000000003e-06 2024-09-17T16:16:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310360+0000", 2024-09-17T16:16:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2383000000000003e-05 2024-09-17T16:16:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310366+0000", 2024-09-17T16:16:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4410000000000001e-06 2024-09-17T16:16:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310452+0000", 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6802e-05 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:152973 2.19 2:9e1f786b:::benchmark_data_smithi173_55121_object76484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 150.645287545, 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59801266799999997, 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 152973 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867039+0000", 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.581e-06 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867044+0000", 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0089999999999998e-06 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867057+0000", 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3658999999999999e-05 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867064+0000", 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2520000000000002e-06 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867120+0000", 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5937999999999999e-05 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867194+0000", 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3467000000000007e-05 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:16:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867254+0000", 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0510000000000002e-05 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.869276+0000", 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002021882 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.869436+0000", 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000159214 2024-09-17T16:16:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.464981+0000", 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59554505000000002 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.465020+0000", 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9674e-05 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.465050+0000", 2024-09-17T16:16:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9431999999999999e-05 2024-09-17T16:16:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25036.0:7257 3.7 e214/208 ondisk, result = 0)", 2024-09-17T16:16:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:14:19.965514+0000", 2024-09-17T16:16:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 124.546810063, 2024-09-17T16:16:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29917786299999999, 2024-09-17T16:16:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:16:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:19.965514+0000", 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:19.965514+0000", 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:19.965515+0000", 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9900000000000001e-07 2024-09-17T16:16:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:19.965518+0000", 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1e-06 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:20.264561+0000", 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29904340099999999 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:20.264568+0000", 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5119999999999997e-06 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:20.264610+0000", 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1563999999999997e-05 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:20.264615+0000", 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2100000000000001e-06 2024-09-17T16:16:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:20.264692+0000", 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7577000000000006e-05 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:8110 3.b 3:de8160e2:::benchmark_data_smithi173_66272_object8109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:14:34.926038+0000", 2024-09-17T16:16:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 109.586286626, 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29379559100000002, 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 8110 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926038+0000", 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926038+0000", 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926039+0000", 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.257e-06 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926084+0000", 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4758000000000001e-05 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926098+0000", 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4229e-05 2024-09-17T16:16:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926105+0000", 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2640000000000002e-06 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926162+0000", 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6264000000000002e-05 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926253+0000", 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1812999999999997e-05 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926348+0000", 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4568999999999995e-05 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926897+0000", 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054859500000000003 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.078517+0000", 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.15162066900000001 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219736+0000", 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.14121818799999999 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219772+0000", 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6078000000000003e-05 2024-09-17T16:16:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219834+0000", 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1907000000000002e-05 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:8122 3.2 3:4938ef6e:::benchmark_data_smithi173_66272_object8121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:14:34.933835+0000", 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 109.578489174, 2024-09-17T16:16:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28931940099999998, 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 8122 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933835+0000", 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933835+0000", 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933837+0000", 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8819999999999999e-06 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933884+0000", 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6499999999999999e-05 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933895+0000", 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1576000000000001e-05 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933900+0000", 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9289999999999997e-06 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933954+0000", 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3393999999999997e-05 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.934052+0000", 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7985000000000005e-05 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9", 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.934149+0000", 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7007000000000001e-05 2024-09-17T16:16:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.934624+0000", 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047536799999999999 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.222931+0000", 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28830738099999997 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.223078+0000", 2024-09-17T16:16:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014623899999999999 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.223112+0000", 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4405e-05 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.223155+0000", 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2735000000000003e-05 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(3.7 240/239 [PushReplyOp(3:e198b3f2:::benchmark_data_smithi173_66272_object478:head)])", 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.499171251000007, 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43133256199999997, 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:16:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013155+0000", 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:16:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013161+0000", 2024-09-17T16:16:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6830000000000002e-06 2024-09-17T16:16:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:16:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441372+0000", 2024-09-17T16:16:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42821048099999998 2024-09-17T16:16:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441385+0000", 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.311e-05 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441470+0000", 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5185000000000005e-05 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:16:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.444486+0000", 2024-09-17T16:16:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030157669999999999 2024-09-17T16:16:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:16:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:16:24.643 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:16:24.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:24.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2160232432' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:24 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 184/126822 objects degraded (0.145%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:24.671 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:24.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:24.772 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:24.802 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:24.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:24.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2160232432' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:24.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:24 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 184/126822 objects degraded (0.145%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:24.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:24.935 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:16:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:24.937 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:24.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:25.038 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:25.068 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:25.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:25.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:25.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:25.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:25.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:25.320 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:25.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:25.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:25.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:25.446 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:25.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:25.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:25.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:25.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:25.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:25 smithi173 ceph-mon[30530]: pgmap v631: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 970 op/s; 184/123753 objects degraded (0.149%); 5236/123753 objects misplaced (4.231%); 4.4 MiB/s, 69 objects/s recovering 2024-09-17T16:16:25.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:25.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:25 smithi017 ceph-mon[27571]: pgmap v631: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 970 op/s; 184/123753 objects degraded (0.149%); 5236/123753 objects misplaced (4.231%); 4.4 MiB/s, 69 objects/s recovering 2024-09-17T16:16:25.699 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:25.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:25.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:25.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:25 smithi071 ceph-mon[30002]: pgmap v631: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 970 op/s; 184/123753 objects degraded (0.149%); 5236/123753 objects misplaced (4.231%); 4.4 MiB/s, 69 objects/s recovering 2024-09-17T16:16:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:25.834 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:25.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:25.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:25.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:25.971 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:25.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:26.072 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:26.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:26.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:26.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:26.229 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:26.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:26.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:26.358 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:26.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:26.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:26.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:26.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:26.496 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:16:26.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:26.622 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:26.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:26.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:26.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:26.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:26.766 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:26.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:26.851 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:16:26.868 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:26.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:26.897 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:26.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:26.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:27.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:27.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:27.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:27.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:27.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:27.251 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:27.289 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:27.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:27.340 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:27.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:27.423 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:27.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:27.525 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:16:27.552 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:16:27.560 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:27.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:27 smithi071 ceph-mon[30002]: pgmap v632: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.4 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 972 op/s; 255/117210 objects degraded (0.218%); 4604/117210 objects misplaced (3.928%); 8.7 MiB/s, 139 objects/s recovering 2024-09-17T16:16:27.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:27 smithi173 ceph-mon[30530]: pgmap v632: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.4 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 972 op/s; 255/117210 objects degraded (0.218%); 4604/117210 objects misplaced (3.928%); 8.7 MiB/s, 139 objects/s recovering 2024-09-17T16:16:27.649 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:27 smithi017 ceph-mon[27571]: pgmap v632: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.4 GiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 972 op/s; 255/117210 objects degraded (0.218%); 4604/117210 objects misplaced (3.928%); 8.7 MiB/s, 139 objects/s recovering 2024-09-17T16:16:27.662 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:27.665 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:16:27.665 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:16:27.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:108826 3.e 3:7dbdc236:::benchmark_data_smithi173_66272_object36870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:27.644207+0000", 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.014582332, 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.014612811999999999, 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 108826 2024-09-17T16:16:27.666 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.644207+0000", 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.644207+0000", 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.667 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.644208+0000", 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0499999999999999e-06 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.644211+0000", 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.236e-06 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.644222+0000", 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0284e-05 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.644229+0000", 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5000000000000002e-06 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.645035+0000", 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00080553700000000005 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:27.669 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.645160+0000", 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012517100000000001 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.645272+0000", 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000111903 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.648185+0000", 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029132020000000002 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:27.670 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:108824 3.e 3:741724e9:::benchmark_data_smithi173_66272_object36868:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:27.639775+0000", 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.019014535999999999, 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.019118937999999999, 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:16:27.671 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 108824 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.639775+0000", 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.639775+0000", 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.639776+0000", 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.85e-07 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.639781+0000", 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0860000000000001e-06 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.639795+0000", 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3674e-05 2024-09-17T16:16:27.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.639802+0000", 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3250000000000002e-06 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.641801+0000", 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001998826 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.641910+0000", 2024-09-17T16:16:27.674 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000108911 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.642010+0000", 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0001008 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.645010+0000", 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029992030000000002 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:27.675 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:27.648398+0000", 2024-09-17T16:16:27.676 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0033884420000000002 2024-09-17T16:16:27.676 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:27.676 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:27.676 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:27.676 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:27.676 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:16:27.676 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 2 2024-09-17T16:16:27.676 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:16:27.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:27.692 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:27.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:27.737 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:16:27.793 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:27.825 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:27.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:27.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:27.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:27.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:28.053 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:28.083 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:28.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:28.185 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:28.221 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:28.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:28.323 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:28.356 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:28.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:28.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:28.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:28.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:16:28.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:28.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:28.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:28.616 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:28.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:28.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:28.743 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":632,"stamp":"2024-09-17T16:16:25.797115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2560885304,"num_objects":39070,"num_object_clones":0,"num_object_copies":117210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":255,"num_objects_misplaced":4604,"num_objects_unfound":0,"num_objects_dirty":39070,"num_whiteouts":0,"num_read":78,"num_read_kb":66,"num_write":104963,"num_write_kb":4607143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23308,"num_bytes_recovered":1528177728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104932,"ondisk_log_size":104932,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":9275236,"kb_used_data":7826540,"kb_used_omap":57,"kb_used_meta":1448454,"kb_avail":1021921436,"statfs":{"total":1055945392128,"available":1046447550464,"internally_reserved":0,"allocated":8014376960,"data_stored":8011249619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58657,"internal_metadata":1483217631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-509804544,"num_objects":-7779,"num_object_clones":0,"num_object_copies":-23337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":79,"num_objects_misplaced":-1197,"num_objects_unfound":0,"num_objects_dirty":-7779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7779,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":73269248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002553"},"pg_stats":[{"pgid":"3.f","version":"264'6483","reported_seq":12367,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:22.889778+0000","last_change":"2024-09-17T16:15:46.346980+0000","last_active":"2024-09-17T16:16:22.889778+0000","last_peered":"2024-09-17T16:16:22.889778+0000","last_clean":"2024-09-17T16:16:22.889778+0000","last_became_active":"2024-09-17T16:15:46.341307+0000","last_became_peered":"2024-09-17T16:15:46.341307+0000","last_unstale":"2024-09-17T16:16:22.889778+0000","last_undegraded":"2024-09-17T16:16:22.889778+0000","last_fullsized":"2024-09-17T16:16:22.889778+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6483,"log_dups_size":0,"ondisk_log_size":6483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164167680,"num_objects":2505,"num_object_clones":0,"num_object_copies":7515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6483,"num_write_kb":287616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2876,"num_bytes_recovered":188481536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,9],"acting":[7,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"264'6685","reported_seq":12693,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:24.478974+0000","last_change":"2024-09-17T16:15:45.346764+0000","last_active":"2024-09-17T16:16:24.478974+0000","last_peered":"2024-09-17T16:16:24.478974+0000","last_clean":"2024-09-17T16:16:24.478974+0000","last_became_active":"2024-09-17T16:15:45.341444+0000","last_became_peered":"2024-09-17T16:15:45.341444+0000","last_unstale":"2024-09-17T16:16:24.478974+0000","last_undegraded":"2024-09-17T16:16:24.478974+0000","last_fullsized":"2024-09-17T16:16:24.478974+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6685,"log_dups_size":0,"ondisk_log_size":6685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:54.837424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160628736,"num_objects":2451,"num_object_clones":0,"num_object_copies":7353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":292352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2937,"num_bytes_recovered":192479232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"264'6453","reported_seq":6620,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:22.880677+0000","last_change":"2024-09-17T16:15:58.646089+0000","last_active":"2024-09-17T16:16:22.880677+0000","last_peered":"2024-09-17T16:16:22.880677+0000","last_clean":"2024-09-17T16:16:22.880677+0000","last_became_active":"2024-09-17T16:14:08.487146+0000","last_became_peered":"2024-09-17T16:14:08.487146+0000","last_unstale":"2024-09-17T16:16:22.880677+0000","last_undegraded":"2024-09-17T16:16:22.880677+0000","last_fullsized":"2024-09-17T16:16:22.880677+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'5072","last_scrub_stamp":"2024-09-17T16:15:58.645836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:15:58.645836+0000","objects_scrubbed":3823,"log_size":6453,"log_dups_size":0,"ondisk_log_size":6453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:03.598654+0000","scrub_duration":678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158466072,"num_objects":2419,"num_object_clones":0,"num_object_copies":7257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":6453,"num_write_kb":283841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"264'6617","reported_seq":6738,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:25.424948+0000","last_change":"2024-09-17T16:14:08.389726+0000","last_active":"2024-09-17T16:16:25.424948+0000","last_peered":"2024-09-17T16:16:25.424948+0000","last_clean":"2024-09-17T16:16:25.424948+0000","last_became_active":"2024-09-17T16:14:08.389315+0000","last_became_peered":"2024-09-17T16:14:08.389315+0000","last_unstale":"2024-09-17T16:16:25.424948+0000","last_undegraded":"2024-09-17T16:16:25.424948+0000","last_fullsized":"2024-09-17T16:16:25.424948+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6617,"log_dups_size":0,"ondisk_log_size":6617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:51.788974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152633344,"num_objects":2329,"num_object_clones":0,"num_object_copies":6987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6617,"num_write_kb":286272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"263'6412","reported_seq":6532,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-17T16:16:21.267825+0000","last_change":"2024-09-17T16:14:08.429967+0000","last_active":"2024-09-17T16:16:21.267825+0000","last_peered":"2024-09-17T16:16:21.267825+0000","last_clean":"2024-09-17T16:16:21.267825+0000","last_became_active":"2024-09-17T16:14:08.429641+0000","last_became_peered":"2024-09-17T16:14:08.429641+0000","last_unstale":"2024-09-17T16:16:21.267825+0000","last_undegraded":"2024-09-17T16:16:21.267825+0000","last_fullsized":"2024-09-17T16:16:21.267825+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6412,"log_dups_size":0,"ondisk_log_size":6412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:51.309421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168296448,"num_objects":2568,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6412,"num_write_kb":287360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"264'6765","reported_seq":6889,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:25.533672+0000","last_change":"2024-09-17T16:14:08.401229+0000","last_active":"2024-09-17T16:16:25.533672+0000","last_peered":"2024-09-17T16:16:25.533672+0000","last_clean":"2024-09-17T16:16:25.533672+0000","last_became_active":"2024-09-17T16:14:08.400841+0000","last_became_peered":"2024-09-17T16:14:08.400841+0000","last_unstale":"2024-09-17T16:16:25.533672+0000","last_undegraded":"2024-09-17T16:16:25.533672+0000","last_fullsized":"2024-09-17T16:16:25.533672+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6765,"log_dups_size":0,"ondisk_log_size":6765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:12.654193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152502272,"num_objects":2327,"num_object_clones":0,"num_object_copies":6981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6765,"num_write_kb":290944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"264'6520","reported_seq":15292,"reported_epoch":264,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-17T16:16:24.481314+0000","last_change":"2024-09-17T16:15:58.065462+0000","last_active":"2024-09-17T16:16:24.481314+0000","last_peered":"2024-09-17T16:16:24.481314+0000","last_clean":"2024-09-17T16:15:55.844817+0000","last_became_active":"2024-09-17T16:15:57.995368+0000","last_became_peered":"2024-09-17T16:15:57.995368+0000","last_unstale":"2024-09-17T16:16:24.481314+0000","last_undegraded":"2024-09-17T16:15:57.880662+0000","last_fullsized":"2024-09-17T16:15:57.855955+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6520,"log_dups_size":0,"ondisk_log_size":6520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151519232,"num_objects":2312,"num_object_clones":0,"num_object_copies":6936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":105,"num_objects_misplaced":2286,"num_objects_unfound":0,"num_objects_dirty":2312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6520,"num_write_kb":282624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4317,"num_bytes_recovered":282918912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,5],"acting":[8,7],"avail_no_missing":["8","7","9","11"],"object_location_counts":[{"shards":"7,8","objects":-79},{"shards":"7,8,9","objects":2286}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"264'6321","reported_seq":12210,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:21.745048+0000","last_change":"2024-09-17T16:16:02.968795+0000","last_active":"2024-09-17T16:16:21.745048+0000","last_peered":"2024-09-17T16:16:21.745048+0000","last_clean":"2024-09-17T16:16:21.745048+0000","last_became_active":"2024-09-17T16:15:50.356572+0000","last_became_peered":"2024-09-17T16:15:50.356572+0000","last_unstale":"2024-09-17T16:16:21.745048+0000","last_undegraded":"2024-09-17T16:16:21.745048+0000","last_fullsized":"2024-09-17T16:16:21.745048+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'5214","last_scrub_stamp":"2024-09-17T16:16:02.968730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:16:02.968730+0000","objects_scrubbed":3616,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:24.624989+0000","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163250176,"num_objects":2491,"num_object_clones":0,"num_object_copies":7473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":281984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2855,"num_bytes_recovered":187039744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"263'6561","reported_seq":12633,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-17T16:16:21.268059+0000","last_change":"2024-09-17T16:15:51.350611+0000","last_active":"2024-09-17T16:16:21.268059+0000","last_peered":"2024-09-17T16:16:21.268059+0000","last_clean":"2024-09-17T16:16:21.268059+0000","last_became_active":"2024-09-17T16:15:51.344007+0000","last_became_peered":"2024-09-17T16:15:51.344007+0000","last_unstale":"2024-09-17T16:16:21.268059+0000","last_undegraded":"2024-09-17T16:16:21.268059+0000","last_fullsized":"2024-09-17T16:16:21.268059+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:53.300502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164560896,"num_objects":2511,"num_object_clones":0,"num_object_copies":7533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6561,"num_write_kb":290304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2971,"num_bytes_recovered":194707456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"264'6504","reported_seq":6662,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:21.740451+0000","last_change":"2024-09-17T16:15:14.282285+0000","last_active":"2024-09-17T16:16:21.740451+0000","last_peered":"2024-09-17T16:16:21.740451+0000","last_clean":"2024-09-17T16:16:21.740451+0000","last_became_active":"2024-09-17T16:14:08.429516+0000","last_became_peered":"2024-09-17T16:14:08.429516+0000","last_unstale":"2024-09-17T16:16:21.740451+0000","last_undegraded":"2024-09-17T16:16:21.740451+0000","last_fullsized":"2024-09-17T16:16:21.740451+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'2873","last_scrub_stamp":"2024-09-17T16:15:14.282192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:15:14.282192+0000","objects_scrubbed":2873,"log_size":6504,"log_dups_size":0,"ondisk_log_size":6504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:25.319637+0000","scrub_duration":649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169869312,"num_objects":2592,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6504,"num_write_kb":291072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"264'6686","reported_seq":6808,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:25.410292+0000","last_change":"2024-09-17T16:14:08.391300+0000","last_active":"2024-09-17T16:16:25.410292+0000","last_peered":"2024-09-17T16:16:25.410292+0000","last_clean":"2024-09-17T16:16:25.410292+0000","last_became_active":"2024-09-17T16:14:08.390988+0000","last_became_peered":"2024-09-17T16:14:08.390988+0000","last_unstale":"2024-09-17T16:16:25.410292+0000","last_undegraded":"2024-09-17T16:16:25.410292+0000","last_fullsized":"2024-09-17T16:16:25.410292+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6686,"log_dups_size":0,"ondisk_log_size":6686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:53.750323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151388160,"num_objects":2310,"num_object_clones":0,"num_object_copies":6930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6686,"num_write_kb":287872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"264'6473","reported_seq":12520,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:21.744136+0000","last_change":"2024-09-17T16:16:00.949803+0000","last_active":"2024-09-17T16:16:21.744136+0000","last_peered":"2024-09-17T16:16:21.744136+0000","last_clean":"2024-09-17T16:16:21.744136+0000","last_became_active":"2024-09-17T16:15:50.340714+0000","last_became_peered":"2024-09-17T16:15:50.340714+0000","last_unstale":"2024-09-17T16:16:21.744136+0000","last_undegraded":"2024-09-17T16:16:21.744136+0000","last_fullsized":"2024-09-17T16:16:21.744136+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'5216","last_scrub_stamp":"2024-09-17T16:16:00.949575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:16:00.949575+0000","objects_scrubbed":3776,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:26.505852+0000","scrub_duration":723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164954112,"num_objects":2517,"num_object_clones":0,"num_object_copies":7551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6473,"num_write_kb":287680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2935,"num_bytes_recovered":192348160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"264'6715","reported_seq":6839,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:24.543149+0000","last_change":"2024-09-17T16:14:08.430014+0000","last_active":"2024-09-17T16:16:24.543149+0000","last_peered":"2024-09-17T16:16:24.543149+0000","last_clean":"2024-09-17T16:16:24.543149+0000","last_became_active":"2024-09-17T16:14:08.429652+0000","last_became_peered":"2024-09-17T16:14:08.429652+0000","last_unstale":"2024-09-17T16:16:24.543149+0000","last_undegraded":"2024-09-17T16:16:24.543149+0000","last_fullsized":"2024-09-17T16:16:24.543149+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6715,"log_dups_size":0,"ondisk_log_size":6715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:16:51.949292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155648000,"num_objects":2375,"num_object_clones":0,"num_object_copies":7125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6715,"num_write_kb":290880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"264'6757","reported_seq":9836,"reported_epoch":264,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-17T16:16:25.704145+0000","last_change":"2024-09-17T16:15:58.104384+0000","last_active":"2024-09-17T16:16:25.704145+0000","last_peered":"2024-09-17T16:16:25.704145+0000","last_clean":"2024-09-17T16:15:55.823864+0000","last_became_active":"2024-09-17T16:15:58.059534+0000","last_became_peered":"2024-09-17T16:15:58.059534+0000","last_unstale":"2024-09-17T16:16:25.704145+0000","last_undegraded":"2024-09-17T16:15:57.882288+0000","last_fullsized":"2024-09-17T16:15:57.856865+0000","mapping_epoch":256,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6757,"log_dups_size":0,"ondisk_log_size":6757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153288704,"num_objects":2339,"num_object_clones":0,"num_object_copies":7017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":150,"num_objects_misplaced":2318,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6757,"num_write_kb":291072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1478,"num_bytes_recovered":96796672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,5],"acting":[10,0],"avail_no_missing":["10","0","4"],"object_location_counts":[{"shards":"0,4,10","objects":2318},{"shards":"0,10","objects":-129}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"264'6517","reported_seq":12565,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:22.223530+0000","last_change":"2024-09-17T16:16:02.459251+0000","last_active":"2024-09-17T16:16:22.223530+0000","last_peered":"2024-09-17T16:16:22.223530+0000","last_clean":"2024-09-17T16:16:22.223530+0000","last_became_active":"2024-09-17T16:16:00.879134+0000","last_became_peered":"2024-09-17T16:16:00.879134+0000","last_unstale":"2024-09-17T16:16:22.223530+0000","last_undegraded":"2024-09-17T16:16:22.223530+0000","last_fullsized":"2024-09-17T16:16:22.223530+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'5353","last_scrub_stamp":"2024-09-17T16:16:02.459059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:16:02.459059+0000","objects_scrubbed":3654,"log_size":6517,"log_dups_size":0,"ondisk_log_size":6517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:22.865264+0000","scrub_duration":896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161021952,"num_objects":2457,"num_object_clones":0,"num_object_copies":7371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6517,"num_write_kb":287168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2935,"num_bytes_recovered":192348160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,1],"acting":[9,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2","version":"263'6371","reported_seq":6492,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-17T16:16:21.249027+0000","last_change":"2024-09-17T16:14:08.487206+0000","last_active":"2024-09-17T16:16:21.249027+0000","last_peered":"2024-09-17T16:16:21.249027+0000","last_clean":"2024-09-17T16:16:21.249027+0000","last_became_active":"2024-09-17T16:14:08.486914+0000","last_became_peered":"2024-09-17T16:14:08.486914+0000","last_unstale":"2024-09-17T16:16:21.249027+0000","last_undegraded":"2024-09-17T16:16:21.249027+0000","last_fullsized":"2024-09-17T16:16:21.249027+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:14.294941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168099840,"num_objects":2565,"num_object_clones":0,"num_object_copies":7695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6371,"num_write_kb":285952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":407,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-17T16:16:21.362532+0000","last_change":"2024-09-17T16:15:13.851150+0000","last_active":"2024-09-17T16:16:21.362532+0000","last_peered":"2024-09-17T16:16:21.362532+0000","last_clean":"2024-09-17T16:16:21.362532+0000","last_became_active":"2024-09-17T16:15:13.845965+0000","last_became_peered":"2024-09-17T16:15:13.845965+0000","last_unstale":"2024-09-17T16:16:21.362532+0000","last_undegraded":"2024-09-17T16:16:21.362532+0000","last_fullsized":"2024-09-17T16:16:21.362532+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2560294936,"num_objects":39068,"num_object_clones":0,"num_object_copies":117204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":255,"num_objects_misplaced":4604,"num_objects_unfound":0,"num_objects_dirty":39068,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":104840,"num_write_kb":4604993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23304,"num_bytes_recovered":1527119872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8816242688,"data_stored":8816230472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104840,"ondisk_log_size":104840,"up":48,"acting":46,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":0,"seq":356482285651,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547287,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":231160,"kb_used_data":151400,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93513992,"statfs":{"total":95995035648,"available":95758327808,"internally_reserved":0,"allocated":155033600,"data_stored":154749761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":81656256},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743516,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1260616,"kb_used_data":1062836,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92484536,"statfs":{"total":95995035648,"available":94704164864,"internally_reserved":0,"allocated":1088344064,"data_stored":1088052601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":202504640},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005153,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416696,"kb_used_data":305000,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93328456,"statfs":{"total":95995035648,"available":95568338944,"internally_reserved":0,"allocated":312320000,"data_stored":312036161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963630,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":745032,"kb_used_data":624372,"kb_used_omap":41,"kb_used_meta":120598,"kb_avail":93000120,"statfs":{"total":95995035648,"available":95232122880,"internally_reserved":0,"allocated":639356928,"data_stored":639065465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":123492659},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495724,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860288,"kb_used_data":756912,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92884864,"statfs":{"total":95995035648,"available":95114100736,"internally_reserved":0,"allocated":775077888,"data_stored":774790497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724655,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894780,"kb_used_data":751148,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92850372,"statfs":{"total":95995035648,"available":95078780928,"internally_reserved":0,"allocated":769175552,"data_stored":768887641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986293,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1330104,"kb_used_data":1186472,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92415048,"statfs":{"total":95995035648,"available":94633009152,"internally_reserved":0,"allocated":1214947328,"data_stored":1214663489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":980460,"kb_used_data":840988,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92764692,"statfs":{"total":95995035648,"available":94991044608,"internally_reserved":0,"allocated":861171712,"data_stored":860892313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142801351},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411455,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":440812,"kb_used_data":329116,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93304340,"statfs":{"total":95995035648,"available":95543644160,"internally_reserved":0,"allocated":337014784,"data_stored":336735385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574980,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":976812,"kb_used_data":819292,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92768340,"statfs":{"total":95995035648,"available":94994780160,"internally_reserved":0,"allocated":838955008,"data_stored":838675609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1138476,"kb_used_data":999004,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92606676,"statfs":{"total":95995035648,"available":94829236224,"internally_reserved":0,"allocated":1022980096,"data_stored":1022700697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1020788736,"data_stored":1020788736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":827195392,"data_stored":827195392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331677696,"data_stored":331677696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":838664192,"data_stored":838664192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1196687360,"data_stored":1196687360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":753733632,"data_stored":753729560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":756940800,"data_stored":756940800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":637800448,"data_stored":637796376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":309985280,"data_stored":309985280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1073287168,"data_stored":1073283096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":153288704,"data_stored":153288704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":916193280,"data_stored":916193280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:16:28.745 INFO:tasks.thrashosds.thrasher:Setting 3.c to [8, 3, 1] 2024-09-17T16:16:28.745 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.c', '8', '3', '1'] 2024-09-17T16:16:28.745 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 3.c 8 3 1 2024-09-17T16:16:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:28.746 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:28.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:28.801 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:16:28.813 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:28.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:28.877 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:28.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:28.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:29.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:29.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:29.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:29.146 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:29.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:29.248 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:29.278 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:29.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:29.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:29.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:29.380 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:16:29.407 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:16:29.407 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:16:29.407 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:16:29.407 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:16:29.407 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:16:29.415 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:29.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:29.436 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:16:29.436 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:16:29.467 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:16:29.467 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:16:29.492 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:16:29.492 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:16:29.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:29 smithi017 ceph-mon[27571]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:16:29.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:29 smithi017 ceph-mon[27571]: pgmap v633: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 255/111186 objects degraded (0.229%); 4604/111186 objects misplaced (4.141%); 7.1 MiB/s, 113 objects/s recovering 2024-09-17T16:16:29.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1405588437' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:16:29.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:29 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:16:29.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:29 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 255/111186 objects degraded (0.229%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:29.516 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:16:29.516 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:16:29.517 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:29.541 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:16:29.542 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:16:29.543 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:29.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:29.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:29 smithi071 ceph-mon[30002]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:16:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:29 smithi071 ceph-mon[30002]: pgmap v633: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 255/111186 objects degraded (0.229%); 4604/111186 objects misplaced (4.141%); 7.1 MiB/s, 113 objects/s recovering 2024-09-17T16:16:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1405588437' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:16:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:29 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:16:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:29 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 255/111186 objects degraded (0.229%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:29.567 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:16:29.567 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:16:29.596 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:16:29.596 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:16:29.621 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:16:29.621 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:16:29.644 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:29 smithi173 ceph-mon[30530]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:16:29.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:29 smithi173 ceph-mon[30530]: pgmap v633: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.3 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 255/111186 objects degraded (0.229%); 4604/111186 objects misplaced (4.141%); 7.1 MiB/s, 113 objects/s recovering 2024-09-17T16:16:29.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1405588437' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:16:29.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:29 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:16:29.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:29 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 255/111186 objects degraded (0.229%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:29.646 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:16:29.646 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:16:29.670 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:29.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:29.672 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:16:29.672 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:16:29.697 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:16:29.698 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:16:29.726 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:16:29.726 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:16:29.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:29.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:29.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:29.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:29.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:29.922 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:29.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:30.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:30.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:30.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:30.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:16:30.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:30.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:30.182 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:30.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:30.283 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:30.324 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:30.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:30.392 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:30 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:30.393 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:30 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:16:30.393 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:30 smithi071 ceph-mon[30002]: osdmap e265: 12 total, 12 up, 11 in 2024-09-17T16:16:30.393 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2778945853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:30.393 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:30 smithi071 ceph-mon[30002]: pgmap v635: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 895 op/s; 255/106905 objects degraded (0.239%); 4317/106905 objects misplaced (4.038%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T16:16:30.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:30.449 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:30.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:30.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:30.496 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:16:30.551 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:30.582 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:30.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:30.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:30 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:30 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:16:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:30 smithi173 ceph-mon[30530]: osdmap e265: 12 total, 12 up, 11 in 2024-09-17T16:16:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2778945853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:30 smithi173 ceph-mon[30530]: pgmap v635: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 895 op/s; 255/106905 objects degraded (0.239%); 4317/106905 objects misplaced (4.038%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T16:16:30.683 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:30.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:30 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:30.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:30 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:16:30.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:30 smithi017 ceph-mon[27571]: osdmap e265: 12 total, 12 up, 11 in 2024-09-17T16:16:30.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2778945853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:30.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:30 smithi017 ceph-mon[27571]: pgmap v635: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 895 op/s; 255/106905 objects degraded (0.239%); 4317/106905 objects misplaced (4.038%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T16:16:30.710 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:30.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:30.726 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:30.812 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:30.837 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:30.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:30.939 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:30.963 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:30.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:31.058 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:31.058 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:31.065 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:31.089 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:31.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:31.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:31.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:31.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:31.319 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:31.347 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:31.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:31.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:31.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:31.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:31.577 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:31.608 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:31.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:31.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:31.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:31.737 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:31.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:31.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:16:31.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:31.866 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:31.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:31.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:31.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:31.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:32.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:32.116 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:32.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:32.154 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:16:32.218 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:32.245 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:32.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:32.348 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:32.379 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:32.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:32.480 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:32.518 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:32.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:32.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:32.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:32.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:16:32.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:32.774 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:32.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:32.782 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:32.782 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:32.859 INFO:teuthology.orchestra.run.smithi017.stderr:set 3.c pg_upmap mapping to [8,3,1] 2024-09-17T16:16:32.876 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:32.904 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:32.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:33.006 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:33.028 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:33.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:32 smithi071 ceph-mon[30002]: pgmap v636: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 940 op/s; 255/98613 objects degraded (0.259%); 4003/98613 objects misplaced (4.059%); 7.3 MiB/s, 116 objects/s recovering 2024-09-17T16:16:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/337620596' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.c", "id": [8, 3, 1]} : dispatch 2024-09-17T16:16:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:32 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.c", "id": [8, 3, 1]} : dispatch 2024-09-17T16:16:33.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:33.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:32 smithi173 ceph-mon[30530]: pgmap v636: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 940 op/s; 255/98613 objects degraded (0.259%); 4003/98613 objects misplaced (4.059%); 7.3 MiB/s, 116 objects/s recovering 2024-09-17T16:16:33.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/337620596' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.c", "id": [8, 3, 1]} : dispatch 2024-09-17T16:16:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:32 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.c", "id": [8, 3, 1]} : dispatch 2024-09-17T16:16:33.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:33.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:32 smithi017 ceph-mon[27571]: pgmap v636: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.0 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 940 op/s; 255/98613 objects degraded (0.259%); 4003/98613 objects misplaced (4.059%); 7.3 MiB/s, 116 objects/s recovering 2024-09-17T16:16:33.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/337620596' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.c", "id": [8, 3, 1]} : dispatch 2024-09-17T16:16:33.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:32 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.c", "id": [8, 3, 1]} : dispatch 2024-09-17T16:16:33.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:33.281 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:33.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:33.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:33.403 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:33.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:33.460 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:16:33.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:33.526 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:33.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:33.567 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:33.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:33.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:16:33.650 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:33.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:33.753 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:33.775 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:33.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:33.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:33.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:33.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:33.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:33.912 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:16:33.986 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:33 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [8, 3, 1]}]': finished 2024-09-17T16:16:33.987 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:33 smithi017 ceph-mon[27571]: osdmap e266: 12 total, 12 up, 11 in 2024-09-17T16:16:34.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:34.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:34.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:34.031 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:33 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [8, 3, 1]}]': finished 2024-09-17T16:16:34.031 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:33 smithi071 ceph-mon[30002]: osdmap e266: 12 total, 12 up, 11 in 2024-09-17T16:16:34.118 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:34.125 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:33 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [8, 3, 1]}]': finished 2024-09-17T16:16:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:33 smithi173 ceph-mon[30530]: osdmap e266: 12 total, 12 up, 11 in 2024-09-17T16:16:34.147 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:34.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:34.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:34.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:34.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:34.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:34.396 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:34.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:34.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:34.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:34.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:34.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:34.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:34.535 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:34.536 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:34.621 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:34.624 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:16:34.625 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:16:34.643 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:34.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:34.645 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:16:34.646 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:16:34.666 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:16:34.666 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:16:34.687 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:16:34.688 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:16:34.709 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:16:34.709 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:16:34.730 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:16:34.731 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:16:34.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:34.752 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:16:34.752 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:16:34.767 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:34.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:34.781 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:16:34.781 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:16:34.802 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:16:34.802 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:16:34.824 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:16:34.824 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:16:34.846 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:16:34.846 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:16:34.863 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:16:34.868 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:16:34.869 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:16:34.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:34.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:34.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:34.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:34.992 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:35.020 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:35.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:35.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: pgmap v638: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.0 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 884 op/s; 255/95970 objects degraded (0.266%); 4003/95970 objects misplaced (4.171%); 4.7 MiB/s, 75 objects/s recovering 2024-09-17T16:16:35.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: osdmap e267: 12 total, 12 up, 11 in 2024-09-17T16:16:35.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:16:35.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:16:35.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:16:35.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:16:35.111 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3381606609' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:35.111 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:16:35.111 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:16:35.111 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 255/95970 objects degraded (0.266%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:35.111 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/243635596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:35.111 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:16:35.111 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:16:35.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: pgmap v638: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.0 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 884 op/s; 255/95970 objects degraded (0.266%); 4003/95970 objects misplaced (4.171%); 4.7 MiB/s, 75 objects/s recovering 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: osdmap e267: 12 total, 12 up, 11 in 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3381606609' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 255/95970 objects degraded (0.266%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/243635596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:35.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:16:35.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:16:35.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:35.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:35.223 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: pgmap v638: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.0 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 884 op/s; 255/95970 objects degraded (0.266%); 4003/95970 objects misplaced (4.171%); 4.7 MiB/s, 75 objects/s recovering 2024-09-17T16:16:35.223 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: osdmap e267: 12 total, 12 up, 11 in 2024-09-17T16:16:35.223 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:16:35.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:16:35.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:16:35.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:16:35.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3381606609' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:35.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:16:35.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:16:35.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 255/95970 objects degraded (0.266%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:35.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/243635596' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:35.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:16:35.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:16:35.249 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:35.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:16:35.296 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:35.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:35.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:35.399 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:35.431 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:35.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:35.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:35.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:35.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:35.659 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:35.675 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:16:35.687 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:35.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:35.790 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:35.814 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:35.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:35.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:35.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:35.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:35.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:36.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:36.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:35 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:16:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3381606609' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:35 smithi071 ceph-mon[30002]: osdmap e268: 12 total, 12 up, 11 in 2024-09-17T16:16:36.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:36.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:36.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:36.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:35 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:16:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3381606609' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:35 smithi173 ceph-mon[30530]: osdmap e268: 12 total, 12 up, 11 in 2024-09-17T16:16:36.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:36.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:36.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:36.236 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:16:36.236 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:16:36.236 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:16:36.236 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:16:36.236 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.236 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:21741 2.3 2:c4bdb6e7:::benchmark_data_smithi173_55121_object21740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:16:36.236 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:16:36.236 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 309.56546523999998, 2024-09-17T16:16:36.236 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55429691999999997, 2024-09-17T16:16:36.237 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.237 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.237 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.237 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.237 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.237 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.237 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 21741 2024-09-17T16:16:36.237 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.237 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.237 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662972+0000", 2024-09-17T16:16:36.238 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.722e-06 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663024+0000", 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2175000000000001e-05 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663036+0000", 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2131e-05 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.239 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663042+0000", 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9649999999999996e-06 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663084+0000", 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2354999999999997e-05 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.240 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663166+0000", 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1699999999999994e-05 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663249+0000", 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2714999999999997e-05 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663804+0000", 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055499599999999998 2024-09-17T16:16:36.241 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.664173+0000", 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000369181 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217193+0000", 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55302014600000005 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217234+0000", 2024-09-17T16:16:36.242 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1149000000000002e-05 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217267+0000", 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2685000000000003e-05 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:23782 2.2 2:41257b37:::benchmark_data_smithi173_55121_object23781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:16:36.243 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 307.540146961, 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51828190799999996, 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 23782 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.244 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688289+0000", 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7400000000000002e-07 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.245 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688376+0000", 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6744999999999997e-05 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688387+0000", 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0499000000000001e-05 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.246 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688393+0000", 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9689999999999997e-06 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688434+0000", 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0694e-05 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688511+0000", 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6656000000000006e-05 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.247 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688594+0000", 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2682999999999994e-05 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689092+0000", 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049850100000000004 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689369+0000", 2024-09-17T16:16:36.248 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027721100000000003 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206489+0000", 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51712006899999996 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206524+0000", 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4516e-05 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.249 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206570+0000", 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6490999999999999e-05 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:23782 2.2 e119/104 ondisk, result = 0)", 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 307.53905417599998, 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51719464199999998, 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.250 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.251 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:16:36.251 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.251 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.251 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.251 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:16:36.251 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689382+0000", 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.252 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689385+0000", 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3569999999999998e-06 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206448+0000", 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51706313699999995 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206453+0000", 2024-09-17T16:16:36.253 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4360000000000004e-06 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206480+0000", 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7770999999999999e-05 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206486+0000", 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6779999999999997e-06 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.254 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206576+0000", 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9701000000000001e-05 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24369 2.3 2:c67d9130:::benchmark_data_smithi173_55121_object24368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 306.54092764000001, 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55710953100000005, 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.255 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24369 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.256 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687509+0000", 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1399999999999995e-07 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687596+0000", 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7288999999999997e-05 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.257 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687607+0000", 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1505000000000001e-05 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687613+0000", 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1440000000000002e-06 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687655+0000", 2024-09-17T16:16:36.258 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2240000000000002e-05 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687782+0000", 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000127043 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687864+0000", 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2064000000000006e-05 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.259 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688444+0000", 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058032600000000004 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688721+0000", 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000276734 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244551+0000", 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55582960199999998 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.260 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244588+0000", 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7004e-05 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244617+0000", 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9666e-05 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.261 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:24369 2.3 e120/104 ondisk, result = 0)", 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 306.53972044900001, 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55590799999999996, 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.262 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688716+0000", 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8999999999999996e-07 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688719+0000", 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5980000000000001e-06 2024-09-17T16:16:36.263 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244478+0000", 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55575852299999995 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244483+0000", 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244541+0000", 2024-09-17T16:16:36.264 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8377e-05 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244547+0000", 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8980000000000001e-06 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244623+0000", 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5780999999999998e-05 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.265 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:29877 2.2 2:40729eb6:::benchmark_data_smithi173_55121_object29876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 286.02456606300001, 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.874997425, 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 29877 2024-09-17T16:16:36.266 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.267 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203870+0000", 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203910+0000", 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9477999999999997e-05 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203918+0000", 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:16:36.268 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203922+0000", 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6369999999999998e-06 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203958+0000", 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5045000000000001e-05 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.269 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204026+0000", 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8693000000000001e-05 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204105+0000", 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8458e-05 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204758+0000", 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00065323300000000003 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.270 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.205073+0000", 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031491100000000002 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078782+0000", 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87370871400000005 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078827+0000", 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5219999999999997e-05 2024-09-17T16:16:36.271 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078867+0000", 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9953000000000002e-05 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:88652 2.2 2:45ffdfe0:::benchmark_data_smithi173_55121_object12163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 232.27168046899999, 2024-09-17T16:16:36.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29260301500000002, 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 88652 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956757+0000", 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7859999999999999e-06 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956762+0000", 2024-09-17T16:16:36.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.31e-06 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956773+0000", 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0832999999999999e-05 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956778+0000", 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2320000000000001e-06 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956829+0000", 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1124e-05 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956915+0000", 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5541e-05 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.957003+0000", 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8332999999999996e-05 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.958148+0000", 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001144937 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.961994+0000", 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038457320000000001 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249267+0000", 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28727286400000002 2024-09-17T16:16:36.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249320+0000", 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3016000000000002e-05 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249358+0000", 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8306999999999998e-05 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(2.3 158/155 [PushReplyOp(2:c6916317:::benchmark_data_smithi173_55121_object76243:head)])", 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 227.50538821999999, 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45253040700000002, 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:16:36.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.281 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:35 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:16:36.281 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3381606609' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:36.281 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:35 smithi017 ceph-mon[27571]: osdmap e268: 12 total, 12 up, 11 in 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723048+0000", 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4300000000000003e-07 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723052+0000", 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9310000000000001e-06 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175226+0000", 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45217462899999999 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175233+0000", 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4099999999999996e-06 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175322+0000", 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9011000000000005e-05 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175578+0000", 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025588299999999999 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:94077 2.16 2:6e055a89:::benchmark_data_smithi173_55121_object17588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 226.14996259700001, 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28469862699999998, 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.284 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 94077 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078474+0000", 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.274e-06 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078479+0000", 2024-09-17T16:16:36.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6310000000000002e-06 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078490+0000", 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1799e-05 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078496+0000", 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7579999999999998e-06 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078540+0000", 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3686999999999997e-05 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078624+0000", 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4153000000000006e-05 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078694+0000", 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9533e-05 2024-09-17T16:16:36.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.079805+0000", 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011117550000000001 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.085499+0000", 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0056938969999999998 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363107+0000", 2024-09-17T16:16:36.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27760825700000002 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363140+0000", 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2055999999999999e-05 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363171+0000", 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1826999999999999e-05 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97425 2.3 2:c4562d4f:::benchmark_data_smithi173_55121_object20936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 222.38136424300001, 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29697053200000001, 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.291 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97425 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847072+0000", 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1399999999999997e-07 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847074+0000", 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7329999999999999e-06 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847082+0000", 2024-09-17T16:16:36.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6830000000000008e-06 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847088+0000", 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0499999999999997e-06 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848028+0000", 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093989799999999999 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848133+0000", 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000104874 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848231+0000", 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8121999999999993e-05 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.849551+0000", 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00131985 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143633+0000", 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29408256900000002 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143940+0000", 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030690399999999999 2024-09-17T16:16:36.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143994+0000", 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3161999999999997e-05 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.144042+0000", 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8072999999999999e-05 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.297 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:129838 2.2 2:4555d77c:::benchmark_data_smithi173_55121_object53349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 188.44190373199999, 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.288916852, 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.298 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 129838 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:16:36.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786533+0000", 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.728e-06 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786540+0000", 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4320000000000004e-06 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786556+0000", 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6302000000000001e-05 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786563+0000", 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.934e-06 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786626+0000", 2024-09-17T16:16:36.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2988000000000001e-05 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786633+0000", 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6150000000000002e-06 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069502+0000", 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.282869606 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069572+0000", 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0201000000000001e-05 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069639+0000", 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6323999999999997e-05 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075296+0000", 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0056576639999999997 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075309+0000", 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2621e-05 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075356+0000", 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6653000000000002e-05 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075405+0000", 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9438000000000003e-05 2024-09-17T16:16:36.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075448+0000", 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3346000000000001e-05 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:129850 2.16 2:6f12710d:::benchmark_data_smithi173_55121_object53361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 188.42608892300001, 2024-09-17T16:16:36.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27163233199999998, 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 129850 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802348+0000", 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0549999999999998e-06 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802356+0000", 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6539999999999999e-06 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802372+0000", 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6314999999999999e-05 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802379+0000", 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0419999999999997e-06 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802428+0000", 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8367e-05 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802433+0000", 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6519999999999997e-06 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069491+0000", 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26705805599999999 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069572+0000", 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1174999999999994e-05 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069639+0000", 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6946000000000006e-05 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073713+0000", 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040741709999999997 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073743+0000", 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9740999999999999e-05 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073843+0000", 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100497 2024-09-17T16:16:36.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073930+0000", 2024-09-17T16:16:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6835999999999993e-05 2024-09-17T16:16:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073979+0000", 2024-09-17T16:16:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8825000000000001e-05 2024-09-17T16:16:36.312 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:150176 2.16 2:6f747f77:::benchmark_data_smithi173_55121_object73687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:50.480487+0000", 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 165.74794841400001, 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25565104700000002, 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.313 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 150176 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480487+0000", 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480487+0000", 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480488+0000", 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.06e-06 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480493+0000", 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6700000000000002e-06 2024-09-17T16:16:36.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480503+0000", 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0052e-05 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480510+0000", 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8820000000000003e-06 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480554+0000", 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4697000000000003e-05 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480627+0000", 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2498999999999998e-05 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.480688+0000", 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1081999999999996e-05 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.482798+0000", 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021097479999999998 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.483396+0000", 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059859300000000002 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.736057+0000", 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25266103699999998 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.736109+0000", 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.168e-05 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.736138+0000", 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9047000000000002e-05 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:150269 2.16 2:6d6b4de7:::benchmark_data_smithi173_55121_object73780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 165.63990164399999, 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29341146400000001, 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 150269 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588535+0000", 2024-09-17T16:16:36.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.751e-06 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588542+0000", 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7479999999999996e-06 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588558+0000", 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5885000000000001e-05 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588582+0000", 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4042000000000001e-05 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588646+0000", 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4103999999999995e-05 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588789+0000", 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000143094 2024-09-17T16:16:36.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588910+0000", 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120751 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.590898+0000", 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001988058 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.590948+0000", 2024-09-17T16:16:36.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9959000000000001e-05 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881725+0000", 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29077654800000002 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881900+0000", 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017578599999999999 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881945+0000", 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4737999999999998e-05 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:152907 2.3 2:c65c8e4e:::benchmark_data_smithi173_55121_object76418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 162.43101557899999, 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87037030500000001, 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.326 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152907 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4099999999999999e-07 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797423+0000", 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:16:36.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797430+0000", 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1119999999999998e-06 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797436+0000", 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6189999999999999e-06 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797479+0000", 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3050999999999998e-05 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797597+0000", 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000117637 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797696+0000", 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9079999999999993e-05 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.799730+0000", 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020345239999999998 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.801020+0000", 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012897270000000001 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667698+0000", 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.866677794 2024-09-17T16:16:36.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667733+0000", 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4953999999999998e-05 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667790+0000", 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7265999999999997e-05 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25036.0:17923 3.c e223/208 ondisk, result = 0)", 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:14:41.811355+0000", 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 114.417080481, 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25393519799999997, 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.811355+0000", 2024-09-17T16:16:36.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.811355+0000", 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.811355+0000", 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:41.811358+0000", 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.027e-06 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.065167+0000", 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25380819199999999 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.065172+0000", 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.677e-06 2024-09-17T16:16:36.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.065217+0000", 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4876999999999997e-05 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.065223+0000", 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4410000000000001e-06 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:42.065290+0000", 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7422000000000006e-05 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(3.8 249/239 [PushOp(3:153a1894:::benchmark_data_smithi173_66272_object39120:head, version: 230'2406, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:153a1894:::benchmark_data_smithi173_66272_object39120:head@230'2406, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:45.102605+0000", 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 51.125830006000001, 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25303556599999999, 2024-09-17T16:16:36.337 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.102605+0000", 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.102605+0000", 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.102607+0000", 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9139999999999998e-06 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.102686+0000", 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8385000000000003e-05 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.354223+0000", 2024-09-17T16:16:36.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25153691 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.354234+0000", 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.097e-05 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.354324+0000", 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0335999999999997e-05 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.354338+0000", 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3756e-05 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.355641+0000", 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013032950000000001 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(3.8 250/239 [PushOp(3:1a7a7bf1:::benchmark_data_smithi173_66272_object39562:head, version: 230'2428, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a7a7bf1:::benchmark_data_smithi173_66272_object39562:head@230'2428, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:45.409030+0000", 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 50.819404984000002, 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.263688647, 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409030+0000", 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409030+0000", 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409032+0000", 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.513e-06 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409173+0000", 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000140915 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671742+0000", 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26256918299999998 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671788+0000", 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5719000000000001e-05 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671881+0000", 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3509000000000002e-05 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671890+0000", 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5290000000000008e-06 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.672719+0000", 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00082927900000000004 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:81509 3.5 3:ac456fc5:::benchmark_data_smithi173_66272_object9553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:56.268230+0000", 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 39.960205772000002, 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27300864699999999, 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.346 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 81509 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268230+0000", 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268230+0000", 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268231+0000", 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3260000000000001e-06 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268236+0000", 2024-09-17T16:16:36.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8040000000000001e-06 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268248+0000", 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2014999999999999e-05 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268253+0000", 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4500000000000003e-06 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268296+0000", 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2879000000000001e-05 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268370+0000", 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3892000000000003e-05 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268433+0000", 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2953999999999994e-05 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270351+0000", 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00191843 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270909+0000", 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055809299999999996 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541148+0000", 2024-09-17T16:16:36.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27023824299999999 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541188+0000", 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0006000000000003e-05 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541238+0000", 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0555e-05 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25036.0:81509 3.5 e255/208 ondisk, result = 0)", 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:56.270755+0000", 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 39.957679931999998, 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27048895299999998, 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270755+0000", 2024-09-17T16:16:36.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270755+0000", 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270756+0000", 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7499999999999998e-07 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270762+0000", 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9649999999999996e-06 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541053+0000", 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27029036699999998 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541078+0000", 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5367e-05 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541134+0000", 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6329000000000002e-05 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541143+0000", 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3059999999999998e-06 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541244+0000", 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000101644 2024-09-17T16:16:36.356 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.357 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.357 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.357 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:36.357 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:36.357 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:16:36.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:36.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:36.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:36.481 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:36.509 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:36.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:36.611 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:36.640 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:36.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:36.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:36.768 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:36.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:36.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:36.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:36.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:36 smithi017 ceph-mon[27571]: pgmap v641: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.9 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 788 op/s; 257/92712 objects degraded (0.277%); 3343/92712 objects misplaced (3.606%); 10 MiB/s, 161 objects/s recovering 2024-09-17T16:16:36.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:37.024 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:37.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:37.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:37.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:37.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:36 smithi173 ceph-mon[30530]: pgmap v641: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.9 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 788 op/s; 257/92712 objects degraded (0.277%); 3343/92712 objects misplaced (3.606%); 10 MiB/s, 161 objects/s recovering 2024-09-17T16:16:37.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:37.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:37.171 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:36 smithi071 ceph-mon[30002]: pgmap v641: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.9 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 788 op/s; 257/92712 objects degraded (0.277%); 3343/92712 objects misplaced (3.606%); 10 MiB/s, 161 objects/s recovering 2024-09-17T16:16:37.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:16:37.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:37.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:37.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:37.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:37.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:37.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:37.505 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:37.529 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:37.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:37.546 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:16:37.631 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:37.661 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:37.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:37.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:37.795 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:37.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:37.897 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:37.920 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:37.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:38.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:38.052 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:38.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:38.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:38.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:38.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:38.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:38.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:38.306 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:38.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:38.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:16:38.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:38.435 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:38.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:38.493 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:16:38.493 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:16:38.493 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-17T16:16:38.493 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T16:16:38.494 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T16:16:38.494 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-17T16:16:38.537 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:38.582 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:38.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:38.683 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:38.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:38.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:38.762 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:38.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:38.849 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:38.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:38.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:16:38.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:38.952 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:16:38.952 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:16:38.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.952 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:116577 3.a 3:530d6efa:::benchmark_data_smithi173_66272_object44621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-17T16:16:38.952 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:38.940804+0000", 2024-09-17T16:16:38.952 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0089903029999999998, 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0090123129999999992, 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 116577 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:16:38.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.940804+0000", 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.940804+0000", 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.940805+0000", 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.705e-06 2024-09-17T16:16:38.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.940812+0000", 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6930000000000001e-06 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.940828+0000", 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.645e-05 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.940836+0000", 2024-09-17T16:16:38.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0570000000000004e-06 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.940897+0000", 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1029999999999997e-05 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.941014+0000", 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011791800000000001 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,4", 2024-09-17T16:16:38.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.941132+0000", 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000117435 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.945324+0000", 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0041921390000000001 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:38.945414+0000", 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0191000000000004e-05 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:16:38.957 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:38.958 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:16:38.958 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:16:38.958 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 1 2024-09-17T16:16:38.958 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:16:38.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:38.974 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:39.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:39.107 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:39.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:39.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:38 smithi173 ceph-mon[30530]: pgmap v642: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.8 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 257/88917 objects degraded (0.289%); 7041/88917 objects misplaced (7.919%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:16:39.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:38 smithi017 ceph-mon[27571]: pgmap v642: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.8 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 257/88917 objects degraded (0.289%); 7041/88917 objects misplaced (7.919%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:16:39.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:39.237 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:39.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:38 smithi071 ceph-mon[30002]: pgmap v642: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.8 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 257/88917 objects degraded (0.289%); 7041/88917 objects misplaced (7.919%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:16:39.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:39.383 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:39.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:39.484 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:39.515 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:39.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:39.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:39.617 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:39.647 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:39.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:39.680 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:39.719 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:16:39.719 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:16:39.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:39.755 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:16:39.755 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:16:39.772 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:39.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:39.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:39.785 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:16:39.786 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:16:39.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:16:39.816 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:16:39.842 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:16:39.842 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:16:39.873 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:16:39.873 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:16:39.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:39.901 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:16:39.902 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:16:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:39.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:39.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:39.927 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:16:39.927 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:16:39.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:16:39.961 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:16:39.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:16:39.993 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:16:39.993 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:16:40.003 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:40.039 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:16:40.039 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:16:40.043 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:40.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:40.074 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:16:40.074 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:16:40.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:39 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 257/88917 objects degraded (0.289%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:40.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4061048730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:40.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:39 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 257/88917 objects degraded (0.289%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4061048730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:40.145 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:40.179 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:40.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:40.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:40.298 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:16:40.298 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T16:16:40.298 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:16:40.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:40.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:40.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:40.308 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:16:40.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:39 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 257/88917 objects degraded (0.289%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:40.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4061048730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:40.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:40.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:40.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:40.538 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:40.565 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:40.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:40.667 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:40.693 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:40.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:40.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:40.823 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:40.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:40.884 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:40.884 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:40.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:40.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:40.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:41.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:40 smithi071 ceph-mon[30002]: pgmap v643: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.8 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 257/86235 objects degraded (0.298%); 6703/86235 objects misplaced (7.773%); 9.6 MiB/s, 153 objects/s recovering 2024-09-17T16:16:41.077 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:41.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:40 smithi173 ceph-mon[30530]: pgmap v643: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.8 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 257/86235 objects degraded (0.298%); 6703/86235 objects misplaced (7.773%); 9.6 MiB/s, 153 objects/s recovering 2024-09-17T16:16:41.179 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:41.184 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:40 smithi017 ceph-mon[27571]: pgmap v643: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.8 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 257/86235 objects degraded (0.298%); 6703/86235 objects misplaced (7.773%); 9.6 MiB/s, 153 objects/s recovering 2024-09-17T16:16:41.206 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:41.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:41.308 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:41.338 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:41.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:41.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:41.440 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:41.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:41.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:41.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:41.604 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:41.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:41.656 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:41.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:41.732 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:41.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:41.834 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:41.865 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:41.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:41.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:41.998 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:41.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:42.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:16:42.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:42.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:42.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:42.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:42.260 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:42.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:42.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:42.362 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:42.389 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:42.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:42.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:42.397 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:16:42.491 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:42.518 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:42.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:42.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:42.645 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:42.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:42.747 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:42.776 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:42.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:42.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:42.907 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:42.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:42.931 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:42.931 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:43.008 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:43.035 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:43.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:43.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:42 smithi071 ceph-mon[30002]: pgmap v644: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.6 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 257/78840 objects degraded (0.326%); 5933/78840 objects misplaced (7.525%); 13 MiB/s, 200 objects/s recovering 2024-09-17T16:16:43.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:43.166 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:43.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:43.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:42 smithi017 ceph-mon[27571]: pgmap v644: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.6 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 257/78840 objects degraded (0.326%); 5933/78840 objects misplaced (7.525%); 13 MiB/s, 200 objects/s recovering 2024-09-17T16:16:43.267 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:43.294 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:43.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:43.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:42 smithi173 ceph-mon[30530]: pgmap v644: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.6 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 257/78840 objects degraded (0.326%); 5933/78840 objects misplaced (7.525%); 13 MiB/s, 200 objects/s recovering 2024-09-17T16:16:43.396 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:43.421 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:43.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:43.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:43.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:43.550 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:43.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:43.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:43.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:43.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:43.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:43.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:43.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:43.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:43.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:43.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:43.904 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:16:43.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:43.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:43.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:16:44.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:44.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:44.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:44.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:44.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:44.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:44.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:16:44.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/972291568' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:16:44.275 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:16:44.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:44.304 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:44.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:44.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/972291568' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:16:44.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/972291568' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:16:44.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:44.433 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:44.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:44.535 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:44.569 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:44.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:44.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:44.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:44.698 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:44.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:44.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:44.800 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:44.815 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:44.816 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:44.822 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:44.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:44.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:44.879 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:44.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:44.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:44.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:44 smithi017 ceph-mon[27571]: pgmap v645: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.6 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 257/76653 objects degraded (0.335%); 5933/76653 objects misplaced (7.740%); 10 MiB/s, 161 objects/s recovering 2024-09-17T16:16:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:44 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/972291568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:16:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:44 smithi017 ceph-mon[27571]: osdmap e269: 12 total, 12 up, 11 in 2024-09-17T16:16:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:44 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 257/76653 objects degraded (0.335%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:44 smithi017 ceph-mon[27571]: osdmap e270: 12 total, 12 up, 11 in 2024-09-17T16:16:45.050 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:45.076 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:45.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:45.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:44 smithi173 ceph-mon[30530]: pgmap v645: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.6 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 257/76653 objects degraded (0.335%); 5933/76653 objects misplaced (7.740%); 10 MiB/s, 161 objects/s recovering 2024-09-17T16:16:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:44 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/972291568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:16:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:44 smithi173 ceph-mon[30530]: osdmap e269: 12 total, 12 up, 11 in 2024-09-17T16:16:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:44 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 257/76653 objects degraded (0.335%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:44 smithi173 ceph-mon[30530]: osdmap e270: 12 total, 12 up, 11 in 2024-09-17T16:16:45.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:45.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:45.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:45.303 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:44 smithi071 ceph-mon[30002]: pgmap v645: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.6 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 257/76653 objects degraded (0.335%); 5933/76653 objects misplaced (7.740%); 10 MiB/s, 161 objects/s recovering 2024-09-17T16:16:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:44 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/972291568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:16:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:44 smithi071 ceph-mon[30002]: osdmap e269: 12 total, 12 up, 11 in 2024-09-17T16:16:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:44 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 257/76653 objects degraded (0.335%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:44 smithi071 ceph-mon[30002]: osdmap e270: 12 total, 12 up, 11 in 2024-09-17T16:16:45.334 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:45.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:45.436 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:45.471 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:45.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:45.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:16:45.513 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:16:45.513 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:16:45.548 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:16:45.549 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:16:45.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:45.582 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:16:45.582 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:16:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:45.608 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:45.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:45.618 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:16:45.618 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:16:45.648 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:16:45.649 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:16:45.680 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:16:45.681 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:16:45.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:45.714 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:16:45.715 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:16:45.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:45.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:45.748 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:16:45.748 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:16:45.783 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:16:45.783 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:16:45.816 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:16:45.817 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:16:45.817 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:16:45.838 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:45.849 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:16:45.849 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:16:45.867 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:45.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:45.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:45.878 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:16:45.878 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:16:45.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:45.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:45.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:46.095 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:46.121 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:46.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:46.213 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:45 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3520196191' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:46.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:46.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:45 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3520196191' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:46.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:46.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:46.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:46.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:45 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3520196191' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:46.344 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:16:46.345 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:16:46.345 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:16:46.345 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:16:46.345 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:16:46.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:46.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:46.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:46.488 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:46.522 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:46.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:46.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:46.638 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:16:46.638 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:16:46.639 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-09-17T16:16:46.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd in 11 2024-09-17T16:16:46.651 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:46.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:46.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:46.752 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:46.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:16:46.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:46.800 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:46.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:46.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:46.955 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:46.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:46.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:46 smithi017 ceph-mon[27571]: pgmap v648: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.5 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 691 op/s; 256/72306 objects degraded (0.354%); 5213/72306 objects misplaced (7.210%); 12 MiB/s, 199 objects/s recovering 2024-09-17T16:16:47.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:47.088 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:47.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:47.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:46 smithi173 ceph-mon[30530]: pgmap v648: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.5 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 691 op/s; 256/72306 objects degraded (0.354%); 5213/72306 objects misplaced (7.210%); 12 MiB/s, 199 objects/s recovering 2024-09-17T16:16:47.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:47.220 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:47.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:47.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:46 smithi071 ceph-mon[30002]: pgmap v648: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.5 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 691 op/s; 256/72306 objects degraded (0.354%); 5213/72306 objects misplaced (7.210%); 12 MiB/s, 199 objects/s recovering 2024-09-17T16:16:47.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:47.351 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:47.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:47.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:16:47.453 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:47.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:47.480 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:47.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:47.581 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:47.610 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:47.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:47.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:47.693 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:16:47.701 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:47.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:47.735 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:47.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:47.749 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:47.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:47.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:47.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:47.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:47.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:47.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:47.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:48.082 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:47 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:16:48.082 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:47 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:16:48.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:48.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:48.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:48.133 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:47 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:16:48.133 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:47 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:16:48.215 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:16:48.215 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:16:48.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:16:48.215 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:16:48.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.215 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:13514 2.f 2:f9c4878d:::benchmark_data_smithi173_55121_object13513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:12.318561+0000", 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 335.89301247399999, 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34231690599999998, 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 13514 2024-09-17T16:16:48.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318561+0000", 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318561+0000", 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318562+0000", 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318587+0000", 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4413000000000002e-05 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318599+0000", 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2449999999999999e-05 2024-09-17T16:16:48.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318605+0000", 2024-09-17T16:16:48.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0059999999999998e-06 2024-09-17T16:16:48.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318661+0000", 2024-09-17T16:16:48.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5299e-05 2024-09-17T16:16:48.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318781+0000", 2024-09-17T16:16:48.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120522 2024-09-17T16:16:48.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:16:48.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.318927+0000", 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000145594 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.319619+0000", 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000692777 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.341876+0000", 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.022256811000000001 2024-09-17T16:16:48.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.660723+0000", 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31884684699999999 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.660839+0000", 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115951 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.660878+0000", 2024-09-17T16:16:48.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8899000000000001e-05 2024-09-17T16:16:48.223 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.223 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.223 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.223 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:13492 2.15 2:a809b404:::benchmark_data_smithi173_55121_object13491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:16:48.223 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:12.334855+0000", 2024-09-17T16:16:48.223 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 335.876718633, 2024-09-17T16:16:48.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.019215271, 2024-09-17T16:16:48.223 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 13492 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334855+0000", 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334855+0000", 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334856+0000", 2024-09-17T16:16:48.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6700000000000002e-07 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334879+0000", 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3359000000000001e-05 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334901+0000", 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2294999999999999e-05 2024-09-17T16:16:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334911+0000", 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3649999999999993e-06 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.343520+0000", 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0086088640000000008 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:16:48.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.343525+0000", 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9930000000000003e-06 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.353911+0000", 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0103866720000001 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.354070+0000", 2024-09-17T16:16:48.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000158856 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24380 2.f 2:ff1cebfd:::benchmark_data_smithi173_55121_object24379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.697276+0000", 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.51429747200001, 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39030066800000002, 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.229 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24380 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697276+0000", 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697276+0000", 2024-09-17T16:16:48.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697277+0000", 2024-09-17T16:16:48.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.13e-06 2024-09-17T16:16:48.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.231 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:48.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697302+0000", 2024-09-17T16:16:48.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4533999999999999e-05 2024-09-17T16:16:48.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697316+0000", 2024-09-17T16:16:48.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4592e-05 2024-09-17T16:16:48.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697324+0000", 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9109999999999993e-06 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697380+0000", 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6162999999999997e-05 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697512+0000", 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013168899999999999 2024-09-17T16:16:48.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697679+0000", 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016729599999999999 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.698584+0000", 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090433600000000005 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.762716+0000", 2024-09-17T16:16:48.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.064132583000000007 2024-09-17T16:16:48.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087510+0000", 2024-09-17T16:16:48.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32479419700000001 2024-09-17T16:16:48.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087549+0000", 2024-09-17T16:16:48.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8278999999999997e-05 2024-09-17T16:16:48.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087577+0000", 2024-09-17T16:16:48.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7957999999999999e-05 2024-09-17T16:16:48.237 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.238 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.238 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.238 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24385 2.b 2:d672fe3e:::benchmark_data_smithi173_55121_object24384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:16:48.238 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.704153+0000", 2024-09-17T16:16:48.238 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.507420309, 2024-09-17T16:16:48.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53404779700000005, 2024-09-17T16:16:48.238 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24385 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704153+0000", 2024-09-17T16:16:48.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704153+0000", 2024-09-17T16:16:48.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704154+0000", 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704177+0000", 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2127e-05 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704189+0000", 2024-09-17T16:16:48.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2241e-05 2024-09-17T16:16:48.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704195+0000", 2024-09-17T16:16:48.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4280000000000001e-06 2024-09-17T16:16:48.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704239+0000", 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3775e-05 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704326+0000", 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7198000000000001e-05 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704414+0000", 2024-09-17T16:16:48.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7652999999999996e-05 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705316+0000", 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090249000000000004 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705635+0000", 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031902200000000002 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.238124+0000", 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53248838600000004 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.238158+0000", 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4789000000000002e-05 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.238201+0000", 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2468000000000001e-05 2024-09-17T16:16:48.245 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24386 2.10 2:0afcc277:::benchmark_data_smithi173_55121_object24385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.705747+0000", 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.50582626099998, 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43386687200000001, 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.246 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24386 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705747+0000", 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705747+0000", 2024-09-17T16:16:48.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705748+0000", 2024-09-17T16:16:48.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5099999999999999e-07 2024-09-17T16:16:48.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705770+0000", 2024-09-17T16:16:48.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1563999999999999e-05 2024-09-17T16:16:48.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705779+0000", 2024-09-17T16:16:48.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3249999999999997e-06 2024-09-17T16:16:48.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705784+0000", 2024-09-17T16:16:48.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.203e-06 2024-09-17T16:16:48.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705832+0000", 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8180000000000003e-05 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705930+0000", 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7617999999999999e-05 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:16:48.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706020+0000", 2024-09-17T16:16:48.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9890999999999997e-05 2024-09-17T16:16:48.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706544+0000", 2024-09-17T16:16:48.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00052466899999999996 2024-09-17T16:16:48.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706882+0000", 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000337833 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139550+0000", 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43266755299999998 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139585+0000", 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5136999999999999e-05 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139614+0000", 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9147999999999999e-05 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24388 2.f 2:f7f1158d:::benchmark_data_smithi173_55121_object24387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.761151+0000", 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.45042247600003, 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37970383499999999, 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24388 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761151+0000", 2024-09-17T16:16:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761151+0000", 2024-09-17T16:16:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761152+0000", 2024-09-17T16:16:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4619999999999999e-06 2024-09-17T16:16:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761176+0000", 2024-09-17T16:16:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3319000000000002e-05 2024-09-17T16:16:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761190+0000", 2024-09-17T16:16:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3916e-05 2024-09-17T16:16:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761198+0000", 2024-09-17T16:16:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1280000000000008e-06 2024-09-17T16:16:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761269+0000", 2024-09-17T16:16:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0862e-05 2024-09-17T16:16:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761423+0000", 2024-09-17T16:16:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015397499999999999 2024-09-17T16:16:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:16:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761596+0000", 2024-09-17T16:16:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017338799999999999 2024-09-17T16:16:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.762762+0000", 2024-09-17T16:16:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001166325 2024-09-17T16:16:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767453+0000", 2024-09-17T16:16:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046909999999999999 2024-09-17T16:16:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.140795+0000", 2024-09-17T16:16:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37334170999999999 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.140830+0000", 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4625999999999997e-05 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.140855+0000", 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5123999999999999e-05 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24391 2.5 2:a6a580fc:::benchmark_data_smithi173_55121_object24390:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.766124+0000", 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.44544968500003, 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36991581699999998, 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24391 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766124+0000", 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766124+0000", 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766125+0000", 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6300000000000004e-07 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766146+0000", 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1365e-05 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766157+0000", 2024-09-17T16:16:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1294000000000001e-05 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766163+0000", 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3170000000000001e-06 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766215+0000", 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2367999999999997e-05 2024-09-17T16:16:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766306+0000", 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0994000000000003e-05 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766395+0000", 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9352999999999996e-05 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766965+0000", 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056975199999999998 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767311+0000", 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000345621 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135856+0000", 2024-09-17T16:16:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36854523 2024-09-17T16:16:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135970+0000", 2024-09-17T16:16:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114548 2024-09-17T16:16:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.136040+0000", 2024-09-17T16:16:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9111999999999998e-05 2024-09-17T16:16:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:24391 2.5 e120/104 ondisk, result = 0)", 2024-09-17T16:16:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:16:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.44418082300001, 2024-09-17T16:16:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36866083100000002, 2024-09-17T16:16:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:16:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:16:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:16:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:16:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5199999999999996e-07 2024-09-17T16:16:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767397+0000", 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.27e-06 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135721+0000", 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36832436600000001 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135734+0000", 2024-09-17T16:16:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.323e-05 2024-09-17T16:16:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135830+0000", 2024-09-17T16:16:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5587999999999994e-05 2024-09-17T16:16:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135843+0000", 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3699999999999999e-05 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.136053+0000", 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000210025 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25180 2.10 2:092fb28c:::benchmark_data_smithi173_55121_object25179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.740770+0000", 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.47080344099999, 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48461674700000001, 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25180 2024-09-17T16:16:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740770+0000", 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740770+0000", 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740772+0000", 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5969999999999999e-06 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740793+0000", 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1379999999999999e-05 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740807+0000", 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4222e-05 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740814+0000", 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7830000000000001e-06 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740877+0000", 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2996999999999993e-05 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740984+0000", 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106656 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.741069+0000", 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5752000000000002e-05 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.741666+0000", 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059629500000000005 2024-09-17T16:16:48.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.742062+0000", 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000396301 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225328+0000", 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48326640700000001 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225362+0000", 2024-09-17T16:16:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3317e-05 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225387+0000", 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5040000000000001e-05 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25215 2.10 2:0e4d5290:::benchmark_data_smithi173_55121_object25214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.775331+0000", 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.43624272699998, 2024-09-17T16:16:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47406391599999997, 2024-09-17T16:16:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25215 2024-09-17T16:16:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775331+0000", 2024-09-17T16:16:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775331+0000", 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775332+0000", 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9300000000000006e-07 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775355+0000", 2024-09-17T16:16:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3303e-05 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775369+0000", 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3896999999999999e-05 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775377+0000", 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1699999999999997e-06 2024-09-17T16:16:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775446+0000", 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8745000000000007e-05 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775561+0000", 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115616 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775705+0000", 2024-09-17T16:16:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000143151 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776312+0000", 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00060773299999999995 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776660+0000", 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034724700000000002 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.249329+0000", 2024-09-17T16:16:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47266967799999998 2024-09-17T16:16:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.249367+0000", 2024-09-17T16:16:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7233000000000001e-05 2024-09-17T16:16:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.249395+0000", 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.815e-05 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:25180 2.10 e121/111 ondisk, result = 0)", 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:16:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.43510095699997, 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44891908400000002, 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3200000000000004e-07 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776477+0000", 2024-09-17T16:16:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6660000000000001e-06 2024-09-17T16:16:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225282+0000", 2024-09-17T16:16:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44880505700000001 2024-09-17T16:16:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225287+0000", 2024-09-17T16:16:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8659999999999997e-06 2024-09-17T16:16:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225320+0000", 2024-09-17T16:16:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3055000000000001e-05 2024-09-17T16:16:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225324+0000", 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3599999999999998e-06 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225392+0000", 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7348000000000006e-05 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25237 2.5 2:a0e164cf:::benchmark_data_smithi173_55121_object25236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.800457+0000", 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.41111673900002, 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33304292899999999, 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25237 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800457+0000", 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800457+0000", 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800458+0000", 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800475+0000", 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7628000000000001e-05 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800486+0000", 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0669999999999999e-05 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800491+0000", 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.22e-06 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800536+0000", 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4712000000000002e-05 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800616+0000", 2024-09-17T16:16:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0365999999999996e-05 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.800697+0000", 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0935999999999999e-05 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.801225+0000", 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000528128 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.801667+0000", 2024-09-17T16:16:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044215299999999998 2024-09-17T16:16:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133437+0000", 2024-09-17T16:16:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33176945000000002 2024-09-17T16:16:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133473+0000", 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6189999999999997e-05 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.133500+0000", 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6701999999999999e-05 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:26993 2.10 2:0b00f2f4:::benchmark_data_smithi173_55121_object26992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.983151+0000", 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 315.22842204300002, 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51500823299999998, 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 26993 2024-09-17T16:16:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983151+0000", 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983151+0000", 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983153+0000", 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.111e-06 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983170+0000", 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7805e-05 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983182+0000", 2024-09-17T16:16:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1391e-05 2024-09-17T16:16:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983187+0000", 2024-09-17T16:16:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4809999999999997e-06 2024-09-17T16:16:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983229+0000", 2024-09-17T16:16:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1817999999999998e-05 2024-09-17T16:16:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983316+0000", 2024-09-17T16:16:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6978999999999995e-05 2024-09-17T16:16:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:16:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983399+0000", 2024-09-17T16:16:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2720999999999996e-05 2024-09-17T16:16:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983972+0000", 2024-09-17T16:16:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057330599999999999 2024-09-17T16:16:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.984444+0000", 2024-09-17T16:16:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047245000000000002 2024-09-17T16:16:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.498086+0000", 2024-09-17T16:16:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51364169000000004 2024-09-17T16:16:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.498123+0000", 2024-09-17T16:16:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6358e-05 2024-09-17T16:16:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.498160+0000", 2024-09-17T16:16:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7122999999999997e-05 2024-09-17T16:16:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:92960 2.b 2:d44e9bf2:::benchmark_data_smithi173_55121_object16471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:16:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.636820+0000", 2024-09-17T16:16:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 239.57475339800001, 2024-09-17T16:16:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63467018600000003, 2024-09-17T16:16:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 92960 2024-09-17T16:16:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636820+0000", 2024-09-17T16:16:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636820+0000", 2024-09-17T16:16:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636822+0000", 2024-09-17T16:16:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1969999999999999e-06 2024-09-17T16:16:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636830+0000", 2024-09-17T16:16:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4490000000000002e-06 2024-09-17T16:16:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636849+0000", 2024-09-17T16:16:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9029999999999999e-05 2024-09-17T16:16:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636859+0000", 2024-09-17T16:16:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0183999999999999e-05 2024-09-17T16:16:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636929+0000", 2024-09-17T16:16:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0017000000000004e-05 2024-09-17T16:16:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.637101+0000", 2024-09-17T16:16:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017157500000000001 2024-09-17T16:16:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:16:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.637326+0000", 2024-09-17T16:16:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022522300000000001 2024-09-17T16:16:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640616+0000", 2024-09-17T16:16:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032902130000000002 2024-09-17T16:16:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004722+0000", 2024-09-17T16:16:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36410627899999998 2024-09-17T16:16:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269780+0000", 2024-09-17T16:16:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.265057773 2024-09-17T16:16:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269830+0000", 2024-09-17T16:16:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9772999999999999e-05 2024-09-17T16:16:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.271490+0000", 2024-09-17T16:16:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001660473 2024-09-17T16:16:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:92960 2.b e158/155 ondisk, result = 0)", 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.640613+0000", 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 239.570960877, 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36412291800000002, 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640613+0000", 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640613+0000", 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640614+0000", 2024-09-17T16:16:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2500000000000004e-07 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640619+0000", 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5350000000000004e-06 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004492+0000", 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36387301399999999 2024-09-17T16:16:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004506+0000", 2024-09-17T16:16:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4403000000000001e-05 2024-09-17T16:16:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004662+0000", 2024-09-17T16:16:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015578400000000001 2024-09-17T16:16:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004694+0000", 2024-09-17T16:16:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1482000000000001e-05 2024-09-17T16:16:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004736+0000", 2024-09-17T16:16:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1774999999999999e-05 2024-09-17T16:16:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:92960 2.b e158/155 ondisk, result = 0)", 2024-09-17T16:16:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.640951+0000", 2024-09-17T16:16:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 239.57062200799999, 2024-09-17T16:16:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.630555843, 2024-09-17T16:16:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:16:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640951+0000", 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640951+0000", 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640952+0000", 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.54e-07 2024-09-17T16:16:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640960+0000", 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9570000000000002e-06 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.266220+0000", 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62525987900000002 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.266226+0000", 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9800000000000003e-06 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269764+0000", 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035381520000000001 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269773+0000", 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5450000000000003e-06 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.271507+0000", 2024-09-17T16:16:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001734576 2024-09-17T16:16:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:135561 2.6 e191/104 2:612899dc:::benchmark_data_smithi173_55121_object59072:head v 191'4677, mlcod=191'4675)", 2024-09-17T16:16:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:35.921733+0000", 2024-09-17T16:16:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 192.289840489, 2024-09-17T16:16:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36610857200000002, 2024-09-17T16:16:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921733+0000", 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921733+0000", 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921734+0000", 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921739+0000", 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2630000000000003e-06 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.285858+0000", 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.364118357 2024-09-17T16:16:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.285878+0000", 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0152999999999998e-05 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.286066+0000", 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000188016 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.286106+0000", 2024-09-17T16:16:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0309999999999999e-05 2024-09-17T16:16:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.287803+0000", 2024-09-17T16:16:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016966290000000001 2024-09-17T16:16:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.287842+0000", 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8945000000000001e-05 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:150435 2.5 2:a4c0aa4f:::benchmark_data_smithi173_55121_object73946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:51.117314+0000", 2024-09-17T16:16:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 177.09425972, 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.349455405, 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 150435 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.117314+0000", 2024-09-17T16:16:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.117314+0000", 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.117315+0000", 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.066e-06 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.117320+0000", 2024-09-17T16:16:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8080000000000004e-06 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.117330+0000", 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0346999999999999e-05 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.117335+0000", 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5110000000000003e-06 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.117390+0000", 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4898000000000002e-05 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.117528+0000", 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000137835 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.117654+0000", 2024-09-17T16:16:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012562200000000001 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.119464+0000", 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018101829999999999 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.119962+0000", 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049821499999999999 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.466666+0000", 2024-09-17T16:16:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.346703335 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.466722+0000", 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6332000000000002e-05 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.466769+0000", 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7253000000000003e-05 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:150435 2.5 e197/104 ondisk, result = 0)", 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:51.119793+0000", 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 177.09178091000001, 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34698340700000002, 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.119793+0000", 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.119793+0000", 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.119793+0000", 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1800000000000005e-07 2024-09-17T16:16:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.119797+0000", 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4340000000000001e-06 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.466590+0000", 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34679301400000001 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.466596+0000", 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5610000000000004e-06 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.466652+0000", 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5390000000000003e-05 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.466660+0000", 2024-09-17T16:16:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1000000000000004e-06 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:51.466776+0000", 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011619 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(3.9 240/239 [PushReplyOp(3:9e32e3c9:::benchmark_data_smithi173_66272_object1144:head)])", 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 93.132035928999997, 2024-09-17T16:16:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90253544399999996, 2024-09-17T16:16:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:16:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:16:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:16:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:16:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:16:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:16:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6799999999999999e-07 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079543+0000", 2024-09-17T16:16:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6809999999999997e-06 2024-09-17T16:16:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:16:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979238+0000", 2024-09-17T16:16:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89969455399999998 2024-09-17T16:16:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:16:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979247+0000", 2024-09-17T16:16:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5659999999999996e-06 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979307+0000", 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9338e-05 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.982073+0000", 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027664370000000001 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:16:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:16:48.360 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:16:48.361 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:47 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:16:48.361 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:47 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:16:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:48.362 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:48.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:48.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:48.490 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:48.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:48.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:48.628 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:48.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:48.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:48.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:48 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:16:48.642+0000 7f086a07d640 -1 mon.a@0(leader).osd e271 definitely_dead 0 2024-09-17T16:16:48.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:48.754 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:48.755 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:48.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:48.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:48.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:48.914 INFO:teuthology.orchestra.run.smithi017.stderr:marked in osd.11. 2024-09-17T16:16:48.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:48 smithi017 ceph-mon[27571]: pgmap v649: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.4 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 805 op/s; 256/66906 objects degraded (0.383%); 4757/66906 objects misplaced (7.110%); 12 MiB/s, 186 objects/s recovering 2024-09-17T16:16:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:48 smithi017 ceph-mon[27571]: osdmap e271: 12 total, 12 up, 11 in 2024-09-17T16:16:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2730244445' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-09-17T16:16:49.010 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:49.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:49.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:49.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:48 smithi173 ceph-mon[30530]: pgmap v649: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.4 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 805 op/s; 256/66906 objects degraded (0.383%); 4757/66906 objects misplaced (7.110%); 12 MiB/s, 186 objects/s recovering 2024-09-17T16:16:49.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:48 smithi173 ceph-mon[30530]: osdmap e271: 12 total, 12 up, 11 in 2024-09-17T16:16:49.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2730244445' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-09-17T16:16:49.112 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:49.134 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:49.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:49.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:49.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:16:49.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:49.258 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:49.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:49.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:49.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:48 smithi071 ceph-mon[30002]: pgmap v649: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.4 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 805 op/s; 256/66906 objects degraded (0.383%); 4757/66906 objects misplaced (7.110%); 12 MiB/s, 186 objects/s recovering 2024-09-17T16:16:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:48 smithi071 ceph-mon[30002]: osdmap e271: 12 total, 12 up, 11 in 2024-09-17T16:16:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2730244445' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-09-17T16:16:49.352 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:16:49.360 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:49.389 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:49.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:49.393 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:16:49.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:49.514 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:49.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:49.572 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:16:49.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:49.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:49.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:49.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:49.741 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:49.767 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:49.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:49.868 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:49.891 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:49.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:49.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:50.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:50.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:50.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:50.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2730244445' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-09-17T16:16:50.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:49 smithi173 ceph-mon[30530]: osdmap e272: 12 total, 12 up, 12 in 2024-09-17T16:16:50.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3080118844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:50.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:49 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 256/66906 objects degraded (0.383%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:50.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:49 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:50.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3080118844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:50.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:49 smithi173 ceph-mon[30530]: osdmap e273: 12 total, 12 up, 12 in 2024-09-17T16:16:50.071 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:16:50.071 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:16:50.071 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:16:50.071 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:16:50.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:50.144 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:50.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:50.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2730244445' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-09-17T16:16:50.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:49 smithi017 ceph-mon[27571]: osdmap e272: 12 total, 12 up, 12 in 2024-09-17T16:16:50.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3080118844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:50.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:49 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 256/66906 objects degraded (0.383%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:50.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:49 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:50.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3080118844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:50.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:49 smithi017 ceph-mon[27571]: osdmap e273: 12 total, 12 up, 12 in 2024-09-17T16:16:50.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:50.273 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:50.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:50.289 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-09-17T16:16:50.309 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:50.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2730244445' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-09-17T16:16:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:49 smithi071 ceph-mon[30002]: osdmap e272: 12 total, 12 up, 12 in 2024-09-17T16:16:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3080118844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:16:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:49 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 256/66906 objects degraded (0.383%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:16:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:49 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3080118844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:16:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:49 smithi071 ceph-mon[30002]: osdmap e273: 12 total, 12 up, 12 in 2024-09-17T16:16:50.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:50.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:50.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:50.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:50.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:50.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:50.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:50.647 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:50.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:50.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:50.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:50.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:50.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:16:50.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:50.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:50.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:51.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:51.061 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:51.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:51.163 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:51.195 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:51.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:51.221 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:16:51.296 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:51.339 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:51.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:51.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:16:51.383 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:16:51.422 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:16:51.422 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:16:51.441 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:51.456 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:16:51.456 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:16:51.466 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:51.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:51.491 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:16:51.491 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:16:51.525 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:16:51.525 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:16:51.559 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:16:51.559 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:16:51.559 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:51 smithi017 ceph-mon[27571]: pgmap v653: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 150/64314 objects degraded (0.233%); 3784/64314 objects misplaced (5.884%); 15 MiB/s, 236 objects/s recovering 2024-09-17T16:16:51.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:51 smithi017 ceph-mon[27571]: 3.9 starting backfill to osd.11 from (0'0,237'2886] MIN to 271'7639 2024-09-17T16:16:51.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:51 smithi017 ceph-mon[27571]: 3.e starting backfill to osd.11 from (0'0,237'2937] MIN to 271'7875 2024-09-17T16:16:51.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:51 smithi017 ceph-mon[27571]: 3.4 starting backfill to osd.11 from (0'0,237'2935] MIN to 271'7791 2024-09-17T16:16:51.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:51 smithi017 ceph-mon[27571]: osdmap e274: 12 total, 12 up, 12 in 2024-09-17T16:16:51.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:51 smithi017 ceph-mon[27571]: 3.0 starting backfill to osd.11 from (0'0,237'2935] MIN to 271'7787 2024-09-17T16:16:51.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:51 smithi017 ceph-mon[27571]: 3.7 starting backfill to osd.11 from (0'0,237'2970] MIN to 271'7860 2024-09-17T16:16:51.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:51 smithi017 ceph-mon[27571]: 3.f starting backfill to osd.11 from (0'0,237'2876] MIN to 271'7749 2024-09-17T16:16:51.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:51 smithi017 ceph-mon[27571]: 3.8 starting backfill to osd.11 from (0'0,237'2854] MIN to 271'7587 2024-09-17T16:16:51.560 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2895000059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:51.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:51.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:16:51.591 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:16:51.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:51.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:51.624 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:16:51.625 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:16:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:51 smithi173 ceph-mon[30530]: pgmap v653: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 150/64314 objects degraded (0.233%); 3784/64314 objects misplaced (5.884%); 15 MiB/s, 236 objects/s recovering 2024-09-17T16:16:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:51 smithi173 ceph-mon[30530]: 3.9 starting backfill to osd.11 from (0'0,237'2886] MIN to 271'7639 2024-09-17T16:16:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:51 smithi173 ceph-mon[30530]: 3.e starting backfill to osd.11 from (0'0,237'2937] MIN to 271'7875 2024-09-17T16:16:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:51 smithi173 ceph-mon[30530]: 3.4 starting backfill to osd.11 from (0'0,237'2935] MIN to 271'7791 2024-09-17T16:16:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:51 smithi173 ceph-mon[30530]: osdmap e274: 12 total, 12 up, 12 in 2024-09-17T16:16:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:51 smithi173 ceph-mon[30530]: 3.0 starting backfill to osd.11 from (0'0,237'2935] MIN to 271'7787 2024-09-17T16:16:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:51 smithi173 ceph-mon[30530]: 3.7 starting backfill to osd.11 from (0'0,237'2970] MIN to 271'7860 2024-09-17T16:16:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:51 smithi173 ceph-mon[30530]: 3.f starting backfill to osd.11 from (0'0,237'2876] MIN to 271'7749 2024-09-17T16:16:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:51 smithi173 ceph-mon[30530]: 3.8 starting backfill to osd.11 from (0'0,237'2854] MIN to 271'7587 2024-09-17T16:16:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2895000059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:51.668 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:16:51.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:51.703 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:16:51.703 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:16:51.737 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:16:51.738 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:16:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:51.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:51.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:51.771 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:16:51.772 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:16:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:51 smithi071 ceph-mon[30002]: pgmap v653: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.3 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 150/64314 objects degraded (0.233%); 3784/64314 objects misplaced (5.884%); 15 MiB/s, 236 objects/s recovering 2024-09-17T16:16:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:51 smithi071 ceph-mon[30002]: 3.9 starting backfill to osd.11 from (0'0,237'2886] MIN to 271'7639 2024-09-17T16:16:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:51 smithi071 ceph-mon[30002]: 3.e starting backfill to osd.11 from (0'0,237'2937] MIN to 271'7875 2024-09-17T16:16:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:51 smithi071 ceph-mon[30002]: 3.4 starting backfill to osd.11 from (0'0,237'2935] MIN to 271'7791 2024-09-17T16:16:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:51 smithi071 ceph-mon[30002]: osdmap e274: 12 total, 12 up, 12 in 2024-09-17T16:16:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:51 smithi071 ceph-mon[30002]: 3.0 starting backfill to osd.11 from (0'0,237'2935] MIN to 271'7787 2024-09-17T16:16:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:51 smithi071 ceph-mon[30002]: 3.7 starting backfill to osd.11 from (0'0,237'2970] MIN to 271'7860 2024-09-17T16:16:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:51 smithi071 ceph-mon[30002]: 3.f starting backfill to osd.11 from (0'0,237'2876] MIN to 271'7749 2024-09-17T16:16:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:51 smithi071 ceph-mon[30002]: 3.8 starting backfill to osd.11 from (0'0,237'2854] MIN to 271'7587 2024-09-17T16:16:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2895000059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:51.822 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:51.822 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:51.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:51.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:51.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:51.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:52.002 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:52.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:52.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:52.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:52.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:52.133 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:52.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:52.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:52.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:52.261 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:52.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:52.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:52.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:52.394 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:52.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:52.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:52.517 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:52.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:52.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:52.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:52.643 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:52.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:52 smithi017 ceph-mon[27571]: pgmap v655: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 9 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 1/58368 objects degraded (0.002%); 9145/58368 objects misplaced (15.668%); 17 MiB/s, 269 objects/s recovering 2024-09-17T16:16:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:52 smithi017 ceph-mon[27571]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:16:52.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:52.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:16:52.772 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:52 smithi173 ceph-mon[30530]: pgmap v655: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 9 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 1/58368 objects degraded (0.002%); 9145/58368 objects misplaced (15.668%); 17 MiB/s, 269 objects/s recovering 2024-09-17T16:16:52.772 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:52 smithi173 ceph-mon[30530]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:16:52.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:52.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:52 smithi071 ceph-mon[30002]: pgmap v655: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 9 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 1/58368 objects degraded (0.002%); 9145/58368 objects misplaced (15.668%); 17 MiB/s, 269 objects/s recovering 2024-09-17T16:16:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:52 smithi071 ceph-mon[30002]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:16:52.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:52.903 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:52.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:53.006 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:53.031 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:53.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:53.130 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:53.133 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:53.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:16:53.160 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:53.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:53.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:53.288 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:53.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:53.390 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:53.417 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:53.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:53.498 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:16:53.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:53.541 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:53.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:53.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:53.665 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:53.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:53.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:53.792 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:53.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:53.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:53.926 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:53.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:54.029 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:54.058 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:54.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:54.091 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:16:54.091 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:16:54.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:54.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:54.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:54.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:54.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:54.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:54.421 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:16:54.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:54.469 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:54.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:54.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:54.593 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:54.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:54.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:54.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:16:54.681 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:54 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:16:54.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:54 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:16:54.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:54 smithi017 ceph-mon[27571]: osdmap e275: 12 total, 12 up, 12 in 2024-09-17T16:16:54.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:54 smithi017 ceph-mon[27571]: pgmap v657: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1/57381 objects degraded (0.002%); 10277/57381 objects misplaced (17.910%); 14 MiB/s, 220 objects/s recovering 2024-09-17T16:16:54.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2503204623' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:16:54.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:54 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/58368 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:16:54.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:54.718 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:54.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:54.759 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:54 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:16:54.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:54 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:16:54.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:54 smithi173 ceph-mon[30530]: osdmap e275: 12 total, 12 up, 12 in 2024-09-17T16:16:54.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:54 smithi173 ceph-mon[30530]: pgmap v657: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1/57381 objects degraded (0.002%); 10277/57381 objects misplaced (17.910%); 14 MiB/s, 220 objects/s recovering 2024-09-17T16:16:54.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2503204623' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:16:54.760 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:54 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/58368 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:16:54.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:16:54.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:54 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:16:54.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:54 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:16:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:54 smithi071 ceph-mon[30002]: osdmap e275: 12 total, 12 up, 12 in 2024-09-17T16:16:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:54 smithi071 ceph-mon[30002]: pgmap v657: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 1.2 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 1/57381 objects degraded (0.002%); 10277/57381 objects misplaced (17.910%); 14 MiB/s, 220 objects/s recovering 2024-09-17T16:16:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2503204623' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:16:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:54 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/58368 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:16:54.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:54.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:54.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:54.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:55.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:55.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:55.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:55.151 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:55.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:55.172 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:16:55.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:55.278 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:55.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:55.290 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:16:55.291 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:16:55.291 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-17T16:16:55.291 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T16:16:55.291 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T16:16:55.291 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-17T16:16:55.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:55.408 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:55.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:55.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:55.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:55.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:55.630 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:16:55.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:55 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2503204623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:16:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:55 smithi173 ceph-mon[30530]: osdmap e276: 12 total, 12 up, 12 in 2024-09-17T16:16:55.654 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:16:55.689 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:55.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:55 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2503204623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:16:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:55 smithi017 ceph-mon[27571]: osdmap e276: 12 total, 12 up, 12 in 2024-09-17T16:16:55.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:55 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:16:55.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2503204623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:16:55.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:55 smithi071 ceph-mon[30002]: osdmap e276: 12 total, 12 up, 12 in 2024-09-17T16:16:55.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:55.827 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:55.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:55.928 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:55.959 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:55.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:56.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:56.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:56.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:56.186 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:16:56.186 INFO:teuthology.orchestra.run.smithi071.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T16:16:56.186 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:16:56.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:56.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:56.219 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:56.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:56.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:56.355 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:56.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:56.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:56.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:56.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:56.585 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:56.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:56 smithi017 ceph-mon[27571]: 3.3 scrub starts 2024-09-17T16:16:56.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:56 smithi017 ceph-mon[27571]: pgmap v659: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 1/54996 objects degraded (0.002%); 10343/54996 objects misplaced (18.807%); 14 MiB/s, 216 objects/s recovering 2024-09-17T16:16:56.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:56 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:16:56.611 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:56 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:16:56.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:56 smithi017 ceph-mon[27571]: 3.3 scrub ok 2024-09-17T16:16:56.621 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:56.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:56.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:16:56.744 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:56.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:16:56.804 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:56 smithi071 ceph-mon[30002]: 3.3 scrub starts 2024-09-17T16:16:56.804 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:56 smithi071 ceph-mon[30002]: pgmap v659: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 1/54996 objects degraded (0.002%); 10343/54996 objects misplaced (18.807%); 14 MiB/s, 216 objects/s recovering 2024-09-17T16:16:56.804 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:56 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:16:56.804 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:56 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:16:56.804 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:56 smithi071 ceph-mon[30002]: 3.3 scrub ok 2024-09-17T16:16:56.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:56.868 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:56.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:56 smithi173 ceph-mon[30530]: 3.3 scrub starts 2024-09-17T16:16:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:56 smithi173 ceph-mon[30530]: pgmap v659: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 1/54996 objects degraded (0.002%); 10343/54996 objects misplaced (18.807%); 14 MiB/s, 216 objects/s recovering 2024-09-17T16:16:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:56 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:16:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:56 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:16:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:56 smithi173 ceph-mon[30530]: 3.3 scrub ok 2024-09-17T16:16:56.969 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:16:57.018 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:57.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:16:57.119 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:16:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:57.171 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:57.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:16:57.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:57.300 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:57.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:57.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:16:57.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:57.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:16:57.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:57.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:16:57.553 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:57.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:57.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:57.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:57.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:57 smithi017 ceph-mon[27571]: osdmap e277: 12 total, 12 up, 12 in 2024-09-17T16:16:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:57 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:16:57.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:57 smithi173 ceph-mon[30530]: osdmap e277: 12 total, 12 up, 12 in 2024-09-17T16:16:57.767 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:57 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:16:57.794 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:57.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:57 smithi071 ceph-mon[30002]: osdmap e277: 12 total, 12 up, 12 in 2024-09-17T16:16:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:57 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:16:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:57.830 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:57.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:57.930 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:57.965 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:57.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:58.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:58.091 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:58.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:58.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:58.217 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:58.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:58.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:16:58.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:58.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:16:58.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:58.474 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:58.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:58.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:58.603 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:58.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:58.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:16:58.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:58.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:16:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:58 smithi017 ceph-mon[27571]: pgmap v661: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 1003 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 1/48120 objects degraded (0.002%); 8509/48120 objects misplaced (17.683%); 13 MiB/s, 200 objects/s recovering 2024-09-17T16:16:58.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:58 smithi071 ceph-mon[30002]: pgmap v661: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 1003 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 1/48120 objects degraded (0.002%); 8509/48120 objects misplaced (17.683%); 13 MiB/s, 200 objects/s recovering 2024-09-17T16:16:58.829 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:16:58.854 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:58.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:58.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:16:58.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:58 smithi173 ceph-mon[30530]: pgmap v661: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 1003 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 1/48120 objects degraded (0.002%); 8509/48120 objects misplaced (17.683%); 13 MiB/s, 200 objects/s recovering 2024-09-17T16:16:58.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:16:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:58.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:58.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:16:59.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:16:59.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:59.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:16:59.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:59.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:59.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:16:59.354 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:59.387 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:59.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:59.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:16:59.510 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:16:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:59.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:16:59.612 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:59.643 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:59.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:59.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1095694329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:16:59 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/48120 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:16:59.745 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:16:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:16:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:59.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:16:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1095694329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:16:59 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/48120 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:16:59.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:16:59.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1095694329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:16:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:16:59 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/48120 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:16:59.904 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:16:59.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:16:59.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:00.006 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:00.033 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:00.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:00.135 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:00.163 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:17:00.163 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:17:00.163 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:17:00.163 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:17:00.163 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:17:00.163 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:17:00.163 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:17:00.163 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:17:00.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:17:00.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:17:00.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:17:00.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:17:00.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:17:00.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:17:00.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:17:00.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:17:00.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:17:00.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:17:00.165 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:17:00.165 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:17:00.165 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:17:00.165 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:17:00.165 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:17:00.165 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:17:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:00.166 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:00.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:00.216 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:17:00.216 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:17:00.266 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:00.290 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:00.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:00.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:00.415 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:00.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:00.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:00.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:00.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:00.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:00 smithi173 ceph-mon[30530]: pgmap v662: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 950 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 1/45570 objects degraded (0.002%); 7933/45570 objects misplaced (17.408%); 15 MiB/s, 273 objects/s recovering 2024-09-17T16:17:00.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:00.674 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:00.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:00.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:00.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:00 smithi017 ceph-mon[27571]: pgmap v662: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 950 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 1/45570 objects degraded (0.002%); 7933/45570 objects misplaced (17.408%); 15 MiB/s, 273 objects/s recovering 2024-09-17T16:17:00.776 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:00.803 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:00.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:00.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:00 smithi071 ceph-mon[30002]: pgmap v662: 17 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 950 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 1/45570 objects degraded (0.002%); 7933/45570 objects misplaced (17.408%); 15 MiB/s, 273 objects/s recovering 2024-09-17T16:17:00.905 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:00.931 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:00.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:01.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:01.062 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:01.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:01.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:01.190 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:01.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:01.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:01.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:01.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:01.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:01.423 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:01.445 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:01.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:01.547 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:01.571 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:01.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:01.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:01.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:01.705 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:01.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:01.807 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:01.837 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:01.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:01.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:01.965 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:01.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:02.021 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:17:02.021 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:17:02.022 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-09-17T16:17:02.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:17:02.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:02.094 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:02.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:02.195 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:02.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:02.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:02.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:02.371 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:02.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:02.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:02.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:02.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:02.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:02.633 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:02.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:02.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:02.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:02.760 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:02.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:02.864 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:02.899 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:02.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:03.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:03.026 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:03.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:03.128 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:03.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:02 smithi173 ceph-mon[30530]: pgmap v663: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 852 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 1/40884 objects degraded (0.002%); 6783/40884 objects misplaced (16.591%); 16 MiB/s, 284 objects/s recovering 2024-09-17T16:17:03.150 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:03.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:02 smithi017 ceph-mon[27571]: pgmap v663: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 852 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 1/40884 objects degraded (0.002%); 6783/40884 objects misplaced (16.591%); 16 MiB/s, 284 objects/s recovering 2024-09-17T16:17:03.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:03.274 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:03.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:03.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:03.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:02 smithi071 ceph-mon[30002]: pgmap v663: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 852 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 1/40884 objects degraded (0.002%); 6783/40884 objects misplaced (16.591%); 16 MiB/s, 284 objects/s recovering 2024-09-17T16:17:03.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:03.399 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:03.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:03.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:03.526 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:03.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:03.628 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:03.651 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:03.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:03.753 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:03.775 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:03.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:03.877 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:03.900 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:03.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:04.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:04.025 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:04.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:04.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:03 smithi173 ceph-mon[30530]: osdmap e278: 12 total, 12 up, 12 in 2024-09-17T16:17:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:04.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:04.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:04.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:03 smithi017 ceph-mon[27571]: osdmap e278: 12 total, 12 up, 12 in 2024-09-17T16:17:04.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:04.293 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:04.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:04.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:03 smithi071 ceph-mon[30002]: osdmap e278: 12 total, 12 up, 12 in 2024-09-17T16:17:04.396 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:04.428 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:04.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:04.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:04.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:04.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:04.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:04.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:04.687 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:04.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:04.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:04.821 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:04.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:04.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:04.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:04.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:05.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:04 smithi071 ceph-mon[30002]: pgmap v665: 17 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 778 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 735 op/s; 1/37347 objects degraded (0.003%); 5674/37347 objects misplaced (15.193%); 16 MiB/s, 274 objects/s recovering 2024-09-17T16:17:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:04 smithi071 ceph-mon[30002]: osdmap e279: 12 total, 12 up, 12 in 2024-09-17T16:17:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:04 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/37347 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:05.081 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:05.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:05.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:04 smithi173 ceph-mon[30530]: pgmap v665: 17 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 778 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 735 op/s; 1/37347 objects degraded (0.003%); 5674/37347 objects misplaced (15.193%); 16 MiB/s, 274 objects/s recovering 2024-09-17T16:17:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:04 smithi173 ceph-mon[30530]: osdmap e279: 12 total, 12 up, 12 in 2024-09-17T16:17:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:04 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/37347 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:05.182 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:05.204 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:05.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:04 smithi017 ceph-mon[27571]: pgmap v665: 17 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 778 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 735 op/s; 1/37347 objects degraded (0.003%); 5674/37347 objects misplaced (15.193%); 16 MiB/s, 274 objects/s recovering 2024-09-17T16:17:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:04 smithi017 ceph-mon[27571]: osdmap e279: 12 total, 12 up, 12 in 2024-09-17T16:17:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:04 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/37347 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:05.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:05.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:05.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:05.438 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:05.469 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:05.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:05.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:05.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:05.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:05.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:05.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:05.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:05.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:05.866 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:05.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:05.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:05.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:05.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:06.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:06.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:06.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:06.219 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:06.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:06.242 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:06.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:06.343 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:06.373 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:06.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:06.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:06.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:06.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:06.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:06.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:06.630 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:06.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:06.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:06.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:06.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:06.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:06.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:06.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:06.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:06.947 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:06.963 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:06.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:07.019 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:07.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:07.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:07.086 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:06 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:17:07.086 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:06 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:17:07.086 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:06 smithi017 ceph-mon[27571]: pgmap v667: 17 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 722 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 1/34656 objects degraded (0.003%); 5956/34656 objects misplaced (17.186%); 13 MiB/s, 302 objects/s recovering 2024-09-17T16:17:07.087 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:06 smithi017 ceph-mon[27571]: osdmap e280: 12 total, 12 up, 12 in 2024-09-17T16:17:07.120 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:07.142 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:07.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:07.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:06 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:17:07.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:06 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:17:07.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:06 smithi173 ceph-mon[30530]: pgmap v667: 17 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 722 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 1/34656 objects degraded (0.003%); 5956/34656 objects misplaced (17.186%); 13 MiB/s, 302 objects/s recovering 2024-09-17T16:17:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:06 smithi173 ceph-mon[30530]: osdmap e280: 12 total, 12 up, 12 in 2024-09-17T16:17:07.244 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:07.277 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:07.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:07.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:06 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:17:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:06 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:17:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:06 smithi071 ceph-mon[30002]: pgmap v667: 17 pgs: 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 8 active+clean; 722 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 1/34656 objects degraded (0.003%); 5956/34656 objects misplaced (17.186%); 13 MiB/s, 302 objects/s recovering 2024-09-17T16:17:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:06 smithi071 ceph-mon[30002]: osdmap e280: 12 total, 12 up, 12 in 2024-09-17T16:17:07.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:07.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:07.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:07.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:07.527 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:07.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:07.629 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:07.659 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:07.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:07.761 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:07.782 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:07.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:07.884 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:07.903 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:07 smithi017 ceph-mon[27571]: osdmap e281: 12 total, 12 up, 12 in 2024-09-17T16:17:07.903 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:07 smithi017 ceph-mon[27571]: 3.5 scrub starts 2024-09-17T16:17:07.916 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:07.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:07.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:08.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:08.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:08.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:08.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:08.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:17:08.131 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-17T16:17:08.121+0000 7f95b5d9b640 0 --2- 172.21.15.17:0/1105856835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95b8108790 0x7f95b810d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T16:17:08.142 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:07 smithi173 ceph-mon[30530]: osdmap e281: 12 total, 12 up, 12 in 2024-09-17T16:17:08.142 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:07 smithi173 ceph-mon[30530]: 3.5 scrub starts 2024-09-17T16:17:08.164 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:08.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:08.203 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:08.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:08.305 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:08.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:07 smithi071 ceph-mon[30002]: osdmap e281: 12 total, 12 up, 12 in 2024-09-17T16:17:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:07 smithi071 ceph-mon[30002]: 3.5 scrub starts 2024-09-17T16:17:08.341 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:08.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:08.443 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:08.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:08.477 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:08.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:08.509 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:17:08.529 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:17:08.530 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:17:08.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:08.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:08.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:08.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:08.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:08.749 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:08.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:08.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:08.876 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:17:08.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:08.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:08.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:08.898 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:17:08.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:09.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:09.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:09.118 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:09.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:08 smithi173 ceph-mon[30530]: 3.5 scrub ok 2024-09-17T16:17:09.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:08 smithi173 ceph-mon[30530]: pgmap v670: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 9 active+clean; 618 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 1/29652 objects degraded (0.003%); 4620/29652 objects misplaced (15.581%); 12 MiB/s, 287 objects/s recovering 2024-09-17T16:17:09.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:08 smithi173 ceph-mon[30530]: osdmap e282: 12 total, 12 up, 12 in 2024-09-17T16:17:09.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1942577417' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:09.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:08 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:09.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:08 smithi017 ceph-mon[27571]: 3.5 scrub ok 2024-09-17T16:17:09.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:08 smithi017 ceph-mon[27571]: pgmap v670: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 9 active+clean; 618 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 1/29652 objects degraded (0.003%); 4620/29652 objects misplaced (15.581%); 12 MiB/s, 287 objects/s recovering 2024-09-17T16:17:09.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:08 smithi017 ceph-mon[27571]: osdmap e282: 12 total, 12 up, 12 in 2024-09-17T16:17:09.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1942577417' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:09.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:08 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:09.160 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:09.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:09.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:09.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:09.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:09.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:08 smithi071 ceph-mon[30002]: 3.5 scrub ok 2024-09-17T16:17:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:08 smithi071 ceph-mon[30002]: pgmap v670: 17 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 9 active+clean; 618 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 1/29652 objects degraded (0.003%); 4620/29652 objects misplaced (15.581%); 12 MiB/s, 287 objects/s recovering 2024-09-17T16:17:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:08 smithi071 ceph-mon[30002]: osdmap e282: 12 total, 12 up, 12 in 2024-09-17T16:17:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1942577417' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:08 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:09.395 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:09.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:09.420 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:09.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:09.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:09.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:09.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:09.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:09.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:09.677 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:09.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:09.753 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:17:09.754 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:17:09.754 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:17:09.754 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:17:09.754 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:17:09.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:09.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:09.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:09.909 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:09.939 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:09.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:10.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:10.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:10.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:10.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:10.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:10.191 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:09 smithi017 ceph-mon[27571]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:17:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:09 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:17:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:09 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:09 smithi017 ceph-mon[27571]: osdmap e283: 12 total, 12 up, 12 in 2024-09-17T16:17:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:09 smithi017 ceph-mon[27571]: 3.f scrub starts 2024-09-17T16:17:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:09 smithi017 ceph-mon[27571]: 3.f scrub ok 2024-09-17T16:17:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:09 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/29652 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:09 smithi017 ceph-mon[27571]: osdmap e284: 12 total, 12 up, 12 in 2024-09-17T16:17:10.291 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:09 smithi071 ceph-mon[30002]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:17:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:09 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:17:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:09 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:09 smithi071 ceph-mon[30002]: osdmap e283: 12 total, 12 up, 12 in 2024-09-17T16:17:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:09 smithi071 ceph-mon[30002]: 3.f scrub starts 2024-09-17T16:17:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:09 smithi071 ceph-mon[30002]: 3.f scrub ok 2024-09-17T16:17:10.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:09 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/29652 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:10.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:09 smithi071 ceph-mon[30002]: osdmap e284: 12 total, 12 up, 12 in 2024-09-17T16:17:10.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:10.316 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:10.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:10.388 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:09 smithi173 ceph-mon[30530]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:17:10.388 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:09 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:17:10.388 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:09 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:10.388 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:09 smithi173 ceph-mon[30530]: osdmap e283: 12 total, 12 up, 12 in 2024-09-17T16:17:10.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:09 smithi173 ceph-mon[30530]: 3.f scrub starts 2024-09-17T16:17:10.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:09 smithi173 ceph-mon[30530]: 3.f scrub ok 2024-09-17T16:17:10.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:09 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/29652 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:10.389 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:09 smithi173 ceph-mon[30530]: osdmap e284: 12 total, 12 up, 12 in 2024-09-17T16:17:10.399 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":671,"stamp":"2024-09-17T16:17:07.885363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":648151608,"num_objects":9884,"num_object_clones":0,"num_object_copies":29652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":4620,"num_objects_unfound":0,"num_objects_dirty":9884,"num_whiteouts":0,"num_read":78,"num_read_kb":66,"num_write":134149,"num_write_kb":4607143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31614,"num_bytes_recovered":2026387040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134118,"ondisk_log_size":134118,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":4863988,"kb_used_data":3220224,"kb_used_omap":58,"kb_used_meta":1643461,"kb_avail":1120077836,"statfs":{"total":1151940427776,"available":1146959704064,"internally_reserved":0,"allocated":3297509376,"data_stored":3293306266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":1682904227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168099840,"num_objects":-2565,"num_object_clones":0,"num_object_copies":-7695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1054,"num_objects_unfound":0,"num_objects_dirty":-2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2565,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1307,"num_bytes_recovered":52494336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.079228"},"pg_stats":[{"pgid":"3.f","version":"278'8280","reported_seq":14421,"reported_epoch":278,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T16:17:02.911645+0000","last_change":"2024-09-17T16:17:01.975619+0000","last_active":"2024-09-17T16:17:02.911645+0000","last_peered":"2024-09-17T16:17:02.911645+0000","last_clean":"2024-09-17T16:16:48.906063+0000","last_became_active":"2024-09-17T16:16:50.865202+0000","last_became_peered":"2024-09-17T16:16:50.865202+0000","last_unstale":"2024-09-17T16:17:02.911645+0000","last_undegraded":"2024-09-17T16:17:02.911645+0000","last_fullsized":"2024-09-17T16:17:02.911645+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46399488,"num_objects":708,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":604,"num_objects_unfound":0,"num_objects_dirty":708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":287616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2988,"num_bytes_recovered":195821568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,8,9],"avail_no_missing":["7","8","9"],"object_location_counts":[{"shards":"7,8,9","objects":708}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"280'8538","reported_seq":14583,"reported_epoch":280,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-17T16:17:06.009836+0000","last_change":"2024-09-17T16:16:50.924833+0000","last_active":"2024-09-17T16:17:06.009836+0000","last_peered":"2024-09-17T16:17:06.009836+0000","last_clean":"2024-09-17T16:16:48.886812+0000","last_became_active":"2024-09-17T16:16:50.885818+0000","last_became_peered":"2024-09-17T16:16:50.885818+0000","last_unstale":"2024-09-17T16:17:06.009836+0000","last_undegraded":"2024-09-17T16:17:06.009836+0000","last_fullsized":"2024-09-17T16:17:06.009836+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":8538,"log_dups_size":0,"ondisk_log_size":8538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":598,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":598,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8538,"num_write_kb":292352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2937,"num_bytes_recovered":192479232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[0,4,9],"avail_no_missing":["0","4","9"],"object_location_counts":[{"shards":"0,4,9","objects":598}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"277'8182","reported_seq":8380,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-17T16:17:02.873299+0000","last_change":"2024-09-17T16:15:58.646089+0000","last_active":"2024-09-17T16:17:02.873299+0000","last_peered":"2024-09-17T16:17:02.873299+0000","last_clean":"2024-09-17T16:17:02.873299+0000","last_became_active":"2024-09-17T16:14:08.487146+0000","last_became_peered":"2024-09-17T16:14:08.487146+0000","last_unstale":"2024-09-17T16:17:02.873299+0000","last_undegraded":"2024-09-17T16:17:02.873299+0000","last_fullsized":"2024-09-17T16:17:02.873299+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'5072","last_scrub_stamp":"2024-09-17T16:15:58.645836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:15:58.645836+0000","objects_scrubbed":3823,"log_size":8182,"log_dups_size":0,"ondisk_log_size":8182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:03.598654+0000","scrub_duration":678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45154328,"num_objects":690,"num_object_clones":0,"num_object_copies":2070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":690,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8182,"num_write_kb":283841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"280'8320","reported_seq":12420,"reported_epoch":280,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:17:06.246634+0000","last_change":"2024-09-17T16:16:35.155286+0000","last_active":"2024-09-17T16:17:06.246634+0000","last_peered":"2024-09-17T16:17:06.246634+0000","last_clean":"2024-09-17T16:16:32.853565+0000","last_became_active":"2024-09-17T16:16:35.104980+0000","last_became_peered":"2024-09-17T16:16:35.104980+0000","last_unstale":"2024-09-17T16:17:06.246634+0000","last_undegraded":"2024-09-17T16:17:06.246634+0000","last_fullsized":"2024-09-17T16:16:34.865822+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":8320,"log_dups_size":0,"ondisk_log_size":8320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41025536,"num_objects":626,"num_object_clones":0,"num_object_copies":1878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8320,"num_write_kb":286272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1314,"num_bytes_recovered":86048768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[3,1],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"1,3","objects":19},{"shards":"3,5,6","objects":606},{"shards":"3,5,6,8","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"280'8409","reported_seq":8564,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-17T16:17:06.179162+0000","last_change":"2024-09-17T16:14:08.429967+0000","last_active":"2024-09-17T16:17:06.179162+0000","last_peered":"2024-09-17T16:17:06.179162+0000","last_clean":"2024-09-17T16:17:06.179162+0000","last_became_active":"2024-09-17T16:14:08.429641+0000","last_became_peered":"2024-09-17T16:14:08.429641+0000","last_unstale":"2024-09-17T16:17:06.179162+0000","last_undegraded":"2024-09-17T16:17:06.179162+0000","last_fullsized":"2024-09-17T16:17:06.179162+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":8409,"log_dups_size":0,"ondisk_log_size":8409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37421056,"num_objects":571,"num_object_clones":0,"num_object_copies":1713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8409,"num_write_kb":287360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"279'8469","reported_seq":8623,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-17T16:17:05.554538+0000","last_change":"2024-09-17T16:14:08.401229+0000","last_active":"2024-09-17T16:17:05.554538+0000","last_peered":"2024-09-17T16:17:05.554538+0000","last_clean":"2024-09-17T16:17:05.554538+0000","last_became_active":"2024-09-17T16:14:08.400841+0000","last_became_peered":"2024-09-17T16:14:08.400841+0000","last_unstale":"2024-09-17T16:17:05.554538+0000","last_undegraded":"2024-09-17T16:17:05.554538+0000","last_fullsized":"2024-09-17T16:17:05.554538+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":8469,"log_dups_size":0,"ondisk_log_size":8469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:12.654193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40828928,"num_objects":623,"num_object_clones":0,"num_object_copies":1869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8469,"num_write_kb":290944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"279'8193","reported_seq":22972,"reported_epoch":279,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-17T16:17:04.477779+0000","last_change":"2024-09-17T16:17:04.422525+0000","last_active":"2024-09-17T16:17:04.477779+0000","last_peered":"2024-09-17T16:17:04.477779+0000","last_clean":"2024-09-17T16:15:55.844817+0000","last_became_active":"2024-09-17T16:16:50.845605+0000","last_became_peered":"2024-09-17T16:16:50.845605+0000","last_unstale":"2024-09-17T16:17:04.477779+0000","last_undegraded":"2024-09-17T16:17:04.477779+0000","last_fullsized":"2024-09-17T16:17:04.477779+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":8193,"log_dups_size":0,"ondisk_log_size":8193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41877504,"num_objects":639,"num_object_clones":0,"num_object_copies":1917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":639,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8193,"num_write_kb":282624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7301,"num_bytes_recovered":432406528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,7,5],"avail_no_missing":["8","5","7","9"],"object_location_counts":[{"shards":"5,7,8","objects":639}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"280'8256","reported_seq":14186,"reported_epoch":280,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-17T16:17:06.764131+0000","last_change":"2024-09-17T16:16:50.989050+0000","last_active":"2024-09-17T16:17:06.764131+0000","last_peered":"2024-09-17T16:17:06.764131+0000","last_clean":"2024-09-17T16:16:48.909634+0000","last_became_active":"2024-09-17T16:16:50.864905+0000","last_became_peered":"2024-09-17T16:16:50.864905+0000","last_unstale":"2024-09-17T16:17:06.764131+0000","last_undegraded":"2024-09-17T16:17:06.764131+0000","last_fullsized":"2024-09-17T16:17:06.764131+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'5214","last_scrub_stamp":"2024-09-17T16:16:02.968730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:16:02.968730+0000","objects_scrubbed":3616,"log_size":8256,"log_dups_size":0,"ondisk_log_size":8256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":556,"num_object_clones":0,"num_object_copies":1668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":556,"num_objects_unfound":0,"num_objects_dirty":556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":281984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2855,"num_bytes_recovered":187039744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":556}],"blocked_by":[],"up_primary":11,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"280'8494","reported_seq":14614,"reported_epoch":280,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-17T16:17:06.317303+0000","last_change":"2024-09-17T16:16:50.934368+0000","last_active":"2024-09-17T16:17:06.317303+0000","last_peered":"2024-09-17T16:17:06.317303+0000","last_clean":"2024-09-17T16:16:48.900600+0000","last_became_active":"2024-09-17T16:16:50.886035+0000","last_became_peered":"2024-09-17T16:16:50.886035+0000","last_unstale":"2024-09-17T16:17:06.317303+0000","last_undegraded":"2024-09-17T16:17:06.317303+0000","last_fullsized":"2024-09-17T16:17:06.317303+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":8494,"log_dups_size":0,"ondisk_log_size":8494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37879808,"num_objects":578,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":578,"num_objects_unfound":0,"num_objects_dirty":578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8494,"num_write_kb":290304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2972,"num_bytes_recovered":194707456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,7,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":578}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"280'8489","reported_seq":8678,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-17T16:17:06.806544+0000","last_change":"2024-09-17T16:15:14.282285+0000","last_active":"2024-09-17T16:17:06.806544+0000","last_peered":"2024-09-17T16:17:06.806544+0000","last_clean":"2024-09-17T16:17:06.806544+0000","last_became_active":"2024-09-17T16:14:08.429516+0000","last_became_peered":"2024-09-17T16:14:08.429516+0000","last_unstale":"2024-09-17T16:17:06.806544+0000","last_undegraded":"2024-09-17T16:17:06.806544+0000","last_fullsized":"2024-09-17T16:17:06.806544+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'2873","last_scrub_stamp":"2024-09-17T16:15:14.282192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:15:14.282192+0000","objects_scrubbed":2873,"log_size":8489,"log_dups_size":0,"ondisk_log_size":8489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:28.102293+0000","scrub_duration":649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39780352,"num_objects":607,"num_object_clones":0,"num_object_copies":1821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8489,"num_write_kb":291072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"279'8347","reported_seq":8499,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-17T16:17:05.443119+0000","last_change":"2024-09-17T16:14:08.391300+0000","last_active":"2024-09-17T16:17:05.443119+0000","last_peered":"2024-09-17T16:17:05.443119+0000","last_clean":"2024-09-17T16:17:05.443119+0000","last_became_active":"2024-09-17T16:14:08.390988+0000","last_became_peered":"2024-09-17T16:14:08.390988+0000","last_unstale":"2024-09-17T16:17:05.443119+0000","last_undegraded":"2024-09-17T16:17:05.443119+0000","last_fullsized":"2024-09-17T16:17:05.443119+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42532864,"num_objects":649,"num_object_clones":0,"num_object_copies":1947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8347,"num_write_kb":287872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"280'8423","reported_seq":16417,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-17T16:17:06.793519+0000","last_change":"2024-09-17T16:17:03.903775+0000","last_active":"2024-09-17T16:17:06.793519+0000","last_peered":"2024-09-17T16:17:06.793519+0000","last_clean":"2024-09-17T16:17:06.793519+0000","last_became_active":"2024-09-17T16:17:03.875461+0000","last_became_peered":"2024-09-17T16:17:03.875461+0000","last_unstale":"2024-09-17T16:17:06.793519+0000","last_undegraded":"2024-09-17T16:17:06.793519+0000","last_fullsized":"2024-09-17T16:17:06.793519+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'5216","last_scrub_stamp":"2024-09-17T16:16:00.949575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:16:00.949575+0000","objects_scrubbed":3776,"log_size":8423,"log_dups_size":0,"ondisk_log_size":8423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:27.794196+0000","scrub_duration":723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37158912,"num_objects":567,"num_object_clones":0,"num_object_copies":1701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":287680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3885,"num_bytes_recovered":254607360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"279'8376","reported_seq":8549,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-17T16:17:04.497653+0000","last_change":"2024-09-17T16:16:56.012393+0000","last_active":"2024-09-17T16:17:04.497653+0000","last_peered":"2024-09-17T16:17:04.497653+0000","last_clean":"2024-09-17T16:17:04.497653+0000","last_became_active":"2024-09-17T16:14:08.429652+0000","last_became_peered":"2024-09-17T16:14:08.429652+0000","last_unstale":"2024-09-17T16:17:04.497653+0000","last_undegraded":"2024-09-17T16:17:04.497653+0000","last_fullsized":"2024-09-17T16:17:04.497653+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'8071","last_scrub_stamp":"2024-09-17T16:16:56.012243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:16:56.012243+0000","objects_scrubbed":1028,"log_size":8376,"log_dups_size":0,"ondisk_log_size":8376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:16.689807+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46792704,"num_objects":714,"num_object_clones":0,"num_object_copies":2142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8376,"num_write_kb":290880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"279'8458","reported_seq":16928,"reported_epoch":279,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-17T16:17:05.703947+0000","last_change":"2024-09-17T16:16:49.583933+0000","last_active":"2024-09-17T16:17:05.703947+0000","last_peered":"2024-09-17T16:17:05.703947+0000","last_clean":"2024-09-17T16:15:55.823864+0000","last_became_active":"2024-09-17T16:16:49.551604+0000","last_became_peered":"2024-09-17T16:16:49.551604+0000","last_unstale":"2024-09-17T16:17:05.703947+0000","last_undegraded":"2024-09-17T16:16:49.448254+0000","last_fullsized":"2024-09-17T16:16:49.356087+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":8458,"log_dups_size":0,"ondisk_log_size":8458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41811968,"num_objects":638,"num_object_clones":0,"num_object_copies":1914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":136,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8458,"num_write_kb":291072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4158,"num_bytes_recovered":272105472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,11],"acting":[10,0],"avail_no_missing":["10","0","4"],"object_location_counts":[{"shards":"0,4,10","objects":136},{"shards":"0,10","objects":501}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"280'8436","reported_seq":15048,"reported_epoch":280,"state":"active+remapped+backfilling","last_fresh":"2024-09-17T16:17:06.794371+0000","last_change":"2024-09-17T16:17:04.433777+0000","last_active":"2024-09-17T16:17:06.794371+0000","last_peered":"2024-09-17T16:17:06.794371+0000","last_clean":"2024-09-17T16:16:48.885282+0000","last_became_active":"2024-09-17T16:16:50.890724+0000","last_became_peered":"2024-09-17T16:16:50.890724+0000","last_unstale":"2024-09-17T16:17:06.794371+0000","last_undegraded":"2024-09-17T16:17:06.794371+0000","last_fullsized":"2024-09-17T16:17:06.794371+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'5353","last_scrub_stamp":"2024-09-17T16:16:02.459059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:16:02.459059+0000","objects_scrubbed":3654,"log_size":8436,"log_dups_size":0,"ondisk_log_size":8436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":538,"num_object_clones":0,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":296,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":287168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3198,"num_bytes_recovered":209584128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[1,9,6],"avail_no_missing":["1","6","9"],"object_location_counts":[{"shards":"1,6,9","objects":538}],"blocked_by":[],"up_primary":11,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"280'8356","reported_seq":8511,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-17T16:17:06.305263+0000","last_change":"2024-09-17T16:14:08.487206+0000","last_active":"2024-09-17T16:17:06.305263+0000","last_peered":"2024-09-17T16:17:06.305263+0000","last_clean":"2024-09-17T16:17:06.305263+0000","last_became_active":"2024-09-17T16:14:08.486914+0000","last_became_peered":"2024-09-17T16:14:08.486914+0000","last_unstale":"2024-09-17T16:17:06.305263+0000","last_undegraded":"2024-09-17T16:17:06.305263+0000","last_fullsized":"2024-09-17T16:17:06.305263+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:14:07.369036+0000","last_clean_scrub_stamp":"2024-09-17T16:14:07.369036+0000","objects_scrubbed":0,"log_size":8356,"log_dups_size":0,"ondisk_log_size":8356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:14.294941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38010880,"num_objects":580,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":285952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":449,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-17T16:17:02.873224+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:17:02.873224+0000","last_peered":"2024-09-17T16:17:02.873224+0000","last_clean":"2024-09-17T16:17:02.873224+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:17:02.873224+0000","last_undegraded":"2024-09-17T16:17:02.873224+0000","last_fullsized":"2024-09-17T16:17:02.873224+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:17:49.956805+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":647561240,"num_objects":9882,"num_object_clones":0,"num_object_copies":29646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":4620,"num_objects_unfound":0,"num_objects_dirty":9882,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":134026,"num_write_kb":4604993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31608,"num_bytes_recovered":2024800256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3225497600,"data_stored":3225485384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134026,"ondisk_log_size":134026,"up":48,"acting":46,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285659,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":761832,"kb_used_data":626520,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92983320,"statfs":{"total":95995035648,"available":95214919680,"internally_reserved":0,"allocated":641556480,"data_stored":641209146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":138541508},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110516,"kb_used_data":43228,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93634636,"statfs":{"total":95995035648,"available":95881867264,"internally_reserved":0,"allocated":44265472,"data_stored":43917459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743525,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":635392,"kb_used_data":433452,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93109760,"statfs":{"total":95995035648,"available":95344394240,"internally_reserved":0,"allocated":443854848,"data_stored":443498554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":206764480},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005161,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193524,"kb_used_data":90140,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93551628,"statfs":{"total":95995035648,"available":95796867072,"internally_reserved":0,"allocated":92303360,"data_stored":91955347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963638,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319008,"kb_used_data":190032,"kb_used_omap":41,"kb_used_meta":128918,"kb_avail":93426144,"statfs":{"total":95995035648,"available":95668371456,"internally_reserved":0,"allocated":194592768,"data_stored":194245612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":132012339},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495732,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405692,"kb_used_data":284260,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93339460,"statfs":{"total":95995035648,"available":95579607040,"internally_reserved":0,"allocated":291082240,"data_stored":290730675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724663,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543152,"kb_used_data":367584,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93202000,"statfs":{"total":95995035648,"available":95438848000,"internally_reserved":0,"allocated":376406016,"data_stored":376053931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179763654},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986301,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659172,"kb_used_data":523856,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93085980,"statfs":{"total":95995035648,"available":95320043520,"internally_reserved":0,"allocated":536428544,"data_stored":536084782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339008,"kb_used_data":191208,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93406144,"statfs":{"total":95995035648,"available":95647891456,"internally_reserved":0,"allocated":195796992,"data_stored":195444770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":151932,"kb_used_data":40232,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93593220,"statfs":{"total":95995035648,"available":95839457280,"internally_reserved":0,"allocated":41197568,"data_stored":40845346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574989,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":370240,"kb_used_data":194664,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93374912,"statfs":{"total":95995035648,"available":95615909888,"internally_reserved":0,"allocated":199335936,"data_stored":198983714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738514,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":374520,"kb_used_data":235048,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93370632,"statfs":{"total":95995035648,"available":95611527168,"internally_reserved":0,"allocated":240689152,"data_stored":240336930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238944256,"data_stored":238944256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186318848,"data_stored":186318848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":38076416,"data_stored":38076416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":194183168,"data_stored":194183168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":525336576,"data_stored":525336576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":361893888,"data_stored":361889816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":288423936,"data_stored":288423936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":180752384,"data_stored":180748312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":88604672,"data_stored":88604672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":428478464,"data_stored":428474392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":41811968,"data_stored":41811968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":652673024,"data_stored":652673024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:17:10.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-09-17T16:17:10.418 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:10.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:10.550 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:10.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:17:10.602 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:10.667 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:10.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:10.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:10.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:10.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:10.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:10.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:10.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:10.965 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:17:11.041 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:11.066 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:11.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:11.168 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:11.192 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:11.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:11.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:11.332 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:11.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:11.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:11.464 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:11.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:11.527 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:21834 2.12 2:4e087699:::benchmark_data_smithi173_55121_object21833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 344.53572939499998, 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23867154600000001, 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.528 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21834 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987494+0000", 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2919999999999998e-06 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987518+0000", 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3598e-05 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987534+0000", 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5896000000000002e-05 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987542+0000", 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1829999999999994e-06 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987592+0000", 2024-09-17T16:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0033e-05 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987671+0000", 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8993999999999996e-05 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987759+0000", 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7751999999999996e-05 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988272+0000", 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051340700000000001 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988663+0000", 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039071199999999998 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226103+0000", 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23744021000000001 2024-09-17T16:17:11.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226137+0000", 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3929e-05 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226164+0000", 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.654e-05 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:21834 2.12 e117/114 ondisk, result = 0)", 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 344.534619835, 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23756740700000001, 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:17:11.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988605+0000", 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0079999999999998e-06 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226056+0000", 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23745111999999999 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226062+0000", 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9739999999999999e-06 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226095+0000", 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2503999999999999e-05 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226100+0000", 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9049999999999996e-06 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226169+0000", 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9303999999999999e-05 2024-09-17T16:17:11.538 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:26989 2.12 2:48690473:::benchmark_data_smithi173_55121_object26988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 338.60200569800003, 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22401436399999999, 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:17:11.539 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 26989 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921217+0000", 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5e-06 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921237+0000", 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9777000000000001e-05 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921250+0000", 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2704999999999999e-05 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921256+0000", 2024-09-17T16:17:11.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4080000000000003e-06 2024-09-17T16:17:11.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921298+0000", 2024-09-17T16:17:11.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2169000000000002e-05 2024-09-17T16:17:11.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921385+0000", 2024-09-17T16:17:11.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6536e-05 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921467+0000", 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2657999999999994e-05 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922015+0000", 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054718699999999998 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922501+0000", 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048639400000000002 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145060+0000", 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.222559429 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145161+0000", 2024-09-17T16:17:11.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100058 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145230+0000", 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9542999999999995e-05 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:26989 2.12 e121/114 ondisk, result = 0)", 2024-09-17T16:17:11.546 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:17:11.547 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 338.60074406400003, 2024-09-17T16:17:11.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22276984499999999, 2024-09-17T16:17:11.547 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.547 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.547 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:17:11.547 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922478+0000", 2024-09-17T16:17:11.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9100000000000004e-07 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922484+0000", 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.144e-06 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.144878+0000", 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.222394432 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.144910+0000", 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1365999999999998e-05 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145033+0000", 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122717 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145050+0000", 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.753e-05 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145247+0000", 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019706500000000001 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 136/134 [PushReplyOp(2:482f57f9:::benchmark_data_smithi173_55121_object24323:head)])", 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 309.35925773100001, 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.229086917, 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.551 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:17:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0200000000000002e-07 2024-09-17T16:17:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163968+0000", 2024-09-17T16:17:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9369999999999997e-06 2024-09-17T16:17:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392756+0000", 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22878752399999999 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392762+0000", 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0959999999999997e-06 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392828+0000", 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5746000000000004e-05 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.393051+0000", 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000223012 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:93035 2.12 2:4eb811d6:::benchmark_data_smithi173_55121_object16546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.696790+0000", 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 262.82643142299997, 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22142989699999999, 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.555 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93035 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696790+0000", 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696790+0000", 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696792+0000", 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.592e-06 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696796+0000", 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4859999999999999e-06 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696806+0000", 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0261999999999999e-05 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696812+0000", 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4960000000000004e-06 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696851+0000", 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9588000000000001e-05 2024-09-17T16:17:11.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.696952+0000", 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100528 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.697033+0000", 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1314000000000001e-05 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.698646+0000", 2024-09-17T16:17:11.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016126459999999999 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.698655+0000", 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6290000000000005e-06 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917751+0000", 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21909500900000001 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917859+0000", 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108412 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.918220+0000", 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000360935 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.561 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:93035 2.12 e158/155 ondisk, result = 0)", 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.699652+0000", 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 262.82356911300002, 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21857501500000001, 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699652+0000", 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699652+0000", 2024-09-17T16:17:11.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699653+0000", 2024-09-17T16:17:11.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5600000000000005e-07 2024-09-17T16:17:11.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.699657+0000", 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.092e-06 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917601+0000", 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21794412199999999 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917615+0000", 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3397000000000001e-05 2024-09-17T16:17:11.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917723+0000", 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108371 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.917739+0000", 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6042999999999999e-05 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.918227+0000", 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048833399999999999 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 158/155 [PushReplyOp(2:4d5499b6:::benchmark_data_smithi173_55121_object58972:head)])", 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 262.76154620400001, 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46593107700000003, 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:17:11.566 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761676+0000", 2024-09-17T16:17:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1900000000000002e-07 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761680+0000", 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1710000000000004e-06 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226484+0000", 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46480413500000001 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226502+0000", 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7895999999999999e-05 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226671+0000", 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016863300000000001 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.227606+0000", 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093562300000000001 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.569 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:81435 3.1 3:829f7929:::benchmark_data_smithi173_66272_object9479:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:57.882513+0000", 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.640708683, 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.216077243, 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.570 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81435 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882513+0000", 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882513+0000", 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882513+0000", 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3300000000000002e-07 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882516+0000", 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1310000000000002e-06 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882525+0000", 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3950000000000001e-06 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882530+0000", 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6149999999999996e-06 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884219+0000", 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001688143 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884225+0000", 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6220000000000003e-06 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.070751+0000", 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.186525778 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.072299+0000", 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001547688 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.072460+0000", 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016171600000000001 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.077846+0000", 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0053857389999999996 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.082941+0000", 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0050951089999999996 2024-09-17T16:17:11.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.096818+0000", 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013876767999999999 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.096858+0000", 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0460999999999998e-05 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.098590+0000", 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017315449999999999 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:81438 3.1 3:802eaca6:::benchmark_data_smithi173_66272_object9482:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:57.882539+0000", 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.640682733000006, 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22175815900000001, 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.577 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81438 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882539+0000", 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882539+0000", 2024-09-17T16:17:11.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882539+0000", 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3799999999999998e-07 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882541+0000", 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2069999999999998e-06 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882549+0000", 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.233e-06 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882553+0000", 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9370000000000003e-06 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884267+0000", 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001713932 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884276+0000", 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1499999999999999e-06 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.074278+0000", 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.190002897 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.074432+0000", 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015365800000000001 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.074590+0000", 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000157987 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.081496+0000", 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006906321 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.089622+0000", 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.008125742 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.103020+0000", 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.01339811 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.103089+0000", 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8745000000000007e-05 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.104297+0000", 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001207802 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:81489 3.1 3:8126b738:::benchmark_data_smithi173_66272_object9533:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T16:17:11.586 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:57.882560+0000", 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.640661078999997, 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22406166199999999, 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81489 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882560+0000", 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882560+0000", 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882561+0000", 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0400000000000002e-07 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882563+0000", 2024-09-17T16:17:11.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.345e-06 2024-09-17T16:17:11.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882569+0000", 2024-09-17T16:17:11.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4909999999999996e-06 2024-09-17T16:17:11.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:11.590 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:11 smithi017 ceph-mon[27571]: pgmap v674: 17 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 10 active+clean; 569 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 1/27309 objects degraded (0.004%); 3307/27309 objects misplaced (12.110%); 25 MiB/s, 393 objects/s recovering 2024-09-17T16:17:11.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:11 smithi017 ceph-mon[27571]: 3.d scrub starts 2024-09-17T16:17:11.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:11 smithi017 ceph-mon[27571]: 3.d scrub ok 2024-09-17T16:17:11.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:11 smithi017 ceph-mon[27571]: osdmap e285: 12 total, 12 up, 12 in 2024-09-17T16:17:11.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:17:11.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:11 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:17:11.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2589671585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882573+0000", 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6099999999999999e-06 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884308+0000", 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001735222 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:17:11.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884315+0000", 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7170000000000002e-06 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.076161+0000", 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.191846344 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.076292+0000", 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013085899999999999 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.076378+0000", 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5451999999999994e-05 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.082952+0000", 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006574084 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.089673+0000", 2024-09-17T16:17:11.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067210669999999998 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.105569+0000", 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015896065000000001 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.105632+0000", 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2650000000000005e-05 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.106622+0000", 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0009903519999999999 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:81512 3.1 3:87dcd0a7:::benchmark_data_smithi173_66272_object9556:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:57.882579+0000", 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.640641985000002, 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225002383, 2024-09-17T16:17:11.596 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81512 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882579+0000", 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882579+0000", 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882580+0000", 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.72e-07 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882582+0000", 2024-09-17T16:17:11.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0169999999999999e-06 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882588+0000", 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8939999999999998e-06 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882597+0000", 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2569999999999996e-06 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884361+0000", 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001764072 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884370+0000", 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6519999999999995e-06 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.077907+0000", 2024-09-17T16:17:11.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19353769500000001 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.078069+0000", 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016133599999999999 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.078214+0000", 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000145251 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.082957+0000", 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047430140000000003 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.092025+0000", 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0090680489999999999 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.106670+0000", 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014645228 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.106707+0000", 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7264000000000001e-05 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.107582+0000", 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00087428199999999999 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:81529 3.1 3:8973215e:::benchmark_data_smithi173_66272_object9573:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T16:17:11.603 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:57.882603+0000", 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.640618214, 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225980187, 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81529 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882603+0000", 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882603+0000", 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882604+0000", 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0999999999999999e-07 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882606+0000", 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2840000000000001e-06 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882613+0000", 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8580000000000002e-06 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882617+0000", 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6179999999999997e-06 2024-09-17T16:17:11.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884416+0000", 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017984640000000001 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884420+0000", 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4070000000000003e-06 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.079807+0000", 2024-09-17T16:17:11.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19538671799999999 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.079918+0000", 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111414 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.080023+0000", 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104528 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.083012+0000", 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029894969999999998 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.092069+0000", 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0090566880000000002 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.107623+0000", 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015553502 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.107658+0000", 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5256999999999998e-05 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.108583+0000", 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00092554200000000001 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:81555 3.1 3:89638963:::benchmark_data_smithi173_66272_object9599:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:57.882624+0000", 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.640597595000003, 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.236389865, 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81555 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882624+0000", 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882624+0000", 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882624+0000", 2024-09-17T16:17:11.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9000000000000002e-07 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882626+0000", 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1950000000000002e-06 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882633+0000", 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9330000000000004e-06 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882638+0000", 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7940000000000002e-06 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884436+0000", 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017976209999999999 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884440+0000", 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1930000000000003e-06 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.081537+0000", 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19709721599999999 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.081626+0000", 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8968999999999995e-05 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.081704+0000", 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8005999999999998e-05 2024-09-17T16:17:11.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.083065+0000", 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001360485 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.092102+0000", 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0090376789999999999 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.117339+0000", 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025236811000000001 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.117377+0000", 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7514e-05 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.119014+0000", 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001637059 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.617 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:89674 3.1 3:8f13ebaf:::benchmark_data_smithi173_66272_object17718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:07.253083+0000", 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 64.270138430000003, 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22595691500000001, 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 89674 2024-09-17T16:17:11.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253083+0000", 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253083+0000", 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253085+0000", 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.872e-06 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253092+0000", 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9789999999999996e-06 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253103+0000", 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1595e-05 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253113+0000", 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8120000000000002e-06 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.258528+0000", 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.005414355 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.258645+0000", 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117006 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.258798+0000", 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000153518 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.268089+0000", 2024-09-17T16:17:11.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0092910890000000006 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.281456+0000", 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013367005 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.477474+0000", 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.196018055 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.477520+0000", 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5707000000000001e-05 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.479040+0000", 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015199219999999999 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:89698 3.1 3:881eb31e:::benchmark_data_smithi173_66272_object17742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:07.276389+0000", 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 64.246831904999993, 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21076109100000001, 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 89698 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.625 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.276389+0000", 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.276389+0000", 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.276391+0000", 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.39e-06 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.276397+0000", 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2450000000000003e-06 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.276410+0000", 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2648e-05 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.276416+0000", 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3090000000000001e-06 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.276470+0000", 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3451e-05 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.276572+0000", 2024-09-17T16:17:11.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102038 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.276705+0000", 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013372699999999999 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.281375+0000", 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046692640000000002 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.299821+0000", 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.018446799 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.485624+0000", 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.185803152 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.485662+0000", 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7673000000000003e-05 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.487151+0000", 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001488395 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:92396 3.1 3:87df27fa:::benchmark_data_smithi173_66272_object20440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:10.408630+0000", 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 61.114591472999997, 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.259313497, 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 92396 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408630+0000", 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408630+0000", 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408632+0000", 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.108e-06 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408640+0000", 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2470000000000008e-06 2024-09-17T16:17:11.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408657+0000", 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6461999999999999e-05 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408666+0000", 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7979999999999995e-06 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408715+0000", 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9468000000000001e-05 2024-09-17T16:17:11.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408797+0000", 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2330000000000006e-05 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.453598+0000", 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.044800409999999999 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.453661+0000", 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2887999999999998e-05 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.453753+0000", 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2545000000000002e-05 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.461059+0000", 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0073062550000000002 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.462616+0000", 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001556778 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.665956+0000", 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20334012100000001 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.665996+0000", 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9227999999999998e-05 2024-09-17T16:17:11.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.667943+0000", 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019478589999999999 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124023 3.1 3:860559f8:::benchmark_data_smithi173_66272_object52067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.587616+0000", 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 22.935605444, 2024-09-17T16:17:11.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23693346200000001, 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124023 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587616+0000", 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587616+0000", 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587617+0000", 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.16e-07 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587621+0000", 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3630000000000004e-06 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587633+0000", 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1579999999999999e-05 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587639+0000", 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3419999999999996e-06 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.591590+0000", 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039506660000000002 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.591736+0000", 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014601800000000001 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.591880+0000", 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000144475 2024-09-17T16:17:11.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.596612+0000", 2024-09-17T16:17:11.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047316169999999996 2024-09-17T16:17:11.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.603533+0000", 2024-09-17T16:17:11.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069211009999999998 2024-09-17T16:17:11.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.823789+0000", 2024-09-17T16:17:11.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.220256128 2024-09-17T16:17:11.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.823822+0000", 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3254999999999999e-05 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.824549+0000", 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00072700100000000001 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.645 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25036.0:124023 3.1 e271/256 ondisk, result = 0)", 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.597700+0000", 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 22.925521684, 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22685955299999999, 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.597700+0000", 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.597700+0000", 2024-09-17T16:17:11.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.597700+0000", 2024-09-17T16:17:11.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4700000000000001e-07 2024-09-17T16:17:11.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.597703+0000", 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2499999999999998e-06 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.820323+0000", 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22261969100000001 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.820328+0000", 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5709999999999999e-06 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.823776+0000", 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034483769999999999 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.823784+0000", 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.503e-06 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.824559+0000", 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077561399999999999 2024-09-17T16:17:11.649 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:134243 3.1 3:8c1e41ff:::benchmark_data_smithi173_66272_object62287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:05.911449+0000", 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.6117721879999998, 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27136616899999999, 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:17:11.650 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 134243 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911449+0000", 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911449+0000", 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:17:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911451+0000", 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3430000000000001e-06 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911455+0000", 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.036e-06 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911465+0000", 2024-09-17T16:17:11.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0064000000000001e-05 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911470+0000", 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9849999999999997e-06 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911512+0000", 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2001e-05 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:17:11.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911657+0000", 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000145494 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911804+0000", 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000146997 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.915415+0000", 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036104900000000001 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.922245+0000", 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0068306850000000004 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:06.182270+0000", 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26002487800000001 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:06.182337+0000", 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7174000000000004e-05 2024-09-17T16:17:11.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:17:11.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:17:11.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:17:11.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:06.182815+0000", 2024-09-17T16:17:11.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047802199999999998 2024-09-17T16:17:11.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.656 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:17:11.656 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:17:11.656 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:17:11.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:11 smithi173 ceph-mon[30530]: pgmap v674: 17 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 10 active+clean; 569 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 1/27309 objects degraded (0.004%); 3307/27309 objects misplaced (12.110%); 25 MiB/s, 393 objects/s recovering 2024-09-17T16:17:11.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:11 smithi173 ceph-mon[30530]: 3.d scrub starts 2024-09-17T16:17:11.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:11 smithi173 ceph-mon[30530]: 3.d scrub ok 2024-09-17T16:17:11.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:11 smithi173 ceph-mon[30530]: osdmap e285: 12 total, 12 up, 12 in 2024-09-17T16:17:11.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:17:11.658 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:11 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:17:11.658 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2589671585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:11.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:11.680 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:11.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:11.781 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:11.811 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:11.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:11.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:11 smithi071 ceph-mon[30002]: pgmap v674: 17 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 10 active+clean; 569 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 1/27309 objects degraded (0.004%); 3307/27309 objects misplaced (12.110%); 25 MiB/s, 393 objects/s recovering 2024-09-17T16:17:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:11 smithi071 ceph-mon[30002]: 3.d scrub starts 2024-09-17T16:17:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:11 smithi071 ceph-mon[30002]: 3.d scrub ok 2024-09-17T16:17:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:11 smithi071 ceph-mon[30002]: osdmap e285: 12 total, 12 up, 12 in 2024-09-17T16:17:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:17:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:11 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:17:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2589671585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:11.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:11.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:11.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:11.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:17:12.045 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:12.070 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:17:12.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:17:12.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:17:12.073 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:17:12.078 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:12.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:12.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:12.180 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:12.207 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:12.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:12.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:12.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:12.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:12.437 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:12.462 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:12.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:12.564 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:12.588 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:12.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:12.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:12.712 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:12.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:12.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:17:12.814 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:12.838 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:12.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:12.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:12.961 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:12.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:13.063 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:13.064 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:17:13.086 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:13.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:13.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:13.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:13.209 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:13.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:13.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:13.332 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:13.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:13.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:13.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:13.462 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:13.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:13.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:13.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:13 smithi173 ceph-mon[30530]: pgmap v676: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 10 active+clean; 485 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 532 op/s; 1/23262 objects degraded (0.004%); 2385/23262 objects misplaced (10.253%); 23 MiB/s, 432 objects/s recovering 2024-09-17T16:17:13.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:13 smithi173 ceph-mon[30530]: osdmap e286: 12 total, 12 up, 12 in 2024-09-17T16:17:13.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:13.600 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:13.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:13.608 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:17:13.608 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:17:13.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:13.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:13.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:13 smithi017 ceph-mon[27571]: pgmap v676: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 10 active+clean; 485 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 532 op/s; 1/23262 objects degraded (0.004%); 2385/23262 objects misplaced (10.253%); 23 MiB/s, 432 objects/s recovering 2024-09-17T16:17:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:13 smithi017 ceph-mon[27571]: osdmap e286: 12 total, 12 up, 12 in 2024-09-17T16:17:13.744 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:13 smithi071 ceph-mon[30002]: pgmap v676: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 10 active+clean; 485 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 532 op/s; 1/23262 objects degraded (0.004%); 2385/23262 objects misplaced (10.253%); 23 MiB/s, 432 objects/s recovering 2024-09-17T16:17:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:13 smithi071 ceph-mon[30002]: osdmap e286: 12 total, 12 up, 12 in 2024-09-17T16:17:13.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:13.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:13.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:13.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:14.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:14.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:14.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:14.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:14.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:14.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:14.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:14.265 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:14.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:14.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:17:14.366 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:14.392 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:14.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:14.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:14.520 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:14.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:14.564 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:17:14.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:17:14.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:14 smithi017 ceph-mon[27571]: 3.a scrub starts 2024-09-17T16:17:14.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:14 smithi017 ceph-mon[27571]: 3.a scrub ok 2024-09-17T16:17:14.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:14 smithi017 ceph-mon[27571]: osdmap e287: 12 total, 12 up, 12 in 2024-09-17T16:17:14.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:14 smithi017 ceph-mon[27571]: pgmap v679: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 10 active+clean; 429 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 1/20586 objects degraded (0.005%); 2218/20586 objects misplaced (10.774%); 13 MiB/s, 263 objects/s recovering 2024-09-17T16:17:14.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:14 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/23262 objects degraded (0.004%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:14.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:17:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:17:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:14 smithi173 ceph-mon[30530]: 3.a scrub starts 2024-09-17T16:17:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:14 smithi173 ceph-mon[30530]: 3.a scrub ok 2024-09-17T16:17:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:14 smithi173 ceph-mon[30530]: osdmap e287: 12 total, 12 up, 12 in 2024-09-17T16:17:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:14 smithi173 ceph-mon[30530]: pgmap v679: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 10 active+clean; 429 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 1/20586 objects degraded (0.005%); 2218/20586 objects misplaced (10.774%); 13 MiB/s, 263 objects/s recovering 2024-09-17T16:17:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:14 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/23262 objects degraded (0.004%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:14.653 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:14.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:14.682 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:17:14.755 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:17:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:17:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:14 smithi071 ceph-mon[30002]: 3.a scrub starts 2024-09-17T16:17:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:14 smithi071 ceph-mon[30002]: 3.a scrub ok 2024-09-17T16:17:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:14 smithi071 ceph-mon[30002]: osdmap e287: 12 total, 12 up, 12 in 2024-09-17T16:17:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:14 smithi071 ceph-mon[30002]: pgmap v679: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 peering, 3 active+remapped+backfill_toofull, 10 active+clean; 429 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 1/20586 objects degraded (0.005%); 2218/20586 objects misplaced (10.774%); 13 MiB/s, 263 objects/s recovering 2024-09-17T16:17:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:14 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/23262 objects degraded (0.004%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:14.819 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:14.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:14.898 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:14.921 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:14.953 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:14.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:15.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:15.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:15.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:15.188 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:15.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:15.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:15.288 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:17:15.289 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:17:15.336 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:15.372 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:15.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:15.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:15.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:15 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:17:15.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:15 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:17:15.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:15 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:17:15.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2795691735' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:17:15.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:15 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:17:15.497 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:15.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:15.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:15.600 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:15.629 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:15.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:15.673 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:15 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:17:15.673 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:15 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:17:15.673 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:15 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:17:15.673 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2795691735' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:17:15.674 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:15 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:17:15.730 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:15.760 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:15.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:15 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:17:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:15 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:17:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:15 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:17:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2795691735' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:17:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:15 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:17:15.822 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 3 pgp_num to 16 2024-09-17T16:17:15.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:15.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:15.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:15.890 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:15.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:16.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:16.025 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:16.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:16.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:17:16.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:16.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:16.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:16.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:16.276 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:16.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:16.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:17:16.378 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:17:16.378 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:16.441 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:16.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:16.543 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:16.577 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:16.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:16.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:16.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:16.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:16.749 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:16.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:16.823 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:17:16.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:16.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:16.840 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:16 smithi017 ceph-mon[27571]: pgmap v680: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 12 active+clean; 352 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 1/16857 objects degraded (0.006%); 1992/16857 objects misplaced (11.817%); 5.9 MiB/s, 226 objects/s recovering 2024-09-17T16:17:16.840 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:16 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T16:17:16.840 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:16 smithi017 ceph-mon[27571]: osdmap e288: 12 total, 12 up, 12 in 2024-09-17T16:17:16.840 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1246440463' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:17:16.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:16.963 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:16.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:16.979 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:17:16.979 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:17:17.064 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:17.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:17.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:17.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:16 smithi173 ceph-mon[30530]: pgmap v680: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 12 active+clean; 352 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 1/16857 objects degraded (0.006%); 1992/16857 objects misplaced (11.817%); 5.9 MiB/s, 226 objects/s recovering 2024-09-17T16:17:17.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:16 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T16:17:17.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:16 smithi173 ceph-mon[30530]: osdmap e288: 12 total, 12 up, 12 in 2024-09-17T16:17:17.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1246440463' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:17:17.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:17.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:17.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:17.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:16 smithi071 ceph-mon[30002]: pgmap v680: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_toofull, 12 active+clean; 352 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 1/16857 objects degraded (0.006%); 1992/16857 objects misplaced (11.817%); 5.9 MiB/s, 226 objects/s recovering 2024-09-17T16:17:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:16 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T16:17:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:16 smithi071 ceph-mon[30002]: osdmap e288: 12 total, 12 up, 12 in 2024-09-17T16:17:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1246440463' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:17:17.330 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:17.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:17.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:17.391 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:17:17.391 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:17:17.423 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:17:17.423 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:17:17.450 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:17:17.451 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:17:17.466 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:17.481 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:17:17.481 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:17:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:17.490 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:17.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:17.508 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:17:17.509 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:17:17.535 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:17:17.536 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:17:17.565 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:17:17.565 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:17:17.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:17.595 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:17:17.595 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:17:17.616 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:17.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:17.627 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:17:17.628 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:17:17.658 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:17:17.659 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:17:17.691 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:17:17.691 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:17:17.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:17.726 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:17:17.726 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:17:17.754 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:17.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:17.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:17.888 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:17.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:17.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:17 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:17.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1246440463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:17:17.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:17 smithi017 ceph-mon[27571]: osdmap e289: 12 total, 12 up, 12 in 2024-09-17T16:17:17.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2374367508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:17.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:18.028 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:18.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:18.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:18.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:17 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:18.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1246440463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:17:18.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:17 smithi173 ceph-mon[30530]: osdmap e289: 12 total, 12 up, 12 in 2024-09-17T16:17:18.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2374367508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:18.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:18.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:18.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:18.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:17:18.288 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:18.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:18.295 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:17 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:18.295 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1246440463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:17:18.295 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:17 smithi071 ceph-mon[30002]: osdmap e289: 12 total, 12 up, 12 in 2024-09-17T16:17:18.295 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2374367508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:18.390 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:18.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:18.429 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:18.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:18.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:18.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:18.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:18.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:18.595 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:17:18.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:18.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:18.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:18.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:18.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:18.806 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:18.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:18.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:18.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:18.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:18.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:18.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:18 smithi017 ceph-mon[27571]: pgmap v683: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 12 active+clean; 241 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 1/11568 objects degraded (0.009%); 826/11568 objects misplaced (7.140%); 5.5 MiB/s, 183 objects/s recovering 2024-09-17T16:17:19.013 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:18 smithi071 ceph-mon[30002]: pgmap v683: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 12 active+clean; 241 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 1/11568 objects degraded (0.009%); 826/11568 objects misplaced (7.140%); 5.5 MiB/s, 183 objects/s recovering 2024-09-17T16:17:19.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:19.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:19.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:19.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:18 smithi173 ceph-mon[30530]: pgmap v683: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 12 active+clean; 241 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 1/11568 objects degraded (0.009%); 826/11568 objects misplaced (7.140%); 5.5 MiB/s, 183 objects/s recovering 2024-09-17T16:17:19.146 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:17:19.146 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:142832 3.f 3:fbd92678:::benchmark_data_smithi173_66272_object70876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:19.123366+0000", 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.02094153, 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.020962690999999999, 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:19.147 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 142832 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.123366+0000", 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.123366+0000", 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:17:19.148 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.123368+0000", 2024-09-17T16:17:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.542e-06 2024-09-17T16:17:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:17:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.123374+0000", 2024-09-17T16:17:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.229e-06 2024-09-17T16:17:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:17:19.149 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.123388+0000", 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4365999999999999e-05 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.123397+0000", 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2390000000000002e-06 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.123444+0000", 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7586000000000001e-05 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:17:19.150 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.123534+0000", 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9876000000000005e-05 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,11", 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.123609+0000", 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5137999999999998e-05 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.127943+0000", 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.004333979 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:19.151 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.130969+0000", 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0030260600000000001 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25036.0:142845 3.9 e289/286 3:9a118fd6:::benchmark_data_smithi173_66272_object70889:head v 289'8762, mlcod=289'8760)", 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:19.141468+0000", 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0028396559999999999, 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029150769999999999, 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:17:19.152 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.141468+0000", 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.141468+0000", 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.141469+0000", 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.144e-06 2024-09-17T16:17:19.153 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.141477+0000", 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6469999999999998e-06 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.141498+0000", 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0852999999999998e-05 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.154 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.141507+0000", 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9209999999999998e-06 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.141556+0000", 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9045999999999998e-05 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:17:19.155 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:19.141570+0000", 2024-09-17T16:17:19.156 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4739000000000001e-05 2024-09-17T16:17:19.156 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:17:19.156 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:17:19.156 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:17:19.156 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:17:19.156 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:17:19.156 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 2 2024-09-17T16:17:19.156 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:17:19.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:19.192 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:19.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:19.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:19.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:19.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:19.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:19.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:19.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:19.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:19.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:19.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:19.702 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:19.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:19.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:19.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:19.856 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:19.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:19.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:19.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:19.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:19.985 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:17:19.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:20.013 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:19 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/11568 objects degraded (0.009%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:20.050 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:17:20.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:19 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/11568 objects degraded (0.009%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:20.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:20.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:20.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:20.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:19 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/11568 objects degraded (0.009%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:20.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:20.251 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:20.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:20.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:20.323 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:17:20.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:20.389 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:20.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:20.428 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:20.491 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:20.514 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:20.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:20.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:20.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:20.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:20.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:20.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:17:20.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:20.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:20.839 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:17:20.839 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:17:20.839 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:17:20.839 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:17:20.839 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:17:20.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:20.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:20.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:20.989 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:21.012 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:21.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:21.037 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:21.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:21.135 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:21.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:21.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:21.237 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:21.260 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:21.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:21.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:21.334 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 71953 objects 2024-09-17T16:17:21.334 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :95.1313 2024-09-17T16:17:21.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:21.375 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:17:21.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:21.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:21.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:21.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:21 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:17:21.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:21 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:17:21.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:21 smithi017 ceph-mon[27571]: pgmap v684: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 12 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 1/8034 objects degraded (0.012%); 826/8034 objects misplaced (10.281%); 4.7 MiB/s, 155 objects/s recovering 2024-09-17T16:17:21.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:21 smithi017 ceph-mon[27571]: osdmap e290: 12 total, 12 up, 12 in 2024-09-17T16:17:21.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1354860593' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:21.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:21.510 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:21.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:21.580 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:17:21.581 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:17:21.604 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:17:21.604 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:17:21.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:21.631 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:17:21.631 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:17:21.631 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T16:17:21.631 INFO:tasks.thrashosds.thrasher:Setting osd 11 primary_affinity to 0.000000 2024-09-17T16:17:21.631 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 11 0 2024-09-17T16:17:21.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:21.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:21.639 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:17:21.640 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:17:21.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:21 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:17:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:21 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:17:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:21 smithi173 ceph-mon[30530]: pgmap v684: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 12 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 1/8034 objects degraded (0.012%); 826/8034 objects misplaced (10.281%); 4.7 MiB/s, 155 objects/s recovering 2024-09-17T16:17:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:21 smithi173 ceph-mon[30530]: osdmap e290: 12 total, 12 up, 12 in 2024-09-17T16:17:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1354860593' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:21.668 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:17:21.669 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:17:21.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:17:21.693 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:17:21.716 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:17:21.716 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:17:21.727 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:21 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:17:21.727 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:21 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:17:21.727 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:21 smithi071 ceph-mon[30002]: pgmap v684: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_toofull, 12 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 1/8034 objects degraded (0.012%); 826/8034 objects misplaced (10.281%); 4.7 MiB/s, 155 objects/s recovering 2024-09-17T16:17:21.727 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:21 smithi071 ceph-mon[30002]: osdmap e290: 12 total, 12 up, 12 in 2024-09-17T16:17:21.727 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1354860593' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:21.736 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:21.739 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:17:21.739 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:17:21.758 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:17:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:21.760 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:21.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:21.761 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:17:21.762 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:17:21.784 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:17:21.784 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:17:21.807 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:17:21.808 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:17:21.835 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:17:21.835 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:17:21.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:21.867 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:17:21.867 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:17:21.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:21.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:21.985 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:22.000 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:17:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:22.008 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:22.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:22.109 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:22.132 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:22.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:22.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:22.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:22.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:22.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:22.381 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:22.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:22.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:22.422 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:17:22.422 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:17:22.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:22.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:22.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:22.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:22.607 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:22.630 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:22.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:22.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:22.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1354860593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:22.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:22 smithi017 ceph-mon[27571]: osdmap e291: 12 total, 12 up, 12 in 2024-09-17T16:17:22.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/262168023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1354860593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:22.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:22 smithi173 ceph-mon[30530]: osdmap e291: 12 total, 12 up, 12 in 2024-09-17T16:17:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/262168023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:22.724 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1354860593' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:22.724 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:22 smithi071 ceph-mon[30002]: osdmap e291: 12 total, 12 up, 12 in 2024-09-17T16:17:22.724 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/262168023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:22.732 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:22.755 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:22.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:22.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:22.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:22.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:22.981 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:23.003 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:23.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:23.059 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:17:23.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:23.125 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:23.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:23.227 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:23.250 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:23.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:23.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:23.376 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:23.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:23.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:23.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:23.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:23.601 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:23.625 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:23.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:23 smithi173 ceph-mon[30530]: pgmap v687: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 12 active+clean; 60 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 1/2859 objects degraded (0.035%); 196/2859 objects misplaced (6.856%); 4.1 MiB/s, 136 objects/s recovering 2024-09-17T16:17:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:23 smithi173 ceph-mon[30530]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:17:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:23 smithi173 ceph-mon[30530]: osdmap e292: 12 total, 12 up, 12 in 2024-09-17T16:17:23.703 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:23 smithi017 ceph-mon[27571]: pgmap v687: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 12 active+clean; 60 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 1/2859 objects degraded (0.035%); 196/2859 objects misplaced (6.856%); 4.1 MiB/s, 136 objects/s recovering 2024-09-17T16:17:23.703 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:23 smithi017 ceph-mon[27571]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:17:23.703 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:23 smithi017 ceph-mon[27571]: osdmap e292: 12 total, 12 up, 12 in 2024-09-17T16:17:23.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:23.748 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:23.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:23.786 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:23.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:23 smithi071 ceph-mon[30002]: pgmap v687: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 12 active+clean; 60 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 1/2859 objects degraded (0.035%); 196/2859 objects misplaced (6.856%); 4.1 MiB/s, 136 objects/s recovering 2024-09-17T16:17:23.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:23 smithi071 ceph-mon[30002]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:17:23.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:23 smithi071 ceph-mon[30002]: osdmap e292: 12 total, 12 up, 12 in 2024-09-17T16:17:23.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:23.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:23.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:23.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:23.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:23.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:24.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:24.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:24.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:24.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:24.248 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:24.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:24.302 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-17T16:17:24.302 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-17T16:17:24.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:24.375 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:24.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:24.450 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:24.476 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:24.500 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:24.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:24.581 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:24 smithi017 ceph-mon[27571]: osdmap e293: 12 total, 12 up, 12 in 2024-09-17T16:17:24.581 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:24 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:17:24.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:24 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:17:24.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:24 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:24.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:24 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/2859 objects degraded (0.035%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:17:24.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:24.625 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:24.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:24.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:17:24.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:24.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:24.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:24.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:24 smithi071 ceph-mon[30002]: osdmap e293: 12 total, 12 up, 12 in 2024-09-17T16:17:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:24 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:17:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:24 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:17:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:24 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:24 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/2859 objects degraded (0.035%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:17:24.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:24.874 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:24.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:24.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:24 smithi173 ceph-mon[30530]: osdmap e293: 12 total, 12 up, 12 in 2024-09-17T16:17:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:24 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:17:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:24 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:17:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:24 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:24 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/2859 objects degraded (0.035%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:17:24.975 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:24.996 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:17:24.996 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:17:24.997 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:24.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:25.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:25.121 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:25.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:25.223 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:25.246 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:25.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:25.348 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:25.371 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:25.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:25.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:25.398 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.11 primary-affinity to 0 (802) 2024-09-17T16:17:25.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:25.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:25.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:25.599 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:25.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:25 smithi173 ceph-mon[30530]: pgmap v690: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 12 active+clean; 21 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 587 op/s; 1/984 objects degraded (0.102%); 196/984 objects misplaced (19.919%); 1.9 MiB/s, 137 objects/s recovering 2024-09-17T16:17:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:25 smithi173 ceph-mon[30530]: osdmap e294: 12 total, 12 up, 12 in 2024-09-17T16:17:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3028036518' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 0.0} : dispatch 2024-09-17T16:17:25.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:25 smithi017 ceph-mon[27571]: pgmap v690: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 12 active+clean; 21 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 587 op/s; 1/984 objects degraded (0.102%); 196/984 objects misplaced (19.919%); 1.9 MiB/s, 137 objects/s recovering 2024-09-17T16:17:25.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:25 smithi017 ceph-mon[27571]: osdmap e294: 12 total, 12 up, 12 in 2024-09-17T16:17:25.716 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3028036518' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 0.0} : dispatch 2024-09-17T16:17:25.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:25.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:17:25.744 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:17:25.750 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:25.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:25 smithi071 ceph-mon[30002]: pgmap v690: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 12 active+clean; 21 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 587 op/s; 1/984 objects degraded (0.102%); 196/984 objects misplaced (19.919%); 1.9 MiB/s, 137 objects/s recovering 2024-09-17T16:17:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:25 smithi071 ceph-mon[30002]: osdmap e294: 12 total, 12 up, 12 in 2024-09-17T16:17:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3028036518' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 0.0} : dispatch 2024-09-17T16:17:25.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:25.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:25.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:25.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:25.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:25.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:25.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:25.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:26.100 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:26.124 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:26.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:26.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:26.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:26.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:26.249 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:26.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:26.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:26.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:26.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3028036518' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]': finished 2024-09-17T16:17:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:26 smithi017 ceph-mon[27571]: osdmap e295: 12 total, 12 up, 12 in 2024-09-17T16:17:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:26 smithi017 ceph-mon[27571]: pgmap v693: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 13 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 237 op/s; 1/6 objects degraded (16.667%); 0 B/s, 287 objects/s recovering 2024-09-17T16:17:26.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:26.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:26.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:26.604 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:26.624 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:26.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:26.648 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:26.649 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:26.652 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:26.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:26.750 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:26.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:26.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3028036518' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]': finished 2024-09-17T16:17:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:26 smithi071 ceph-mon[30002]: osdmap e295: 12 total, 12 up, 12 in 2024-09-17T16:17:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:26 smithi071 ceph-mon[30002]: pgmap v693: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 13 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 237 op/s; 1/6 objects degraded (16.667%); 0 B/s, 287 objects/s recovering 2024-09-17T16:17:26.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:26.876 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:26.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:26.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3028036518' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]': finished 2024-09-17T16:17:26.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:26 smithi173 ceph-mon[30530]: osdmap e295: 12 total, 12 up, 12 in 2024-09-17T16:17:26.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:26 smithi173 ceph-mon[30530]: pgmap v693: 17 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 13 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 237 op/s; 1/6 objects degraded (16.667%); 0 B/s, 287 objects/s recovering 2024-09-17T16:17:26.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:27.002 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:27.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:27.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:27.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:27.126 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:27.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:27.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:27.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:27.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:27.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:27.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:27.376 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:27.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:27.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:27.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:27.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:27.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:27.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:27.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:27 smithi017 ceph-mon[27571]: osdmap e296: 12 total, 12 up, 12 in 2024-09-17T16:17:27.622 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:17:27.622 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:17:27.626 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:27.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:27.646 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:17:27.646 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:17:27.668 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:17:27.668 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:17:27.690 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:17:27.690 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:17:27.715 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:17:27.715 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:17:27.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:27.742 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:17:27.743 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:17:27.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:27.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:27.765 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:17:27.765 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:17:27.788 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:17:27.788 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:17:27.813 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:17:27.814 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:17:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:27 smithi071 ceph-mon[30002]: osdmap e296: 12 total, 12 up, 12 in 2024-09-17T16:17:27.836 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:17:27.837 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:17:27.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:27.861 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:17:27.861 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:17:27.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:27.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:27.889 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:17:27.889 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:17:27.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:27 smithi173 ceph-mon[30530]: osdmap e296: 12 total, 12 up, 12 in 2024-09-17T16:17:27.977 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:28.003 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:28.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:28.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:28.105 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:28.132 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:28.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:28.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:28.234 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:28.263 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:28.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:28.363 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:28.386 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:28.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:28.424 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_1' removed 2024-09-17T16:17:28.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:28.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:28.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:28.611 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:28.634 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:28.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:28.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:28 smithi017 ceph-mon[27571]: 3.e continuing backfill to osd.11 from (0'0,290'9136] MIN to 290'9136 2024-09-17T16:17:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:28 smithi017 ceph-mon[27571]: osdmap e297: 12 total, 12 up, 12 in 2024-09-17T16:17:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1921847895' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:17:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1390986636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:28 smithi017 ceph-mon[27571]: pgmap v696: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 81 op/s; 1/6 objects degraded (16.667%); 0 B/s, 408 objects/s recovering 2024-09-17T16:17:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:28.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:28.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:28 smithi071 ceph-mon[30002]: 3.e continuing backfill to osd.11 from (0'0,290'9136] MIN to 290'9136 2024-09-17T16:17:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:28 smithi071 ceph-mon[30002]: osdmap e297: 12 total, 12 up, 12 in 2024-09-17T16:17:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1921847895' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:17:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1390986636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:28 smithi071 ceph-mon[30002]: pgmap v696: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 81 op/s; 1/6 objects degraded (16.667%); 0 B/s, 408 objects/s recovering 2024-09-17T16:17:28.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:28.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:28.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:28 smithi173 ceph-mon[30530]: 3.e continuing backfill to osd.11 from (0'0,290'9136] MIN to 290'9136 2024-09-17T16:17:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:28 smithi173 ceph-mon[30530]: osdmap e297: 12 total, 12 up, 12 in 2024-09-17T16:17:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1921847895' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:17:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1390986636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:28 smithi173 ceph-mon[30530]: pgmap v696: 17 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 14 active+clean; 577 KiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 81 op/s; 1/6 objects degraded (16.667%); 0 B/s, 408 objects/s recovering 2024-09-17T16:17:28.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:29.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:29.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:29.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:29.129 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:29.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:29.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:29.257 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:29.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:29.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:29.378 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:17:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:17:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:17:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:17:29.378 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:17:29.384 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:29.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:29.486 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:29.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:29.509 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:29.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:29.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:29.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:29.635 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:29.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:29 smithi017 ceph-mon[27571]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T16:17:29.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1921847895' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:17:29.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:29 smithi017 ceph-mon[27571]: osdmap e298: 12 total, 12 up, 12 in 2024-09-17T16:17:29.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:29 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:29.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:29 smithi017 ceph-mon[27571]: osdmap e299: 12 total, 12 up, 12 in 2024-09-17T16:17:29.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:29.760 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:29.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:29 smithi071 ceph-mon[30002]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T16:17:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1921847895' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:17:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:29 smithi071 ceph-mon[30002]: osdmap e298: 12 total, 12 up, 12 in 2024-09-17T16:17:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:29 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:29 smithi071 ceph-mon[30002]: osdmap e299: 12 total, 12 up, 12 in 2024-09-17T16:17:29.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:29.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:29.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:29 smithi173 ceph-mon[30530]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T16:17:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1921847895' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:17:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:29 smithi173 ceph-mon[30530]: osdmap e298: 12 total, 12 up, 12 in 2024-09-17T16:17:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:29 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:17:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:29 smithi173 ceph-mon[30530]: osdmap e299: 12 total, 12 up, 12 in 2024-09-17T16:17:29.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:30.011 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:30.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:30.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:30.136 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:30.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:30.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:30.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:30.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:30.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:30.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:30.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:30.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:30.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:30.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:30.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:30.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:30.644 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:30.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:30.734 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:30 smithi017 ceph-mon[27571]: pgmap v699: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:30.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:30.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:17:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:30.772 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:30.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:30.787 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:30 smithi071 ceph-mon[30002]: pgmap v699: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:30.873 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:30.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:30 smithi173 ceph-mon[30530]: pgmap v699: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:30.895 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:30.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:30.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:31.018 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:31.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:31.120 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:31.141 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:31.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:31.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:31.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:31.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:31.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:31.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:31.397 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:31.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:31.474 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T16:17:31.475 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T16:17:31.475 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-17T16:17:31.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-17T16:17:31.498 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:31.525 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:31.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:31.531 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:17:31.531 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:17:31.531 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-17T16:17:31.531 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T16:17:31.531 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T16:17:31.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-17T16:17:31.598 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:31 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded) 2024-09-17T16:17:31.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:31.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:31.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:31.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:31.754 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:31.762 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:31.777 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:31.783 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:31.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:31 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded) 2024-09-17T16:17:31.817 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:17:31.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:31.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:31 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded) 2024-09-17T16:17:31.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:31.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:32.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:32.034 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:32.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:32.038 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:17:32.135 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:32.158 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:32.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:32.259 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:32.284 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:32.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:32.286 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:17:32.286 INFO:teuthology.orchestra.run.smithi071.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T16:17:32.286 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:17:32.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:32.407 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:32.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:32.437 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:17:32.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:32.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:32.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:32 smithi071 ceph-mon[30002]: pgmap v700: 1 pgs: 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2598662888' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:17:32.636 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:32.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:32.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:32 smithi017 ceph-mon[27571]: pgmap v700: 1 pgs: 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:32.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2598662888' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:17:32.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:32.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:32.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:32.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:32.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:32 smithi173 ceph-mon[30530]: pgmap v700: 1 pgs: 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:32.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2598662888' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:17:32.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:32.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:32.927 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:32.952 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:17:32.952 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:17:32.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:17:32.952 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:17:32.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25217 2.19 e121/114 2:9a6d6ce1:::benchmark_data_smithi173_55121_object25216:head v 121'1238, mlcod=121'1235)", 2024-09-17T16:17:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.777156+0000", 2024-09-17T16:17:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 362.170855421, 2024-09-17T16:17:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25947270300000003, 2024-09-17T16:17:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777156+0000", 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777156+0000", 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777157+0000", 2024-09-17T16:17:32.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7300000000000005e-07 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777182+0000", 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5307e-05 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035767+0000", 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25858533500000003 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035781+0000", 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3567000000000001e-05 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035903+0000", 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012237899999999999 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:32.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.035938+0000", 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4215999999999999e-05 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.036602+0000", 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00066465100000000004 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.036629+0000", 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6475e-05 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.957 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25228 2.1 e121/104 2:8352dd7d:::benchmark_data_smithi173_55121_object25227:head v 121'1242, mlcod=121'1239)", 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.786569+0000", 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 362.161442153, 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27958702099999999, 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786569+0000", 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786569+0000", 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786570+0000", 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1399999999999997e-07 2024-09-17T16:17:32.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786589+0000", 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8754999999999999e-05 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065420+0000", 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.278831679 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065429+0000", 2024-09-17T16:17:32.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4149999999999999e-06 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065499+0000", 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9944000000000007e-05 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.065517+0000", 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8499e-05 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066120+0000", 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060275699999999999 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066156+0000", 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6358e-05 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:29460 2.1 e128/104 2:83aee601:::benchmark_data_smithi173_55121_object29459:head v 128'1369, mlcod=128'1366)", 2024-09-17T16:17:32.962 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.198210+0000", 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 343.74980111999997, 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31777874900000003, 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198210+0000", 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:32.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198210+0000", 2024-09-17T16:17:32.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:32.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198211+0000", 2024-09-17T16:17:32.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-17T16:17:32.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198235+0000", 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3884999999999999e-05 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515223+0000", 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31698817499999998 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515231+0000", 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9189999999999999e-06 2024-09-17T16:17:32.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.966 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:32.966 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515321+0000", 2024-09-17T16:17:32.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9240000000000006e-05 2024-09-17T16:17:32.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515344+0000", 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3362000000000001e-05 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515954+0000", 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000610291 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.515989+0000", 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4758000000000002e-05 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.968 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:32.969 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.969 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:29625 2.5 e129/104 2:a21e4127:::benchmark_data_smithi173_55121_object29624:head v 129'1363, mlcod=129'1360)", 2024-09-17T16:17:32.969 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.564207+0000", 2024-09-17T16:17:32.969 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 343.38380435900001, 2024-09-17T16:17:32.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29265457499999997, 2024-09-17T16:17:32.970 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564207+0000", 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564207+0000", 2024-09-17T16:17:32.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:32.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564208+0000", 2024-09-17T16:17:32.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2399999999999998e-07 2024-09-17T16:17:32.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:32.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.564411+0000", 2024-09-17T16:17:32.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020328800000000001 2024-09-17T16:17:32.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:32.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856209+0000", 2024-09-17T16:17:32.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29179771999999998 2024-09-17T16:17:32.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:32.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856213+0000", 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0579999999999999e-06 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856281+0000", 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7911999999999996e-05 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856295+0000", 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4559000000000001e-05 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856830+0000", 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00053516399999999997 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.856862+0000", 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1350000000000003e-05 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.975 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:88644 2.8 2:10ace813:::benchmark_data_smithi173_55121_object12155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:43.950796+0000", 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.997215767, 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31997915599999999, 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 88644 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:32.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950796+0000", 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950796+0000", 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950797+0000", 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4079999999999999e-06 2024-09-17T16:17:32.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950804+0000", 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2490000000000004e-06 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950817+0000", 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.234e-05 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:32.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950823+0000", 2024-09-17T16:17:32.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2199999999999997e-06 2024-09-17T16:17:32.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:32.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.950884+0000", 2024-09-17T16:17:32.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0770000000000003e-05 2024-09-17T16:17:32.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.951019+0000", 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134907 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.951145+0000", 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126663 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.952746+0000", 2024-09-17T16:17:32.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016012859999999999 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.953782+0000", 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010357210000000001 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270185+0000", 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31640305000000002 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270223+0000", 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7607000000000001e-05 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270775+0000", 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00055193500000000003 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:88644 2.8 e157/155 ondisk, result = 0)", 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:43.952810+0000", 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 288.995201495, 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31796888800000001, 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.952810+0000", 2024-09-17T16:17:32.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.952810+0000", 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.952811+0000", 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.85e-07 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:32.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:43.952815+0000", 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0740000000000003e-06 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270131+0000", 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31731594699999999 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270135+0000", 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.374e-06 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270176+0000", 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0914000000000002e-05 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270182+0000", 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6659999999999996e-06 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:44.270779+0000", 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00059712800000000005 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.986 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:32.987 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.987 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:90287 2.8 2:13eb058f:::benchmark_data_smithi173_55121_object13798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:17:32.987 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:45.941984+0000", 2024-09-17T16:17:32.987 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 287.006027668, 2024-09-17T16:17:32.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25228158000000001, 2024-09-17T16:17:32.987 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90287 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.941984+0000", 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:32.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.941984+0000", 2024-09-17T16:17:32.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:32.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.941987+0000", 2024-09-17T16:17:32.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4230000000000001e-06 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942001+0000", 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3647e-05 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942041+0000", 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9731999999999998e-05 2024-09-17T16:17:32.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942064+0000", 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3604e-05 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942178+0000", 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113508 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942270+0000", 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2823000000000002e-05 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.942366+0000", 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5445999999999994e-05 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:32.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.943598+0000", 2024-09-17T16:17:32.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001232089 2024-09-17T16:17:32.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:32.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:45.944062+0000", 2024-09-17T16:17:32.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046364399999999998 2024-09-17T16:17:32.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.193517+0000", 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.249455344 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.193584+0000", 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7328000000000003e-05 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:46.194265+0000", 2024-09-17T16:17:32.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00068099199999999995 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:94044 2.8 2:165f9939:::benchmark_data_smithi173_55121_object17555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:49.924944+0000", 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 283.02306768400001, 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35947178600000002, 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:32.995 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 94044 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924944+0000", 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924944+0000", 2024-09-17T16:17:32.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924946+0000", 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.21e-06 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924966+0000", 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9626000000000001e-05 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:32.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924978+0000", 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.29e-05 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.924987+0000", 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2930000000000001e-06 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.925037+0000", 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0133999999999998e-05 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.925123+0000", 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6194999999999997e-05 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.925201+0000", 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8362999999999995e-05 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.927427+0000", 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022257879999999998 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:32.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.927457+0000", 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9961999999999999e-05 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.283840+0000", 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35638287000000002 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.283883+0000", 2024-09-17T16:17:33.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2812999999999998e-05 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.284416+0000", 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00053263200000000001 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:104874 2.8 2:13fbf047:::benchmark_data_smithi173_55121_object28385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:01.984418+0000", 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 270.96359313900001, 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.254036761, 2024-09-17T16:17:33.001 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 104874 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984418+0000", 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984418+0000", 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984419+0000", 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7900000000000007e-07 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984427+0000", 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2029999999999992e-06 2024-09-17T16:17:33.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984439+0000", 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1938e-05 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.984446+0000", 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8979999999999999e-06 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.985736+0000", 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001289622 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.985889+0000", 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015309500000000001 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.986033+0000", 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014426799999999999 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.987137+0000", 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011038039999999999 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:01.991811+0000", 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046738090000000001 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.237979+0000", 2024-09-17T16:17:33.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.246168528 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.238037+0000", 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7225000000000002e-05 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.238455+0000", 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00041839200000000002 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:152973 2.19 e199/114 2:9e1f786b:::benchmark_data_smithi173_55121_object76484:head v 199'5124, mlcod=199'5121)", 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.867324+0000", 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 219.08068787299999, 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59753164299999995, 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867324+0000", 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867324+0000", 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867325+0000", 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.78e-07 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867331+0000", 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6239999999999996e-06 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462706+0000", 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59537525499999999 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462713+0000", 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4019999999999999e-06 2024-09-17T16:17:33.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462789+0000", 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6006000000000004e-05 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462816+0000", 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7087e-05 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.464821+0000", 2024-09-17T16:17:33.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002004731 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.464855+0000", 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4560000000000001e-05 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:8110 3.b e220/208 3:de8160e2:::benchmark_data_smithi173_66272_object8109:head v 220'485, mlcod=220'481)", 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:14:34.926435+0000", 2024-09-17T16:17:33.012 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 178.02157666400001, 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.293198399, 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926435+0000", 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926435+0000", 2024-09-17T16:17:33.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926436+0000", 2024-09-17T16:17:33.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.049e-06 2024-09-17T16:17:33.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926456+0000", 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0636e-05 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219071+0000", 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29261461100000002 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219077+0000", 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9880000000000001e-06 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219126+0000", 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8807000000000002e-05 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219142+0000", 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5846e-05 2024-09-17T16:17:33.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219593+0000", 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045157699999999999 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219633+0000", 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9885000000000002e-05 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124293 3.e 3:74c76eb5:::benchmark_data_smithi173_66272_object52337:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.576096401999997, 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.55229465499999997, 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124293 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.018 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371917+0000", 2024-09-17T16:17:33.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1430000000000001e-06 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371923+0000", 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1709999999999999e-06 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371936+0000", 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2947e-05 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371951+0000", 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4547e-05 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.372152+0000", 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020128999999999999 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.372168+0000", 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6218999999999998e-05 2024-09-17T16:17:33.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.916603+0000", 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54443496499999999 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.919612+0000", 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030085120000000001 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.919736+0000", 2024-09-17T16:17:33.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012435899999999999 2024-09-17T16:17:33.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.923884+0000", 2024-09-17T16:17:33.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041481460000000001 2024-09-17T16:17:33.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.923928+0000", 2024-09-17T16:17:33.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4159999999999997e-05 2024-09-17T16:17:33.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924005+0000", 2024-09-17T16:17:33.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6178000000000001e-05 2024-09-17T16:17:33.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924024+0000", 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9318e-05 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924094+0000", 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9605999999999997e-05 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924210+0000", 2024-09-17T16:17:33.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116094 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:132741 3.e 3:729d5b9e:::benchmark_data_smithi173_66272_object60785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 29.316087699000001, 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43202703199999998, 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:33.026 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 132741 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:17:33.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631925+0000", 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3239999999999999e-06 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631930+0000", 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6700000000000002e-06 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631942+0000", 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1857e-05 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631947+0000", 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8980000000000001e-06 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631998+0000", 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0365000000000003e-05 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.632091+0000", 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3592999999999997e-05 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.632178+0000", 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6648000000000001e-05 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635422+0000", 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003244289 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.636256+0000", 2024-09-17T16:17:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083370299999999998 2024-09-17T16:17:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.636486+0000", 2024-09-17T16:17:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022978699999999999 2024-09-17T16:17:33.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063733+0000", 2024-09-17T16:17:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42724667900000002 2024-09-17T16:17:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:33.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063826+0000", 2024-09-17T16:17:33.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3579999999999995e-05 2024-09-17T16:17:33.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063951+0000", 2024-09-17T16:17:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012463900000000001 2024-09-17T16:17:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.035 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25036.0:132741 3.e e278/273 ondisk, result = 0)", 2024-09-17T16:17:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:17:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 29.312076329, 2024-09-17T16:17:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42802421600000001, 2024-09-17T16:17:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:17:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:17:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635936+0000", 2024-09-17T16:17:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.023e-06 2024-09-17T16:17:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635941+0000", 2024-09-17T16:17:33.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5000000000000001e-06 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.060074+0000", 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42413389000000001 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.060083+0000", 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6670000000000002e-06 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063717+0000", 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036338640000000001 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063726+0000", 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.789e-06 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063959+0000", 2024-09-17T16:17:33.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023348299999999999 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:134243 3.1 e280/272 3:8c1e41ff:::benchmark_data_smithi173_66272_object62287:head v 280'8469, mlcod=247'4427)", 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:05.911868+0000", 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.036143591999998, 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26671571799999999, 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.041 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.042 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:33.042 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911868+0000", 2024-09-17T16:17:33.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911868+0000", 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911869+0000", 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911874+0000", 2024-09-17T16:17:33.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4380000000000003e-06 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169900+0000", 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25802566399999999 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169906+0000", 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4180000000000002e-06 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169967+0000", 2024-09-17T16:17:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1247000000000005e-05 2024-09-17T16:17:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169991+0000", 2024-09-17T16:17:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4151999999999998e-05 2024-09-17T16:17:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.178494+0000", 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0085021780000000009 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.178584+0000", 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9865999999999996e-05 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.046 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:135118 3.e 3:7ce4ae2e:::benchmark_data_smithi173_66272_object63162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-17T16:17:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:17:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 25.677602965999998, 2024-09-17T16:17:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34112115999999998, 2024-09-17T16:17:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:33.047 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 135118 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270411+0000", 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3520000000000001e-06 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270420+0000", 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270443+0000", 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2789000000000002e-05 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270453+0000", 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0322e-05 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270692+0000", 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023867100000000001 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270833+0000", 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014147500000000001 2024-09-17T16:17:33.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.271004+0000", 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017102299999999999 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279153+0000", 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081486939999999997 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279218+0000", 2024-09-17T16:17:33.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5320999999999993e-05 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279235+0000", 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6402999999999999e-05 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611337+0000", 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33210225300000001 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611397+0000", 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9757999999999999e-05 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611530+0000", 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013292299999999999 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.053 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:136353 3.e 3:75c6ba64:::benchmark_data_smithi173_66272_object64397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.999218753000001, 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34635525499999997, 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 136353 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948795+0000", 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4269999999999998e-06 2024-09-17T16:17:33.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948804+0000", 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948823+0000", 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8892000000000001e-05 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948832+0000", 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0629999999999995e-06 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948903+0000", 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0796000000000005e-05 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.949058+0000", 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015499099999999999 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.949219+0000", 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016127900000000001 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953488+0000", 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042689440000000002 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953702+0000", 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021406499999999999 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953721+0000", 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8936999999999998e-05 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294938+0000", 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.341217465 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295013+0000", 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4389999999999998e-05 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295148+0000", 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000135148 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25036.0:136353 3.e e284/273 ondisk, result = 0)", 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.995546434000001, 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.342690457, 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952466+0000", 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0389999999999999e-06 2024-09-17T16:17:33.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952472+0000", 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6910000000000002e-06 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294851+0000", 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34237905899999999 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294856+0000", 2024-09-17T16:17:33.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3499999999999996e-06 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294916+0000", 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9580000000000002e-05 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294928+0000", 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2571999999999999e-05 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295155+0000", 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000227166 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:143899 3.1 e290/272 3:8d15b72a:::benchmark_data_smithi173_66272_object71943:head v 290'9096, mlcod=261'5689)", 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:17:33.065 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 12.468891853000001, 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37496784399999999, 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9699999999999996e-07 2024-09-17T16:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479128+0000", 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850526+0000", 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37139821699999997 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850534+0000", 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1760000000000004e-06 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850583+0000", 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9013e-05 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850601+0000", 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8916e-05 2024-09-17T16:17:33.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.854050+0000", 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034480639999999998 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.854087+0000", 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7846000000000003e-05 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.070 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:143908 3.e 3:70cae513:::benchmark_data_smithi173_66272_object71952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 12.458840690000001, 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35428895300000002, 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:33.071 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 143908 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:33.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:17:33.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:33.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:33.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489172+0000", 2024-09-17T16:17:33.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6700000000000001e-06 2024-09-17T16:17:33.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:33.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489177+0000", 2024-09-17T16:17:33.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.651e-06 2024-09-17T16:17:33.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:33.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489191+0000", 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3497e-05 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489197+0000", 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0719999999999996e-06 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489244+0000", 2024-09-17T16:17:33.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7037999999999999e-05 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489341+0000", 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7664000000000005e-05 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489424+0000", 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2916000000000004e-05 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493180+0000", 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037554379999999998 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493210+0000", 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0076000000000001e-05 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.494016+0000", 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080592900000000004 2024-09-17T16:17:33.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843310+0000", 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34929427400000002 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843362+0000", 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2549000000000001e-05 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843460+0000", 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7178999999999999e-05 2024-09-17T16:17:33.078 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.079 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:33.079 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:33.079 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:17:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:33.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:33.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:33.182 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:33.203 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:33.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:33.305 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:33.326 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:33.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:33.428 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:33.439 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_2' created 2024-09-17T16:17:33.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:33 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:33.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2598662888' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:17:33.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:33 smithi017 ceph-mon[27571]: osdmap e300: 12 total, 12 up, 12 in 2024-09-17T16:17:33.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/99493990' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-17T16:17:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:33.455 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:33.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:33.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:33.577 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:33.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:33.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:33.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:33.702 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:33.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:33.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:33.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:33 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:33.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2598662888' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:17:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:33 smithi071 ceph-mon[30002]: osdmap e300: 12 total, 12 up, 12 in 2024-09-17T16:17:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/99493990' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-17T16:17:33.825 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:33.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:33 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2598662888' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:17:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:33 smithi173 ceph-mon[30530]: osdmap e300: 12 total, 12 up, 12 in 2024-09-17T16:17:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/99493990' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-17T16:17:33.927 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:33.950 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:33.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:34.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:34.075 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:34.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:34.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:34.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:34.198 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:34.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:34.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:34.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:34.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:34.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:34.453 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:34.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:34.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:34.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:34.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:34.684 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:34.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/99493990' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-17T16:17:34.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:34 smithi017 ceph-mon[27571]: osdmap e301: 12 total, 12 up, 12 in 2024-09-17T16:17:34.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:34 smithi017 ceph-mon[27571]: pgmap v703: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:34.706 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:34.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:34.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:34.808 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:34.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/99493990' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-17T16:17:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:34 smithi071 ceph-mon[30002]: osdmap e301: 12 total, 12 up, 12 in 2024-09-17T16:17:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:34 smithi071 ceph-mon[30002]: pgmap v703: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:34.830 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:34.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:34.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/99493990' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-17T16:17:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:34 smithi173 ceph-mon[30530]: osdmap e301: 12 total, 12 up, 12 in 2024-09-17T16:17:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:34 smithi173 ceph-mon[30530]: pgmap v703: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:34.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:34.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:34.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:34.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:35.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-17T16:17:35.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:35.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:35.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:35.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:35.145 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:17:35.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:35.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:35.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:35.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:35.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:35.341 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:35.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:35.360 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:17:35.360 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:17:35.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:17:35.383 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:17:35.403 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:17:35.406 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:17:35.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:17:35.424 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:17:35.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:35.445 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:17:35.446 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:17:35.467 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:17:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:17:35.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:35.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:35.489 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:17:35.489 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:17:35.511 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:17:35.511 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:17:35.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:35 smithi017 ceph-mon[27571]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:17:35.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:35 smithi017 ceph-mon[27571]: osdmap e302: 12 total, 12 up, 12 in 2024-09-17T16:17:35.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3658047036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:35.533 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:17:35.533 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:17:35.555 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:17:35.555 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:17:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:35 smithi071 ceph-mon[30002]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:17:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:35 smithi071 ceph-mon[30002]: osdmap e302: 12 total, 12 up, 12 in 2024-09-17T16:17:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3658047036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:35.570 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:35.577 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:17:35.577 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:17:35.594 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:35.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:35.599 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:17:35.599 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:17:35.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:35.716 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:35.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:35.818 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:35.841 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:35.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:35.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:35 smithi173 ceph-mon[30530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:17:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:35 smithi173 ceph-mon[30530]: osdmap e302: 12 total, 12 up, 12 in 2024-09-17T16:17:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3658047036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:35.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:35.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:35.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:36.071 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:36.093 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:36.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:36.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:36.217 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:36.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:36.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:36.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:36.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:36.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:36.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:36.471 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:36.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:36.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:36.597 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:36.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:36.699 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:36.721 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:36.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:36.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:36 smithi017 ceph-mon[27571]: osdmap e303: 12 total, 12 up, 12 in 2024-09-17T16:17:36.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:36 smithi017 ceph-mon[27571]: pgmap v706: 17 pgs: 10 unknown, 7 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:36.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:36 smithi071 ceph-mon[30002]: osdmap e303: 12 total, 12 up, 12 in 2024-09-17T16:17:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:36 smithi071 ceph-mon[30002]: pgmap v706: 17 pgs: 10 unknown, 7 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:36.817 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:36.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:36.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:36.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:36.854 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:36 smithi173 ceph-mon[30530]: osdmap e303: 12 total, 12 up, 12 in 2024-09-17T16:17:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:36 smithi173 ceph-mon[30530]: pgmap v706: 17 pgs: 10 unknown, 7 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:36.950 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:36.977 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:36.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:37.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:37.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:37.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:37.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:37.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:37.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:37.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:37.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:37.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:37.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:37.485 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:37.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:37.513 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:17:37.514 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:17:37.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:37.612 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:37.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:37.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:37.742 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:37.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:37.843 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:37.866 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:37.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:37.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:37.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:37.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:37.995 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:38.080 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:17:38.080 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:17:38.081 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-17T16:17:38.081 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd out 1 2024-09-17T16:17:38.081 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:17:38.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:38.149 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:38.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:38.251 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:38.277 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:38.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:38.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:17:38.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:38.406 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:38.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:38.507 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:38.531 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:38.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:38.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:38.649 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:38.654 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:38.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:38.659 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:38.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:38.782 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:38.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:38.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:17:38.870 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-17T16:17:38.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:38.910 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:38.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:38.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:39.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:39.037 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:39.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:39.044 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:38 smithi017 ceph-mon[27571]: pgmap v707: 17 pgs: 2 unknown, 15 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:39.044 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3640095806' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:17:39.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:39.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:38 smithi173 ceph-mon[30530]: pgmap v707: 17 pgs: 2 unknown, 15 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:39.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3640095806' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:17:39.164 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:39.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:39.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:39.288 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:39.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:38 smithi071 ceph-mon[30002]: pgmap v707: 17 pgs: 2 unknown, 15 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3640095806' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:17:39.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:39.416 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:39.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:39.519 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:39.544 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:39.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:39.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:39.672 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:39.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:39.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:39.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:39.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:39.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:39.922 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:39.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3640095806' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:17:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:39 smithi017 ceph-mon[27571]: osdmap e304: 12 total, 12 up, 12 in 2024-09-17T16:17:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:39 smithi017 ceph-mon[27571]: osdmap e305: 12 total, 12 up, 12 in 2024-09-17T16:17:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1893526191' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:39 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:39 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:17:39.881+0000 7f086a07d640 -1 mon.a@0(leader).osd e305 definitely_dead 0 2024-09-17T16:17:40.024 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:40.049 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:40.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:40.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3640095806' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:17:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:39 smithi173 ceph-mon[30530]: osdmap e304: 12 total, 12 up, 12 in 2024-09-17T16:17:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:39 smithi173 ceph-mon[30530]: osdmap e305: 12 total, 12 up, 12 in 2024-09-17T16:17:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1893526191' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:39 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:40.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:40.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:40.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:40.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:40.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:40.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:40.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:40.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3640095806' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:17:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:39 smithi071 ceph-mon[30002]: osdmap e304: 12 total, 12 up, 12 in 2024-09-17T16:17:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:39 smithi071 ceph-mon[30002]: osdmap e305: 12 total, 12 up, 12 in 2024-09-17T16:17:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1893526191' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:39 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:40.372 INFO:teuthology.orchestra.run.smithi017.stderr:marked out osd.1. 2024-09-17T16:17:40.380 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:17:40.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:40.422 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:40.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:40.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:40.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:40.547 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:40.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:40.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:40.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:40.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:40.774 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:40.799 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:40.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:40.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:40.923 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:40.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:41.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:41.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:41.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:41.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:40 smithi017 ceph-mon[27571]: pgmap v710: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:41.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1223141231' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-17T16:17:41.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:40 smithi017 ceph-mon[27571]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-17T16:17:41.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:40 smithi017 ceph-mon[27571]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:17:41.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:40 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:17:41.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:40 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:41.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1223141231' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-17T16:17:41.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:40 smithi017 ceph-mon[27571]: osdmap e306: 12 total, 12 up, 11 in 2024-09-17T16:17:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:40 smithi173 ceph-mon[30530]: pgmap v710: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1223141231' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-17T16:17:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:40 smithi173 ceph-mon[30530]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-17T16:17:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:40 smithi173 ceph-mon[30530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:17:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:40 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:17:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:40 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1223141231' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-17T16:17:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:40 smithi173 ceph-mon[30530]: osdmap e306: 12 total, 12 up, 11 in 2024-09-17T16:17:41.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:41.173 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:41.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:41.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:41.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:41.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:40 smithi071 ceph-mon[30002]: pgmap v710: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:17:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1223141231' entity='client.admin' cmd={"prefix": "osd out", "ids": ["1"]} : dispatch 2024-09-17T16:17:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:40 smithi071 ceph-mon[30002]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-17T16:17:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:40 smithi071 ceph-mon[30002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:17:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:40 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:17:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:40 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:41.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1223141231' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-17T16:17:41.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:40 smithi071 ceph-mon[30002]: osdmap e306: 12 total, 12 up, 11 in 2024-09-17T16:17:41.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:41.429 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:41.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:41.456 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-17T16:17:41.459 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T16:17:41.459 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T16:17:41.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-09-17T16:17:41.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T16:17:41.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_77474 2024-09-17T16:17:41.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:17:41.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-09-17T16:17:41.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:41.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:41.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:41.663 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:41.695 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:41.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:41.796 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:41.825 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:41.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:41.927 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:41.955 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:41.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:42.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:42.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:42.088 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:42.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:42.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:42.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:42.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:42.339 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:42.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:42.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:42.463 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:42.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:42.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 15 2505 2490 155.617 155.625 0.0127701 0.00638949 2024-09-17T16:17:42.565 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:42.591 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:42.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:42.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:42 smithi173 ceph-mon[30530]: osdmap e307: 12 total, 12 up, 11 in 2024-09-17T16:17:42.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:42.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:42.720 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:42.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:42 smithi017 ceph-mon[27571]: osdmap e307: 12 total, 12 up, 11 in 2024-09-17T16:17:42.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:42 smithi071 ceph-mon[30002]: osdmap e307: 12 total, 12 up, 11 in 2024-09-17T16:17:42.821 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:42.843 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:42.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:42.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:42.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:42.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:43.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:43.102 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:43.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:43.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:43.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:17:43.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:43.226 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:43.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:43.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:43.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:43.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:43.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:43.483 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:43.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:43.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:43 smithi017 ceph-mon[27571]: pgmap v713: 17 pgs: 4 peering, 13 active+clean; 3.9 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 848 KiB/s wr, 13 op/s 2024-09-17T16:17:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:43 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:17:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:43 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:17:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:43 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:17:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:43 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:17:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:43 smithi017 ceph-mon[27571]: osdmap e308: 12 total, 12 up, 11 in 2024-09-17T16:17:43.522 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 4320 4304 134.491 113.375 0.0052286 0.00741101 2024-09-17T16:17:43.585 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:43.611 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:43.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:43 smithi173 ceph-mon[30530]: pgmap v713: 17 pgs: 4 peering, 13 active+clean; 3.9 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 848 KiB/s wr, 13 op/s 2024-09-17T16:17:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:43 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:17:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:43 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:17:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:43 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:17:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:43 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:17:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:43 smithi173 ceph-mon[30530]: osdmap e308: 12 total, 12 up, 11 in 2024-09-17T16:17:43.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:43.739 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:43.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:43 smithi071 ceph-mon[30002]: pgmap v713: 17 pgs: 4 peering, 13 active+clean; 3.9 MiB data, 1.8 GiB used, 982 GiB / 983 GiB avail; 848 KiB/s wr, 13 op/s 2024-09-17T16:17:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:43 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:17:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:43 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:17:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:43 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:17:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:43 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:17:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:43 smithi071 ceph-mon[30002]: osdmap e308: 12 total, 12 up, 11 in 2024-09-17T16:17:43.841 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:43.862 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:43.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:43.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:43.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:43.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:44.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:44.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:44.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:44.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:44.236 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:44.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:44.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:44.361 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:44.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:44.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:44.486 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:44.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:44.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 5784 5768 120.158 91.5 0.00433638 0.00829536 2024-09-17T16:17:44.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:44.618 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:44.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:44.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:44.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:44.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:44.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:44.848 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:44.875 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:44.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:44.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:45.005 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:45.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:45.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:45.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:45.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:45.138 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:45.192 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:45.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:45.268 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:45.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:45.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:45.396 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:45.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:45.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:45.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:45.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:45.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 7927 7911 123.6 133.938 0.139351 0.00782078 2024-09-17T16:17:45.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:45 smithi173 ceph-mon[30530]: pgmap v715: 17 pgs: 4 peering, 13 active+clean; 36 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 8.0 MiB/s wr, 128 op/s 2024-09-17T16:17:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:45 smithi173 ceph-mon[30530]: osdmap e309: 12 total, 12 up, 11 in 2024-09-17T16:17:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:45.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:45.675 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:45 smithi017 ceph-mon[27571]: pgmap v715: 17 pgs: 4 peering, 13 active+clean; 36 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 8.0 MiB/s wr, 128 op/s 2024-09-17T16:17:45.676 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:45 smithi017 ceph-mon[27571]: osdmap e309: 12 total, 12 up, 11 in 2024-09-17T16:17:45.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:45.779 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:45.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:45 smithi071 ceph-mon[30002]: pgmap v715: 17 pgs: 4 peering, 13 active+clean; 36 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 8.0 MiB/s wr, 128 op/s 2024-09-17T16:17:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:45 smithi071 ceph-mon[30002]: osdmap e309: 12 total, 12 up, 11 in 2024-09-17T16:17:45.881 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:45.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:45.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:45.956 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:17:45.956 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:17:46.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:46.039 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:46.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:46.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:46.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:17:46.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:17:46.169 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:46.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:46.192 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:17:46.192 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:17:46.220 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:17:46.220 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:17:46.244 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:17:46.244 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:17:46.271 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:46.273 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:17:46.273 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:17:46.298 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:46.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:46.301 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:17:46.302 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:17:46.329 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:17:46.330 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:17:46.355 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:17:46.355 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:17:46.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:17:46.383 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:17:46.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:46.406 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:17:46.406 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:17:46.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:46.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:46.431 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:17:46.432 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:17:46.455 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:17:46.456 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:17:46.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3956852626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:46.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 9984 9968 124.59 128.562 0.00507012 0.00797461 2024-09-17T16:17:46.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:46.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:46.546 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3956852626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:46.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:46.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:46.671 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:46.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:46.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:46.796 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:46.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3956852626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:46.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:46.929 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:46.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:46.944 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:17:47.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:47.055 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:47.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:47.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:47.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:47.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:47.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:47.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:47.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:47.415 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:47.443 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:47.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:47.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 11156 11140 116.033 73.25 0.0536355 0.00857861 2024-09-17T16:17:47.545 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:47.569 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:47.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:47 smithi173 ceph-mon[30530]: pgmap v717: 17 pgs: 2 peering, 15 active+clean; 252 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 46 MiB/s wr, 739 op/s 2024-09-17T16:17:47.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:47.692 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:47.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:47 smithi017 ceph-mon[27571]: pgmap v717: 17 pgs: 2 peering, 15 active+clean; 252 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 46 MiB/s wr, 739 op/s 2024-09-17T16:17:47.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:47.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:47 smithi071 ceph-mon[30002]: pgmap v717: 17 pgs: 2 peering, 15 active+clean; 252 MiB data, 2.6 GiB used, 981 GiB / 983 GiB avail; 46 MiB/s wr, 739 op/s 2024-09-17T16:17:47.822 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:47.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:47.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:47.832 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:47.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:47.951 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:47.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:48.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:48.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:48.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:48.182 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:48.206 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:48.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:48.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:48.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:48.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:48.428 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:48.436 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:48.473 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:48.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:48.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:48.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 15 13298 13283 118.589 133.938 0.00345655 0.00842363 2024-09-17T16:17:48.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:48.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:48.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:48.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:48.702 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:48.732 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:48.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:48.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:48.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:48.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:48.964 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:48.969 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:17:48.969 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:17:48.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:17:48.969 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:17:48.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.969 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:13527 2.19 2:9902b078:::benchmark_data_smithi173_55121_object13526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:17:48.969 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.335470+0000", 2024-09-17T16:17:48.969 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 396.62967425900001, 2024-09-17T16:17:48.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0334308889999999, 2024-09-17T16:17:48.969 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 13527 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335470+0000", 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:48.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335470+0000", 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335471+0000", 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.89e-07 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335500+0000", 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8971000000000001e-05 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335507+0000", 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.401e-06 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:48.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335513+0000", 2024-09-17T16:17:48.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.366e-06 2024-09-17T16:17:48.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:48.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.362278+0000", 2024-09-17T16:17:48.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.026764929999999999 2024-09-17T16:17:48.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.362286+0000", 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.038e-06 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.368810+0000", 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.006524102 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:48.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.368901+0000", 2024-09-17T16:17:48.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1592e-05 2024-09-17T16:17:48.975 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:48.975 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:48.975 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:48.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.975 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:13539 2.14 2:295cc8f1:::benchmark_data_smithi173_55121_object13538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.759315+0000", 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 396.20582920200002, 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.60567000500000001, 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 13539 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:48.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759315+0000", 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759315+0000", 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759324+0000", 2024-09-17T16:17:48.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.14e-06 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759364+0000", 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0513999999999999e-05 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759390+0000", 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6265e-05 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:48.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759403+0000", 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3088000000000001e-05 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759483+0000", 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9049999999999997e-05 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.759492+0000", 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7119999999999995e-06 2024-09-17T16:17:48.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.360226+0000", 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.60073335000000005 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.360357+0000", 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013103999999999999 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.360500+0000", 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014305899999999999 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364416+0000", 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039165040000000003 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364455+0000", 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9304000000000002e-05 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:48.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364915+0000", 2024-09-17T16:17:48.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045950800000000002 2024-09-17T16:17:48.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:48.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364954+0000", 2024-09-17T16:17:48.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8639000000000001e-05 2024-09-17T16:17:48.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:48.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.364985+0000", 2024-09-17T16:17:48.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1832000000000003e-05 2024-09-17T16:17:48.983 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:48.983 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:48.983 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:48.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.983 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:25215 2.10 e121/111 2:0e4d5290:::benchmark_data_smithi173_55121_object25214:head v 121'1162, mlcod=121'1158)", 2024-09-17T16:17:48.983 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.775819+0000", 2024-09-17T16:17:48.984 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 378.18932535499999, 2024-09-17T16:17:48.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44938945499999999, 2024-09-17T16:17:48.985 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:48.985 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:48.985 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775819+0000", 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775819+0000", 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:48.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775820+0000", 2024-09-17T16:17:48.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:17:48.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.775853+0000", 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3364000000000001e-05 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224477+0000", 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44862400299999999 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224483+0000", 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9730000000000002e-06 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224548+0000", 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4199e-05 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.224566+0000", 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8686999999999999e-05 2024-09-17T16:17:48.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.225168+0000", 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060204500000000005 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.225209+0000", 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0392999999999998e-05 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:48.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25240 2.9 2:9323c63b:::benchmark_data_smithi173_55121_object25239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.803581+0000", 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 378.16156331899998, 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.437608569, 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:48.991 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25240 2024-09-17T16:17:48.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.992 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:48.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:48.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803581+0000", 2024-09-17T16:17:48.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:48.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:48.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803581+0000", 2024-09-17T16:17:48.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:48.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:48.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803583+0000", 2024-09-17T16:17:48.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.265e-06 2024-09-17T16:17:48.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:48.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803637+0000", 2024-09-17T16:17:48.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4765999999999997e-05 2024-09-17T16:17:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803649+0000", 2024-09-17T16:17:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1304999999999999e-05 2024-09-17T16:17:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803655+0000", 2024-09-17T16:17:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2899999999999999e-06 2024-09-17T16:17:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803708+0000", 2024-09-17T16:17:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2627999999999998e-05 2024-09-17T16:17:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803793+0000", 2024-09-17T16:17:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5376000000000004e-05 2024-09-17T16:17:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:17:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.803906+0000", 2024-09-17T16:17:48.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011330100000000001 2024-09-17T16:17:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804549+0000", 2024-09-17T16:17:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00064244400000000004 2024-09-17T16:17:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:48.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.804821+0000", 2024-09-17T16:17:48.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027272499999999997 2024-09-17T16:17:48.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:48.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.241122+0000", 2024-09-17T16:17:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43630034000000001 2024-09-17T16:17:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.241159+0000", 2024-09-17T16:17:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7251e-05 2024-09-17T16:17:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.241190+0000", 2024-09-17T16:17:49.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0877999999999998e-05 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:28853 2.2 e128/104 2:4387b46d:::benchmark_data_smithi173_55121_object28852:head v 128'1332, mlcod=128'1329)", 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:48.540329+0000", 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 360.42481532800002, 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.364975192, 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.002 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540329+0000", 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540329+0000", 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540330+0000", 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3300000000000002e-07 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.540380+0000", 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.02e-05 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904638+0000", 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36425802000000002 2024-09-17T16:17:49.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904645+0000", 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6579999999999996e-06 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904700+0000", 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4855999999999998e-05 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.904714+0000", 2024-09-17T16:17:49.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3902e-05 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.905261+0000", 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054778599999999998 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:48.905305+0000", 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3136999999999997e-05 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.006 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:29877 2.2 e129/104 2:40729eb6:::benchmark_data_smithi173_55121_object29876:head v 129'1356, mlcod=129'1354)", 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 358.76095387100003, 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.87425371299999999, 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:17:49.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:17:49.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204191+0000", 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5700000000000002e-07 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:50.204236+0000", 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4901999999999998e-05 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077424+0000", 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.87318755800000003 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077440+0000", 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5753999999999999e-05 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077544+0000", 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104652 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.077590+0000", 2024-09-17T16:17:49.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5359999999999999e-05 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.078411+0000", 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082190400000000004 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:51.078445+0000", 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3025999999999999e-05 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(2.9 158/155 [PushReplyOp(2:9287a5b2:::benchmark_data_smithi173_55121_object60862:head)])", 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:48.880360+0000", 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 300.08478465000002, 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38735041999999997, 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880360+0000", 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880360+0000", 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880361+0000", 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3200000000000005e-07 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:48.880365+0000", 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3189999999999996e-06 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267083+0000", 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38671788000000001 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267091+0000", 2024-09-17T16:17:49.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8050000000000007e-06 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267182+0000", 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1217000000000004e-05 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.267710+0000", 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00052856699999999997 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:135561 2.6 2:612899dc:::benchmark_data_smithi173_55121_object59072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 253.04376820100001, 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36676542899999998, 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 135561 2024-09-17T16:17:49.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921378+0000", 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.897e-06 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921386+0000", 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.17e-06 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921398+0000", 2024-09-17T16:17:49.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2408999999999999e-05 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921406+0000", 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6179999999999997e-06 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921457+0000", 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1743999999999998e-05 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921550+0000", 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2318000000000006e-05 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,8", 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921623+0000", 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3337999999999995e-05 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.923435+0000", 2024-09-17T16:17:49.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001811654 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.923947+0000", 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051277600000000005 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288065+0000", 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36411789 2024-09-17T16:17:49.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288107+0000", 2024-09-17T16:17:49.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1479e-05 2024-09-17T16:17:49.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288142+0000", 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5135999999999997e-05 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:152966 2.c e199/95 2:3484f1c7:::benchmark_data_smithi173_55121_object76477:head v 199'9438, mlcod=199'9436)", 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:17:49.023 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 235.10901837500001, 2024-09-17T16:17:49.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58877864599999996, 2024-09-17T16:17:49.024 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.024 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.024 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.024 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:17:49.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856128+0000", 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3740000000000001e-06 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856136+0000", 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6950000000000006e-06 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441396+0000", 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58525992800000004 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441405+0000", 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9770000000000006e-06 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441463+0000", 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8025000000000001e-05 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441483+0000", 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9678999999999999e-05 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.444869+0000", 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033864070000000001 2024-09-17T16:17:49.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.444905+0000", 2024-09-17T16:17:49.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5561000000000001e-05 2024-09-17T16:17:49.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.028 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(3.0 256/239 [PushOp(3:0ca39f28:::benchmark_data_smithi173_66272_object41987:head, version: 231'2635, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0ca39f28:::benchmark_data_smithi173_66272_object41987:head@231'2635, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 111.924481012, 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59916758699999995, 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040665+0000", 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0100000000000003e-07 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040719+0000", 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4487000000000002e-05 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638102+0000", 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.597383255 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638121+0000", 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8595999999999999e-05 2024-09-17T16:17:49.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638205+0000", 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4042999999999995e-05 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638216+0000", 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1299e-05 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.639831+0000", 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001615006 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124280 3.0 3:099812ca:::benchmark_data_smithi173_66272_object52324:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377919+0000", 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.587225701999998, 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63194678999999998, 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124280 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377919+0000", 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377919+0000", 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377921+0000", 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.793e-06 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377925+0000", 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5859999999999998e-06 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377937+0000", 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1168999999999999e-05 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377944+0000", 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5209999999999997e-06 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391301+0000", 2024-09-17T16:17:49.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013356989999999999 2024-09-17T16:17:49.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:49.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391307+0000", 2024-09-17T16:17:49.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0939999999999996e-06 2024-09-17T16:17:49.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.983230+0000", 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59192265799999999 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.983309+0000", 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9083000000000002e-05 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:17:49.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.983395+0000", 2024-09-17T16:17:49.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6595999999999996e-05 2024-09-17T16:17:49.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:49.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991593+0000", 2024-09-17T16:17:49.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081975050000000008 2024-09-17T16:17:49.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991633+0000", 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9579000000000002e-05 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991675+0000", 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2407999999999998e-05 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009712+0000", 2024-09-17T16:17:49.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.018037553000000001 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009777+0000", 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4436000000000004e-05 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009866+0000", 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8819000000000005e-05 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124284 3.0 3:04f57090:::benchmark_data_smithi173_66272_object52328:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377957+0000", 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.587187274000001, 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63216316299999997, 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:49.042 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124284 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377957+0000", 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377957+0000", 2024-09-17T16:17:49.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377958+0000", 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377960+0000", 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9209999999999999e-06 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377964+0000", 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6569999999999996e-06 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377970+0000", 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3290000000000001e-06 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391358+0000", 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013388304 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391368+0000", 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0570999999999999e-05 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.986075+0000", 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.594706819 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.986194+0000", 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118577 2024-09-17T16:17:49.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.986314+0000", 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000119782 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991687+0000", 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053729240000000003 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996674+0000", 2024-09-17T16:17:49.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00498761 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.000224+0000", 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035494630000000001 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009919+0000", 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096958189999999996 2024-09-17T16:17:49.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009978+0000", 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8242000000000002e-05 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.010121+0000", 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014286999999999999 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124286 3.0 3:01b0fd2d:::benchmark_data_smithi173_66272_object52330:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377976+0000", 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.587168378999998, 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63695855000000001, 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124286 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.050 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377976+0000", 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377976+0000", 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377977+0000", 2024-09-17T16:17:49.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4699999999999998e-07 2024-09-17T16:17:49.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377978+0000", 2024-09-17T16:17:49.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.342e-06 2024-09-17T16:17:49.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377982+0000", 2024-09-17T16:17:49.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6389999999999998e-06 2024-09-17T16:17:49.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:49.054 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:49.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:49.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377985+0000", 2024-09-17T16:17:49.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2789999999999999e-06 2024-09-17T16:17:49.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391399+0000", 2024-09-17T16:17:49.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013414272 2024-09-17T16:17:49.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:49.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391403+0000", 2024-09-17T16:17:49.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.799e-06 2024-09-17T16:17:49.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.989045+0000", 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59764173499999995 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.989160+0000", 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000115654 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.989270+0000", 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000109574 2024-09-17T16:17:49.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996735+0000", 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074655440000000002 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996747+0000", 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2125e-05 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.010180+0000", 2024-09-17T16:17:49.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013432908 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.014804+0000", 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046233350000000001 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.014845+0000", 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1390999999999997e-05 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.014935+0000", 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9705999999999998e-05 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124287 3.0 3:022daafd:::benchmark_data_smithi173_66272_object52331:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.587156024000002, 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63722835200000005, 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.060 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124287 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:17:49.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:17:49.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2399999999999999e-07 2024-09-17T16:17:49.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377990+0000", 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1489999999999999e-06 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377993+0000", 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.19e-06 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377996+0000", 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8289999999999999e-06 2024-09-17T16:17:49.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391438+0000", 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013442154 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391448+0000", 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9159999999999996e-06 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991732+0000", 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.600283593 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991823+0000", 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1145999999999998e-05 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991905+0000", 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1664999999999998e-05 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996755+0000", 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048505700000000002 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.000183+0000", 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034277359999999998 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.010393+0000", 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010210558 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015049+0000", 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046556740000000003 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015106+0000", 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7339000000000001e-05 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015217+0000", 2024-09-17T16:17:49.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110609 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124291 3.0 3:0f520599:::benchmark_data_smithi173_66272_object52335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.587143830000002, 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.64518043599999997, 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.068 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124291 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0800000000000001e-07 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378002+0000", 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1620000000000001e-06 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378006+0000", 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9910000000000004e-06 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378009+0000", 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.785e-06 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391464+0000", 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013454729 2024-09-17T16:17:49.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391467+0000", 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4750000000000002e-06 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.994266+0000", 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.602799058 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.994361+0000", 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4487999999999994e-05 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.994432+0000", 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1518000000000002e-05 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.002826+0000", 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083936540000000004 2024-09-17T16:17:49.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.002845+0000", 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9128999999999999e-05 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015269+0000", 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012424269 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.022862+0000", 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075930640000000001 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.022977+0000", 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114539 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.023181+0000", 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000204367 2024-09-17T16:17:49.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124263 3.4 3:2eabebde:::benchmark_data_smithi173_66272_object52307:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382419+0000", 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.582725226999997, 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.499722689, 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:17:49.076 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124263 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382419+0000", 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382419+0000", 2024-09-17T16:17:49.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382421+0000", 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.536e-06 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382427+0000", 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6500000000000001e-06 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382441+0000", 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4368e-05 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382449+0000", 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3680000000000002e-06 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387414+0000", 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049646159999999998 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387421+0000", 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5889999999999999e-06 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882026+0000", 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49460579599999999 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882142+0000", 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000115766 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.080 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124265 3.4 3:2dc750d7:::benchmark_data_smithi173_66272_object52309:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.582656127, 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49990343999999998, 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:49.081 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124265 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5300000000000004e-07 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382492+0000", 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2370000000000001e-06 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382502+0000", 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0037000000000001e-05 2024-09-17T16:17:49.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382509+0000", 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9149999999999999e-06 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387622+0000", 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051124600000000001 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387630+0000", 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9510000000000006e-06 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882341+0000", 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49471140899999999 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882392+0000", 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0878000000000003e-05 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.085 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124266 3.4 3:2de8446c:::benchmark_data_smithi173_66272_object52310:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.582625806000003, 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.51201529199999996, 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:49.086 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124266 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:17:49.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2899999999999999e-07 2024-09-17T16:17:49.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382521+0000", 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8959999999999999e-06 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382527+0000", 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9950000000000002e-06 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382532+0000", 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1749999999999996e-06 2024-09-17T16:17:49.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387669+0000", 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051369980000000003 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387680+0000", 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1e-05 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882422+0000", 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49474148299999998 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882507+0000", 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4802999999999994e-05 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,5,11", 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882611+0000", 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104527 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.890862+0000", 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082508949999999994 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893785+0000", 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029228700000000002 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893832+0000", 2024-09-17T16:17:49.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7620999999999997e-05 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.894360+0000", 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000527605 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.894418+0000", 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8057000000000003e-05 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.894534+0000", 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116038 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124283 3.4 3:2189d34a:::benchmark_data_smithi173_66272_object52327:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382540+0000", 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.582604492999998, 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.52263484000000004, 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:49.094 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124283 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382540+0000", 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382540+0000", 2024-09-17T16:17:49.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382541+0000", 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6899999999999998e-07 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382543+0000", 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0530000000000001e-06 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382549+0000", 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0580000000000003e-06 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382554+0000", 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0300000000000001e-06 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387707+0000", 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051537110000000001 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387715+0000", 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8220000000000007e-06 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.886356+0000", 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49864073799999997 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.886497+0000", 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014137500000000001 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,5,11", 2024-09-17T16:17:49.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.886624+0000", 2024-09-17T16:17:49.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126551 2024-09-17T16:17:49.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:49.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893848+0000", 2024-09-17T16:17:49.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007223648 2024-09-17T16:17:49.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893892+0000", 2024-09-17T16:17:49.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4084e-05 2024-09-17T16:17:49.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.896419+0000", 2024-09-17T16:17:49.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002527502 2024-09-17T16:17:49.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:49.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.904985+0000", 2024-09-17T16:17:49.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0085655720000000005 2024-09-17T16:17:49.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.905056+0000", 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1730999999999995e-05 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.905175+0000", 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118596 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.102 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(3.c e290,267 3:3e4663e2:::benchmark_data_smithi173_66272_object807:head,251'4519)", 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.493506+0000", 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.471638798000001, 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62512154200000003, 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493506+0000", 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493506+0000", 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493507+0000", 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.164e-06 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493514+0000", 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3549999999999999e-06 2024-09-17T16:17:49.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118475+0000", 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62496048599999998 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118483+0000", 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6769999999999995e-06 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118554+0000", 2024-09-17T16:17:49.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1210000000000004e-05 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118561+0000", 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.029e-06 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118627+0000", 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6620999999999998e-05 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.107 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:49.107 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:49.107 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:17:49.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:49.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:49.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:49.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:49.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:49.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:49.379 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:17:49.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:49.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:49.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:49.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 14442 14426 112.694 71.4375 0.00219838 0.00883558 2024-09-17T16:17:49.527 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:49.553 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:49.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:49.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:49.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:49 smithi173 ceph-mon[30530]: pgmap v718: 17 pgs: 17 active+clean; 477 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-17T16:17:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2061250098' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:17:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:49 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:17:49.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:49.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:49.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:49 smithi017 ceph-mon[27571]: pgmap v718: 17 pgs: 17 active+clean; 477 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-17T16:17:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2061250098' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:17:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:49 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:17:49.781 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:49.804 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:49.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:49.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:49.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:49 smithi071 ceph-mon[30002]: pgmap v718: 17 pgs: 17 active+clean; 477 MiB data, 3.0 GiB used, 980 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-17T16:17:49.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2061250098' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:17:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:49 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:17:49.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:17:49.907 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:49.928 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:49.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:49.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:49.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:17:50.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:50.060 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:50.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:50.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:50.168 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:17:50.169 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:17:50.169 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T16:17:50.169 INFO:tasks.thrashosds.thrasher:Setting osd 9 primary_affinity to 0.282132 2024-09-17T16:17:50.169 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 9 0.28213196454631584 2024-09-17T16:17:50.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:50.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:50.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:50.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:50.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:50.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:50.442 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:50.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:50.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:50 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:50 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:17:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:50 smithi017 ceph-mon[27571]: osdmap e310: 12 total, 12 up, 11 in 2024-09-17T16:17:50.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 15736 15720 109.158 80.875 0.00150775 0.00912719 2024-09-17T16:17:50.544 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:50.569 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:50.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:50 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:50 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:17:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:50 smithi173 ceph-mon[30530]: osdmap e310: 12 total, 12 up, 11 in 2024-09-17T16:17:50.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:17:50.696 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:50.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:17:50.745 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:50.765 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:50.798 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:50.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:50 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:50 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:17:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:50 smithi071 ceph-mon[30002]: osdmap e310: 12 total, 12 up, 11 in 2024-09-17T16:17:50.823 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:50.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:50.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:50.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:50.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:50.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:51.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:51.076 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:51.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:51.179 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:51.205 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:51.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:51.307 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:51.328 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:51.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:51.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:51.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:51.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:51.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:51.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 17125 17109 106.923 86.8125 0.0629698 0.00932398 2024-09-17T16:17:51.556 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:51.583 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:51.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:51.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:51.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:51 smithi173 ceph-mon[30530]: pgmap v720: 17 pgs: 17 active+clean; 679 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-17T16:17:51.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:51.698 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:17:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:17:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:17085 4.4 4:2c21b0f2:::benchmark_data_smithi173_77474_object17084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-09-17T16:17:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:51.457518+0000", 2024-09-17T16:17:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.031328504, 2024-09-17T16:17:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.031358218, 2024-09-17T16:17:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:17:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:17:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:17:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17085 2024-09-17T16:17:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.457518+0000", 2024-09-17T16:17:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.457518+0000", 2024-09-17T16:17:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.457520+0000", 2024-09-17T16:17:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.916e-06 2024-09-17T16:17:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.457562+0000", 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1446000000000003e-05 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.457576+0000", 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4008e-05 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.457585+0000", 2024-09-17T16:17:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9989999999999997e-06 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.457637+0000", 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2565999999999998e-05 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.457730+0000", 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2968999999999997e-05 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:17:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.457821+0000", 2024-09-17T16:17:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1101999999999996e-05 2024-09-17T16:17:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.458649+0000", 2024-09-17T16:17:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082818599999999996 2024-09-17T16:17:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.458952+0000", 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00030232499999999999 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:17098 4.4 4:20b1acf6:::benchmark_data_smithi173_77474_object17097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:51.477171+0000", 2024-09-17T16:17:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.011675899999999999, 2024-09-17T16:17:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011787219, 2024-09-17T16:17:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:17:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:17:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:17:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17098 2024-09-17T16:17:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.477171+0000", 2024-09-17T16:17:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.477171+0000", 2024-09-17T16:17:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.477172+0000", 2024-09-17T16:17:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3880000000000001e-06 2024-09-17T16:17:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.477204+0000", 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1313000000000003e-05 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.477220+0000", 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6755000000000001e-05 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.477227+0000", 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7900000000000002e-06 2024-09-17T16:17:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.477284+0000", 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7067000000000001e-05 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.477411+0000", 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000127296 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:17:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.477513+0000", 2024-09-17T16:17:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000101975 2024-09-17T16:17:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.478417+0000", 2024-09-17T16:17:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090331799999999996 2024-09-17T16:17:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.478797+0000", 2024-09-17T16:17:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00038066199999999998 2024-09-17T16:17:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:17:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 2 2024-09-17T16:17:51.712 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:17:51.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:51 smithi017 ceph-mon[27571]: pgmap v720: 17 pgs: 17 active+clean; 679 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-17T16:17:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:51.714 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:51.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:51.795 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:17:51.795 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:17:51.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:51 smithi071 ceph-mon[30002]: pgmap v720: 17 pgs: 17 active+clean; 679 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-17T16:17:51.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:51.817 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:17:51.817 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:17:51.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:51.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:51.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:51.961 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:51.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:52.038 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:17:52.038 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:17:52.061 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:17:52.061 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:17:52.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:52.082 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:17:52.082 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:17:52.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:52.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:52.103 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:17:52.103 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:17:52.124 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:17:52.124 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:17:52.148 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:17:52.149 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:17:52.170 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:17:52.170 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:17:52.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:52.193 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:17:52.193 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:17:52.218 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:52.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:52.223 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:17:52.224 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:17:52.255 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:17:52.256 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:17:52.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:52.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:52.353 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:52.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:52.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:52.483 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:52.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:52.508 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:17:52.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 18096 18080 102.719 60.6875 0.0033175 0.00971904 2024-09-17T16:17:52.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:52.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:52.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:52.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:52.722 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:52.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:17:52.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:52.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:52.735 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:52.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/59013470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/59013470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:52.835 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:52.858 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:52.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:52.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:52.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/59013470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:52.960 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:52.986 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:52.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:52.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:53.088 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:53.113 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:53.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:53.215 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:53.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:53.342 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:53.367 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:53.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:53.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:53.491 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:53.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:53.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:53.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 19192 19176 99.8673 68.5 0.0408424 0.00999292 2024-09-17T16:17:53.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:53.605 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:53 smithi017 ceph-mon[27571]: pgmap v721: 17 pgs: 17 active+clean; 946 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-17T16:17:53.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:53 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:17:53.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:53 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:17:53.620 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:53.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:53.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:53.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:53.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:53.785 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:53.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:53 smithi071 ceph-mon[30002]: pgmap v721: 17 pgs: 17 active+clean; 946 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-17T16:17:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:53 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:17:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:53 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:17:53.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:53.880 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:53.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:53.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:53 smithi173 ceph-mon[30530]: pgmap v721: 17 pgs: 17 active+clean; 946 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-17T16:17:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:53 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:17:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:53 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:17:53.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:54.006 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:54.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:54.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:54.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:54.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:54.232 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:54.255 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:54.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:54.331 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:54.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:54.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:54.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:54.407 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.9 primary-affinity to 0.282132 (8184892) 2024-09-17T16:17:54.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:54.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:54 smithi017 ceph-mon[27571]: osdmap e311: 12 total, 12 up, 11 in 2024-09-17T16:17:54.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:54 smithi017 ceph-mon[27571]: pgmap v723: 17 pgs: 17 active+clean; 1020 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-17T16:17:54.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4166985636' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.28213196454631584} : dispatch 2024-09-17T16:17:54.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:54 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.28213196454631584} : dispatch 2024-09-17T16:17:54.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:54.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:54.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:54.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 20332 20316 97.6656 71.25 0.00166172 0.0101762 2024-09-17T16:17:54.607 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:54.629 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:54.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:54.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:54.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:54.755 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:54.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:54 smithi071 ceph-mon[30002]: osdmap e311: 12 total, 12 up, 11 in 2024-09-17T16:17:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:54 smithi071 ceph-mon[30002]: pgmap v723: 17 pgs: 17 active+clean; 1020 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-17T16:17:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4166985636' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.28213196454631584} : dispatch 2024-09-17T16:17:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:54 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.28213196454631584} : dispatch 2024-09-17T16:17:54.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:54.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:54.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:54.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:54 smithi173 ceph-mon[30530]: osdmap e311: 12 total, 12 up, 11 in 2024-09-17T16:17:54.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:54 smithi173 ceph-mon[30530]: pgmap v723: 17 pgs: 17 active+clean; 1020 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-17T16:17:54.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4166985636' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.28213196454631584} : dispatch 2024-09-17T16:17:54.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:54 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.28213196454631584} : dispatch 2024-09-17T16:17:54.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:55.003 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:55.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:55.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:55.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:55.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:55.194 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:17:55.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:55.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:55.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:55.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:55.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:55.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:55.413 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:17:55.426 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:55 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.28213196454631584}]': finished 2024-09-17T16:17:55.426 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:55 smithi017 ceph-mon[27571]: osdmap e312: 12 total, 12 up, 11 in 2024-09-17T16:17:55.426 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2906394198' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:55.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:55.505 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:55.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:55.523 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 20759 20743 92.5956 26.6875 0.00308822 0.0107935 2024-09-17T16:17:55.607 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:55.636 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:55.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:55.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:55.767 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:55.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:55 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.28213196454631584}]': finished 2024-09-17T16:17:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:55 smithi071 ceph-mon[30002]: osdmap e312: 12 total, 12 up, 11 in 2024-09-17T16:17:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2906394198' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:55.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:55.893 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:55.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:55.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:55 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.28213196454631584}]': finished 2024-09-17T16:17:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:55 smithi173 ceph-mon[30530]: osdmap e312: 12 total, 12 up, 11 in 2024-09-17T16:17:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2906394198' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:17:55.965 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:17:55.965 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:17:55.995 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:56.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:56.020 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:56.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:56.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:56.147 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:56.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:56.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:56.270 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:56.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:56.372 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:56.399 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:56.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:56.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:56.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 22068 22052 91.8763 81.8125 0.00192794 0.0108618 2024-09-17T16:17:56.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:56.525 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:56.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:56.626 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:56.647 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:56.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:56.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:56.729 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:56 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:56.729 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2906394198' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:56.729 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:56 smithi017 ceph-mon[27571]: osdmap e313: 12 total, 12 up, 11 in 2024-09-17T16:17:56.729 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:56 smithi017 ceph-mon[27571]: pgmap v726: 17 pgs: 17 active+clean; 1.2 GiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:17:56.750 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:56.773 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:56.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:56.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:56.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:56 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2906394198' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:56 smithi071 ceph-mon[30002]: osdmap e313: 12 total, 12 up, 11 in 2024-09-17T16:17:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:56 smithi071 ceph-mon[30002]: pgmap v726: 17 pgs: 17 active+clean; 1.2 GiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:17:56.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:17:56.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:56 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:17:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2906394198' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:17:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:56 smithi173 ceph-mon[30530]: osdmap e313: 12 total, 12 up, 11 in 2024-09-17T16:17:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:56 smithi173 ceph-mon[30530]: pgmap v726: 17 pgs: 17 active+clean; 1.2 GiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 87 MiB/s wr, 1.39k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:17:56.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:56.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:56.902 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:17:57.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:57.028 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:57.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:57.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:57.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:57.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:57.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:57.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:57.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:57.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:57.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:57.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:57.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:17:57.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:17:57.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 23055 23039 89.989 61.6875 0.00326241 0.0110868 2024-09-17T16:17:57.528 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:57.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:17:57.630 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:57.652 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:57.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:57.754 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:57.784 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:57.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:57.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:57.886 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:57.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:17:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:57.917 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:57.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:57.941 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:58.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:58.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:58.042 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:58.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:58.168 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:58.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:58.270 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:58.295 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:58.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:58.397 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:17:58.424 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:58.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:17:58.504 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:17:58.505 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:17:58.505 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:17:58.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 24424 24408 89.7283 85.5625 0.00915172 0.0111378 2024-09-17T16:17:58.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:17:58.546 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:17:58.546 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:17:58.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:58.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:17:58.582 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:17:58.583 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:17:58.619 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:17:58.619 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:17:58.646 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:17:58.647 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:17:58.653 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:17:58.678 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:17:58.678 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:17:58.681 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:58.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:17:58.700 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:17:58.701 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:17:58.723 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:17:58.724 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:17:58.745 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:17:58.745 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:17:58.767 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:17:58.767 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:17:58.783 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:58.788 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:17:58.788 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:17:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:58.805 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:58.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:58.822 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:17:58.822 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:17:58.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:17:58.930 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:17:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:58.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:17:59.032 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:17:59.060 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:59.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:17:59.095 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:17:59.095 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:17:59.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:17:59.095 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:17:59.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.095 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:13535 2.11 2:8808aac6:::benchmark_data_smithi173_55121_object13534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:17:59.095 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:17:59.095 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 406.756247051, 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314941879999999, 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 13535 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335348+0000", 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335349+0000", 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3599999999999999e-06 2024-09-17T16:17:59.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335378+0000", 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8965000000000001e-05 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335391+0000", 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.257e-05 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.335398+0000", 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3499999999999999e-06 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.367429+0000", 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.032030422000000003 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.367439+0000", 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0451e-05 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.366280+0000", 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.99884132299999995 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:13.366842+0000", 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00056174700000000003 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.100 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:14925 2.e 2:743cbf17:::benchmark_data_smithi173_55121_object14924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.264388+0000", 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 404.827207004, 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43099893299999997, 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.101 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 14925 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264388+0000", 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264388+0000", 2024-09-17T16:17:59.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264390+0000", 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.677e-06 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264432+0000", 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2292000000000001e-05 2024-09-17T16:17:59.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264447+0000", 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.518e-05 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264455+0000", 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2490000000000001e-06 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264541+0000", 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5760000000000006e-05 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264676+0000", 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134487 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.264784+0000", 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108402 2024-09-17T16:17:59.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265461+0000", 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067649699999999997 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265775+0000", 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031408899999999998 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695211+0000", 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42943639900000002 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695308+0000", 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7176000000000006e-05 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695387+0000", 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8725000000000003e-05 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.107 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:14925 2.e e106/95 ondisk, result = 0)", 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 404.82575952899998, 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42955775600000001, 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265836+0000", 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.265844+0000", 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2450000000000001e-06 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695035+0000", 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42919093699999999 2024-09-17T16:17:59.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695052+0000", 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.772e-05 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695186+0000", 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013379200000000001 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695202+0000", 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5519000000000002e-05 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.695393+0000", 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019176600000000001 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:14944 2.e 2:79e7edec:::benchmark_data_smithi173_55121_object14943:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.358500+0000", 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 404.73309491499998, 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34235612100000001, 2024-09-17T16:17:59.113 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 14944 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358500+0000", 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358500+0000", 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358502+0000", 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8899999999999999e-06 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358530+0000", 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8248000000000001e-05 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358548+0000", 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8068e-05 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358555+0000", 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.934e-06 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358614+0000", 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8488e-05 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358718+0000", 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103991 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.358818+0000", 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100374 2024-09-17T16:17:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359536+0000", 2024-09-17T16:17:59.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00071760900000000002 2024-09-17T16:17:59.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359808+0000", 2024-09-17T16:17:59.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027181699999999999 2024-09-17T16:17:59.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700797+0000", 2024-09-17T16:17:59.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.340989443 2024-09-17T16:17:59.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700830+0000", 2024-09-17T16:17:59.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2543999999999999e-05 2024-09-17T16:17:59.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700856+0000", 2024-09-17T16:17:59.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6715000000000001e-05 2024-09-17T16:17:59.120 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.120 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.120 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:14944 2.e e106/95 ondisk, result = 0)", 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:14.359723+0000", 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 404.73187222000001, 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.341137735, 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359723+0000", 2024-09-17T16:17:59.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359723+0000", 2024-09-17T16:17:59.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359724+0000", 2024-09-17T16:17:59.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9299999999999996e-07 2024-09-17T16:17:59.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.359729+0000", 2024-09-17T16:17:59.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5770000000000001e-06 2024-09-17T16:17:59.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700733+0000", 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34100362499999998 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700739+0000", 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3199999999999996e-06 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700789+0000", 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9845000000000001e-05 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700794+0000", 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6299999999999997e-06 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:14.700861+0000", 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6845000000000001e-05 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(2.4 116/112 [PushOp(2:23763086:::benchmark_data_smithi173_55121_object9285:head, version: 101'619, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:23763086:::benchmark_data_smithi173_55121_object9285:head@101'619, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:25.348190+0000", 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 393.74340468499997, 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61457114000000002, 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348190+0000", 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348190+0000", 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348191+0000", 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3000000000000001e-07 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.348211+0000", 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9811999999999999e-05 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.961507+0000", 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61329585900000005 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.961527+0000", 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0744999999999999e-05 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.962424+0000", 2024-09-17T16:17:59.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089668899999999999 2024-09-17T16:17:59.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.962438+0000", 2024-09-17T16:17:59.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3458000000000001e-05 2024-09-17T16:17:59.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.962762+0000", 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000324047 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(2.4 117/112 [PushOp(2:26e71621:::benchmark_data_smithi173_55121_object17115:head, version: 108'1004, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:26e71621:::benchmark_data_smithi173_55121_object17115:head@108'1004, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:26.624251+0000", 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 392.46734415499998, 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.252196103, 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624251+0000", 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624251+0000", 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624252+0000", 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.029e-06 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.624283+0000", 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0960000000000002e-05 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875818+0000", 2024-09-17T16:17:59.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25153555500000002 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875841+0000", 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2894000000000001e-05 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875950+0000", 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108758 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.875961+0000", 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0784e-05 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:26.876447+0000", 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048612299999999998 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.134 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:24327 2.17 e120/104 2:ecf8c341:::benchmark_data_smithi173_55121_object24326:head v 120'1226, mlcod=120'1223)", 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 389.67369556599999, 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.245189986, 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417900+0000", 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8199999999999999e-07 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.417946+0000", 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6263000000000001e-05 2024-09-17T16:17:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662362+0000", 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24441564700000001 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662376+0000", 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3573e-05 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662484+0000", 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000108559 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.662517+0000", 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2755000000000001e-05 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.663062+0000", 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054487399999999999 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:29.663090+0000", 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7633000000000001e-05 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25228 2.1 2:8352dd7d:::benchmark_data_smithi173_55121_object25227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.786229+0000", 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 388.30536562700001, 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28014749900000002, 2024-09-17T16:17:59.139 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25228 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786229+0000", 2024-09-17T16:17:59.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786229+0000", 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786230+0000", 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9199999999999999e-07 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786266+0000", 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5939999999999998e-05 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786275+0000", 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0140000000000004e-06 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786280+0000", 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4710000000000001e-06 2024-09-17T16:17:59.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786324+0000", 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3829999999999999e-05 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786410+0000", 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6013000000000005e-05 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.786496+0000", 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5784000000000004e-05 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.787047+0000", 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00055157999999999997 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.788017+0000", 2024-09-17T16:17:59.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097012900000000002 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066298+0000", 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27828037 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066347+0000", 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9508000000000001e-05 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.066377+0000", 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9867999999999999e-05 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25249 2.e 2:7b694fa0:::benchmark_data_smithi173_55121_object25248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.912148+0000", 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 388.17944738599999, 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33397465900000001, 2024-09-17T16:17:59.146 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25249 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912148+0000", 2024-09-17T16:17:59.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912148+0000", 2024-09-17T16:17:59.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912149+0000", 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912174+0000", 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5080999999999999e-05 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912187+0000", 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3311e-05 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912193+0000", 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6110000000000001e-06 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912270+0000", 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6598000000000001e-05 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912380+0000", 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110389 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.912476+0000", 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5493000000000002e-05 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913408+0000", 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093208600000000005 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.965233+0000", 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.051825489000000002 2024-09-17T16:17:59.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246005+0000", 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28077230800000003 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246071+0000", 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6000999999999994e-05 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246122+0000", 2024-09-17T16:17:59.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0955999999999998e-05 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:25249 2.e e121/95 ondisk, result = 0)", 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.913496+0000", 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 388.17809917099999, 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.332636237, 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913496+0000", 2024-09-17T16:17:59.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913496+0000", 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913497+0000", 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.913500+0000", 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5449999999999999e-06 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245930+0000", 2024-09-17T16:17:59.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33242972799999998 2024-09-17T16:17:59.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.157 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:58 smithi173 ceph-mon[30530]: pgmap v727: 17 pgs: 17 active+clean; 1.3 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 59 MiB/s wr, 942 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:17:59.157 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:17:59.157 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:17:59.157 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/991051292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:59.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245935+0000", 2024-09-17T16:17:59.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9479999999999999e-06 2024-09-17T16:17:59.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245989+0000", 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4187999999999997e-05 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.245999+0000", 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9729999999999992e-06 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.246132+0000", 2024-09-17T16:17:59.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000133118 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:26909 2.e 2:76c214cd:::benchmark_data_smithi173_55121_object26908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.325897+0000", 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.76569782899998, 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23800827699999999, 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.159 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26909 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325897+0000", 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325897+0000", 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325898+0000", 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1379999999999999e-06 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325939+0000", 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0466000000000002e-05 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325950+0000", 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1287000000000001e-05 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.325955+0000", 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1730000000000003e-06 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326000+0000", 2024-09-17T16:17:59.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4674e-05 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326082+0000", 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1706999999999995e-05 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326170+0000", 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7903000000000002e-05 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.326971+0000", 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080154300000000005 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327360+0000", 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00038908699999999998 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563817+0000", 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23645692900000001 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563856+0000", 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9150000000000003e-05 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.563906+0000", 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.922e-05 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.165 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:26910 2.1 2:838cca45:::benchmark_data_smithi173_55121_object26909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.76412257200002, 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26261527099999998, 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26910 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327473+0000", 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.23e-07 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327529+0000", 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5284000000000001e-05 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327538+0000", 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7790000000000007e-06 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327543+0000", 2024-09-17T16:17:59.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4700000000000004e-06 2024-09-17T16:17:59.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327599+0000", 2024-09-17T16:17:59.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6100999999999997e-05 2024-09-17T16:17:59.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327722+0000", 2024-09-17T16:17:59.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012263399999999999 2024-09-17T16:17:59.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.327808+0000", 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6803000000000002e-05 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.328412+0000", 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060394300000000001 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.328847+0000", 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00043514399999999999 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.590005+0000", 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26115713200000001 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.590043+0000", 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.824e-05 2024-09-17T16:17:59.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.590088+0000", 2024-09-17T16:17:59.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5018000000000002e-05 2024-09-17T16:17:59.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.173 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:26911 2.e 2:7501b6f7:::benchmark_data_smithi173_55121_object26910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:32.329019+0000", 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.762576314, 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26290611800000002, 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26911 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329019+0000", 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329019+0000", 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329020+0000", 2024-09-17T16:17:59.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329082+0000", 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2477000000000005e-05 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329093+0000", 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0814e-05 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329100+0000", 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1339999999999998e-06 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329141+0000", 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0915999999999999e-05 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329219+0000", 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7902000000000001e-05 2024-09-17T16:17:59.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.329304+0000", 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4820000000000004e-05 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330001+0000", 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00069732400000000003 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.330213+0000", 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000212563 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.591852+0000", 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26163829100000002 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.591886+0000", 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4662000000000001e-05 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:32.591925+0000", 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8470000000000003e-05 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:27009 2.e 2:7c49ab26:::benchmark_data_smithi173_55121_object27008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:33.169267+0000", 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 385.92232796899998, 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33189007700000001, 2024-09-17T16:17:59.180 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 27009 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169267+0000", 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169267+0000", 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169270+0000", 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9469999999999999e-06 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169342+0000", 2024-09-17T16:17:59.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1785999999999993e-05 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169372+0000", 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0399000000000002e-05 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169387+0000", 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4323e-05 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169516+0000", 2024-09-17T16:17:59.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012973600000000001 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169614+0000", 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8103000000000006e-05 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.169705+0000", 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0989000000000006e-05 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.170561+0000", 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085536800000000001 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.170823+0000", 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026205999999999999 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.501034+0000", 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33021098799999998 2024-09-17T16:17:59.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.501104+0000", 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0321000000000007e-05 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:33.501157+0000", 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3056999999999997e-05 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.187 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:29460 2.1 2:83aee601:::benchmark_data_smithi173_55121_object29459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-17T16:17:59.187 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:17:59.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:58 smithi017 ceph-mon[27571]: pgmap v727: 17 pgs: 17 active+clean; 1.3 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 59 MiB/s wr, 942 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:17:59.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:17:59.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:17:59.188 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/991051292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:59.188 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 369.89393262200002, 2024-09-17T16:17:59.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31850449400000003, 2024-09-17T16:17:59.188 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.188 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.188 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.188 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.188 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 29460 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:17:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197664+0000", 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1289999999999999e-06 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197740+0000", 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6361999999999998e-05 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197753+0000", 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3453e-05 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197759+0000", 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7610000000000004e-06 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197820+0000", 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0735e-05 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197960+0000", 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013987799999999999 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198120+0000", 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016034 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198880+0000", 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075943099999999997 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.322818+0000", 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.123938883 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516108+0000", 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.193289187 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516141+0000", 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3691e-05 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516167+0000", 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5644e-05 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.194 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:129847 2.1 2:86a30a1b:::benchmark_data_smithi173_55121_object53358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 271.29255265199998, 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29781534999999998, 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 129847 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799048+0000", 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6289999999999998e-06 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799056+0000", 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3809999999999999e-06 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799074+0000", 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7864e-05 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799085+0000", 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1076e-05 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799177+0000", 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1402000000000003e-05 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799338+0000", 2024-09-17T16:17:59.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000161263 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799482+0000", 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000143638 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801720+0000", 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022381739999999999 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.901264+0000", 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.099544493999999997 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096745+0000", 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.195480396 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096804+0000", 2024-09-17T16:17:59.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8990000000000003e-05 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096858+0000", 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4042999999999997e-05 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:129847 2.1 e184/104 ondisk, result = 0)", 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:17:59.201 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 271.289595824, 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29486439199999998, 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:17:59.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.802000+0000", 2024-09-17T16:17:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1600000000000001e-07 2024-09-17T16:17:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.802005+0000", 2024-09-17T16:17:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9060000000000001e-06 2024-09-17T16:17:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096600+0000", 2024-09-17T16:17:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.294595249 2024-09-17T16:17:59.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096606+0000", 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.733e-06 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096731+0000", 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000124735 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096739+0000", 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096864+0000", 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012515399999999999 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(3.4 241/239 [PushOp(3:20540438:::benchmark_data_smithi173_66272_object5569:head, version: 213'343, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:20540438:::benchmark_data_smithi173_66272_object5569:head@213'343, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:17:59.207 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 161.538801138, 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56341482399999998, 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552796+0000", 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.888e-06 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552825+0000", 2024-09-17T16:17:59.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9417999999999999e-05 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115364+0000", 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56253852800000004 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115383+0000", 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9055999999999998e-05 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115470+0000", 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7520000000000002e-05 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115481+0000", 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0329000000000001e-05 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.116209+0000", 2024-09-17T16:17:59.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072808499999999995 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:17197 4.4 4:27b87537:::benchmark_data_smithi173_77474_object17196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:51.585134+0000", 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 7.5064613859999998, 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27644972000000001, 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:17:59.212 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17197 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.585134+0000", 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:17:59.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.585134+0000", 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.585135+0000", 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.097e-06 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.585178+0000", 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3489000000000003e-05 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.585191+0000", 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.238e-05 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.585196+0000", 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.784e-06 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.585246+0000", 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9422999999999998e-05 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.585339+0000", 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3480000000000006e-05 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.585433+0000", 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3468999999999996e-05 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.586164+0000", 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073132299999999998 2024-09-17T16:17:59.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.586540+0000", 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00037550700000000002 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.861523+0000", 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27498326400000001 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:17:59.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.861556+0000", 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3302000000000001e-05 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:51.861583+0000", 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7202000000000001e-05 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:17:59.218 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:17:59.219 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:59.250 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:59.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:59.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:58 smithi071 ceph-mon[30002]: pgmap v727: 17 pgs: 17 active+clean; 1.3 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 59 MiB/s wr, 942 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:17:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:17:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:17:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/991051292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:17:59.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:59.384 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:59.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:59.421 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:17:59.488 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:17:59.516 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:59.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:17:59.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 26161 26145 90.7742 108.562 0.013882 0.0110099 2024-09-17T16:17:59.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:17:59.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:59.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:17:59.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:17:59.776 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:17:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:59.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:17:59.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:17:59.906 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:17:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:17:59.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:00.008 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:00.036 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:00.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:00.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:00.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:59 smithi173 ceph-mon[30530]: osdmap e314: 12 total, 12 up, 11 in 2024-09-17T16:18:00.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:17:59 smithi173 ceph-mon[30530]: osdmap e315: 12 total, 12 up, 11 in 2024-09-17T16:18:00.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:00.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:59 smithi017 ceph-mon[27571]: osdmap e314: 12 total, 12 up, 11 in 2024-09-17T16:18:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:17:59 smithi017 ceph-mon[27571]: osdmap e315: 12 total, 12 up, 11 in 2024-09-17T16:18:00.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:00.286 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:00.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:00.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:59 smithi071 ceph-mon[30002]: osdmap e314: 12 total, 12 up, 11 in 2024-09-17T16:18:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:17:59 smithi071 ceph-mon[30002]: osdmap e315: 12 total, 12 up, 11 in 2024-09-17T16:18:00.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:00.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:00.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:00.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:00.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:00.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 27626 27610 90.8153 91.5625 0.0095142 0.0110041 2024-09-17T16:18:00.543 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:00.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:00.646 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:00.671 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:00.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:00.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:00.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:00.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:00.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:00.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:00.920 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:00.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:01.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:01.046 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:01.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:01.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:01.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:01.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:01.277 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:01.309 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:01.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:01.411 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:01.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:18:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:01.439 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:01.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:01.521 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:01.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:18:01.523215+0000 min lat: 0.00106175 max lat: 1.04372 avg lat: 0.010958 2024-09-17T16:18:01.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:18:01.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 29198 29182 91.1866 98.25 0.00229727 0.010958 2024-09-17T16:18:01.540 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:01.576 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:01.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:01.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:01 smithi173 ceph-mon[30530]: pgmap v730: 17 pgs: 17 active+clean; 1.4 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s; 12 KiB/s, 0 objects/s recovering 2024-09-17T16:18:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:01 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:18:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:01 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:18:01.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:01 smithi017 ceph-mon[27571]: pgmap v730: 17 pgs: 17 active+clean; 1.4 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s; 12 KiB/s, 0 objects/s recovering 2024-09-17T16:18:01.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:01 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:18:01.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:01 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:18:01.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:01.704 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:01.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:01.765 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:18:01.766 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T16:18:01.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:01.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:01 smithi071 ceph-mon[30002]: pgmap v730: 17 pgs: 17 active+clean; 1.4 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s; 12 KiB/s, 0 objects/s recovering 2024-09-17T16:18:01.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:01 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:18:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:01 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:18:01.842 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:01.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:01.944 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:01.969 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:01.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:02.071 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:02.094 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:02.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:02.180 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:02.189 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:02.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:02.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:02.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:02.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:02.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:02.351 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:02.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:02.388 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:18:02.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:02.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:02.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:02.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 30720 30704 91.3738 95.125 0.00881084 0.0109378 2024-09-17T16:18:02.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:02.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:02.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:02.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:02 smithi173 ceph-mon[30530]: osdmap e316: 12 total, 12 up, 11 in 2024-09-17T16:18:02.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2889869822' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:18:02.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:02.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:02 smithi017 ceph-mon[27571]: osdmap e316: 12 total, 12 up, 11 in 2024-09-17T16:18:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2889869822' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:18:02.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:02.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:02.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:02.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:02 smithi071 ceph-mon[30002]: osdmap e316: 12 total, 12 up, 11 in 2024-09-17T16:18:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2889869822' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:18:02.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:02.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:02.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:02.964 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:18:02.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:03.000 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:03.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:03.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:03.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:03.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:03.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:03.256 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:03.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:03.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:03.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:03.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:03.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:03.486 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:03.510 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:03 smithi017 ceph-mon[27571]: pgmap v732: 17 pgs: 17 active+clean; 1.7 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:18:03.510 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:03 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:03.510 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2889869822' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:18:03.510 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:03 smithi017 ceph-mon[27571]: osdmap e317: 12 total, 12 up, 11 in 2024-09-17T16:18:03.513 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:03.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:03.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 32222 32206 91.4872 93.875 0.025443 0.0109228 2024-09-17T16:18:03.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:03.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:03.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:03.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:03 smithi173 ceph-mon[30530]: pgmap v732: 17 pgs: 17 active+clean; 1.7 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:18:03.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:03 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:03.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2889869822' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:18:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:03 smithi173 ceph-mon[30530]: osdmap e317: 12 total, 12 up, 11 in 2024-09-17T16:18:03.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:03 smithi071 ceph-mon[30002]: pgmap v732: 17 pgs: 17 active+clean; 1.7 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:18:03.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:03 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:03.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2889869822' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:18:03.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:03 smithi071 ceph-mon[30002]: osdmap e317: 12 total, 12 up, 11 in 2024-09-17T16:18:03.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:03.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:03.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:03.866 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:03.898 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:03.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:03.917 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:18:03.917 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:18:03.917 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:18:03.917 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:18:04.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:04.026 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:04.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:04.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:04.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:04.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:04.251 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:04.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:04.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:04.376 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:04.402 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:04.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:04.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:04.404 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T16:18:04.504 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:04.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 33499 33483 90.9793 79.8125 0.0119663 0.0109849 2024-09-17T16:18:04.531 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:04.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:04.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:04.659 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:04.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3773853741' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:18:04.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:04.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:04.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:04.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3773853741' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:18:04.839 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:18:04.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3773853741' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:18:04.912 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:04.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:05.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:05.034 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:05.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:05.073 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:05.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:05.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:05.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:05.256 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:18:05.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:05.299 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:05.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:05.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:05.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:05.426 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:05.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:05 smithi017 ceph-mon[27571]: pgmap v734: 17 pgs: 17 active+clean; 1.7 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-17T16:18:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3773853741' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:18:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:05 smithi017 ceph-mon[27571]: osdmap e318: 12 total, 12 up, 11 in 2024-09-17T16:18:05.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 35007 34991 91.1153 94.25 0.003072 0.0109697 2024-09-17T16:18:05.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:05.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:05.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:05.659 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:05.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T16:18:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:05.688 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:05.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:05.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:05.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:05 smithi071 ceph-mon[30002]: pgmap v734: 17 pgs: 17 active+clean; 1.7 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-17T16:18:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3773853741' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:18:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:05 smithi071 ceph-mon[30002]: osdmap e318: 12 total, 12 up, 11 in 2024-09-17T16:18:05.817 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:05.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:05 smithi173 ceph-mon[30530]: pgmap v734: 17 pgs: 17 active+clean; 1.7 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-17T16:18:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3773853741' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:18:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:05 smithi173 ceph-mon[30530]: osdmap e318: 12 total, 12 up, 11 in 2024-09-17T16:18:05.920 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:05.945 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:05.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:06.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:06.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:06.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:06.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:06.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:06.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:06.298 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:06.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:06.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:06.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:06.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:06.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:06.524 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 36070 36054 90.128 66.4375 0.00286785 0.0110874 2024-09-17T16:18:06.547 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:06.576 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:06.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:06.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:06.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:06.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:06.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:06.720 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:06 smithi017 ceph-mon[27571]: pgmap v736: 17 pgs: 17 active+clean; 2.0 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-17T16:18:06.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:18:06.744 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:18:06.775 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:18:06.775 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:18:06.779 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:06.785 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:06.801 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:06.802 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:18:06.803 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:18:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:06 smithi071 ceph-mon[30002]: pgmap v736: 17 pgs: 17 active+clean; 2.0 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-17T16:18:06.832 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:18:06.832 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:18:06.833 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:06.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:06.854 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:18:06.854 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:18:06.875 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:18:06.876 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:18:06.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:06 smithi173 ceph-mon[30530]: pgmap v736: 17 pgs: 17 active+clean; 2.0 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-17T16:18:06.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:18:06.898 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:18:06.920 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:18:06.920 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:18:06.934 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:06.941 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:18:06.941 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:18:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:06.956 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:06.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:06.963 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:18:06.963 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:18:06.986 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:18:06.987 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:18:07.008 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:18:07.008 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:18:07.058 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:07.079 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:07.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:07.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:07.205 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:07.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:07.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:07.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:07.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:07.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:07.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:07.434 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:07.459 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:07.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:07.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 37050 37034 89.0171 61.25 0.00795834 0.0112268 2024-09-17T16:18:07.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:07.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:07.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:07.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:07.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:07.712 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:07.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2558429429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:07.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:07.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2558429429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:07.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:07.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:07.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2558429429' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:07.946 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:07.971 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:07.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:08.073 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:08.100 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:08.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:08.202 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:08.225 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:08.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:08.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:08.354 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:08.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:08.374 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:08.375 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:08.456 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:08.488 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:08.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:08.500 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:08 smithi017 ceph-mon[27571]: pgmap v737: 17 pgs: 17 active+clean; 2.1 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-17T16:18:08.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 38378 38362 88.794 83 0.00521365 0.0112558 2024-09-17T16:18:08.589 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:08.618 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:08.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:08.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:08.624 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:08.722 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:08.752 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:08.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:08 smithi071 ceph-mon[30002]: pgmap v737: 17 pgs: 17 active+clean; 2.1 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-17T16:18:08.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:08.879 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:08.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:08.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:08 smithi173 ceph-mon[30530]: pgmap v737: 17 pgs: 17 active+clean; 2.1 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-17T16:18:08.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:09.017 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:09.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:09.119 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:09.143 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:09.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:09.248 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:09.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:09.272 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:09.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:09.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:09.374 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:18:09.406 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:09.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:09.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:09.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 39531 39515 88.1963 72.0625 0.00384227 0.0113299 2024-09-17T16:18:09.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:09.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:09.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:09.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:09.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:09.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:09.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1582078400' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:09.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:09 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:18:09.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:09 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:18:09.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1582078400' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:09.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:09 smithi017 ceph-mon[27571]: osdmap e319: 12 total, 12 up, 11 in 2024-09-17T16:18:09.761 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:09.783 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:09.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:09.797 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1582078400' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:09.797 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:09 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:18:09.797 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:09 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:18:09.797 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1582078400' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:09.797 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:09 smithi173 ceph-mon[30530]: osdmap e319: 12 total, 12 up, 11 in 2024-09-17T16:18:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1582078400' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:09 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:18:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:09 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:18:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1582078400' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:09 smithi071 ceph-mon[30002]: osdmap e319: 12 total, 12 up, 11 in 2024-09-17T16:18:09.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:09.893 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:18:09.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:18:09.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:09.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:09.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:10.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:10.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:10.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:10.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:10.171 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:10.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:10.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:10.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:10.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:10.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:10.405 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:10.423 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T16:18:10.434 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:10.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:10.493 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2414683349' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:18:10.493 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:10 smithi173 ceph-mon[30530]: pgmap v739: 17 pgs: 17 active+clean; 2.3 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-17T16:18:10.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 40919 40903 88.1462 86.75 0.0207941 0.0113393 2024-09-17T16:18:10.535 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:10.565 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:10.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:10.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2414683349' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:18:10.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:10 smithi017 ceph-mon[27571]: pgmap v739: 17 pgs: 17 active+clean; 2.3 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-17T16:18:10.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:10.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:10.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:10.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:10.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2414683349' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:18:10.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:10 smithi071 ceph-mon[30002]: pgmap v739: 17 pgs: 17 active+clean; 2.3 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-17T16:18:10.818 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:10.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:10.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:10.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:10.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:11.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:11.072 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:11.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:11.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:11.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:11.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:11.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:11.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:11.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:11.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:11.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:11.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-17T16:18:11.453 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:11.459 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:11.483 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:11.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:11.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 42040 42024 87.5432 70.0625 0.0048101 0.0114165 2024-09-17T16:18:11.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:11.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:11.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:11.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:11.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2414683349' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:18:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:11 smithi017 ceph-mon[27571]: osdmap e320: 12 total, 12 up, 11 in 2024-09-17T16:18:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:11.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2414683349' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:18:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:11 smithi071 ceph-mon[30002]: osdmap e320: 12 total, 12 up, 11 in 2024-09-17T16:18:11.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:11.869 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:11.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:11.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2414683349' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:18:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:11 smithi173 ceph-mon[30530]: osdmap e320: 12 total, 12 up, 11 in 2024-09-17T16:18:11.970 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:11.998 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:11.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:12.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:12.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:12.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:12.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:12.232 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:12.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:12.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:12.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:12.386 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:12.386 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:12.391 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:12.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:12.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:12.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:12.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:12.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 43873 43857 88.4145 114.562 0.0202106 0.0113047 2024-09-17T16:18:12.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:12.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:12.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:12.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:18:12.735 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:12 smithi017 ceph-mon[27571]: pgmap v741: 17 pgs: 17 active+clean; 2.5 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-17T16:18:12.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:12.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:12.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:12 smithi071 ceph-mon[30002]: pgmap v741: 17 pgs: 17 active+clean; 2.5 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-17T16:18:12.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:12.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:12 smithi173 ceph-mon[30530]: pgmap v741: 17 pgs: 17 active+clean; 2.5 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-17T16:18:12.905 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:12.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:13.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:13.029 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:13.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:13.131 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:13.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:18:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:13.162 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:13.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:13.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:13.292 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:13.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:13.394 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:13.422 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:13.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:13.497 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:13.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:13.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 45087 45071 88.0224 75.875 0.00367279 0.0113391 2024-09-17T16:18:13.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:13.557 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:13.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:13 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:18:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:13 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:18:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:13 smithi071 ceph-mon[30002]: osdmap e321: 12 total, 12 up, 11 in 2024-09-17T16:18:13.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:13 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:18:13.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:13 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:18:13.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:13 smithi017 ceph-mon[27571]: osdmap e321: 12 total, 12 up, 11 in 2024-09-17T16:18:13.634 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:13.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:13.673 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:13.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:13.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:13.781 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:13.807 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:13.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:13.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:13 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:18:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:13 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:18:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:13 smithi173 ceph-mon[30530]: osdmap e321: 12 total, 12 up, 11 in 2024-09-17T16:18:13.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:13.934 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:13.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:14.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:14.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:14.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:14.164 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:14.191 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:14.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:14.201 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:14.201 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:14.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:14.320 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:14.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:14.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:14.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:14.458 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:14.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:14.474 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:18:14.475 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:18:14.505 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:18:14.505 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:18:14.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 46272 46256 87.5992 74.0625 0.0223813 0.0114091 2024-09-17T16:18:14.531 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:18:14.531 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:18:14.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:14.569 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:18:14.570 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:18:14.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:14.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:14.604 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:18:14.604 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:18:14.632 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:18:14.632 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:18:14.659 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:18:14.660 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:18:14.685 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:18:14.685 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:18:14.689 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:14.712 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:18:14.713 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2839750799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:14.713 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:14 smithi017 ceph-mon[27571]: pgmap v743: 17 pgs: 17 active+clean; 2.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:18:14.713 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:18:14.716 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:14.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:14.740 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:18:14.740 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:18:14.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2839750799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:14.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:14 smithi071 ceph-mon[30002]: pgmap v743: 17 pgs: 17 active+clean; 2.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:18:14.766 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:18:14.766 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:18:14.796 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:18:14.796 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:18:14.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:14.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:14.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:14.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2839750799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:14 smithi173 ceph-mon[30530]: pgmap v743: 17 pgs: 17 active+clean; 2.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:18:14.945 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:14.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:14.988 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:14.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:15.089 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:15.112 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:15.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:15.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:15.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:15.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:15.338 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:15.362 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:15.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:15.452 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:18:15.452 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T16:18:15.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:15.489 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:15.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:15.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 47223 47207 86.7705 59.4375 0.00258493 0.0114608 2024-09-17T16:18:15.591 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:15.613 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:18:15.627 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:15.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:15.729 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:15.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/392579703' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:18:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1183626072' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:18:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:15 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:18:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:15.796 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:15.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/392579703' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:18:15.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1183626072' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:18:15.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:15 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:18:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/392579703' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:18:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1183626072' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:18:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:15 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:18:15.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:15.923 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:15.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:16.025 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:16.048 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:16.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:16.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:16.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:16.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:16.285 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:16.319 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:16.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:16.421 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:16.451 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:16.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:16.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 48582 48566 86.7178 84.9375 0.00384131 0.0115265 2024-09-17T16:18:16.553 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:16.588 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:16.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:16.691 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:16.692 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:16.718 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:16.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:16 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/392579703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:18:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1183626072' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:18:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:16 smithi017 ceph-mon[27571]: osdmap e322: 12 total, 12 up, 11 in 2024-09-17T16:18:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:16 smithi017 ceph-mon[27571]: pgmap v745: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-17T16:18:16.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:16 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/392579703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:18:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1183626072' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:18:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:16 smithi071 ceph-mon[30002]: osdmap e322: 12 total, 12 up, 11 in 2024-09-17T16:18:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:16 smithi071 ceph-mon[30002]: pgmap v745: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-17T16:18:16.820 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:16.845 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:16.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:16.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:16 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/392579703' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:18:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1183626072' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:18:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:16 smithi173 ceph-mon[30530]: osdmap e322: 12 total, 12 up, 11 in 2024-09-17T16:18:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:16 smithi173 ceph-mon[30530]: pgmap v745: 17 pgs: 17 active+clean; 2.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-17T16:18:16.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:16.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:16.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:16.973 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:17.073 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:17.108 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:17.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:17.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:17.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:17.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:17.283 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:18:17.283 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:18:17.283 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:18:17.283 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:18:17.283 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:18:17.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:17.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:17.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:17.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:17.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:17.500 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:17.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:17.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 49984 49968 86.7428 87.625 0.0165504 0.0115226 2024-09-17T16:18:17.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:17.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:18:17.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:17.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:17.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:17.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:17.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:17.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:17.877 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:17.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:17.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:18:17.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:18.018 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:18.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:18.120 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:18.160 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:18.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:18.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:18.307 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:18.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:18.333 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T16:18:18.339 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:18.408 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:18.446 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:18.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:18.469 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:18.525 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 15 51362 51347 86.7276 86.1875 0.0166592 0.0115243 2024-09-17T16:18:18.547 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:18.582 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:18.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:18.599 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:18:18.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:18:18.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:18:18.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:18:18.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:18:18.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:18:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:18:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:18:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:18:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:18:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:18:18.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:18.688 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:18:18.688 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:18:18.688 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:18:18.688 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:18:18.688 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:18:18.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:18.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:18.811 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:18.847 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:18.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:18.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:18.979 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:18.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:19.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:19.107 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:19.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:19.209 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:19.236 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:19.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:19.306 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:18:19.306 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:18:19.324 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:19.334 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:18:19.334 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:18:19.338 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:19.366 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:19.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:19.368 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:18:19.368 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:18:19.399 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:18:19.400 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:18:19.425 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:18:19.426 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:18:19.426 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:19 smithi017 ceph-mon[27571]: pgmap v746: 17 pgs: 17 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-17T16:18:19.426 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2961406239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:19.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:18:19.455 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:18:19.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:19.486 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:18:19.486 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:18:19.494 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:19.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:19.516 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:18:19.517 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:18:19.527 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 15 52690 52675 86.6294 83 0.0106473 0.0115386 2024-09-17T16:18:19.542 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:18:19.543 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:18:19.567 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:18:19.567 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:18:19.580 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:19.580 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:19.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:18:19.591 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:18:19.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:19.615 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:18:19.615 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:18:19.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:19.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:19.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:19 smithi173 ceph-mon[30530]: pgmap v746: 17 pgs: 17 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-17T16:18:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2961406239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:19.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:19.746 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:19.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:19.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:19 smithi071 ceph-mon[30002]: pgmap v746: 17 pgs: 17 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-17T16:18:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2961406239' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:19.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:19.872 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:19.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:19.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:19.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:20.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:20.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:20.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:20.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:20.195 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:20.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:20.253 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:20.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:20.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:18:20.355 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:20.387 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:20.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:20.489 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:20.514 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:20.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:20.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 53858 53842 86.2782 72.9375 0.00251766 0.0115827 2024-09-17T16:18:20.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:20 smithi071 ceph-mon[30002]: osdmap e323: 12 total, 12 up, 11 in 2024-09-17T16:18:20.606 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:20.616 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:20.640 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:20.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:20.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:20 smithi173 ceph-mon[30530]: osdmap e323: 12 total, 12 up, 11 in 2024-09-17T16:18:20.681 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:20 smithi017 ceph-mon[27571]: osdmap e323: 12 total, 12 up, 11 in 2024-09-17T16:18:20.741 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:20.772 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:20.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:20.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:20.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:20.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:21.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:21.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:21.025 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:21.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:21.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:21.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:21.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:21.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:21.283 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:21.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:21.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:21.385 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:21.390 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:18:21.414 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:21.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:21.516 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:21.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:18:21.524908+0000 min lat: 0.00106175 max lat: 1.04372 avg lat: 0.011625 2024-09-17T16:18:21.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:18:21.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 55034 55018 85.9586 73.5 0.00252048 0.011625 2024-09-17T16:18:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:21.543 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:21.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:21.645 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:21.668 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:21.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:21 smithi017 ceph-mon[27571]: pgmap v748: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-17T16:18:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:21 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:18:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:21 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:18:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:21 smithi017 ceph-mon[27571]: osdmap e324: 12 total, 12 up, 11 in 2024-09-17T16:18:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3630803434' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:21.742 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:21 smithi071 ceph-mon[30002]: pgmap v748: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-17T16:18:21.743 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:21 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:18:21.743 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:21 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:18:21.743 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:21 smithi071 ceph-mon[30002]: osdmap e324: 12 total, 12 up, 11 in 2024-09-17T16:18:21.743 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3630803434' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:21.770 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:21.795 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:21.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:21.858 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:21.858 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:21.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:21 smithi173 ceph-mon[30530]: pgmap v748: 17 pgs: 17 active+clean; 3.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-17T16:18:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:21 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:18:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:21 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:18:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:21 smithi173 ceph-mon[30530]: osdmap e324: 12 total, 12 up, 11 in 2024-09-17T16:18:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3630803434' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:21.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:21.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:21.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:22.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:22.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:22.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:22.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:22.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:22.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:22.281 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:22.310 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:22.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:22.399 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T16:18:22.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:22.446 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:22.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:22.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 56184 56168 85.615 71.875 0.00163753 0.0116609 2024-09-17T16:18:22.547 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:22.569 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:18:22.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:22.572 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:22.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:22.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:22.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:22.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:22.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:22 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:22.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3630803434' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:22.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:22 smithi017 ceph-mon[27571]: osdmap e325: 12 total, 12 up, 11 in 2024-09-17T16:18:22.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:22 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:18:22.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:22 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:18:22.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/133037960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:18:22.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:18:22.803 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:22 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3630803434' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:22 smithi071 ceph-mon[30002]: osdmap e325: 12 total, 12 up, 11 in 2024-09-17T16:18:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:22 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:18:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:22 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:18:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/133037960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:18:22.838 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:22.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:22.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:22.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:22 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3630803434' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:22 smithi173 ceph-mon[30530]: osdmap e325: 12 total, 12 up, 11 in 2024-09-17T16:18:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:22 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:18:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:22 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:18:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/133037960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:18:22.914 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:22.941 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:22.962 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:22.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:23.064 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:23.094 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:23.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:23.195 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:23.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T16:18:23.227 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:23.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:23.264 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:23.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:23.356 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:23.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:23.458 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:23.469 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:23.469 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:23.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:23.486 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:23.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:23.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:23 smithi017 ceph-mon[27571]: pgmap v751: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-17T16:18:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/133037960' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:18:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:23 smithi017 ceph-mon[27571]: osdmap e326: 12 total, 12 up, 11 in 2024-09-17T16:18:23.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 57325 57309 85.2743 71.3125 0.00354899 0.0117193 2024-09-17T16:18:23.587 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:23.612 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:23.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:23.714 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:23.738 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:23.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:23.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:23 smithi071 ceph-mon[30002]: pgmap v751: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-17T16:18:23.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/133037960' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:18:23.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:23 smithi071 ceph-mon[30002]: osdmap e326: 12 total, 12 up, 11 in 2024-09-17T16:18:23.840 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:23.862 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:23.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:23.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:23 smithi173 ceph-mon[30530]: pgmap v751: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-09-17T16:18:23.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/133037960' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:18:23.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:23 smithi173 ceph-mon[30530]: osdmap e326: 12 total, 12 up, 11 in 2024-09-17T16:18:23.948 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:18:23.948 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:18:23.960 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:23.963 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:23.982 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:18:23.982 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:18:23.992 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:23.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:24.010 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:18:24.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:18:24.037 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:18:24.037 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:18:24.067 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:18:24.067 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:18:24.094 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:24.097 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:18:24.097 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:18:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:24.123 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:24.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:24.124 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:18:24.124 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:18:24.149 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:18:24.149 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:18:24.172 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:18:24.172 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:18:24.197 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:18:24.197 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:18:24.222 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:18:24.222 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:18:24.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:24.245 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:18:24.246 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:18:24.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:24.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:24.295 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:18:24.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:24.376 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:24.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:24.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:24.502 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:24.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:24.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 58235 58219 84.6137 56.875 0.00983663 0.0118132 2024-09-17T16:18:24.604 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:24.626 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:24.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:24.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:24.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:24 smithi017 ceph-mon[27571]: pgmap v753: 17 pgs: 17 active+clean; 3.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-17T16:18:24.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/865668113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:24.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:24 smithi017 ceph-mon[27571]: osdmap e327: 12 total, 12 up, 11 in 2024-09-17T16:18:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:24.758 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:24.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:24 smithi071 ceph-mon[30002]: pgmap v753: 17 pgs: 17 active+clean; 3.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-17T16:18:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/865668113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:24 smithi071 ceph-mon[30002]: osdmap e327: 12 total, 12 up, 11 in 2024-09-17T16:18:24.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:24.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:24.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:24.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:24.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:24 smithi173 ceph-mon[30530]: pgmap v753: 17 pgs: 17 active+clean; 3.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-17T16:18:24.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/865668113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:24 smithi173 ceph-mon[30530]: osdmap e327: 12 total, 12 up, 11 in 2024-09-17T16:18:24.905 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:18:24.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:25.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:25.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:25.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:25.149 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:25.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:25.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:25.279 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:25.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:25.297 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:25.341 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:25.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:25.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:25.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:25.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:25.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 59451 59435 84.4178 76 0.00499707 0.0118378 2024-09-17T16:18:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:25.536 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:25.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:25.567 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T16:18:25.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:25.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:25.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:25.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4063496011' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:18:25.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:25.787 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:25.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:25.887 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4063496011' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:18:25.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:25.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4063496011' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:18:25.919 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:25.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:21741 2.3 2:c4bdb6e7:::benchmark_data_smithi173_55121_object21740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 419.26633346699998, 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55429691999999997, 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.933 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 21741 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662972+0000", 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.722e-06 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663024+0000", 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2175000000000001e-05 2024-09-17T16:18:25.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663036+0000", 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2131e-05 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663042+0000", 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9649999999999996e-06 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663084+0000", 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2354999999999997e-05 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663166+0000", 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1699999999999994e-05 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663249+0000", 2024-09-17T16:18:25.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2714999999999997e-05 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663804+0000", 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055499599999999998 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.664173+0000", 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000369181 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.938 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217193+0000", 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55302014600000005 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217234+0000", 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1149000000000002e-05 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217267+0000", 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2685000000000003e-05 2024-09-17T16:18:25.939 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:23782 2.2 2:41257b37:::benchmark_data_smithi173_55121_object23781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 417.24101518800001, 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51828190799999996, 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:25.940 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 23782 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:18:25.941 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688289+0000", 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7400000000000002e-07 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688376+0000", 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6744999999999997e-05 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688387+0000", 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0499000000000001e-05 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688393+0000", 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9689999999999997e-06 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688434+0000", 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0694e-05 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688511+0000", 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6656000000000006e-05 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688594+0000", 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2682999999999994e-05 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689092+0000", 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049850100000000004 2024-09-17T16:18:25.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689369+0000", 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027721100000000003 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206489+0000", 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51712006899999996 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:25.945 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206524+0000", 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4516e-05 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206570+0000", 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6490999999999999e-05 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:23782 2.2 e119/104 ondisk, result = 0)", 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 417.23992240299998, 2024-09-17T16:18:25.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51719464199999998, 2024-09-17T16:18:25.947 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.947 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.947 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:18:25.947 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:18:25.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:18:25.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689382+0000", 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689385+0000", 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3569999999999998e-06 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206448+0000", 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51706313699999995 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.949 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206453+0000", 2024-09-17T16:18:25.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4360000000000004e-06 2024-09-17T16:18:25.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.950 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206480+0000", 2024-09-17T16:18:25.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7770999999999999e-05 2024-09-17T16:18:25.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.950 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206486+0000", 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6779999999999997e-06 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206576+0000", 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9701000000000001e-05 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24369 2.3 2:c67d9130:::benchmark_data_smithi173_55121_object24368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 416.24179586700001, 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55710953100000005, 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24369 2024-09-17T16:18:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687509+0000", 2024-09-17T16:18:25.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1399999999999995e-07 2024-09-17T16:18:25.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687596+0000", 2024-09-17T16:18:25.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7288999999999997e-05 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687607+0000", 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1505000000000001e-05 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687613+0000", 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1440000000000002e-06 2024-09-17T16:18:25.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687655+0000", 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2240000000000002e-05 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687782+0000", 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000127043 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:18:25.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687864+0000", 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2064000000000006e-05 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688444+0000", 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058032600000000004 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688721+0000", 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000276734 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244551+0000", 2024-09-17T16:18:25.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55582960199999998 2024-09-17T16:18:25.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:25.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244588+0000", 2024-09-17T16:18:25.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7004e-05 2024-09-17T16:18:25.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244617+0000", 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9666e-05 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:24369 2.3 e120/104 ondisk, result = 0)", 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:18:25.959 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 416.24058867600002, 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55590799999999996, 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688716+0000", 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8999999999999996e-07 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688719+0000", 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5980000000000001e-06 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244478+0000", 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55575852299999995 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244483+0000", 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244541+0000", 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8377e-05 2024-09-17T16:18:25.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244547+0000", 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8980000000000001e-06 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244623+0000", 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5780999999999998e-05 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.963 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:29877 2.2 2:40729eb6:::benchmark_data_smithi173_55121_object29876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T16:18:25.964 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:18:25.964 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 395.72543429000001, 2024-09-17T16:18:25.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.874997425, 2024-09-17T16:18:25.964 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.964 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.964 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:25.964 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:25.964 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:25.964 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:25.964 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 29877 2024-09-17T16:18:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:18:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203870+0000", 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203910+0000", 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9477999999999997e-05 2024-09-17T16:18:25.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203918+0000", 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203922+0000", 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6369999999999998e-06 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203958+0000", 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5045000000000001e-05 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204026+0000", 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8693000000000001e-05 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204105+0000", 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8458e-05 2024-09-17T16:18:25.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204758+0000", 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00065323300000000003 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.205073+0000", 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031491100000000002 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078782+0000", 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87370871400000005 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078827+0000", 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5219999999999997e-05 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078867+0000", 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9953000000000002e-05 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:25.970 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:88652 2.2 2:45ffdfe0:::benchmark_data_smithi173_55121_object12163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 341.97254869599999, 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29260301500000002, 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 88652 2024-09-17T16:18:25.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956757+0000", 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7859999999999999e-06 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956762+0000", 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.31e-06 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956773+0000", 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0832999999999999e-05 2024-09-17T16:18:25.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956778+0000", 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2320000000000001e-06 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956829+0000", 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1124e-05 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956915+0000", 2024-09-17T16:18:25.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5541e-05 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.957003+0000", 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8332999999999996e-05 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.958148+0000", 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001144937 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.961994+0000", 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038457320000000001 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249267+0000", 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28727286400000002 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249320+0000", 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3016000000000002e-05 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249358+0000", 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8306999999999998e-05 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(2.3 158/155 [PushReplyOp(2:c6916317:::benchmark_data_smithi173_55121_object76243:head)])", 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 337.20625644699999, 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45253040700000002, 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.977 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723048+0000", 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4300000000000003e-07 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723052+0000", 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9310000000000001e-06 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175226+0000", 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45217462899999999 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.980 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.980 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175233+0000", 2024-09-17T16:18:25.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4099999999999996e-06 2024-09-17T16:18:25.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.980 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.980 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175322+0000", 2024-09-17T16:18:25.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9011000000000005e-05 2024-09-17T16:18:25.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175578+0000", 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025588299999999999 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:94077 2.16 2:6e055a89:::benchmark_data_smithi173_55121_object17588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 335.85083082400001, 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28469862699999998, 2024-09-17T16:18:25.981 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 94077 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078474+0000", 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.274e-06 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078479+0000", 2024-09-17T16:18:25.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6310000000000002e-06 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078490+0000", 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1799e-05 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078496+0000", 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7579999999999998e-06 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078540+0000", 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3686999999999997e-05 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078624+0000", 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4153000000000006e-05 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078694+0000", 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9533e-05 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.079805+0000", 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011117550000000001 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.085499+0000", 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0056938969999999998 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363107+0000", 2024-09-17T16:18:25.986 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27760825700000002 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363140+0000", 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2055999999999999e-05 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363171+0000", 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1826999999999999e-05 2024-09-17T16:18:25.987 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97425 2.3 2:c4562d4f:::benchmark_data_smithi173_55121_object20936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 332.08223247000001, 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29697053200000001, 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:25.988 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97425 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847072+0000", 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1399999999999997e-07 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847074+0000", 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7329999999999999e-06 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847082+0000", 2024-09-17T16:18:25.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6830000000000008e-06 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847088+0000", 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0499999999999997e-06 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848028+0000", 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093989799999999999 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848133+0000", 2024-09-17T16:18:25.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000104874 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848231+0000", 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8121999999999993e-05 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.849551+0000", 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00131985 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.992 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.993 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143633+0000", 2024-09-17T16:18:25.993 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29408256900000002 2024-09-17T16:18:25.993 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.993 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.993 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:25.993 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143940+0000", 2024-09-17T16:18:25.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030690399999999999 2024-09-17T16:18:25.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.994 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.994 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:25.994 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143994+0000", 2024-09-17T16:18:25.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3161999999999997e-05 2024-09-17T16:18:25.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.144042+0000", 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8072999999999999e-05 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:129838 2.2 2:4555d77c:::benchmark_data_smithi173_55121_object53349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:18:25.995 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 298.14277195900002, 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.288916852, 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 129838 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:25.996 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:18:25.997 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.997 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.997 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.997 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:25.997 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:18:25.997 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:25.997 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.997 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.997 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786533+0000", 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.728e-06 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786540+0000", 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4320000000000004e-06 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786556+0000", 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6302000000000001e-05 2024-09-17T16:18:25.998 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786563+0000", 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.934e-06 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786626+0000", 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2988000000000001e-05 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:18:25.999 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786633+0000", 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6150000000000002e-06 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069502+0000", 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.282869606 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069572+0000", 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0201000000000001e-05 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.000 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069639+0000", 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6323999999999997e-05 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075296+0000", 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0056576639999999997 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075309+0000", 2024-09-17T16:18:26.001 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2621e-05 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075356+0000", 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6653000000000002e-05 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075405+0000", 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9438000000000003e-05 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:26.002 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075448+0000", 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3346000000000001e-05 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:129850 2.16 2:6f12710d:::benchmark_data_smithi173_55121_object53361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 298.12695715000001, 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27163233199999998, 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:26.003 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 129850 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.004 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802348+0000", 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0549999999999998e-06 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:26.005 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802356+0000", 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6539999999999999e-06 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802372+0000", 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6314999999999999e-05 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802379+0000", 2024-09-17T16:18:26.006 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0419999999999997e-06 2024-09-17T16:18:26.007 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.007 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.007 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.007 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802428+0000", 2024-09-17T16:18:26.007 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8367e-05 2024-09-17T16:18:26.007 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.007 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802433+0000", 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6519999999999997e-06 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069491+0000", 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26705805599999999 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:26.008 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069572+0000", 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1174999999999994e-05 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069639+0000", 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6946000000000006e-05 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073713+0000", 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040741709999999997 2024-09-17T16:18:26.009 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.010 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.010 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.010 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073743+0000", 2024-09-17T16:18:26.010 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9740999999999999e-05 2024-09-17T16:18:26.010 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.010 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.010 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.010 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073843+0000", 2024-09-17T16:18:26.010 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100497 2024-09-17T16:18:26.010 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073930+0000", 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6835999999999993e-05 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073979+0000", 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8825000000000001e-05 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.011 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:150269 2.16 2:6d6b4de7:::benchmark_data_smithi173_55121_object73780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 275.34076987100002, 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29341146400000001, 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 150269 2024-09-17T16:18:26.012 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.013 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588535+0000", 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.751e-06 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588542+0000", 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7479999999999996e-06 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588558+0000", 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5885000000000001e-05 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.014 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588582+0000", 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4042000000000001e-05 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588646+0000", 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4103999999999995e-05 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588789+0000", 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000143094 2024-09-17T16:18:26.015 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588910+0000", 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120751 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.590898+0000", 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001988058 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.016 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.590948+0000", 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9959000000000001e-05 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881725+0000", 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29077654800000002 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881900+0000", 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017578599999999999 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.017 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881945+0000", 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4737999999999998e-05 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:152907 2.3 2:c65c8e4e:::benchmark_data_smithi173_55121_object76418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 272.13188380600002, 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87037030500000001, 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:26.018 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:26.019 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:26.019 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:26.019 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:18:26.019 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:26.019 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152907 2024-09-17T16:18:26.019 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.019 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:26.019 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.019 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:26.019 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4099999999999999e-07 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.020 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797423+0000", 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797430+0000", 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1119999999999998e-06 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797436+0000", 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6189999999999999e-06 2024-09-17T16:18:26.021 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797479+0000", 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3050999999999998e-05 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797597+0000", 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000117637 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797696+0000", 2024-09-17T16:18:26.022 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9079999999999993e-05 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.799730+0000", 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020345239999999998 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.801020+0000", 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012897270000000001 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.023 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667698+0000", 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.866677794 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667733+0000", 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4953999999999998e-05 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667790+0000", 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7265999999999997e-05 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.024 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(3.8 250/239 [PushOp(3:1a7a7bf1:::benchmark_data_smithi173_66272_object39562:head, version: 230'2428, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a7a7bf1:::benchmark_data_smithi173_66272_object39562:head@230'2428, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:45.409030+0000", 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 160.52027321099999, 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.263688647, 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:26.025 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409030+0000", 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409030+0000", 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409032+0000", 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.513e-06 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.026 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409173+0000", 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000140915 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671742+0000", 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26256918299999998 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671788+0000", 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5719000000000001e-05 2024-09-17T16:18:26.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671881+0000", 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3509000000000002e-05 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671890+0000", 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5290000000000008e-06 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.672719+0000", 2024-09-17T16:18:26.028 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00082927900000000004 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:81509 3.5 3:ac456fc5:::benchmark_data_smithi173_66272_object9553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:56.268230+0000", 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 149.661073999, 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27300864699999999, 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:26.029 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 81509 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268230+0000", 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268230+0000", 2024-09-17T16:18:26.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268231+0000", 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3260000000000001e-06 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268236+0000", 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8040000000000001e-06 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268248+0000", 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2014999999999999e-05 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268253+0000", 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4500000000000003e-06 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268296+0000", 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2879000000000001e-05 2024-09-17T16:18:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268370+0000", 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3892000000000003e-05 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268433+0000", 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2953999999999994e-05 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270351+0000", 2024-09-17T16:18:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00191843 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270909+0000", 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055809299999999996 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541148+0000", 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27023824299999999 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541188+0000", 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0006000000000003e-05 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541238+0000", 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0555e-05 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25036.0:81509 3.5 e255/208 ondisk, result = 0)", 2024-09-17T16:18:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:56.270755+0000", 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 149.65854815899999, 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27048895299999998, 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270755+0000", 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270755+0000", 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270756+0000", 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7499999999999998e-07 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270762+0000", 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9649999999999996e-06 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541053+0000", 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27029036699999998 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541078+0000", 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5367e-05 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541134+0000", 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6329000000000002e-05 2024-09-17T16:18:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541143+0000", 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3059999999999998e-06 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541244+0000", 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000101644 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDelete(3.9 e278,273 3:90341f46:::benchmark_data_smithi173_66272_object7671:head,254'4892)", 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 82.777822955999994, 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30541183, 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5000000000000001e-07 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151489+0000", 2024-09-17T16:18:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.52e-06 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456713+0000", 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.305224035 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456737+0000", 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.429e-05 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456817+0000", 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9598000000000006e-05 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456828+0000", 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0740000000000001e-05 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456892+0000", 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4796999999999996e-05 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:136335 3.5 3:aa93c047:::benchmark_data_smithi173_66272_object64379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:09.807143+0000", 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 76.122160827000002, 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37544051899999997, 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 136335 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807143+0000", 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807143+0000", 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807144+0000", 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6899999999999999e-06 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807151+0000", 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6880000000000002e-06 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807166+0000", 2024-09-17T16:18:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4883e-05 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807173+0000", 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1690000000000003e-06 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807229+0000", 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5337999999999998e-05 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807354+0000", 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000125007 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807468+0000", 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011404900000000001 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.811073+0000", 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036052879999999999 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.811379+0000", 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030657599999999997 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:10.182478+0000", 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37109837699999998 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:10.182538+0000", 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9840000000000003e-05 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:10.182583+0000", 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5614000000000001e-05 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:14455 4.1 4:8aec1aca:::benchmark_data_smithi173_77474_object14454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:49.531449+0000", 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 36.397854715999998, 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26431682000000001, 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14455 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531449+0000", 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531449+0000", 2024-09-17T16:18:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531451+0000", 2024-09-17T16:18:26.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6500000000000001e-06 2024-09-17T16:18:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531501+0000", 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0448999999999998e-05 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531516+0000", 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4687e-05 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531523+0000", 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2690000000000002e-06 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531592+0000", 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9431999999999995e-05 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531718+0000", 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000125897 2024-09-17T16:18:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531913+0000", 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019508500000000001 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.533462+0000", 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001549148 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.669390+0000", 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13592713300000001 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.795690+0000", 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.126300828 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.795733+0000", 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2750000000000002e-05 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:18:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.795766+0000", 2024-09-17T16:18:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2492e-05 2024-09-17T16:18:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:26.058 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:18:26.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:26.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:26.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:26.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:26.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:26.302 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:26.326 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:26.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:26.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:26.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:26.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:26.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:26.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 60421 60405 83.889 60.625 0.00957716 0.0119133 2024-09-17T16:18:26.554 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:26.579 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:26.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:26.586 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:18:26.607 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T16:18:26.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:26.718 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:26.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:26.795 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4063496011' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:18:26.796 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:26 smithi071 ceph-mon[30002]: osdmap e328: 12 total, 12 up, 11 in 2024-09-17T16:18:26.796 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:26 smithi071 ceph-mon[30002]: pgmap v756: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-17T16:18:26.796 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1164233646' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:18:26.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:26.839 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:18:26.853 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:26.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:26.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4063496011' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:18:26.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:26 smithi173 ceph-mon[30530]: osdmap e328: 12 total, 12 up, 11 in 2024-09-17T16:18:26.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:26 smithi173 ceph-mon[30530]: pgmap v756: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-17T16:18:26.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1164233646' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:18:26.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:26.981 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:26.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4063496011' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:18:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:26 smithi017 ceph-mon[27571]: osdmap e328: 12 total, 12 up, 11 in 2024-09-17T16:18:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:26 smithi017 ceph-mon[27571]: pgmap v756: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-17T16:18:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1164233646' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:18:27.061 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:18:27.083 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:27.131 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:27.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:27.232 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:27.233 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:27.266 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:27.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:27.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:27.394 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:27.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:27.496 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:27.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:27.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:27.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 61734 61718 83.8491 82.0625 0.00927997 0.0119215 2024-09-17T16:18:27.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:27.649 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:27 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:18:27.649 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:27 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:18:27.650 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:27 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:18:27.650 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:27 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:18:27.650 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:27 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:27.650 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1164233646' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:18:27.650 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:27 smithi071 ceph-mon[30002]: osdmap e329: 12 total, 12 up, 11 in 2024-09-17T16:18:27.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:27.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:27.669 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:27 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:18:27.669 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:27 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:18:27.669 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:27 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:18:27.669 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:27 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:18:27.669 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:27 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:27.669 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1164233646' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:18:27.670 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:27 smithi017 ceph-mon[27571]: osdmap e329: 12 total, 12 up, 11 in 2024-09-17T16:18:27.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:27.783 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:18:27.783 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:18:27.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.783 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:62046 4.5 4:a7872e2c:::benchmark_data_smithi173_77474_object62045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-17T16:18:27.783 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:18:27.776671+0000", 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0043245649999999998, 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0043548019999999996, 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 62046 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.776671+0000", 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.776671+0000", 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.776673+0000", 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5110000000000002e-06 2024-09-17T16:18:27.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.776718+0000", 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4688000000000003e-05 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.776738+0000", 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9831e-05 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:27.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.776753+0000", 2024-09-17T16:18:27.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4737e-05 2024-09-17T16:18:27.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:27.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.776831+0000", 2024-09-17T16:18:27.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8578999999999994e-05 2024-09-17T16:18:27.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:27.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.776958+0000", 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000126654 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.777169+0000", 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00021150300000000001 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.779484+0000", 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023150409999999999 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.779554+0000", 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0024000000000006e-05 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:62033 4.c e329/301 4:39dcd887:::benchmark_data_smithi173_77474_object62032:head v 329'3908, mlcod=329'3905)", 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:18:27.762654+0000", 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.018341653999999999, 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.018445702000000001, 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:18:27.789 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.762654+0000", 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.762654+0000", 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:18:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.762654+0000", 2024-09-17T16:18:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0200000000000001e-07 2024-09-17T16:18:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:18:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.762676+0000", 2024-09-17T16:18:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1747e-05 2024-09-17T16:18:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.780543+0000", 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.017867069999999999 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.780549+0000", 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1249999999999998e-06 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.780636+0000", 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6755999999999994e-05 2024-09-17T16:18:27.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:18:27.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:18:27.793 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:18:27.793 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:27.780676+0000", 2024-09-17T16:18:27.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9857000000000001e-05 2024-09-17T16:18:27.793 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:27.793 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:18:27.793 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:27.793 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:18:27.794 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:18:27.794 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 2 2024-09-17T16:18:27.794 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:18:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:27.795 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:27.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:27.803 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:27.832 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:27 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:18:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:27 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:18:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:27 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:18:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:27 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:18:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:27 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1164233646' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:18:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:27 smithi173 ceph-mon[30530]: osdmap e329: 12 total, 12 up, 11 in 2024-09-17T16:18:27.896 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:27.921 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:27.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:28.024 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:28.057 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:28.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:28.158 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:28.186 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:28.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:28.289 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:28.315 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:28.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:28.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:28.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:28.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:28.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:18:28.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 63017 63001 83.7711 80.1875 0.0192461 0.0119327 2024-09-17T16:18:28.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:28.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:28.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:28.676 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:28.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:28.722 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:28.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:28 smithi071 ceph-mon[30002]: pgmap v758: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-17T16:18:28.823 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:28.853 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:28.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:28.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:28 smithi173 ceph-mon[30530]: pgmap v758: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-17T16:18:28.913 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:28.936 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:28 smithi017 ceph-mon[27571]: pgmap v758: 17 pgs: 17 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-17T16:18:28.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:28.968 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-17T16:18:28.964+0000 7f27fb7fe640 0 --2- 172.21.15.17:0/3703123634 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f27fc1017d0 0x7f27fc196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T16:18:28.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:28.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:29.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:29.115 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:29.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:29.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:29.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:29.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:29.348 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:29.375 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:29.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:29.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:29.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:29.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:29.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 64519 64503 83.9814 93.875 0.00582364 0.0119008 2024-09-17T16:18:29.547 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:18:29.548 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:18:29.548 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:18:29.548 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:18:29.548 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:18:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T16:18:29.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:29.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:29.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:29.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:18:29.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:29.756 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:29.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/11330096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3703123634' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:18:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:29 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:18:29.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:29.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:29.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/11330096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3703123634' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:18:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:29 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:18:29.905 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:18:29.906 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:18:29.906 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/11330096' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:29.906 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3703123634' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:18:29.906 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:29 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:18:29.934 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:18:29.934 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:18:29.962 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:18:29.962 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:18:29.987 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:18:29.987 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:18:29.987 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:30.012 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:18:30.013 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:18:30.017 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:30.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:30.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:30.039 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:18:30.039 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:18:30.067 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:18:30.067 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:18:30.095 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:18:30.096 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:18:30.118 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:18:30.118 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:18:30.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:30.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:30.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:30.144 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:18:30.144 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:18:30.172 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:18:30.172 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:18:30.195 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:18:30.195 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:18:30.244 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:30.267 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:30.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:30.369 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:30.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:18:30.388 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T16:18:30.395 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:30.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:30.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:30.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:30.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:30.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 65788 65772 83.886 79.3125 0.00516592 0.011914 2024-09-17T16:18:30.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:30.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:30.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:30.728 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:30.753 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:30.795 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:30.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:30.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:30.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:30 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:18:30.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:30 smithi071 ceph-mon[30002]: osdmap e330: 12 total, 12 up, 11 in 2024-09-17T16:18:30.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:30 smithi071 ceph-mon[30002]: pgmap v760: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-17T16:18:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:30 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:18:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:30 smithi173 ceph-mon[30530]: osdmap e330: 12 total, 12 up, 11 in 2024-09-17T16:18:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:30 smithi173 ceph-mon[30530]: pgmap v760: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-17T16:18:30.896 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:30.918 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:30.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:30.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:30 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:18:30.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:30 smithi017 ceph-mon[27571]: osdmap e330: 12 total, 12 up, 11 in 2024-09-17T16:18:30.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:30 smithi017 ceph-mon[27571]: pgmap v760: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-17T16:18:31.019 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:31.041 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:31.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:31.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:31.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:31.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:31.233 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:31.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:31.279 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:31.302 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:31.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:31.404 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:31.429 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:31.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:31.526 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 66852 66836 83.5382 66.5 0.00316634 0.011963 2024-09-17T16:18:31.530 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:31.555 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:31.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:31.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:31.619 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:31.619 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:31.657 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:31.687 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:31.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:31.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:31.814 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:31.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:31.916 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:31.941 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:31.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:32.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:32.073 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:32.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:32.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:32.205 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:32.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:32.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:32.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:32.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:32.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:18:32.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:32.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:32.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:32.527 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 68119 68103 83.4528 79.1875 0.0248305 0.0119778 2024-09-17T16:18:32.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:32.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:32.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:32.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:32.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:32.710 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:32.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:32.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:32.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:32.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:32.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:32.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:32.888 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:18:32.888 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T16:18:32.953 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:32.978 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:32.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:33.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:33.109 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:33.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:33.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:32 smithi017 ceph-mon[27571]: pgmap v761: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-17T16:18:33.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3094161748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:18:33.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/402865807' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:33.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:32 smithi173 ceph-mon[30530]: pgmap v761: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-17T16:18:33.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3094161748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:18:33.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/402865807' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:33.168 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:18:33.209 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:33.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:33.237 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:33.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:32 smithi071 ceph-mon[30002]: pgmap v761: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-17T16:18:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3094161748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:18:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/402865807' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:33.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:33.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:33.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:33.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:33.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:33.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:33.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:33.527 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 69193 69177 83.1386 67.125 0.00439677 0.0120218 2024-09-17T16:18:33.557 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:33.557 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:33.594 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:33.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:33.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:33.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:33.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:33.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:33.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:33.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:33.976 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:33 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:18:33.977 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:33 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:18:33.977 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3094161748' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:18:33.977 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/402865807' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:33.977 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:33 smithi071 ceph-mon[30002]: osdmap e331: 12 total, 12 up, 11 in 2024-09-17T16:18:33.987 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:34.013 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:34.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:34.115 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:33 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:18:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:33 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:18:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3094161748' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:18:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/402865807' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:33 smithi173 ceph-mon[30530]: osdmap e331: 12 total, 12 up, 11 in 2024-09-17T16:18:34.149 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:34.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:34.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:33 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:18:34.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:33 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:18:34.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3094161748' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:18:34.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/402865807' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:34.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:33 smithi017 ceph-mon[27571]: osdmap e331: 12 total, 12 up, 11 in 2024-09-17T16:18:34.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-17T16:18:34.251 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:34.284 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:34.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:34.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:18:34.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:34.449 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:34.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:34.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:34.527 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 70514 70498 83.1276 82.5625 0.00669937 0.012023 2024-09-17T16:18:34.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:34.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:34.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:34.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:34.683 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:34.694 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:34.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:34.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:34.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:34.715 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:34.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:34.830 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:34.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:34.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:34.960 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:34.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:35.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:35.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:35.089 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:35.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:35.107 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:34 smithi173 ceph-mon[30530]: pgmap v763: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:18:35.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:34 smithi017 ceph-mon[27571]: pgmap v763: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:18:35.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:35.222 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:35.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:35.238 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:35.239 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:35.280 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:18:35.281 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:18:35.306 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:18:35.306 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:18:35.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:34 smithi071 ceph-mon[30002]: pgmap v763: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:18:35.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:35.332 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:18:35.332 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:18:35.350 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:35.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:35.357 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:18:35.357 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:18:35.382 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:18:35.383 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:18:35.413 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:18:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:18:35.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:18:35.438 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:18:35.453 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:35.465 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:18:35.465 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:18:35.476 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:35.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:35.488 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:18:35.488 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:18:35.516 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:18:35.516 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:18:35.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 71425 71409 82.6425 56.9375 0.00850211 0.012095 2024-09-17T16:18:35.546 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:18:35.547 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:18:35.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:35.590 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:18:35.590 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:18:35.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:35.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:35.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:35.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:35.736 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:35.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:35.839 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:35.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:35.866 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:35.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3062307872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:35 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:18:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:35 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:18:35.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:36.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:36.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:36.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:18:36.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:36.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:36.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:36.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:36.263 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:36.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:36.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:36.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3062307872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:36.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:35 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:18:36.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:35 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:18:36.366 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:36.391 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:36.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:36.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:36.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3062307872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:36.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:35 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:18:36.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:35 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:18:36.396 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:36.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:36.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:36.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:36.527 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 72805 72789 82.708 86.25 0.00330351 0.0120853 2024-09-17T16:18:36.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:36.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:36.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:36.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:36.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:18:36.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:36.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:36.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:36.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.8 to 1 (10000) 2024-09-17T16:18:36.901 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:36.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:37.003 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:37.028 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:37.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:37.130 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:37.143 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:18:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:18:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:13492 2.15 2:a809b404:::benchmark_data_smithi173_55121_object13491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:12.334855+0000", 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 444.67800419100001, 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.019215271, 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 13492 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334855+0000", 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334855+0000", 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334856+0000", 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6700000000000002e-07 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334879+0000", 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3359000000000001e-05 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334901+0000", 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2294999999999999e-05 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.334911+0000", 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3649999999999993e-06 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.343520+0000", 2024-09-17T16:18:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0086088640000000008 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.343525+0000", 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9930000000000003e-06 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.353911+0000", 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0103866720000001 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:13.354070+0000", 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000158856 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24380 2.f 2:ff1cebfd:::benchmark_data_smithi173_55121_object24379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.697276+0000", 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 427.31558303000003, 2024-09-17T16:18:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39030066800000002, 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24380 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697276+0000", 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697276+0000", 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697277+0000", 2024-09-17T16:18:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.13e-06 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697302+0000", 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4533999999999999e-05 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697316+0000", 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4592e-05 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697324+0000", 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9109999999999993e-06 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697380+0000", 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6162999999999997e-05 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697512+0000", 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013168899999999999 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.697679+0000", 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016729599999999999 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.698584+0000", 2024-09-17T16:18:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090433600000000005 2024-09-17T16:18:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.762716+0000", 2024-09-17T16:18:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.064132583000000007 2024-09-17T16:18:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087510+0000", 2024-09-17T16:18:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32479419700000001 2024-09-17T16:18:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087549+0000", 2024-09-17T16:18:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8278999999999997e-05 2024-09-17T16:18:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.087577+0000", 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7957999999999999e-05 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24385 2.b 2:d672fe3e:::benchmark_data_smithi173_55121_object24384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.704153+0000", 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 427.30870586700001, 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53404779700000005, 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24385 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704153+0000", 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704153+0000", 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704154+0000", 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704177+0000", 2024-09-17T16:18:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2127e-05 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704189+0000", 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2241e-05 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704195+0000", 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4280000000000001e-06 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704239+0000", 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3775e-05 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704326+0000", 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7198000000000001e-05 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.704414+0000", 2024-09-17T16:18:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7652999999999996e-05 2024-09-17T16:18:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705316+0000", 2024-09-17T16:18:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090249000000000004 2024-09-17T16:18:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705635+0000", 2024-09-17T16:18:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031902200000000002 2024-09-17T16:18:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.238124+0000", 2024-09-17T16:18:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53248838600000004 2024-09-17T16:18:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.238158+0000", 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4789000000000002e-05 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.238201+0000", 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2468000000000001e-05 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24386 2.10 2:0afcc277:::benchmark_data_smithi173_55121_object24385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.705747+0000", 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 427.307111819, 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43386687200000001, 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24386 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705747+0000", 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705747+0000", 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705748+0000", 2024-09-17T16:18:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5099999999999999e-07 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705770+0000", 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1563999999999999e-05 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705779+0000", 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3249999999999997e-06 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705784+0000", 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.203e-06 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705832+0000", 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8180000000000003e-05 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.705930+0000", 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7617999999999999e-05 2024-09-17T16:18:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706020+0000", 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9890999999999997e-05 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706544+0000", 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00052466899999999996 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.706882+0000", 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000337833 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139550+0000", 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43266755299999998 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139585+0000", 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5136999999999999e-05 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.139614+0000", 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9147999999999999e-05 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24388 2.f 2:f7f1158d:::benchmark_data_smithi173_55121_object24387:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.761151+0000", 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 427.25170803399999, 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37970383499999999, 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24388 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761151+0000", 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761151+0000", 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761152+0000", 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4619999999999999e-06 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761176+0000", 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3319000000000002e-05 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761190+0000", 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3916e-05 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761198+0000", 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1280000000000008e-06 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761269+0000", 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0862e-05 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761423+0000", 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015397499999999999 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.761596+0000", 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017338799999999999 2024-09-17T16:18:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.762762+0000", 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001166325 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767453+0000", 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046909999999999999 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.140795+0000", 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37334170999999999 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.140830+0000", 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4625999999999997e-05 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.140855+0000", 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5123999999999999e-05 2024-09-17T16:18:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:24391 2.5 2:a6a580fc:::benchmark_data_smithi173_55121_object24390:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.766124+0000", 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 427.24673524299999, 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36991581699999998, 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 24391 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766124+0000", 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766124+0000", 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766125+0000", 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6300000000000004e-07 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766146+0000", 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1365e-05 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766157+0000", 2024-09-17T16:18:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1294000000000001e-05 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766163+0000", 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3170000000000001e-06 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766215+0000", 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2367999999999997e-05 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766306+0000", 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0994000000000003e-05 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766395+0000", 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9352999999999996e-05 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.766965+0000", 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056975199999999998 2024-09-17T16:18:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767311+0000", 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000345621 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135856+0000", 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36854523 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135970+0000", 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114548 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.136040+0000", 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9111999999999998e-05 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:24391 2.5 e120/104 ondisk, result = 0)", 2024-09-17T16:18:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 427.24546638100003, 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36866083100000002, 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767393+0000", 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5199999999999996e-07 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:29.767397+0000", 2024-09-17T16:18:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.27e-06 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135721+0000", 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36832436600000001 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135734+0000", 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.323e-05 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135830+0000", 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5587999999999994e-05 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.135843+0000", 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3699999999999999e-05 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.136053+0000", 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000210025 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25180 2.10 2:092fb28c:::benchmark_data_smithi173_55121_object25179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.740770+0000", 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 426.272088999, 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48461674700000001, 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25180 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740770+0000", 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740770+0000", 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740772+0000", 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5969999999999999e-06 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740793+0000", 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1379999999999999e-05 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740807+0000", 2024-09-17T16:18:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4222e-05 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740814+0000", 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7830000000000001e-06 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740877+0000", 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2996999999999993e-05 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.740984+0000", 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106656 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.741069+0000", 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5752000000000002e-05 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.741666+0000", 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059629500000000005 2024-09-17T16:18:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.742062+0000", 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000396301 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225328+0000", 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48326640700000001 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225362+0000", 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3317e-05 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225387+0000", 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5040000000000001e-05 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25215 2.10 2:0e4d5290:::benchmark_data_smithi173_55121_object25214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.775331+0000", 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 426.237528285, 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47406391599999997, 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25215 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775331+0000", 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775331+0000", 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775332+0000", 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9300000000000006e-07 2024-09-17T16:18:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775355+0000", 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3303e-05 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775369+0000", 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3896999999999999e-05 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775377+0000", 2024-09-17T16:18:37.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1699999999999997e-06 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775446+0000", 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8745000000000007e-05 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775561+0000", 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115616 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:18:37.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.775705+0000", 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000143151 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776312+0000", 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00060773299999999995 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776660+0000", 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034724700000000002 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.249329+0000", 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47266967799999998 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.249367+0000", 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7233000000000001e-05 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.249395+0000", 2024-09-17T16:18:37.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.815e-05 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:25180 2.10 e121/111 ondisk, result = 0)", 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 426.23638651499999, 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44891908400000002, 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.203 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776473+0000", 2024-09-17T16:18:37.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3200000000000004e-07 2024-09-17T16:18:37.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.776477+0000", 2024-09-17T16:18:37.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6660000000000001e-06 2024-09-17T16:18:37.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225282+0000", 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44880505700000001 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225287+0000", 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8659999999999997e-06 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225320+0000", 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3055000000000001e-05 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225324+0000", 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3599999999999998e-06 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.225392+0000", 2024-09-17T16:18:37.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7348000000000006e-05 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:26993 2.10 2:0b00f2f4:::benchmark_data_smithi173_55121_object26992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.983151+0000", 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 424.02970760099998, 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51500823299999998, 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.208 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 26993 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983151+0000", 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983151+0000", 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983153+0000", 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.111e-06 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983170+0000", 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7805e-05 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983182+0000", 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1391e-05 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983187+0000", 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4809999999999997e-06 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983229+0000", 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1817999999999998e-05 2024-09-17T16:18:37.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983316+0000", 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6978999999999995e-05 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983399+0000", 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2720999999999996e-05 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.983972+0000", 2024-09-17T16:18:37.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057330599999999999 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.984444+0000", 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047245000000000002 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.498086+0000", 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51364169000000004 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.498123+0000", 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6358e-05 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.498160+0000", 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7122999999999997e-05 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:92960 2.b 2:d44e9bf2:::benchmark_data_smithi173_55121_object16471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.636820+0000", 2024-09-17T16:18:37.214 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 348.376038956, 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63467018600000003, 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 92960 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636820+0000", 2024-09-17T16:18:37.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636820+0000", 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636822+0000", 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1969999999999999e-06 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636830+0000", 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4490000000000002e-06 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636849+0000", 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9029999999999999e-05 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636859+0000", 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0183999999999999e-05 2024-09-17T16:18:37.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636929+0000", 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0017000000000004e-05 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.637101+0000", 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017157500000000001 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.637326+0000", 2024-09-17T16:18:37.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022522300000000001 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640616+0000", 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032902130000000002 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004722+0000", 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36410627899999998 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269780+0000", 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.265057773 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269830+0000", 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9772999999999999e-05 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.271490+0000", 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001660473 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.220 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:92960 2.b e158/155 ondisk, result = 0)", 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.640613+0000", 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 348.37224643500002, 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36412291800000002, 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640613+0000", 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640613+0000", 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640614+0000", 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2500000000000004e-07 2024-09-17T16:18:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640619+0000", 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5350000000000004e-06 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004492+0000", 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36387301399999999 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004506+0000", 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4403000000000001e-05 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004662+0000", 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015578400000000001 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004694+0000", 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1482000000000001e-05 2024-09-17T16:18:37.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004736+0000", 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1774999999999999e-05 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:92960 2.b e158/155 ondisk, result = 0)", 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.640951+0000", 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 348.371907566, 2024-09-17T16:18:37.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.630555843, 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640951+0000", 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640951+0000", 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640952+0000", 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.54e-07 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640960+0000", 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9570000000000002e-06 2024-09-17T16:18:37.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.266220+0000", 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62525987900000002 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.266226+0000", 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9800000000000003e-06 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269764+0000", 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035381520000000001 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269773+0000", 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5450000000000003e-06 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.271507+0000", 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001734576 2024-09-17T16:18:37.229 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:135561 2.6 e191/104 2:612899dc:::benchmark_data_smithi173_55121_object59072:head v 191'4677, mlcod=191'4675)", 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:35.921733+0000", 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 301.09112604699999, 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36610857200000002, 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921733+0000", 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921733+0000", 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921734+0000", 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:18:37.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921739+0000", 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2630000000000003e-06 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.285858+0000", 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.364118357 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.285878+0000", 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0152999999999998e-05 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.286066+0000", 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000188016 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.286106+0000", 2024-09-17T16:18:37.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0309999999999999e-05 2024-09-17T16:18:37.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.287803+0000", 2024-09-17T16:18:37.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016966290000000001 2024-09-17T16:18:37.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.287842+0000", 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8945000000000001e-05 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(3.9 240/239 [PushReplyOp(3:9e32e3c9:::benchmark_data_smithi173_66272_object1144:head)])", 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 201.933321487, 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90253544399999996, 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.235 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6799999999999999e-07 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079543+0000", 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6809999999999997e-06 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979238+0000", 2024-09-17T16:18:37.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89969455399999998 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979247+0000", 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5659999999999996e-06 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979307+0000", 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9338e-05 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.982073+0000", 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027664370000000001 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124278 3.9 3.d65c03a9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.930793+0000", 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 108.08206649900001, 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42595735800000001, 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.239 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124278 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930793+0000", 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930793+0000", 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930794+0000", 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4890000000000001e-06 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930801+0000", 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5699999999999998e-06 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930816+0000", 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4953000000000001e-05 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930823+0000", 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6279999999999996e-06 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930874+0000", 2024-09-17T16:18:37.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0333000000000001e-05 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930880+0000", 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8090000000000004e-06 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.356716+0000", 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42583603799999997 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.356750+0000", 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3538000000000003e-05 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124282 3.9 3.c1fc19b9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.955157+0000", 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 108.057701882, 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.402016861, 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.244 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124282 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955157+0000", 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955157+0000", 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955158+0000", 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.274e-06 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955164+0000", 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9240000000000004e-06 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955175+0000", 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0222e-05 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955181+0000", 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1709999999999999e-06 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955232+0000", 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0927999999999998e-05 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955240+0000", 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4039999999999995e-06 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.357130+0000", 2024-09-17T16:18:37.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40189000200000002 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.357174+0000", 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3936000000000001e-05 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124278 3.9 3:95c03a6b:::benchmark_data_smithi173_66272_object52322:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:50.410673+0000", 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 106.60218591, 2024-09-17T16:18:37.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47593780800000002, 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124278 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410673+0000", 2024-09-17T16:18:37.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410673+0000", 2024-09-17T16:18:37.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410675+0000", 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6199999999999999e-06 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410689+0000", 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4664000000000001e-05 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410704+0000", 2024-09-17T16:18:37.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4174e-05 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410714+0000", 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0659000000000001e-05 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411252+0000", 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053726700000000002 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:18:37.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411263+0000", 2024-09-17T16:18:37.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1297999999999999e-05 2024-09-17T16:18:37.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.870528+0000", 2024-09-17T16:18:37.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45926520500000001 2024-09-17T16:18:37.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.870665+0000", 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013726100000000001 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7,11", 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.870758+0000", 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2767000000000001e-05 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875501+0000", 2024-09-17T16:18:37.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047434310000000002 2024-09-17T16:18:37.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875552+0000", 2024-09-17T16:18:37.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0096000000000003e-05 2024-09-17T16:18:37.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875561+0000", 2024-09-17T16:18:37.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0690000000000008e-06 2024-09-17T16:18:37.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.885797+0000", 2024-09-17T16:18:37.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010236413999999999 2024-09-17T16:18:37.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.885869+0000", 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2149000000000003e-05 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.886611+0000", 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074173399999999997 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124282 3.9 3:9d983f83:::benchmark_data_smithi173_66272_object52326:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:50.410739+0000", 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 106.602119563, 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47645007299999997, 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124282 2024-09-17T16:18:37.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410739+0000", 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410739+0000", 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410740+0000", 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1099999999999995e-07 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410746+0000", 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0170000000000002e-06 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410754+0000", 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6499999999999996e-06 2024-09-17T16:18:37.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410760+0000", 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5309999999999998e-06 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411348+0000", 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00058806500000000003 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:18:37.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411359+0000", 2024-09-17T16:18:37.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0681e-05 2024-09-17T16:18:37.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:37.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875583+0000", 2024-09-17T16:18:37.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46422414299999998 2024-09-17T16:18:37.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:37.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875678+0000", 2024-09-17T16:18:37.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4757000000000001e-05 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7,11", 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875760+0000", 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2109999999999998e-05 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.878759+0000", 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029989209999999999 2024-09-17T16:18:37.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.879450+0000", 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069113699999999998 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.879505+0000", 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4765999999999997e-05 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.886692+0000", 2024-09-17T16:18:37.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0071871299999999999 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.886735+0000", 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3312999999999997e-05 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.887190+0000", 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045414099999999997 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:37.268 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:37.268 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:36 smithi173 ceph-mon[30530]: pgmap v764: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-17T16:18:37.268 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:36 smithi173 ceph-mon[30530]: osdmap e332: 12 total, 12 up, 11 in 2024-09-17T16:18:37.268 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/116023042' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:18:37.268 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:18:37.269 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:36 smithi017 ceph-mon[27571]: pgmap v764: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-17T16:18:37.269 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:36 smithi017 ceph-mon[27571]: osdmap e332: 12 total, 12 up, 11 in 2024-09-17T16:18:37.269 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/116023042' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:18:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:37.270 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:37.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:36 smithi071 ceph-mon[30002]: pgmap v764: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-17T16:18:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:36 smithi071 ceph-mon[30002]: osdmap e332: 12 total, 12 up, 11 in 2024-09-17T16:18:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/116023042' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:18:37.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:37.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:37.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:37.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:37.527 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 74033 74017 82.6015 76.75 0.00448614 0.0121014 2024-09-17T16:18:37.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:37.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:37.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:37.663 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:37.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:37.755 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-17T16:18:37.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:37.791 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:37.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:37.824 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:37.893 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:37.918 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:37.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:37.984 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/116023042' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:18:37.984 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:37 smithi017 ceph-mon[27571]: osdmap e333: 12 total, 12 up, 11 in 2024-09-17T16:18:38.021 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:38.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:38.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:38.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:38.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/116023042' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:18:38.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:37 smithi173 ceph-mon[30530]: osdmap e333: 12 total, 12 up, 11 in 2024-09-17T16:18:38.149 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:38.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:38.178 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:38.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:38.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:38.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:38.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:38.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:38.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/116023042' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:18:38.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:37 smithi071 ceph-mon[30002]: osdmap e333: 12 total, 12 up, 11 in 2024-09-17T16:18:38.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:38.441 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:38.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:38.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:38.467 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:38.527 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 75201 75185 82.433 73 0.0346944 0.0121257 2024-09-17T16:18:38.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:38.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:38.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:38.643 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:18:38.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:38.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:38.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:38.799 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:38.831 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:38.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:38.920 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:18:38.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:38.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:38.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:38.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:39.056 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:39.083 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:39.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:39.186 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:39.210 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:39.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:38 smithi017 ceph-mon[27571]: pgmap v767: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 61 MiB/s wr, 983 op/s 2024-09-17T16:18:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/680359732' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:18:39.312 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:39.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:38 smithi071 ceph-mon[30002]: pgmap v767: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 61 MiB/s wr, 983 op/s 2024-09-17T16:18:39.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/680359732' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:18:39.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:39.339 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:39.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:38 smithi173 ceph-mon[30530]: pgmap v767: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 61 MiB/s wr, 983 op/s 2024-09-17T16:18:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/680359732' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:18:39.440 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:39.463 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:39.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:39.527 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 76035 76019 81.9102 52.125 0.00265693 0.0121882 2024-09-17T16:18:39.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:39.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:18:39.586 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:39.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:39.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:39.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:39.711 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:39.812 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:39.837 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:39.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:39.847 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:39.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:39.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:39.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:39.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:40.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:40.085 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:40.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:40.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:40.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:39 smithi173 ceph-mon[30530]: 4.7 scrub starts 2024-09-17T16:18:40.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:39 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:40.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/680359732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:18:40.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:39 smithi173 ceph-mon[30530]: osdmap e334: 12 total, 12 up, 11 in 2024-09-17T16:18:40.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1468305887' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:18:40.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:39 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:18:40.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:39 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:18:40.115 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:39 smithi173 ceph-mon[30530]: osdmap e335: 12 total, 12 up, 11 in 2024-09-17T16:18:40.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:40.210 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:40.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:40.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:39 smithi017 ceph-mon[27571]: 4.7 scrub starts 2024-09-17T16:18:40.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:39 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:40.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/680359732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:18:40.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:39 smithi017 ceph-mon[27571]: osdmap e334: 12 total, 12 up, 11 in 2024-09-17T16:18:40.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1468305887' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:18:40.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:39 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:18:40.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:39 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:18:40.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:39 smithi017 ceph-mon[27571]: osdmap e335: 12 total, 12 up, 11 in 2024-09-17T16:18:40.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:39 smithi071 ceph-mon[30002]: 4.7 scrub starts 2024-09-17T16:18:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:39 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/680359732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:18:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:39 smithi071 ceph-mon[30002]: osdmap e334: 12 total, 12 up, 11 in 2024-09-17T16:18:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1468305887' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:18:40.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:39 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:18:40.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:39 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:18:40.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:39 smithi071 ceph-mon[30002]: osdmap e335: 12 total, 12 up, 11 in 2024-09-17T16:18:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:40.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:40.335 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:18:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:18:40.335 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:18:40.335 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:18:40.406 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.9 to 1 (10000) 2024-09-17T16:18:40.435 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:40.458 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:40.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:40.527 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 76094 76078 80.5844 3.6875 0.00290595 0.0122266 2024-09-17T16:18:40.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:40.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:40.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:40.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:40.707 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:40.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:40.809 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:40.831 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:40.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:40.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:40.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:40.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:41.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-17T16:18:41.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:41.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:41.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:41.088 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:41.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:18:41.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:41.204 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:41.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:41.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:41.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:41.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:41.329 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:41.377 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:41.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:41.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:41.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:41.527 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:18:41.526670+0000 min lat: 0.00106175 max lat: 1.04372 avg lat: 0.0122278 2024-09-17T16:18:41.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:18:41.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 76097 76081 79.2444 0.1875 0.00314354 0.0122278 2024-09-17T16:18:41.553 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:41.576 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:41.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:41.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:41 smithi017 ceph-mon[27571]: pgmap v770: 17 pgs: 1 peering, 16 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 61 MiB/s wr, 979 op/s 2024-09-17T16:18:41.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:41 smithi017 ceph-mon[27571]: osdmap e336: 12 total, 12 up, 11 in 2024-09-17T16:18:41.637 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:18:41.637 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:18:41.666 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:41.667 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:18:41.667 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:18:41.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:41.682 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:41 smithi173 ceph-mon[30530]: pgmap v770: 17 pgs: 1 peering, 16 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 61 MiB/s wr, 979 op/s 2024-09-17T16:18:41.683 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:41 smithi173 ceph-mon[30530]: osdmap e336: 12 total, 12 up, 11 in 2024-09-17T16:18:41.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:18:41.694 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:18:41.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:41.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:41.717 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:18:41.717 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:18:41.739 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:18:41.739 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:18:41.763 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:18:41.763 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:18:41.785 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:18:41.785 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:18:41.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:41.806 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:18:41.807 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:18:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:41 smithi071 ceph-mon[30002]: pgmap v770: 17 pgs: 1 peering, 16 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 61 MiB/s wr, 979 op/s 2024-09-17T16:18:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:41 smithi071 ceph-mon[30002]: osdmap e336: 12 total, 12 up, 11 in 2024-09-17T16:18:41.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:41.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:41.831 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:18:41.831 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:18:41.854 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:18:41.854 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:18:41.876 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:18:41.876 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:18:41.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:18:41.898 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:18:41.909 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:41.909 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:41.930 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:41.954 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:41.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:41.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:42.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:42.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:42.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:42.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:42.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:42.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:42.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:18:42.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:42.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:42.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:42.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:42.465 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:42.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:42.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 76586 76570 78.4463 30.5625 0.0206741 0.0127396 2024-09-17T16:18:42.566 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:42.591 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:42.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:42.627 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:42 smithi017 ceph-mon[27571]: osdmap e337: 12 total, 12 up, 11 in 2024-09-17T16:18:42.628 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3279254066' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:42.628 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:42 smithi017 ceph-mon[27571]: pgmap v773: 17 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 15 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s; 4670/227598 objects degraded (2.052%) 2024-09-17T16:18:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:42 smithi173 ceph-mon[30530]: osdmap e337: 12 total, 12 up, 11 in 2024-09-17T16:18:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3279254066' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:42 smithi173 ceph-mon[30530]: pgmap v773: 17 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 15 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s; 4670/227598 objects degraded (2.052%) 2024-09-17T16:18:42.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:42.715 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:42.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:42.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:42 smithi071 ceph-mon[30002]: osdmap e337: 12 total, 12 up, 11 in 2024-09-17T16:18:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3279254066' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:42 smithi071 ceph-mon[30002]: pgmap v773: 17 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 15 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s; 4670/227598 objects degraded (2.052%) 2024-09-17T16:18:42.817 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:42.840 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:42.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:42.943 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:42.968 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:42.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:42.982 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:43.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:43.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:43.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:43.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:43.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:43.223 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:43.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:43.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:43.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:18:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:43.357 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:43.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:43.450 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.10 to 1 (10000) 2024-09-17T16:18:43.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:43.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:43.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:43.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:43.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 77082 77066 77.681 31 0.00533711 0.012868 2024-09-17T16:18:43.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:43.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:43.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:43 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 4670/227598 objects degraded (2.052%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:18:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2515933862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:18:43.677 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:43.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:43 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 4670/227598 objects degraded (2.052%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:18:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2515933862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:18:43.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:43.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:43.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:43 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 4670/227598 objects degraded (2.052%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:18:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2515933862' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:18:43.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:43.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:43.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:43.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:44.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:44.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:44.102 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:44.126 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:44.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:44.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:44.247 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:44.247 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:44.256 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-17T16:18:44.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:44.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:44.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:18:44.391 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:44.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:44 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:18:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:44 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:18:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2515933862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:18:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:44 smithi017 ceph-mon[27571]: osdmap e338: 12 total, 12 up, 11 in 2024-09-17T16:18:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:44 smithi017 ceph-mon[27571]: from='client.16044 172.21.15.17:0/1456657036' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:44 smithi017 ceph-mon[27571]: from='client.16044 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:44 smithi017 ceph-mon[27571]: pgmap v775: 17 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 15 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 924 op/s; 4670/228390 objects degraded (2.045%) 2024-09-17T16:18:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:44 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:44 smithi017 ceph-mon[27571]: from='client.16044 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:44.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:44 smithi017 ceph-mon[27571]: osdmap e339: 12 total, 12 up, 11 in 2024-09-17T16:18:44.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:44.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:44.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:44.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 78025 78009 77.3834 58.9375 0.0261031 0.0129159 2024-09-17T16:18:44.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:44.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:44.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:44.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:44.712 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:18:44.751 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:44.775 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:44.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:44 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:18:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:44 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:18:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2515933862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:18:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:44 smithi071 ceph-mon[30002]: osdmap e338: 12 total, 12 up, 11 in 2024-09-17T16:18:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:44 smithi071 ceph-mon[30002]: from='client.16044 172.21.15.17:0/1456657036' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:44 smithi071 ceph-mon[30002]: from='client.16044 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:44 smithi071 ceph-mon[30002]: pgmap v775: 17 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 15 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 924 op/s; 4670/228390 objects degraded (2.045%) 2024-09-17T16:18:44.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:44 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:44.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:44 smithi071 ceph-mon[30002]: from='client.16044 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:44.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:44 smithi071 ceph-mon[30002]: osdmap e339: 12 total, 12 up, 11 in 2024-09-17T16:18:44.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:44 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:18:44.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:44 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:18:44.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2515933862' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:18:44.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:44 smithi173 ceph-mon[30530]: osdmap e338: 12 total, 12 up, 11 in 2024-09-17T16:18:44.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:44 smithi173 ceph-mon[30530]: from='client.16044 172.21.15.17:0/1456657036' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:44.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:44 smithi173 ceph-mon[30530]: from='client.16044 ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:44.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:44 smithi173 ceph-mon[30530]: pgmap v775: 17 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 15 active+clean; 4.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 924 op/s; 4670/228390 objects degraded (2.045%) 2024-09-17T16:18:44.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:44 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:44.845 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:44 smithi173 ceph-mon[30530]: from='client.16044 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:44.845 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:44 smithi173 ceph-mon[30530]: osdmap e339: 12 total, 12 up, 11 in 2024-09-17T16:18:44.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:44.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:44.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:45.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:45.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:45.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:45.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:18:45.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:45.167 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:45.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:45.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:45.293 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:45.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:45.396 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:45.423 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:45.423 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:45.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:45.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:45.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:45.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 78953 78937 77.0805 58 0.0088125 0.0129672 2024-09-17T16:18:45.551 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:45.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:45.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:45.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:45.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:45.785 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:45.815 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:45.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:45.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:45.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:45.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:45.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:45.983 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:45.984 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:46.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:46.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:46.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:46.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:46.179 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:46.183 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:46.217 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:46.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:46.319 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:46.350 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:46.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:46.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:46.481 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:46.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:46.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:46.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 80019 80003 76.9196 66.625 0.040945 0.012994 2024-09-17T16:18:46.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:46.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:46.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:46.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:18:46.714 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:46.728 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:46.744 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:46.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:46.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:46.874 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:46.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:46.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:46.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:47.010 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:47.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:47.042 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:47.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:47.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:47.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:47.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:46 smithi173 ceph-mon[30530]: pgmap v777: 17 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 15 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 48 MiB/s wr, 760 op/s; 4670/232548 objects degraded (2.008%); 4712/232548 objects misplaced (2.026%); 1.1 MiB/s, 17 objects/s recovering 2024-09-17T16:18:47.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:46 smithi017 ceph-mon[27571]: pgmap v777: 17 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 15 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 48 MiB/s wr, 760 op/s; 4670/232548 objects degraded (2.008%); 4712/232548 objects misplaced (2.026%); 1.1 MiB/s, 17 objects/s recovering 2024-09-17T16:18:47.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:47.269 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:47.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:47.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:18:47.282 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:18:47.305 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:18:47.305 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:18:47.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:46 smithi071 ceph-mon[30002]: pgmap v777: 17 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 15 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 48 MiB/s wr, 760 op/s; 4670/232548 objects degraded (2.008%); 4712/232548 objects misplaced (2.026%); 1.1 MiB/s, 17 objects/s recovering 2024-09-17T16:18:47.333 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:18:47.333 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:18:47.359 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:18:47.359 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:18:47.371 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:47.394 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:18:47.394 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:18:47.397 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:47.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:47.417 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:18:47.417 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:18:47.445 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:18:47.445 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:18:47.471 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:18:47.472 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:18:47.499 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:47.501 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:18:47.501 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:18:47.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:47.527 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:18:47.527 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:18:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:47.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:47.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 80921 80905 76.6082 56.375 0.00547726 0.0130464 2024-09-17T16:18:47.553 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:18:47.553 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:18:47.581 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:18:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:18:47.599 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:18:47.599 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:18:47.599 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:18:47.600 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:18:47.600 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:18:47.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:47.655 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:47.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:47.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:47.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:47.783 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:47.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:47.885 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:47.891 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.11 to 1 (10000) 2024-09-17T16:18:47.913 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:47.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:48.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:48.043 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:48.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:48.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1795456963' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:18:48.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/259090063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:48.145 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:48.171 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:48.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:48.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:18:48.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1795456963' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:18:48.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/259090063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:48.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:48.304 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:48.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:48.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:48.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1795456963' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:18:48.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/259090063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:48.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:18:48.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:48.431 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:48.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:48.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 81952 81936 76.4265 64.4375 0.0121063 0.0130784 2024-09-17T16:18:48.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:48.561 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:48.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:48.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:48.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:48.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:48.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:48.724 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:48.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:48.818 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:48.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:48.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:48.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:48.947 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:48.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:49.049 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:49.071 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:49.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:48 smithi173 ceph-mon[30530]: pgmap v778: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 32 MiB/s wr, 511 op/s; 9173/236247 objects misplaced (3.883%); 3.0 MiB/s, 47 objects/s recovering 2024-09-17T16:18:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:48 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4670/232548 objects degraded (2.008%), 1 pg degraded) 2024-09-17T16:18:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1795456963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:18:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:48 smithi173 ceph-mon[30530]: osdmap e340: 12 total, 12 up, 11 in 2024-09-17T16:18:49.172 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:49.198 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:49.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:49.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:48 smithi017 ceph-mon[27571]: pgmap v778: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 32 MiB/s wr, 511 op/s; 9173/236247 objects misplaced (3.883%); 3.0 MiB/s, 47 objects/s recovering 2024-09-17T16:18:49.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:48 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4670/232548 objects degraded (2.008%), 1 pg degraded) 2024-09-17T16:18:49.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1795456963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:18:49.230 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:48 smithi017 ceph-mon[27571]: osdmap e340: 12 total, 12 up, 11 in 2024-09-17T16:18:49.270 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T16:18:49.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:18:49.300 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:49.311 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:48 smithi071 ceph-mon[30002]: pgmap v778: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 32 MiB/s wr, 511 op/s; 9173/236247 objects misplaced (3.883%); 3.0 MiB/s, 47 objects/s recovering 2024-09-17T16:18:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:48 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4670/232548 objects degraded (2.008%), 1 pg degraded) 2024-09-17T16:18:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1795456963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:18:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:48 smithi071 ceph-mon[30002]: osdmap e340: 12 total, 12 up, 11 in 2024-09-17T16:18:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:49.345 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:49.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:49.445 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:18:49.445 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:18:49.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:18:49.445 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:21834 2.12 2:4e087699:::benchmark_data_smithi173_55121_object21833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 442.45413988400003, 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23867154600000001, 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21834 2024-09-17T16:18:49.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987492+0000", 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987494+0000", 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2919999999999998e-06 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987518+0000", 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3598e-05 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987534+0000", 2024-09-17T16:18:49.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5896000000000002e-05 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987542+0000", 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1829999999999994e-06 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987592+0000", 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0033e-05 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987671+0000", 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8993999999999996e-05 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.987759+0000", 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7751999999999996e-05 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988272+0000", 2024-09-17T16:18:49.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051340700000000001 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988663+0000", 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039071199999999998 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226103+0000", 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23744021000000001 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226137+0000", 2024-09-17T16:18:49.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3929e-05 2024-09-17T16:18:49.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226164+0000", 2024-09-17T16:18:49.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.654e-05 2024-09-17T16:18:49.452 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.453 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.453 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.453 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:21834 2.12 e117/114 ondisk, result = 0)", 2024-09-17T16:18:49.453 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:18:49.453 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 442.453030324, 2024-09-17T16:18:49.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23756740700000001, 2024-09-17T16:18:49.453 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988602+0000", 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:26.988605+0000", 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0079999999999998e-06 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226056+0000", 2024-09-17T16:18:49.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23745111999999999 2024-09-17T16:18:49.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226062+0000", 2024-09-17T16:18:49.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9739999999999999e-06 2024-09-17T16:18:49.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226095+0000", 2024-09-17T16:18:49.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2503999999999999e-05 2024-09-17T16:18:49.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226100+0000", 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9049999999999996e-06 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:27.226169+0000", 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9303999999999999e-05 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.459 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:26989 2.12 2:48690473:::benchmark_data_smithi173_55121_object26988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:18:49.459 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:18:49.459 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 436.52041618700002, 2024-09-17T16:18:49.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22401436399999999, 2024-09-17T16:18:49.459 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.459 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.459 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.459 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 26989 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921216+0000", 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921217+0000", 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5e-06 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921237+0000", 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9777000000000001e-05 2024-09-17T16:18:49.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921250+0000", 2024-09-17T16:18:49.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2704999999999999e-05 2024-09-17T16:18:49.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921256+0000", 2024-09-17T16:18:49.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4080000000000003e-06 2024-09-17T16:18:49.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:49.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921298+0000", 2024-09-17T16:18:49.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2169000000000002e-05 2024-09-17T16:18:49.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921385+0000", 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6536e-05 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.921467+0000", 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2657999999999994e-05 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922015+0000", 2024-09-17T16:18:49.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054718699999999998 2024-09-17T16:18:49.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922501+0000", 2024-09-17T16:18:49.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048639400000000002 2024-09-17T16:18:49.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145060+0000", 2024-09-17T16:18:49.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.222559429 2024-09-17T16:18:49.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145161+0000", 2024-09-17T16:18:49.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100058 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145230+0000", 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9542999999999995e-05 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:26989 2.12 e121/114 ondisk, result = 0)", 2024-09-17T16:18:49.468 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 436.51915455300002, 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22276984499999999, 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922477+0000", 2024-09-17T16:18:49.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922478+0000", 2024-09-17T16:18:49.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9100000000000004e-07 2024-09-17T16:18:49.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.922484+0000", 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.144e-06 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.144878+0000", 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.222394432 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.144910+0000", 2024-09-17T16:18:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1365999999999998e-05 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145033+0000", 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122717 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145050+0000", 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.753e-05 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:33.145247+0000", 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019706500000000001 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 136/134 [PushReplyOp(2:482f57f9:::benchmark_data_smithi173_55121_object24323:head)])", 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 407.27766822000001, 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.229086917, 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.473 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163964+0000", 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0200000000000002e-07 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.163968+0000", 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9369999999999997e-06 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392756+0000", 2024-09-17T16:18:49.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22878752399999999 2024-09-17T16:18:49.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392762+0000", 2024-09-17T16:18:49.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0959999999999997e-06 2024-09-17T16:18:49.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.392828+0000", 2024-09-17T16:18:49.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5746000000000004e-05 2024-09-17T16:18:49.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:02.393051+0000", 2024-09-17T16:18:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000223012 2024-09-17T16:18:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(2.12 158/155 [PushReplyOp(2:4d5499b6:::benchmark_data_smithi173_55121_object58972:head)])", 2024-09-17T16:18:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:18:49.478 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 360.67995669300001, 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46593107700000003, 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761675+0000", 2024-09-17T16:18:49.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761676+0000", 2024-09-17T16:18:49.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1900000000000002e-07 2024-09-17T16:18:49.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.761680+0000", 2024-09-17T16:18:49.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1710000000000004e-06 2024-09-17T16:18:49.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226484+0000", 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46480413500000001 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226502+0000", 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7895999999999999e-05 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.226671+0000", 2024-09-17T16:18:49.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016863300000000001 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.227606+0000", 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093562300000000001 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:81489 3.1 3:8126b738:::benchmark_data_smithi173_66272_object9533:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T16:18:49.483 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:57.882560+0000", 2024-09-17T16:18:49.484 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 171.55907156800001, 2024-09-17T16:18:49.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22406166199999999, 2024-09-17T16:18:49.484 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.484 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.484 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:49.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:49.485 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81489 2024-09-17T16:18:49.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.485 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882560+0000", 2024-09-17T16:18:49.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882560+0000", 2024-09-17T16:18:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882561+0000", 2024-09-17T16:18:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0400000000000002e-07 2024-09-17T16:18:49.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882563+0000", 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.345e-06 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882569+0000", 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4909999999999996e-06 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882573+0000", 2024-09-17T16:18:49.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6099999999999999e-06 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884308+0000", 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001735222 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884315+0000", 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7170000000000002e-06 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.076161+0000", 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.191846344 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.076292+0000", 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013085899999999999 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.076378+0000", 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5451999999999994e-05 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.082952+0000", 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006574084 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.089673+0000", 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067210669999999998 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.105569+0000", 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015896065000000001 2024-09-17T16:18:49.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.105632+0000", 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2650000000000005e-05 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.106622+0000", 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0009903519999999999 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.492 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:81512 3.1 3:87dcd0a7:::benchmark_data_smithi173_66272_object9556:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T16:18:49.492 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:57.882579+0000", 2024-09-17T16:18:49.492 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 171.559052474, 2024-09-17T16:18:49.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225002383, 2024-09-17T16:18:49.492 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.492 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.492 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81512 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882579+0000", 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882579+0000", 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882580+0000", 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.72e-07 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882582+0000", 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0169999999999999e-06 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882588+0000", 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8939999999999998e-06 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882597+0000", 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2569999999999996e-06 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884361+0000", 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001764072 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:18:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884370+0000", 2024-09-17T16:18:49.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6519999999999995e-06 2024-09-17T16:18:49.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.077907+0000", 2024-09-17T16:18:49.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19353769500000001 2024-09-17T16:18:49.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.078069+0000", 2024-09-17T16:18:49.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016133599999999999 2024-09-17T16:18:49.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.078214+0000", 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000145251 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.082957+0000", 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047430140000000003 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.092025+0000", 2024-09-17T16:18:49.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0090680489999999999 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.106670+0000", 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014645228 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.106707+0000", 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7264000000000001e-05 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.107582+0000", 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00087428199999999999 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:81529 3.1 3:8973215e:::benchmark_data_smithi173_66272_object9573:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:57.882603+0000", 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 171.559028703, 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225980187, 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.500 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81529 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882603+0000", 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882603+0000", 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882604+0000", 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0999999999999999e-07 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882606+0000", 2024-09-17T16:18:49.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2840000000000001e-06 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882613+0000", 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8580000000000002e-06 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882617+0000", 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6179999999999997e-06 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884416+0000", 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017984640000000001 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884420+0000", 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4070000000000003e-06 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.079807+0000", 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19538671799999999 2024-09-17T16:18:49.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.079918+0000", 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111414 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.080023+0000", 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104528 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.083012+0000", 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029894969999999998 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.092069+0000", 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0090566880000000002 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.107623+0000", 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015553502 2024-09-17T16:18:49.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.107658+0000", 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5256999999999998e-05 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.108583+0000", 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00092554200000000001 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:81555 3.1 3:89638963:::benchmark_data_smithi173_66272_object9599:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e256)", 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:57.882624+0000", 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 171.559008084, 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.236389865, 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 81555 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.508 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882624+0000", 2024-09-17T16:18:49.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882624+0000", 2024-09-17T16:18:49.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882624+0000", 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9000000000000002e-07 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882626+0000", 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1950000000000002e-06 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882633+0000", 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9330000000000004e-06 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.882638+0000", 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7940000000000002e-06 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884436+0000", 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017976209999999999 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:18:49.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:57.884440+0000", 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1930000000000003e-06 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.081537+0000", 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19709721599999999 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.081626+0000", 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8968999999999995e-05 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.081704+0000", 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8005999999999998e-05 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.083065+0000", 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001360485 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.092102+0000", 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0090376789999999999 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.117339+0000", 2024-09-17T16:18:49.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025236811000000001 2024-09-17T16:18:49.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.117377+0000", 2024-09-17T16:18:49.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7514e-05 2024-09-17T16:18:49.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:58.119014+0000", 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001637059 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:89674 3.1 3:8f13ebaf:::benchmark_data_smithi173_66272_object17718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:07.253083+0000", 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 162.188548919, 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22595691500000001, 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.516 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 89674 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253083+0000", 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253083+0000", 2024-09-17T16:18:49.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253085+0000", 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.872e-06 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253092+0000", 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9789999999999996e-06 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253103+0000", 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1595e-05 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.253113+0000", 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8120000000000002e-06 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.258528+0000", 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.005414355 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.258645+0000", 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117006 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.258798+0000", 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000153518 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.268089+0000", 2024-09-17T16:18:49.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0092910890000000006 2024-09-17T16:18:49.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.281456+0000", 2024-09-17T16:18:49.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013367005 2024-09-17T16:18:49.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.477474+0000", 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.196018055 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.477520+0000", 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5707000000000001e-05 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:07.479040+0000", 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015199219999999999 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.522 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:92396 3.1 3:87df27fa:::benchmark_data_smithi173_66272_object20440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:10.408630+0000", 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 159.03300196199999, 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.259313497, 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 92396 2024-09-17T16:18:49.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.524 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408630+0000", 2024-09-17T16:18:49.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408630+0000", 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408632+0000", 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.108e-06 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408640+0000", 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2470000000000008e-06 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408657+0000", 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6461999999999999e-05 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408666+0000", 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7979999999999995e-06 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408715+0000", 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9468000000000001e-05 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.408797+0000", 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2330000000000006e-05 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.453598+0000", 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.044800409999999999 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.453661+0000", 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2887999999999998e-05 2024-09-17T16:18:49.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.453753+0000", 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2545000000000002e-05 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.461059+0000", 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0073062550000000002 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.462616+0000", 2024-09-17T16:18:49.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001556778 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.665956+0000", 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20334012100000001 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.665996+0000", 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9227999999999998e-05 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:10.667943+0000", 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019478589999999999 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124023 3.1 3:860559f8:::benchmark_data_smithi173_66272_object52067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.587616+0000", 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 120.854015933, 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23693346200000001, 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.530 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124023 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587616+0000", 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587616+0000", 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587617+0000", 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.16e-07 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587621+0000", 2024-09-17T16:18:49.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3630000000000004e-06 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587633+0000", 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1579999999999999e-05 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.587639+0000", 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3419999999999996e-06 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.591590+0000", 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039506660000000002 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.591736+0000", 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014601800000000001 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.591880+0000", 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000144475 2024-09-17T16:18:49.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.596612+0000", 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047316169999999996 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.603533+0000", 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069211009999999998 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.823789+0000", 2024-09-17T16:18:49.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.220256128 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.823822+0000", 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3254999999999999e-05 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.824549+0000", 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00072700100000000001 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.536 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25036.0:124023 3.1 e271/256 ondisk, result = 0)", 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.597700+0000", 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 120.843932173, 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22685955299999999, 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.597700+0000", 2024-09-17T16:18:49.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.597700+0000", 2024-09-17T16:18:49.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.597700+0000", 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4700000000000001e-07 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.597703+0000", 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2499999999999998e-06 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.820323+0000", 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22261969100000001 2024-09-17T16:18:49.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.820328+0000", 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5709999999999999e-06 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.823776+0000", 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034483769999999999 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.823784+0000", 2024-09-17T16:18:49.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.503e-06 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.824559+0000", 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077561399999999999 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:134243 3.1 3:8c1e41ff:::benchmark_data_smithi173_66272_object62287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:05.911449+0000", 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 103.530182677, 2024-09-17T16:18:49.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27136616899999999, 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 134243 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911449+0000", 2024-09-17T16:18:49.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911449+0000", 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911451+0000", 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3430000000000001e-06 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911455+0000", 2024-09-17T16:18:49.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.036e-06 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911465+0000", 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0064000000000001e-05 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911470+0000", 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9849999999999997e-06 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911512+0000", 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2001e-05 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911657+0000", 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000145494 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.911804+0000", 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000146997 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.915415+0000", 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036104900000000001 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:05.922245+0000", 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0068306850000000004 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:06.182270+0000", 2024-09-17T16:18:49.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26002487800000001 2024-09-17T16:18:49.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:06.182337+0000", 2024-09-17T16:18:49.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7174000000000004e-05 2024-09-17T16:18:49.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:06.182815+0000", 2024-09-17T16:18:49.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047802199999999998 2024-09-17T16:18:49.548 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.548 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.548 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.548 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:143899 3.1 3:8d15b72a:::benchmark_data_smithi173_66272_object71943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-17T16:18:49.548 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.475047+0000", 2024-09-17T16:18:49.549 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 88.966584983999994, 2024-09-17T16:18:49.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37967805900000001, 2024-09-17T16:18:49.549 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.549 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.549 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.549 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.549 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:18:49.549 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:18:49.549 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 143899 2024-09-17T16:18:49.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.550 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475047+0000", 2024-09-17T16:18:49.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475047+0000", 2024-09-17T16:18:49.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475048+0000", 2024-09-17T16:18:49.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.065e-06 2024-09-17T16:18:49.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475053+0000", 2024-09-17T16:18:49.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8330000000000001e-06 2024-09-17T16:18:49.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475067+0000", 2024-09-17T16:18:49.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.416e-05 2024-09-17T16:18:49.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475075+0000", 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7049999999999999e-06 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.478661+0000", 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00358662 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.478828+0000", 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016719200000000001 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.478988+0000", 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015920299999999999 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.487257+0000", 2024-09-17T16:18:49.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0082690630000000001 2024-09-17T16:18:49.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.498341+0000", 2024-09-17T16:18:49.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011084053999999999 2024-09-17T16:18:49.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.854410+0000", 2024-09-17T16:18:49.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35606905799999999 2024-09-17T16:18:49.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.854451+0000", 2024-09-17T16:18:49.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.155e-05 2024-09-17T16:18:49.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.854725+0000", 2024-09-17T16:18:49.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00027355599999999999 2024-09-17T16:18:49.557 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.557 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.557 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.557 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1 e290,272 3:8b678a21:::benchmark_data_smithi173_66272_object17829:head,261'5698)", 2024-09-17T16:18:49.557 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.665092+0000", 2024-09-17T16:18:49.558 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 88.776539735, 2024-09-17T16:18:49.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46414988299999999, 2024-09-17T16:18:49.558 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.558 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.558 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:18:49.558 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665092+0000", 2024-09-17T16:18:49.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665092+0000", 2024-09-17T16:18:49.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665093+0000", 2024-09-17T16:18:49.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9599999999999999e-07 2024-09-17T16:18:49.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665097+0000", 2024-09-17T16:18:49.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2830000000000003e-06 2024-09-17T16:18:49.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.128735+0000", 2024-09-17T16:18:49.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46363770199999998 2024-09-17T16:18:49.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.128747+0000", 2024-09-17T16:18:49.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2779000000000001e-05 2024-09-17T16:18:49.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.128815+0000", 2024-09-17T16:18:49.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7749999999999993e-05 2024-09-17T16:18:49.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.129242+0000", 2024-09-17T16:18:49.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042677299999999997 2024-09-17T16:18:49.563 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.563 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.563 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.563 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25516.0:14363 4.7 4:ee0b4ef2:::benchmark_data_smithi173_77474_object14362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-17T16:18:49.563 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:49.268098+0000", 2024-09-17T16:18:49.564 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 60.173534320999998, 2024-09-17T16:18:49.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22570928100000001, 2024-09-17T16:18:49.564 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.564 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.564 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.564 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.564 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25516", 2024-09-17T16:18:49.564 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:18:49.564 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14363 2024-09-17T16:18:49.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.565 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.268098+0000", 2024-09-17T16:18:49.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.268098+0000", 2024-09-17T16:18:49.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.268099+0000", 2024-09-17T16:18:49.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7699999999999992e-07 2024-09-17T16:18:49.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.268116+0000", 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7581999999999999e-05 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.268127+0000", 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0851e-05 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.268132+0000", 2024-09-17T16:18:49.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2599999999999996e-06 2024-09-17T16:18:49.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.268176+0000", 2024-09-17T16:18:49.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.367e-05 2024-09-17T16:18:49.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.268269+0000", 2024-09-17T16:18:49.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2936999999999995e-05 2024-09-17T16:18:49.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:18:49.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.268371+0000", 2024-09-17T16:18:49.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101759 2024-09-17T16:18:49.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.268944+0000", 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057297499999999998 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.341404+0000", 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.072460744999999993 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.493745+0000", 2024-09-17T16:18:49.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15234054 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.493779+0000", 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4591999999999997e-05 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:49.493807+0000", 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7393e-05 2024-09-17T16:18:49.571 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.572 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.572 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.572 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25516.0:47838 4.7 4:ed7bbe38:::benchmark_data_smithi173_77474_object47837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-17T16:18:49.572 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:18:15.948400+0000", 2024-09-17T16:18:49.572 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 33.493231917999999, 2024-09-17T16:18:49.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22655581599999999, 2024-09-17T16:18:49.572 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25516", 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 47838 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.948400+0000", 2024-09-17T16:18:49.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.948400+0000", 2024-09-17T16:18:49.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.948401+0000", 2024-09-17T16:18:49.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.545e-06 2024-09-17T16:18:49.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.948423+0000", 2024-09-17T16:18:49.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1807999999999999e-05 2024-09-17T16:18:49.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.948439+0000", 2024-09-17T16:18:49.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5201999999999999e-05 2024-09-17T16:18:49.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.948446+0000", 2024-09-17T16:18:49.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1180000000000003e-06 2024-09-17T16:18:49.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.948494+0000", 2024-09-17T16:18:49.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8108000000000001e-05 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.948598+0000", 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010433 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.948698+0000", 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9457000000000007e-05 2024-09-17T16:18:49.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:18:49.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.950007+0000", 2024-09-17T16:18:49.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013096889999999999 2024-09-17T16:18:49.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:15.950531+0000", 2024-09-17T16:18:49.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00052343500000000005 2024-09-17T16:18:49.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:18:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:16.174859+0000", 2024-09-17T16:18:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22432813400000001 2024-09-17T16:18:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:16.174906+0000", 2024-09-17T16:18:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7595000000000001e-05 2024-09-17T16:18:49.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:16.174956+0000", 2024-09-17T16:18:49.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9394999999999997e-05 2024-09-17T16:18:49.580 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.580 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.580 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.580 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:76068 4.5 e335/301 4:a01bffc8:::benchmark_data_smithi173_77474_object76067:head v 335'4877, mlcod=335'4875)", 2024-09-17T16:18:49.581 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:18:40.016492+0000", 2024-09-17T16:18:49.581 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.4251397739999998, 2024-09-17T16:18:49.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39477241400000002, 2024-09-17T16:18:49.581 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:49.581 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:49.581 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:18:49.581 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:49.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:49.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016492+0000", 2024-09-17T16:18:49.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:49.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016492+0000", 2024-09-17T16:18:49.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:49.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:49.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016493+0000", 2024-09-17T16:18:49.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3659999999999999e-06 2024-09-17T16:18:49.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:49.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016567+0000", 2024-09-17T16:18:49.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3520000000000001e-05 2024-09-17T16:18:49.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:49.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409280+0000", 2024-09-17T16:18:49.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39271265799999999 2024-09-17T16:18:49.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:49.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409289+0000", 2024-09-17T16:18:49.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8789999999999992e-06 2024-09-17T16:18:49.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:49.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409338+0000", 2024-09-17T16:18:49.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9029000000000001e-05 2024-09-17T16:18:49.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:49.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409355+0000", 2024-09-17T16:18:49.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7727000000000002e-05 2024-09-17T16:18:49.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:18:49.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.411210+0000", 2024-09-17T16:18:49.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018547570000000001 2024-09-17T16:18:49.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:49.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:49.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:18:49.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.411265+0000", 2024-09-17T16:18:49.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4478000000000003e-05 2024-09-17T16:18:49.587 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.587 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.587 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.587 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:49.587 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:49.587 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:18:49.589 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 82705 82689 75.9946 47.0625 0.00440959 0.0131517 2024-09-17T16:18:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:49.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:49.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:49.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:49.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:49.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:49.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:49.848 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:49.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:49.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:49.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:49.979 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:49.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:50.081 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:50.107 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:50.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:50.165 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:50.209 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:50.238 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:50.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:50.340 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:50.369 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:50.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:50.399 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:18:50.470 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:50.500 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:50.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:50.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 83686 83670 75.7818 61.3125 0.0738581 0.0131903 2024-09-17T16:18:50.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:50.627 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:50.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:50.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:50 smithi173 ceph-mon[30530]: osdmap e341: 12 total, 12 up, 11 in 2024-09-17T16:18:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1049355026' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:18:50.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:18:50.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:50 smithi017 ceph-mon[27571]: osdmap e341: 12 total, 12 up, 11 in 2024-09-17T16:18:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1049355026' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:18:50.755 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:50.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:50.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:50.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:50 smithi071 ceph-mon[30002]: osdmap e341: 12 total, 12 up, 11 in 2024-09-17T16:18:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1049355026' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:18:50.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:18:50.857 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:50.878 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:50.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:50.962 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:50.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:51.002 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:51.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:51.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:18:51.114 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:18:51.128 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:51.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:51.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:51.255 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:51.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:51.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:51.356 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:51.378 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:51.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:51.480 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:51.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:51 smithi173 ceph-mon[30530]: pgmap v781: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 47 MiB/s wr, 744 op/s; 8998/241800 objects misplaced (3.721%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:18:51.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:51 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:51.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1049355026' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:18:51.484 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:51 smithi173 ceph-mon[30530]: osdmap e342: 12 total, 12 up, 11 in 2024-09-17T16:18:51.505 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:51.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:51.508 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:18:51.508 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:18:51.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:51.508 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:84309 4.2 e342/312 4:417d299b:::benchmark_data_smithi173_77474_object84308:head v 342'5205, mlcod=342'5203)", 2024-09-17T16:18:51.508 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:18:51.502853+0000", 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.00039067699999999997, 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040939999999999998, 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:51.502853+0000", 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:51.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:51.502853+0000", 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:51.502854+0000", 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5199999999999995e-07 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:51.503049+0000", 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000194632 2024-09-17T16:18:51.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:51.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:51.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:18:51.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:51.503076+0000", 2024-09-17T16:18:51.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7691e-05 2024-09-17T16:18:51.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:51.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:51.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:18:51.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:51.503084+0000", 2024-09-17T16:18:51.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6790000000000005e-06 2024-09-17T16:18:51.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:51.503147+0000", 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2901e-05 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:51.503184+0000", 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7437000000000001e-05 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:18:51.512 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:51.513 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:18:51.513 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:18:51.513 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-09-17T16:18:51.513 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:18:51.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 84336 84320 75.2795 40.625 0.015202 0.0132778 2024-09-17T16:18:51.606 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:51.633 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:51.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:51.670 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:51 smithi017 ceph-mon[27571]: pgmap v781: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 47 MiB/s wr, 744 op/s; 8998/241800 objects misplaced (3.721%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:18:51.671 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:51 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:51.671 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1049355026' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:18:51.671 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:51 smithi017 ceph-mon[27571]: osdmap e342: 12 total, 12 up, 11 in 2024-09-17T16:18:51.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:51.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:51.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:51 smithi071 ceph-mon[30002]: pgmap v781: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 47 MiB/s wr, 744 op/s; 8998/241800 objects misplaced (3.721%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:18:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:51 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:18:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1049355026' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:18:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:51 smithi071 ceph-mon[30002]: osdmap e342: 12 total, 12 up, 11 in 2024-09-17T16:18:51.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:51.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:51.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:51.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:52.018 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:52.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:52.121 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:52.151 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:52.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:52.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:18:52.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:52.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:52.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:52.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:52.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:52.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:52.459 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":782,"stamp":"2024-09-17T16:18:50.399758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5282660896,"num_objects":80600,"num_object_clones":0,"num_object_copies":241800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8998,"num_objects_unfound":0,"num_objects_dirty":80600,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":80721,"num_write_kb":5160422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":33044064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80690,"ondisk_log_size":80690,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":17442612,"kb_used_data":15459364,"kb_used_omap":57,"kb_used_meta":1982982,"kb_avail":1013754060,"statfs":{"total":1055945392128,"available":1038084157440,"internally_reserved":0,"allocated":15830388736,"data_stored":15823545886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2030574302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":292945920,"num_objects":4470,"num_object_clones":0,"num_object_copies":13410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4670,"num_objects_misplaced":8998,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4470,"num_write_kb":286080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":31391744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.015197"},"pg_stats":[{"pgid":"4.f","version":"339'4991","reported_seq":5068,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-17T16:18:46.361221+0000","last_change":"2024-09-17T16:17:41.531577+0000","last_active":"2024-09-17T16:18:46.361221+0000","last_peered":"2024-09-17T16:18:46.361221+0000","last_clean":"2024-09-17T16:18:46.361221+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:18:46.361221+0000","last_undegraded":"2024-09-17T16:18:46.361221+0000","last_fullsized":"2024-09-17T16:18:46.361221+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:01.010428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327090176,"num_objects":4991,"num_object_clones":0,"num_object_copies":14973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4991,"num_write_kb":319424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"341'5203","reported_seq":5293,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-17T16:18:49.538305+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:18:49.538305+0000","last_peered":"2024-09-17T16:18:49.538305+0000","last_clean":"2024-09-17T16:18:49.538305+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:18:49.538305+0000","last_undegraded":"2024-09-17T16:18:49.538305+0000","last_fullsized":"2024-09-17T16:18:49.538305+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5203,"log_dups_size":0,"ondisk_log_size":5203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:58.008105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340983808,"num_objects":5203,"num_object_clones":0,"num_object_copies":15609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5203,"num_write_kb":332992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"341'5044","reported_seq":5141,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-17T16:18:49.537572+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:18:49.537572+0000","last_peered":"2024-09-17T16:18:49.537572+0000","last_clean":"2024-09-17T16:18:49.537572+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:18:49.537572+0000","last_undegraded":"2024-09-17T16:18:49.537572+0000","last_fullsized":"2024-09-17T16:18:49.537572+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:51.524778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330563584,"num_objects":5044,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":322816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"341'5268","reported_seq":5358,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-17T16:18:49.561092+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:18:49.561092+0000","last_peered":"2024-09-17T16:18:49.561092+0000","last_clean":"2024-09-17T16:18:49.561092+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:18:49.561092+0000","last_undegraded":"2024-09-17T16:18:49.561092+0000","last_fullsized":"2024-09-17T16:18:49.561092+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345243648,"num_objects":5268,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5268,"num_write_kb":337152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"341'5214","reported_seq":5837,"reported_epoch":341,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:18:49.592558+0000","last_change":"2024-09-17T16:18:41.641380+0000","last_active":"2024-09-17T16:18:49.592558+0000","last_peered":"2024-09-17T16:18:49.592558+0000","last_clean":"2024-09-17T16:18:39.384812+0000","last_became_active":"2024-09-17T16:18:41.595599+0000","last_became_peered":"2024-09-17T16:18:41.595599+0000","last_unstale":"2024-09-17T16:18:49.592558+0000","last_undegraded":"2024-09-17T16:18:49.592558+0000","last_fullsized":"2024-09-17T16:18:41.426452+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341704704,"num_objects":5214,"num_object_clones":0,"num_object_copies":15642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4537,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5214,"num_write_kb":333696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":17694720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4","objects":677},{"shards":"0,4,5","objects":4537}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"339'5000","reported_seq":5091,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-17T16:18:45.569608+0000","last_change":"2024-09-17T16:17:41.379235+0000","last_active":"2024-09-17T16:18:45.569608+0000","last_peered":"2024-09-17T16:18:45.569608+0000","last_clean":"2024-09-17T16:18:45.569608+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:18:45.569608+0000","last_undegraded":"2024-09-17T16:18:45.569608+0000","last_fullsized":"2024-09-17T16:18:45.569608+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5000,"log_dups_size":0,"ondisk_log_size":5000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:00.640175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680000,"num_objects":5000,"num_object_clones":0,"num_object_copies":15000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5000,"num_write_kb":320000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"339'4906","reported_seq":4983,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-17T16:18:46.342372+0000","last_change":"2024-09-17T16:17:41.530942+0000","last_active":"2024-09-17T16:18:46.342372+0000","last_peered":"2024-09-17T16:18:46.342372+0000","last_clean":"2024-09-17T16:18:46.342372+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:18:46.342372+0000","last_undegraded":"2024-09-17T16:18:46.342372+0000","last_fullsized":"2024-09-17T16:18:46.342372+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:56.403839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321519616,"num_objects":4906,"num_object_clones":0,"num_object_copies":14718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4906,"num_write_kb":313984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"340'5132","reported_seq":5219,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-17T16:18:47.908705+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:18:47.908705+0000","last_peered":"2024-09-17T16:18:47.908705+0000","last_clean":"2024-09-17T16:18:47.908705+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:18:47.908705+0000","last_undegraded":"2024-09-17T16:18:47.908705+0000","last_fullsized":"2024-09-17T16:18:47.908705+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336330752,"num_objects":5132,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5132,"num_write_kb":328448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"339'4873","reported_seq":4965,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-17T16:18:45.686082+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:18:45.686082+0000","last_peered":"2024-09-17T16:18:45.686082+0000","last_clean":"2024-09-17T16:18:45.686082+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:18:45.686082+0000","last_undegraded":"2024-09-17T16:18:45.686082+0000","last_fullsized":"2024-09-17T16:18:45.686082+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:14.213522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319356928,"num_objects":4873,"num_object_clones":0,"num_object_copies":14619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4873,"num_write_kb":311872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"339'4936","reported_seq":5433,"reported_epoch":339,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:18:46.700677+0000","last_change":"2024-09-17T16:18:42.321603+0000","last_active":"2024-09-17T16:18:46.700677+0000","last_peered":"2024-09-17T16:18:46.700677+0000","last_clean":"2024-09-17T16:18:39.337421+0000","last_became_active":"2024-09-17T16:18:42.219868+0000","last_became_peered":"2024-09-17T16:18:42.219868+0000","last_unstale":"2024-09-17T16:18:46.700677+0000","last_undegraded":"2024-09-17T16:18:46.700677+0000","last_fullsized":"2024-09-17T16:18:41.426290+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323485696,"num_objects":4936,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4461,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4936,"num_write_kb":315904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":13697024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[2,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":4461},{"shards":"2,7","objects":475}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"339'4968","reported_seq":5045,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-17T16:18:46.661874+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:18:46.661874+0000","last_peered":"2024-09-17T16:18:46.661874+0000","last_clean":"2024-09-17T16:18:46.661874+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:18:46.661874+0000","last_undegraded":"2024-09-17T16:18:46.661874+0000","last_fullsized":"2024-09-17T16:18:46.661874+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":4968,"log_dups_size":0,"ondisk_log_size":4968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:55.543723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325582848,"num_objects":4968,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":317952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"340'5149","reported_seq":5244,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-17T16:18:47.926048+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:18:47.926048+0000","last_peered":"2024-09-17T16:18:47.926048+0000","last_clean":"2024-09-17T16:18:47.926048+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:18:47.926048+0000","last_undegraded":"2024-09-17T16:18:47.926048+0000","last_fullsized":"2024-09-17T16:18:47.926048+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5149,"log_dups_size":0,"ondisk_log_size":5149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:00.611452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337444864,"num_objects":5149,"num_object_clones":0,"num_object_copies":15447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5149,"num_write_kb":329536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"339'4859","reported_seq":4954,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-17T16:18:45.495897+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:18:45.495897+0000","last_peered":"2024-09-17T16:18:45.495897+0000","last_clean":"2024-09-17T16:18:45.495897+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:18:45.495897+0000","last_undegraded":"2024-09-17T16:18:45.495897+0000","last_fullsized":"2024-09-17T16:18:45.495897+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:44.325955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318439424,"num_objects":4859,"num_object_clones":0,"num_object_copies":14577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4859,"num_write_kb":310976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"339'4976","reported_seq":5063,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-17T16:18:45.485411+0000","last_change":"2024-09-17T16:17:34.582545+0000","last_active":"2024-09-17T16:18:45.485411+0000","last_peered":"2024-09-17T16:18:45.485411+0000","last_clean":"2024-09-17T16:18:45.485411+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:18:45.485411+0000","last_undegraded":"2024-09-17T16:18:45.485411+0000","last_fullsized":"2024-09-17T16:18:45.485411+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:58.497255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326107136,"num_objects":4976,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":318464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"339'4996","reported_seq":5081,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-17T16:18:46.371954+0000","last_change":"2024-09-17T16:17:34.582002+0000","last_active":"2024-09-17T16:18:46.371954+0000","last_peered":"2024-09-17T16:18:46.371954+0000","last_clean":"2024-09-17T16:18:46.371954+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:18:46.371954+0000","last_undegraded":"2024-09-17T16:18:46.371954+0000","last_fullsized":"2024-09-17T16:18:46.371954+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":4996,"log_dups_size":0,"ondisk_log_size":4996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:58.969242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327417856,"num_objects":4996,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4996,"num_write_kb":319744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"339'5083","reported_seq":5166,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-17T16:18:45.570366+0000","last_change":"2024-09-17T16:17:34.582684+0000","last_active":"2024-09-17T16:18:45.570366+0000","last_peered":"2024-09-17T16:18:45.570366+0000","last_clean":"2024-09-17T16:18:45.570366+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:18:45.570366+0000","last_undegraded":"2024-09-17T16:18:45.570366+0000","last_fullsized":"2024-09-17T16:18:45.570366+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:59.603290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333119488,"num_objects":5083,"num_object_clones":0,"num_object_copies":15249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5083,"num_write_kb":325312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":573,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-09-17T16:18:47.894010+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:18:47.894010+0000","last_peered":"2024-09-17T16:18:47.894010+0000","last_clean":"2024-09-17T16:18:47.894010+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:18:47.894010+0000","last_undegraded":"2024-09-17T16:18:47.894010+0000","last_fullsized":"2024-09-17T16:18:47.894010+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:50.229084+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":5282070528,"num_objects":80598,"num_object_clones":0,"num_object_copies":241794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8998,"num_objects_unfound":0,"num_objects_dirty":80598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80598,"num_write_kb":5158272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":480,"num_bytes_recovered":31457280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15875244032,"data_stored":15875244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80598,"ondisk_log_size":80598,"up":48,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285680,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2867556,"kb_used_data":2647568,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":90877596,"statfs":{"total":95995035648,"available":93058658304,"internally_reserved":0,"allocated":2711109632,"data_stored":2710478020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":225245636},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547316,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713408,"kb_used_data":1587824,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92031744,"statfs":{"total":95995035648,"available":94240505856,"internally_reserved":0,"allocated":1625931776,"data_stored":1625312708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":128580032},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743545,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267200,"kb_used_data":1001392,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92477952,"statfs":{"total":95995035648,"available":94697422848,"internally_reserved":0,"allocated":1025425408,"data_stored":1024806340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":272169408},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005182,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":123088,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622064,"statfs":{"total":95995035648,"available":95868993536,"internally_reserved":0,"allocated":1699840,"data_stored":1076290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963659,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2136968,"kb_used_data":1935800,"kb_used_omap":41,"kb_used_meta":201110,"kb_avail":91608184,"statfs":{"total":95995035648,"available":93806780416,"internally_reserved":0,"allocated":1982259200,"data_stored":1981636580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":205936947},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2402376,"kb_used_data":2230968,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91342776,"statfs":{"total":95995035648,"available":93535002624,"internally_reserved":0,"allocated":2284511232,"data_stored":2283888612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724683,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1451784,"kb_used_data":1235952,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92293368,"statfs":{"total":95995035648,"available":94508408832,"internally_reserved":0,"allocated":1265614848,"data_stored":1264995780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":220985798},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986322,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497952,"kb_used_data":1326536,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92247200,"statfs":{"total":95995035648,"available":94461132800,"internally_reserved":0,"allocated":1358372864,"data_stored":1357744824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1801480,"kb_used_data":1585648,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91943672,"statfs":{"total":95995035648,"available":94150320128,"internally_reserved":0,"allocated":1623703552,"data_stored":1623084484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":220985799},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411484,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":733312,"kb_used_data":629936,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93011840,"statfs":{"total":95995035648,"available":95244124160,"internally_reserved":0,"allocated":645054464,"data_stored":644435396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738534,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1447488,"kb_used_data":1276080,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92297664,"statfs":{"total":95995035648,"available":94512807936,"internally_reserved":0,"allocated":1306705920,"data_stored":1306086852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1306329088,"data_stored":1306329088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":649199616,"data_stored":649199616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1632829440,"data_stored":1632829440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1358692352,"data_stored":1358692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1265631232,"data_stored":1265631232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2284847104,"data_stored":2284847104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1998651392,"data_stored":1998651392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1041629184,"data_stored":1041629184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1627389952,"data_stored":1627389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2710044672,"data_stored":2710044672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:18:52.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:18:52.506 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:52.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:52.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 85117 85101 74.9066 48.8125 0.0473001 0.0133396 2024-09-17T16:18:52.534 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:52.549 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:52.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:52.651 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:52.657 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:52 smithi173 ceph-mon[30530]: from='client.16068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:18:52.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:52 smithi017 ceph-mon[27571]: from='client.16068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:18:52.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:52.682 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:52.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:52.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:52 smithi071 ceph-mon[30002]: from='client.16068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:18:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:52.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:52.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:52.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:52.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:52.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:53.064 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:18:53.064 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:18:53.064 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:18:53.065 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:18:53.065 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:18:53.066 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:53.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:53.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:53.142 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:53.154 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:18:53.154 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:18:53.184 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:18:53.184 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:18:53.191 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:53.211 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:18:53.211 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:18:53.224 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:53.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:53.236 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:18:53.236 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:18:53.264 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:18:53.265 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:18:53.293 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:18:53.293 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:18:53.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:18:53.321 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:18:53.325 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:53.355 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:18:53.355 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:18:53.358 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:53.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:53.384 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:18:53.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:18:53.410 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:18:53.411 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:18:53.441 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:18:53.442 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:18:53.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:53.473 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:18:53.473 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:18:53.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:53.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:53.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 86154 86138 74.7663 64.8125 0.00526027 0.0133704 2024-09-17T16:18:53.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:53.619 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:53.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:53.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:53 smithi017 ceph-mon[27571]: pgmap v783: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 61 MiB/s wr, 974 op/s; 8729/250101 objects misplaced (3.490%); 6.8 MiB/s, 108 objects/s recovering 2024-09-17T16:18:53.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3472488151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:53.712 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:18:53.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:53.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:53.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:53.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:53.785 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:53 smithi173 ceph-mon[30530]: pgmap v783: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 61 MiB/s wr, 974 op/s; 8729/250101 objects misplaced (3.490%); 6.8 MiB/s, 108 objects/s recovering 2024-09-17T16:18:53.786 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3472488151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:53.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:53 smithi071 ceph-mon[30002]: pgmap v783: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 61 MiB/s wr, 974 op/s; 8729/250101 objects misplaced (3.490%); 6.8 MiB/s, 108 objects/s recovering 2024-09-17T16:18:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3472488151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:18:53.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:18:53.849 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:53.878 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:53.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:53.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:54.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:54.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:54.091 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:18:54.092 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:18:54.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:54.137 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:54.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:54.209 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:18:54.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:54.270 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:54.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:54.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:54.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:54.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:54.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:54.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 87249 87233 74.6796 68.4375 0.0143301 0.0133848 2024-09-17T16:18:54.529 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:54.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:54.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:54.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:54 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:18:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:54 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:18:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:54 smithi173 ceph-mon[30530]: osdmap e343: 12 total, 12 up, 11 in 2024-09-17T16:18:54.659 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:54.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:54.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:54 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:18:54.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:54 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:18:54.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:54 smithi017 ceph-mon[27571]: osdmap e343: 12 total, 12 up, 11 in 2024-09-17T16:18:54.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:54.786 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:54.786 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:54.786 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:54.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:54 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:18:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:54 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:18:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:54 smithi071 ceph-mon[30002]: osdmap e343: 12 total, 12 up, 11 in 2024-09-17T16:18:54.888 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:54.912 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:54.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:55.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:55.044 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:55.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:55.148 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:55.175 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:55.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:55.277 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:55.306 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:55.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:55.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:55.412 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":784,"stamp":"2024-09-17T16:18:53.830458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5499322912,"num_objects":83906,"num_object_clones":0,"num_object_copies":251718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8729,"num_objects_unfound":0,"num_objects_dirty":83906,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":84027,"num_write_kb":5372006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":755,"num_bytes_recovered":50673248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83996,"ondisk_log_size":83996,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":18201784,"kb_used_data":16165764,"kb_used_omap":57,"kb_used_meta":2035718,"kb_avail":1012994888,"statfs":{"total":1055945392128,"available":1037306765312,"internally_reserved":0,"allocated":16553742336,"data_stored":16546791790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2084575966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,1,1,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":337969152,"num_objects":5157,"num_object_clones":0,"num_object_copies":15471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-444,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":330048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":29097984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001207"},"pg_stats":[{"pgid":"4.f","version":"342'5235","reported_seq":5317,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:51.229010+0000","last_change":"2024-09-17T16:17:41.531577+0000","last_active":"2024-09-17T16:18:51.229010+0000","last_peered":"2024-09-17T16:18:51.229010+0000","last_clean":"2024-09-17T16:18:51.229010+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:18:51.229010+0000","last_undegraded":"2024-09-17T16:18:51.229010+0000","last_fullsized":"2024-09-17T16:18:51.229010+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:01.010428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343080960,"num_objects":5235,"num_object_clones":0,"num_object_copies":15705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5235,"num_write_kb":335040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"341'5203","reported_seq":5293,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-17T16:18:49.538305+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:18:49.538305+0000","last_peered":"2024-09-17T16:18:49.538305+0000","last_clean":"2024-09-17T16:18:49.538305+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:18:49.538305+0000","last_undegraded":"2024-09-17T16:18:49.538305+0000","last_fullsized":"2024-09-17T16:18:49.538305+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5203,"log_dups_size":0,"ondisk_log_size":5203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:58.008105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340983808,"num_objects":5203,"num_object_clones":0,"num_object_copies":15609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5203,"num_write_kb":332992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"341'5044","reported_seq":5141,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-17T16:18:49.537572+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:18:49.537572+0000","last_peered":"2024-09-17T16:18:49.537572+0000","last_clean":"2024-09-17T16:18:49.537572+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:18:49.537572+0000","last_undegraded":"2024-09-17T16:18:49.537572+0000","last_fullsized":"2024-09-17T16:18:49.537572+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:51.524778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330563584,"num_objects":5044,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":322816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"341'5268","reported_seq":5358,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-09-17T16:18:49.561092+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:18:49.561092+0000","last_peered":"2024-09-17T16:18:49.561092+0000","last_clean":"2024-09-17T16:18:49.561092+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:18:49.561092+0000","last_undegraded":"2024-09-17T16:18:49.561092+0000","last_fullsized":"2024-09-17T16:18:49.561092+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345243648,"num_objects":5268,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5268,"num_write_kb":337152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"341'5214","reported_seq":5837,"reported_epoch":341,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:18:49.592558+0000","last_change":"2024-09-17T16:18:41.641380+0000","last_active":"2024-09-17T16:18:49.592558+0000","last_peered":"2024-09-17T16:18:49.592558+0000","last_clean":"2024-09-17T16:18:39.384812+0000","last_became_active":"2024-09-17T16:18:41.595599+0000","last_became_peered":"2024-09-17T16:18:41.595599+0000","last_unstale":"2024-09-17T16:18:49.592558+0000","last_undegraded":"2024-09-17T16:18:49.592558+0000","last_fullsized":"2024-09-17T16:18:41.426452+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341704704,"num_objects":5214,"num_object_clones":0,"num_object_copies":15642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4537,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5214,"num_write_kb":333696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":17694720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4","objects":677},{"shards":"0,4,5","objects":4537}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"342'5259","reported_seq":5356,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:50.547979+0000","last_change":"2024-09-17T16:17:41.379235+0000","last_active":"2024-09-17T16:18:50.547979+0000","last_peered":"2024-09-17T16:18:50.547979+0000","last_clean":"2024-09-17T16:18:50.547979+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:18:50.547979+0000","last_undegraded":"2024-09-17T16:18:50.547979+0000","last_fullsized":"2024-09-17T16:18:50.547979+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5259,"log_dups_size":0,"ondisk_log_size":5259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:00.640175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344653824,"num_objects":5259,"num_object_clones":0,"num_object_copies":15777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":336576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"342'5174","reported_seq":5257,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:51.219008+0000","last_change":"2024-09-17T16:17:41.530942+0000","last_active":"2024-09-17T16:18:51.219008+0000","last_peered":"2024-09-17T16:18:51.219008+0000","last_clean":"2024-09-17T16:18:51.219008+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:18:51.219008+0000","last_undegraded":"2024-09-17T16:18:51.219008+0000","last_fullsized":"2024-09-17T16:18:51.219008+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:56.403839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339083264,"num_objects":5174,"num_object_clones":0,"num_object_copies":15522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5174,"num_write_kb":331136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"342'5400","reported_seq":5492,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:52.920781+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:18:52.920781+0000","last_peered":"2024-09-17T16:18:52.920781+0000","last_clean":"2024-09-17T16:18:52.920781+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:18:52.920781+0000","last_undegraded":"2024-09-17T16:18:52.920781+0000","last_fullsized":"2024-09-17T16:18:52.920781+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5400,"log_dups_size":0,"ondisk_log_size":5400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353894400,"num_objects":5400,"num_object_clones":0,"num_object_copies":16200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5400,"num_write_kb":345600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"342'5186","reported_seq":5285,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:50.729060+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:18:50.729060+0000","last_peered":"2024-09-17T16:18:50.729060+0000","last_clean":"2024-09-17T16:18:50.729060+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:18:50.729060+0000","last_undegraded":"2024-09-17T16:18:50.729060+0000","last_fullsized":"2024-09-17T16:18:50.729060+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":5186,"log_dups_size":0,"ondisk_log_size":5186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:14.213522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339869696,"num_objects":5186,"num_object_clones":0,"num_object_copies":15558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5186,"num_write_kb":331904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"342'5190","reported_seq":6230,"reported_epoch":342,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:18:51.708481+0000","last_change":"2024-09-17T16:18:42.321603+0000","last_active":"2024-09-17T16:18:51.708481+0000","last_peered":"2024-09-17T16:18:51.708481+0000","last_clean":"2024-09-17T16:18:39.337421+0000","last_became_active":"2024-09-17T16:18:42.219868+0000","last_became_peered":"2024-09-17T16:18:42.219868+0000","last_unstale":"2024-09-17T16:18:51.708481+0000","last_undegraded":"2024-09-17T16:18:51.708481+0000","last_fullsized":"2024-09-17T16:18:41.426290+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5190,"log_dups_size":0,"ondisk_log_size":5190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340131840,"num_objects":5190,"num_object_clones":0,"num_object_copies":15570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5190,"num_write_kb":332160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":31326208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[2,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":4192},{"shards":"2,7","objects":998}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"342'5255","reported_seq":5337,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:51.676865+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:18:51.676865+0000","last_peered":"2024-09-17T16:18:51.676865+0000","last_clean":"2024-09-17T16:18:51.676865+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:18:51.676865+0000","last_undegraded":"2024-09-17T16:18:51.676865+0000","last_fullsized":"2024-09-17T16:18:51.676865+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5255,"log_dups_size":0,"ondisk_log_size":5255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:55.543723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344391680,"num_objects":5255,"num_object_clones":0,"num_object_copies":15765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5255,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"342'5420","reported_seq":5517,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:52.920112+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:18:52.920112+0000","last_peered":"2024-09-17T16:18:52.920112+0000","last_clean":"2024-09-17T16:18:52.920112+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:18:52.920112+0000","last_undegraded":"2024-09-17T16:18:52.920112+0000","last_fullsized":"2024-09-17T16:18:52.920112+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5420,"log_dups_size":0,"ondisk_log_size":5420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:00.611452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355205120,"num_objects":5420,"num_object_clones":0,"num_object_copies":16260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5420,"num_write_kb":346880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"342'5161","reported_seq":5263,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:50.475739+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:18:50.475739+0000","last_peered":"2024-09-17T16:18:50.475739+0000","last_clean":"2024-09-17T16:18:50.475739+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:18:50.475739+0000","last_undegraded":"2024-09-17T16:18:50.475739+0000","last_fullsized":"2024-09-17T16:18:50.475739+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5161,"log_dups_size":0,"ondisk_log_size":5161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:44.325955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338231296,"num_objects":5161,"num_object_clones":0,"num_object_copies":15483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":330304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"342'5273","reported_seq":5366,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:50.475829+0000","last_change":"2024-09-17T16:17:34.582545+0000","last_active":"2024-09-17T16:18:50.475829+0000","last_peered":"2024-09-17T16:18:50.475829+0000","last_clean":"2024-09-17T16:18:50.475829+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:18:50.475829+0000","last_undegraded":"2024-09-17T16:18:50.475829+0000","last_fullsized":"2024-09-17T16:18:50.475829+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5273,"log_dups_size":0,"ondisk_log_size":5273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:58.497255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345571328,"num_objects":5273,"num_object_clones":0,"num_object_copies":15819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":337472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"342'5257","reported_seq":5347,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:51.277832+0000","last_change":"2024-09-17T16:17:34.582002+0000","last_active":"2024-09-17T16:18:51.277832+0000","last_peered":"2024-09-17T16:18:51.277832+0000","last_clean":"2024-09-17T16:18:51.277832+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:18:51.277832+0000","last_undegraded":"2024-09-17T16:18:51.277832+0000","last_fullsized":"2024-09-17T16:18:51.277832+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:58.969242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344522752,"num_objects":5257,"num_object_clones":0,"num_object_copies":15771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5257,"num_write_kb":336448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"342'5365","reported_seq":5455,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:50.576209+0000","last_change":"2024-09-17T16:17:34.582684+0000","last_active":"2024-09-17T16:18:50.576209+0000","last_peered":"2024-09-17T16:18:50.576209+0000","last_clean":"2024-09-17T16:18:50.576209+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:18:50.576209+0000","last_undegraded":"2024-09-17T16:18:50.576209+0000","last_fullsized":"2024-09-17T16:18:50.576209+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5365,"log_dups_size":0,"ondisk_log_size":5365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:18:59.603290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351600640,"num_objects":5365,"num_object_clones":0,"num_object_copies":16095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5365,"num_write_kb":343360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":577,"reported_epoch":342,"state":"active+clean","last_fresh":"2024-09-17T16:18:50.404123+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:18:50.404123+0000","last_peered":"2024-09-17T16:18:50.404123+0000","last_clean":"2024-09-17T16:18:50.404123+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:18:50.404123+0000","last_undegraded":"2024-09-17T16:18:50.404123+0000","last_fullsized":"2024-09-17T16:18:50.404123+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":5498732544,"num_objects":83904,"num_object_clones":0,"num_object_copies":251712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8729,"num_objects_unfound":0,"num_objects_dirty":83904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83904,"num_write_kb":5369856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":749,"num_bytes_recovered":49086464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16569335808,"data_stored":16569335808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83904,"ondisk_log_size":83904,"up":48,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285680,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2867556,"kb_used_data":2647568,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":90877596,"statfs":{"total":95995035648,"available":93058658304,"internally_reserved":0,"allocated":2711109632,"data_stored":2710478020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":225245636},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547317,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1802096,"kb_used_data":1684820,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":91943056,"statfs":{"total":95995035648,"available":94149689344,"internally_reserved":0,"allocated":1725255680,"data_stored":1724623150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":120060352},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743546,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1407596,"kb_used_data":1115412,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92337556,"statfs":{"total":95995035648,"available":94553657344,"internally_reserved":0,"allocated":1142181888,"data_stored":1141549358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":299170240},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005182,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":123088,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622064,"statfs":{"total":95995035648,"available":95868993536,"internally_reserved":0,"allocated":1699840,"data_stored":1076290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963660,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2246572,"kb_used_data":2053724,"kb_used_omap":41,"kb_used_meta":192790,"kb_avail":91498580,"statfs":{"total":95995035648,"available":93694545920,"internally_reserved":0,"allocated":2103013376,"data_stored":2102377294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":197417267},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495753,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2545528,"kb_used_data":2365788,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91199624,"statfs":{"total":95995035648,"available":93388414976,"internally_reserved":0,"allocated":2422566912,"data_stored":2421930830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724684,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1495208,"kb_used_data":1275220,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92249944,"statfs":{"total":95995035648,"available":94463942656,"internally_reserved":0,"allocated":1305825280,"data_stored":1305192750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":225245638},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986322,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497952,"kb_used_data":1326536,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92247200,"statfs":{"total":95995035648,"available":94461132800,"internally_reserved":0,"allocated":1358372864,"data_stored":1357744824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1897516,"kb_used_data":1677524,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":91847636,"statfs":{"total":95995035648,"available":94051979264,"internally_reserved":0,"allocated":1717784576,"data_stored":1717152046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":225245639},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411485,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793700,"kb_used_data":668116,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92951452,"statfs":{"total":95995035648,"available":95182286848,"internally_reserved":0,"allocated":684150784,"data_stored":683518254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1524972,"kb_used_data":1349396,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92220180,"statfs":{"total":95995035648,"available":94433464320,"internally_reserved":0,"allocated":1381781504,"data_stored":1381148974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1382219776,"data_stored":1382219776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":684523520,"data_stored":684523520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1721827328,"data_stored":1721827328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1358692352,"data_stored":1358692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1304887296,"data_stored":1304887296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2420637696,"data_stored":2420637696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2101542912,"data_stored":2101542912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1160052736,"data_stored":1160052736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1724907520,"data_stored":1724907520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2710044672,"data_stored":2710044672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:18:55.414 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-17T16:18:55.436 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:55.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:55.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:55.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:18:55.525 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:18:55.528 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 88138 88122 74.4212 55.5625 0.00382136 0.0134316 2024-09-17T16:18:55.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:55.564 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:55.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:55.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:55 smithi017 ceph-mon[27571]: pgmap v784: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 54 MiB/s wr, 859 op/s; 8729/251718 objects misplaced (3.468%); 4.6 MiB/s, 73 objects/s recovering 2024-09-17T16:18:55.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:55 smithi017 ceph-mon[27571]: from='client.16080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:18:55.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:55.691 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:55.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:55.793 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:55.795 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:55 smithi071 ceph-mon[30002]: pgmap v784: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 54 MiB/s wr, 859 op/s; 8729/251718 objects misplaced (3.468%); 4.6 MiB/s, 73 objects/s recovering 2024-09-17T16:18:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:55 smithi071 ceph-mon[30002]: from='client.16080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:18:55.817 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:55.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:55.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:55 smithi173 ceph-mon[30530]: pgmap v784: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 54 MiB/s wr, 859 op/s; 8729/251718 objects misplaced (3.468%); 4.6 MiB/s, 73 objects/s recovering 2024-09-17T16:18:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:55 smithi173 ceph-mon[30530]: from='client.16080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:18:55.918 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:18:55.951 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:55.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:18:56.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:56.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:56.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:56.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:56.217 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:56.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:56.319 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:18:56.343 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:56.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:18:56.444 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:56.478 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:56.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:56.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 89091 89075 74.223 59.5625 0.0125921 0.0134675 2024-09-17T16:18:56.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:56 smithi071 ceph-mon[30002]: pgmap v786: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 8525/259422 objects misplaced (3.286%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T16:18:56.581 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:56.593 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:56.594 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:56.608 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:56.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:56.619 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:18:56.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:56 smithi017 ceph-mon[27571]: pgmap v786: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 8525/259422 objects misplaced (3.286%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T16:18:56.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:56.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:56.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:56.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:56.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:56.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:56.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:56 smithi173 ceph-mon[30530]: pgmap v786: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 8525/259422 objects misplaced (3.286%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T16:18:56.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:56.987 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:56.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:57.089 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:57.115 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:57.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:57.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:57.242 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:57.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:57.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:57.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:57.374 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:57.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:57.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:18:57.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:57.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:18:57.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 89997 89981 73.9914 56.625 0.00855322 0.0135101 2024-09-17T16:18:57.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:18:57.634 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:57.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:18:57.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:57.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:57.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:57.776 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:18:57.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:57.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:57.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:57.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:58.023 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:58.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:58.115 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:18:58.126 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:58.155 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:58.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:58.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:18:58.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:58.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:18:58.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:58.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:58.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:58.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:58.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 90818 90802 73.6968 51.3125 0.00406283 0.0135621 2024-09-17T16:18:58.536 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:58.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:58.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:18:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:18:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:58 smithi071 ceph-mon[30002]: osdmap e344: 12 total, 12 up, 11 in 2024-09-17T16:18:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:58 smithi071 ceph-mon[30002]: pgmap v788: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 62 MiB/s wr, 986 op/s; 8279/263784 objects misplaced (3.139%); 6.0 MiB/s, 96 objects/s recovering 2024-09-17T16:18:58.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:58.666 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:58.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:58.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:18:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:18:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:58 smithi017 ceph-mon[27571]: osdmap e344: 12 total, 12 up, 11 in 2024-09-17T16:18:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:58 smithi017 ceph-mon[27571]: pgmap v788: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 62 MiB/s wr, 986 op/s; 8279/263784 objects misplaced (3.139%); 6.0 MiB/s, 96 objects/s recovering 2024-09-17T16:18:58.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:58.791 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:58.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:58.893 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:18:58.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:18:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:18:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:58 smithi173 ceph-mon[30530]: osdmap e344: 12 total, 12 up, 11 in 2024-09-17T16:18:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:58 smithi173 ceph-mon[30530]: pgmap v788: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 62 MiB/s wr, 986 op/s; 8279/263784 objects misplaced (3.139%); 6.0 MiB/s, 96 objects/s recovering 2024-09-17T16:18:58.919 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:58.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:18:59.021 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:18:59.047 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:59.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:18:59.149 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:59.175 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:59.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:59.242 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:18:59.243 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:18:59.277 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:18:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:59.305 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:18:59.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:18:59.407 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:18:59.435 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:59.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:18:59.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 91615 91599 73.3905 49.8125 0.0139146 0.0136201 2024-09-17T16:18:59.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:18:59.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:59.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:18:59.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:18:59.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:18:59.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:59.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:18:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/739345892' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:59 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:18:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:59 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:18:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/739345892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:18:59 smithi017 ceph-mon[27571]: osdmap e345: 12 total, 12 up, 11 in 2024-09-17T16:18:59.754 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:18:59.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/739345892' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:59 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:18:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:59 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:18:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/739345892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:18:59 smithi071 ceph-mon[30002]: osdmap e345: 12 total, 12 up, 11 in 2024-09-17T16:18:59.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:59.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:59.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:18:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/739345892' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:18:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:59 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:18:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:59 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:18:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/739345892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:18:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:18:59 smithi173 ceph-mon[30530]: osdmap e345: 12 total, 12 up, 11 in 2024-09-17T16:18:59.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:18:59.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:18:59.941 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:18:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:18:59.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:00.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:00.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:00.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:00.173 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:00.201 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:00.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:00.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:00.331 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:19:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:00.345 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:00.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:00.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:00.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:00.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:00.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 92546 92530 73.198 58.1875 0.0178674 0.0136561 2024-09-17T16:19:00.566 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:00.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:00.606 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:00.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:00.609 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:00.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:00 smithi017 ceph-mon[27571]: 4.5 scrub starts 2024-09-17T16:19:00.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:00 smithi017 ceph-mon[27571]: pgmap v790: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 59 MiB/s wr, 943 op/s; 8085/268707 objects misplaced (3.009%); 6.7 MiB/s, 107 objects/s recovering 2024-09-17T16:19:00.708 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:00.734 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:00.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:00.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:00.766 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:00 smithi071 ceph-mon[30002]: 4.5 scrub starts 2024-09-17T16:19:00.766 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:00 smithi071 ceph-mon[30002]: pgmap v790: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 59 MiB/s wr, 943 op/s; 8085/268707 objects misplaced (3.009%); 6.7 MiB/s, 107 objects/s recovering 2024-09-17T16:19:00.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:00.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:00.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:00 smithi173 ceph-mon[30530]: 4.5 scrub starts 2024-09-17T16:19:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:00 smithi173 ceph-mon[30530]: pgmap v790: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 59 MiB/s wr, 943 op/s; 8085/268707 objects misplaced (3.009%); 6.7 MiB/s, 107 objects/s recovering 2024-09-17T16:19:00.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:00.992 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:19:00.992 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:19:00.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:00.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:01.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:01.125 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:01.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:01.227 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:01.255 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:01.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:01.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:01.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:01.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:01.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:01.483 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:01.513 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:01.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:01.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:19:01.528310+0000 min lat: 0.00106175 max lat: 2.86134 avg lat: 0.0137197 2024-09-17T16:19:01.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:19:01.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 93271 93255 72.8494 45.3125 0.0129763 0.0137197 2024-09-17T16:19:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:01 smithi071 ceph-mon[30002]: 4.a scrub starts 2024-09-17T16:19:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:01 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:19:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:01 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:19:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:01 smithi071 ceph-mon[30002]: 4.5 scrub ok 2024-09-17T16:19:01.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:01 smithi017 ceph-mon[27571]: 4.a scrub starts 2024-09-17T16:19:01.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:01 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:19:01.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:01 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:19:01.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:01 smithi017 ceph-mon[27571]: 4.5 scrub ok 2024-09-17T16:19:01.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:01.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:01.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:01.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:01.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:19:01.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:01.775 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:01.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:01.797 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:19:01.797 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:19:01.835 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:19:01.836 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:19:01.869 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:19:01.869 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:19:01.875 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:19:01.875 INFO:teuthology.orchestra.run.smithi017.stdout:{"fsid":"19341aec-750e-11ef-bceb-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":859,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":346,"num_osds":12,"num_up_osds":12,"osd_up_since":1726589558,"num_in_osds":11,"osd_in_since":1726589860,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+remapped","count":2}],"num_pgs":17,"num_pools":2,"num_objects":89569,"data_bytes":5870453280,"bytes_used":19771232256,"bytes_avail":1036174159872,"bytes_total":1055945392128,"misplaced_objects":8085,"misplaced_total":268707,"misplaced_ratio":0.030088535095847899,"recovering_objects_per_sec":107,"recovering_bytes_per_sec":7032941,"recovering_keys_per_sec":0,"num_objects_recovered":644,"num_bytes_recovered":42205184,"num_keys_recovered":0,"write_bytes_sec":61844015,"read_op_per_sec":0,"write_op_per_sec":943},"fsmap":{"epoch":1,"btime":"2024-09-17T16:02:46:978559+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.17:8443/"}},"servicemap":{"epoch":22,"modified":"2024-09-17T16:18:45.828885+0000","services":{}},"progress_events":{}} 2024-09-17T16:19:01.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:01 smithi173 ceph-mon[30530]: 4.a scrub starts 2024-09-17T16:19:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:01 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:19:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:01 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:19:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:01 smithi173 ceph-mon[30530]: 4.5 scrub ok 2024-09-17T16:19:01.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:01.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:01.911 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:19:01.911 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:19:01.942 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:19:01.942 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:19:01.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:19:01.974 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:19:02.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:02.006 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:19:02.006 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:19:02.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:02.036 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:02.038 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:19:02.038 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:19:02.065 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:19:02.065 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:19:02.089 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:19:02.090 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:19:02.127 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:19:02.127 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:19:02.137 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:02.165 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:19:02.165 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:19:02.172 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:02.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:02.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:02.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:02.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:02.397 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:02.425 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:02.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:02.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:02.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 94171 94155 72.6444 56.25 0.0227379 0.0137599 2024-09-17T16:19:02.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:02.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:02.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:02.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:19:02.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:02.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:02.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:02.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:02.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:02 smithi017 ceph-mon[27571]: 4.9 scrub starts 2024-09-17T16:19:02.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:02 smithi017 ceph-mon[27571]: osdmap e346: 12 total, 12 up, 11 in 2024-09-17T16:19:02.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/997845582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:02.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:02 smithi017 ceph-mon[27571]: pgmap v792: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 13 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s; 7833/277017 objects misplaced (2.828%); 7.2 MiB/s, 115 objects/s recovering 2024-09-17T16:19:02.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1932967877' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:19:02.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:02 smithi017 ceph-mon[27571]: 4.a scrub ok 2024-09-17T16:19:02.783 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:02.810 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:02.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:02 smithi071 ceph-mon[30002]: 4.9 scrub starts 2024-09-17T16:19:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:02 smithi071 ceph-mon[30002]: osdmap e346: 12 total, 12 up, 11 in 2024-09-17T16:19:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/997845582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:02 smithi071 ceph-mon[30002]: pgmap v792: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 13 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s; 7833/277017 objects misplaced (2.828%); 7.2 MiB/s, 115 objects/s recovering 2024-09-17T16:19:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1932967877' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:19:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:02 smithi071 ceph-mon[30002]: 4.a scrub ok 2024-09-17T16:19:02.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:02 smithi173 ceph-mon[30530]: 4.9 scrub starts 2024-09-17T16:19:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:02 smithi173 ceph-mon[30530]: osdmap e346: 12 total, 12 up, 11 in 2024-09-17T16:19:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/997845582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:02 smithi173 ceph-mon[30530]: pgmap v792: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 13 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s; 7833/277017 objects misplaced (2.828%); 7.2 MiB/s, 115 objects/s recovering 2024-09-17T16:19:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1932967877' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:19:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:02 smithi173 ceph-mon[30530]: 4.a scrub ok 2024-09-17T16:19:02.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:02.940 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:02.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:03.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:03.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:03.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:03.153 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-17T16:19:03.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:19:03.170 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:03.198 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:03.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:03.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:03.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:03.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:03.431 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:03.459 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:03.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:03.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 95183 95167 72.5298 63.25 0.0244649 0.0137832 2024-09-17T16:19:03.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:03.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:03.595 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:03.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:03.697 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:03.715 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:03 smithi071 ceph-mon[30002]: 4.9 scrub ok 2024-09-17T16:19:03.726 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:03.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:03 smithi017 ceph-mon[27571]: 4.9 scrub ok 2024-09-17T16:19:03.828 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:03.857 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:03.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:03.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:03 smithi173 ceph-mon[30530]: 4.9 scrub ok 2024-09-17T16:19:03.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:03.985 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:03.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:03.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:04.039 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:04.087 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:04.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:04.120 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:04.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:04.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:04.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:04.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:04.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:04.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:04.381 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:04.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:04.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:04 smithi017 ceph-mon[27571]: pgmap v793: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 13 active+clean; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 48 MiB/s wr, 762 op/s; 7833/278481 objects misplaced (2.813%); 4.3 MiB/s, 69 objects/s recovering 2024-09-17T16:19:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:04 smithi017 ceph-mon[27571]: osdmap e347: 12 total, 12 up, 11 in 2024-09-17T16:19:04.515 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:04.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:04.529 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 96105 96089 72.3502 57.625 0.0214022 0.0138168 2024-09-17T16:19:04.609 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:04.609 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:04.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:04.632 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:19:04.632 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:19:04.632 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:19:04.632 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:19:04.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:04.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:04.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:04.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:04.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:04.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:04.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:04 smithi071 ceph-mon[30002]: pgmap v793: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 13 active+clean; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 48 MiB/s wr, 762 op/s; 7833/278481 objects misplaced (2.813%); 4.3 MiB/s, 69 objects/s recovering 2024-09-17T16:19:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:04 smithi071 ceph-mon[30002]: osdmap e347: 12 total, 12 up, 11 in 2024-09-17T16:19:04.870 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:04.891 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:04.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:04 smithi173 ceph-mon[30530]: pgmap v793: 17 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 13 active+clean; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 48 MiB/s wr, 762 op/s; 7833/278481 objects misplaced (2.813%); 4.3 MiB/s, 69 objects/s recovering 2024-09-17T16:19:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:04 smithi173 ceph-mon[30530]: osdmap e347: 12 total, 12 up, 11 in 2024-09-17T16:19:04.993 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:05.021 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:05.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:05.122 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:05.154 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:05.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:05.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:05.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:05.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:05.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:05.384 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:05.410 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:05.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:05.454 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:19:05.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:05.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 96864 96848 72.0535 47.4375 0.00614047 0.0138735 2024-09-17T16:19:05.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:05.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:05.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:05.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:05.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:05.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:05.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:05.801 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:05.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:05.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:05.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:05.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:05.981 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:19:06.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:06.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:06.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:06.069 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:06.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:06.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:06.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:06.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:06.315 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:06.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:06.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:06.447 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:06.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:06.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 97693 97677 71.8153 51.8125 0.0297096 0.0139196 2024-09-17T16:19:06.548 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:06.581 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:06.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:06.683 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:06.704 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:06.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:06.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:06.838 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:06.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:06.896 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:19:06.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:06.967 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:06.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:06.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:06.976 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:19:06.976 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:19:06.976 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:19:06.976 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:19:06.976 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:19:07.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:07.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:07.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:06 smithi173 ceph-mon[30530]: pgmap v795: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 14 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s; 7657/285942 objects misplaced (2.678%); 6.0 MiB/s, 96 objects/s recovering 2024-09-17T16:19:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:06 smithi173 ceph-mon[30530]: 4.0 scrub starts 2024-09-17T16:19:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:06 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:19:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:06 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:19:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/308162086' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:19:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:06 smithi173 ceph-mon[30530]: 4.1 scrub starts 2024-09-17T16:19:07.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:07.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:07.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:06 smithi017 ceph-mon[27571]: pgmap v795: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 14 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s; 7657/285942 objects misplaced (2.678%); 6.0 MiB/s, 96 objects/s recovering 2024-09-17T16:19:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:06 smithi017 ceph-mon[27571]: 4.0 scrub starts 2024-09-17T16:19:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:06 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:19:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:06 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:19:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/308162086' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:19:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:06 smithi017 ceph-mon[27571]: 4.1 scrub starts 2024-09-17T16:19:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:06 smithi071 ceph-mon[30002]: pgmap v795: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 14 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s; 7657/285942 objects misplaced (2.678%); 6.0 MiB/s, 96 objects/s recovering 2024-09-17T16:19:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:06 smithi071 ceph-mon[30002]: 4.0 scrub starts 2024-09-17T16:19:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:06 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:19:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:06 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:19:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/308162086' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:19:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:06 smithi071 ceph-mon[30002]: 4.1 scrub starts 2024-09-17T16:19:07.323 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:07.348 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:07.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:07.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:07.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:07.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:07.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 98548 98532 71.6016 53.4375 0.00664366 0.0139604 2024-09-17T16:19:07.579 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:07.614 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:07.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:07.715 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:07.748 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:07.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:07.850 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:07.855 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:19:07.856 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:19:07.878 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:07.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:07.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:08.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:08.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:08.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:08.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:08.135 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:08.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:08.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:08.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:07 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:08.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/308162086' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:19:08.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:07 smithi173 ceph-mon[30530]: osdmap e348: 12 total, 12 up, 11 in 2024-09-17T16:19:08.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:07 smithi173 ceph-mon[30530]: from='client.16107 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:08.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:07 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:08.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/308162086' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:19:08.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:07 smithi017 ceph-mon[27571]: osdmap e348: 12 total, 12 up, 11 in 2024-09-17T16:19:08.195 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:07 smithi017 ceph-mon[27571]: from='client.16107 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:08.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:08.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:08.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:08.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:08.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:07 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/308162086' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:19:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:07 smithi071 ceph-mon[30002]: osdmap e348: 12 total, 12 up, 11 in 2024-09-17T16:19:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:07 smithi071 ceph-mon[30002]: from='client.16107 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:08.364 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:08.389 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:08.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:08.491 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:08.515 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:08.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:08.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:08.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 99520 99504 71.4768 60.75 0.0163848 0.0139849 2024-09-17T16:19:08.617 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:08.644 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:08.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:08.683 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":796,"stamp":"2024-09-17T16:19:06.896670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6334513696,"num_objects":96650,"num_object_clones":0,"num_object_copies":289950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7400,"num_objects_unfound":0,"num_objects_dirty":96650,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":96771,"num_write_kb":6187622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":137770592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96740,"ondisk_log_size":96740,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":20707944,"kb_used_data":18544280,"kb_used_omap":57,"kb_used_meta":2163398,"kb_avail":1010488728,"statfs":{"total":1055945392128,"available":1034740457472,"internally_reserved":0,"allocated":18989342720,"data_stored":18982146874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2215320286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":464060416,"num_objects":7081,"num_object_clones":0,"num_object_copies":21243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-685,"num_objects_unfound":0,"num_objects_dirty":7081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":453184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":685,"num_bytes_recovered":44892160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.065139"},"pg_stats":[{"pgid":"4.f","version":"347'6038","reported_seq":6130,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:06.393102+0000","last_change":"2024-09-17T16:17:41.531577+0000","last_active":"2024-09-17T16:19:06.393102+0000","last_peered":"2024-09-17T16:19:06.393102+0000","last_clean":"2024-09-17T16:19:06.393102+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:19:06.393102+0000","last_undegraded":"2024-09-17T16:19:06.393102+0000","last_fullsized":"2024-09-17T16:19:06.393102+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395706368,"num_objects":6038,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":386432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"347'5986","reported_seq":6088,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:04.613186+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:19:04.613186+0000","last_peered":"2024-09-17T16:19:04.613186+0000","last_clean":"2024-09-17T16:19:04.613186+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:19:04.613186+0000","last_undegraded":"2024-09-17T16:19:04.613186+0000","last_fullsized":"2024-09-17T16:19:04.613186+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5986,"log_dups_size":0,"ondisk_log_size":5986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392298496,"num_objects":5986,"num_object_clones":0,"num_object_copies":17958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":383104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"347'5852","reported_seq":5960,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:04.578307+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:19:04.578307+0000","last_peered":"2024-09-17T16:19:04.578307+0000","last_clean":"2024-09-17T16:19:04.578307+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:19:04.578307+0000","last_undegraded":"2024-09-17T16:19:04.578307+0000","last_fullsized":"2024-09-17T16:19:04.578307+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5852,"log_dups_size":0,"ondisk_log_size":5852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383516672,"num_objects":5852,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":374528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"347'6148","reported_seq":6250,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:04.615175+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:19:04.615175+0000","last_peered":"2024-09-17T16:19:04.615175+0000","last_clean":"2024-09-17T16:19:04.615175+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:19:04.615175+0000","last_undegraded":"2024-09-17T16:19:04.615175+0000","last_fullsized":"2024-09-17T16:19:04.615175+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402915328,"num_objects":6148,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"347'6005","reported_seq":7789,"reported_epoch":347,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:04.624982+0000","last_change":"2024-09-17T16:18:41.641380+0000","last_active":"2024-09-17T16:19:04.624982+0000","last_peered":"2024-09-17T16:19:04.624982+0000","last_clean":"2024-09-17T16:18:39.384812+0000","last_became_active":"2024-09-17T16:18:41.595599+0000","last_became_peered":"2024-09-17T16:18:41.595599+0000","last_unstale":"2024-09-17T16:19:04.624982+0000","last_undegraded":"2024-09-17T16:19:04.624982+0000","last_fullsized":"2024-09-17T16:18:41.426452+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6005,"log_dups_size":0,"ondisk_log_size":6005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393543680,"num_objects":6005,"num_object_clones":0,"num_object_copies":18015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3963,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":384320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":55312384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4","objects":2042},{"shards":"0,4,5","objects":3963}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"347'6148","reported_seq":6321,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:05.595608+0000","last_change":"2024-09-17T16:19:02.221681+0000","last_active":"2024-09-17T16:19:05.595608+0000","last_peered":"2024-09-17T16:19:05.595608+0000","last_clean":"2024-09-17T16:19:05.595608+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:19:05.595608+0000","last_undegraded":"2024-09-17T16:19:05.595608+0000","last_fullsized":"2024-09-17T16:19:05.595608+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5925","last_scrub_stamp":"2024-09-17T16:19:02.221472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.221472+0000","objects_scrubbed":5879,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.368910+0000","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402915328,"num_objects":6148,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"347'5990","reported_seq":6150,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:06.354791+0000","last_change":"2024-09-17T16:19:02.418259+0000","last_active":"2024-09-17T16:19:06.354791+0000","last_peered":"2024-09-17T16:19:06.354791+0000","last_clean":"2024-09-17T16:19:06.354791+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:19:06.354791+0000","last_undegraded":"2024-09-17T16:19:06.354791+0000","last_fullsized":"2024-09-17T16:19:06.354791+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5797","last_scrub_stamp":"2024-09-17T16:19:02.418176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.418176+0000","objects_scrubbed":5756,"log_size":5990,"log_dups_size":0,"ondisk_log_size":5990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:17.281193+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392560640,"num_objects":5990,"num_object_clones":0,"num_object_copies":17970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5990,"num_write_kb":383360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"346'5912","reported_seq":6012,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T16:19:02.888814+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:19:02.888814+0000","last_peered":"2024-09-17T16:19:02.888814+0000","last_clean":"2024-09-17T16:19:02.888814+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:19:02.888814+0000","last_undegraded":"2024-09-17T16:19:02.888814+0000","last_fullsized":"2024-09-17T16:19:02.888814+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387448832,"num_objects":5912,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5912,"num_write_kb":378368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"347'6038","reported_seq":6147,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:05.710152+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:19:05.710152+0000","last_peered":"2024-09-17T16:19:05.710152+0000","last_clean":"2024-09-17T16:19:05.710152+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:19:05.710152+0000","last_undegraded":"2024-09-17T16:19:05.710152+0000","last_fullsized":"2024-09-17T16:19:05.710152+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:14.213522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395706368,"num_objects":6038,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":386432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"347'5983","reported_seq":8543,"reported_epoch":347,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:06.722578+0000","last_change":"2024-09-17T16:18:42.321603+0000","last_active":"2024-09-17T16:19:06.722578+0000","last_peered":"2024-09-17T16:19:06.722578+0000","last_clean":"2024-09-17T16:18:39.337421+0000","last_became_active":"2024-09-17T16:18:42.219868+0000","last_became_peered":"2024-09-17T16:18:42.219868+0000","last_unstale":"2024-09-17T16:19:06.722578+0000","last_undegraded":"2024-09-17T16:19:06.722578+0000","last_fullsized":"2024-09-17T16:18:41.426290+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5983,"log_dups_size":0,"ondisk_log_size":5983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392101888,"num_objects":5983,"num_object_clones":0,"num_object_copies":17949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3437,"num_objects_unfound":0,"num_objects_dirty":5983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5983,"num_write_kb":382912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":80805888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[2,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":3437},{"shards":"2,7","objects":2546}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"347'6115","reported_seq":6208,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:06.688665+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:19:06.688665+0000","last_peered":"2024-09-17T16:19:06.688665+0000","last_clean":"2024-09-17T16:19:06.688665+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:19:06.688665+0000","last_undegraded":"2024-09-17T16:19:06.688665+0000","last_fullsized":"2024-09-17T16:19:06.688665+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6115,"log_dups_size":0,"ondisk_log_size":6115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400752640,"num_objects":6115,"num_object_clones":0,"num_object_copies":18345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6115,"num_write_kb":391360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"346'6022","reported_seq":6126,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T16:19:02.932410+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:19:02.932410+0000","last_peered":"2024-09-17T16:19:02.932410+0000","last_clean":"2024-09-17T16:19:02.932410+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:19:02.932410+0000","last_undegraded":"2024-09-17T16:19:02.932410+0000","last_fullsized":"2024-09-17T16:19:02.932410+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6022,"log_dups_size":0,"ondisk_log_size":6022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394657792,"num_objects":6022,"num_object_clones":0,"num_object_copies":18066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6022,"num_write_kb":385408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"347'5963","reported_seq":6075,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:05.461084+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:19:05.461084+0000","last_peered":"2024-09-17T16:19:05.461084+0000","last_clean":"2024-09-17T16:19:05.461084+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:19:05.461084+0000","last_undegraded":"2024-09-17T16:19:05.461084+0000","last_fullsized":"2024-09-17T16:19:05.461084+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":5963,"log_dups_size":0,"ondisk_log_size":5963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:44.325955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390791168,"num_objects":5963,"num_object_clones":0,"num_object_copies":17889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5963,"num_write_kb":381632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"347'6112","reported_seq":6215,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:05.489503+0000","last_change":"2024-09-17T16:17:34.582545+0000","last_active":"2024-09-17T16:19:05.489503+0000","last_peered":"2024-09-17T16:19:05.489503+0000","last_clean":"2024-09-17T16:19:05.489503+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:19:05.489503+0000","last_undegraded":"2024-09-17T16:19:05.489503+0000","last_fullsized":"2024-09-17T16:19:05.489503+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6112,"log_dups_size":0,"ondisk_log_size":6112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400556032,"num_objects":6112,"num_object_clones":0,"num_object_copies":18336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6112,"num_write_kb":391168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"347'6101","reported_seq":6220,"reported_epoch":347,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T16:19:06.392639+0000","last_change":"2024-09-17T16:19:06.065248+0000","last_active":"2024-09-17T16:19:06.392639+0000","last_peered":"2024-09-17T16:19:06.392639+0000","last_clean":"2024-09-17T16:19:06.392639+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:19:06.392639+0000","last_undegraded":"2024-09-17T16:19:06.392639+0000","last_fullsized":"2024-09-17T16:19:06.392639+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":6101,"log_dups_size":0,"ondisk_log_size":6101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399835136,"num_objects":6101,"num_object_clones":0,"num_object_copies":18303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6101,"num_write_kb":390464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"347'6235","reported_seq":6405,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-09-17T16:19:05.615522+0000","last_change":"2024-09-17T16:19:01.159281+0000","last_active":"2024-09-17T16:19:05.615522+0000","last_peered":"2024-09-17T16:19:05.615522+0000","last_clean":"2024-09-17T16:19:05.615522+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:19:05.615522+0000","last_undegraded":"2024-09-17T16:19:05.615522+0000","last_fullsized":"2024-09-17T16:19:05.615522+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5950","last_scrub_stamp":"2024-09-17T16:19:01.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:01.159158+0000","objects_scrubbed":5911,"log_size":6235,"log_dups_size":0,"ondisk_log_size":6235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:27.590602+0000","scrub_duration":1399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408616960,"num_objects":6235,"num_object_clones":0,"num_object_copies":18705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6235,"num_write_kb":399040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":585,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-09-17T16:19:01.435442+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:19:01.435442+0000","last_peered":"2024-09-17T16:19:01.435442+0000","last_clean":"2024-09-17T16:19:01.435442+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:19:01.435442+0000","last_undegraded":"2024-09-17T16:19:01.435442+0000","last_fullsized":"2024-09-17T16:19:01.435442+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":6333923328,"num_objects":96648,"num_object_clones":0,"num_object_copies":289944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7400,"num_objects_unfound":0,"num_objects_dirty":96648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96648,"num_write_kb":6185472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2078,"num_bytes_recovered":136183808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19050397696,"data_stored":19050397696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96648,"ondisk_log_size":96648,"up":48,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285683,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3345644,"kb_used_data":3075672,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":90399508,"statfs":{"total":95995035648,"available":92569096192,"internally_reserved":0,"allocated":3149488128,"data_stored":3148829726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":276429252},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547320,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2076772,"kb_used_data":1942864,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":91668380,"statfs":{"total":95995035648,"available":93868421120,"internally_reserved":0,"allocated":1989492736,"data_stored":1988837906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":137099712},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743548,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1654424,"kb_used_data":1344196,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92090728,"statfs":{"total":95995035648,"available":94300905472,"internally_reserved":0,"allocated":1376456704,"data_stored":1375806364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":317651392},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":123164,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93621988,"statfs":{"total":95995035648,"available":95868915712,"internally_reserved":0,"allocated":1773568,"data_stored":1123228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963662,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2476060,"kb_used_data":2270732,"kb_used_omap":41,"kb_used_meta":205270,"kb_avail":91269092,"statfs":{"total":95995035648,"available":93459550208,"internally_reserved":0,"allocated":2325229568,"data_stored":2324575676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":210196787},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495756,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2910316,"kb_used_data":2722264,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":90834836,"statfs":{"total":95995035648,"available":93014872064,"internally_reserved":0,"allocated":2787598336,"data_stored":2786939954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":192543174},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724687,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1592160,"kb_used_data":1380496,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92152992,"statfs":{"total":95995035648,"available":94364663808,"internally_reserved":0,"allocated":1413627904,"data_stored":1412973074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986326,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1716308,"kb_used_data":1532420,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92028844,"statfs":{"total":95995035648,"available":94237536256,"internally_reserved":0,"allocated":1569198080,"data_stored":1568547740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2167780,"kb_used_data":1947792,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":91577372,"statfs":{"total":95995035648,"available":93775228928,"internally_reserved":0,"allocated":1994539008,"data_stored":1993884178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":225245639},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411488,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902496,"kb_used_data":772752,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92842656,"statfs":{"total":95995035648,"available":95070879744,"internally_reserved":0,"allocated":791298048,"data_stored":790643218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1742820,"kb_used_data":1553360,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92002332,"statfs":{"total":95995035648,"available":94210387968,"internally_reserved":0,"allocated":1590640640,"data_stored":1589985810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1604321280,"data_stored":1604321280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":792854528,"data_stored":792854528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1996881920,"data_stored":1996881920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1572339712,"data_stored":1572339712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1412497408,"data_stored":1412497408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2802188288,"data_stored":2802188288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2326331392,"data_stored":2326331392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1392508928,"data_stored":1392508928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2001928192,"data_stored":2001928192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3148546048,"data_stored":3148546048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:19:08.747 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:08.771 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:08.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:08.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:08.873 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:08.898 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:08 smithi017 ceph-mon[27571]: pgmap v797: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 14 active+clean; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 47 MiB/s wr, 756 op/s; 7400/289950 objects misplaced (2.552%); 6.7 MiB/s, 106 objects/s recovering 2024-09-17T16:19:08.898 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:08 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:19:08.898 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:08 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:19:08.900 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:08.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:09.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:09.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:09.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:09.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:09.158 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:09.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:09.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:09.274 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:08 smithi173 ceph-mon[30530]: pgmap v797: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 14 active+clean; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 47 MiB/s wr, 756 op/s; 7400/289950 objects misplaced (2.552%); 6.7 MiB/s, 106 objects/s recovering 2024-09-17T16:19:09.274 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:08 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:19:09.274 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:08 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:19:09.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:09.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:09.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:08 smithi071 ceph-mon[30002]: pgmap v797: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 14 active+clean; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 47 MiB/s wr, 756 op/s; 7400/289950 objects misplaced (2.552%); 6.7 MiB/s, 106 objects/s recovering 2024-09-17T16:19:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:08 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:19:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:08 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:19:09.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:09.427 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:09.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:09.529 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:09.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 100511 100495 71.3684 61.9375 0.0310003 0.0140073 2024-09-17T16:19:09.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:09.557 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:09.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:09.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:09.680 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:09.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:09.769 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:19:09.771 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:19:09.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:09.843 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:09.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:09.846 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:09.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:09.965 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:09.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:09.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:09 smithi017 ceph-mon[27571]: osdmap e349: 12 total, 12 up, 11 in 2024-09-17T16:19:10.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:10.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:10.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:10.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:10.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:10.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:09 smithi071 ceph-mon[30002]: osdmap e349: 12 total, 12 up, 11 in 2024-09-17T16:19:10.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:10.363 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:10.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:09 smithi173 ceph-mon[30530]: osdmap e349: 12 total, 12 up, 11 in 2024-09-17T16:19:10.465 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:10.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:10.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:10.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 101141 101125 71.0088 39.375 0.00519404 0.014078 2024-09-17T16:19:10.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:10.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:10.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:10.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:10.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:10.755 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:10.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:10.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:10.880 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:10.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:10.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:11.014 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:11.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:11.116 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:11.144 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:11.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:10 smithi173 ceph-mon[30530]: pgmap v799: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 14 active+clean; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 57 MiB/s wr, 914 op/s; 7191/294948 objects misplaced (2.438%); 6.7 MiB/s, 106 objects/s recovering 2024-09-17T16:19:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:11.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:10 smithi017 ceph-mon[27571]: pgmap v799: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 14 active+clean; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 57 MiB/s wr, 914 op/s; 7191/294948 objects misplaced (2.438%); 6.7 MiB/s, 106 objects/s recovering 2024-09-17T16:19:11.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:11.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:11.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:11.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:10 smithi071 ceph-mon[30002]: pgmap v799: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 14 active+clean; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 57 MiB/s wr, 914 op/s; 7191/294948 objects misplaced (2.438%); 6.7 MiB/s, 106 objects/s recovering 2024-09-17T16:19:11.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:11.398 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:11.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:11.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:11.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:11.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:11.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 9 102172 102163 70.9406 64.875 0.0105203 0.0140907 2024-09-17T16:19:11.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:11.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:11.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:11.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:11.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:19:11.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:11.773 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:11.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:11.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:11.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:11.897 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:11.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:11.999 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:12.020 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:12.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:12.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:12.144 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:12.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:12.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:12.267 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:12.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:12.370 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:12.391 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:12.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:12.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:12.516 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:12.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:12.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.8898 2024-09-17T16:19:12.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 102172 2024-09-17T16:19:12.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-09-17T16:19:12.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-09-17T16:19:12.530 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 70.2582 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 24.6381 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 155.625 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0.1875 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1124 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 394.209 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2490 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 3 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0141205 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.0384201 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 2.86134 2024-09-17T16:19:12.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00106175 2024-09-17T16:19:12.535 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T16:19:12.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:12.650 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:12.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:12.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:12.782 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:12.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:12.885 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:12.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:12.923 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:12.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:13.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:13.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:13.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:13.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:13.116 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:12 smithi017 ceph-mon[27571]: pgmap v800: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 6934/303147 objects misplaced (2.287%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:19:13.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:13.185 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:13.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:13.276 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:19:13.276 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:19:13.276 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:19:13.277 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:19:13.277 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:19:13.277 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:19:13.286 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:13.296 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:19:13.297 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:19:13.309 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:13.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:13.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:12 smithi071 ceph-mon[30002]: pgmap v800: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 6934/303147 objects misplaced (2.287%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:19:13.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:19:13.321 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:19:13.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:19:13.348 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:19:13.375 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:19:13.375 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:19:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:12 smithi173 ceph-mon[30530]: pgmap v800: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s; 6934/303147 objects misplaced (2.287%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:19:13.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:13.413 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:19:13.413 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:19:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:13.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:13.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:13.443 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:13.448 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:19:13.449 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:13.449 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:19:13.450 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:13.485 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:19:13.485 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:19:13.515 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:19:13.516 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:19:13.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:13.543 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:19:13.543 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:19:13.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:13.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:13.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:13.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:13.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:13.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:13.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:13.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:13.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:13.820 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:13.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:13.922 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:13.958 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:13.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:14.060 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:14.097 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:14.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:14.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3120516852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:14.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:14.227 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:14.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:14.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3120516852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:14.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:14.356 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:14.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:14.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3120516852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:14.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:14.485 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:14.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:14.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:14.615 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:14.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:14.717 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:14.744 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:14.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:14.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:14.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:14.870 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:14.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:14.972 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:14.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:14.994 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:14.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:15.095 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:15.117 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:15.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:15.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:15.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:15.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:15.348 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:15.376 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:15.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:15.479 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:15.512 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:15.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:15.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:15.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:15.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:15.653 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:19:15.653 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:19:15.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:15 smithi017 ceph-mon[27571]: pgmap v801: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 48 MiB/s wr, 772 op/s; 6934/304047 objects misplaced (2.281%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T16:19:15.739 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:15.765 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:15.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:15 smithi071 ceph-mon[30002]: pgmap v801: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 48 MiB/s wr, 772 op/s; 6934/304047 objects misplaced (2.281%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T16:19:15.867 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:15.891 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:15.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:15 smithi173 ceph-mon[30530]: pgmap v801: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 48 MiB/s wr, 772 op/s; 6934/304047 objects misplaced (2.281%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T16:19:15.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:16.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:16.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:16.121 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:16.150 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:16.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:16.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:16.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:16.278 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:16.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:16.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:16.407 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:19:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:16.408 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:16.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:16.508 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:19:16.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:16.538 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:16.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:16.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:16.640 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:16.662 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:16.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:16.722 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:19:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2401836230' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:16 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:16.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:16.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:16.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2401836230' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:16 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:16.889 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:16.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2401836230' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:16 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:16.920 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:16.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:16.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:17.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:17.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:17.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:17.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:17.152 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:19:17.153 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:19:17.177 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:17.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:17.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:17.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:17.312 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:17.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:17.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:17.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:17.439 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:17.540 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:17.569 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:17.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:17.671 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:17.695 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:17.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:17 smithi017 ceph-mon[27571]: pgmap v802: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 114 B/s rd, 38 MiB/s wr, 782 op/s; 6624/301974 objects misplaced (2.194%); 5.4 MiB/s, 86 objects/s recovering 2024-09-17T16:19:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:17 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:17 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:19:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:17 smithi017 ceph-mon[27571]: osdmap e350: 12 total, 12 up, 11 in 2024-09-17T16:19:17.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:17.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:17 smithi071 ceph-mon[30002]: pgmap v802: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 114 B/s rd, 38 MiB/s wr, 782 op/s; 6624/301974 objects misplaced (2.194%); 5.4 MiB/s, 86 objects/s recovering 2024-09-17T16:19:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:17 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:17 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:19:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:17 smithi071 ceph-mon[30002]: osdmap e350: 12 total, 12 up, 11 in 2024-09-17T16:19:17.824 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:17.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:17 smithi173 ceph-mon[30530]: pgmap v802: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 114 B/s rd, 38 MiB/s wr, 782 op/s; 6624/301974 objects misplaced (2.194%); 5.4 MiB/s, 86 objects/s recovering 2024-09-17T16:19:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:17 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:17 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:19:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:17 smithi173 ceph-mon[30530]: osdmap e350: 12 total, 12 up, 11 in 2024-09-17T16:19:17.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:17.956 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:17.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:18.057 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:18.085 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:18.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:18.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:18.216 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:18.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:18.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:18.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:18.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:18.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:18.447 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:18.475 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:18.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:18.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:18.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:18.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:18.701 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:18.723 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:18.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:18.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:18 smithi017 ceph-mon[27571]: from='client.25771 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:18.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:19:18.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:18 smithi071 ceph-mon[30002]: from='client.25771 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:19:18.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:18.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:18.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:18 smithi173 ceph-mon[30530]: from='client.25771 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:19:18.950 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:18.973 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:18.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:19.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:19.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:19.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:19.201 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:19.226 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:19.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:19.328 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:19.353 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:19.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:19.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:19.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:19.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:19.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:19.571 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:19:19.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:19.616 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:19.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:19.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:19 smithi017 ceph-mon[27571]: pgmap v804: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 114 B/s rd, 35 MiB/s wr, 856 op/s; 6246/298425 objects misplaced (2.093%); 8.1 MiB/s, 129 objects/s recovering 2024-09-17T16:19:19.750 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:19.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:19.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:19 smithi071 ceph-mon[30002]: pgmap v804: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 114 B/s rd, 35 MiB/s wr, 856 op/s; 6246/298425 objects misplaced (2.093%); 8.1 MiB/s, 129 objects/s recovering 2024-09-17T16:19:19.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:19.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:19.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:19 smithi173 ceph-mon[30530]: pgmap v804: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 114 B/s rd, 35 MiB/s wr, 856 op/s; 6246/298425 objects misplaced (2.093%); 8.1 MiB/s, 129 objects/s recovering 2024-09-17T16:19:19.987 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:20.013 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:20.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:20.115 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:20.140 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:20.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:20.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:20.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:20.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:20.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:20.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:20.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:20.428 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:20.462 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:20.463 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":803,"stamp":"2024-09-17T16:19:16.402205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6597116472,"num_objects":100658,"num_object_clones":0,"num_object_copies":301974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6624,"num_objects_unfound":0,"num_objects_dirty":100658,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":103761,"num_write_kb":6539495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2860,"num_bytes_recovered":188626528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103730,"ondisk_log_size":103730,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":21664332,"kb_used_data":19507608,"kb_used_omap":57,"kb_used_meta":2156486,"kb_avail":1009532340,"statfs":{"total":1055945392128,"available":1033761116160,"internally_reserved":0,"allocated":19975790592,"data_stored":19968470962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2208242398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,7,3,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":262602776,"num_objects":4008,"num_object_clones":0,"num_object_copies":12024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-776,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6990,"num_write_kb":351873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":776,"num_bytes_recovered":50855936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.569073"},"pg_stats":[{"pgid":"4.f","version":"349'6349","reported_seq":6445,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:11.304169+0000","last_change":"2024-09-17T16:17:41.531577+0000","last_active":"2024-09-17T16:19:11.304169+0000","last_peered":"2024-09-17T16:19:11.304169+0000","last_clean":"2024-09-17T16:19:11.304169+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:19:11.304169+0000","last_undegraded":"2024-09-17T16:19:11.304169+0000","last_fullsized":"2024-09-17T16:19:11.304169+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.077147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416088064,"num_objects":6349,"num_object_clones":0,"num_object_copies":19047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6349,"num_write_kb":406336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"349'6498","reported_seq":6604,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:14.635112+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:19:14.635112+0000","last_peered":"2024-09-17T16:19:14.635112+0000","last_clean":"2024-09-17T16:19:14.635112+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:19:14.635112+0000","last_undegraded":"2024-09-17T16:19:14.635112+0000","last_fullsized":"2024-09-17T16:19:14.635112+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6498,"log_dups_size":0,"ondisk_log_size":6498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408158208,"num_objects":6228,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6498,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"349'6347","reported_seq":6461,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:14.620341+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:19:14.620341+0000","last_peered":"2024-09-17T16:19:14.620341+0000","last_clean":"2024-09-17T16:19:14.620341+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:19:14.620341+0000","last_undegraded":"2024-09-17T16:19:14.620341+0000","last_fullsized":"2024-09-17T16:19:14.620341+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399769624,"num_objects":6101,"num_object_clones":0,"num_object_copies":18303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6347,"num_write_kb":398273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"349'6637","reported_seq":6743,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:14.646247+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:19:14.646247+0000","last_peered":"2024-09-17T16:19:14.646247+0000","last_clean":"2024-09-17T16:19:14.646247+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:19:14.646247+0000","last_undegraded":"2024-09-17T16:19:14.646247+0000","last_fullsized":"2024-09-17T16:19:14.646247+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6637,"log_dups_size":0,"ondisk_log_size":6637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418185216,"num_objects":6381,"num_object_clones":0,"num_object_copies":19143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6637,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"349'6548","reported_seq":9373,"reported_epoch":349,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:14.627483+0000","last_change":"2024-09-17T16:18:41.641380+0000","last_active":"2024-09-17T16:19:14.627483+0000","last_peered":"2024-09-17T16:19:14.627483+0000","last_clean":"2024-09-17T16:18:39.384812+0000","last_became_active":"2024-09-17T16:18:41.595599+0000","last_became_peered":"2024-09-17T16:18:41.595599+0000","last_unstale":"2024-09-17T16:19:14.627483+0000","last_undegraded":"2024-09-17T16:19:14.627483+0000","last_fullsized":"2024-09-17T16:18:41.426452+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6548,"log_dups_size":0,"ondisk_log_size":6548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409206784,"num_objects":6244,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3444,"num_objects_unfound":0,"num_objects_dirty":6244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6548,"num_write_kb":409344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1363,"num_bytes_recovered":89325568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4","objects":2800},{"shards":"0,4,5","objects":3444}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"349'6667","reported_seq":6844,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:15.595816+0000","last_change":"2024-09-17T16:19:02.221681+0000","last_active":"2024-09-17T16:19:15.595816+0000","last_peered":"2024-09-17T16:19:15.595816+0000","last_clean":"2024-09-17T16:19:15.595816+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:19:15.595816+0000","last_undegraded":"2024-09-17T16:19:15.595816+0000","last_fullsized":"2024-09-17T16:19:15.595816+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5925","last_scrub_stamp":"2024-09-17T16:19:02.221472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.221472+0000","objects_scrubbed":5879,"log_size":6667,"log_dups_size":0,"ondisk_log_size":6667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.368910+0000","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410714112,"num_objects":6267,"num_object_clones":0,"num_object_copies":18801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6667,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"349'6279","reported_seq":6442,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:11.377691+0000","last_change":"2024-09-17T16:19:02.418259+0000","last_active":"2024-09-17T16:19:11.377691+0000","last_peered":"2024-09-17T16:19:11.377691+0000","last_clean":"2024-09-17T16:19:11.377691+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:19:11.377691+0000","last_undegraded":"2024-09-17T16:19:11.377691+0000","last_fullsized":"2024-09-17T16:19:11.377691+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5797","last_scrub_stamp":"2024-09-17T16:19:02.418176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.418176+0000","objects_scrubbed":5756,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:17.281193+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411500544,"num_objects":6279,"num_object_clones":0,"num_object_copies":18837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6279,"num_write_kb":401856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"349'6412","reported_seq":6518,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:12.903073+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:19:12.903073+0000","last_peered":"2024-09-17T16:19:12.903073+0000","last_clean":"2024-09-17T16:19:12.903073+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:19:12.903073+0000","last_undegraded":"2024-09-17T16:19:12.903073+0000","last_fullsized":"2024-09-17T16:19:12.903073+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6412,"log_dups_size":0,"ondisk_log_size":6412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415891456,"num_objects":6346,"num_object_clones":0,"num_object_copies":19038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6412,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"349'6556","reported_seq":6670,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:15.745786+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:19:15.745786+0000","last_peered":"2024-09-17T16:19:15.745786+0000","last_clean":"2024-09-17T16:19:15.745786+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:19:15.745786+0000","last_undegraded":"2024-09-17T16:19:15.745786+0000","last_fullsized":"2024-09-17T16:19:15.745786+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":6556,"log_dups_size":0,"ondisk_log_size":6556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406192128,"num_objects":6198,"num_object_clones":0,"num_object_copies":18594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"349'6260","reported_seq":9339,"reported_epoch":349,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:11.524887+0000","last_change":"2024-09-17T16:18:42.321603+0000","last_active":"2024-09-17T16:19:11.524887+0000","last_peered":"2024-09-17T16:19:11.524887+0000","last_clean":"2024-09-17T16:18:39.337421+0000","last_became_active":"2024-09-17T16:18:42.219868+0000","last_became_peered":"2024-09-17T16:18:42.219868+0000","last_unstale":"2024-09-17T16:19:11.524887+0000","last_undegraded":"2024-09-17T16:19:11.524887+0000","last_fullsized":"2024-09-17T16:18:41.426290+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410255360,"num_objects":6260,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3180,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1490,"num_bytes_recovered":97648640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[2,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":3180},{"shards":"2,7","objects":3080}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"349'6392","reported_seq":6489,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:11.523936+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:19:11.523936+0000","last_peered":"2024-09-17T16:19:11.523936+0000","last_clean":"2024-09-17T16:19:11.523936+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:19:11.523936+0000","last_undegraded":"2024-09-17T16:19:11.523936+0000","last_fullsized":"2024-09-17T16:19:11.523936+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6392,"log_dups_size":0,"ondisk_log_size":6392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418906112,"num_objects":6392,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6392,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"349'6543","reported_seq":6656,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:12.949003+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:19:12.949003+0000","last_peered":"2024-09-17T16:19:12.949003+0000","last_clean":"2024-09-17T16:19:12.949003+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:19:12.949003+0000","last_undegraded":"2024-09-17T16:19:12.949003+0000","last_fullsized":"2024-09-17T16:19:12.949003+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423297024,"num_objects":6459,"num_object_clones":0,"num_object_copies":19377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6543,"num_write_kb":416064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"349'6435","reported_seq":6550,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:15.494699+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:19:15.494699+0000","last_peered":"2024-09-17T16:19:15.494699+0000","last_clean":"2024-09-17T16:19:15.494699+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:19:15.494699+0000","last_undegraded":"2024-09-17T16:19:15.494699+0000","last_fullsized":"2024-09-17T16:19:15.494699+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6435,"log_dups_size":0,"ondisk_log_size":6435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:44.325955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400883712,"num_objects":6117,"num_object_clones":0,"num_object_copies":18351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"349'6615","reported_seq":6738,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:15.493415+0000","last_change":"2024-09-17T16:19:06.911296+0000","last_active":"2024-09-17T16:19:15.493415+0000","last_peered":"2024-09-17T16:19:15.493415+0000","last_clean":"2024-09-17T16:19:15.493415+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:19:15.493415+0000","last_undegraded":"2024-09-17T16:19:15.493415+0000","last_fullsized":"2024-09-17T16:19:15.493415+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1124,"log_size":6615,"log_dups_size":0,"ondisk_log_size":6615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:36.906787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412811264,"num_objects":6299,"num_object_clones":0,"num_object_copies":18897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6615,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"349'6356","reported_seq":6499,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:11.318776+0000","last_change":"2024-09-17T16:19:06.928683+0000","last_active":"2024-09-17T16:19:11.318776+0000","last_peered":"2024-09-17T16:19:11.318776+0000","last_clean":"2024-09-17T16:19:11.318776+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:19:11.318776+0000","last_undegraded":"2024-09-17T16:19:11.318776+0000","last_fullsized":"2024-09-17T16:19:11.318776+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":3399,"log_size":6356,"log_dups_size":0,"ondisk_log_size":6356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:36.915694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416546816,"num_objects":6356,"num_object_clones":0,"num_object_copies":19068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6356,"num_write_kb":406784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"349'6744","reported_seq":6917,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:15.599182+0000","last_change":"2024-09-17T16:19:01.159281+0000","last_active":"2024-09-17T16:19:15.599182+0000","last_peered":"2024-09-17T16:19:15.599182+0000","last_clean":"2024-09-17T16:19:15.599182+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:19:15.599182+0000","last_undegraded":"2024-09-17T16:19:15.599182+0000","last_fullsized":"2024-09-17T16:19:15.599182+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5950","last_scrub_stamp":"2024-09-17T16:19:01.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:01.159158+0000","objects_scrubbed":5911,"log_size":6744,"log_dups_size":0,"ondisk_log_size":6744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:27.590602+0000","scrub_duration":1399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418119680,"num_objects":6380,"num_object_clones":0,"num_object_copies":19140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6744,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":591,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-17T16:19:08.907000+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:19:08.907000+0000","last_peered":"2024-09-17T16:19:08.907000+0000","last_clean":"2024-09-17T16:19:08.907000+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:19:08.907000+0000","last_undegraded":"2024-09-17T16:19:08.907000+0000","last_fullsized":"2024-09-17T16:19:08.907000+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":6596526104,"num_objects":100656,"num_object_clones":0,"num_object_copies":301968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6624,"num_objects_unfound":0,"num_objects_dirty":100656,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103638,"num_write_kb":6537345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2854,"num_bytes_recovered":187039744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19921248256,"data_stored":19921240112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103638,"ondisk_log_size":103638,"up":48,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285685,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3474436,"kb_used_data":3208628,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":90270716,"statfs":{"total":95995035648,"available":92437213184,"internally_reserved":0,"allocated":3285635072,"data_stored":3284963718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":272169412},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547322,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2136888,"kb_used_data":2011304,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91608264,"statfs":{"total":95995035648,"available":93806862336,"internally_reserved":0,"allocated":2059575296,"data_stored":2058911586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":128580032},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743550,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1884216,"kb_used_data":1578152,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91860936,"statfs":{"total":95995035648,"available":94065598464,"internally_reserved":0,"allocated":1616027648,"data_stored":1615363938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":313391552},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":123196,"kb_used_data":1768,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93621956,"statfs":{"total":95995035648,"available":95868882944,"internally_reserved":0,"allocated":1810432,"data_stored":1146722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963664,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2647552,"kb_used_data":2446384,"kb_used_omap":41,"kb_used_meta":201110,"kb_avail":91097600,"statfs":{"total":95995035648,"available":93283942400,"internally_reserved":0,"allocated":2505097216,"data_stored":2504429954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":205936947},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3023616,"kb_used_data":2821680,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":90721536,"statfs":{"total":95995035648,"available":92898852864,"internally_reserved":0,"allocated":2889400320,"data_stored":2888733058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206764486},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724689,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1619384,"kb_used_data":1403560,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92125768,"statfs":{"total":95995035648,"available":94336786432,"internally_reserved":0,"allocated":1437245440,"data_stored":1436581730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":220985798},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986328,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1780156,"kb_used_data":1604588,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91964996,"statfs":{"total":95995035648,"available":94172155904,"internally_reserved":0,"allocated":1643098112,"data_stored":1642430330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2245496,"kb_used_data":2033832,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":91499656,"statfs":{"total":95995035648,"available":93695647744,"internally_reserved":0,"allocated":2082643968,"data_stored":2081980258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216725959},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":943736,"kb_used_data":809832,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":92801416,"statfs":{"total":95995035648,"available":95028649984,"internally_reserved":0,"allocated":829267968,"data_stored":828604258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738540,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1785656,"kb_used_data":1587880,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91959496,"statfs":{"total":95995035648,"available":94166523904,"internally_reserved":0,"allocated":1625989120,"data_stored":1625325410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1620180992,"data_stored":1620180992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":829161472,"data_stored":829161472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2088632320,"data_stored":2088632320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1635127296,"data_stored":1635123224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1434714112,"data_stored":1434714112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2866020352,"data_stored":2866020352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2499608576,"data_stored":2499608576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1626079232,"data_stored":1626079232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2044723200,"data_stored":2044723200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3277000704,"data_stored":3276996632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:19:20.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-17T16:19:20.498 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:20.525 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:20.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:20.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:20.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:20.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:19:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:19:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:19:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:20 smithi017 ceph-mon[27571]: pgmap v805: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 102 B/s rd, 24 MiB/s wr, 815 op/s; 5996/293628 objects misplaced (2.042%); 7.5 MiB/s, 119 objects/s recovering 2024-09-17T16:19:20.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:20.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:20.791 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:20.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:19:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:19:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:19:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:20 smithi071 ceph-mon[30002]: pgmap v805: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 102 B/s rd, 24 MiB/s wr, 815 op/s; 5996/293628 objects misplaced (2.042%); 7.5 MiB/s, 119 objects/s recovering 2024-09-17T16:19:20.892 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:19:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:19:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:19:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:20 smithi173 ceph-mon[30530]: pgmap v805: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 102 B/s rd, 24 MiB/s wr, 815 op/s; 5996/293628 objects misplaced (2.042%); 7.5 MiB/s, 119 objects/s recovering 2024-09-17T16:19:20.920 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:20.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:21.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:21.054 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:21.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:21.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:21.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:21.182 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:21.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:21.285 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:21.310 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:19:21.310 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:19:21.312 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:21.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:21.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:21.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:21.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:21.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:21.569 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:21.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:21.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:21.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:21.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:21.797 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:21.825 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:21.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:21.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:21.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:21.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:21.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:22.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:22.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:22.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:22.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:22.212 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:22.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:22.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:19:22.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:22.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:22.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:22.423 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:19:22.423 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:19:22.423 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:19:22.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:19:22.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:19:22.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:19:22.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:19:22.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:19:22.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:19:22.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:19:22.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:19:22.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:19:22.424 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:19:22.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:19:22.451 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:22.478 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:22.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:22 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:19:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:22 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:19:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:22 smithi017 ceph-mon[27571]: osdmap e351: 12 total, 12 up, 11 in 2024-09-17T16:19:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4270462989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:22 smithi017 ceph-mon[27571]: pgmap v807: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 5.9 MiB/s wr, 961 op/s; 5710/285477 objects misplaced (2.000%); 9.6 MiB/s, 152 objects/s recovering 2024-09-17T16:19:22.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:22.609 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:22.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:22.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:22.742 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:22.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:22 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:19:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:22 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:19:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:22 smithi071 ceph-mon[30002]: osdmap e351: 12 total, 12 up, 11 in 2024-09-17T16:19:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4270462989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:22 smithi071 ceph-mon[30002]: pgmap v807: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 5.9 MiB/s wr, 961 op/s; 5710/285477 objects misplaced (2.000%); 9.6 MiB/s, 152 objects/s recovering 2024-09-17T16:19:22.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:22.877 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:22.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:22 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:19:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:22 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:19:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:22 smithi173 ceph-mon[30530]: osdmap e351: 12 total, 12 up, 11 in 2024-09-17T16:19:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4270462989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:22 smithi173 ceph-mon[30530]: pgmap v807: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 5.9 MiB/s wr, 961 op/s; 5710/285477 objects misplaced (2.000%); 9.6 MiB/s, 152 objects/s recovering 2024-09-17T16:19:22.978 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:23.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:23.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:23.104 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:23.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:23.130 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:23.159 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:23.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:23.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:23.287 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:23.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:23.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:23.419 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:23.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:23.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:23.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:23.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:23.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:23.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:23.677 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:23.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:23.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:23.801 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:23.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:23.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:23.934 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:23.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:23.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:24.035 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:19:24.035 INFO:teuthology.orchestra.run.smithi017.stdout:{"fsid":"19341aec-750e-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":882,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":351,"num_osds":12,"num_up_osds":12,"osd_up_since":1726589558,"num_in_osds":11,"osd_in_since":1726589860,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+remapped","count":2}],"num_pgs":17,"num_pools":2,"num_objects":94501,"data_bytes":6193611320,"bytes_used":21174513664,"bytes_avail":1034770878464,"bytes_total":1055945392128,"misplaced_objects":5710,"misplaced_total":283503,"misplaced_ratio":0.020140880343417888,"recovering_objects_per_sec":114,"recovering_bytes_per_sec":7486026,"recovering_keys_per_sec":0,"num_objects_recovered":914,"num_bytes_recovered":59899904,"num_keys_recovered":0,"write_bytes_sec":212950,"read_op_per_sec":0,"write_op_per_sec":775},"fsmap":{"epoch":1,"btime":"2024-09-17T16:02:46:978559+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.17:8443/"}},"servicemap":{"epoch":22,"modified":"2024-09-17T16:18:45.828885+0000","services":{}},"progress_events":{}} 2024-09-17T16:19:24.036 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:24.065 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:24.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:24.166 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:24.193 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:24.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:24.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:24.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:24.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:24.418 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:24.446 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:24.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:24.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:24.548 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:24.582 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:24.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:24.684 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:24.712 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:24.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:24.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:24.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:24.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:24.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:24.896 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:19:24.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:24.967 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:24.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:24.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:25.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:25.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:25.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:25.135 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:19:25.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:24 smithi173 ceph-mon[30530]: pgmap v808: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 208 KiB/s wr, 775 op/s; 5710/283503 objects misplaced (2.014%); 7.1 MiB/s, 114 objects/s recovering 2024-09-17T16:19:25.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4216678066' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:19:25.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/39233042' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:19:25.160 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:24 smithi017 ceph-mon[27571]: pgmap v808: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 208 KiB/s wr, 775 op/s; 5710/283503 objects misplaced (2.014%); 7.1 MiB/s, 114 objects/s recovering 2024-09-17T16:19:25.160 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4216678066' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:19:25.160 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/39233042' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:19:25.194 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:25.222 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:25.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:24 smithi071 ceph-mon[30002]: pgmap v808: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 208 KiB/s wr, 775 op/s; 5710/283503 objects misplaced (2.014%); 7.1 MiB/s, 114 objects/s recovering 2024-09-17T16:19:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4216678066' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:19:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/39233042' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:19:25.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:25.335 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-17T16:19:25.336 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:19:25.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:25.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:25.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:25.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:25.477 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:25.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:25.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:25.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:25.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:25.666 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:25.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:25.735 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:25.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:25.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:25.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:25.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:25.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:25.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:25.992 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:25.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:26.094 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:26.125 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:26.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:26.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:25 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/39233042' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:19:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:25 smithi173 ceph-mon[30530]: osdmap e352: 12 total, 12 up, 11 in 2024-09-17T16:19:26.227 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:26.234 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:25 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:26.234 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/39233042' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:19:26.234 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:25 smithi017 ceph-mon[27571]: osdmap e352: 12 total, 12 up, 11 in 2024-09-17T16:19:26.250 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:26.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:26.280 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:26.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:25 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/39233042' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:19:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:25 smithi071 ceph-mon[30002]: osdmap e352: 12 total, 12 up, 11 in 2024-09-17T16:19:26.331 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:26.352 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:26.380 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:26.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:26.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:26.482 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:26.507 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:26.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:26.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:26.632 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:26.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:26.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:26.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:26.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:26.860 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:26.883 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:26.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:26.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:27.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:27.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:27.089 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:26 smithi017 ceph-mon[27571]: pgmap v810: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 959 op/s; 5440/275400 objects misplaced (1.975%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T16:19:27.112 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:27.121 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:19:27.121 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:19:27.121 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:19:27.121 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:19:27.121 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:19:27.142 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:27.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:27.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:27.244 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:27.275 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:27.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:26 smithi071 ceph-mon[30002]: pgmap v810: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 959 op/s; 5440/275400 objects misplaced (1.975%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T16:19:27.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:27.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:26 smithi173 ceph-mon[30530]: pgmap v810: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.6 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 959 op/s; 5440/275400 objects misplaced (1.975%); 6.3 MiB/s, 100 objects/s recovering 2024-09-17T16:19:27.407 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:27.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:27.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:27.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:27.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:27.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:19:27.611 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:19:27.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:27.665 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:27.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:27.769 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:27.800 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:27.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:27.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:27.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:27.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:27.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:28.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:28.060 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:28.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:28.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:28.189 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:28.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:28.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:28.271 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:19:28.291 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:28.322 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:28.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:28.423 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:28.450 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:28.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:28.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:28.571 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":810,"stamp":"2024-09-17T16:19:25.836786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6016598584,"num_objects":91800,"num_object_clones":0,"num_object_copies":275400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5440,"num_objects_unfound":0,"num_objects_dirty":91800,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":112671,"num_write_kb":6541159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4044,"num_bytes_recovered":266221152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112640,"ondisk_log_size":112640,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":20195276,"kb_used_data":17989244,"kb_used_omap":57,"kb_used_meta":2205062,"kb_avail":1011001396,"statfs":{"total":1055945392128,"available":1035265429504,"internally_reserved":0,"allocated":18420985856,"data_stored":18414930295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2257984222},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-502988800,"num_objects":-7675,"num_object_clones":0,"num_object_copies":-23025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-806,"num_objects_unfound":0,"num_objects_dirty":-7675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":806,"num_bytes_recovered":52822016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001563"},"pg_stats":[{"pgid":"4.f","version":"350'6853","reported_seq":6952,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-17T16:19:21.349986+0000","last_change":"2024-09-17T16:17:41.531577+0000","last_active":"2024-09-17T16:19:21.349986+0000","last_peered":"2024-09-17T16:19:21.349986+0000","last_clean":"2024-09-17T16:19:21.349986+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:19:21.349986+0000","last_undegraded":"2024-09-17T16:19:21.349986+0000","last_fullsized":"2024-09-17T16:19:21.349986+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.077147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383975424,"num_objects":5859,"num_object_clones":0,"num_object_copies":17577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":406784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"351'7077","reported_seq":7185,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-17T16:19:24.667768+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:19:24.667768+0000","last_peered":"2024-09-17T16:19:24.667768+0000","last_clean":"2024-09-17T16:19:24.667768+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:19:24.667768+0000","last_undegraded":"2024-09-17T16:19:24.667768+0000","last_fullsized":"2024-09-17T16:19:24.667768+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370212864,"num_objects":5649,"num_object_clones":0,"num_object_copies":16947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7077,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"351'6894","reported_seq":7012,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-17T16:19:24.650185+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:19:24.650185+0000","last_peered":"2024-09-17T16:19:24.650185+0000","last_clean":"2024-09-17T16:19:24.650185+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:19:24.650185+0000","last_undegraded":"2024-09-17T16:19:24.650185+0000","last_fullsized":"2024-09-17T16:19:24.650185+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6894,"log_dups_size":0,"ondisk_log_size":6894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363921432,"num_objects":5554,"num_object_clones":0,"num_object_copies":16662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6894,"num_write_kb":398273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"351'7164","reported_seq":7274,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-17T16:19:24.643557+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:19:24.643557+0000","last_peered":"2024-09-17T16:19:24.643557+0000","last_clean":"2024-09-17T16:19:24.643557+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:19:24.643557+0000","last_undegraded":"2024-09-17T16:19:24.643557+0000","last_fullsized":"2024-09-17T16:19:24.643557+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7164,"log_dups_size":0,"ondisk_log_size":7164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383647744,"num_objects":5854,"num_object_clones":0,"num_object_copies":17562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"351'7114","reported_seq":10985,"reported_epoch":351,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:24.650102+0000","last_change":"2024-09-17T16:18:41.641380+0000","last_active":"2024-09-17T16:19:24.650102+0000","last_peered":"2024-09-17T16:19:24.650102+0000","last_clean":"2024-09-17T16:18:39.384812+0000","last_became_active":"2024-09-17T16:18:41.595599+0000","last_became_peered":"2024-09-17T16:18:41.595599+0000","last_unstale":"2024-09-17T16:19:24.650102+0000","last_undegraded":"2024-09-17T16:19:24.650102+0000","last_fullsized":"2024-09-17T16:18:41.426452+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7114,"log_dups_size":0,"ondisk_log_size":7114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372113408,"num_objects":5678,"num_object_clones":0,"num_object_copies":17034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":5678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7114,"num_write_kb":409344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1883,"num_bytes_recovered":123404288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4","objects":2754},{"shards":"0,4,5","objects":2924}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"352'7246","reported_seq":7429,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-17T16:19:25.558518+0000","last_change":"2024-09-17T16:19:02.221681+0000","last_active":"2024-09-17T16:19:25.558518+0000","last_peered":"2024-09-17T16:19:25.558518+0000","last_clean":"2024-09-17T16:19:25.558518+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:19:25.558518+0000","last_undegraded":"2024-09-17T16:19:25.558518+0000","last_fullsized":"2024-09-17T16:19:25.558518+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5925","last_scrub_stamp":"2024-09-17T16:19:02.221472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.221472+0000","objects_scrubbed":5879,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.368910+0000","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372768768,"num_objects":5688,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7246,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"350'6816","reported_seq":6982,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-17T16:19:21.375425+0000","last_change":"2024-09-17T16:19:02.418259+0000","last_active":"2024-09-17T16:19:21.375425+0000","last_peered":"2024-09-17T16:19:21.375425+0000","last_clean":"2024-09-17T16:19:21.375425+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:19:21.375425+0000","last_undegraded":"2024-09-17T16:19:21.375425+0000","last_fullsized":"2024-09-17T16:19:21.375425+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5797","last_scrub_stamp":"2024-09-17T16:19:02.418176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.418176+0000","objects_scrubbed":5756,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:17.281193+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377225216,"num_objects":5756,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"351'6982","reported_seq":7092,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-17T16:19:22.920869+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:19:22.920869+0000","last_peered":"2024-09-17T16:19:22.920869+0000","last_clean":"2024-09-17T16:19:22.920869+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:19:22.920869+0000","last_undegraded":"2024-09-17T16:19:22.920869+0000","last_fullsized":"2024-09-17T16:19:22.920869+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6982,"log_dups_size":0,"ondisk_log_size":6982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378535936,"num_objects":5776,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6982,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"352'7122","reported_seq":7241,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-17T16:19:25.750458+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:19:25.750458+0000","last_peered":"2024-09-17T16:19:25.750458+0000","last_clean":"2024-09-17T16:19:25.750458+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:19:25.750458+0000","last_undegraded":"2024-09-17T16:19:25.750458+0000","last_fullsized":"2024-09-17T16:19:25.750458+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":7122,"log_dups_size":0,"ondisk_log_size":7122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:18.005666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369098752,"num_objects":5632,"num_object_clones":0,"num_object_copies":16896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"351'6804","reported_seq":11215,"reported_epoch":351,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:21.723703+0000","last_change":"2024-09-17T16:18:42.321603+0000","last_active":"2024-09-17T16:19:21.723703+0000","last_peered":"2024-09-17T16:19:21.723703+0000","last_clean":"2024-09-17T16:18:39.337421+0000","last_became_active":"2024-09-17T16:18:42.219868+0000","last_became_peered":"2024-09-17T16:18:42.219868+0000","last_unstale":"2024-09-17T16:19:21.723703+0000","last_undegraded":"2024-09-17T16:19:21.723703+0000","last_fullsized":"2024-09-17T16:18:41.426290+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6804,"log_dups_size":0,"ondisk_log_size":6804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374603776,"num_objects":5716,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2516,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6804,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2154,"num_bytes_recovered":141164544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[2,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":2516},{"shards":"2,7","objects":3200}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"351'6940","reported_seq":7041,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-17T16:19:21.714428+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:19:21.714428+0000","last_peered":"2024-09-17T16:19:21.714428+0000","last_clean":"2024-09-17T16:19:21.714428+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:19:21.714428+0000","last_undegraded":"2024-09-17T16:19:21.714428+0000","last_fullsized":"2024-09-17T16:19:21.714428+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6940,"log_dups_size":0,"ondisk_log_size":6940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382992384,"num_objects":5844,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6940,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"351'7172","reported_seq":7289,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-17T16:19:22.956936+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:19:22.956936+0000","last_peered":"2024-09-17T16:19:22.956936+0000","last_clean":"2024-09-17T16:19:22.956936+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:19:22.956936+0000","last_undegraded":"2024-09-17T16:19:22.956936+0000","last_fullsized":"2024-09-17T16:19:22.956936+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7172,"log_dups_size":0,"ondisk_log_size":7172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382074880,"num_objects":5830,"num_object_clones":0,"num_object_copies":17490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7172,"num_write_kb":416064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"352'6993","reported_seq":7115,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-17T16:19:25.444311+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:19:25.444311+0000","last_peered":"2024-09-17T16:19:25.444311+0000","last_clean":"2024-09-17T16:19:25.444311+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:19:25.444311+0000","last_undegraded":"2024-09-17T16:19:25.444311+0000","last_fullsized":"2024-09-17T16:19:25.444311+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":6993,"log_dups_size":0,"ondisk_log_size":6993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:44.325955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364314624,"num_objects":5559,"num_object_clones":0,"num_object_copies":16677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6993,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"352'7162","reported_seq":7291,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-17T16:19:25.462900+0000","last_change":"2024-09-17T16:19:06.911296+0000","last_active":"2024-09-17T16:19:25.462900+0000","last_peered":"2024-09-17T16:19:25.462900+0000","last_clean":"2024-09-17T16:19:25.462900+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:19:25.462900+0000","last_undegraded":"2024-09-17T16:19:25.462900+0000","last_fullsized":"2024-09-17T16:19:25.462900+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1124,"log_size":7162,"log_dups_size":0,"ondisk_log_size":7162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:36.906787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376963072,"num_objects":5752,"num_object_clones":0,"num_object_copies":17256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7162,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"350'6853","reported_seq":6997,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-17T16:19:21.386844+0000","last_change":"2024-09-17T16:19:06.928683+0000","last_active":"2024-09-17T16:19:21.386844+0000","last_peered":"2024-09-17T16:19:21.386844+0000","last_clean":"2024-09-17T16:19:21.386844+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:19:21.386844+0000","last_undegraded":"2024-09-17T16:19:21.386844+0000","last_fullsized":"2024-09-17T16:19:21.386844+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":3399,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:36.915694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385548288,"num_objects":5883,"num_object_clones":0,"num_object_copies":17649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":407552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"352'7356","reported_seq":7536,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-17T16:19:25.606230+0000","last_change":"2024-09-17T16:19:01.159281+0000","last_active":"2024-09-17T16:19:25.606230+0000","last_peered":"2024-09-17T16:19:25.606230+0000","last_clean":"2024-09-17T16:19:25.606230+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:19:25.606230+0000","last_undegraded":"2024-09-17T16:19:25.606230+0000","last_fullsized":"2024-09-17T16:19:25.606230+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5950","last_scrub_stamp":"2024-09-17T16:19:01.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:01.159158+0000","objects_scrubbed":5911,"log_size":7356,"log_dups_size":0,"ondisk_log_size":7356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:27.590602+0000","scrub_duration":1399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378011648,"num_objects":5768,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7356,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":595,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-17T16:19:21.414917+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:19:21.414917+0000","last_peered":"2024-09-17T16:19:21.414917+0000","last_clean":"2024-09-17T16:19:21.414917+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:19:21.414917+0000","last_undegraded":"2024-09-17T16:19:21.414917+0000","last_fullsized":"2024-09-17T16:19:21.414917+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":6016008216,"num_objects":91798,"num_object_clones":0,"num_object_copies":275394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5440,"num_objects_unfound":0,"num_objects_dirty":91798,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112548,"num_write_kb":6539009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4038,"num_bytes_recovered":264634368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18314309632,"data_stored":18314297416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112548,"ondisk_log_size":112548,"up":48,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285687,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3190644,"kb_used_data":2920676,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":90554508,"statfs":{"total":95995035648,"available":92727816192,"internally_reserved":0,"allocated":2990772224,"data_stored":2990228263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":276429252},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547324,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1966648,"kb_used_data":1826992,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91778504,"statfs":{"total":95995035648,"available":93981188096,"internally_reserved":0,"allocated":1870839808,"data_stored":1870364216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743552,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1804564,"kb_used_data":1498500,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91940588,"statfs":{"total":95995035648,"available":94147162112,"internally_reserved":0,"allocated":1534464000,"data_stored":1533859953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":313391552},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":123040,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622112,"statfs":{"total":95995035648,"available":95869042688,"internally_reserved":0,"allocated":1466368,"data_stored":930051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963666,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2428584,"kb_used_data":2227232,"kb_used_omap":41,"kb_used_meta":201110,"kb_avail":91316568,"statfs":{"total":95995035648,"available":93508165632,"internally_reserved":0,"allocated":2280685568,"data_stored":2280145699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":205936947},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2769136,"kb_used_data":2567200,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":90976016,"statfs":{"total":95995035648,"available":93159440384,"internally_reserved":0,"allocated":2628812800,"data_stored":2628272931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206764486},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724691,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1553592,"kb_used_data":1333424,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92191560,"statfs":{"total":95995035648,"available":94404157440,"internally_reserved":0,"allocated":1365426176,"data_stored":1364950584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":225245638},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986330,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1663212,"kb_used_data":1461276,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92081940,"statfs":{"total":95995035648,"available":94291906560,"internally_reserved":0,"allocated":1496346624,"data_stored":1495806235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2108560,"kb_used_data":1888576,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":91636592,"statfs":{"total":95995035648,"available":93835870208,"internally_reserved":0,"allocated":1933901824,"data_stored":1933301849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":225245639},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411491,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866640,"kb_used_data":745216,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92878512,"statfs":{"total":95995035648,"available":95107596288,"internally_reserved":0,"allocated":763101184,"data_stored":762501209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738541,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1720656,"kb_used_data":1518720,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92024496,"statfs":{"total":95995035648,"available":94233083904,"internally_reserved":0,"allocated":1555169280,"data_stored":1554569305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1549205504,"data_stored":1549205504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":757465088,"data_stored":757465088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1926430720,"data_stored":1926430720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1489702912,"data_stored":1489698840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1362231296,"data_stored":1362231296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2604335104,"data_stored":2604335104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2272002048,"data_stored":2272002048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1518669824,"data_stored":1518665752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1854799872,"data_stored":1854799872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2979467264,"data_stored":2979463192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:19:28.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:28.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:28.686 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:28.711 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:28.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:28.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:28.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:28.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:28.909 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:19:28.931 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:28.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:28.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:28.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:29.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:29.110 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:29.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:28 smithi017 ceph-mon[27571]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:28 smithi017 ceph-mon[27571]: pgmap v811: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 941 op/s; 5138/271029 objects misplaced (1.896%); 6.7 MiB/s, 107 objects/s recovering 2024-09-17T16:19:29.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:29.257 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:29.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:28 smithi071 ceph-mon[30002]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:28 smithi071 ceph-mon[30002]: pgmap v811: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 941 op/s; 5138/271029 objects misplaced (1.896%); 6.7 MiB/s, 107 objects/s recovering 2024-09-17T16:19:29.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:29.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:29.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:29.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:28 smithi173 ceph-mon[30530]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:28 smithi173 ceph-mon[30530]: pgmap v811: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 941 op/s; 5138/271029 objects misplaced (1.896%); 6.7 MiB/s, 107 objects/s recovering 2024-09-17T16:19:29.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:29.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:29.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:29.615 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:29.643 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:29.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:29.721 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:19:29.721 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:19:29.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:29.750 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:19:29.750 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:19:29.772 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:19:29.772 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:19:29.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:29.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:29.798 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:19:29.798 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:19:29.828 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:19:29.829 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:19:29.856 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:19:29.856 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:19:29.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:29.884 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:19:29.884 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:19:29.904 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:29.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:29.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:19:29.913 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:19:29.942 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:19:29.942 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:19:29.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:19:29.971 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:19:29.999 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:30.000 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:19:30.001 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:19:30.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:30.029 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:19:30.029 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:19:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:30.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:30.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:30.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:30.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:30.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:30.258 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:30.282 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:30.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:30.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:30.385 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:30.412 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:30.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:30.426 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:19:30.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:30.540 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:30.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:30.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:30.670 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:30.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3662265927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1710144640' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:30.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:30.797 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:30.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:30.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3662265927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1710144640' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3662265927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1710144640' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:30.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:30.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:30.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:31.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:31.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:31.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:31.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:31.180 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:31.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:31.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:31.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:31.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:31.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:31.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:31.434 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:31.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:31.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:31.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:31.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:31.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:31.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:19:31.664 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:31.691 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:31.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:31.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:31 smithi017 ceph-mon[27571]: pgmap v812: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 849 op/s; 4919/265743 objects misplaced (1.851%); 8.0 MiB/s, 127 objects/s recovering 2024-09-17T16:19:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:31 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:19:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:31 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:19:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1710144640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:19:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:31 smithi017 ceph-mon[27571]: osdmap e353: 12 total, 12 up, 11 in 2024-09-17T16:19:31.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:31 smithi071 ceph-mon[30002]: pgmap v812: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 849 op/s; 4919/265743 objects misplaced (1.851%); 8.0 MiB/s, 127 objects/s recovering 2024-09-17T16:19:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:31 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:19:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:31 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:19:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1710144640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:19:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:31 smithi071 ceph-mon[30002]: osdmap e353: 12 total, 12 up, 11 in 2024-09-17T16:19:31.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:31.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:31 smithi173 ceph-mon[30530]: pgmap v812: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 849 op/s; 4919/265743 objects misplaced (1.851%); 8.0 MiB/s, 127 objects/s recovering 2024-09-17T16:19:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:31 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:19:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:31 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:19:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1710144640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:19:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:31 smithi173 ceph-mon[30530]: osdmap e353: 12 total, 12 up, 11 in 2024-09-17T16:19:31.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:31.953 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:31.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:32.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:32.088 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:32.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:32.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:32.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:32.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:32.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:32.271 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:19:32.271 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:19:32.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:32.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:32.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:32.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:32.473 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:32.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:32.520 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:32.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:32.601 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:32.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:32.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:32.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:32 smithi017 ceph-mon[27571]: pgmap v814: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 4619/256815 objects misplaced (1.799%); 8.5 MiB/s, 136 objects/s recovering 2024-09-17T16:19:32.703 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:32.728 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:32.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:32.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:32.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:32 smithi071 ceph-mon[30002]: pgmap v814: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 4619/256815 objects misplaced (1.799%); 8.5 MiB/s, 136 objects/s recovering 2024-09-17T16:19:32.830 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:32.853 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:32.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:32 smithi173 ceph-mon[30530]: pgmap v814: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.11k op/s; 4619/256815 objects misplaced (1.799%); 8.5 MiB/s, 136 objects/s recovering 2024-09-17T16:19:32.956 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:32.984 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:32.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:33.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:33.113 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:33.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:33.148 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:19:33.186 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:19:33.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:33.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:33.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:33.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:33.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:33.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:33.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:33.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:33.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:33.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:33.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:33.635 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:33.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:33.737 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:33.775 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:33.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:33.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:33.796 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:33.800 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:33.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:33.907 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:33.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:33.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:33.997 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:19:34.009 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:34.024 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:19:34.025 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:19:34.036 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:34.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:34.140 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:34.203 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:34.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:34.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:34.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:34.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:34.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:34.461 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:34.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:34.563 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:34.592 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:34.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:34.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:34.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:34.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:34.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:34.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:34.920 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":815,"stamp":"2024-09-17T16:19:33.838509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5575213624,"num_objects":85065,"num_object_clones":0,"num_object_copies":255195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4619,"num_objects_unfound":0,"num_objects_dirty":85065,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":119406,"num_write_kb":6541159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4865,"num_bytes_recovered":320026208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119375,"ondisk_log_size":119375,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":18921540,"kb_used_data":16730924,"kb_used_omap":57,"kb_used_meta":2189830,"kb_avail":1012275132,"statfs":{"total":1055945392128,"available":1036569735168,"internally_reserved":0,"allocated":17132466176,"data_stored":17127522272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2242386654},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,3,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-542310400,"num_objects":-8275,"num_object_clones":0,"num_object_copies":-24825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-821,"num_objects_unfound":0,"num_objects_dirty":-8275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8275,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":821,"num_bytes_recovered":53805056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.943827"},"pg_stats":[{"pgid":"4.f","version":"353'7465","reported_seq":7570,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:31.391887+0000","last_change":"2024-09-17T16:17:41.531577+0000","last_active":"2024-09-17T16:19:31.391887+0000","last_peered":"2024-09-17T16:19:31.391887+0000","last_clean":"2024-09-17T16:19:31.391887+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:19:31.391887+0000","last_undegraded":"2024-09-17T16:19:31.391887+0000","last_fullsized":"2024-09-17T16:19:31.391887+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.077147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343867392,"num_objects":5247,"num_object_clones":0,"num_object_copies":15741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7465,"num_write_kb":406784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"352'7370","reported_seq":7481,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-17T16:19:29.653599+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:19:29.653599+0000","last_peered":"2024-09-17T16:19:29.653599+0000","last_clean":"2024-09-17T16:19:29.653599+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:19:29.653599+0000","last_undegraded":"2024-09-17T16:19:29.653599+0000","last_fullsized":"2024-09-17T16:19:29.653599+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7370,"log_dups_size":0,"ondisk_log_size":7370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351010816,"num_objects":5356,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7370,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"352'7190","reported_seq":7310,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-17T16:19:29.659231+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:19:29.659231+0000","last_peered":"2024-09-17T16:19:29.659231+0000","last_clean":"2024-09-17T16:19:29.659231+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:19:29.659231+0000","last_undegraded":"2024-09-17T16:19:29.659231+0000","last_fullsized":"2024-09-17T16:19:29.659231+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7190,"log_dups_size":0,"ondisk_log_size":7190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344522776,"num_objects":5258,"num_object_clones":0,"num_object_copies":15774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7190,"num_write_kb":398273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"352'7479","reported_seq":7591,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-17T16:19:29.586694+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:19:29.586694+0000","last_peered":"2024-09-17T16:19:29.586694+0000","last_clean":"2024-09-17T16:19:29.586694+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:19:29.586694+0000","last_undegraded":"2024-09-17T16:19:29.586694+0000","last_fullsized":"2024-09-17T16:19:29.586694+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363003904,"num_objects":5539,"num_object_clones":0,"num_object_copies":16617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7479,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"352'7420","reported_seq":11731,"reported_epoch":352,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:29.670092+0000","last_change":"2024-09-17T16:18:41.641380+0000","last_active":"2024-09-17T16:19:29.670092+0000","last_peered":"2024-09-17T16:19:29.670092+0000","last_clean":"2024-09-17T16:18:39.384812+0000","last_became_active":"2024-09-17T16:18:41.595599+0000","last_became_peered":"2024-09-17T16:18:41.595599+0000","last_unstale":"2024-09-17T16:19:29.670092+0000","last_undegraded":"2024-09-17T16:19:29.670092+0000","last_fullsized":"2024-09-17T16:18:41.426452+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7420,"log_dups_size":0,"ondisk_log_size":7420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352059392,"num_objects":5372,"num_object_clones":0,"num_object_copies":16116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":5372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7420,"num_write_kb":409344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2102,"num_bytes_recovered":137756672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4","objects":2667},{"shards":"0,4,5","objects":2705}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"353'7561","reported_seq":7746,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:30.624044+0000","last_change":"2024-09-17T16:19:02.221681+0000","last_active":"2024-09-17T16:19:30.624044+0000","last_peered":"2024-09-17T16:19:30.624044+0000","last_clean":"2024-09-17T16:19:30.624044+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:19:30.624044+0000","last_undegraded":"2024-09-17T16:19:30.624044+0000","last_fullsized":"2024-09-17T16:19:30.624044+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5925","last_scrub_stamp":"2024-09-17T16:19:02.221472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.221472+0000","objects_scrubbed":5879,"log_size":7561,"log_dups_size":0,"ondisk_log_size":7561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.368910+0000","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352124928,"num_objects":5373,"num_object_clones":0,"num_object_copies":16119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7561,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"353'7401","reported_seq":7573,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:31.362808+0000","last_change":"2024-09-17T16:19:02.418259+0000","last_active":"2024-09-17T16:19:31.362808+0000","last_peered":"2024-09-17T16:19:31.362808+0000","last_clean":"2024-09-17T16:19:31.362808+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:19:31.362808+0000","last_undegraded":"2024-09-17T16:19:31.362808+0000","last_fullsized":"2024-09-17T16:19:31.362808+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5797","last_scrub_stamp":"2024-09-17T16:19:02.418176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.418176+0000","objects_scrubbed":5756,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:17.281193+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338886656,"num_objects":5171,"num_object_clones":0,"num_object_copies":15513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7401,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"353'7522","reported_seq":7636,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:32.884409+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:19:32.884409+0000","last_peered":"2024-09-17T16:19:32.884409+0000","last_clean":"2024-09-17T16:19:32.884409+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:19:32.884409+0000","last_undegraded":"2024-09-17T16:19:32.884409+0000","last_fullsized":"2024-09-17T16:19:32.884409+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7522,"log_dups_size":0,"ondisk_log_size":7522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343146496,"num_objects":5236,"num_object_clones":0,"num_object_copies":15708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7522,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"353'7406","reported_seq":7527,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:30.715240+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:19:30.715240+0000","last_peered":"2024-09-17T16:19:30.715240+0000","last_clean":"2024-09-17T16:19:30.715240+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:19:30.715240+0000","last_undegraded":"2024-09-17T16:19:30.715240+0000","last_fullsized":"2024-09-17T16:19:30.715240+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":7406,"log_dups_size":0,"ondisk_log_size":7406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:18.005666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350486528,"num_objects":5348,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7406,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"353'7379","reported_seq":12997,"reported_epoch":353,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:31.731598+0000","last_change":"2024-09-17T16:18:42.321603+0000","last_active":"2024-09-17T16:19:31.731598+0000","last_peered":"2024-09-17T16:19:31.731598+0000","last_clean":"2024-09-17T16:18:39.337421+0000","last_became_active":"2024-09-17T16:18:42.219868+0000","last_became_peered":"2024-09-17T16:18:42.219868+0000","last_unstale":"2024-09-17T16:19:31.731598+0000","last_undegraded":"2024-09-17T16:19:31.731598+0000","last_fullsized":"2024-09-17T16:18:41.426290+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7379,"log_dups_size":0,"ondisk_log_size":7379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336920576,"num_objects":5141,"num_object_clones":0,"num_object_copies":15423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1914,"num_objects_unfound":0,"num_objects_dirty":5141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7379,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2756,"num_bytes_recovered":180617216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[2,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":1914},{"shards":"2,7","objects":3227}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"353'7510","reported_seq":7615,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:31.713252+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:19:31.713252+0000","last_peered":"2024-09-17T16:19:31.713252+0000","last_clean":"2024-09-17T16:19:31.713252+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:19:31.713252+0000","last_undegraded":"2024-09-17T16:19:31.713252+0000","last_fullsized":"2024-09-17T16:19:31.713252+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7510,"log_dups_size":0,"ondisk_log_size":7510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345636864,"num_objects":5274,"num_object_clones":0,"num_object_copies":15822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7510,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"353'7724","reported_seq":7845,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:32.946541+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:19:32.946541+0000","last_peered":"2024-09-17T16:19:32.946541+0000","last_clean":"2024-09-17T16:19:32.946541+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:19:32.946541+0000","last_undegraded":"2024-09-17T16:19:32.946541+0000","last_fullsized":"2024-09-17T16:19:32.946541+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7724,"log_dups_size":0,"ondisk_log_size":7724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345899008,"num_objects":5278,"num_object_clones":0,"num_object_copies":15834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7724,"num_write_kb":416064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"353'7299","reported_seq":7423,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:30.487716+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:19:30.487716+0000","last_peered":"2024-09-17T16:19:30.487716+0000","last_clean":"2024-09-17T16:19:30.487716+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:19:30.487716+0000","last_undegraded":"2024-09-17T16:19:30.487716+0000","last_fullsized":"2024-09-17T16:19:30.487716+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7299,"log_dups_size":0,"ondisk_log_size":7299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:44.325955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344260608,"num_objects":5253,"num_object_clones":0,"num_object_copies":15759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7299,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"353'7491","reported_seq":7621,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:30.501828+0000","last_change":"2024-09-17T16:19:06.911296+0000","last_active":"2024-09-17T16:19:30.501828+0000","last_peered":"2024-09-17T16:19:30.501828+0000","last_clean":"2024-09-17T16:19:30.501828+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:19:30.501828+0000","last_undegraded":"2024-09-17T16:19:30.501828+0000","last_fullsized":"2024-09-17T16:19:30.501828+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1124,"log_size":7491,"log_dups_size":0,"ondisk_log_size":7491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:36.906787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355401728,"num_objects":5423,"num_object_clones":0,"num_object_copies":16269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7491,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"353'7439","reported_seq":7589,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:31.307987+0000","last_change":"2024-09-17T16:19:06.928683+0000","last_active":"2024-09-17T16:19:31.307987+0000","last_peered":"2024-09-17T16:19:31.307987+0000","last_clean":"2024-09-17T16:19:31.307987+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:19:31.307987+0000","last_undegraded":"2024-09-17T16:19:31.307987+0000","last_fullsized":"2024-09-17T16:19:31.307987+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":3399,"log_size":7439,"log_dups_size":0,"ondisk_log_size":7439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:19:36.915694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347144192,"num_objects":5297,"num_object_clones":0,"num_object_copies":15891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":407552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"353'7627","reported_seq":7808,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:30.550739+0000","last_change":"2024-09-17T16:19:01.159281+0000","last_active":"2024-09-17T16:19:30.550739+0000","last_peered":"2024-09-17T16:19:30.550739+0000","last_clean":"2024-09-17T16:19:30.550739+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:19:30.550739+0000","last_undegraded":"2024-09-17T16:19:30.550739+0000","last_fullsized":"2024-09-17T16:19:30.550739+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5950","last_scrub_stamp":"2024-09-17T16:19:01.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:01.159158+0000","objects_scrubbed":5911,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:27.590602+0000","scrub_duration":1399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360251392,"num_objects":5497,"num_object_clones":0,"num_object_copies":16491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":599,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-17T16:19:30.433244+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:19:30.433244+0000","last_peered":"2024-09-17T16:19:30.433244+0000","last_clean":"2024-09-17T16:19:30.433244+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:19:30.433244+0000","last_undegraded":"2024-09-17T16:19:30.433244+0000","last_fullsized":"2024-09-17T16:19:30.433244+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":5574623256,"num_objects":85063,"num_object_clones":0,"num_object_copies":255189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4619,"num_objects_unfound":0,"num_objects_dirty":85063,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119283,"num_write_kb":6539009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4859,"num_bytes_recovered":318439424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17040158720,"data_stored":17040146504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119283,"ondisk_log_size":119283,"up":48,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285688,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3023820,"kb_used_data":2762172,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":90721332,"statfs":{"total":95995035648,"available":92898643968,"internally_reserved":0,"allocated":2828464128,"data_stored":2827980892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":267909572},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547325,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1878080,"kb_used_data":1734448,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91867072,"statfs":{"total":95995035648,"available":94071881728,"internally_reserved":0,"allocated":1776074752,"data_stored":1775599160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743554,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1688284,"kb_used_data":1382220,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92056868,"statfs":{"total":95995035648,"available":94266232832,"internally_reserved":0,"allocated":1415393280,"data_stored":1414975716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":313391552},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005190,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122872,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622280,"statfs":{"total":95995035648,"available":95869214720,"internally_reserved":0,"allocated":1294336,"data_stored":818744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963668,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2201248,"kb_used_data":2004240,"kb_used_omap":41,"kb_used_meta":196950,"kb_avail":91543904,"statfs":{"total":95995035648,"available":93740957696,"internally_reserved":0,"allocated":2052341760,"data_stored":2051924716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":201677107},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2626760,"kb_used_data":2433144,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":91118392,"statfs":{"total":95995035648,"available":93305233408,"internally_reserved":0,"allocated":2491539456,"data_stored":2491060312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":198244806},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724692,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1512192,"kb_used_data":1292208,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92232960,"statfs":{"total":95995035648,"available":94446551040,"internally_reserved":0,"allocated":1323220992,"data_stored":1322745400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":225245638},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986331,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1575172,"kb_used_data":1385716,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92169980,"statfs":{"total":95995035648,"available":94382059520,"internally_reserved":0,"allocated":1418973184,"data_stored":1418493520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1914000,"kb_used_data":1697992,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91831152,"statfs":{"total":95995035648,"available":94035099648,"internally_reserved":0,"allocated":1738743808,"data_stored":1738330316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":220985799},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411493,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":815376,"kb_used_data":671560,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92929776,"statfs":{"total":95995035648,"available":95160090624,"internally_reserved":0,"allocated":687677440,"data_stored":687263948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738543,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1563736,"kb_used_data":1365960,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92181416,"statfs":{"total":95995035648,"available":94393769984,"internally_reserved":0,"allocated":1398743040,"data_stored":1398329548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1395589120,"data_stored":1395589120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":682491904,"data_stored":682491904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1735065600,"data_stored":1735065600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1410469888,"data_stored":1410465816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1320681472,"data_stored":1320681472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2467037184,"data_stored":2467037184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2047344640,"data_stored":2047344640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1401163776,"data_stored":1401159704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1758855168,"data_stored":1758855168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2821459968,"data_stored":2821455896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:19:34.922 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-17T16:19:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:34.948 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:34.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:35.049 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:35.081 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:35.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:35.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:34 smithi173 ceph-mon[30530]: pgmap v815: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 925 op/s; 4619/255195 objects misplaced (1.810%); 5.7 MiB/s, 91 objects/s recovering 2024-09-17T16:19:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:34 smithi173 ceph-mon[30530]: from='client.16167 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:35.182 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:35.209 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:35.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:35.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:34 smithi017 ceph-mon[27571]: pgmap v815: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 925 op/s; 4619/255195 objects misplaced (1.810%); 5.7 MiB/s, 91 objects/s recovering 2024-09-17T16:19:35.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:34 smithi017 ceph-mon[27571]: from='client.16167 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:35.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:34 smithi071 ceph-mon[30002]: pgmap v815: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 925 op/s; 4619/255195 objects misplaced (1.810%); 5.7 MiB/s, 91 objects/s recovering 2024-09-17T16:19:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:34 smithi071 ceph-mon[30002]: from='client.16167 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:35.340 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:35.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:35.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:35.475 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:35.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:35.577 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:35.616 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:35.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:35.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:35.742 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:35.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:35.844 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:35.874 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:35.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:35.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:36.000 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:35 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:19:36.000 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:35 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:19:36.000 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:35 smithi071 ceph-mon[30002]: osdmap e354: 12 total, 12 up, 11 in 2024-09-17T16:19:36.004 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:36.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:36.076 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:35 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:19:36.077 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:35 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:19:36.077 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:35 smithi017 ceph-mon[27571]: osdmap e354: 12 total, 12 up, 11 in 2024-09-17T16:19:36.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:36.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:36.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:36.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:36.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:36.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:36.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:36.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:36.386 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:36.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:36.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:35 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:19:36.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:35 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:19:36.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:35 smithi173 ceph-mon[30530]: osdmap e354: 12 total, 12 up, 11 in 2024-09-17T16:19:36.489 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:36.516 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:36.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:36.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:36.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:36.645 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:36.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:36.668 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:19:36.668 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:19:36.668 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:19:36.668 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:19:36.668 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:19:36.669 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:19:36.669 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:19:36.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:19:36.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:19:36.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:19:36.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:19:36.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:19:36.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:19:36.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:19:36.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:19:36.670 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:19:36.670 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:19:36.670 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:19:36.670 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:19:36.670 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:19:36.670 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:19:36.670 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:19:36.670 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:19:36.670 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:19:36.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:36.774 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:36.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:36.875 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:36.907 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:36.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:36.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:36 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:19:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:36 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:19:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:36 smithi017 ceph-mon[27571]: pgmap v817: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 984 op/s; 4341/247395 objects misplaced (1.755%); 6.2 MiB/s, 99 objects/s recovering 2024-09-17T16:19:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:36 smithi017 ceph-mon[27571]: osdmap e355: 12 total, 12 up, 11 in 2024-09-17T16:19:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1412117903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:37.003 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:19:37.003 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:19:37.008 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:37.035 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:37.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:37.138 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:37.163 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:37.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:37.265 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:37.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:37.293 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:37.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:36 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:19:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:36 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:19:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:36 smithi071 ceph-mon[30002]: pgmap v817: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 984 op/s; 4341/247395 objects misplaced (1.755%); 6.2 MiB/s, 99 objects/s recovering 2024-09-17T16:19:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:36 smithi071 ceph-mon[30002]: osdmap e355: 12 total, 12 up, 11 in 2024-09-17T16:19:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1412117903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:37.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:36 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:19:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:36 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:19:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:36 smithi173 ceph-mon[30530]: pgmap v817: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 5.0 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 984 op/s; 4341/247395 objects misplaced (1.755%); 6.2 MiB/s, 99 objects/s recovering 2024-09-17T16:19:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:36 smithi173 ceph-mon[30530]: osdmap e355: 12 total, 12 up, 11 in 2024-09-17T16:19:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1412117903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:37.421 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:37.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:37.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:37.523 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:37.551 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:37.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:37.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:37.686 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:37.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:37.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:37.788 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:37.811 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:19:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:37.823 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:37.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:37.824 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:37.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:37.957 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:37.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:38.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:38.086 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:38.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:38.188 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:38.212 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:38.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:38.236 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:37 smithi017 ceph-mon[27571]: 4.1 scrub starts 2024-09-17T16:19:38.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:37 smithi071 ceph-mon[30002]: 4.1 scrub starts 2024-09-17T16:19:38.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:38.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:38.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:37 smithi173 ceph-mon[30530]: 4.1 scrub starts 2024-09-17T16:19:38.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:38.499 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:38.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:38.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:38.630 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:38.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:38.732 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:38.766 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:38.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:38.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:38.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:38.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:38.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:38 smithi017 ceph-mon[27571]: pgmap v819: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 4038/242778 objects misplaced (1.663%); 7.4 MiB/s, 118 objects/s recovering 2024-09-17T16:19:38.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:38 smithi017 ceph-mon[27571]: 4.1 scrub ok 2024-09-17T16:19:38.934 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:38.990 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:38.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:39.030 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:39.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:39.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:39.132 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:39.160 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:39.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:39.262 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:39.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:39.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:39.295 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:38 smithi173 ceph-mon[30530]: pgmap v819: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 4038/242778 objects misplaced (1.663%); 7.4 MiB/s, 118 objects/s recovering 2024-09-17T16:19:39.295 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:38 smithi173 ceph-mon[30530]: 4.1 scrub ok 2024-09-17T16:19:39.304 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:19:39.304 INFO:teuthology.orchestra.run.smithi017.stdout:{"fsid":"19341aec-750e-11ef-bceb-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":897,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":355,"num_osds":12,"num_up_osds":12,"osd_up_since":1726589558,"num_in_osds":11,"osd_in_since":1726589860,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+remapped","count":2}],"num_pgs":17,"num_pools":2,"num_objects":80926,"data_bytes":5303960120,"bytes_used":18632183808,"bytes_avail":1037313208320,"bytes_total":1055945392128,"misplaced_objects":4038,"misplaced_total":242778,"misplaced_ratio":0.01663247905493908,"recovering_objects_per_sec":118,"recovering_bytes_per_sec":7786713,"recovering_keys_per_sec":0,"num_objects_recovered":881,"num_bytes_recovered":57737216,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1032},"fsmap":{"epoch":1,"btime":"2024-09-17T16:02:46:978559+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.17:8443/"}},"servicemap":{"epoch":22,"modified":"2024-09-17T16:18:45.828885+0000","services":{}},"progress_events":{}} 2024-09-17T16:19:39.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:38 smithi071 ceph-mon[30002]: pgmap v819: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 4038/242778 objects misplaced (1.663%); 7.4 MiB/s, 118 objects/s recovering 2024-09-17T16:19:39.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:38 smithi071 ceph-mon[30002]: 4.1 scrub ok 2024-09-17T16:19:39.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:39.421 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:39.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:39.521 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:39.553 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:39.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:39.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:39.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:39.683 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:39.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:39.784 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:39.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:39.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:39.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:39.947 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:39.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:40.000 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:19:40.000 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:19:40.000 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:19:40.000 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:19:40.000 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:19:40.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:40.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:40.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:40.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:40.214 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:40.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:40.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/918085305' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:19:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/918085305' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:19:40.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:40.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:40.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:40.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:40.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/918085305' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:19:40.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:19:40.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:40.478 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:40.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:40.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:40.616 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:40.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:40.718 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:40.749 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:40.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:40.850 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:40.876 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:40.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:40.928 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:19:40.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:41.004 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:41.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:41.061 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-17T16:19:41.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:19:41.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:41.131 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:41.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:41.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:41.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:40 smithi017 ceph-mon[27571]: pgmap v820: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 798 op/s; 3754/237636 objects misplaced (1.580%); 6.8 MiB/s, 108 objects/s recovering 2024-09-17T16:19:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3849352961' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:19:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:40 smithi017 ceph-mon[27571]: 4.0 scrub starts 2024-09-17T16:19:41.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:41.260 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:40 smithi071 ceph-mon[30002]: pgmap v820: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 798 op/s; 3754/237636 objects misplaced (1.580%); 6.8 MiB/s, 108 objects/s recovering 2024-09-17T16:19:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3849352961' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:19:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:40 smithi071 ceph-mon[30002]: 4.0 scrub starts 2024-09-17T16:19:41.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:41.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:41.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:41.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:40 smithi173 ceph-mon[30530]: pgmap v820: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 798 op/s; 3754/237636 objects misplaced (1.580%); 6.8 MiB/s, 108 objects/s recovering 2024-09-17T16:19:41.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3849352961' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:19:41.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:40 smithi173 ceph-mon[30530]: 4.0 scrub starts 2024-09-17T16:19:41.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:41.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:41.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:41.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:41.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:41.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:41.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:41.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:41.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:41.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:41.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:41.895 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:41.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:41.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:42.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:42.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:42.131 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:42.163 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:42.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:41 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3849352961' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:19:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:41 smithi017 ceph-mon[27571]: osdmap e356: 12 total, 12 up, 11 in 2024-09-17T16:19:42.264 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:42.291 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:42.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:42.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:41 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3849352961' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:19:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:41 smithi071 ceph-mon[30002]: osdmap e356: 12 total, 12 up, 11 in 2024-09-17T16:19:42.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:42.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:41 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:42.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3849352961' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:19:42.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:41 smithi173 ceph-mon[30530]: osdmap e356: 12 total, 12 up, 11 in 2024-09-17T16:19:42.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:42.424 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:42.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:42.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:42.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:42.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:42.676 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:42.677 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:42.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:42.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:42.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:42.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:42.930 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:42.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:43.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:43.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:43.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:43.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:43.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:43.191 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:19:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:19:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: pgmap v822: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 3377/228987 objects misplaced (1.475%); 8.7 MiB/s, 138 objects/s recovering 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: osdmap e357: 12 total, 12 up, 11 in 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=54,logm=3} crc {auth=2177375717,config=1059521453,health=3079269128,kv=420973415,logm=3738862875}) 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906617587}) 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2377789491}) 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2237723371}) 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659663829}) 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1571373277}) 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3731141147}) 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3805424563}) 2024-09-17T16:19:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3273012725}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508872391}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263597814}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069994879}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870994419}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899488714}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850967252}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466115059}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128063089}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416515822}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612570454}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018220424}) 2024-09-17T16:19:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094150758}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3010394247}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3010929325}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3409118182}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38098201}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3540394655}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1099896466}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=102094452}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443039166}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2483068486}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059000873}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466200765}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1978711023}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3513013153}) 2024-09-17T16:19:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3752808804}) 2024-09-17T16:19:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=63,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13} crc {logm=3039075746,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=2350606054}) 2024-09-17T16:19:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=4191902087,mon_config_key=1345730000}) 2024-09-17T16:19:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=163066068}) 2024-09-17T16:19:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,nvmeofgw=3,osd_metadata=5} crc {mon_config_key=2434274148,monmap=2143941632,nvmeofgw=598337686,osd_metadata=547839073}) 2024-09-17T16:19:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=1701853976,osd_pg_creating=2714581885,osdmap=4245895138}) 2024-09-17T16:19:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=29} crc {osdmap=133688214}) 2024-09-17T16:19:43.292 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:19:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: pgmap v822: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 3377/228987 objects misplaced (1.475%); 8.7 MiB/s, 138 objects/s recovering 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: osdmap e357: 12 total, 12 up, 11 in 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=54,logm=3} crc {auth=2177375717,config=1059521453,health=3079269128,kv=420973415,logm=3738862875}) 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906617587}) 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2377789491}) 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2237723371}) 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659663829}) 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1571373277}) 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3731141147}) 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3805424563}) 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3273012725}) 2024-09-17T16:19:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508872391}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263597814}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069994879}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870994419}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899488714}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850967252}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466115059}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128063089}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416515822}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612570454}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018220424}) 2024-09-17T16:19:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094150758}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3010394247}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3010929325}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3409118182}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38098201}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3540394655}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1099896466}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=102094452}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443039166}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2483068486}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059000873}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466200765}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1978711023}) 2024-09-17T16:19:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3513013153}) 2024-09-17T16:19:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3752808804}) 2024-09-17T16:19:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=63,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13} crc {logm=3039075746,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=2350606054}) 2024-09-17T16:19:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=4191902087,mon_config_key=1345730000}) 2024-09-17T16:19:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=163066068}) 2024-09-17T16:19:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,nvmeofgw=3,osd_metadata=5} crc {mon_config_key=2434274148,monmap=2143941632,nvmeofgw=598337686,osd_metadata=547839073}) 2024-09-17T16:19:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=1701853976,osd_pg_creating=2714581885,osdmap=4245895138}) 2024-09-17T16:19:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=29} crc {osdmap=133688214}) 2024-09-17T16:19:43.320 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:43.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: pgmap v822: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 3377/228987 objects misplaced (1.475%); 8.7 MiB/s, 138 objects/s recovering 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: osdmap e357: 12 total, 12 up, 11 in 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=54,logm=3} crc {auth=2177375717,config=1059521453,health=3079269128,kv=420973415,logm=3738862875}) 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1906617587}) 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2377789491}) 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2237723371}) 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3659663829}) 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1571373277}) 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3731141147}) 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3805424563}) 2024-09-17T16:19:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3273012725}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508872391}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263597814}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069994879}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870994419}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899488714}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850967252}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466115059}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128063089}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416515822}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612570454}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018220424}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094150758}) 2024-09-17T16:19:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3010394247}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3010929325}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3409118182}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38098201}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3540394655}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1099896466}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=102094452}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443039166}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2483068486}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4059000873}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466200765}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1978711023}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3513013153}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3752808804}) 2024-09-17T16:19:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=63,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13} crc {logm=3039075746,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=2350606054}) 2024-09-17T16:19:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=2,mon_config_key=98} crc {mgrstat=4191902087,mon_config_key=1345730000}) 2024-09-17T16:19:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=163066068}) 2024-09-17T16:19:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,nvmeofgw=3,osd_metadata=5} crc {mon_config_key=2434274148,monmap=2143941632,nvmeofgw=598337686,osd_metadata=547839073}) 2024-09-17T16:19:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=1701853976,osd_pg_creating=2714581885,osdmap=4245895138}) 2024-09-17T16:19:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=29} crc {osdmap=133688214}) 2024-09-17T16:19:43.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:43.450 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:43.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:43.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:43.581 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:43.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:43.683 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:43.707 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:43.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:43.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:43.841 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:43.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:43.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:43.966 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:43.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:44.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:44.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:44.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:44.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:44.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:44.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:44.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:44.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:44.337 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:44.363 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:44.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:44.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:44.466 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:44.493 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:44.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:44.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:44.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:44.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:44.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:44.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:44.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:44.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:44.848 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:44.877 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:44.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:44.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:45.015 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:45.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:45.117 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:45.149 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:45.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:45.251 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:45.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:45.275 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:45.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:45.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:45.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:45.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:45.509 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:45.535 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:45.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:45.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:45.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:45.667 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:45 smithi017 ceph-mon[27571]: pgmap v824: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 843 op/s; 3377/227319 objects misplaced (1.486%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:19:45.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:45.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:45.798 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:45 smithi071 ceph-mon[30002]: pgmap v824: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 843 op/s; 3377/227319 objects misplaced (1.486%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:19:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:45 smithi173 ceph-mon[30530]: pgmap v824: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 843 op/s; 3377/227319 objects misplaced (1.486%); 7.6 MiB/s, 121 objects/s recovering 2024-09-17T16:19:45.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:45.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:45.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:45.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:46.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:46.062 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:46.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:46.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:46.070 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:19:46.085 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:46.086 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:46.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:46.189 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:46.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:46.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:46.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:46.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:46.420 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:46.447 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:46.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:46.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:46.575 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:46.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:46.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:46.707 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:46.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:46.809 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:46.838 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:46.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:46.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:46.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:46.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:46.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:47.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:47.114 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:47.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:47.205 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:19:47.205 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:19:47.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:47.235 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:19:47.235 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:19:47.238 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:19:47.238 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:19:47.242 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:47.243 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:47.244 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:47.274 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:19:47.275 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:19:47.306 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:19:47.307 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:19:47.338 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:19:47.339 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:19:47.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:47.367 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:19:47.368 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:19:47.369 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:19:47.382 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:47.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:47.399 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:19:47.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:19:47.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:19:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:19:47.468 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:19:47.468 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:19:47.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:47.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:47 smithi017 ceph-mon[27571]: pgmap v825: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 975 op/s; 3098/219372 objects misplaced (1.412%); 7.3 MiB/s, 117 objects/s recovering 2024-09-17T16:19:47.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:47 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:19:47.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:47 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:19:47.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:47 smithi017 ceph-mon[27571]: osdmap e358: 12 total, 12 up, 11 in 2024-09-17T16:19:47.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:47 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:19:47.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:47 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:19:47.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1541214498' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:47.509 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:19:47.510 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:19:47.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:47.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:47.541 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:19:47.541 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:19:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:47 smithi071 ceph-mon[30002]: pgmap v825: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 975 op/s; 3098/219372 objects misplaced (1.412%); 7.3 MiB/s, 117 objects/s recovering 2024-09-17T16:19:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:47 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:19:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:47 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:19:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:47 smithi071 ceph-mon[30002]: osdmap e358: 12 total, 12 up, 11 in 2024-09-17T16:19:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:47 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:19:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:47 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:19:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1541214498' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:47.582 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:19:47.582 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:19:47.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:47.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:47.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:47.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:47.739 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:47.766 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:47.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:47.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:47.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:47.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:47.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:47 smithi173 ceph-mon[30530]: pgmap v825: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 975 op/s; 3098/219372 objects misplaced (1.412%); 7.3 MiB/s, 117 objects/s recovering 2024-09-17T16:19:47.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:47 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:19:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:47 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:19:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:47 smithi173 ceph-mon[30530]: osdmap e358: 12 total, 12 up, 11 in 2024-09-17T16:19:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:47 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:19:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:47 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:19:47.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1541214498' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:47.994 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:48.024 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:48.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:48.126 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:48.147 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:48.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:48.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:48.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:48.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:48.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:48.298 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":826,"stamp":"2024-09-17T16:19:46.408712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4792648248,"num_objects":73124,"num_object_clones":0,"num_object_copies":219372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3098,"num_objects_unfound":0,"num_objects_dirty":73124,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":131347,"num_write_kb":6541159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6386,"num_bytes_recovered":419706464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131316,"ondisk_log_size":131316,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":16780900,"kb_used_data":14457256,"kb_used_omap":57,"kb_used_meta":2323078,"kb_avail":1014415772,"statfs":{"total":1055945392128,"available":1038761750528,"internally_reserved":0,"allocated":14804230144,"data_stored":14801134119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2378832606},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,9,5,5,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-398983168,"num_objects":-6088,"num_object_clones":0,"num_object_copies":-18264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-656,"num_objects_unfound":0,"num_objects_dirty":-6088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6088,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":656,"num_bytes_recovered":42991616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.569031"},"pg_stats":[{"pgid":"4.f","version":"356'8049","reported_seq":8160,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-17T16:19:41.310941+0000","last_change":"2024-09-17T16:17:41.531577+0000","last_active":"2024-09-17T16:19:41.310941+0000","last_peered":"2024-09-17T16:19:41.310941+0000","last_clean":"2024-09-17T16:19:41.310941+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:19:41.310941+0000","last_undegraded":"2024-09-17T16:19:41.310941+0000","last_fullsized":"2024-09-17T16:19:41.310941+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8049,"log_dups_size":0,"ondisk_log_size":8049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.077147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305594368,"num_objects":4663,"num_object_clones":0,"num_object_copies":13989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8049,"num_write_kb":406784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"357'8194","reported_seq":8316,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:44.626543+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:19:44.626543+0000","last_peered":"2024-09-17T16:19:44.626543+0000","last_clean":"2024-09-17T16:19:44.626543+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:19:44.626543+0000","last_undegraded":"2024-09-17T16:19:44.626543+0000","last_fullsized":"2024-09-17T16:19:44.626543+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8194,"log_dups_size":0,"ondisk_log_size":8194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297009152,"num_objects":4532,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8194,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"357'7975","reported_seq":8103,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:44.622228+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:19:44.622228+0000","last_peered":"2024-09-17T16:19:44.622228+0000","last_clean":"2024-09-17T16:19:44.622228+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:19:44.622228+0000","last_undegraded":"2024-09-17T16:19:44.622228+0000","last_fullsized":"2024-09-17T16:19:44.622228+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293077016,"num_objects":4473,"num_object_clones":0,"num_object_copies":13419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7975,"num_write_kb":398273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"357'8370","reported_seq":8492,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:44.617395+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:19:44.617395+0000","last_peered":"2024-09-17T16:19:44.617395+0000","last_clean":"2024-09-17T16:19:44.617395+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:19:44.617395+0000","last_undegraded":"2024-09-17T16:19:44.617395+0000","last_fullsized":"2024-09-17T16:19:44.617395+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8370,"log_dups_size":0,"ondisk_log_size":8370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304611328,"num_objects":4648,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8370,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"357'8270","reported_seq":14273,"reported_epoch":357,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:44.633174+0000","last_change":"2024-09-17T16:18:41.641380+0000","last_active":"2024-09-17T16:19:44.633174+0000","last_peered":"2024-09-17T16:19:44.633174+0000","last_clean":"2024-09-17T16:18:39.384812+0000","last_became_active":"2024-09-17T16:18:41.595599+0000","last_became_peered":"2024-09-17T16:18:41.595599+0000","last_unstale":"2024-09-17T16:19:44.633174+0000","last_undegraded":"2024-09-17T16:19:44.633174+0000","last_fullsized":"2024-09-17T16:18:41.426452+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8270,"log_dups_size":0,"ondisk_log_size":8270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296353792,"num_objects":4522,"num_object_clones":0,"num_object_copies":13566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1864,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8270,"num_write_kb":409344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2943,"num_bytes_recovered":192872448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4","objects":2658},{"shards":"0,4,5","objects":1864}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"357'8421","reported_seq":8614,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:45.662818+0000","last_change":"2024-09-17T16:19:02.221681+0000","last_active":"2024-09-17T16:19:45.662818+0000","last_peered":"2024-09-17T16:19:45.662818+0000","last_clean":"2024-09-17T16:19:45.662818+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:19:45.662818+0000","last_undegraded":"2024-09-17T16:19:45.662818+0000","last_fullsized":"2024-09-17T16:19:45.662818+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5925","last_scrub_stamp":"2024-09-17T16:19:02.221472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.221472+0000","objects_scrubbed":5879,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.368910+0000","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295763968,"num_objects":4513,"num_object_clones":0,"num_object_copies":13539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"356'8002","reported_seq":8180,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-17T16:19:41.386645+0000","last_change":"2024-09-17T16:19:02.418259+0000","last_active":"2024-09-17T16:19:41.386645+0000","last_peered":"2024-09-17T16:19:41.386645+0000","last_clean":"2024-09-17T16:19:41.386645+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:19:41.386645+0000","last_undegraded":"2024-09-17T16:19:41.386645+0000","last_fullsized":"2024-09-17T16:19:41.386645+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5797","last_scrub_stamp":"2024-09-17T16:19:02.418176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.418176+0000","objects_scrubbed":5756,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:17.281193+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299499520,"num_objects":4570,"num_object_clones":0,"num_object_copies":13710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8002,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"357'8136","reported_seq":8258,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:42.885190+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:19:42.885190+0000","last_peered":"2024-09-17T16:19:42.885190+0000","last_clean":"2024-09-17T16:19:42.885190+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:19:42.885190+0000","last_undegraded":"2024-09-17T16:19:42.885190+0000","last_fullsized":"2024-09-17T16:19:42.885190+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302907392,"num_objects":4622,"num_object_clones":0,"num_object_copies":13866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8136,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"357'8307","reported_seq":8436,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:45.769523+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:19:45.769523+0000","last_peered":"2024-09-17T16:19:45.769523+0000","last_clean":"2024-09-17T16:19:45.769523+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:19:45.769523+0000","last_undegraded":"2024-09-17T16:19:45.769523+0000","last_fullsized":"2024-09-17T16:19:45.769523+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":8307,"log_dups_size":0,"ondisk_log_size":8307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:18.005666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291438592,"num_objects":4447,"num_object_clones":0,"num_object_copies":13341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"356'7981","reported_seq":14965,"reported_epoch":356,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:41.729568+0000","last_change":"2024-09-17T16:18:42.321603+0000","last_active":"2024-09-17T16:19:41.729568+0000","last_peered":"2024-09-17T16:19:41.729568+0000","last_clean":"2024-09-17T16:18:39.337421+0000","last_became_active":"2024-09-17T16:18:42.219868+0000","last_became_peered":"2024-09-17T16:18:42.219868+0000","last_unstale":"2024-09-17T16:19:41.729568+0000","last_undegraded":"2024-09-17T16:19:41.729568+0000","last_fullsized":"2024-09-17T16:18:41.426290+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":7981,"log_dups_size":0,"ondisk_log_size":7981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297467904,"num_objects":4539,"num_object_clones":0,"num_object_copies":13617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1234,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7981,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3436,"num_bytes_recovered":225181696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[2,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":1234},{"shards":"2,7","objects":3305}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"356'8099","reported_seq":8210,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-17T16:19:41.733166+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:19:41.733166+0000","last_peered":"2024-09-17T16:19:41.733166+0000","last_clean":"2024-09-17T16:19:41.733166+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:19:41.733166+0000","last_undegraded":"2024-09-17T16:19:41.733166+0000","last_fullsized":"2024-09-17T16:19:41.733166+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8099,"log_dups_size":0,"ondisk_log_size":8099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307036160,"num_objects":4685,"num_object_clones":0,"num_object_copies":14055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8099,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"357'8293","reported_seq":8423,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:42.939882+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:19:42.939882+0000","last_peered":"2024-09-17T16:19:42.939882+0000","last_clean":"2024-09-17T16:19:42.939882+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:19:42.939882+0000","last_undegraded":"2024-09-17T16:19:42.939882+0000","last_fullsized":"2024-09-17T16:19:42.939882+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8293,"log_dups_size":0,"ondisk_log_size":8293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308609024,"num_objects":4709,"num_object_clones":0,"num_object_copies":14127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8293,"num_write_kb":416064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"357'8158","reported_seq":8289,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:45.491193+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:19:45.491193+0000","last_peered":"2024-09-17T16:19:45.491193+0000","last_clean":"2024-09-17T16:19:45.491193+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:19:45.491193+0000","last_undegraded":"2024-09-17T16:19:45.491193+0000","last_fullsized":"2024-09-17T16:19:45.491193+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8158,"log_dups_size":0,"ondisk_log_size":8158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:45.334658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287965184,"num_objects":4394,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"357'8394","reported_seq":8591,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:45.481088+0000","last_change":"2024-09-17T16:19:38.397911+0000","last_active":"2024-09-17T16:19:45.481088+0000","last_peered":"2024-09-17T16:19:45.481088+0000","last_clean":"2024-09-17T16:19:45.481088+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:19:45.481088+0000","last_undegraded":"2024-09-17T16:19:45.481088+0000","last_fullsized":"2024-09-17T16:19:45.481088+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'7961","last_scrub_stamp":"2024-09-17T16:19:38.397698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:38.397698+0000","objects_scrubbed":4979,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:39.699266+0000","scrub_duration":1058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296222720,"num_objects":4520,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8394,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"356'8039","reported_seq":8225,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-17T16:19:41.384913+0000","last_change":"2024-09-17T16:19:40.984613+0000","last_active":"2024-09-17T16:19:41.384913+0000","last_peered":"2024-09-17T16:19:41.384913+0000","last_clean":"2024-09-17T16:19:41.384913+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:19:41.384913+0000","last_undegraded":"2024-09-17T16:19:41.384913+0000","last_fullsized":"2024-09-17T16:19:41.384913+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":2499,"log_size":8039,"log_dups_size":0,"ondisk_log_size":8039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:10.939951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307822592,"num_objects":4697,"num_object_clones":0,"num_object_copies":14091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8039,"num_write_kb":407552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"357'8536","reported_seq":8725,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:45.622868+0000","last_change":"2024-09-17T16:19:01.159281+0000","last_active":"2024-09-17T16:19:45.622868+0000","last_peered":"2024-09-17T16:19:45.622868+0000","last_clean":"2024-09-17T16:19:45.622868+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:19:45.622868+0000","last_undegraded":"2024-09-17T16:19:45.622868+0000","last_fullsized":"2024-09-17T16:19:45.622868+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5950","last_scrub_stamp":"2024-09-17T16:19:01.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:01.159158+0000","objects_scrubbed":5911,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:27.590602+0000","scrub_duration":1399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300679168,"num_objects":4588,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8536,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":607,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-09-17T16:19:41.938489+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:19:41.938489+0000","last_peered":"2024-09-17T16:19:41.938489+0000","last_clean":"2024-09-17T16:19:41.938489+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:19:41.938489+0000","last_undegraded":"2024-09-17T16:19:41.938489+0000","last_fullsized":"2024-09-17T16:19:41.938489+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4792057880,"num_objects":73122,"num_object_clones":0,"num_object_copies":219366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3098,"num_objects_unfound":0,"num_objects_dirty":73122,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131224,"num_write_kb":6539009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6380,"num_bytes_recovered":418119680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14722871296,"data_stored":14722859080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131224,"ondisk_log_size":131224,"up":48,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285691,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2616432,"kb_used_data":2324192,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91128720,"statfs":{"total":95995035648,"available":93315809280,"internally_reserved":0,"allocated":2379972608,"data_stored":2379684781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":299170244},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547328,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1602276,"kb_used_data":1454484,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92142876,"statfs":{"total":95995035648,"available":94354305024,"internally_reserved":0,"allocated":1489391616,"data_stored":1489111433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743556,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1608300,"kb_used_data":1266072,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92136852,"statfs":{"total":95995035648,"available":94348136448,"internally_reserved":0,"allocated":1296457728,"data_stored":1296173473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":350353856},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122152,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93623000,"statfs":{"total":95995035648,"available":95869952000,"internally_reserved":0,"allocated":741376,"data_stored":461193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963670,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1970544,"kb_used_data":1777692,"kb_used_omap":41,"kb_used_meta":192790,"kb_avail":91774608,"statfs":{"total":95995035648,"available":93977198592,"internally_reserved":0,"allocated":1820356608,"data_stored":1820072873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":197417267},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2262444,"kb_used_data":2042460,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":91482708,"statfs":{"total":95995035648,"available":93678292992,"internally_reserved":0,"allocated":2091479040,"data_stored":2091195305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":225245638},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724695,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1390756,"kb_used_data":1179092,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92354396,"statfs":{"total":95995035648,"available":94570901504,"internally_reserved":0,"allocated":1207390208,"data_stored":1207110025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986334,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1375848,"kb_used_data":1169752,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":92369304,"statfs":{"total":95995035648,"available":94586167296,"internally_reserved":0,"allocated":1197826048,"data_stored":1197541793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":211024325},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1753880,"kb_used_data":1506056,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91991272,"statfs":{"total":95995035648,"available":94199062528,"internally_reserved":0,"allocated":1542201344,"data_stored":1541925601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":253688263},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411495,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742808,"kb_used_data":594952,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93002344,"statfs":{"total":95995035648,"available":95234400256,"internally_reserved":0,"allocated":609230848,"data_stored":608955105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1335460,"kb_used_data":1141780,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92409692,"statfs":{"total":95995035648,"available":94627524608,"internally_reserved":0,"allocated":1169182720,"data_stored":1168902537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1165623296,"data_stored":1165623296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":604438528,"data_stored":604438528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1537540096,"data_stored":1537540096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1191055360,"data_stored":1191051288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1205207040,"data_stored":1205207040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2064777216,"data_stored":2064777216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1817772032,"data_stored":1817772032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1294995456,"data_stored":1294991384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1471086592,"data_stored":1471086592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2370375680,"data_stored":2370371608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:19:48.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:48.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:48.407 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:48.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:48.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:48.537 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:48.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:48.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:48.668 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:48.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:48 smithi017 ceph-mon[27571]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:48 smithi017 ceph-mon[27571]: osdmap e359: 12 total, 12 up, 11 in 2024-09-17T16:19:48.770 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:48.797 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:48.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:48 smithi071 ceph-mon[30002]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:48 smithi071 ceph-mon[30002]: osdmap e359: 12 total, 12 up, 11 in 2024-09-17T16:19:48.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:48 smithi173 ceph-mon[30530]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:48 smithi173 ceph-mon[30530]: osdmap e359: 12 total, 12 up, 11 in 2024-09-17T16:19:48.899 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:48.928 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:48.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:49.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:49.059 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:49.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:49.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:49.187 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:49.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:49.289 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:49.313 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:19:49.313 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:19:49.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:49.316 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:49.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:49.418 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:49.451 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:49.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:49.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:49.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:49.576 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:49.579 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:49.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:49.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:49 smithi017 ceph-mon[27571]: pgmap v828: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 2636/215205 objects misplaced (1.225%); 7.7 MiB/s, 123 objects/s recovering 2024-09-17T16:19:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:49.711 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:49.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:49.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:49 smithi071 ceph-mon[30002]: pgmap v828: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 2636/215205 objects misplaced (1.225%); 7.7 MiB/s, 123 objects/s recovering 2024-09-17T16:19:49.839 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:49.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:49.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:49 smithi173 ceph-mon[30530]: pgmap v828: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 2636/215205 objects misplaced (1.225%); 7.7 MiB/s, 123 objects/s recovering 2024-09-17T16:19:49.941 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:49.966 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:49.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:50.068 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:50.095 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:50.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:50.166 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:19:50.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:50.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:50.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:50.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:50.352 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:50.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:50.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:50.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:50.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:50.583 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:50.615 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:50.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:50.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:50 smithi017 ceph-mon[27571]: pgmap v829: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 805 op/s; 2353/209877 objects misplaced (1.121%); 8.1 MiB/s, 129 objects/s recovering 2024-09-17T16:19:50.744 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:50.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:50.780 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:19:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:50 smithi071 ceph-mon[30002]: pgmap v829: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 805 op/s; 2353/209877 objects misplaced (1.121%); 8.1 MiB/s, 129 objects/s recovering 2024-09-17T16:19:50.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:50.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:50.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:50 smithi173 ceph-mon[30530]: pgmap v829: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 805 op/s; 2353/209877 objects misplaced (1.121%); 8.1 MiB/s, 129 objects/s recovering 2024-09-17T16:19:50.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:51.003 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:51.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:51.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:51.132 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:51.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:51.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:51.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:51.259 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:51.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:51.300 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:19:51.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:51.370 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:51.379 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:51.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:51.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:51.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:51.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:51.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:51.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:51.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:51.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:51.658 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:51.699 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:19:51.759 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:51.787 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:51.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:51.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:51.793 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/27250373' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:51.794 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:51 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:51.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:51.915 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:51.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:52.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:52.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:52.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:52.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/27250373' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:52.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:51 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:52.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/27250373' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:52.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:51 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:19:52.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:52.179 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:52.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:52.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:52.306 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:52.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:52.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:52.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:52.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:52.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:52.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:52.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:52.664 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:52.691 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:19:52.692 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:19:52.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:52 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:52.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:52 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:19:52.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:52 smithi017 ceph-mon[27571]: osdmap e360: 12 total, 12 up, 11 in 2024-09-17T16:19:52.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:52 smithi017 ceph-mon[27571]: pgmap v831: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 995 op/s; 1770/201456 objects misplaced (0.879%); 14 MiB/s, 221 objects/s recovering 2024-09-17T16:19:52.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2824941945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:52.694 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:52.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:52.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:52.719 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:19:52.719 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:19:52.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:19:52.745 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:19:52.771 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:19:52.772 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:19:52.795 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:19:52.795 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:19:52.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:52.819 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:19:52.819 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:19:52.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:52.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:52.855 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:19:52.856 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:19:52.894 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:19:52.894 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:19:52.923 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:19:52.923 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:19:52.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:52.953 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:19:52.954 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:19:52.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:52.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:52.966 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:19:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:19:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:139498 4.0 4:05b841c6:::benchmark_data_smithi173_77474_object37323:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:19:52.923688+0000", 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.040158341, 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.040194264, 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 139498 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:19:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:19:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.923688+0000", 2024-09-17T16:19:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:19:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:19:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.923688+0000", 2024-09-17T16:19:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.923689+0000", 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0279999999999999e-06 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.923694+0000", 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0679999999999996e-06 2024-09-17T16:19:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:19:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.923707+0000", 2024-09-17T16:19:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2914999999999999e-05 2024-09-17T16:19:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:19:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.923713+0000", 2024-09-17T16:19:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6869999999999997e-06 2024-09-17T16:19:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:19:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.925922+0000", 2024-09-17T16:19:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002208804 2024-09-17T16:19:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:19:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.925997+0000", 2024-09-17T16:19:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4605999999999997e-05 2024-09-17T16:19:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:19:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.926060+0000", 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3439e-05 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.929332+0000", 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032714950000000001 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.929615+0000", 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00028383999999999999 2024-09-17T16:19:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:139522 4.f 4:f2ee6c87:::benchmark_data_smithi173_77474_object37347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:19:52.955494+0000", 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.0083519470000000002, 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0084885550000000001, 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:19:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 139522 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.955494+0000", 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.955494+0000", 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:19:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:19:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.955495+0000", 2024-09-17T16:19:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.364e-06 2024-09-17T16:19:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:19:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.955500+0000", 2024-09-17T16:19:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0919999999999997e-06 2024-09-17T16:19:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:19:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.955514+0000", 2024-09-17T16:19:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3409e-05 2024-09-17T16:19:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:19:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.955521+0000", 2024-09-17T16:19:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7950000000000002e-06 2024-09-17T16:19:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:19:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.955584+0000", 2024-09-17T16:19:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3169000000000005e-05 2024-09-17T16:19:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:19:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.955693+0000", 2024-09-17T16:19:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010908699999999999 2024-09-17T16:19:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:19:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.955786+0000", 2024-09-17T16:19:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3516000000000004e-05 2024-09-17T16:19:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:19:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.959908+0000", 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041216120000000002 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:52.960862+0000", 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00095410099999999995 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 2 2024-09-17T16:19:52.980 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:19:52.986 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:19:52.987 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:19:53.016 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:19:53.016 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:19:53.061 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:53.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:52 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:53.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:52 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:19:53.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:52 smithi071 ceph-mon[30002]: osdmap e360: 12 total, 12 up, 11 in 2024-09-17T16:19:53.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:52 smithi071 ceph-mon[30002]: pgmap v831: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 995 op/s; 1770/201456 objects misplaced (0.879%); 14 MiB/s, 221 objects/s recovering 2024-09-17T16:19:53.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2824941945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:53.086 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:53.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:53.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:52 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:19:53.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:52 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:19:53.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:52 smithi173 ceph-mon[30530]: osdmap e360: 12 total, 12 up, 11 in 2024-09-17T16:19:53.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:52 smithi173 ceph-mon[30530]: pgmap v831: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 995 op/s; 1770/201456 objects misplaced (0.879%); 14 MiB/s, 221 objects/s recovering 2024-09-17T16:19:53.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2824941945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:19:53.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:53.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:53.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:53.317 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:53.343 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:53.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:53.444 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:53.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:53.470 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:53.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:53.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:19:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:53.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:53.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:19:53.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:53.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:53.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:53.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:53.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:53.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:53.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:53.987 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:53.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:54.089 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:54.111 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:54.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:54.213 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:54.235 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:54.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:54.305 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:19:54.338 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:54.360 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:54.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:54.462 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:54.484 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:54.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:54.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:54.612 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:54.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:54.714 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:54.739 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:54.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:54.840 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:54.875 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:54.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:54.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:54.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:54 smithi017 ceph-mon[27571]: osdmap e361: 12 total, 12 up, 11 in 2024-09-17T16:19:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:54 smithi017 ceph-mon[27571]: pgmap v833: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 796 op/s; 1770/199836 objects misplaced (0.886%); 8.4 MiB/s, 134 objects/s recovering 2024-09-17T16:19:55.001 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:55.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:55.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:54 smithi071 ceph-mon[30002]: osdmap e361: 12 total, 12 up, 11 in 2024-09-17T16:19:55.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:54 smithi071 ceph-mon[30002]: pgmap v833: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 796 op/s; 1770/199836 objects misplaced (0.886%); 8.4 MiB/s, 134 objects/s recovering 2024-09-17T16:19:55.102 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:55.128 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:55.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:55.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:55.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:54 smithi173 ceph-mon[30530]: osdmap e361: 12 total, 12 up, 11 in 2024-09-17T16:19:55.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:54 smithi173 ceph-mon[30530]: pgmap v833: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 796 op/s; 1770/199836 objects misplaced (0.886%); 8.4 MiB/s, 134 objects/s recovering 2024-09-17T16:19:55.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:55.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:55.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:55.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:19:55.386 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:55.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:55.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:19:55.488 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:55.513 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:55.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:55.604 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:19:55.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:55.641 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:55.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:55.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:55.684 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:55.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:55.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:55.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:55.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:55.870 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:55.897 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:55.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:55.969 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:55 smithi017 ceph-mon[27571]: osdmap e362: 12 total, 12 up, 11 in 2024-09-17T16:19:55.994 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:19:55.995 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:19:55.998 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:56.026 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:56.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:56.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:56.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:55 smithi071 ceph-mon[30002]: osdmap e362: 12 total, 12 up, 11 in 2024-09-17T16:19:56.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:56.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:55 smithi173 ceph-mon[30530]: osdmap e362: 12 total, 12 up, 11 in 2024-09-17T16:19:56.152 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:56.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:56.240 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:19:56.240 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:19:56.240 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:19:56.240 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:19:56.240 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:19:56.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:56.280 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:56.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:56.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:19:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:56.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:19:56.511 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:56.537 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:56.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:56.639 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:56.674 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:56.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:56.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:56.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:56.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:56.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:56.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:56.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:56 smithi017 ceph-mon[27571]: pgmap v835: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 929 op/s; 1321/193137 objects misplaced (0.684%); 11 MiB/s, 171 objects/s recovering 2024-09-17T16:19:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:56 smithi017 ceph-mon[27571]: from='client.25868 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:57.039 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:56 smithi071 ceph-mon[30002]: pgmap v835: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 929 op/s; 1321/193137 objects misplaced (0.684%); 11 MiB/s, 171 objects/s recovering 2024-09-17T16:19:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:56 smithi071 ceph-mon[30002]: from='client.25868 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:57.070 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:57.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:57.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:57.108 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:19:57.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:56 smithi173 ceph-mon[30530]: pgmap v835: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 929 op/s; 1321/193137 objects misplaced (0.684%); 11 MiB/s, 171 objects/s recovering 2024-09-17T16:19:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:56 smithi173 ceph-mon[30530]: from='client.25868 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:19:57.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:57.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:57.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:57.301 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":835,"stamp":"2024-09-17T16:19:55.842896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4219535928,"num_objects":64379,"num_object_clones":0,"num_object_copies":193137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1321,"num_objects_unfound":0,"num_objects_dirty":64379,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":140092,"num_write_kb":6541159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8163,"num_bytes_recovered":536163936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140061,"ondisk_log_size":140061,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":15298712,"kb_used_data":12933332,"kb_used_omap":57,"kb_used_meta":2364742,"kb_avail":1015897960,"statfs":{"total":1055945392128,"available":1040279511040,"internally_reserved":0,"allocated":13243731968,"data_stored":13240811222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2421496542},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-365690880,"num_objects":-5580,"num_object_clones":0,"num_object_copies":-16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1032,"num_objects_unfound":0,"num_objects_dirty":-5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5580,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":67633152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001265"},"pg_stats":[{"pgid":"4.f","version":"359'8600","reported_seq":8717,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-17T16:19:51.380173+0000","last_change":"2024-09-17T16:17:41.531577+0000","last_active":"2024-09-17T16:19:51.380173+0000","last_peered":"2024-09-17T16:19:51.380173+0000","last_clean":"2024-09-17T16:19:51.380173+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:19:51.380173+0000","last_undegraded":"2024-09-17T16:19:51.380173+0000","last_fullsized":"2024-09-17T16:19:51.380173+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.077147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269484032,"num_objects":4112,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":406784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"361'8750","reported_seq":8883,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-17T16:19:54.694286+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:19:54.694286+0000","last_peered":"2024-09-17T16:19:54.694286+0000","last_clean":"2024-09-17T16:19:54.694286+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:19:54.694286+0000","last_undegraded":"2024-09-17T16:19:54.694286+0000","last_fullsized":"2024-09-17T16:19:54.694286+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8750,"log_dups_size":0,"ondisk_log_size":8750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260571136,"num_objects":3976,"num_object_clones":0,"num_object_copies":11928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8750,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"361'8493","reported_seq":8634,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-17T16:19:54.694284+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:19:54.694284+0000","last_peered":"2024-09-17T16:19:54.694284+0000","last_clean":"2024-09-17T16:19:54.694284+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:19:54.694284+0000","last_undegraded":"2024-09-17T16:19:54.694284+0000","last_fullsized":"2024-09-17T16:19:54.694284+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8493,"log_dups_size":0,"ondisk_log_size":8493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259129368,"num_objects":3955,"num_object_clones":0,"num_object_copies":11865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3955,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8493,"num_write_kb":398273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"361'8896","reported_seq":9028,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-17T16:19:54.694284+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:19:54.694284+0000","last_peered":"2024-09-17T16:19:54.694284+0000","last_clean":"2024-09-17T16:19:54.694284+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:19:54.694284+0000","last_undegraded":"2024-09-17T16:19:54.694284+0000","last_fullsized":"2024-09-17T16:19:54.694284+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8896,"log_dups_size":0,"ondisk_log_size":8896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270139392,"num_objects":4122,"num_object_clones":0,"num_object_copies":12366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8896,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"361'8827","reported_seq":16304,"reported_epoch":362,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:54.700769+0000","last_change":"2024-09-17T16:18:41.641380+0000","last_active":"2024-09-17T16:19:54.700769+0000","last_peered":"2024-09-17T16:19:54.700769+0000","last_clean":"2024-09-17T16:18:39.384812+0000","last_became_active":"2024-09-17T16:18:41.595599+0000","last_became_peered":"2024-09-17T16:18:41.595599+0000","last_unstale":"2024-09-17T16:19:54.700769+0000","last_undegraded":"2024-09-17T16:19:54.700769+0000","last_fullsized":"2024-09-17T16:18:41.426452+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8827,"log_dups_size":0,"ondisk_log_size":8827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259850240,"num_objects":3965,"num_object_clones":0,"num_object_copies":11895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1132,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":409344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3675,"num_bytes_recovered":240844800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4","objects":2833},{"shards":"0,4,5","objects":1132}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"362'8965","reported_seq":9169,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-17T16:19:55.649362+0000","last_change":"2024-09-17T16:19:02.221681+0000","last_active":"2024-09-17T16:19:55.649362+0000","last_peered":"2024-09-17T16:19:55.649362+0000","last_clean":"2024-09-17T16:19:55.649362+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:19:55.649362+0000","last_undegraded":"2024-09-17T16:19:55.649362+0000","last_fullsized":"2024-09-17T16:19:55.649362+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5925","last_scrub_stamp":"2024-09-17T16:19:02.221472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.221472+0000","objects_scrubbed":5879,"log_size":8965,"log_dups_size":0,"ondisk_log_size":8965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.368910+0000","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260112384,"num_objects":3969,"num_object_clones":0,"num_object_copies":11907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8965,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"359'8526","reported_seq":8710,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-17T16:19:51.384514+0000","last_change":"2024-09-17T16:19:02.418259+0000","last_active":"2024-09-17T16:19:51.384514+0000","last_peered":"2024-09-17T16:19:51.384514+0000","last_clean":"2024-09-17T16:19:51.384514+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:19:51.384514+0000","last_undegraded":"2024-09-17T16:19:51.384514+0000","last_fullsized":"2024-09-17T16:19:51.384514+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5797","last_scrub_stamp":"2024-09-17T16:19:02.418176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.418176+0000","objects_scrubbed":5756,"log_size":8526,"log_dups_size":0,"ondisk_log_size":8526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:17.281193+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265158656,"num_objects":4046,"num_object_clones":0,"num_object_copies":12138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8526,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"360'8707","reported_seq":8836,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-17T16:19:52.905085+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:19:52.905085+0000","last_peered":"2024-09-17T16:19:52.905085+0000","last_clean":"2024-09-17T16:19:52.905085+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:19:52.905085+0000","last_undegraded":"2024-09-17T16:19:52.905085+0000","last_fullsized":"2024-09-17T16:19:52.905085+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8707,"log_dups_size":0,"ondisk_log_size":8707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265486336,"num_objects":4051,"num_object_clones":0,"num_object_copies":12153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"362'8796","reported_seq":8936,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-17T16:19:55.734681+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:19:55.734681+0000","last_peered":"2024-09-17T16:19:55.734681+0000","last_clean":"2024-09-17T16:19:55.734681+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:19:55.734681+0000","last_undegraded":"2024-09-17T16:19:55.734681+0000","last_fullsized":"2024-09-17T16:19:55.734681+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":8796,"log_dups_size":0,"ondisk_log_size":8796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:18.005666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259391488,"num_objects":3958,"num_object_clones":0,"num_object_copies":11874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8796,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"360'8526","reported_seq":17609,"reported_epoch":360,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:19:51.739769+0000","last_change":"2024-09-17T16:18:42.321603+0000","last_active":"2024-09-17T16:19:51.739769+0000","last_peered":"2024-09-17T16:19:51.739769+0000","last_clean":"2024-09-17T16:18:39.337421+0000","last_became_active":"2024-09-17T16:18:42.219868+0000","last_became_peered":"2024-09-17T16:18:42.219868+0000","last_unstale":"2024-09-17T16:19:51.739769+0000","last_undegraded":"2024-09-17T16:19:51.739769+0000","last_fullsized":"2024-09-17T16:18:41.426290+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8526,"log_dups_size":0,"ondisk_log_size":8526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261750784,"num_objects":3994,"num_object_clones":0,"num_object_copies":11982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":189,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8526,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4481,"num_bytes_recovered":293666816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[2,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":189},{"shards":"2,7","objects":3805}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"360'8692","reported_seq":8810,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-17T16:19:51.734581+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:19:51.734581+0000","last_peered":"2024-09-17T16:19:51.734581+0000","last_clean":"2024-09-17T16:19:51.734581+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:19:51.734581+0000","last_undegraded":"2024-09-17T16:19:51.734581+0000","last_fullsized":"2024-09-17T16:19:51.734581+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8692,"log_dups_size":0,"ondisk_log_size":8692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268173312,"num_objects":4092,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8692,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"360'8885","reported_seq":9021,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-17T16:19:52.967811+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:19:52.967811+0000","last_peered":"2024-09-17T16:19:52.967811+0000","last_clean":"2024-09-17T16:19:52.967811+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:19:52.967811+0000","last_undegraded":"2024-09-17T16:19:52.967811+0000","last_fullsized":"2024-09-17T16:19:52.967811+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8885,"log_dups_size":0,"ondisk_log_size":8885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269811712,"num_objects":4117,"num_object_clones":0,"num_object_copies":12351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":416064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"362'8697","reported_seq":8839,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-17T16:19:55.509576+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:19:55.509576+0000","last_peered":"2024-09-17T16:19:55.509576+0000","last_clean":"2024-09-17T16:19:55.509576+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:19:55.509576+0000","last_undegraded":"2024-09-17T16:19:55.509576+0000","last_fullsized":"2024-09-17T16:19:55.509576+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":8697,"log_dups_size":0,"ondisk_log_size":8697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:45.334658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252641280,"num_objects":3855,"num_object_clones":0,"num_object_copies":11565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8697,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"362'8933","reported_seq":9140,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-17T16:19:55.476420+0000","last_change":"2024-09-17T16:19:38.397911+0000","last_active":"2024-09-17T16:19:55.476420+0000","last_peered":"2024-09-17T16:19:55.476420+0000","last_clean":"2024-09-17T16:19:55.476420+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:19:55.476420+0000","last_undegraded":"2024-09-17T16:19:55.476420+0000","last_fullsized":"2024-09-17T16:19:55.476420+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'7961","last_scrub_stamp":"2024-09-17T16:19:38.397698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:38.397698+0000","objects_scrubbed":4979,"log_size":8933,"log_dups_size":0,"ondisk_log_size":8933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:39.699266+0000","scrub_duration":1058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260898816,"num_objects":3981,"num_object_clones":0,"num_object_copies":11943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8933,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"359'8581","reported_seq":8773,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-09-17T16:19:51.393798+0000","last_change":"2024-09-17T16:19:40.984613+0000","last_active":"2024-09-17T16:19:51.393798+0000","last_peered":"2024-09-17T16:19:51.393798+0000","last_clean":"2024-09-17T16:19:51.393798+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:19:51.393798+0000","last_undegraded":"2024-09-17T16:19:51.393798+0000","last_fullsized":"2024-09-17T16:19:51.393798+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":2499,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:10.939951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272302080,"num_objects":4155,"num_object_clones":0,"num_object_copies":12465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":407552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"362'9095","reported_seq":9293,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-17T16:19:55.627538+0000","last_change":"2024-09-17T16:19:01.159281+0000","last_active":"2024-09-17T16:19:55.627538+0000","last_peered":"2024-09-17T16:19:55.627538+0000","last_clean":"2024-09-17T16:19:55.627538+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:19:55.627538+0000","last_undegraded":"2024-09-17T16:19:55.627538+0000","last_fullsized":"2024-09-17T16:19:55.627538+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5950","last_scrub_stamp":"2024-09-17T16:19:01.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:01.159158+0000","objects_scrubbed":5911,"log_size":9095,"log_dups_size":0,"ondisk_log_size":9095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:27.590602+0000","scrub_duration":1399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264044544,"num_objects":4029,"num_object_clones":0,"num_object_copies":12087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":613,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-09-17T16:19:51.683832+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:19:51.683832+0000","last_peered":"2024-09-17T16:19:51.683832+0000","last_clean":"2024-09-17T16:19:51.683832+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:19:51.683832+0000","last_undegraded":"2024-09-17T16:19:51.683832+0000","last_fullsized":"2024-09-17T16:19:51.683832+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4218945560,"num_objects":64377,"num_object_clones":0,"num_object_copies":193131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1321,"num_objects_unfound":0,"num_objects_dirty":64377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139969,"num_write_kb":6539009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8157,"num_bytes_recovered":534577152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13180612608,"data_stored":13180600392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139969,"ondisk_log_size":139969,"up":48,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285693,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2334764,"kb_used_data":2042584,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91410388,"statfs":{"total":95995035648,"available":93604237312,"internally_reserved":0,"allocated":2091606016,"data_stored":2091345108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":299170244},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547330,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1421100,"kb_used_data":1281624,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92324052,"statfs":{"total":95995035648,"available":94539829248,"internally_reserved":0,"allocated":1312382976,"data_stored":1312125201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1534608,"kb_used_data":1192324,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92210544,"statfs":{"total":95995035648,"available":94423597056,"internally_reserved":0,"allocated":1220939776,"data_stored":1220686930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":350353856},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005195,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122080,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93623072,"statfs":{"total":95995035648,"available":95870025728,"internally_reserved":0,"allocated":667648,"data_stored":414384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963672,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1788052,"kb_used_data":1560392,"kb_used_omap":41,"kb_used_meta":227478,"kb_avail":91957100,"statfs":{"total":95995035648,"available":94164070400,"internally_reserved":0,"allocated":1597841408,"data_stored":1597589082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":232937779},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2053492,"kb_used_data":1801568,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91691660,"statfs":{"total":95995035648,"available":93892259840,"internally_reserved":0,"allocated":1844805632,"data_stored":1844544305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257948102},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724697,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1324144,"kb_used_data":1108248,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92421008,"statfs":{"total":95995035648,"available":94639112192,"internally_reserved":0,"allocated":1134845952,"data_stored":1134588177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":220985798},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986336,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1223780,"kb_used_data":1026000,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92521372,"statfs":{"total":95995035648,"available":94741884928,"internally_reserved":0,"allocated":1050624000,"data_stored":1050366664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1573820,"kb_used_data":1330220,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":92171332,"statfs":{"total":95995035648,"available":94383443968,"internally_reserved":0,"allocated":1362145280,"data_stored":1361856217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":249428423},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":655356,"kb_used_data":520044,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93089796,"statfs":{"total":95995035648,"available":95323951104,"internally_reserved":0,"allocated":532525056,"data_stored":532235993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738547,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267516,"kb_used_data":1069676,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92477636,"statfs":{"total":95995035648,"available":94697099264,"internally_reserved":0,"allocated":1095348224,"data_stored":1095059161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1092485120,"data_stored":1092485120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":529924096,"data_stored":529924096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1358036992,"data_stored":1358036992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1049694208,"data_stored":1049690136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1132658688,"data_stored":1132658688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1817313280,"data_stored":1817313280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1595473920,"data_stored":1595473920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1219497984,"data_stored":1219493912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1295319040,"data_stored":1295319040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2090209280,"data_stored":2090205208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:19:57.303 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-17T16:19:57.307 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:57.373 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:57.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:57.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:57.475 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:57.503 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:57.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:57.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:57.629 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:57.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:57.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:57.754 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:57.755 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:57.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:57.879 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:57.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:57.966 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:57.980 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:58.019 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:58.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:58.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:19:58.144 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:58.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:58.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:19:58.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:57 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-17T16:19:58.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:58.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:58.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:57 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-17T16:19:58.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:58.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:57 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-17T16:19:58.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:58.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:58.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:58.525 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:58.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:58.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:58.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:58.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:58.751 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:19:58.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:58.781 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:58.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:58.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:19:58.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:58.912 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:58.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:19:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:58 smithi017 ceph-mon[27571]: pgmap v836: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 727 op/s; 1132/189672 objects misplaced (0.597%); 12 MiB/s, 197 objects/s recovering 2024-09-17T16:19:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:19:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:19:59.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:19:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:59.046 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:59.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:19:59.147 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:19:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:59.172 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:59.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:19:59.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:19:59.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:59.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:59.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:19:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:58 smithi071 ceph-mon[30002]: pgmap v836: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 727 op/s; 1132/189672 objects misplaced (0.597%); 12 MiB/s, 197 objects/s recovering 2024-09-17T16:19:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:19:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:19:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:58 smithi173 ceph-mon[30530]: pgmap v836: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 727 op/s; 1132/189672 objects misplaced (0.597%); 12 MiB/s, 197 objects/s recovering 2024-09-17T16:19:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:19:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:19:59.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:59.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:59.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:19:59.520 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:59.523 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:59.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:19:59.525 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:19:59.547 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:59.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:19:59.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:19:59.671 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:19:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:59.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:19:59.773 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:19:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:59.796 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:19:59.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:19:59.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:19:59.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:19:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:19:59.929 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:00.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:00.056 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:00.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:13526 2.d e103/95 2:b276e7f5:::benchmark_data_smithi173_55121_object13525:head v 103'845, mlcod=103'842)", 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.327883+0000", 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 527.787177638, 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41382291199999999, 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.119 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.120 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.120 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.327883+0000", 2024-09-17T16:20:00.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.327883+0000", 2024-09-17T16:20:00.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.327884+0000", 2024-09-17T16:20:00.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5899999999999996e-07 2024-09-17T16:20:00.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.328029+0000", 2024-09-17T16:20:00.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014476699999999999 2024-09-17T16:20:00.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.740919+0000", 2024-09-17T16:20:00.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41289076499999999 2024-09-17T16:20:00.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.740932+0000", 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2248e-05 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741023+0000", 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1221e-05 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741050+0000", 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7132e-05 2024-09-17T16:20:00.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741673+0000", 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062324600000000004 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.741706+0000", 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2974e-05 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:13531 2.a e103/95 2:59aa3d24:::benchmark_data_smithi173_55121_object13530:head v 103'829, mlcod=103'826)", 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:12.331447+0000", 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 527.78361347500004, 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.507364445, 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331447+0000", 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331447+0000", 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331448+0000", 2024-09-17T16:20:00.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8899999999999999e-07 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.331493+0000", 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4728999999999998e-05 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838043+0000", 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.50655012099999996 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838053+0000", 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7850000000000003e-06 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838135+0000", 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2113000000000005e-05 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838163+0000", 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8181999999999999e-05 2024-09-17T16:20:00.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838781+0000", 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061825400000000005 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:12.838812+0000", 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0571999999999998e-05 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:20838 2.e e116/95 2:7e96b7fd:::benchmark_data_smithi173_55121_object20837:head v 116'1252, mlcod=116'1249)", 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:25.455525+0000", 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 514.65953604000003, 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31922061699999998, 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455525+0000", 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455525+0000", 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455526+0000", 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.33e-07 2024-09-17T16:20:00.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.455549+0000", 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3686999999999999e-05 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773668+0000", 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31811887700000002 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773674+0000", 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.037e-06 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773734+0000", 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9619e-05 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.773758+0000", 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4097999999999999e-05 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.774701+0000", 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00094260999999999998 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:25.774745+0000", 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4756000000000003e-05 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.135 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:25217 2.19 2:9a6d6ce1:::benchmark_data_smithi173_55121_object25216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:20:00.135 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.776796+0000", 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 509.33826468400002, 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47785892200000002, 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25217 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776796+0000", 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776796+0000", 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776798+0000", 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8899999999999999e-06 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776843+0000", 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5306999999999998e-05 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776856+0000", 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2858e-05 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776862+0000", 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3120000000000001e-06 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.776906+0000", 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4311000000000003e-05 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777001+0000", 2024-09-17T16:20:00.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4927999999999996e-05 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777093+0000", 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1878000000000004e-05 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.777671+0000", 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00057851599999999995 2024-09-17T16:20:00.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.036749+0000", 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.259078209 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254584+0000", 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21783461500000001 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254624+0000", 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9915e-05 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254655+0000", 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1183000000000003e-05 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:25217 2.19 e121/114 ondisk, result = 0)", 2024-09-17T16:20:00.142 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:30.781044+0000", 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 509.33401697800002, 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47361829999999999, 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781044+0000", 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781044+0000", 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781045+0000", 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4099999999999998e-07 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:30.781049+0000", 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4859999999999999e-06 2024-09-17T16:20:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254521+0000", 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47347151599999998 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254527+0000", 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5239999999999997e-06 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254576+0000", 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8445000000000001e-05 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254581+0000", 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:31.254662+0000", 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1454999999999999e-05 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:94007 2.19 2:9e42b13d:::benchmark_data_smithi173_55121_object17518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:20:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:20:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 430.22736856699998, 2024-09-17T16:20:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.467128445, 2024-09-17T16:20:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:20:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:20:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 94007 2024-09-17T16:20:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:20:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:20:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887693+0000", 2024-09-17T16:20:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-17T16:20:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887700+0000", 2024-09-17T16:20:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-17T16:20:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887711+0000", 2024-09-17T16:20:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0739e-05 2024-09-17T16:20:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887716+0000", 2024-09-17T16:20:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2000000000000002e-06 2024-09-17T16:20:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887759+0000", 2024-09-17T16:20:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2898999999999997e-05 2024-09-17T16:20:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887839+0000", 2024-09-17T16:20:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9500999999999997e-05 2024-09-17T16:20:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:20:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887937+0000", 2024-09-17T16:20:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8531999999999998e-05 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.889187+0000", 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012492580000000001 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.889505+0000", 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031885699999999997 2024-09-17T16:20:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354736+0000", 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46523019100000002 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354784+0000", 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8776999999999997e-05 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354821+0000", 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6321e-05 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:105470 2.19 2:9b81cd1a:::benchmark_data_smithi173_55121_object28981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:02.872711+0000", 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 417.24234970600003, 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43773533999999997, 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 105470 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872711+0000", 2024-09-17T16:20:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872711+0000", 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872713+0000", 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3689999999999999e-06 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872717+0000", 2024-09-17T16:20:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9400000000000001e-06 2024-09-17T16:20:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872729+0000", 2024-09-17T16:20:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1994000000000001e-05 2024-09-17T16:20:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872735+0000", 2024-09-17T16:20:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.711e-06 2024-09-17T16:20:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872777+0000", 2024-09-17T16:20:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1894000000000002e-05 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872864+0000", 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7322000000000002e-05 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872942+0000", 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7465000000000005e-05 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:20:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874265+0000", 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0013235009999999999 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874550+0000", 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000285059 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310369+0000", 2024-09-17T16:20:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43581851399999999 2024-09-17T16:20:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310403+0000", 2024-09-17T16:20:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3594999999999999e-05 2024-09-17T16:20:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310447+0000", 2024-09-17T16:20:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3976e-05 2024-09-17T16:20:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:105470 2.19 e170/114 ondisk, result = 0)", 2024-09-17T16:20:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:02.874849+0000", 2024-09-17T16:20:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 417.24021216199998, 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43560358700000001, 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874849+0000", 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874849+0000", 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874850+0000", 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2130000000000001e-06 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874857+0000", 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9970000000000001e-06 2024-09-17T16:20:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310301+0000", 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43544382999999998 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310308+0000", 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9210000000000003e-06 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310360+0000", 2024-09-17T16:20:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2383000000000003e-05 2024-09-17T16:20:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310366+0000", 2024-09-17T16:20:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4410000000000001e-06 2024-09-17T16:20:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310452+0000", 2024-09-17T16:20:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6802e-05 2024-09-17T16:20:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:152973 2.19 2:9e1f786b:::benchmark_data_smithi173_55121_object76484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 366.248023897, 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59801266799999997, 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:20:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 152973 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867039+0000", 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.581e-06 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867044+0000", 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0089999999999998e-06 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867057+0000", 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3658999999999999e-05 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867064+0000", 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2520000000000002e-06 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867120+0000", 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5937999999999999e-05 2024-09-17T16:20:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867194+0000", 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3467000000000007e-05 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867254+0000", 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0510000000000002e-05 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.869276+0000", 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002021882 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.869436+0000", 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000159214 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.464981+0000", 2024-09-17T16:20:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59554505000000002 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.465020+0000", 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9674e-05 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.465050+0000", 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9431999999999999e-05 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25036.0:7257 3.7 e214/208 ondisk, result = 0)", 2024-09-17T16:20:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:14:19.965514+0000", 2024-09-17T16:20:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 340.14954641499997, 2024-09-17T16:20:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29917786299999999, 2024-09-17T16:20:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:20:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:19.965514+0000", 2024-09-17T16:20:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:19.965514+0000", 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:19.965515+0000", 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9900000000000001e-07 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:19.965518+0000", 2024-09-17T16:20:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1e-06 2024-09-17T16:20:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:20.264561+0000", 2024-09-17T16:20:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29904340099999999 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:20.264568+0000", 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5119999999999997e-06 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:20.264610+0000", 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1563999999999997e-05 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:20.264615+0000", 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2100000000000001e-06 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:20.264692+0000", 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7577000000000006e-05 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:8110 3.b 3:de8160e2:::benchmark_data_smithi173_66272_object8109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:14:34.926038+0000", 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 325.18902297800003, 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29379559100000002, 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 8110 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926038+0000", 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926038+0000", 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926039+0000", 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.257e-06 2024-09-17T16:20:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926084+0000", 2024-09-17T16:20:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4758000000000001e-05 2024-09-17T16:20:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926098+0000", 2024-09-17T16:20:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4229e-05 2024-09-17T16:20:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926105+0000", 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2640000000000002e-06 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926162+0000", 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6264000000000002e-05 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926253+0000", 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1812999999999997e-05 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926348+0000", 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4568999999999995e-05 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926897+0000", 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054859500000000003 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.195 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.078517+0000", 2024-09-17T16:20:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.15162066900000001 2024-09-17T16:20:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219736+0000", 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.14121818799999999 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219772+0000", 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6078000000000003e-05 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219834+0000", 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1907000000000002e-05 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:8122 3.2 3:4938ef6e:::benchmark_data_smithi173_66272_object8121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:14:34.933835+0000", 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 325.18122552599999, 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28931940099999998, 2024-09-17T16:20:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 8122 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933835+0000", 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933835+0000", 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933837+0000", 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8819999999999999e-06 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933884+0000", 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6499999999999999e-05 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933895+0000", 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1576000000000001e-05 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933900+0000", 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9289999999999997e-06 2024-09-17T16:20:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.933954+0000", 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3393999999999997e-05 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.934052+0000", 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7985000000000005e-05 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9", 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.934149+0000", 2024-09-17T16:20:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7007000000000001e-05 2024-09-17T16:20:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:20:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.934624+0000", 2024-09-17T16:20:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047536799999999999 2024-09-17T16:20:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.222931+0000", 2024-09-17T16:20:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28830738099999997 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.223078+0000", 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014623899999999999 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.223112+0000", 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4405e-05 2024-09-17T16:20:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.223155+0000", 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2735000000000003e-05 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(3.7 240/239 [PushReplyOp(3:e198b3f2:::benchmark_data_smithi173_66272_object478:head)])", 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 285.10190760299997, 2024-09-17T16:20:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43133256199999997, 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013155+0000", 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013161+0000", 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6830000000000002e-06 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441372+0000", 2024-09-17T16:20:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42821048099999998 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441385+0000", 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.311e-05 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441470+0000", 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5185000000000005e-05 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.444486+0000", 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030157669999999999 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124248 3.7 3:e0068a72:::benchmark_data_smithi173_66272_object52292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 191.22032031099999, 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46227925399999997, 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124248 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894742+0000", 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.392e-06 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894747+0000", 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4979999999999997e-06 2024-09-17T16:20:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894763+0000", 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5095e-05 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894769+0000", 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7360000000000004e-06 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894818+0000", 2024-09-17T16:20:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8551000000000003e-05 2024-09-17T16:20:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894928+0000", 2024-09-17T16:20:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110536 2024-09-17T16:20:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:20:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.895033+0000", 2024-09-17T16:20:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104589 2024-09-17T16:20:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.900743+0000", 2024-09-17T16:20:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057098299999999999 2024-09-17T16:20:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:20:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.901009+0000", 2024-09-17T16:20:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000265927 2024-09-17T16:20:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.927407+0000", 2024-09-17T16:20:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.026398688 2024-09-17T16:20:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:20:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.927419+0000", 2024-09-17T16:20:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1154e-05 2024-09-17T16:20:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:49.356994+0000", 2024-09-17T16:20:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.429575227 2024-09-17T16:20:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:49.357020+0000", 2024-09-17T16:20:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6030999999999999e-05 2024-09-17T16:20:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124248 3.7 3:e0068a72:::benchmark_data_smithi173_66272_object52292:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:20:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:20:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 189.738539602, 2024-09-17T16:20:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54499606199999995, 2024-09-17T16:20:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:20:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:20:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:20:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124248 2024-09-17T16:20:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:20:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:20:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376523+0000", 2024-09-17T16:20:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.42e-06 2024-09-17T16:20:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376528+0000", 2024-09-17T16:20:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7200000000000003e-06 2024-09-17T16:20:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376539+0000", 2024-09-17T16:20:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.022e-05 2024-09-17T16:20:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376546+0000", 2024-09-17T16:20:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8390000000000007e-06 2024-09-17T16:20:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376598+0000", 2024-09-17T16:20:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1953000000000002e-05 2024-09-17T16:20:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:20:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376611+0000", 2024-09-17T16:20:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2320000000000001e-05 2024-09-17T16:20:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.921322+0000", 2024-09-17T16:20:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54471168999999997 2024-09-17T16:20:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.921517+0000", 2024-09-17T16:20:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019489999999999999 2024-09-17T16:20:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:136335 3.5 e284/208 3:aa93c047:::benchmark_data_smithi173_66272_object64379:head v 284'8513, mlcod=284'8510)", 2024-09-17T16:20:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:09.807565+0000", 2024-09-17T16:20:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 170.30749636900001, 2024-09-17T16:20:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37472513000000002, 2024-09-17T16:20:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.807565+0000", 2024-09-17T16:20:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.807565+0000", 2024-09-17T16:20:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.807565+0000", 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8299999999999996e-07 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.807572+0000", 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6220000000000003e-06 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.179272+0000", 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37170049900000002 2024-09-17T16:20:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.179277+0000", 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4730000000000002e-06 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.179325+0000", 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7856999999999999e-05 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.179338+0000", 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3723000000000001e-05 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.182255+0000", 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029164239999999999 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.182290+0000", 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4749000000000002e-05 2024-09-17T16:20:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_scan(digest 3.7 MIN-3:ec3efd19:::benchmark_data_smithi173_66272_object4297:head e 290/273)", 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 159.074903202, 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.427333666, 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040160+0000", 2024-09-17T16:20:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0660000000000002e-06 2024-09-17T16:20:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040256+0000", 2024-09-17T16:20:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5756999999999998e-05 2024-09-17T16:20:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467140+0000", 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42688425000000002 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467149+0000", 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2289999999999992e-06 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467244+0000", 2024-09-17T16:20:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709000000000004e-05 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467260+0000", 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6232e-05 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467491+0000", 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000231423 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDeleteReply(3.c e297,267 3:3ad310f1:::benchmark_data_smithi173_66272_object11599:head,259'5193)", 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:28.259471+0000", 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 151.85558987799999, 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28797248800000003, 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "queued for pg", 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.259471+0000", 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.259471+0000", 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.259472+0000", 2024-09-17T16:20:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.259479+0000", 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0520000000000004e-06 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.547348+0000", 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28786951300000002 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.547361+0000", 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2497000000000001e-05 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.547444+0000", 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2683999999999996e-05 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDeleteReply(3.c e297,267 3:350956dd:::benchmark_data_smithi173_66272_object11629:head,259'5194)", 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 151.848046963, 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.990895003, 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "queued for pg", 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:20:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267015+0000", 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0500000000000003e-07 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267021+0000", 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2550000000000003e-06 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257793+0000", 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.99077237299999998 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257810+0000", 2024-09-17T16:20:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7005999999999999e-05 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257909+0000", 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8664000000000003e-05 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:102161 4.9 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-17T16:20:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:20:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 48.606416340000003, 2024-09-17T16:20:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.902645841, 2024-09-17T16:20:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:20:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:20:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:20:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 102161 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:20:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8700000000000005e-07 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508680+0000", 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4382999999999999e-05 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:20:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508688+0000", 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5059999999999995e-06 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508699+0000", 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0781000000000001e-05 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508757+0000", 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8026999999999999e-05 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508920+0000", 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163308 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.509034+0000", 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113961 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.512221+0000", 2024-09-17T16:20:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003186805 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.515116+0000", 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002895299 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411212+0000", 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89609538399999999 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:20:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411258+0000", 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6217999999999997e-05 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411290+0000", 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2481999999999998e-05 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:20:00.251 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:20:00.252 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:59 smithi017 ceph-mon[27571]: osdmap e363: 12 total, 12 up, 11 in 2024-09-17T16:20:00.252 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:19:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2893615265' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:20:00.276 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:00.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:59 smithi071 ceph-mon[30002]: osdmap e363: 12 total, 12 up, 11 in 2024-09-17T16:20:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:19:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2893615265' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:20:00.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:59 smithi173 ceph-mon[30530]: osdmap e363: 12 total, 12 up, 11 in 2024-09-17T16:20:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:19:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2893615265' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:20:00.402 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:00.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:00.413 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:20:00.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:00.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:00.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:00.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:00.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:00.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:00.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:00.801 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:00.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:00.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:00.928 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:00.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:00.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:01.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:01.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:01.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:01.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:01.190 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:01.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:01.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:00 smithi017 ceph-mon[27571]: pgmap v838: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 809 op/s; 730/184890 objects misplaced (0.395%); 11 MiB/s, 168 objects/s recovering 2024-09-17T16:20:01.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:00 smithi017 ceph-mon[27571]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 1 pg undersized 2024-09-17T16:20:01.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:00 smithi017 ceph-mon[27571]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-17T16:20:01.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:00 smithi017 ceph-mon[27571]: [WRN] PG_DEGRADED: Degraded data redundancy: 1 pg undersized 2024-09-17T16:20:01.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:00 smithi017 ceph-mon[27571]: pg 4.b is stuck undersized for 76s, current state active+recovering+undersized+remapped, last acting [4,0] 2024-09-17T16:20:01.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:00 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:01.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2893615265' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:20:01.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:00 smithi017 ceph-mon[27571]: osdmap e364: 12 total, 12 up, 11 in 2024-09-17T16:20:01.292 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:00 smithi071 ceph-mon[30002]: pgmap v838: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 809 op/s; 730/184890 objects misplaced (0.395%); 11 MiB/s, 168 objects/s recovering 2024-09-17T16:20:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:00 smithi071 ceph-mon[30002]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 1 pg undersized 2024-09-17T16:20:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:00 smithi071 ceph-mon[30002]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-17T16:20:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:00 smithi071 ceph-mon[30002]: [WRN] PG_DEGRADED: Degraded data redundancy: 1 pg undersized 2024-09-17T16:20:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:00 smithi071 ceph-mon[30002]: pg 4.b is stuck undersized for 76s, current state active+recovering+undersized+remapped, last acting [4,0] 2024-09-17T16:20:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:00 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2893615265' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:20:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:00 smithi071 ceph-mon[30002]: osdmap e364: 12 total, 12 up, 11 in 2024-09-17T16:20:01.324 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:01.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:00 smithi173 ceph-mon[30530]: pgmap v838: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 809 op/s; 730/184890 objects misplaced (0.395%); 11 MiB/s, 168 objects/s recovering 2024-09-17T16:20:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:00 smithi173 ceph-mon[30530]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Degraded data redundancy: 1 pg undersized 2024-09-17T16:20:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:00 smithi173 ceph-mon[30530]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-17T16:20:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:00 smithi173 ceph-mon[30530]: [WRN] PG_DEGRADED: Degraded data redundancy: 1 pg undersized 2024-09-17T16:20:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:00 smithi173 ceph-mon[30530]: pg 4.b is stuck undersized for 76s, current state active+recovering+undersized+remapped, last acting [4,0] 2024-09-17T16:20:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:00 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2893615265' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:20:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:00 smithi173 ceph-mon[30530]: osdmap e364: 12 total, 12 up, 11 in 2024-09-17T16:20:01.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:01.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:01.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:01.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:01.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:01.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:01.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:01.719 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:01.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:01.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:01.822 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:01.858 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:01.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:01.958 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:01.985 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:01.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:02.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:02.112 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:02.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:02.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:02.241 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:02.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:02.343 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:02.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:02.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:02.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:02.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:02.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:02.599 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:02.621 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:02.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:02.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:20:02.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:02.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:02.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:02.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:02.885 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:02.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:02.899 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:20:02.900 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:20:02.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:03.014 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:03.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:03.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:02 smithi071 ceph-mon[30002]: pgmap v840: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 730/177786 objects misplaced (0.411%); 9.1 MiB/s, 145 objects/s recovering 2024-09-17T16:20:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2390855410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:03.064 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:03.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:03.132 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:20:03.133 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:20:03.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:03.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:03.158 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:20:03.158 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:20:03.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:02 smithi017 ceph-mon[27571]: pgmap v840: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 730/177786 objects misplaced (0.411%); 9.1 MiB/s, 145 objects/s recovering 2024-09-17T16:20:03.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2390855410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:03.181 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:20:03.181 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:20:03.218 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:20:03.219 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:20:03.245 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:20:03.245 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:20:03.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:03.270 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:03.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:03.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:03.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:02 smithi173 ceph-mon[30530]: pgmap v840: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 730/177786 objects misplaced (0.411%); 9.1 MiB/s, 145 objects/s recovering 2024-09-17T16:20:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2390855410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:03.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:03.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:03.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:03.432 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:03.432 INFO:teuthology.orchestra.run.smithi017.stdout:{"fsid":"19341aec-750e-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":921,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":364,"num_osds":12,"num_up_osds":12,"osd_up_since":1726589558,"num_in_osds":11,"osd_in_since":1726589860,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":59262,"data_bytes":3884188216,"bytes_used":14722736128,"bytes_avail":1041222656000,"bytes_total":1055945392128,"misplaced_objects":730,"misplaced_total":177786,"misplaced_ratio":0.0041060600947206193,"recovering_objects_per_sec":145,"recovering_bytes_per_sec":9526222,"recovering_keys_per_sec":0,"num_objects_recovered":1040,"num_bytes_recovered":68157440,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1027},"fsmap":{"epoch":1,"btime":"2024-09-17T16:02:46:978559+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.17:8443/"}},"servicemap":{"epoch":24,"modified":"2024-09-17T16:19:55.842977+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T16:20:03.472 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:20:03.473 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:20:03.498 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:20:03.498 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:20:03.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:03.524 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:20:03.524 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:20:03.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:03.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:03.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:03.533 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:20:03.548 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:20:03.548 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:20:03.574 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:20:03.574 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:20:03.600 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:20:03.600 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:20:03.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:03.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:03.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:03.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:03.680 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:03.680 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:03.754 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:03.785 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:03.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:03.886 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:03.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:03.920 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:03.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:04.021 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:04.054 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:04.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:04.156 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:04.181 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:04.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:04.219 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/586244184' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:20:04.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/586244184' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:20:04.283 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:04.306 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:04.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:04.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/586244184' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:20:04.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:04.447 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:04.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:04.479 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:20:04.499 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-17T16:20:04.499 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:20:04.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:04.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:04.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:04.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:04.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:04.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:04.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:04.826 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:04.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:04.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:04.916 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:04.928 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:04.953 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:04.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:04.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:05.013 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:05.039 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:05.055 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:05.084 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:05.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:05.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:05.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:05.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:05.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:04 smithi017 ceph-mon[27571]: pgmap v841: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 730/175437 objects misplaced (0.416%); 4.6 MiB/s, 73 objects/s recovering 2024-09-17T16:20:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:04 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:20:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:04 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:20:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:04 smithi071 ceph-mon[30002]: pgmap v841: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 730/175437 objects misplaced (0.416%); 4.6 MiB/s, 73 objects/s recovering 2024-09-17T16:20:05.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:04 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:20:05.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:04 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:20:05.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:05.378 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:05.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:05.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:04 smithi173 ceph-mon[30530]: pgmap v841: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 730/175437 objects misplaced (0.416%); 4.6 MiB/s, 73 objects/s recovering 2024-09-17T16:20:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:04 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:20:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:04 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:20:05.479 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:05.502 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:05.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:05.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:05.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:05.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:05.634 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:05.634 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:05.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:05.759 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:05.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:05.860 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:05.893 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:05.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:05.995 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:06.026 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:06.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:06.126 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:06.153 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:06.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:06.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:06.185 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:06.185 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:20:06.186 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:05 smithi017 ceph-mon[27571]: osdmap e365: 12 total, 12 up, 11 in 2024-09-17T16:20:06.255 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:06.264 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:05 smithi071 ceph-mon[30002]: osdmap e365: 12 total, 12 up, 11 in 2024-09-17T16:20:06.278 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:06.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:06.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:20:06.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:06.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:05 smithi173 ceph-mon[30530]: osdmap e365: 12 total, 12 up, 11 in 2024-09-17T16:20:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:06.426 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:06.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:06.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:06.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:06.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:06.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:06.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:06.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:06.697 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:06.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:06.816 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:06.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:06.918 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:06.948 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:06.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:06.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:20:06.991 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:20:07.050 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:07.054 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":844,"stamp":"2024-09-17T16:20:05.983387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3674800696,"num_objects":56067,"num_object_clones":0,"num_object_copies":168201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":302,"num_objects_unfound":0,"num_objects_dirty":56067,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":148404,"num_write_kb":6541159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9182,"num_bytes_recovered":602945120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148373,"ondisk_log_size":148373,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":13779364,"kb_used_data":11368536,"kb_used_omap":57,"kb_used_meta":2410566,"kb_avail":1017417308,"statfs":{"total":1055945392128,"available":1041835323392,"internally_reserved":0,"allocated":11641380864,"data_stored":11638410172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2468420318},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,1,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-364576768,"num_objects":-5563,"num_object_clones":0,"num_object_copies":-16689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-428,"num_objects_unfound":0,"num_objects_dirty":-5563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5563,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":428,"num_bytes_recovered":28049408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.139733"},"pg_stats":[{"pgid":"4.f","version":"364'9113","reported_seq":9240,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:01.380914+0000","last_change":"2024-09-17T16:17:41.531577+0000","last_active":"2024-09-17T16:20:01.380914+0000","last_peered":"2024-09-17T16:20:01.380914+0000","last_clean":"2024-09-17T16:20:01.380914+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:20:01.380914+0000","last_undegraded":"2024-09-17T16:20:01.380914+0000","last_fullsized":"2024-09-17T16:20:01.380914+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9113,"log_dups_size":0,"ondisk_log_size":9113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.077147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235864064,"num_objects":3599,"num_object_clones":0,"num_object_copies":10797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9113,"num_write_kb":406784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"364'9267","reported_seq":9403,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:04.698282+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:20:04.698282+0000","last_peered":"2024-09-17T16:20:04.698282+0000","last_clean":"2024-09-17T16:20:04.698282+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:20:04.698282+0000","last_undegraded":"2024-09-17T16:20:04.698282+0000","last_fullsized":"2024-09-17T16:20:04.698282+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9267,"log_dups_size":0,"ondisk_log_size":9267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226689024,"num_objects":3459,"num_object_clones":0,"num_object_copies":10377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9267,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"364'9014","reported_seq":9158,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:04.615579+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:20:04.615579+0000","last_peered":"2024-09-17T16:20:04.615579+0000","last_clean":"2024-09-17T16:20:04.615579+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:20:04.615579+0000","last_undegraded":"2024-09-17T16:20:04.615579+0000","last_fullsized":"2024-09-17T16:20:04.615579+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9014,"log_dups_size":0,"ondisk_log_size":9014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224985112,"num_objects":3434,"num_object_clones":0,"num_object_copies":10302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9014,"num_write_kb":398273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"364'9455","reported_seq":9592,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:04.666135+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:20:04.666135+0000","last_peered":"2024-09-17T16:20:04.666135+0000","last_clean":"2024-09-17T16:20:04.666135+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:20:04.666135+0000","last_undegraded":"2024-09-17T16:20:04.666135+0000","last_fullsized":"2024-09-17T16:20:04.666135+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9455,"log_dups_size":0,"ondisk_log_size":9455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233504768,"num_objects":3563,"num_object_clones":0,"num_object_copies":10689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9455,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"364'9381","reported_seq":18523,"reported_epoch":364,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:20:04.711182+0000","last_change":"2024-09-17T16:18:41.641380+0000","last_active":"2024-09-17T16:20:04.711182+0000","last_peered":"2024-09-17T16:20:04.711182+0000","last_clean":"2024-09-17T16:18:39.384812+0000","last_became_active":"2024-09-17T16:18:41.595599+0000","last_became_peered":"2024-09-17T16:18:41.595599+0000","last_unstale":"2024-09-17T16:20:04.711182+0000","last_undegraded":"2024-09-17T16:20:04.711182+0000","last_fullsized":"2024-09-17T16:18:41.426452+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9381,"log_dups_size":0,"ondisk_log_size":9381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223543296,"num_objects":3411,"num_object_clones":0,"num_object_copies":10233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":302,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9381,"num_write_kb":409344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4505,"num_bytes_recovered":295239680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0],"avail_no_missing":["4","0","5"],"object_location_counts":[{"shards":"0,4","objects":3109},{"shards":"0,4,5","objects":302}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"365'9500","reported_seq":9710,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-17T16:20:05.608108+0000","last_change":"2024-09-17T16:19:02.221681+0000","last_active":"2024-09-17T16:20:05.608108+0000","last_peered":"2024-09-17T16:20:05.608108+0000","last_clean":"2024-09-17T16:20:05.608108+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:20:05.608108+0000","last_undegraded":"2024-09-17T16:20:05.608108+0000","last_fullsized":"2024-09-17T16:20:05.608108+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5925","last_scrub_stamp":"2024-09-17T16:19:02.221472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.221472+0000","objects_scrubbed":5879,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.368910+0000","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225050624,"num_objects":3434,"num_object_clones":0,"num_object_copies":10302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"364'9005","reported_seq":9199,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:01.352302+0000","last_change":"2024-09-17T16:19:02.418259+0000","last_active":"2024-09-17T16:20:01.352302+0000","last_peered":"2024-09-17T16:20:01.352302+0000","last_clean":"2024-09-17T16:20:01.352302+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:20:01.352302+0000","last_undegraded":"2024-09-17T16:20:01.352302+0000","last_fullsized":"2024-09-17T16:20:01.352302+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5797","last_scrub_stamp":"2024-09-17T16:19:02.418176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.418176+0000","objects_scrubbed":5756,"log_size":9005,"log_dups_size":0,"ondisk_log_size":9005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:17.281193+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233766912,"num_objects":3567,"num_object_clones":0,"num_object_copies":10701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9005,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"364'9215","reported_seq":9352,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:02.958097+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:20:02.958097+0000","last_peered":"2024-09-17T16:20:02.958097+0000","last_clean":"2024-09-17T16:20:02.958097+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:20:02.958097+0000","last_undegraded":"2024-09-17T16:20:02.958097+0000","last_fullsized":"2024-09-17T16:20:02.958097+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9215,"log_dups_size":0,"ondisk_log_size":9215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:56.862671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232194048,"num_objects":3543,"num_object_clones":0,"num_object_copies":10629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9215,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"365'9317","reported_seq":9464,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-17T16:20:05.777051+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:20:05.777051+0000","last_peered":"2024-09-17T16:20:05.777051+0000","last_clean":"2024-09-17T16:20:05.777051+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:20:05.777051+0000","last_undegraded":"2024-09-17T16:20:05.777051+0000","last_fullsized":"2024-09-17T16:20:05.777051+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":9317,"log_dups_size":0,"ondisk_log_size":9317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:18.005666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225247232,"num_objects":3437,"num_object_clones":0,"num_object_copies":10311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9317,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"364'9013","reported_seq":18492,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:02.287036+0000","last_change":"2024-09-17T16:19:54.707874+0000","last_active":"2024-09-17T16:20:02.287036+0000","last_peered":"2024-09-17T16:20:02.287036+0000","last_clean":"2024-09-17T16:20:02.287036+0000","last_became_active":"2024-09-17T16:19:54.693956+0000","last_became_peered":"2024-09-17T16:19:54.693956+0000","last_unstale":"2024-09-17T16:20:02.287036+0000","last_undegraded":"2024-09-17T16:20:02.287036+0000","last_fullsized":"2024-09-17T16:20:02.287036+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9013,"log_dups_size":0,"ondisk_log_size":9013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229834752,"num_objects":3507,"num_object_clones":0,"num_object_copies":10521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9013,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4670,"num_bytes_recovered":306053120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"364'9183","reported_seq":9311,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:01.649165+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:20:01.649165+0000","last_peered":"2024-09-17T16:20:01.649165+0000","last_clean":"2024-09-17T16:20:01.649165+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:20:01.649165+0000","last_undegraded":"2024-09-17T16:20:01.649165+0000","last_fullsized":"2024-09-17T16:20:01.649165+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9183,"log_dups_size":0,"ondisk_log_size":9183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:53.509019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235995136,"num_objects":3601,"num_object_clones":0,"num_object_copies":10803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9183,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"364'9432","reported_seq":9576,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:02.942533+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:20:02.942533+0000","last_peered":"2024-09-17T16:20:02.942533+0000","last_clean":"2024-09-17T16:20:02.942533+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:20:02.942533+0000","last_undegraded":"2024-09-17T16:20:02.942533+0000","last_fullsized":"2024-09-17T16:20:02.942533+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:58.832940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233963520,"num_objects":3570,"num_object_clones":0,"num_object_copies":10710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9432,"num_write_kb":416064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"365'9233","reported_seq":9382,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-17T16:20:05.459752+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:20:05.459752+0000","last_peered":"2024-09-17T16:20:05.459752+0000","last_clean":"2024-09-17T16:20:05.459752+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:20:05.459752+0000","last_undegraded":"2024-09-17T16:20:05.459752+0000","last_fullsized":"2024-09-17T16:20:05.459752+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9233,"log_dups_size":0,"ondisk_log_size":9233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:45.334658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217513984,"num_objects":3319,"num_object_clones":0,"num_object_copies":9957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9233,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"365'9448","reported_seq":9661,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-17T16:20:05.455106+0000","last_change":"2024-09-17T16:19:38.397911+0000","last_active":"2024-09-17T16:20:05.455106+0000","last_peered":"2024-09-17T16:20:05.455106+0000","last_clean":"2024-09-17T16:20:05.455106+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:20:05.455106+0000","last_undegraded":"2024-09-17T16:20:05.455106+0000","last_fullsized":"2024-09-17T16:20:05.455106+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'7961","last_scrub_stamp":"2024-09-17T16:19:38.397698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:38.397698+0000","objects_scrubbed":4979,"log_size":9448,"log_dups_size":0,"ondisk_log_size":9448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:39.699266+0000","scrub_duration":1058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227147776,"num_objects":3466,"num_object_clones":0,"num_object_copies":10398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9448,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"364'9067","reported_seq":9270,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:01.393432+0000","last_change":"2024-09-17T16:19:40.984613+0000","last_active":"2024-09-17T16:20:01.393432+0000","last_peered":"2024-09-17T16:20:01.393432+0000","last_clean":"2024-09-17T16:20:01.393432+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:20:01.393432+0000","last_undegraded":"2024-09-17T16:20:01.393432+0000","last_fullsized":"2024-09-17T16:20:01.393432+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":2499,"log_size":9067,"log_dups_size":0,"ondisk_log_size":9067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:10.939951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240451584,"num_objects":3669,"num_object_clones":0,"num_object_copies":11007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9067,"num_write_kb":407552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"365'9638","reported_seq":9844,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-09-17T16:20:05.651062+0000","last_change":"2024-09-17T16:19:01.159281+0000","last_active":"2024-09-17T16:20:05.651062+0000","last_peered":"2024-09-17T16:20:05.651062+0000","last_clean":"2024-09-17T16:20:05.651062+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:20:05.651062+0000","last_undegraded":"2024-09-17T16:20:05.651062+0000","last_fullsized":"2024-09-17T16:20:05.651062+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5950","last_scrub_stamp":"2024-09-17T16:19:01.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:01.159158+0000","objects_scrubbed":5911,"log_size":9638,"log_dups_size":0,"ondisk_log_size":9638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:27.590602+0000","scrub_duration":1399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228458496,"num_objects":3486,"num_object_clones":0,"num_object_copies":10458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9638,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":621,"reported_epoch":364,"state":"active+clean","last_fresh":"2024-09-17T16:20:00.418796+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:20:00.418796+0000","last_peered":"2024-09-17T16:20:00.418796+0000","last_clean":"2024-09-17T16:20:00.418796+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:20:00.418796+0000","last_undegraded":"2024-09-17T16:20:00.418796+0000","last_fullsized":"2024-09-17T16:20:00.418796+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:55.890376+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":3674210328,"num_objects":56065,"num_object_clones":0,"num_object_copies":168195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":302,"num_objects_unfound":0,"num_objects_dirty":56065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148281,"num_write_kb":6539009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9176,"num_bytes_recovered":601358336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11595821056,"data_stored":11595808840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148281,"ondisk_log_size":148281,"up":48,"acting":47,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285695,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2066188,"kb_used_data":1774012,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91678964,"statfs":{"total":95995035648,"available":93879259136,"internally_reserved":0,"allocated":1816588288,"data_stored":1816313881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":299170244},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547332,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1243920,"kb_used_data":1108604,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92501232,"statfs":{"total":95995035648,"available":94721261568,"internally_reserved":0,"allocated":1135210496,"data_stored":1134939264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743560,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1437636,"kb_used_data":1095476,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92307516,"statfs":{"total":95995035648,"available":94522896384,"internally_reserved":0,"allocated":1121767424,"data_stored":1121496589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":350353856},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005197,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122116,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93623036,"statfs":{"total":95995035648,"available":95869988864,"internally_reserved":0,"allocated":704512,"data_stored":437749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963674,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1607688,"kb_used_data":1366264,"kb_used_omap":41,"kb_used_meta":241366,"kb_avail":92137464,"statfs":{"total":95995035648,"available":94348763136,"internally_reserved":0,"allocated":1399054336,"data_stored":1398784021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":247159091},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495768,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1811224,"kb_used_data":1541252,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91933928,"statfs":{"total":95995035648,"available":94140342272,"internally_reserved":0,"allocated":1578242048,"data_stored":1577967264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":276429254},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724699,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241488,"kb_used_data":1021500,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92503664,"statfs":{"total":95995035648,"available":94723751936,"internally_reserved":0,"allocated":1046016000,"data_stored":1045744768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":225245638},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986338,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1080196,"kb_used_data":890740,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92664956,"statfs":{"total":95995035648,"available":94888914944,"internally_reserved":0,"allocated":912117760,"data_stored":911846925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1439752,"kb_used_data":1169776,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92305400,"statfs":{"total":95995035648,"available":94520729600,"internally_reserved":0,"allocated":1197850624,"data_stored":1197583861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":276429255},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":601156,"kb_used_data":461680,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93143996,"statfs":{"total":95995035648,"available":95379451904,"internally_reserved":0,"allocated":472760320,"data_stored":472493557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738549,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1128000,"kb_used_data":938544,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92617152,"statfs":{"total":95995035648,"available":94839963648,"internally_reserved":0,"allocated":961069056,"data_stored":960802293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":958005248,"data_stored":958005248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":467599360,"data_stored":467599360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1192755200,"data_stored":1192755200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":908464128,"data_stored":908460056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1044119552,"data_stored":1044119552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1576730624,"data_stored":1576730624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1394409472,"data_stored":1394409472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1120276480,"data_stored":1120272408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1122435072,"data_stored":1122435072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1811025920,"data_stored":1811021848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:20:07.080 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:07.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:07.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:06 smithi071 ceph-mon[30002]: pgmap v843: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 962 op/s; 302/168201 objects misplaced (0.180%); 7.5 MiB/s, 120 objects/s recovering 2024-09-17T16:20:07.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:06 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:20:07.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:06 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:20:07.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:06 smithi071 ceph-mon[30002]: osdmap e366: 12 total, 12 up, 11 in 2024-09-17T16:20:07.142 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/866555199' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:07.142 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:06 smithi071 ceph-mon[30002]: from='client.16245 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:07.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:07.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:07.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:07.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:06 smithi017 ceph-mon[27571]: pgmap v843: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 962 op/s; 302/168201 objects misplaced (0.180%); 7.5 MiB/s, 120 objects/s recovering 2024-09-17T16:20:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:06 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:20:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:06 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:20:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:06 smithi017 ceph-mon[27571]: osdmap e366: 12 total, 12 up, 11 in 2024-09-17T16:20:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/866555199' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:06 smithi017 ceph-mon[27571]: from='client.16245 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:07.286 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:07.286 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:07.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:07.339 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:07.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:07.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:06 smithi173 ceph-mon[30530]: pgmap v843: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 962 op/s; 302/168201 objects misplaced (0.180%); 7.5 MiB/s, 120 objects/s recovering 2024-09-17T16:20:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:06 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:20:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:06 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:20:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:06 smithi173 ceph-mon[30530]: osdmap e366: 12 total, 12 up, 11 in 2024-09-17T16:20:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/866555199' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:06 smithi173 ceph-mon[30530]: from='client.16245 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:07.441 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:07.468 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:07.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:07.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:07.593 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:07.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:07.695 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:07.725 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:07.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:07.753 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:07.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:07.850 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:07.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:07.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:07.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:07.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:08.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:20:08.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:08.118 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:08.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:08.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:08.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:08.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:08.271 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:07 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:20:08.271 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/866555199' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:08.271 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:07 smithi017 ceph-mon[27571]: osdmap e367: 12 total, 12 up, 11 in 2024-09-17T16:20:08.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:07 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:20:08.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/866555199' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:08.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:07 smithi071 ceph-mon[30002]: osdmap e367: 12 total, 12 up, 11 in 2024-09-17T16:20:08.339 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:20:08.339 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:20:08.350 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:08.362 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:20:08.362 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:20:08.373 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:08.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:08.390 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:20:08.390 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:20:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:07 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:20:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/866555199' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:07 smithi173 ceph-mon[30530]: osdmap e367: 12 total, 12 up, 11 in 2024-09-17T16:20:08.415 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:20:08.415 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:20:08.433 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:08.445 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:20:08.445 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:20:08.471 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:20:08.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:20:08.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:08.496 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:20:08.497 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:20:08.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:08.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:08.525 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:20:08.526 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:20:08.550 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:20:08.550 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:20:08.574 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:20:08.574 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:20:08.603 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:20:08.603 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:20:08.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:08.632 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:20:08.632 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:20:08.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:08.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:08.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:08.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:08.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:08.865 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:08.906 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:08.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:09.008 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:09.038 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:09.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:09.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:09.165 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:09.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:09.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:09.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:09.276 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:152823 4.1 4:8140d626:::benchmark_data_smithi173_77474_object50648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:08.996022+0000", 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.039944055999999999, 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.039972911, 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:09.277 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152823 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:08.996022+0000", 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:08.996022+0000", 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:09.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:08.996023+0000", 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.406e-06 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:08.996028+0000", 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2329999999999997e-06 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:09.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:08.996039+0000", 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0667e-05 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:08.996045+0000", 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9649999999999996e-06 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:08.996098+0000", 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3494e-05 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:09.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:08.997690+0000", 2024-09-17T16:20:09.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001591425 2024-09-17T16:20:09.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:20:09.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:08.997808+0000", 2024-09-17T16:20:09.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000118093 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.003445+0000", 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0056373739999999997 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.003625+0000", 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000179772 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.282 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:152849 4.2 4:4480fdb9:::benchmark_data_smithi173_77474_object50674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:09.030504+0000", 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0054621460000000002, 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0055673290000000002, 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:20:09.283 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152849 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.030504+0000", 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.030504+0000", 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:09.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.030506+0000", 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.779e-06 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.030514+0000", 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.824e-06 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:09.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.030534+0000", 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9613999999999999e-05 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.030543+0000", 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1050000000000001e-06 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.030597+0000", 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4413999999999998e-05 2024-09-17T16:20:09.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.030718+0000", 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012092299999999999 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.030831+0000", 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000112522 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:152852 4.1 4:89197431:::benchmark_data_smithi173_77474_object50677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:09.034991+0000", 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.000974328, 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001137739, 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152852 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:09.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.034991+0000", 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.034991+0000", 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.034993+0000", 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.858e-06 2024-09-17T16:20:09.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035000+0000", 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.075e-06 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035021+0000", 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1225000000000001e-05 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035031+0000", 2024-09-17T16:20:09.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.098e-06 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035099+0000", 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8641999999999998e-05 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035228+0000", 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012888999999999999 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035327+0000", 2024-09-17T16:20:09.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9297999999999997e-05 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:152849 4.2 e368/312)", 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:09.035228+0000", 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.00073754199999999999, 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000954455, 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "queued for pg", 2024-09-17T16:20:09.292 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035228+0000", 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035228+0000", 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035229+0000", 2024-09-17T16:20:09.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.33e-07 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035234+0000", 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0159999999999999e-06 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035250+0000", 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6274e-05 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035257+0000", 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2999999999999998e-06 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:152849 4.2 e368/312)", 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:09.035645+0000", 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.000321138, 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057093300000000003, 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:09.295 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "queued for pg", 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035645+0000", 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035645+0000", 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:09.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035645+0000", 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4099999999999997e-07 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035649+0000", 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.371e-06 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035663+0000", 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4498000000000001e-05 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:09.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:09.298 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:09.298 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.035669+0000", 2024-09-17T16:20:09.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7729999999999996e-06 2024-09-17T16:20:09.298 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.298 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:09.298 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.298 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:09.298 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:20:09.298 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 5 2024-09-17T16:20:09.298 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:20:09.298 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:08 smithi071 ceph-mon[30002]: pgmap v846: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 302/164340 objects misplaced (0.184%); 4.5 MiB/s, 71 objects/s recovering 2024-09-17T16:20:09.298 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:08 smithi071 ceph-mon[30002]: 4.f scrub starts 2024-09-17T16:20:09.299 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1344077459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:09.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:09.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:09.300 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:08 smithi173 ceph-mon[30530]: pgmap v846: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 302/164340 objects misplaced (0.184%); 4.5 MiB/s, 71 objects/s recovering 2024-09-17T16:20:09.300 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:08 smithi173 ceph-mon[30530]: 4.f scrub starts 2024-09-17T16:20:09.300 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1344077459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:09.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:09.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:09.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:08 smithi017 ceph-mon[27571]: pgmap v846: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 302/164340 objects misplaced (0.184%); 4.5 MiB/s, 71 objects/s recovering 2024-09-17T16:20:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:08 smithi017 ceph-mon[27571]: 4.f scrub starts 2024-09-17T16:20:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1344077459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:09.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:09.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:09.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:09.648 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:09.669 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:09.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:09.754 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:20:09.771 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:09.820 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:09.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:09.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:09.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:09.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:09.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:10.038 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:20:10.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:10.056 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:20:10.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:10.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:10.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:10.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:10.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:10.244 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:10.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:10.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:10 smithi071 ceph-mon[30002]: osdmap e368: 12 total, 12 up, 11 in 2024-09-17T16:20:10.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:10 smithi071 ceph-mon[30002]: 4.f scrub ok 2024-09-17T16:20:10.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:10.336 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:10.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:10 smithi017 ceph-mon[27571]: osdmap e368: 12 total, 12 up, 11 in 2024-09-17T16:20:10.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:10 smithi017 ceph-mon[27571]: 4.f scrub ok 2024-09-17T16:20:10.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:10 smithi173 ceph-mon[30530]: osdmap e368: 12 total, 12 up, 11 in 2024-09-17T16:20:10.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:10 smithi173 ceph-mon[30530]: 4.f scrub ok 2024-09-17T16:20:10.415 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:10.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:10.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:10.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:10.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:10.594 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:10.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:10.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:10.696 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:10.726 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:10.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:10.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:10.841 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:10.853 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:10.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:10.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:10.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:10.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:11.087 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:11.110 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:11.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:11.212 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:11 smithi017 ceph-mon[27571]: pgmap v848: 17 pgs: 1 peering, 16 active+clean; 3.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 854 op/s; 3.9 MiB/s, 61 objects/s recovering 2024-09-17T16:20:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:11 smithi017 ceph-mon[27571]: osdmap e369: 12 total, 12 up, 11 in 2024-09-17T16:20:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:11 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:20:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:11 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:20:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:11 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T16:20:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:11 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:20:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:11.240 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:11.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:11.266 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:11 smithi071 ceph-mon[30002]: pgmap v848: 17 pgs: 1 peering, 16 active+clean; 3.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 854 op/s; 3.9 MiB/s, 61 objects/s recovering 2024-09-17T16:20:11.266 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:11 smithi071 ceph-mon[30002]: osdmap e369: 12 total, 12 up, 11 in 2024-09-17T16:20:11.266 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:11 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:20:11.267 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:11 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:20:11.267 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:11 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T16:20:11.267 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:11 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:20:11.342 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:11.374 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:11.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:11.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:11 smithi173 ceph-mon[30530]: pgmap v848: 17 pgs: 1 peering, 16 active+clean; 3.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 854 op/s; 3.9 MiB/s, 61 objects/s recovering 2024-09-17T16:20:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:11 smithi173 ceph-mon[30530]: osdmap e369: 12 total, 12 up, 11 in 2024-09-17T16:20:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:11 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:20:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:11 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:20:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:11 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T16:20:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:11 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:20:11.476 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:11.517 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:11.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:11.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:11.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:11.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:11.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:11.661 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:11.661 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:20:11.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:11.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:11.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:11.821 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:20:11.821 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:20:11.821 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:20:11.821 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:20:11.821 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:20:11.883 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:11.911 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:11.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:12.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:12.032 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:20:12.042 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:12.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:12.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:12.144 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:12.167 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:12.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:12.225 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:20:12.252 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:12 smithi017 ceph-mon[27571]: osdmap e370: 12 total, 12 up, 11 in 2024-09-17T16:20:12.252 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/69545889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:12.252 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:12 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:20:12.252 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:12 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:20:12.252 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:12 smithi017 ceph-mon[27571]: 4.0 scrub starts 2024-09-17T16:20:12.269 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:12.295 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:12.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:12.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:12.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:12 smithi071 ceph-mon[30002]: osdmap e370: 12 total, 12 up, 11 in 2024-09-17T16:20:12.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/69545889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:12.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:12 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:20:12.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:12 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:20:12.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:12 smithi071 ceph-mon[30002]: 4.0 scrub starts 2024-09-17T16:20:12.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:12 smithi173 ceph-mon[30530]: osdmap e370: 12 total, 12 up, 11 in 2024-09-17T16:20:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/69545889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:12 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:20:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:12 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:20:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:12 smithi173 ceph-mon[30530]: 4.0 scrub starts 2024-09-17T16:20:12.397 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:12.419 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:12.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:12.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:12.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:12.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:12.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:12.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:12.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:12.769 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:12.791 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:12.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:12.893 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:12.922 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:12.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:12.940 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":850,"stamp":"2024-09-17T16:20:11.017588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3406627384,"num_objects":51975,"num_object_clones":0,"num_object_copies":155925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51975,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":152496,"num_write_kb":6541159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9484,"num_bytes_recovered":622736992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152465,"ondisk_log_size":152465,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":13029744,"kb_used_data":10597820,"kb_used_omap":57,"kb_used_meta":2431430,"kb_avail":1018166928,"statfs":{"total":1055945392128,"available":1042602934272,"internally_reserved":0,"allocated":10852167680,"data_stored":10849039731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2489785054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-268173312,"num_objects":-4092,"num_object_clones":0,"num_object_copies":-12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-302,"num_objects_unfound":0,"num_objects_dirty":-4092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4092,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":19791872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.034200"},"pg_stats":[{"pgid":"4.f","version":"366'9400","reported_seq":9531,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T16:20:06.376591+0000","last_change":"2024-09-17T16:17:41.531577+0000","last_active":"2024-09-17T16:20:06.376591+0000","last_peered":"2024-09-17T16:20:06.376591+0000","last_clean":"2024-09-17T16:20:06.376591+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:20:06.376591+0000","last_undegraded":"2024-09-17T16:20:06.376591+0000","last_fullsized":"2024-09-17T16:20:06.376591+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9400,"log_dups_size":0,"ondisk_log_size":9400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:08.077147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217055232,"num_objects":3312,"num_object_clones":0,"num_object_copies":9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9400,"num_write_kb":406784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"368'9557","reported_seq":9702,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T16:20:09.533362+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:20:09.533362+0000","last_peered":"2024-09-17T16:20:09.533362+0000","last_clean":"2024-09-17T16:20:09.533362+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:20:09.533362+0000","last_undegraded":"2024-09-17T16:20:09.533362+0000","last_fullsized":"2024-09-17T16:20:09.533362+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9557,"log_dups_size":0,"ondisk_log_size":9557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207683584,"num_objects":3169,"num_object_clones":0,"num_object_copies":9507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9557,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"368'9282","reported_seq":9435,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T16:20:09.547884+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:20:09.547884+0000","last_peered":"2024-09-17T16:20:09.547884+0000","last_clean":"2024-09-17T16:20:09.547884+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:20:09.547884+0000","last_undegraded":"2024-09-17T16:20:09.547884+0000","last_fullsized":"2024-09-17T16:20:09.547884+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207421464,"num_objects":3166,"num_object_clones":0,"num_object_copies":9498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9282,"num_write_kb":398273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"368'9706","reported_seq":9851,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-17T16:20:09.519413+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:20:09.519413+0000","last_peered":"2024-09-17T16:20:09.519413+0000","last_clean":"2024-09-17T16:20:09.519413+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:20:09.519413+0000","last_undegraded":"2024-09-17T16:20:09.519413+0000","last_fullsized":"2024-09-17T16:20:09.519413+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9706,"log_dups_size":0,"ondisk_log_size":9706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217055232,"num_objects":3312,"num_object_clones":0,"num_object_copies":9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9706,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"367'9596","reported_seq":19350,"reported_epoch":367,"state":"peering","last_fresh":"2024-09-17T16:20:09.001031+0000","last_change":"2024-09-17T16:20:09.013987+0000","last_active":"2024-09-17T16:20:09.013987+0000","last_peered":"2024-09-17T16:20:09.001031+0000","last_clean":"2024-09-17T16:18:39.384812+0000","last_became_active":"2024-09-17T16:18:41.595599+0000","last_became_peered":"2024-09-17T16:18:41.595599+0000","last_unstale":"2024-09-17T16:20:09.001031+0000","last_undegraded":"2024-09-17T16:20:09.001031+0000","last_fullsized":"2024-09-17T16:18:41.426452+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9596,"log_dups_size":0,"ondisk_log_size":9596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209453056,"num_objects":3196,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9596,"num_write_kb":409344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":315031552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5,9],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"369'9749","reported_seq":9967,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-17T16:20:10.626342+0000","last_change":"2024-09-17T16:19:02.221681+0000","last_active":"2024-09-17T16:20:10.626342+0000","last_peered":"2024-09-17T16:20:10.626342+0000","last_clean":"2024-09-17T16:20:10.626342+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:20:10.626342+0000","last_undegraded":"2024-09-17T16:20:10.626342+0000","last_fullsized":"2024-09-17T16:20:10.626342+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5925","last_scrub_stamp":"2024-09-17T16:19:02.221472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.221472+0000","objects_scrubbed":5879,"log_size":9749,"log_dups_size":0,"ondisk_log_size":9749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208732160,"num_objects":3185,"num_object_clones":0,"num_object_copies":9555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9749,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"366'9259","reported_seq":9456,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T16:20:06.387291+0000","last_change":"2024-09-17T16:19:02.418259+0000","last_active":"2024-09-17T16:20:06.387291+0000","last_peered":"2024-09-17T16:20:06.387291+0000","last_clean":"2024-09-17T16:20:06.387291+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:20:06.387291+0000","last_undegraded":"2024-09-17T16:20:06.387291+0000","last_fullsized":"2024-09-17T16:20:06.387291+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5797","last_scrub_stamp":"2024-09-17T16:19:02.418176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.418176+0000","objects_scrubbed":5756,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:17.281193+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217120768,"num_objects":3313,"num_object_clones":0,"num_object_copies":9939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9259,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"367'9471","reported_seq":9614,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T16:20:07.951998+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:20:07.951998+0000","last_peered":"2024-09-17T16:20:07.951998+0000","last_clean":"2024-09-17T16:20:07.951998+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:20:07.951998+0000","last_undegraded":"2024-09-17T16:20:07.951998+0000","last_fullsized":"2024-09-17T16:20:07.951998+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9471,"log_dups_size":0,"ondisk_log_size":9471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:56.862671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215416832,"num_objects":3287,"num_object_clones":0,"num_object_copies":9861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9471,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"369'9585","reported_seq":9738,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-17T16:20:10.780491+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:20:10.780491+0000","last_peered":"2024-09-17T16:20:10.780491+0000","last_clean":"2024-09-17T16:20:10.780491+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:20:10.780491+0000","last_undegraded":"2024-09-17T16:20:10.780491+0000","last_fullsized":"2024-09-17T16:20:10.780491+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":9585,"log_dups_size":0,"ondisk_log_size":9585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:18.005666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207683584,"num_objects":3169,"num_object_clones":0,"num_object_copies":9507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9585,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"367'9253","reported_seq":18739,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T16:20:07.231206+0000","last_change":"2024-09-17T16:19:54.707874+0000","last_active":"2024-09-17T16:20:07.231206+0000","last_peered":"2024-09-17T16:20:07.231206+0000","last_clean":"2024-09-17T16:20:07.231206+0000","last_became_active":"2024-09-17T16:19:54.693956+0000","last_became_peered":"2024-09-17T16:19:54.693956+0000","last_unstale":"2024-09-17T16:20:07.231206+0000","last_undegraded":"2024-09-17T16:20:07.231206+0000","last_fullsized":"2024-09-17T16:20:07.231206+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9253,"log_dups_size":0,"ondisk_log_size":9253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214106112,"num_objects":3267,"num_object_clones":0,"num_object_copies":9801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9253,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4670,"num_bytes_recovered":306053120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"366'9435","reported_seq":9565,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T16:20:06.746569+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:20:06.746569+0000","last_peered":"2024-09-17T16:20:06.746569+0000","last_clean":"2024-09-17T16:20:06.746569+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:20:06.746569+0000","last_undegraded":"2024-09-17T16:20:06.746569+0000","last_fullsized":"2024-09-17T16:20:06.746569+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9435,"log_dups_size":0,"ondisk_log_size":9435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:53.509019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219480064,"num_objects":3349,"num_object_clones":0,"num_object_copies":10047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9435,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"367'9693","reported_seq":9843,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T16:20:07.974751+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:20:07.974751+0000","last_peered":"2024-09-17T16:20:07.974751+0000","last_clean":"2024-09-17T16:20:07.974751+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:20:07.974751+0000","last_undegraded":"2024-09-17T16:20:07.974751+0000","last_fullsized":"2024-09-17T16:20:07.974751+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9693,"log_dups_size":0,"ondisk_log_size":9693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:58.832940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216858624,"num_objects":3309,"num_object_clones":0,"num_object_copies":9927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9693,"num_write_kb":416064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"369'9470","reported_seq":9626,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-17T16:20:10.508539+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:20:10.508539+0000","last_peered":"2024-09-17T16:20:10.508539+0000","last_clean":"2024-09-17T16:20:10.508539+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:20:10.508539+0000","last_undegraded":"2024-09-17T16:20:10.508539+0000","last_fullsized":"2024-09-17T16:20:10.508539+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9470,"log_dups_size":0,"ondisk_log_size":9470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:45.334658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201981952,"num_objects":3082,"num_object_clones":0,"num_object_copies":9246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9470,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"369'9702","reported_seq":9922,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-17T16:20:10.479122+0000","last_change":"2024-09-17T16:19:38.397911+0000","last_active":"2024-09-17T16:20:10.479122+0000","last_peered":"2024-09-17T16:20:10.479122+0000","last_clean":"2024-09-17T16:20:10.479122+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:20:10.479122+0000","last_undegraded":"2024-09-17T16:20:10.479122+0000","last_fullsized":"2024-09-17T16:20:10.479122+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'7961","last_scrub_stamp":"2024-09-17T16:19:38.397698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:38.397698+0000","objects_scrubbed":4979,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:39.699266+0000","scrub_duration":1058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210501632,"num_objects":3212,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9702,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"366'9321","reported_seq":9528,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-09-17T16:20:06.366961+0000","last_change":"2024-09-17T16:19:40.984613+0000","last_active":"2024-09-17T16:20:06.366961+0000","last_peered":"2024-09-17T16:20:06.366961+0000","last_clean":"2024-09-17T16:20:06.366961+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:20:06.366961+0000","last_undegraded":"2024-09-17T16:20:06.366961+0000","last_fullsized":"2024-09-17T16:20:06.366961+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":2499,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:10.939951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223805440,"num_objects":3415,"num_object_clones":0,"num_object_copies":10245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9321,"num_write_kb":407552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"369'9894","reported_seq":10108,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-17T16:20:10.667301+0000","last_change":"2024-09-17T16:19:01.159281+0000","last_active":"2024-09-17T16:20:10.667301+0000","last_peered":"2024-09-17T16:20:10.667301+0000","last_clean":"2024-09-17T16:20:10.667301+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:20:10.667301+0000","last_undegraded":"2024-09-17T16:20:10.667301+0000","last_fullsized":"2024-09-17T16:20:10.667301+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5950","last_scrub_stamp":"2024-09-17T16:19:01.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:01.159158+0000","objects_scrubbed":5911,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:27.590602+0000","scrub_duration":1399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211681280,"num_objects":3230,"num_object_clones":0,"num_object_copies":9690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9894,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":627,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-09-17T16:20:06.997449+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:20:06.997449+0000","last_peered":"2024-09-17T16:20:06.997449+0000","last_clean":"2024-09-17T16:20:06.997449+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:20:06.997449+0000","last_undegraded":"2024-09-17T16:20:06.997449+0000","last_fullsized":"2024-09-17T16:20:06.997449+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:55.890376+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":3406037016,"num_objects":51973,"num_object_clones":0,"num_object_copies":155919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51973,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152373,"num_write_kb":6539009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9478,"num_bytes_recovered":621150208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10807619584,"data_stored":10807607368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152373,"ondisk_log_size":152373,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285696,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1920900,"kb_used_data":1637036,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91824252,"statfs":{"total":95995035648,"available":94028034048,"internally_reserved":0,"allocated":1676324864,"data_stored":1676032481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":290650564},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547333,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1169268,"kb_used_data":1029792,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92575884,"statfs":{"total":95995035648,"available":94797705216,"internally_reserved":0,"allocated":1054507008,"data_stored":1054222269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743561,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1388660,"kb_used_data":1042328,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92356492,"statfs":{"total":95995035648,"available":94573047808,"internally_reserved":0,"allocated":1067343872,"data_stored":1067059580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005198,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122164,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622988,"statfs":{"total":95995035648,"available":95869939712,"internally_reserved":0,"allocated":753664,"data_stored":468925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963675,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497844,"kb_used_data":1268764,"kb_used_omap":41,"kb_used_meta":228886,"kb_avail":92247308,"statfs":{"total":95995035648,"available":94461243392,"internally_reserved":0,"allocated":1299214336,"data_stored":1298930564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":234379571},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713736,"kb_used_data":1431156,"kb_used_omap":1,"kb_used_meta":282494,"kb_avail":92031416,"statfs":{"total":95995035648,"available":94240169984,"internally_reserved":0,"allocated":1465503744,"data_stored":1465210913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":289274310},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724700,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1202948,"kb_used_data":978796,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":92542204,"statfs":{"total":95995035648,"available":94763216896,"internally_reserved":0,"allocated":1002287104,"data_stored":1001997825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":229505478},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986339,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1016952,"kb_used_data":823332,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92728200,"statfs":{"total":95995035648,"available":94953676800,"internally_reserved":0,"allocated":843091968,"data_stored":842803157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1358176,"kb_used_data":1084040,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92386976,"statfs":{"total":95995035648,"available":94604263424,"internally_reserved":0,"allocated":1110056960,"data_stored":1109781259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411500,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":572696,"kb_used_data":429064,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93172456,"statfs":{"total":95995035648,"available":95408594944,"internally_reserved":0,"allocated":439361536,"data_stored":439085835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738550,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1066400,"kb_used_data":872776,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92678752,"statfs":{"total":95995035648,"available":94903042048,"internally_reserved":0,"allocated":893722624,"data_stored":893446923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":889847808,"data_stored":889847808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":435421184,"data_stored":435421184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1105854464,"data_stored":1105854464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":841617408,"data_stored":841613336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1000275968,"data_stored":1000275968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1462632448,"data_stored":1462632448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1293287424,"data_stored":1293287424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1065881600,"data_stored":1065877528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1039925248,"data_stored":1039925248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1672876032,"data_stored":1672871960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:20:12.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-17T16:20:13.025 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:13.054 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:13.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:13.156 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:13.188 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:13.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:13.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:13 smithi017 ceph-mon[27571]: from='client.16263 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:13.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:13 smithi017 ceph-mon[27571]: pgmap v851: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 3.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 816 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-09-17T16:20:13.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:13 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:13.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/69545889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:20:13.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:13 smithi017 ceph-mon[27571]: osdmap e371: 12 total, 12 up, 11 in 2024-09-17T16:20:13.278 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:13 smithi071 ceph-mon[30002]: from='client.16263 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:13.278 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:13 smithi071 ceph-mon[30002]: pgmap v851: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 3.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 816 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-09-17T16:20:13.279 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:13 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:13.279 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/69545889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:20:13.279 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:13 smithi071 ceph-mon[30002]: osdmap e371: 12 total, 12 up, 11 in 2024-09-17T16:20:13.292 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:13.331 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:20:13.334 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:13.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:13.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:13 smithi173 ceph-mon[30530]: from='client.16263 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:13 smithi173 ceph-mon[30530]: pgmap v851: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 3.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 816 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-09-17T16:20:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:13 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/69545889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:20:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:13 smithi173 ceph-mon[30530]: osdmap e371: 12 total, 12 up, 11 in 2024-09-17T16:20:13.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:13.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:13.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:13.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:13.590 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:13.616 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:13.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:13.620 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:13.709 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:13.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:13.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:13.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:13.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:13.880 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:13.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:13.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:14.015 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:14.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:14.117 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:14.150 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:20:14.150 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:20:14.151 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:14.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:14.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:14.287 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:14.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:14.303 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:20:14.303 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:20:14.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:21741 2.3 2:c4bdb6e7:::benchmark_data_smithi173_55121_object21740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 527.63543076600001, 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55429691999999997, 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.304 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 21741 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662970+0000", 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.662972+0000", 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.722e-06 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663024+0000", 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2175000000000001e-05 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663036+0000", 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2131e-05 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663042+0000", 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9649999999999996e-06 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663084+0000", 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2354999999999997e-05 2024-09-17T16:20:14.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663166+0000", 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1699999999999994e-05 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663249+0000", 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2714999999999997e-05 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.663804+0000", 2024-09-17T16:20:14.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055499599999999998 2024-09-17T16:20:14.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:26.664173+0000", 2024-09-17T16:20:14.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000369181 2024-09-17T16:20:14.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217193+0000", 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55302014600000005 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217234+0000", 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1149000000000002e-05 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:27.217267+0000", 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2685000000000003e-05 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:23782 2.2 2:41257b37:::benchmark_data_smithi173_55121_object23781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 525.61011248700004, 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51828190799999996, 2024-09-17T16:20:14.311 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 23782 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688288+0000", 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688289+0000", 2024-09-17T16:20:14.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7400000000000002e-07 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688376+0000", 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6744999999999997e-05 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688387+0000", 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0499000000000001e-05 2024-09-17T16:20:14.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688393+0000", 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9689999999999997e-06 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688434+0000", 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0694e-05 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688511+0000", 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6656000000000006e-05 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.688594+0000", 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2682999999999994e-05 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689092+0000", 2024-09-17T16:20:14.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049850100000000004 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689369+0000", 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027721100000000003 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206489+0000", 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51712006899999996 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206524+0000", 2024-09-17T16:20:14.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4516e-05 2024-09-17T16:20:14.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206570+0000", 2024-09-17T16:20:14.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6490999999999999e-05 2024-09-17T16:20:14.320 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.320 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.320 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.320 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:23782 2.2 e119/104 ondisk, result = 0)", 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 525.60901970199996, 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51719464199999998, 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689381+0000", 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689382+0000", 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:28.689385+0000", 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3569999999999998e-06 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206448+0000", 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51706313699999995 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206453+0000", 2024-09-17T16:20:14.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4360000000000004e-06 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206480+0000", 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7770999999999999e-05 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206486+0000", 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6779999999999997e-06 2024-09-17T16:20:14.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.206576+0000", 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9701000000000001e-05 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24369 2.3 2:c67d9130:::benchmark_data_smithi173_55121_object24368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:20:14.325 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 524.61089316599998, 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55710953100000005, 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24369 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687508+0000", 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687509+0000", 2024-09-17T16:20:14.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1399999999999995e-07 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687596+0000", 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7288999999999997e-05 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687607+0000", 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1505000000000001e-05 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687613+0000", 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1440000000000002e-06 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687655+0000", 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2240000000000002e-05 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687782+0000", 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000127043 2024-09-17T16:20:14.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.687864+0000", 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2064000000000006e-05 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688444+0000", 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058032600000000004 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688721+0000", 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000276734 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244551+0000", 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55582960199999998 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244588+0000", 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7004e-05 2024-09-17T16:20:14.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244617+0000", 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9666e-05 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:24369 2.3 e120/104 ondisk, result = 0)", 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:20:14.332 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 524.60968597500005, 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55590799999999996, 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688715+0000", 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688716+0000", 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8999999999999996e-07 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.688719+0000", 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5980000000000001e-06 2024-09-17T16:20:14.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244478+0000", 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55575852299999995 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244483+0000", 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244541+0000", 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8377e-05 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244547+0000", 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8980000000000001e-06 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.244623+0000", 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5780999999999998e-05 2024-09-17T16:20:14.336 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.337 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.337 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.337 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.337 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:29877 2.2 2:40729eb6:::benchmark_data_smithi173_55121_object29876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-17T16:20:14.338 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:20:14.338 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:20:14.338 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:20:14.338 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 504.09453158899998, 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.874997425, 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 29877 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203869+0000", 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203870+0000", 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:20:14.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203910+0000", 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9477999999999997e-05 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203918+0000", 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203922+0000", 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6369999999999998e-06 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.203958+0000", 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5045000000000001e-05 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204026+0000", 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8693000000000001e-05 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204105+0000", 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8458e-05 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.204758+0000", 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00065323300000000003 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:50.205073+0000", 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031491100000000002 2024-09-17T16:20:14.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078782+0000", 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87370871400000005 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078827+0000", 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5219999999999997e-05 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:51.078867+0000", 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9953000000000002e-05 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:88652 2.2 2:45ffdfe0:::benchmark_data_smithi173_55121_object12163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 450.34164599500002, 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29260301500000002, 2024-09-17T16:20:14.345 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 88652 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956755+0000", 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956757+0000", 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7859999999999999e-06 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956762+0000", 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.31e-06 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956773+0000", 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0832999999999999e-05 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956778+0000", 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2320000000000001e-06 2024-09-17T16:20:14.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956829+0000", 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1124e-05 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.956915+0000", 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5541e-05 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:14.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.957003+0000", 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8332999999999996e-05 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.958148+0000", 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001144937 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:43.961994+0000", 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038457320000000001 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249267+0000", 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28727286400000002 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249320+0000", 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3016000000000002e-05 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:44.249358+0000", 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8306999999999998e-05 2024-09-17T16:20:14.351 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(2.3 158/155 [PushReplyOp(2:c6916317:::benchmark_data_smithi173_55121_object76243:head)])", 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 445.57535374600002, 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45253040700000002, 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723048+0000", 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4300000000000003e-07 2024-09-17T16:20:14.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723052+0000", 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9310000000000001e-06 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175226+0000", 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45217462899999999 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175233+0000", 2024-09-17T16:20:14.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4099999999999996e-06 2024-09-17T16:20:14.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175322+0000", 2024-09-17T16:20:14.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9011000000000005e-05 2024-09-17T16:20:14.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175578+0000", 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025588299999999999 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:94077 2.16 2:6e055a89:::benchmark_data_smithi173_55121_object17588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 444.21992812299999, 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28469862699999998, 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.356 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 94077 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:20:14.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078473+0000", 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078474+0000", 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.274e-06 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078479+0000", 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6310000000000002e-06 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078490+0000", 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1799e-05 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078496+0000", 2024-09-17T16:20:14.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7579999999999998e-06 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078540+0000", 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3686999999999997e-05 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078624+0000", 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4153000000000006e-05 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.078694+0000", 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9533e-05 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.079805+0000", 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011117550000000001 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.085499+0000", 2024-09-17T16:20:14.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0056938969999999998 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363107+0000", 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27760825700000002 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363140+0000", 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2055999999999999e-05 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.363171+0000", 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1826999999999999e-05 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97425 2.3 2:c4562d4f:::benchmark_data_smithi173_55121_object20936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 440.45132976899998, 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29697053200000001, 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.363 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97425 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847072+0000", 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1399999999999997e-07 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847074+0000", 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7329999999999999e-06 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847082+0000", 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6830000000000008e-06 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847088+0000", 2024-09-17T16:20:14.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0499999999999997e-06 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848028+0000", 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093989799999999999 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848133+0000", 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000104874 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848231+0000", 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8121999999999993e-05 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.849551+0000", 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00131985 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143633+0000", 2024-09-17T16:20:14.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29408256900000002 2024-09-17T16:20:14.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.369 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143940+0000", 2024-09-17T16:20:14.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030690399999999999 2024-09-17T16:20:14.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143994+0000", 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3161999999999997e-05 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.144042+0000", 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8072999999999999e-05 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:129838 2.2 2:4555d77c:::benchmark_data_smithi173_55121_object53349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 406.51186925799999, 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.288916852, 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.371 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 129838 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786532+0000", 2024-09-17T16:20:14.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786533+0000", 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.728e-06 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786540+0000", 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4320000000000004e-06 2024-09-17T16:20:14.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786556+0000", 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6302000000000001e-05 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786563+0000", 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.934e-06 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786626+0000", 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2988000000000001e-05 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.786633+0000", 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6150000000000002e-06 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069502+0000", 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.282869606 2024-09-17T16:20:14.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069572+0000", 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0201000000000001e-05 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069639+0000", 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6323999999999997e-05 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075296+0000", 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0056576639999999997 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075309+0000", 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2621e-05 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075356+0000", 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6653000000000002e-05 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075405+0000", 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9438000000000003e-05 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.075448+0000", 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3346000000000001e-05 2024-09-17T16:20:14.378 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:129850 2.16 2:6f12710d:::benchmark_data_smithi173_55121_object53361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 406.49605444899998, 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27163233199999998, 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.379 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 129850 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802346+0000", 2024-09-17T16:20:14.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802348+0000", 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0549999999999998e-06 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802356+0000", 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6539999999999999e-06 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802372+0000", 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6314999999999999e-05 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802379+0000", 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0419999999999997e-06 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802428+0000", 2024-09-17T16:20:14.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8367e-05 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:27.802433+0000", 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6519999999999997e-06 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069491+0000", 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26705805599999999 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069572+0000", 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1174999999999994e-05 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.069639+0000", 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6946000000000006e-05 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073713+0000", 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040741709999999997 2024-09-17T16:20:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073743+0000", 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9740999999999999e-05 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073843+0000", 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100497 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073930+0000", 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6835999999999993e-05 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:28.073979+0000", 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8825000000000001e-05 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.386 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:150269 2.16 2:6d6b4de7:::benchmark_data_smithi173_55121_object73780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 383.70986717, 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29341146400000001, 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.387 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 150269 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588534+0000", 2024-09-17T16:20:14.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588535+0000", 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.751e-06 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588542+0000", 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7479999999999996e-06 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588558+0000", 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5885000000000001e-05 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588582+0000", 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4042000000000001e-05 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588646+0000", 2024-09-17T16:20:14.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4103999999999995e-05 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588789+0000", 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000143094 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.588910+0000", 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120751 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.590898+0000", 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001988058 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.590948+0000", 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9959000000000001e-05 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881725+0000", 2024-09-17T16:20:14.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29077654800000002 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881900+0000", 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017578599999999999 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:50.881945+0000", 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4737999999999998e-05 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.393 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:152907 2.3 2:c65c8e4e:::benchmark_data_smithi173_55121_object76418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 380.50098110499999, 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87037030500000001, 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:14.394 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152907 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:20:14.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:20:14.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4099999999999999e-07 2024-09-17T16:20:14.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797423+0000", 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797430+0000", 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1119999999999998e-06 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797436+0000", 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6189999999999999e-06 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797479+0000", 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3050999999999998e-05 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797597+0000", 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000117637 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:20:14.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797696+0000", 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9079999999999993e-05 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.799730+0000", 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020345239999999998 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.801020+0000", 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012897270000000001 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667698+0000", 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.866677794 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667733+0000", 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4953999999999998e-05 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667790+0000", 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7265999999999997e-05 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.400 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.401 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:14.402 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:20:14.402 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:20:14.402 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:20:14.403 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:20:14.403 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(3.8 250/239 [PushOp(3:1a7a7bf1:::benchmark_data_smithi173_66272_object39562:head, version: 230'2428, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:1a7a7bf1:::benchmark_data_smithi173_66272_object39562:head@230'2428, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:20:14.403 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:45.409030+0000", 2024-09-17T16:20:14.403 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 268.88937050999999, 2024-09-17T16:20:14.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.263688647, 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409030+0000", 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409030+0000", 2024-09-17T16:20:14.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409032+0000", 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.513e-06 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.409173+0000", 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000140915 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671742+0000", 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26256918299999998 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671788+0000", 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5719000000000001e-05 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671881+0000", 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3509000000000002e-05 2024-09-17T16:20:14.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.671890+0000", 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5290000000000008e-06 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:45.672719+0000", 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00082927900000000004 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:81509 3.5 3:ac456fc5:::benchmark_data_smithi173_66272_object9553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:56.268230+0000", 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 258.03017129800003, 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27300864699999999, 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:20:14.408 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 81509 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268230+0000", 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268230+0000", 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268231+0000", 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3260000000000001e-06 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268236+0000", 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8040000000000001e-06 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268248+0000", 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2014999999999999e-05 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268253+0000", 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4500000000000003e-06 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268296+0000", 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2879000000000001e-05 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268370+0000", 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3892000000000003e-05 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.268433+0000", 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2953999999999994e-05 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270351+0000", 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00191843 2024-09-17T16:20:14.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270909+0000", 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055809299999999996 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541148+0000", 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27023824299999999 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541188+0000", 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0006000000000003e-05 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541238+0000", 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0555e-05 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.414 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25036.0:81509 3.5 e255/208 ondisk, result = 0)", 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:56.270755+0000", 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 258.02764545799999, 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27048895299999998, 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270755+0000", 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270755+0000", 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270756+0000", 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7499999999999998e-07 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.270762+0000", 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9649999999999996e-06 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541053+0000", 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27029036699999998 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541078+0000", 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5367e-05 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541134+0000", 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6329000000000002e-05 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541143+0000", 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3059999999999998e-06 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:56.541244+0000", 2024-09-17T16:20:14.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000101644 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDelete(3.9 e278,273 3:90341f46:::benchmark_data_smithi173_66272_object7671:head,254'4892)", 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 191.146920255, 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30541183, 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.419 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:20:14.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5000000000000001e-07 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151489+0000", 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.52e-06 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456713+0000", 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.305224035 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456737+0000", 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.429e-05 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456817+0000", 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9598000000000006e-05 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456828+0000", 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0740000000000001e-05 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456892+0000", 2024-09-17T16:20:14.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4796999999999996e-05 2024-09-17T16:20:14.424 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.424 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.424 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.424 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:136335 3.5 3:aa93c047:::benchmark_data_smithi173_66272_object64379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-17T16:20:14.424 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:09.807143+0000", 2024-09-17T16:20:14.424 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 184.49125812599999, 2024-09-17T16:20:14.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37544051899999997, 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 136335 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807143+0000", 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807143+0000", 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807144+0000", 2024-09-17T16:20:14.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6899999999999999e-06 2024-09-17T16:20:14.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807151+0000", 2024-09-17T16:20:14.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6880000000000002e-06 2024-09-17T16:20:14.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807166+0000", 2024-09-17T16:20:14.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4883e-05 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807173+0000", 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1690000000000003e-06 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807229+0000", 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5337999999999998e-05 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807354+0000", 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000125007 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807468+0000", 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011404900000000001 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.811073+0000", 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036052879999999999 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.811379+0000", 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030657599999999997 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:10.182478+0000", 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37109837699999998 2024-09-17T16:20:14.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:10.182538+0000", 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9840000000000003e-05 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:10.182583+0000", 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5614000000000001e-05 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.431 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:14455 4.1 4:8aec1aca:::benchmark_data_smithi173_77474_object14454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:49.531449+0000", 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 144.76695201499999, 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26431682000000001, 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:14.432 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14455 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531449+0000", 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531449+0000", 2024-09-17T16:20:14.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531451+0000", 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6500000000000001e-06 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531501+0000", 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0448999999999998e-05 2024-09-17T16:20:14.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531516+0000", 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4687e-05 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531523+0000", 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2690000000000002e-06 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531592+0000", 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9431999999999995e-05 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531718+0000", 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000125897 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.531913+0000", 2024-09-17T16:20:14.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019508500000000001 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.533462+0000", 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001549148 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.669390+0000", 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13592713300000001 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.795690+0000", 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.126300828 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.795733+0000", 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2750000000000002e-05 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:49.795766+0000", 2024-09-17T16:20:14.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2492e-05 2024-09-17T16:20:14.439 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.439 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.439 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.439 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:14.439 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:14.439 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:20:14.439 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:20:14.440 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:20:14.440 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:20:14.440 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:20:14.441 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:20:14.441 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:20:14.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:20:14.454 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:20:14.467 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:14.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:14.484 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:20:14.484 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:20:14.509 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:20:14.509 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:20:14.534 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:20:14.534 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:20:14.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:20:14.560 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:20:14.569 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:14.583 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:20:14.584 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:20:14.595 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:14.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:14.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:14.608 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:20:14.608 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:20:14.634 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:20:14.634 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:20:14.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:14.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:14.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:14.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:14.852 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:14.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:14.953 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:14.977 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:14.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:15.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:15.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:15.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:15.206 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:15.234 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:15.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:15 smithi071 ceph-mon[30002]: pgmap v853: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 3.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 795 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-09-17T16:20:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:15 smithi071 ceph-mon[30002]: osdmap e372: 12 total, 12 up, 11 in 2024-09-17T16:20:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3418495671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:15.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:15.362 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:15.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:15 smithi173 ceph-mon[30530]: pgmap v853: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 3.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 795 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-09-17T16:20:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:15 smithi173 ceph-mon[30530]: osdmap e372: 12 total, 12 up, 11 in 2024-09-17T16:20:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3418495671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:15.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:15.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:15.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:15.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:15 smithi017 ceph-mon[27571]: pgmap v853: 17 pgs: 1 active+clean+scrubbing, 1 peering, 15 active+clean; 3.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 795 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-09-17T16:20:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:15 smithi017 ceph-mon[27571]: osdmap e372: 12 total, 12 up, 11 in 2024-09-17T16:20:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3418495671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:15.517 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:20:15.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:15.616 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:15.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:15.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:15.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:15.757 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:15.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:15.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:15.868 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:15.898 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:15.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:16.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:16.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:16.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:16.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:16.157 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:16.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:16.164 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:16.164 INFO:teuthology.orchestra.run.smithi017.stdout:{"fsid":"19341aec-750e-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":934,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":372,"num_osds":12,"num_up_osds":12,"osd_up_since":1726589558,"num_in_osds":11,"osd_in_since":1726589860,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+clean+scrubbing","count":1}],"num_pgs":17,"num_pools":2,"num_objects":48103,"data_bytes":3152871992,"bytes_used":12572504064,"bytes_avail":1043372888064,"bytes_total":1055945392128,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":879},"fsmap":{"epoch":1,"btime":"2024-09-17T16:02:46:978559+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.17:8443/"}},"servicemap":{"epoch":24,"modified":"2024-09-17T16:19:55.842977+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T16:20:16.259 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:16.287 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:16.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:16.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:16.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:16.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:16.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:16.543 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:16.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:16.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:16.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:16.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:16.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:16.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:16.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:16.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:16.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:16.931 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:16.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:17.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:17.041 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:20:17.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:17.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:17.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:17.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:17.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:17.291 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:17.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:17 smithi017 ceph-mon[27571]: pgmap v855: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-09-17T16:20:17.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1083796385' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:20:17.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3144028128' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:17.311 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T16:20:17.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:20:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:17 smithi071 ceph-mon[30002]: pgmap v855: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-09-17T16:20:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1083796385' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:20:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3144028128' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:17.315 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:17.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:17.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:17 smithi173 ceph-mon[30530]: pgmap v855: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-09-17T16:20:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1083796385' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-17T16:20:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3144028128' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:17.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:17.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:17.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:17.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:17.554 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:20:17.572 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:17.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:17.674 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:17.700 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:17.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:17.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:17.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:17.833 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:17.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:17.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:17.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:17.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:18.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:18.093 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:18.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:18.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:18.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:18.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:18 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3144028128' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:18 smithi071 ceph-mon[30002]: osdmap e373: 12 total, 12 up, 11 in 2024-09-17T16:20:18.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:18.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:18.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:18 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3144028128' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:18 smithi173 ceph-mon[30530]: osdmap e373: 12 total, 12 up, 11 in 2024-09-17T16:20:18.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:18.477 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:18.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:18.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:18 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3144028128' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:18 smithi017 ceph-mon[27571]: osdmap e373: 12 total, 12 up, 11 in 2024-09-17T16:20:18.579 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:18.601 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:18.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:18.625 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:18.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:18.731 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:18.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:18.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:18.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:18.865 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:18.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:18.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:18.996 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:18.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:19.098 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:19.133 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:19.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:19.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:19 smithi017 ceph-mon[27571]: pgmap v857: 17 pgs: 17 active+clean; 2.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 705 op/s 2024-09-17T16:20:19.235 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:19.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:20:19.261 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:20:19.262 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:20:19.263 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:19.272 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:19.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:19.300 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:20:19.301 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:20:19.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:19 smithi071 ceph-mon[30002]: pgmap v857: 17 pgs: 17 active+clean; 2.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 705 op/s 2024-09-17T16:20:19.329 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:20:19.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:20:19.359 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:20:19.359 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:20:19.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:19.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:20:19.389 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:20:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:19 smithi173 ceph-mon[30530]: pgmap v857: 17 pgs: 17 active+clean; 2.9 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 705 op/s 2024-09-17T16:20:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:19.411 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:19.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:19.418 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:20:19.418 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:20:19.447 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:20:19.448 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:20:19.486 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:20:19.486 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:20:19.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:19.513 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:20:19.513 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:20:19.539 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:20:19.539 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:20:19.549 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:19.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:19.568 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:20:19.568 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:20:19.598 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:20:19.598 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:20:19.626 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:19.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:19.687 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:19.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:19.789 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:19.826 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:19.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:19.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:19.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:19.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:20.056 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:20.092 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:20.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:20.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:20.214 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:161542 4.1 e373/301 4:817bb2ab:::benchmark_data_smithi173_77474_object59367:head v 373'10181, mlcod=373'10178)", 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:20.210335+0000", 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.001494645, 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001512434, 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:20.215 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:20.210335+0000", 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:20.210335+0000", 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:20.210336+0000", 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5999999999999991e-07 2024-09-17T16:20:20.216 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:20.210342+0000", 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9340000000000003e-06 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:20.210372+0000", 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0525999999999999e-05 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:20.210378+0000", 2024-09-17T16:20:20.217 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1510000000000001e-06 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:20.210444+0000", 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5821999999999994e-05 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:20.210459+0000", 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5207e-05 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:20.218 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:20.219 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:20.219 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:20.219 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:20:20.219 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 1 2024-09-17T16:20:20.219 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:20:20.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:20.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:20.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:20.234 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3922745146' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:20.234 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:20:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3922745146' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:20:20.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:20.349 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:20.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3922745146' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:20:20.417 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:20.418 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:20:20.451 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:20.478 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:20.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:20.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:20.603 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:20:20.604 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:20.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:20.705 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:20.734 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:20.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:20.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:20.869 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:20.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:20.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:20:20.970 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:21.023 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:21.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:21.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:21.125 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:21.166 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:21.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:21.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:21.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:21.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:21.319 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:21.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:21 smithi173 ceph-mon[30530]: pgmap v858: 17 pgs: 17 active+clean; 2.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 748 op/s 2024-09-17T16:20:21.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:21.438 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:21.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:21.443 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:21 smithi071 ceph-mon[30002]: pgmap v858: 17 pgs: 17 active+clean; 2.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 748 op/s 2024-09-17T16:20:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:21 smithi017 ceph-mon[27571]: pgmap v858: 17 pgs: 17 active+clean; 2.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 748 op/s 2024-09-17T16:20:21.539 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:21.564 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:21.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:21.581 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":858,"stamp":"2024-09-17T16:20:19.847457+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2963210808,"num_objects":45209,"num_object_clones":0,"num_object_copies":135627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45209,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":159262,"num_write_kb":6541159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9484,"num_bytes_recovered":622736992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158731,"ondisk_log_size":158731,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":11722592,"kb_used_data":9294660,"kb_used_omap":57,"kb_used_meta":2427206,"kb_avail":1019474080,"statfs":{"total":1055945392128,"available":1043941457920,"internally_reserved":0,"allocated":9517731840,"data_stored":9514650165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2485459678},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-383254528,"num_objects":-5848,"num_object_clones":0,"num_object_copies":-17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5848,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.816159"},"pg_stats":[{"pgid":"4.f","version":"372'9922","reported_seq":10104,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-17T16:20:16.380898+0000","last_change":"2024-09-17T16:20:09.115717+0000","last_active":"2024-09-17T16:20:16.380898+0000","last_peered":"2024-09-17T16:20:16.380898+0000","last_clean":"2024-09-17T16:20:16.380898+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:20:16.380898+0000","last_undegraded":"2024-09-17T16:20:16.380898+0000","last_fullsized":"2024-09-17T16:20:16.380898+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'9560","last_scrub_stamp":"2024-09-17T16:20:09.115593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:20:09.115593+0000","objects_scrubbed":3173,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:09.532067+0000","scrub_duration":944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182845440,"num_objects":2790,"num_object_clones":0,"num_object_copies":8370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9922,"num_write_kb":406784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"373'10070","reported_seq":10225,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:19.649608+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:20:19.649608+0000","last_peered":"2024-09-17T16:20:19.649608+0000","last_clean":"2024-09-17T16:20:19.649608+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:20:19.649608+0000","last_undegraded":"2024-09-17T16:20:19.649608+0000","last_fullsized":"2024-09-17T16:20:19.649608+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174063616,"num_objects":2656,"num_object_clones":0,"num_object_copies":7968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10070,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"373'9782","reported_seq":9944,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:19.692821+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:20:19.692821+0000","last_peered":"2024-09-17T16:20:19.692821+0000","last_clean":"2024-09-17T16:20:19.692821+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:20:19.692821+0000","last_undegraded":"2024-09-17T16:20:19.692821+0000","last_fullsized":"2024-09-17T16:20:19.692821+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9782,"log_dups_size":0,"ondisk_log_size":9782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174653464,"num_objects":2666,"num_object_clones":0,"num_object_copies":7998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9782,"num_write_kb":398273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"373'10219","reported_seq":10374,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:19.679248+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:20:19.679248+0000","last_peered":"2024-09-17T16:20:19.679248+0000","last_clean":"2024-09-17T16:20:19.679248+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:20:19.679248+0000","last_undegraded":"2024-09-17T16:20:19.679248+0000","last_fullsized":"2024-09-17T16:20:19.679248+0000","mapping_epoch":301,"log_start":"308'200","ondisk_log_start":"308'200","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183435264,"num_objects":2799,"num_object_clones":0,"num_object_copies":8397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10219,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"373'10131","reported_seq":19903,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:19.705898+0000","last_change":"2024-09-17T16:20:10.038730+0000","last_active":"2024-09-17T16:20:19.705898+0000","last_peered":"2024-09-17T16:20:19.705898+0000","last_clean":"2024-09-17T16:20:19.705898+0000","last_became_active":"2024-09-17T16:20:10.016978+0000","last_became_peered":"2024-09-17T16:20:10.016978+0000","last_unstale":"2024-09-17T16:20:19.705898+0000","last_undegraded":"2024-09-17T16:20:19.705898+0000","last_fullsized":"2024-09-17T16:20:19.705898+0000","mapping_epoch":368,"log_start":"307'100","ondisk_log_start":"307'100","created":301,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174391296,"num_objects":2661,"num_object_clones":0,"num_object_copies":7983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10131,"num_write_kb":409344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":315031552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"372'10018","reported_seq":10242,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-17T16:20:15.655808+0000","last_change":"2024-09-17T16:19:02.221681+0000","last_active":"2024-09-17T16:20:15.655808+0000","last_peered":"2024-09-17T16:20:15.655808+0000","last_clean":"2024-09-17T16:20:15.655808+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:20:15.655808+0000","last_undegraded":"2024-09-17T16:20:15.655808+0000","last_fullsized":"2024-09-17T16:20:15.655808+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5925","last_scrub_stamp":"2024-09-17T16:19:02.221472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.221472+0000","objects_scrubbed":5879,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191102976,"num_objects":2916,"num_object_clones":0,"num_object_copies":8748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10018,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"372'9730","reported_seq":9940,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-17T16:20:16.386481+0000","last_change":"2024-09-17T16:19:02.418259+0000","last_active":"2024-09-17T16:20:16.386481+0000","last_peered":"2024-09-17T16:20:16.386481+0000","last_clean":"2024-09-17T16:20:16.386481+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:20:16.386481+0000","last_undegraded":"2024-09-17T16:20:16.386481+0000","last_fullsized":"2024-09-17T16:20:16.386481+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5797","last_scrub_stamp":"2024-09-17T16:19:02.418176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.418176+0000","objects_scrubbed":5756,"log_size":9730,"log_dups_size":0,"ondisk_log_size":9730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:17.281193+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186253312,"num_objects":2842,"num_object_clones":0,"num_object_copies":8526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9730,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"373'9976","reported_seq":10131,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:17.934932+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:20:17.934932+0000","last_peered":"2024-09-17T16:20:17.934932+0000","last_clean":"2024-09-17T16:20:17.934932+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:20:17.934932+0000","last_undegraded":"2024-09-17T16:20:17.934932+0000","last_fullsized":"2024-09-17T16:20:17.934932+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9976,"log_dups_size":0,"ondisk_log_size":9976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:56.862671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182321152,"num_objects":2782,"num_object_clones":0,"num_object_copies":8346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"372'9852","reported_seq":10013,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-17T16:20:15.797584+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:20:15.797584+0000","last_peered":"2024-09-17T16:20:15.797584+0000","last_clean":"2024-09-17T16:20:15.797584+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:20:15.797584+0000","last_undegraded":"2024-09-17T16:20:15.797584+0000","last_fullsized":"2024-09-17T16:20:15.797584+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":9852,"log_dups_size":0,"ondisk_log_size":9852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:18.005666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190185472,"num_objects":2902,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9852,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"373'9732","reported_seq":19230,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:17.262211+0000","last_change":"2024-09-17T16:19:54.707874+0000","last_active":"2024-09-17T16:20:17.262211+0000","last_peered":"2024-09-17T16:20:17.262211+0000","last_clean":"2024-09-17T16:20:17.262211+0000","last_became_active":"2024-09-17T16:19:54.693956+0000","last_became_peered":"2024-09-17T16:19:54.693956+0000","last_unstale":"2024-09-17T16:20:17.262211+0000","last_undegraded":"2024-09-17T16:20:17.262211+0000","last_fullsized":"2024-09-17T16:20:17.262211+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9732,"log_dups_size":0,"ondisk_log_size":9732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:12.160156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182714368,"num_objects":2788,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9732,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4670,"num_bytes_recovered":306053120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"372'9899","reported_seq":10043,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-17T16:20:16.647906+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:20:16.647906+0000","last_peered":"2024-09-17T16:20:16.647906+0000","last_clean":"2024-09-17T16:20:16.647906+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:20:16.647906+0000","last_undegraded":"2024-09-17T16:20:16.647906+0000","last_fullsized":"2024-09-17T16:20:16.647906+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9899,"log_dups_size":0,"ondisk_log_size":9899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:53.509019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189071360,"num_objects":2885,"num_object_clones":0,"num_object_copies":8655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9899,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"373'10179","reported_seq":10341,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:17.893610+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:20:17.893610+0000","last_peered":"2024-09-17T16:20:17.893610+0000","last_clean":"2024-09-17T16:20:17.893610+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:20:17.893610+0000","last_undegraded":"2024-09-17T16:20:17.893610+0000","last_fullsized":"2024-09-17T16:20:17.893610+0000","mapping_epoch":306,"log_start":"307'100","ondisk_log_start":"307'100","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:58.832940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":8469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10179,"num_write_kb":416064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"372'9697","reported_seq":9859,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-17T16:20:15.468255+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:20:15.468255+0000","last_peered":"2024-09-17T16:20:15.468255+0000","last_clean":"2024-09-17T16:20:15.468255+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:20:15.468255+0000","last_undegraded":"2024-09-17T16:20:15.468255+0000","last_fullsized":"2024-09-17T16:20:15.468255+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":9697,"log_dups_size":0,"ondisk_log_size":9697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:45.334658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187105280,"num_objects":2855,"num_object_clones":0,"num_object_copies":8565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9697,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"372'9954","reported_seq":10181,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-17T16:20:15.436575+0000","last_change":"2024-09-17T16:19:38.397911+0000","last_active":"2024-09-17T16:20:15.436575+0000","last_peered":"2024-09-17T16:20:15.436575+0000","last_clean":"2024-09-17T16:20:15.436575+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:20:15.436575+0000","last_undegraded":"2024-09-17T16:20:15.436575+0000","last_fullsized":"2024-09-17T16:20:15.436575+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'7961","last_scrub_stamp":"2024-09-17T16:19:38.397698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:38.397698+0000","objects_scrubbed":4979,"log_size":9954,"log_dups_size":0,"ondisk_log_size":9954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:39.699266+0000","scrub_duration":1058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193986560,"num_objects":2960,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9954,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"372'9821","reported_seq":10065,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-17T16:20:16.358241+0000","last_change":"2024-09-17T16:20:12.071751+0000","last_active":"2024-09-17T16:20:16.358241+0000","last_peered":"2024-09-17T16:20:16.358241+0000","last_clean":"2024-09-17T16:20:16.358241+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:20:16.358241+0000","last_undegraded":"2024-09-17T16:20:16.358241+0000","last_fullsized":"2024-09-17T16:20:16.358241+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1999,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:42.066678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191037440,"num_objects":2915,"num_object_clones":0,"num_object_copies":8745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9821,"num_write_kb":407552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"372'10157","reported_seq":10377,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-17T16:20:15.692372+0000","last_change":"2024-09-17T16:19:01.159281+0000","last_active":"2024-09-17T16:20:15.692372+0000","last_peered":"2024-09-17T16:20:15.692372+0000","last_clean":"2024-09-17T16:20:15.692372+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:20:15.692372+0000","last_undegraded":"2024-09-17T16:20:15.692372+0000","last_fullsized":"2024-09-17T16:20:15.692372+0000","mapping_epoch":301,"log_start":"307'100","ondisk_log_start":"307'100","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5950","last_scrub_stamp":"2024-09-17T16:19:01.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:01.159158+0000","objects_scrubbed":5911,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:27.590602+0000","scrub_duration":1399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194445312,"num_objects":2967,"num_object_clones":0,"num_object_copies":8901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10157,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":639,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:17.045300+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:20:17.045300+0000","last_peered":"2024-09-17T16:20:17.045300+0000","last_clean":"2024-09-17T16:20:17.045300+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:20:17.045300+0000","last_undegraded":"2024-09-17T16:20:17.045300+0000","last_fullsized":"2024-09-17T16:20:17.045300+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:55.890376+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2962620440,"num_objects":45207,"num_object_clones":0,"num_object_copies":135621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45207,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159139,"num_write_kb":6539009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9478,"num_bytes_recovered":621150208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9451876352,"data_stored":9451864136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158639,"ondisk_log_size":158639,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285698,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1671452,"kb_used_data":1387596,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92073700,"statfs":{"total":95995035648,"available":94283468800,"internally_reserved":0,"allocated":1420898304,"data_stored":1420642231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":290650564},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547334,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1089888,"kb_used_data":946256,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92655264,"statfs":{"total":95995035648,"available":94878990336,"internally_reserved":0,"allocated":968966144,"data_stored":968663371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743563,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1230388,"kb_used_data":888068,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92514764,"statfs":{"total":95995035648,"available":94735118336,"internally_reserved":0,"allocated":909381632,"data_stored":909129131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":350353856},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005200,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122224,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622928,"statfs":{"total":95995035648,"available":95869878272,"internally_reserved":0,"allocated":655360,"data_stored":406931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963677,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1318392,"kb_used_data":1076808,"kb_used_omap":41,"kb_used_meta":241366,"kb_avail":92426760,"statfs":{"total":95995035648,"available":94645002240,"internally_reserved":0,"allocated":1102651392,"data_stored":1102399411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":247159091},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1603244,"kb_used_data":1337432,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92141908,"statfs":{"total":95995035648,"available":94353313792,"internally_reserved":0,"allocated":1369530368,"data_stored":1369224043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":272169414},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724701,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1135460,"kb_used_data":927952,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92609692,"statfs":{"total":95995035648,"available":94832324608,"internally_reserved":0,"allocated":950222848,"data_stored":949920075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":212466118},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986341,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896724,"kb_used_data":694788,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92848428,"statfs":{"total":95995035648,"available":95076790272,"internally_reserved":0,"allocated":711462912,"data_stored":711210411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1194084,"kb_used_data":924112,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92551068,"statfs":{"total":95995035648,"available":94772293632,"internally_reserved":0,"allocated":946290688,"data_stored":945987915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":276429255},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411502,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515680,"kb_used_data":367888,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93229472,"statfs":{"total":95995035648,"available":95466979328,"internally_reserved":0,"allocated":376717312,"data_stored":376414539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738552,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":945056,"kb_used_data":743120,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92800096,"statfs":{"total":95995035648,"available":95027298304,"internally_reserved":0,"allocated":760954880,"data_stored":760652107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":759431168,"data_stored":759431168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":373358592,"data_stored":373358592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":944242688,"data_stored":944242688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":706416640,"data_stored":706412568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":949223424,"data_stored":949223424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1344536576,"data_stored":1344536576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1098645504,"data_stored":1098645504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":908005376,"data_stored":908001304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":954400768,"data_stored":954400768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1413615616,"data_stored":1413611544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:20:21.647 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:21.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:21.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:21.694 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:21.795 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:21.828 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:21.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:21.909 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:20:21.909 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:20:21.909 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:20:21.909 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:20:21.909 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:20:21.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:21.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:21.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:21.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:22.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:22.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:22.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:22.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:22.183 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:22.211 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:22.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:22.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:22.338 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:22.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:22.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:22.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:22.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:22 smithi017 ceph-mon[27571]: from='client.25918 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:20:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:20:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:22 smithi071 ceph-mon[30002]: from='client.25918 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:20:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:20:22.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:22.571 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:22.598 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:22.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:22 smithi173 ceph-mon[30530]: from='client.25918 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:20:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:20:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:22.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:20:22.700 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:22.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:20:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:22.746 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:22.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:22.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:22.873 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:22.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:22.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:22.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:23.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:23.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:23.023 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:23.040 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:23.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:23.127 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:23.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:23.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:23.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:23.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:23.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:23.359 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:23.386 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:20:23.387 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:23.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:23.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:23 smithi017 ceph-mon[27571]: pgmap v859: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 941 op/s 2024-09-17T16:20:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2466133471' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:20:23.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:23.493 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:23 smithi071 ceph-mon[30002]: pgmap v859: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 941 op/s 2024-09-17T16:20:23.493 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2466133471' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:20:23.516 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:23.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:23.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:23 smithi173 ceph-mon[30530]: pgmap v859: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 941 op/s 2024-09-17T16:20:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2466133471' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:20:23.649 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:23.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:23.660 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:20:23.661 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:20:23.688 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:20:23.689 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:20:23.715 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:20:23.715 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:20:23.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:20:23.744 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:20:23.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:23.773 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:20:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:20:23.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:23.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:23.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:23.799 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:13493 2.17 2:ef6c8979:::benchmark_data_smithi173_55121_object13492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:12.334922+0000", 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 551.46142237599997, 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0242685929999999, 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.800 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 13493 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334922+0000", 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334922+0000", 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334923+0000", 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.192e-06 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.334999+0000", 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5327999999999994e-05 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.335013+0000", 2024-09-17T16:20:23.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.42e-05 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.335019+0000", 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0920000000000003e-06 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.350269+0000", 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.01524939 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:12.350275+0000", 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7399999999999998e-06 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:13.359107+0000", 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0088320850000001 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:13.359191+0000", 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3566000000000006e-05 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.804 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14929 2.11 2.db77d491 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.268492+0000", 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 549.52785275899998, 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.99314310400000005, 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:23.805 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14929 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268492+0000", 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268492+0000", 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268493+0000", 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1900000000000001e-07 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268552+0000", 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8807999999999997e-05 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268562+0000", 2024-09-17T16:20:23.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0180000000000001e-05 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268567+0000", 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0549999999999999e-06 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268612+0000", 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5719000000000001e-05 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:20:23.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.268618+0000", 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5650000000000001e-06 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261624+0000", 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.993005519 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261635+0000", 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1439e-05 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.809 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14934 2.11 2.10e332f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.326376+0000", 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 549.46996908400001, 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93529751100000003, 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14934 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.810 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326376+0000", 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326376+0000", 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326378+0000", 2024-09-17T16:20:23.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1849999999999998e-06 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326430+0000", 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2686999999999999e-05 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326448+0000", 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7518999999999999e-05 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326455+0000", 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2230000000000001e-06 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326508+0000", 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2645000000000002e-05 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.326517+0000", 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7679999999999997e-06 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.814 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:20:23.815 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:20:23.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261665+0000", 2024-09-17T16:20:23.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93514807200000005 2024-09-17T16:20:23.815 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.815 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.815 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261673+0000", 2024-09-17T16:20:23.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4120000000000001e-06 2024-09-17T16:20:23.815 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14937 2.11 2.66ac0671 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.333944+0000", 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 549.46240047399999, 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.92790043600000005, 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:23.816 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14937 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.333944+0000", 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.333944+0000", 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.333945+0000", 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6499999999999998e-07 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334020+0000", 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5496999999999999e-05 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334029+0000", 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5909999999999996e-06 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334033+0000", 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2799999999999997e-06 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334071+0000", 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7719999999999998e-05 2024-09-17T16:20:23.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.334076+0000", 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6659999999999999e-06 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261837+0000", 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.92776128899999999 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261845+0000", 2024-09-17T16:20:23.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6279999999999996e-06 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14955 2.11 2.20b690d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.701197+0000", 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 549.09514809100006, 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56068107599999994, 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:20:23.821 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14955 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701197+0000", 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701197+0000", 2024-09-17T16:20:23.822 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.823 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.823 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.823 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.823 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701206+0000", 2024-09-17T16:20:23.823 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6399999999999992e-06 2024-09-17T16:20:23.823 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.823 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.823 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.823 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701258+0000", 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2159e-05 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701289+0000", 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0482999999999999e-05 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701303+0000", 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4576000000000001e-05 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701450+0000", 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014646099999999999 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.701466+0000", 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5913000000000002e-05 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261871+0000", 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56040511900000001 2024-09-17T16:20:23.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261878+0000", 2024-09-17T16:20:23.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.725e-06 2024-09-17T16:20:23.826 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.826 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.826 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.827 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:14965 2.11 2.79d92c71 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:20:23.827 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:14.872030+0000", 2024-09-17T16:20:23.827 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 548.92431447000001, 2024-09-17T16:20:23.827 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.389896574, 2024-09-17T16:20:23.827 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.827 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.827 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:20:23.827 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.827 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 14965 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872030+0000", 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872030+0000", 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872031+0000", 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6899999999999996e-07 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872113+0000", 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1584999999999999e-05 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.829 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872126+0000", 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3181e-05 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872132+0000", 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5390000000000004e-06 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872179+0000", 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7058000000000002e-05 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.830 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:14.872185+0000", 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7699999999999998e-06 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261919+0000", 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38973334999999998 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:15.261927+0000", 2024-09-17T16:20:23.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2220000000000002e-06 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:24378 2.17 2:eca9db3e:::benchmark_data_smithi173_55121_object24377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.695814+0000", 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 534.10053028000004, 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39380668000000002, 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:23.832 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 24378 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695814+0000", 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.833 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695814+0000", 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695816+0000", 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.15e-06 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695864+0000", 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8797000000000001e-05 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695875+0000", 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0795000000000001e-05 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695881+0000", 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7470000000000002e-06 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.695926+0000", 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4669000000000002e-05 2024-09-17T16:20:23.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696011+0000", 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5141000000000004e-05 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696097+0000", 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5805999999999998e-05 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:23.836 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696594+0000", 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049760799999999997 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.697152+0000", 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055830099999999998 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089548+0000", 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39239545599999998 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.837 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089589+0000", 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1539999999999999e-05 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089621+0000", 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.167e-05 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.838 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:24378 2.17 e120/104 ondisk, result = 0)", 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:29.696958+0000", 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 534.09938632299998, 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39266769299999998, 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696958+0000", 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.839 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696958+0000", 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696959+0000", 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.99e-07 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:29.696962+0000", 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5310000000000002e-06 2024-09-17T16:20:23.840 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089489+0000", 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39252651 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089496+0000", 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.165e-06 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.841 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089537+0000", 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0401000000000002e-05 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089543+0000", 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5130000000000002e-06 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:30.089626+0000", 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2973999999999995e-05 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.842 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:27006 2.7 2:e4bc7e40:::benchmark_data_smithi173_55121_object27005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:33.164406+0000", 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 530.63193823999995, 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31356054900000002, 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 27006 2024-09-17T16:20:23.843 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164406+0000", 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164406+0000", 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.844 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164407+0000", 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164434+0000", 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6551999999999999e-05 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164445+0000", 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1683e-05 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.845 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164468+0000", 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2708e-05 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164511+0000", 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2923000000000002e-05 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164588+0000", 2024-09-17T16:20:23.846 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6625000000000006e-05 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.164669+0000", 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1527999999999996e-05 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165334+0000", 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00066428899999999998 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.847 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165780+0000", 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000446799 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477887+0000", 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31210646199999997 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477933+0000", 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6013999999999997e-05 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.848 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477967+0000", 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4167e-05 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:27006 2.7 e121/104 ondisk, result = 0)", 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 530.63058614299996, 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31221570399999998, 2024-09-17T16:20:23.849 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.850 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.850 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:23.850 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.850 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.850 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.850 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:20:23.850 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.850 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.850 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.850 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165759+0000", 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2099999999999996e-07 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.165764+0000", 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3159999999999998e-06 2024-09-17T16:20:23.851 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477726+0000", 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31196209800000002 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477766+0000", 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0238999999999999e-05 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.852 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477875+0000", 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010957499999999999 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477883+0000", 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2860000000000002e-06 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:11:33.477974+0000", 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1469000000000001e-05 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.853 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:94007 2.19 e160/114 2:9e42b13d:::benchmark_data_smithi173_55121_object17518:head v 160'3326, mlcod=160'3323)", 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:49.888089+0000", 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 453.90825601, 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.46654707200000001, 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.854 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.888089+0000", 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.888089+0000", 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.888089+0000", 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0699999999999997e-07 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.855 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.888094+0000", 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.938e-06 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.353318+0000", 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.46522339899999998 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.353323+0000", 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1800000000000004e-06 2024-09-17T16:20:23.856 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.353396+0000", 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3294000000000007e-05 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.353411+0000", 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4942e-05 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:23.857 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.354595+0000", 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011835630000000001 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:50.354636+0000", 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1149000000000002e-05 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97421 2.14 2:2bd549c5:::benchmark_data_smithi173_55121_object20932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.844120+0000", 2024-09-17T16:20:23.858 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 449.95222489600002, 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.341556152, 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97421 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844120+0000", 2024-09-17T16:20:23.859 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.860 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.860 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.860 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.860 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:20:23.861 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:20:23.861 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844120+0000", 2024-09-17T16:20:23.861 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844122+0000", 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9439999999999999e-06 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844128+0000", 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2029999999999997e-06 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.862 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844145+0000", 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7370000000000001e-05 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844153+0000", 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7409999999999992e-06 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844213+0000", 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0003000000000001e-05 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.863 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844307+0000", 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4271000000000006e-05 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 5,9", 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.844381+0000", 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3847e-05 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.845862+0000", 2024-09-17T16:20:23.864 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014811150000000001 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.853013+0000", 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0071510469999999998 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185577+0000", 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33256385799999999 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:23.865 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185630+0000", 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2664000000000003e-05 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185676+0000", 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6088999999999999e-05 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:97421 2.14 e163/117 ondisk, result = 0)", 2024-09-17T16:20:23.866 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.846233+0000", 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 449.95011175000002, 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33944879500000003, 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.846233+0000", 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.867 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.846233+0000", 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.846233+0000", 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6599999999999996e-07 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.846238+0000", 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3970000000000004e-06 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.868 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185505+0000", 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33926690199999998 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185511+0000", 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1489999999999999e-06 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185564+0000", 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3371000000000001e-05 2024-09-17T16:20:23.869 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185571+0000", 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1640000000000004e-06 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.185682+0000", 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000110246 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.870 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25036.0:8122 3.2 e220/208 3:4938ef6e:::benchmark_data_smithi173_66272_object8121:head v 220'496, mlcod=220'493)", 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:14:34.934266+0000", 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 348.86207909799998, 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28873317999999998, 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:34.934266+0000", 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.871 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:34.934266+0000", 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:34.934267+0000", 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9900000000000009e-07 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.872 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:34.934362+0000", 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5050000000000006e-05 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222243+0000", 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28788133799999999 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222250+0000", 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0310000000000001e-06 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.873 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222296+0000", 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6381999999999997e-05 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222317+0000", 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0401e-05 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222944+0000", 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00062735000000000004 2024-09-17T16:20:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222999+0000", 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4629000000000002e-05 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:76068 4.5 4:a01bffc8:::benchmark_data_smithi173_77474_object76067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:18:40.015930+0000", 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 103.780414631, 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39555464400000001, 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 76068 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015930+0000", 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015930+0000", 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015931+0000", 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.44e-06 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015982+0000", 2024-09-17T16:20:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0076e-05 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015997+0000", 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5531000000000001e-05 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016004+0000", 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6930000000000001e-06 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016063+0000", 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9621999999999999e-05 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016194+0000", 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013074899999999999 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016340+0000", 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000145601 2024-09-17T16:20:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.018468+0000", 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021282110000000001 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.018797+0000", 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032921399999999998 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.411385+0000", 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39258796099999999 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.411442+0000", 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6997000000000003e-05 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.411485+0000", 2024-09-17T16:20:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2549000000000002e-05 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:102166 4.a 4:583138b7:::benchmark_data_smithi173_77474_object102165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:11.515734+0000", 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 72.280610736, 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31708639999999999, 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:20:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 102166 2024-09-17T16:20:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515734+0000", 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515734+0000", 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515736+0000", 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.903e-06 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515777+0000", 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1304000000000003e-05 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515791+0000", 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3648e-05 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515798+0000", 2024-09-17T16:20:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4139999999999997e-06 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515850+0000", 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2157000000000002e-05 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515951+0000", 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100321 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:20:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.516061+0000", 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000110636 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.518923+0000", 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028621470000000002 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519104+0000", 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018084500000000001 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832702+0000", 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31359804299999999 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832773+0000", 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0612999999999996e-05 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832820+0000", 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7369e-05 2024-09-17T16:20:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:102166 4.a e349/306 ondisk, result = 0)", 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:11.519407+0000", 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 72.276937199000002, 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31341929800000001, 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519407+0000", 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519407+0000", 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519408+0000", 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5300000000000003e-07 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519417+0000", 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7660000000000004e-06 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832605+0000", 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31318752500000002 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832629+0000", 2024-09-17T16:20:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4039000000000001e-05 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832686+0000", 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7722000000000001e-05 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832696+0000", 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3610000000000007e-06 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832827+0000", 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000131132 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:144445 4.5 4:a72f5a89:::benchmark_data_smithi173_77474_object42270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:59.250222+0000", 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 24.546122319999998, 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32863163499999998, 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 144445 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250222+0000", 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250222+0000", 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250224+0000", 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2240000000000001e-06 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250229+0000", 2024-09-17T16:20:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4720000000000003e-06 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250242+0000", 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2508e-05 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250251+0000", 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0049999999999993e-06 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250311+0000", 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0441e-05 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250442+0000", 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013124299999999999 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250564+0000", 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012152200000000001 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254897+0000", 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0043328849999999999 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.259207+0000", 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0043099599999999998 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578783+0000", 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31957687099999998 2024-09-17T16:20:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578813+0000", 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9745000000000001e-05 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578854+0000", 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0759000000000001e-05 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:144445 4.5 e363/301 ondisk, result = 0)", 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:59.254966+0000", 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 24.541378675000001, 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32389465200000001, 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254966+0000", 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254966+0000", 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254967+0000", 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6899999999999996e-07 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254972+0000", 2024-09-17T16:20:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9259999999999999e-06 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578690+0000", 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32371775800000002 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578694+0000", 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3189999999999996e-06 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578774+0000", 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0020999999999998e-05 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578779+0000", 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0669999999999999e-06 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578861+0000", 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1692000000000003e-05 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:144445 4.5 4:a72f5a89:::benchmark_data_smithi173_77474_object42270:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e363)", 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:59.291351+0000", 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 24.504993639999999, 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28747054399999999, 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for ondisk", 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 144445 2024-09-17T16:20:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.291351+0000", 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.291351+0000", 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:20:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.291352+0000", 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6099999999999999e-07 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.291357+0000", 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7190000000000001e-06 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.291365+0000", 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0399999999999993e-06 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:20:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.291372+0000", 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2300000000000002e-06 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.291426+0000", 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3554999999999998e-05 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for ondisk", 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.291468+0000", 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2781999999999998e-05 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:20:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:20:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578822+0000", 2024-09-17T16:20:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28735325699999997 2024-09-17T16:20:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:20:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:20:23.908 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:20:23.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:23.909 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:20:23.910 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:20:23.910 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:20:23.910 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:20:23.924 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:20:23.925 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:20:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:23.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:23.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:23.952 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:20:23.952 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:20:23.979 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:20:23.980 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:20:24.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:24.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:24.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:24.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:24.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:24.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:24.295 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:24.324 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:24.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:24.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:24.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:24.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:24 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2466133471' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:20:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:24 smithi017 ceph-mon[27571]: osdmap e374: 12 total, 12 up, 11 in 2024-09-17T16:20:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/398447459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:24.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:24.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:24 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:24.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2466133471' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:20:24.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:24 smithi071 ceph-mon[30002]: osdmap e374: 12 total, 12 up, 11 in 2024-09-17T16:20:24.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/398447459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:24.579 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:24.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:24.584 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:20:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:24 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2466133471' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:20:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:24 smithi173 ceph-mon[30530]: osdmap e374: 12 total, 12 up, 11 in 2024-09-17T16:20:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/398447459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:24.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:24.712 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:24.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:24.813 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:24.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:24.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:24.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:24.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:24.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:25.071 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:25.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:20:25.103 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:25.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:25.205 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:25.233 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:25.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:25.332 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:25.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:25.356 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:25.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:25.429 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:25.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:25 smithi017 ceph-mon[27571]: pgmap v861: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 752 op/s 2024-09-17T16:20:25.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:25.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:25.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:25.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:25.611 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:25.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:25.701 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:25 smithi173 ceph-mon[30530]: pgmap v861: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 752 op/s 2024-09-17T16:20:25.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:25.739 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:25.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:25.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:25 smithi071 ceph-mon[30002]: pgmap v861: 17 pgs: 17 active+clean; 2.6 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 752 op/s 2024-09-17T16:20:25.841 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:25.865 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:25.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:25.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:25.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:25.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:26.004 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:26.004 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:26.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:26.120 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:26.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:26.152 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:26.152 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:20:26.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:26.247 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:26.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:26.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:26.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:26.371 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:26.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:26.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:26.498 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:26.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:26.600 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:26.623 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:26.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:26.725 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:26.750 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:26.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:26.793 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:20:26.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:20:26.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:26.877 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:26.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:26.930 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":862,"stamp":"2024-09-17T16:20:25.848606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2596864568,"num_objects":39619,"num_object_clones":0,"num_object_copies":118857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39619,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":164852,"num_write_kb":6541159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9484,"num_bytes_recovered":622736992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160721,"ondisk_log_size":160721,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":10,"kb":1031196672,"kb_used":10644076,"kb_used_data":8219308,"kb_used_omap":57,"kb_used_meta":2424454,"kb_avail":1020552596,"statfs":{"total":1055945392128,"available":1045045858304,"internally_reserved":0,"allocated":8416571392,"data_stored":8413788493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2482641630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,6,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-487915520,"num_objects":-7445,"num_object_clones":0,"num_object_copies":-22335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7445,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.808563"},"pg_stats":[{"pgid":"4.f","version":"373'10207","reported_seq":10393,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:21.389327+0000","last_change":"2024-09-17T16:20:09.115717+0000","last_active":"2024-09-17T16:20:21.389327+0000","last_peered":"2024-09-17T16:20:21.389327+0000","last_clean":"2024-09-17T16:20:21.389327+0000","last_became_active":"2024-09-17T16:17:41.530543+0000","last_became_peered":"2024-09-17T16:17:41.530543+0000","last_unstale":"2024-09-17T16:20:21.389327+0000","last_undegraded":"2024-09-17T16:20:21.389327+0000","last_fullsized":"2024-09-17T16:20:21.389327+0000","mapping_epoch":306,"log_start":"308'200","ondisk_log_start":"308'200","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"368'9560","last_scrub_stamp":"2024-09-17T16:20:09.115593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:20:09.115593+0000","objects_scrubbed":3173,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:09.532067+0000","scrub_duration":944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164167680,"num_objects":2505,"num_object_clones":0,"num_object_copies":7515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10207,"num_write_kb":406784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"374'10334","reported_seq":10491,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-17T16:20:24.717131+0000","last_change":"2024-09-17T16:17:34.566590+0000","last_active":"2024-09-17T16:20:24.717131+0000","last_peered":"2024-09-17T16:20:24.717131+0000","last_clean":"2024-09-17T16:20:24.717131+0000","last_became_active":"2024-09-17T16:17:34.566272+0000","last_became_peered":"2024-09-17T16:17:34.566272+0000","last_unstale":"2024-09-17T16:20:24.717131+0000","last_undegraded":"2024-09-17T16:20:24.717131+0000","last_fullsized":"2024-09-17T16:20:24.717131+0000","mapping_epoch":301,"log_start":"308'300","ondisk_log_start":"308'300","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156762112,"num_objects":2392,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10334,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,6],"acting":[4,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"374'10058","reported_seq":10222,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-17T16:20:24.691608+0000","last_change":"2024-09-17T16:17:41.531372+0000","last_active":"2024-09-17T16:20:24.691608+0000","last_peered":"2024-09-17T16:20:24.691608+0000","last_clean":"2024-09-17T16:20:24.691608+0000","last_became_active":"2024-09-17T16:17:41.530988+0000","last_became_peered":"2024-09-17T16:17:41.530988+0000","last_unstale":"2024-09-17T16:20:24.691608+0000","last_undegraded":"2024-09-17T16:20:24.691608+0000","last_fullsized":"2024-09-17T16:20:24.691608+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156565528,"num_objects":2390,"num_object_clones":0,"num_object_copies":7170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2390,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10058,"num_write_kb":398273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,11],"acting":[4,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"374'10491","reported_seq":10648,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-17T16:20:24.648972+0000","last_change":"2024-09-17T16:17:34.448690+0000","last_active":"2024-09-17T16:20:24.648972+0000","last_peered":"2024-09-17T16:20:24.648972+0000","last_clean":"2024-09-17T16:20:24.648972+0000","last_became_active":"2024-09-17T16:17:34.448354+0000","last_became_peered":"2024-09-17T16:17:34.448354+0000","last_unstale":"2024-09-17T16:20:24.648972+0000","last_undegraded":"2024-09-17T16:20:24.648972+0000","last_fullsized":"2024-09-17T16:20:24.648972+0000","mapping_epoch":301,"log_start":"309'400","ondisk_log_start":"309'400","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:21.409666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165609472,"num_objects":2527,"num_object_clones":0,"num_object_copies":7581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10491,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"374'10431","reported_seq":20205,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-17T16:20:24.656859+0000","last_change":"2024-09-17T16:20:10.038730+0000","last_active":"2024-09-17T16:20:24.656859+0000","last_peered":"2024-09-17T16:20:24.656859+0000","last_clean":"2024-09-17T16:20:24.656859+0000","last_became_active":"2024-09-17T16:20:10.016978+0000","last_became_peered":"2024-09-17T16:20:10.016978+0000","last_unstale":"2024-09-17T16:20:24.656859+0000","last_undegraded":"2024-09-17T16:20:24.656859+0000","last_fullsized":"2024-09-17T16:20:24.656859+0000","mapping_epoch":368,"log_start":"309'400","ondisk_log_start":"309'400","created":301,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154730496,"num_objects":2361,"num_object_clones":0,"num_object_copies":7083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10431,"num_write_kb":409344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4807,"num_bytes_recovered":315031552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,9],"acting":[4,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"374'10501","reported_seq":10729,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-17T16:20:25.681952+0000","last_change":"2024-09-17T16:19:02.221681+0000","last_active":"2024-09-17T16:20:25.681952+0000","last_peered":"2024-09-17T16:20:25.681952+0000","last_clean":"2024-09-17T16:20:25.681952+0000","last_became_active":"2024-09-17T16:17:41.378881+0000","last_became_peered":"2024-09-17T16:17:41.378881+0000","last_unstale":"2024-09-17T16:20:25.681952+0000","last_undegraded":"2024-09-17T16:20:25.681952+0000","last_fullsized":"2024-09-17T16:20:25.681952+0000","mapping_epoch":306,"log_start":"309'500","ondisk_log_start":"309'500","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5925","last_scrub_stamp":"2024-09-17T16:19:02.221472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.221472+0000","objects_scrubbed":5879,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:20.715753+0000","scrub_duration":1440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159449088,"num_objects":2433,"num_object_clones":0,"num_object_copies":7299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10501,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,7],"acting":[6,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"373'10005","reported_seq":10216,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:21.420519+0000","last_change":"2024-09-17T16:19:02.418259+0000","last_active":"2024-09-17T16:20:21.420519+0000","last_peered":"2024-09-17T16:20:21.420519+0000","last_clean":"2024-09-17T16:20:21.420519+0000","last_became_active":"2024-09-17T16:17:41.530630+0000","last_became_peered":"2024-09-17T16:17:41.530630+0000","last_unstale":"2024-09-17T16:20:21.420519+0000","last_undegraded":"2024-09-17T16:20:21.420519+0000","last_fullsized":"2024-09-17T16:20:21.420519+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5797","last_scrub_stamp":"2024-09-17T16:19:02.418176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:02.418176+0000","objects_scrubbed":5756,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:20.490144+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168230912,"num_objects":2567,"num_object_clones":0,"num_object_copies":7701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10005,"num_write_kb":402304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"373'10224","reported_seq":10378,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:22.981230+0000","last_change":"2024-09-17T16:17:34.582854+0000","last_active":"2024-09-17T16:20:22.981230+0000","last_peered":"2024-09-17T16:20:22.981230+0000","last_clean":"2024-09-17T16:20:22.981230+0000","last_became_active":"2024-09-17T16:17:34.582157+0000","last_became_peered":"2024-09-17T16:17:34.582157+0000","last_unstale":"2024-09-17T16:20:22.981230+0000","last_undegraded":"2024-09-17T16:20:22.981230+0000","last_fullsized":"2024-09-17T16:20:22.981230+0000","mapping_epoch":301,"log_start":"308'200","ondisk_log_start":"308'200","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:56.862671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166068224,"num_objects":2534,"num_object_clones":0,"num_object_copies":7602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10224,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"374'10346","reported_seq":10510,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-17T16:20:25.800613+0000","last_change":"2024-09-17T16:18:39.011781+0000","last_active":"2024-09-17T16:20:25.800613+0000","last_peered":"2024-09-17T16:20:25.800613+0000","last_clean":"2024-09-17T16:20:25.800613+0000","last_became_active":"2024-09-17T16:17:41.496860+0000","last_became_peered":"2024-09-17T16:17:41.496860+0000","last_unstale":"2024-09-17T16:20:25.800613+0000","last_undegraded":"2024-09-17T16:20:25.800613+0000","last_fullsized":"2024-09-17T16:20:25.800613+0000","mapping_epoch":306,"log_start":"308'300","ondisk_log_start":"308'300","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1399,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:18.011627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157810688,"num_objects":2408,"num_object_clones":0,"num_object_copies":7224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"373'10013","reported_seq":19511,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:22.266068+0000","last_change":"2024-09-17T16:19:54.707874+0000","last_active":"2024-09-17T16:20:22.266068+0000","last_peered":"2024-09-17T16:20:22.266068+0000","last_clean":"2024-09-17T16:20:22.266068+0000","last_became_active":"2024-09-17T16:19:54.693956+0000","last_became_peered":"2024-09-17T16:19:54.693956+0000","last_unstale":"2024-09-17T16:20:22.266068+0000","last_undegraded":"2024-09-17T16:20:22.266068+0000","last_fullsized":"2024-09-17T16:20:22.266068+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:12.160156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164298752,"num_objects":2507,"num_object_clones":0,"num_object_copies":7521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4670,"num_bytes_recovered":306053120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"373'10154","reported_seq":10300,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:21.715387+0000","last_change":"2024-09-17T16:17:41.378668+0000","last_active":"2024-09-17T16:20:21.715387+0000","last_peered":"2024-09-17T16:20:21.715387+0000","last_clean":"2024-09-17T16:20:21.715387+0000","last_became_active":"2024-09-17T16:17:41.378348+0000","last_became_peered":"2024-09-17T16:17:41.378348+0000","last_unstale":"2024-09-17T16:20:21.715387+0000","last_undegraded":"2024-09-17T16:20:21.715387+0000","last_fullsized":"2024-09-17T16:20:21.715387+0000","mapping_epoch":306,"log_start":"307'100","ondisk_log_start":"307'100","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:53.509019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172359680,"num_objects":2630,"num_object_clones":0,"num_object_copies":7890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10154,"num_write_kb":409088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,10],"acting":[2,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"373'10441","reported_seq":10603,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:22.960779+0000","last_change":"2024-09-17T16:17:41.531655+0000","last_active":"2024-09-17T16:20:22.960779+0000","last_peered":"2024-09-17T16:20:22.960779+0000","last_clean":"2024-09-17T16:20:22.960779+0000","last_became_active":"2024-09-17T16:17:41.531250+0000","last_became_peered":"2024-09-17T16:17:41.531250+0000","last_unstale":"2024-09-17T16:20:22.960779+0000","last_undegraded":"2024-09-17T16:20:22.960779+0000","last_fullsized":"2024-09-17T16:20:22.960779+0000","mapping_epoch":306,"log_start":"309'400","ondisk_log_start":"309'400","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:58.832940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167837696,"num_objects":2561,"num_object_clones":0,"num_object_copies":7683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":416064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"374'10216","reported_seq":10383,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-17T16:20:25.378671+0000","last_change":"2024-09-17T16:17:55.437108+0000","last_active":"2024-09-17T16:20:25.378671+0000","last_peered":"2024-09-17T16:20:25.378671+0000","last_clean":"2024-09-17T16:20:25.378671+0000","last_became_active":"2024-09-17T16:17:55.416879+0000","last_became_peered":"2024-09-17T16:17:55.416879+0000","last_unstale":"2024-09-17T16:20:25.378671+0000","last_undegraded":"2024-09-17T16:20:25.378671+0000","last_fullsized":"2024-09-17T16:20:25.378671+0000","mapping_epoch":312,"log_start":"308'200","ondisk_log_start":"308'200","created":301,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:45.334658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153092096,"num_objects":2336,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10216,"num_write_kb":401664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,11],"acting":[5,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"374'10520","reported_seq":10751,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-17T16:20:25.424855+0000","last_change":"2024-09-17T16:19:38.397911+0000","last_active":"2024-09-17T16:20:25.424855+0000","last_peered":"2024-09-17T16:20:25.424855+0000","last_clean":"2024-09-17T16:20:25.424855+0000","last_became_active":"2024-09-17T16:17:34.582235+0000","last_became_peered":"2024-09-17T16:17:34.582235+0000","last_unstale":"2024-09-17T16:20:25.424855+0000","last_undegraded":"2024-09-17T16:20:25.424855+0000","last_fullsized":"2024-09-17T16:20:25.424855+0000","mapping_epoch":301,"log_start":"309'500","ondisk_log_start":"309'500","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'7961","last_scrub_stamp":"2024-09-17T16:19:38.397698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:38.397698+0000","objects_scrubbed":4979,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:39.699266+0000","scrub_duration":1058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156893184,"num_objects":2394,"num_object_clones":0,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10520,"num_write_kb":413248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6],"acting":[5,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"373'10100","reported_seq":10346,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:21.410092+0000","last_change":"2024-09-17T16:20:12.071751+0000","last_active":"2024-09-17T16:20:21.410092+0000","last_peered":"2024-09-17T16:20:21.410092+0000","last_clean":"2024-09-17T16:20:21.410092+0000","last_became_active":"2024-09-17T16:17:34.581698+0000","last_became_peered":"2024-09-17T16:17:34.581698+0000","last_unstale":"2024-09-17T16:20:21.410092+0000","last_undegraded":"2024-09-17T16:20:21.410092+0000","last_fullsized":"2024-09-17T16:20:21.410092+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:17:33.436404+0000","objects_scrubbed":1999,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:42.066678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172752896,"num_objects":2636,"num_object_clones":0,"num_object_copies":7908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10100,"num_write_kb":407552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"374'10688","reported_seq":10911,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-17T16:20:25.672062+0000","last_change":"2024-09-17T16:19:01.159281+0000","last_active":"2024-09-17T16:20:25.672062+0000","last_peered":"2024-09-17T16:20:25.672062+0000","last_clean":"2024-09-17T16:20:25.672062+0000","last_became_active":"2024-09-17T16:17:34.582335+0000","last_became_peered":"2024-09-17T16:17:34.582335+0000","last_unstale":"2024-09-17T16:20:25.672062+0000","last_undegraded":"2024-09-17T16:20:25.672062+0000","last_fullsized":"2024-09-17T16:20:25.672062+0000","mapping_epoch":301,"log_start":"309'600","ondisk_log_start":"309'600","created":301,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'5950","last_scrub_stamp":"2024-09-17T16:19:01.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:17:33.436404+0000","last_clean_scrub_stamp":"2024-09-17T16:19:01.159158+0000","objects_scrubbed":5911,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:27.590602+0000","scrub_duration":1399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159645696,"num_objects":2436,"num_object_clones":0,"num_object_copies":7308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10688,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":639,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-17T16:20:17.045300+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:20:17.045300+0000","last_peered":"2024-09-17T16:20:17.045300+0000","last_clean":"2024-09-17T16:20:17.045300+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:20:17.045300+0000","last_undegraded":"2024-09-17T16:20:17.045300+0000","last_fullsized":"2024-09-17T16:20:17.045300+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:20:55.890376+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":2596274200,"num_objects":39617,"num_object_clones":0,"num_object_copies":118851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39617,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164729,"num_write_kb":6539009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9478,"num_bytes_recovered":621150208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8335470592,"data_stored":8335458376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160629,"ondisk_log_size":160629,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285699,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1539436,"kb_used_data":1247192,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92205716,"statfs":{"total":95995035648,"available":94418653184,"internally_reserved":0,"allocated":1277124608,"data_stored":1276863987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":299170244},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547336,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":922140,"kb_used_data":782668,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92823012,"statfs":{"total":95995035648,"available":95050764288,"internally_reserved":0,"allocated":801452032,"data_stored":801199055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":142801344},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1150356,"kb_used_data":804036,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92594796,"statfs":{"total":95995035648,"available":94817071104,"internally_reserved":0,"allocated":823332864,"data_stored":823080363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":122080,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93623072,"statfs":{"total":95995035648,"available":95870025728,"internally_reserved":0,"allocated":667648,"data_stored":414671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963678,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1208984,"kb_used_data":980040,"kb_used_omap":41,"kb_used_meta":228886,"kb_avail":92536168,"statfs":{"total":95995035648,"available":94757036032,"internally_reserved":0,"allocated":1003560960,"data_stored":1003308979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":234379571},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1364388,"kb_used_data":1098580,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92380764,"statfs":{"total":95995035648,"available":94597902336,"internally_reserved":0,"allocated":1124945920,"data_stored":1124689391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":272169414},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724703,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1031520,"kb_used_data":819852,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92713632,"statfs":{"total":95995035648,"available":94938759168,"internally_reserved":0,"allocated":839528448,"data_stored":839275471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986342,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830756,"kb_used_data":623248,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92914396,"statfs":{"total":95995035648,"available":95144341504,"internally_reserved":0,"allocated":638205952,"data_stored":637948903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1127120,"kb_used_data":852992,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92618032,"statfs":{"total":95995035648,"available":94840864768,"internally_reserved":0,"allocated":873463808,"data_stored":873215379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411503,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":481808,"kb_used_data":334016,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93263344,"statfs":{"total":95995035648,"available":95501664256,"internally_reserved":0,"allocated":342032384,"data_stored":341783955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575029,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738553,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865488,"kb_used_data":676032,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92879664,"statfs":{"total":95995035648,"available":95108775936,"internally_reserved":0,"allocated":692256768,"data_stored":692008339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":690421760,"data_stored":690421760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":338690048,"data_stored":338690048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":854523904,"data_stored":854523904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":633475072,"data_stored":633471000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":838795264,"data_stored":838795264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1100742656,"data_stored":1100742656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":997982208,"data_stored":997982208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":822218752,"data_stored":822214680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":787283968,"data_stored":787283968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1271336960,"data_stored":1271332888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:20:26.932 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T16:20:26.932 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:20:26.933 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-17T16:20:26.933 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-09-17T16:20:26.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:27.007 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:27.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:27.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:27.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:27.118 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:27.133 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:27.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:27.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:27.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:20:27.243 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:27.264 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:27.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:27.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:27.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:27.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:27.497 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:27.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:27.567 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:27 smithi017 ceph-mon[27571]: pgmap v862: 17 pgs: 17 active+clean; 2.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-09-17T16:20:27.567 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:27 smithi017 ceph-mon[27571]: from='client.16311 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:27.567 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:27 smithi017 ceph-mon[27571]: 4.b scrub starts 2024-09-17T16:20:27.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:27.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:27 smithi173 ceph-mon[30530]: pgmap v862: 17 pgs: 17 active+clean; 2.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-09-17T16:20:27.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:27 smithi173 ceph-mon[30530]: from='client.16311 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:27.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:27 smithi173 ceph-mon[30530]: 4.b scrub starts 2024-09-17T16:20:27.650 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:27.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:27.751 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:27.783 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:27.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:27.804 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:27.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:27 smithi071 ceph-mon[30002]: pgmap v862: 17 pgs: 17 active+clean; 2.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-09-17T16:20:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:27 smithi071 ceph-mon[30002]: from='client.16311 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:20:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:27 smithi071 ceph-mon[30002]: 4.b scrub starts 2024-09-17T16:20:27.833 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:27.840 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:27.840 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:27.884 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:27.911 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:27.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:28.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:28.044 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:28.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:28.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:28.178 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:28.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:28.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:28.304 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:28.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:28.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:28.433 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:28.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:28.454 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:20:28.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: 4.b scrub ok 2024-09-17T16:20:28.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:20:28.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:20:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:20:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:20:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:20:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:20:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: 4.5 scrub starts 2024-09-17T16:20:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3153055902' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:28 smithi017 ceph-mon[27571]: 4.5 scrub ok 2024-09-17T16:20:28.509 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: 4.b scrub ok 2024-09-17T16:20:28.509 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:20:28.509 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:20:28.509 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:20:28.509 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:20:28.509 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:20:28.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:20:28.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:28.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: 4.5 scrub starts 2024-09-17T16:20:28.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3153055902' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:28.510 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:28 smithi173 ceph-mon[30530]: 4.5 scrub ok 2024-09-17T16:20:28.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:28.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:20:28.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:28.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:28.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:28.699 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:28.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:28.801 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: 4.b scrub ok 2024-09-17T16:20:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:20:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:20:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:20:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:20:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:20:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:20:28.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:20:28.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: 4.5 scrub starts 2024-09-17T16:20:28.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3153055902' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:28.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:28 smithi071 ceph-mon[30002]: 4.5 scrub ok 2024-09-17T16:20:28.830 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:28.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:28.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:28.924 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:28.930 INFO:teuthology.orchestra.run.smithi017.stdout:moved 0 / 51 (0%) 2024-09-17T16:20:28.930 INFO:teuthology.orchestra.run.smithi017.stdout:avg 4.63636 2024-09-17T16:20:28.930 INFO:teuthology.orchestra.run.smithi017.stdout:stddev 2.56808 -> 2.56808 (expected baseline 2.05302) 2024-09-17T16:20:28.930 INFO:teuthology.orchestra.run.smithi017.stdout:min osd.8 with 0 -> 0 pgs (0 -> 0 * mean) 2024-09-17T16:20:28.930 INFO:teuthology.orchestra.run.smithi017.stdout:max osd.11 with 9 -> 9 pgs (1.94118 -> 1.94118 * mean) 2024-09-17T16:20:28.930 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:28.930 INFO:teuthology.orchestra.run.smithi017.stdout:oload 1000 2024-09-17T16:20:28.931 INFO:teuthology.orchestra.run.smithi017.stdout:max_change 1 2024-09-17T16:20:28.931 INFO:teuthology.orchestra.run.smithi017.stdout:max_change_osds 4 2024-09-17T16:20:28.931 INFO:teuthology.orchestra.run.smithi017.stdout:average_utilization 0.0100 2024-09-17T16:20:28.931 INFO:teuthology.orchestra.run.smithi017.stdout:overload_utilization 0.0996 2024-09-17T16:20:28.931 INFO:teuthology.orchestra.run.smithi017.stderr:no change 2024-09-17T16:20:28.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:28.959 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:28.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:29.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:29.093 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:29.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:29.196 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:29.228 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:29.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:29.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:29.356 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:29.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:29.459 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:29.486 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:29.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:29.539 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:29.539 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:29.587 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:29.600 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:20:29.601 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:20:29.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:29 smithi017 ceph-mon[27571]: pgmap v863: 17 pgs: 17 active+clean; 2.3 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-09-17T16:20:29.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:29 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:20:29.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:29 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:20:29.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3153055902' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:29.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:29 smithi017 ceph-mon[27571]: osdmap e375: 12 total, 12 up, 11 in 2024-09-17T16:20:29.617 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:29.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:29.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:29.629 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:20:29.629 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:20:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:29 smithi173 ceph-mon[30530]: pgmap v863: 17 pgs: 17 active+clean; 2.3 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-09-17T16:20:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:29 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:20:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:29 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:20:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3153055902' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:29 smithi173 ceph-mon[30530]: osdmap e375: 12 total, 12 up, 11 in 2024-09-17T16:20:29.654 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:20:29.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:20:29.679 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:20:29.679 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:20:29.704 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:20:29.705 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:20:29.719 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:29.733 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:20:29.733 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:20:29.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:29.745 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:29.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:29.756 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:20:29.756 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:20:29.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:20:29.783 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:20:29.808 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:20:29.809 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:20:29.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:29 smithi071 ceph-mon[30002]: pgmap v863: 17 pgs: 17 active+clean; 2.3 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 849 op/s 2024-09-17T16:20:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:29 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:20:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:29 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:20:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3153055902' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:29 smithi071 ceph-mon[30002]: osdmap e375: 12 total, 12 up, 11 in 2024-09-17T16:20:29.831 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:20:29.832 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:20:29.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:29.854 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:20:29.854 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:20:29.872 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:29.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:29.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:29.877 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:20:29.877 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:20:29.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:29.999 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:30.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:30.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:30.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:30.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:30.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:30.257 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:30.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:30.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:20:30.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:30.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:30.388 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:30.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:30.520 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:30.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:30.615 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:30.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:30.644 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:30.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:30 smithi017 ceph-mon[27571]: from='client.16323 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1133061811' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:30.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:30.747 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:30 smithi173 ceph-mon[30530]: from='client.16323 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:30.747 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1133061811' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:30.774 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:30.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:30 smithi071 ceph-mon[30002]: from='client.16323 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1133061811' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:30.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:30.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:30.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:31.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:31.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:31.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:31.037 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:31.138 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:31.166 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:31.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:31.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:31.271 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:20:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:20:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25516.0:170855 4.6 4:61b038cf:::benchmark_data_smithi173_77474_object68680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-17T16:20:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:20:31.253382+0000", 2024-09-17T16:20:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.015182573, 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015211728000000001, 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25516", 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 170855 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.253382+0000", 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.253382+0000", 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.253383+0000", 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.015e-06 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.253387+0000", 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7500000000000003e-06 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.253402+0000", 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4746000000000001e-05 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.253409+0000", 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2259999999999999e-06 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.257395+0000", 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039850780000000004 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.257519+0000", 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012409899999999999 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.257640+0000", 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012095 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.261946+0000", 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043059609999999996 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.262043+0000", 2024-09-17T16:20:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.768e-05 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25516.0:170866 4.6 4:6338e646:::benchmark_data_smithi173_77474_object68691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:20:31.267063+0000", 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0015015670000000001, 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001634183, 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:20:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25516", 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 170866 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.267063+0000", 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.267063+0000", 2024-09-17T16:20:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.267065+0000", 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.928e-06 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.267073+0000", 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6309999999999992e-06 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.267086+0000", 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3149000000000001e-05 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.267096+0000", 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1980000000000007e-06 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.267189+0000", 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3197999999999998e-05 2024-09-17T16:20:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.267396+0000", 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020678299999999999 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.267556+0000", 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016018799999999999 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:170867 4.2 e375/312 4:412066c3:::benchmark_data_smithi173_77474_object68692:head v 375'10507, mlcod=375'10503)", 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:20:31.268437+0000", 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.00012686500000000001, 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032424800000000002, 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.268437+0000", 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.268437+0000", 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.268439+0000", 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1400000000000001e-06 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.268444+0000", 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8579999999999996e-06 2024-09-17T16:20:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.268472+0000", 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7917999999999999e-05 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.268479+0000", 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3779999999999998e-06 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.268526+0000", 2024-09-17T16:20:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6817999999999998e-05 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:31.268680+0000", 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015410099999999999 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 3 2024-09-17T16:20:31.285 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:20:31.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:31.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:31.382 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-17T16:20:31.382 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-17T16:20:31.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:20:31.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:31.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:31.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:31.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:31.570 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:31.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:31.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:31.672 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:31.703 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:31.707 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:31.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:31.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:31.720 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:31 smithi017 ceph-mon[27571]: pgmap v865: 17 pgs: 17 active+clean; 2.2 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 759 op/s 2024-09-17T16:20:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:31 smithi017 ceph-mon[27571]: 4.d scrub starts 2024-09-17T16:20:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:31 smithi017 ceph-mon[27571]: 4.d scrub ok 2024-09-17T16:20:31.770 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:31 smithi173 ceph-mon[30530]: pgmap v865: 17 pgs: 17 active+clean; 2.2 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 759 op/s 2024-09-17T16:20:31.770 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:31 smithi173 ceph-mon[30530]: 4.d scrub starts 2024-09-17T16:20:31.770 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:31 smithi173 ceph-mon[30530]: 4.d scrub ok 2024-09-17T16:20:31.809 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:31 smithi071 ceph-mon[30002]: pgmap v865: 17 pgs: 17 active+clean; 2.2 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 759 op/s 2024-09-17T16:20:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:31 smithi071 ceph-mon[30002]: 4.d scrub starts 2024-09-17T16:20:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:31 smithi071 ceph-mon[30002]: 4.d scrub ok 2024-09-17T16:20:31.833 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:31.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:31.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:31.958 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:31.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:31.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:32.060 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:32.092 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:32.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:32.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:20:32.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:32.223 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:32.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:32.325 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:32.355 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:32.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:32.456 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:32.474 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:32.485 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:32.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:32.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:32.616 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:32.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:32.703 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:32 smithi017 ceph-mon[27571]: 4.e scrub starts 2024-09-17T16:20:32.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:32.744 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:32.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:32.762 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:32 smithi173 ceph-mon[30530]: 4.e scrub starts 2024-09-17T16:20:32.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:32 smithi071 ceph-mon[30002]: 4.e scrub starts 2024-09-17T16:20:32.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:32.878 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:32.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:32.912 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:20:32.980 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:33.012 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:33.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:33.054 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:20:33.054 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:20:33.054 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:20:33.054 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:20:33.054 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:20:33.092 INFO:teuthology.orchestra.run.smithi017.stdout:moved 0 / 51 (0%) 2024-09-17T16:20:33.092 INFO:teuthology.orchestra.run.smithi017.stdout:avg 4.63636 2024-09-17T16:20:33.092 INFO:teuthology.orchestra.run.smithi017.stdout:stddev 2.56808 -> 2.56808 (expected baseline 2.05302) 2024-09-17T16:20:33.093 INFO:teuthology.orchestra.run.smithi017.stdout:min osd.8 with 0 -> 0 pgs (0 -> 0 * mean) 2024-09-17T16:20:33.093 INFO:teuthology.orchestra.run.smithi017.stdout:max osd.11 with 9 -> 9 pgs (1.94118 -> 1.94118 * mean) 2024-09-17T16:20:33.093 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:33.093 INFO:teuthology.orchestra.run.smithi017.stdout:oload 1000 2024-09-17T16:20:33.093 INFO:teuthology.orchestra.run.smithi017.stdout:max_change 0.05 2024-09-17T16:20:33.093 INFO:teuthology.orchestra.run.smithi017.stdout:max_change_osds 4 2024-09-17T16:20:33.094 INFO:teuthology.orchestra.run.smithi017.stdout:average_utilization 0.0094 2024-09-17T16:20:33.094 INFO:teuthology.orchestra.run.smithi017.stdout:overload_utilization 0.0937 2024-09-17T16:20:33.094 INFO:teuthology.orchestra.run.smithi017.stderr:no change 2024-09-17T16:20:33.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:33.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:33.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:33.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:33.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:33.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:33.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:33.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:33.401 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:33.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:33.426 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:20:33.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:33.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:33.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:33.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:33.658 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:33.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:33.704 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:33 smithi173 ceph-mon[30530]: pgmap v866: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 895 op/s 2024-09-17T16:20:33.704 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:33 smithi173 ceph-mon[30530]: 4.e scrub ok 2024-09-17T16:20:33.704 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3338992251' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:33.704 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:33 smithi173 ceph-mon[30530]: from='client.16341 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:33 smithi017 ceph-mon[27571]: pgmap v866: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 895 op/s 2024-09-17T16:20:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:33 smithi017 ceph-mon[27571]: 4.e scrub ok 2024-09-17T16:20:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3338992251' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:33 smithi017 ceph-mon[27571]: from='client.16341 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:33.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:20:33.761 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:33.784 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:33.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:33 smithi071 ceph-mon[30002]: pgmap v866: 17 pgs: 17 active+clean; 2.1 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 895 op/s 2024-09-17T16:20:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:33 smithi071 ceph-mon[30002]: 4.e scrub ok 2024-09-17T16:20:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3338992251' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:33 smithi071 ceph-mon[30002]: from='client.16341 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:33.886 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:33.911 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:33.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:34.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:34.038 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:34.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:34.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:34.070 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:34.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:34.170 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:34.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:34.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:34.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:34.302 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:34.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:34.428 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:34.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:34.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:34 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:34.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3338992251' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:20:34.447 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:34 smithi173 ceph-mon[30530]: osdmap e376: 12 total, 12 up, 11 in 2024-09-17T16:20:34.448 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:34 smithi173 ceph-mon[30530]: pgmap v868: 17 pgs: 17 active+clean; 2.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 738 op/s 2024-09-17T16:20:34.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:34.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:34.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:34.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:34.691 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:34.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:34.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:34.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:34 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:34.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3338992251' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:20:34.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:34 smithi017 ceph-mon[27571]: osdmap e376: 12 total, 12 up, 11 in 2024-09-17T16:20:34.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:34 smithi017 ceph-mon[27571]: pgmap v868: 17 pgs: 17 active+clean; 2.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 738 op/s 2024-09-17T16:20:34.753 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-17T16:20:34.753 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-09-17T16:20:34.755 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:34.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:34 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3338992251' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:20:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:34 smithi071 ceph-mon[30002]: osdmap e376: 12 total, 12 up, 11 in 2024-09-17T16:20:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:34 smithi071 ceph-mon[30002]: pgmap v868: 17 pgs: 17 active+clean; 2.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 738 op/s 2024-09-17T16:20:34.821 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:34.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:34.859 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:20:34.859 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:20:34.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:14931 2.18 2.b4f18d38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:14.294157+0000", 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 560.56139449299997, 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.967921951, 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:34.860 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14931 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294157+0000", 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294157+0000", 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294159+0000", 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1039999999999999e-06 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294178+0000", 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9079000000000001e-05 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294189+0000", 2024-09-17T16:20:34.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0926e-05 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294194+0000", 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4759999999999997e-06 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294244+0000", 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9740000000000001e-05 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:20:34.863 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:14.294251+0000", 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5290000000000003e-06 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:15.262048+0000", 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96779710200000002 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:15.262079+0000", 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0994999999999997e-05 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.864 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:76468 2.15 e149/113 ondisk, result = 0)", 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:32.182636+0000", 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 482.67291586599998, 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.74260242600000004, 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.182636+0000", 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.182636+0000", 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.182637+0000", 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3499999999999996e-07 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.182640+0000", 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2490000000000001e-06 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.925050+0000", 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.74241045500000002 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.925060+0000", 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2930000000000006e-06 2024-09-17T16:20:34.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.925141+0000", 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1599000000000003e-05 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.925149+0000", 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6009999999999997e-06 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.925238+0000", 2024-09-17T16:20:34.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9693999999999999e-05 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:76476 2.d e149/95 ondisk, result = 0)", 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:32.192989+0000", 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 482.66256295800002, 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88641944299999997, 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:34.869 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.192989+0000", 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.192989+0000", 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.192990+0000", 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5499999999999998e-07 2024-09-17T16:20:34.870 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:32.192993+0000", 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1540000000000002e-06 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:33.079232+0000", 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.88623938400000002 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:33.079239+0000", 2024-09-17T16:20:34.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8510000000000001e-06 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:33.079303+0000", 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3652999999999995e-05 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:33.079310+0000", 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0160000000000003e-06 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:33.079408+0000", 2024-09-17T16:20:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8830000000000001e-05 2024-09-17T16:20:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(3.e 240/239 [PushOp(3:7f188a09:::benchmark_data_smithi173_66272_object4256:head, version: 213'268, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:7f188a09:::benchmark_data_smithi173_66272_object4256:head@213'268, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:20:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:16.226711+0000", 2024-09-17T16:20:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.628841411, 2024-09-17T16:20:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.677631184, 2024-09-17T16:20:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:16.226711+0000", 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:16.226711+0000", 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:16.226711+0000", 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2500000000000005e-07 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:16.226754+0000", 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2843999999999998e-05 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:16.903273+0000", 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67651915299999998 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:16.903292+0000", 2024-09-17T16:20:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8754999999999999e-05 2024-09-17T16:20:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:16.903395+0000", 2024-09-17T16:20:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010336 2024-09-17T16:20:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:16.903411+0000", 2024-09-17T16:20:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5387e-05 2024-09-17T16:20:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.878 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:16.904342+0000", 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093106000000000005 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46460 3.0 3:01d0a3c8:::benchmark_data_smithi173_66272_object46459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.158690+0000", 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.696861738, 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0928823219999999, 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:20:34.879 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46460 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158690+0000", 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158690+0000", 2024-09-17T16:20:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158692+0000", 2024-09-17T16:20:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8530000000000001e-06 2024-09-17T16:20:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158718+0000", 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6443000000000001e-05 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158736+0000", 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7501999999999999e-05 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158745+0000", 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5269999999999998e-06 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.246558+0000", 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0878132250000001 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.251572+0000", 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0050147719999999998 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.884 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46493 3.0 3:011aabf5:::benchmark_data_smithi173_66272_object46492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.158765+0000", 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.69678645800002, 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1194322990000001, 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46493 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158765+0000", 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158765+0000", 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158766+0000", 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1799999999999995e-07 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158793+0000", 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6733e-05 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158802+0000", 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2070000000000001e-06 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158809+0000", 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8369999999999999e-06 2024-09-17T16:20:34.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.267714+0000", 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.108904804 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.267825+0000", 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111328 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.267957+0000", 2024-09-17T16:20:34.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000131738 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.272277+0000", 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043203709999999999 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.272297+0000", 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0329000000000001e-05 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.276420+0000", 2024-09-17T16:20:34.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004122377 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.276457+0000", 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.701e-05 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.278198+0000", 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017410469999999999 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.890 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46496 3.0 3:0481a0ad:::benchmark_data_smithi173_66272_object46495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.158819+0000", 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.69673325100001, 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.120540683, 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.891 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46496 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158819+0000", 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158819+0000", 2024-09-17T16:20:34.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158819+0000", 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7800000000000002e-07 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158838+0000", 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9239e-05 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158846+0000", 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.943e-06 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158853+0000", 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4749999999999998e-06 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.269241+0000", 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1103884319999999 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.269389+0000", 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014819799999999999 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.269569+0000", 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017973800000000001 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.272306+0000", 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027364920000000001 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.272368+0000", 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2551999999999993e-05 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.278255+0000", 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058863320000000002 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.278293+0000", 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8418999999999999e-05 2024-09-17T16:20:34.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.279359+0000", 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001066385 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46503 3.0 3:035d4f58:::benchmark_data_smithi173_66272_object46502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.158861+0000", 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.696691139, 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1248135109999999, 2024-09-17T16:20:34.897 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46503 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158861+0000", 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158861+0000", 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158861+0000", 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3500000000000002e-07 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158899+0000", 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8182000000000001e-05 2024-09-17T16:20:34.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158908+0000", 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7800000000000006e-06 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158914+0000", 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6010000000000002e-06 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.271051+0000", 2024-09-17T16:20:34.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1121368620000001 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.271139+0000", 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7959000000000003e-05 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.271279+0000", 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014066299999999999 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.274462+0000", 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003182795 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.274473+0000", 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1368e-05 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.282100+0000", 2024-09-17T16:20:34.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0076270319999999997 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.282152+0000", 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1529000000000001e-05 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.283674+0000", 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015223050000000001 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.903 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46517 3.0 3:071ef6ea:::benchmark_data_smithi173_66272_object46516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.158923+0000", 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.696629164, 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.12985578, 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.904 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46517 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158923+0000", 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158923+0000", 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158923+0000", 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5999999999999999e-07 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158964+0000", 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0540000000000001e-05 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158973+0000", 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7790000000000001e-06 2024-09-17T16:20:34.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158978+0000", 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8039999999999998e-06 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.272416+0000", 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.113437615 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.272540+0000", 2024-09-17T16:20:34.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012416799999999999 2024-09-17T16:20:34.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.272691+0000", 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015070799999999999 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.274505+0000", 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001813898 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.275341+0000", 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083632200000000002 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.286891+0000", 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011549754000000001 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.286964+0000", 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3563e-05 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.288779+0000", 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001814169 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46524 3.0 3:0443c1a4:::benchmark_data_smithi173_66272_object46523:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.158987+0000", 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.69656456199999, 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.131321341, 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.911 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46524 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158987+0000", 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158987+0000", 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.158988+0000", 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3799999999999998e-07 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159016+0000", 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8101000000000001e-05 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159024+0000", 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2779999999999993e-06 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159031+0000", 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6370000000000002e-06 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.273593+0000", 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1145624839999999 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.273673+0000", 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9438999999999996e-05 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.273810+0000", 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000136947 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.276348+0000", 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002538659 2024-09-17T16:20:34.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.276400+0000", 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1249999999999999e-05 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.288849+0000", 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012449884 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.288920+0000", 2024-09-17T16:20:34.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0885999999999999e-05 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.290309+0000", 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001388338 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46532 3.0 3:0e5ff288:::benchmark_data_smithi173_66272_object46531:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.159039+0000", 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.69651267799998, 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1324267690000001, 2024-09-17T16:20:34.917 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46532 2024-09-17T16:20:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159039+0000", 2024-09-17T16:20:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159039+0000", 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159040+0000", 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4000000000000002e-07 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159061+0000", 2024-09-17T16:20:34.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1783000000000001e-05 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159069+0000", 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7470000000000005e-06 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159076+0000", 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7800000000000003e-06 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.274533+0000", 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.115456963 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.274611+0000", 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8251999999999995e-05 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.274705+0000", 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3626000000000001e-05 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.276391+0000", 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001685794 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.279372+0000", 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002980899 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.290515+0000", 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011143082 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.290553+0000", 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8538000000000003e-05 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.291466+0000", 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091286500000000001 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.924 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46547 3.0 3:05216654:::benchmark_data_smithi173_66272_object46546:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.159083+0000", 2024-09-17T16:20:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.696469218, 2024-09-17T16:20:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1336803090000001, 2024-09-17T16:20:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.925 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46547 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159083+0000", 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159083+0000", 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159083+0000", 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9000000000000002e-07 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159108+0000", 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4960000000000002e-05 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159116+0000", 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0169999999999996e-06 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159123+0000", 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5570000000000001e-06 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.275373+0000", 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1162506329999999 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.275457+0000", 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4048999999999994e-05 2024-09-17T16:20:34.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.275559+0000", 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010149600000000001 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.282039+0000", 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006480352 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.283721+0000", 2024-09-17T16:20:34.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016818689999999999 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.291530+0000", 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0078092400000000003 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.291589+0000", 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8363999999999999e-05 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.292763+0000", 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011743820000000001 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46560 3.0 3:052a4b6c:::benchmark_data_smithi173_66272_object46559:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.159131+0000", 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.69642069600002, 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.138316909, 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.931 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46560 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159131+0000", 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159131+0000", 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159132+0000", 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4700000000000002e-07 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159153+0000", 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1087999999999999e-05 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159160+0000", 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6459999999999993e-06 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159166+0000", 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.694e-06 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.279407+0000", 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1202412930000001 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.279508+0000", 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100206 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.279611+0000", 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010319000000000001 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.283730+0000", 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041196360000000003 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.283758+0000", 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7098e-05 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.295835+0000", 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012077684 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.295878+0000", 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2639000000000003e-05 2024-09-17T16:20:34.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.297448+0000", 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001570288 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46563 3.0 3:0b43d8f4:::benchmark_data_smithi173_66272_object46562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.159174+0000", 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.69637794599998, 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1420026000000001, 2024-09-17T16:20:34.937 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46563 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159174+0000", 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159174+0000", 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159174+0000", 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9499999999999998e-07 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159191+0000", 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6949999999999999e-05 2024-09-17T16:20:34.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159198+0000", 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1099999999999997e-06 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159203+0000", 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0409999999999997e-06 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.280886+0000", 2024-09-17T16:20:34.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1216827140000001 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.281021+0000", 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013444800000000001 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.281199+0000", 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000177966 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.285364+0000", 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041655030000000001 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.290432+0000", 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0050674559999999997 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.299298+0000", 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0088664940000000008 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.299360+0000", 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1580000000000003e-05 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.301177+0000", 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018169429999999999 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46570 3.0 3:0228d07c:::benchmark_data_smithi173_66272_object46569:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.159211+0000", 2024-09-17T16:20:34.943 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.696341287, 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1433704069999999, 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46570 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159211+0000", 2024-09-17T16:20:34.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159211+0000", 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159211+0000", 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6600000000000002e-07 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159228+0000", 2024-09-17T16:20:34.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7119e-05 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159235+0000", 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1720000000000001e-06 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159241+0000", 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9920000000000004e-06 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.283785+0000", 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1245437389999999 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.283877+0000", 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2368999999999996e-05 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.283990+0000", 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011237 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.290386+0000", 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0063963910000000004 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.290478+0000", 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1966999999999995e-05 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.301242+0000", 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010764029 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.301283+0000", 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0601e-05 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.302581+0000", 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012982919999999999 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46584 3.0 3:04c9e1f4:::benchmark_data_smithi173_66272_object46583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.949 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.159248+0000", 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.696304091, 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1546924620000001, 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46584 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159248+0000", 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159248+0000", 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159248+0000", 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8000000000000001e-07 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159266+0000", 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8278999999999999e-05 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159274+0000", 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2779999999999996e-06 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159279+0000", 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.101e-06 2024-09-17T16:20:34.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.285402+0000", 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.12612314 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.285491+0000", 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9185999999999996e-05 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.285603+0000", 2024-09-17T16:20:34.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111468 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.290393+0000", 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047907999999999996 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.292832+0000", 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024388510000000001 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.311985+0000", 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.019152413 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.312025+0000", 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9875e-05 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.313940+0000", 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001915691 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.955 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46615 3.0 3:0267e8e4:::benchmark_data_smithi173_66272_object46614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.159286+0000", 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.696265719, 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1561902580000001, 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.956 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46615 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159286+0000", 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159286+0000", 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159287+0000", 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9400000000000001e-07 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159308+0000", 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1152999999999999e-05 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159315+0000", 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3329999999999999e-06 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159320+0000", 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0969999999999996e-06 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.292876+0000", 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1335556920000001 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.292989+0000", 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112909 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.293097+0000", 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108354 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.299232+0000", 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0061346040000000001 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.299249+0000", 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7609e-05 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.313999+0000", 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014749942 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.314048+0000", 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8751000000000001e-05 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.315476+0000", 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014284199999999999 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46618 3.0 3:0094f295:::benchmark_data_smithi173_66272_object46617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.159328+0000", 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.696224202, 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1576202209999999, 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.962 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46618 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159328+0000", 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159328+0000", 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159328+0000", 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8000000000000001e-07 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159346+0000", 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7422999999999999e-05 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159355+0000", 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7240000000000004e-06 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159362+0000", 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5309999999999998e-06 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.294340+0000", 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.134978415 2024-09-17T16:20:34.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.294473+0000", 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013258700000000001 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.294642+0000", 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016905300000000001 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.299258+0000", 2024-09-17T16:20:34.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046156929999999997 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.311828+0000", 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012569983 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.315542+0000", 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037145609999999999 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.315612+0000", 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9846999999999997e-05 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.316948+0000", 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013360239999999999 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46625 3.0 3:08656e84:::benchmark_data_smithi173_66272_object46624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.968 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.159370+0000", 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.69618240300002, 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1586553150000001, 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46625 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159370+0000", 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159370+0000", 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159370+0000", 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5600000000000001e-07 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159418+0000", 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8374999999999997e-05 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159427+0000", 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.337e-06 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159433+0000", 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3620000000000002e-06 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.302624+0000", 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.143191353 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.302726+0000", 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101999 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.302843+0000", 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011695400000000001 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.305265+0000", 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024217409999999998 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.311883+0000", 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0066182439999999997 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.316997+0000", 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0051133150000000002 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.317040+0000", 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3402000000000002e-05 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.318025+0000", 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0009848770000000001 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.974 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:46627 3.0 3:0a052312:::benchmark_data_smithi173_66272_object46626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:17.159441+0000", 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.69611101700002, 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1596414669999999, 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46627 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159441+0000", 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159441+0000", 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159441+0000", 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0499999999999999e-07 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159482+0000", 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0395999999999997e-05 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159489+0000", 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6790000000000005e-06 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:17.159496+0000", 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3509999999999998e-06 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.304105+0000", 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1446088059999999 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.304201+0000", 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5995000000000005e-05 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.304299+0000", 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8900000000000005e-05 2024-09-17T16:20:34.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.306641+0000", 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002342018 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.311947+0000", 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0053056020000000004 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.318062+0000", 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0061150220000000003 2024-09-17T16:20:34.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.318100+0000", 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7966999999999997e-05 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:18.319082+0000", 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098232599999999995 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:34.980 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:34.981 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:20:34.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:35.012 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:35.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:35.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:35.139 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:35.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:35.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:35.269 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:35.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:35.350 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:35.364 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:20:35.364 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:20:35.370 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:35.398 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:20:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:20:35.399 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:35.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:35.425 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:20:35.426 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:20:35.448 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:20:35.448 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:20:35.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:20:35.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:20:35.499 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:20:35.499 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:20:35.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:35.531 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:35.532 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:20:35.532 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:20:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:35.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:35.562 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:20:35.562 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:20:35.589 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:20:35.589 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:20:35.619 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:20:35.619 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:20:35.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1171542152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:35.634 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:35.649 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:20:35.649 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:20:35.660 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:35.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:35.681 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:20:35.681 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:20:35.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1171542152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:35.762 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:35.796 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:35.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:35.806 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1171542152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:35.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:35.930 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:35.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:36.032 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:36.127 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:36.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:36.229 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:36.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:20:36.255 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:36.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:36.303 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:20:36.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:36.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:36.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:36.441 INFO:teuthology.orchestra.run.smithi017.stdout:moved 3 / 51 (5.88235%) 2024-09-17T16:20:36.441 INFO:teuthology.orchestra.run.smithi017.stdout:avg 4.63636 2024-09-17T16:20:36.441 INFO:teuthology.orchestra.run.smithi017.stdout:stddev 2.56808 -> 2.26727 (expected baseline 2.05302) 2024-09-17T16:20:36.441 INFO:teuthology.orchestra.run.smithi017.stdout:min osd.8 with 0 -> 0 pgs (0 -> 0 * mean) 2024-09-17T16:20:36.441 INFO:teuthology.orchestra.run.smithi017.stdout:max osd.11 with 9 -> 8 pgs (1.94118 -> 1.72549 * mean) 2024-09-17T16:20:36.442 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:36.442 INFO:teuthology.orchestra.run.smithi017.stdout:oload 110 2024-09-17T16:20:36.442 INFO:teuthology.orchestra.run.smithi017.stdout:max_change 0.05 2024-09-17T16:20:36.442 INFO:teuthology.orchestra.run.smithi017.stdout:max_change_osds 4 2024-09-17T16:20:36.442 INFO:teuthology.orchestra.run.smithi017.stdout:average_utilization 0.0088 2024-09-17T16:20:36.442 INFO:teuthology.orchestra.run.smithi017.stdout:overload_utilization 0.0096 2024-09-17T16:20:36.442 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11 weight 1.0000 -> 0.9500 2024-09-17T16:20:36.442 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 weight 1.0000 -> 0.9500 2024-09-17T16:20:36.442 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 weight 1.0000 -> 0.9500 2024-09-17T16:20:36.442 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9 weight 1.0000 -> 0.9500 2024-09-17T16:20:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:36 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:20:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:36 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:20:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:36 smithi017 ceph-mon[27571]: osdmap e377: 12 total, 12 up, 11 in 2024-09-17T16:20:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:36 smithi017 ceph-mon[27571]: pgmap v870: 17 pgs: 17 active+clean; 1.9 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-09-17T16:20:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:36 smithi017 ceph-mon[27571]: from='client.16350 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:36 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260','9':'62260','11':'62260'}"} : dispatch 2024-09-17T16:20:36.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:36.519 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:36.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:36.567 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:36.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:36.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:36.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:36.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:36 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:20:36.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:36 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:20:36.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:36 smithi173 ceph-mon[30530]: osdmap e377: 12 total, 12 up, 11 in 2024-09-17T16:20:36.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:36 smithi173 ceph-mon[30530]: pgmap v870: 17 pgs: 17 active+clean; 1.9 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-09-17T16:20:36.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:36 smithi173 ceph-mon[30530]: from='client.16350 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:36.709 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:36 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260','9':'62260','11':'62260'}"} : dispatch 2024-09-17T16:20:36.746 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:36.767 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:36.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:36.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:36.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:36 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:20:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:36 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:20:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:36 smithi071 ceph-mon[30002]: osdmap e377: 12 total, 12 up, 11 in 2024-09-17T16:20:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:36 smithi071 ceph-mon[30002]: pgmap v870: 17 pgs: 17 active+clean; 1.9 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-09-17T16:20:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:36 smithi071 ceph-mon[30002]: from='client.16350 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:36 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260','9':'62260','11':'62260'}"} : dispatch 2024-09-17T16:20:36.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:36.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:36.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:36.995 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:37.017 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:37.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:37.052 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:37.053 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:37.118 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:37.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:37.140 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:37.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:37.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:37.262 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:37.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:37.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:37.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:37.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:37.488 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:37.512 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:37.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:37.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:37.637 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:37.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:37.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:20:37.700 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:37 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260','9':'62260','11':'62260'}"}]': finished 2024-09-17T16:20:37.700 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:37 smithi173 ceph-mon[30530]: osdmap e378: 12 total, 12 up, 11 in 2024-09-17T16:20:37.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:37 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260','9':'62260','11':'62260'}"}]': finished 2024-09-17T16:20:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:37 smithi017 ceph-mon[27571]: osdmap e378: 12 total, 12 up, 11 in 2024-09-17T16:20:37.761 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:37.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:37 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'6':'62260','7':'62260','9':'62260','11':'62260'}"}]': finished 2024-09-17T16:20:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:37 smithi071 ceph-mon[30002]: osdmap e378: 12 total, 12 up, 11 in 2024-09-17T16:20:37.863 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:37.885 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:37.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:37.923 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:37.986 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:38.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:38.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:38.110 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:38.132 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:38.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:38.200 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-17T16:20:38.200 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-09-17T16:20:38.203 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:38.234 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:38.265 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:38.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:38.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:38.349 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:38.349 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:38.367 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:38.389 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:38.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:38.491 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:38.514 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:38.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:38.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:38.642 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:38.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:38 smithi173 ceph-mon[30530]: osdmap e379: 12 total, 12 up, 11 in 2024-09-17T16:20:38.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:38 smithi173 ceph-mon[30530]: pgmap v873: 17 pgs: 1 peering, 16 active+clean; 1.8 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-09-17T16:20:38.731 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:38 smithi017 ceph-mon[27571]: osdmap e379: 12 total, 12 up, 11 in 2024-09-17T16:20:38.731 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:38 smithi017 ceph-mon[27571]: pgmap v873: 17 pgs: 1 peering, 16 active+clean; 1.8 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-09-17T16:20:38.744 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:38.767 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:38.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:38 smithi071 ceph-mon[30002]: osdmap e379: 12 total, 12 up, 11 in 2024-09-17T16:20:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:38 smithi071 ceph-mon[30002]: pgmap v873: 17 pgs: 1 peering, 16 active+clean; 1.8 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-09-17T16:20:38.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:38.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:38.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:38.999 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:39.029 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:39.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:39.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:39.032 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:39.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:20:39.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:39.160 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:39.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:39.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:39.292 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:39.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:39.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:39.423 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:39.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:39.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:39.435 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:39.455 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:20:39.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:20:39.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:39.551 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:39.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:39.653 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:39.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:39.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:39 smithi017 ceph-mon[27571]: osdmap e380: 12 total, 12 up, 11 in 2024-09-17T16:20:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:39 smithi017 ceph-mon[27571]: 4.6 starting backfill to osd.5 from (0'0,334'4670] MIN to 377'10747 2024-09-17T16:20:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:39 smithi017 ceph-mon[27571]: 4.b starting backfill to osd.5 from (0'0,334'4807] MIN to 377'11041 2024-09-17T16:20:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:39 smithi017 ceph-mon[27571]: 4.3 starting backfill to osd.2 from (0'0,0'0] MAX to 377'11157 2024-09-17T16:20:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/383986720' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:39.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:39.791 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:39 smithi173 ceph-mon[30530]: osdmap e380: 12 total, 12 up, 11 in 2024-09-17T16:20:39.792 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:39 smithi173 ceph-mon[30530]: 4.6 starting backfill to osd.5 from (0'0,334'4670] MIN to 377'10747 2024-09-17T16:20:39.792 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:39 smithi173 ceph-mon[30530]: 4.b starting backfill to osd.5 from (0'0,334'4807] MIN to 377'11041 2024-09-17T16:20:39.792 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:39 smithi173 ceph-mon[30530]: 4.3 starting backfill to osd.2 from (0'0,0'0] MAX to 377'11157 2024-09-17T16:20:39.792 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/383986720' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:39.811 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:39.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:39 smithi071 ceph-mon[30002]: osdmap e380: 12 total, 12 up, 11 in 2024-09-17T16:20:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:39 smithi071 ceph-mon[30002]: 4.6 starting backfill to osd.5 from (0'0,334'4670] MIN to 377'10747 2024-09-17T16:20:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:39 smithi071 ceph-mon[30002]: 4.b starting backfill to osd.5 from (0'0,334'4807] MIN to 377'11041 2024-09-17T16:20:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:39 smithi071 ceph-mon[30002]: 4.3 starting backfill to osd.2 from (0'0,0'0] MAX to 377'11157 2024-09-17T16:20:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/383986720' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:39.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:39.938 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:39.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:40.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:40.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:40.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:40.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:40.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:40.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:40.300 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:40.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:40.411 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:40.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:40.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:40.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:moved 12 / 51 (23.5294%) 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:avg 4.63636 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:stddev 2.26727 -> 1.49379 (expected baseline 2.05302) 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:min osd.8 with 0 -> 2 pgs (0 -> 0.431373 * mean) 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:max osd.6 with 8 -> 3 pgs (1.72549 -> 0.647059 * mean) 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:oload 110 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:max_change 1 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:max_change_osds 4 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:average_utilization 0.0084 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:overload_utilization 0.0093 2024-09-17T16:20:40.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11 weight 0.9500 -> 0.5888 2024-09-17T16:20:40.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 weight 0.9500 -> 0.6679 2024-09-17T16:20:40.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 weight 1.0000 -> 0.8457 2024-09-17T16:20:40.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 weight 0.9500 -> 0.8062 2024-09-17T16:20:40.508 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:40.509 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:40.563 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:40.585 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:40.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:40 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/383986720' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:40 smithi173 ceph-mon[30530]: osdmap e381: 12 total, 12 up, 11 in 2024-09-17T16:20:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:40 smithi173 ceph-mon[30530]: from='client.25994 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:40 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'5':'55422','6':'43772','7':'52833','11':'38590'}"} : dispatch 2024-09-17T16:20:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:40 smithi173 ceph-mon[30530]: pgmap v876: 17 pgs: 1 remapped+peering, 1 peering, 1 active+remapped+backfill_toofull, 14 active+clean; 1.8 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 1695/86841 objects misplaced (1.952%) 2024-09-17T16:20:40.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:40 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:40.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/383986720' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:40.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:40 smithi017 ceph-mon[27571]: osdmap e381: 12 total, 12 up, 11 in 2024-09-17T16:20:40.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:40 smithi017 ceph-mon[27571]: from='client.25994 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:40.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:40 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'5':'55422','6':'43772','7':'52833','11':'38590'}"} : dispatch 2024-09-17T16:20:40.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:40 smithi017 ceph-mon[27571]: pgmap v876: 17 pgs: 1 remapped+peering, 1 peering, 1 active+remapped+backfill_toofull, 14 active+clean; 1.8 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 1695/86841 objects misplaced (1.952%) 2024-09-17T16:20:40.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:40.708 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:40.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:40.810 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:40.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:40 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:40.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/383986720' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:40 smithi071 ceph-mon[30002]: osdmap e381: 12 total, 12 up, 11 in 2024-09-17T16:20:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:40 smithi071 ceph-mon[30002]: from='client.25994 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:40 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'5':'55422','6':'43772','7':'52833','11':'38590'}"} : dispatch 2024-09-17T16:20:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:40 smithi071 ceph-mon[30002]: pgmap v876: 17 pgs: 1 remapped+peering, 1 peering, 1 active+remapped+backfill_toofull, 14 active+clean; 1.8 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 1695/86841 objects misplaced (1.952%) 2024-09-17T16:20:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:40.831 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:40.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:40.933 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:40.956 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:40.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:41.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:41.080 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:41.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:41.138 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:20:41.138 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:20:41.159 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:20:41.160 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:20:41.180 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:20:41.180 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:20:41.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:41.203 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:20:41.203 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:20:41.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:41.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:41.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:41.225 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:20:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:20:41.246 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:20:41.246 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:20:41.267 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:20:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:20:41.289 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:20:41.289 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:20:41.307 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:41.309 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:20:41.309 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:20:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:41.331 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:41.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:41.331 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:20:41.332 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:20:41.352 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:20:41.352 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:20:41.373 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:20:41.374 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:20:41.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:41.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:20:41.454 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:41.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:41.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:41.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:41.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:41 smithi017 ceph-mon[27571]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:20:41.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:41 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'5':'55422','6':'43772','7':'52833','11':'38590'}"}]': finished 2024-09-17T16:20:41.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:41 smithi017 ceph-mon[27571]: osdmap e382: 12 total, 12 up, 11 in 2024-09-17T16:20:41.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/47360871' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:41.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:41.605 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-17T16:20:41.606 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-17T16:20:41.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:41.701 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:41.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:41.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:41.805 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:41 smithi071 ceph-mon[30002]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:20:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:41 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'5':'55422','6':'43772','7':'52833','11':'38590'}"}]': finished 2024-09-17T16:20:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:41 smithi071 ceph-mon[30002]: osdmap e382: 12 total, 12 up, 11 in 2024-09-17T16:20:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/47360871' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:41.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:41 smithi173 ceph-mon[30530]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:20:41.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:41 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'5':'55422','6':'43772','7':'52833','11':'38590'}"}]': finished 2024-09-17T16:20:41.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:41 smithi173 ceph-mon[30530]: osdmap e382: 12 total, 12 up, 11 in 2024-09-17T16:20:41.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/47360871' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:41.834 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:41.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:41.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:41.959 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:41.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:41.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:42.061 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:42.084 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:42.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:42.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:42.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:42.208 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:42.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:42.219 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:20:42.219 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:20:42.219 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:20:42.219 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:20:42.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:42.331 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:42.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:42.366 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:42.410 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:20:42.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:42.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:42.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:42.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:42.519 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:42 smithi017 ceph-mon[27571]: osdmap e383: 12 total, 12 up, 11 in 2024-09-17T16:20:42.519 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:42 smithi017 ceph-mon[27571]: 4.6 starting backfill to osd.0 from (0'0,0'0] MAX to 381'10848 2024-09-17T16:20:42.519 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:42 smithi017 ceph-mon[27571]: pgmap v879: 17 pgs: 1 active+remapped+backfilling, 3 unknown, 1 remapped+peering, 4 peering, 1 active+remapped+backfill_toofull, 7 active+clean; 1.6 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 3276/77805 objects misplaced (4.211%); 1.4 MiB/s, 22 objects/s recovering 2024-09-17T16:20:42.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:42.590 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:42.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:42.692 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:42.726 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:42.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:42.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:42 smithi071 ceph-mon[30002]: osdmap e383: 12 total, 12 up, 11 in 2024-09-17T16:20:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:42 smithi071 ceph-mon[30002]: 4.6 starting backfill to osd.0 from (0'0,0'0] MAX to 381'10848 2024-09-17T16:20:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:42 smithi071 ceph-mon[30002]: pgmap v879: 17 pgs: 1 active+remapped+backfilling, 3 unknown, 1 remapped+peering, 4 peering, 1 active+remapped+backfill_toofull, 7 active+clean; 1.6 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 3276/77805 objects misplaced (4.211%); 1.4 MiB/s, 22 objects/s recovering 2024-09-17T16:20:42.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:42.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:42 smithi173 ceph-mon[30530]: osdmap e383: 12 total, 12 up, 11 in 2024-09-17T16:20:42.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:42 smithi173 ceph-mon[30530]: 4.6 starting backfill to osd.0 from (0'0,0'0] MAX to 381'10848 2024-09-17T16:20:42.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:42 smithi173 ceph-mon[30530]: pgmap v879: 17 pgs: 1 active+remapped+backfilling, 3 unknown, 1 remapped+peering, 4 peering, 1 active+remapped+backfill_toofull, 7 active+clean; 1.6 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 3276/77805 objects misplaced (4.211%); 1.4 MiB/s, 22 objects/s recovering 2024-09-17T16:20:42.855 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:42.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:42.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:20:42.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:42.990 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:42.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:42.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:43.080 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:43.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:43.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:43.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:43.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:43.245 INFO:teuthology.orchestra.run.smithi017.stdout:moved 0 / 51 (0%) 2024-09-17T16:20:43.245 INFO:teuthology.orchestra.run.smithi017.stdout:avg 4.63636 2024-09-17T16:20:43.245 INFO:teuthology.orchestra.run.smithi017.stdout:stddev 1.49379 -> 1.49379 (expected baseline 2.05302) 2024-09-17T16:20:43.245 INFO:teuthology.orchestra.run.smithi017.stdout:min osd.8 with 2 -> 2 pgs (0.431373 -> 0.431373 * mean) 2024-09-17T16:20:43.245 INFO:teuthology.orchestra.run.smithi017.stdout:max osd.0 with 7 -> 7 pgs (1.5098 -> 1.5098 * mean) 2024-09-17T16:20:43.246 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:20:43.246 INFO:teuthology.orchestra.run.smithi017.stdout:oload 1000 2024-09-17T16:20:43.246 INFO:teuthology.orchestra.run.smithi017.stdout:max_change 0.05 2024-09-17T16:20:43.246 INFO:teuthology.orchestra.run.smithi017.stdout:max_change_osds 4 2024-09-17T16:20:43.246 INFO:teuthology.orchestra.run.smithi017.stdout:average_utilization 0.0081 2024-09-17T16:20:43.246 INFO:teuthology.orchestra.run.smithi017.stdout:overload_utilization 0.0809 2024-09-17T16:20:43.246 INFO:teuthology.orchestra.run.smithi017.stderr:no change 2024-09-17T16:20:43.266 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:43.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:43.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:43.313 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:43.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:43.401 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:43.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:43.503 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:43.533 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:43.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:43.635 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:43.661 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:43.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.9 starting backfill to osd.0 from (0'0,0'0] MAX to 381'10863 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.f starting backfill to osd.5 from (0'0,0'0] MAX to 381'11037 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 381'11066 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: osdmap e384: 12 total, 12 up, 11 in 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 381'11201 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.5 starting backfill to osd.0 from (0'0,0'0] MAX to 381'11362 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.8 starting backfill to osd.9 from (0'0,0'0] MAX to 381'11042 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.2 starting backfill to osd.8 from (0'0,0'0] MAX to 381'10880 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.0 starting backfill to osd.8 from (0'0,0'0] MAX to 381'11034 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.8 starting backfill to osd.10 from (0'0,0'0] MAX to 381'11042 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.5 starting backfill to osd.7 from (0'0,0'0] MAX to 381'11362 2024-09-17T16:20:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: 4.9 starting backfill to osd.4 from (0'0,0'0] MAX to 381'10863 2024-09-17T16:20:43.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:43 smithi017 ceph-mon[27571]: from='client.16374 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:43.744 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.9 starting backfill to osd.0 from (0'0,0'0] MAX to 381'10863 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.f starting backfill to osd.5 from (0'0,0'0] MAX to 381'11037 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 381'11066 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: osdmap e384: 12 total, 12 up, 11 in 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 381'11201 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.5 starting backfill to osd.0 from (0'0,0'0] MAX to 381'11362 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.8 starting backfill to osd.9 from (0'0,0'0] MAX to 381'11042 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.2 starting backfill to osd.8 from (0'0,0'0] MAX to 381'10880 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.0 starting backfill to osd.8 from (0'0,0'0] MAX to 381'11034 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.8 starting backfill to osd.10 from (0'0,0'0] MAX to 381'11042 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.5 starting backfill to osd.7 from (0'0,0'0] MAX to 381'11362 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: 4.9 starting backfill to osd.4 from (0'0,0'0] MAX to 381'10863 2024-09-17T16:20:43.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:43 smithi173 ceph-mon[30530]: from='client.16374 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:43.763 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:43.792 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:43.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:43.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.9 starting backfill to osd.0 from (0'0,0'0] MAX to 381'10863 2024-09-17T16:20:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.f starting backfill to osd.5 from (0'0,0'0] MAX to 381'11037 2024-09-17T16:20:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.e starting backfill to osd.3 from (0'0,0'0] MAX to 381'11066 2024-09-17T16:20:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: osdmap e384: 12 total, 12 up, 11 in 2024-09-17T16:20:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.a starting backfill to osd.5 from (0'0,0'0] MAX to 381'11201 2024-09-17T16:20:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.5 starting backfill to osd.0 from (0'0,0'0] MAX to 381'11362 2024-09-17T16:20:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.8 starting backfill to osd.9 from (0'0,0'0] MAX to 381'11042 2024-09-17T16:20:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.2 starting backfill to osd.8 from (0'0,0'0] MAX to 381'10880 2024-09-17T16:20:43.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.0 starting backfill to osd.8 from (0'0,0'0] MAX to 381'11034 2024-09-17T16:20:43.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.8 starting backfill to osd.10 from (0'0,0'0] MAX to 381'11042 2024-09-17T16:20:43.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.5 starting backfill to osd.7 from (0'0,0'0] MAX to 381'11362 2024-09-17T16:20:43.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: 4.9 starting backfill to osd.4 from (0'0,0'0] MAX to 381'10863 2024-09-17T16:20:43.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:43 smithi071 ceph-mon[30002]: from='client.16374 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T16:20:43.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:43.899 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:20:43.900 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:20:43.900 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:20:43.900 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:20:43.900 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:20:43.924 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:43.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:44.027 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:44.065 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:44.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:44.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:44.207 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:44.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:44.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:44.271 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:20:44.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:44.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:44.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:44.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:44.493 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:20:44.514 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:44.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:44.617 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:44.620 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:44 smithi173 ceph-mon[30530]: pgmap v881: 17 pgs: 1 active+remapped+backfilling, 3 unknown, 4 peering, 2 active+remapped+backfill_toofull, 7 active+clean; 1.6 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 5031/77535 objects misplaced (6.489%); 1.3 MiB/s, 20 objects/s recovering 2024-09-17T16:20:44.620 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/699817872' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:20:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:44.648 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:44.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:44.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:20:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:44 smithi017 ceph-mon[27571]: pgmap v881: 17 pgs: 1 active+remapped+backfilling, 3 unknown, 4 peering, 2 active+remapped+backfill_toofull, 7 active+clean; 1.6 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 5031/77535 objects misplaced (6.489%); 1.3 MiB/s, 20 objects/s recovering 2024-09-17T16:20:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/699817872' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:20:44.749 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:44.777 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:44.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:44 smithi071 ceph-mon[30002]: pgmap v881: 17 pgs: 1 active+remapped+backfilling, 3 unknown, 4 peering, 2 active+remapped+backfill_toofull, 7 active+clean; 1.6 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 5031/77535 objects misplaced (6.489%); 1.3 MiB/s, 20 objects/s recovering 2024-09-17T16:20:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/699817872' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:20:44.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:44.905 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:44.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:44.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:45.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:45.009 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:20:45.034 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:45.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:45.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:45.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:45.169 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:45.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:45.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:45.301 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:45.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:45.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:45.437 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:45.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:45.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:45.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:45.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:45.587 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:45.618 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:20:45.618 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:20:45.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:20:45.618 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:20:45.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.618 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:13531 2.a 2:59aa3d24:::benchmark_data_smithi173_55121_object13530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-17T16:20:45.618 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:12.331033+0000", 2024-09-17T16:20:45.618 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 573.28283662800004, 2024-09-17T16:20:45.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50813131099999997, 2024-09-17T16:20:45.618 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 13531 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331033+0000", 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331033+0000", 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331034+0000", 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0200000000000001e-07 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331051+0000", 2024-09-17T16:20:45.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7229999999999999e-05 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331061+0000", 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.041e-05 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331067+0000", 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1730000000000003e-06 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331115+0000", 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8053999999999998e-05 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331233+0000", 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118643 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331371+0000", 2024-09-17T16:20:45.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000137973 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.331939+0000", 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056774899999999999 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.332293+0000", 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000353698 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.839080+0000", 2024-09-17T16:20:45.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50678687099999997 2024-09-17T16:20:45.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.839127+0000", 2024-09-17T16:20:45.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7236e-05 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:12.839164+0000", 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7472000000000003e-05 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:20813 2.a e116/95 ondisk, result = 0)", 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:25.337708+0000", 2024-09-17T16:20:45.625 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 560.27616184399994, 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.445626886, 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.337708+0000", 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.337708+0000", 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.337708+0000", 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5200000000000002e-07 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.337719+0000", 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.027e-05 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.783227+0000", 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44550846599999999 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.783230+0000", 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9229999999999998e-06 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.783259+0000", 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8943999999999999e-05 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.783265+0000", 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9750000000000004e-06 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:25.783335+0000", 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9856000000000003e-05 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:25245 2.c 2:3feb22e3:::benchmark_data_smithi173_55121_object25244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.807906+0000", 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 554.80596326900002, 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32403302699999997, 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 25245 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807906+0000", 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807906+0000", 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807907+0000", 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4799999999999997e-07 2024-09-17T16:20:45.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807926+0000", 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8859e-05 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807936+0000", 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0071999999999999e-05 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807941+0000", 2024-09-17T16:20:45.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1270000000000002e-06 2024-09-17T16:20:45.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.807980+0000", 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8279999999999999e-05 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808053+0000", 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3112000000000001e-05 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-09-17T16:20:45.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808131+0000", 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8548999999999996e-05 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.808765+0000", 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00063386300000000003 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.809542+0000", 2024-09-17T16:20:45.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077723799999999997 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131686+0000", 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.322143974 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131783+0000", 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6248000000000004e-05 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131939+0000", 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015675699999999999 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:25245 2.c e121/95 ondisk, result = 0)", 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:30.809328+0000", 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 554.80454136799995, 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32261866900000002, 2024-09-17T16:20:45.637 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.638 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.638 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:45.638 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.809328+0000", 2024-09-17T16:20:45.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.809328+0000", 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.809329+0000", 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:30.809333+0000", 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7069999999999998e-06 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131494+0000", 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32216098300000001 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131509+0000", 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5313000000000001e-05 2024-09-17T16:20:45.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131662+0000", 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015320000000000001 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131677+0000", 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5063e-05 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:31.131947+0000", 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026969400000000001 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:26904 2.a 2:5574ee04:::benchmark_data_smithi173_55121_object26903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.318265+0000", 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 553.295604303, 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45754640299999999, 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.642 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 26904 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318265+0000", 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318265+0000", 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318266+0000", 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.15e-06 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318283+0000", 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6872e-05 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318293+0000", 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0019e-05 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318298+0000", 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8600000000000001e-06 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318339+0000", 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0398999999999997e-05 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318420+0000", 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1275999999999999e-05 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.318505+0000", 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4759000000000006e-05 2024-09-17T16:20:45.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319324+0000", 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081926600000000003 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319648+0000", 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032444199999999998 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775709+0000", 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45606089100000002 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775767+0000", 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8139e-05 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775812+0000", 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4329999999999997e-05 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.648 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:26904 2.a e121/95 ondisk, result = 0)", 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:11:32.319630+0000", 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 553.29423970300002, 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45618829300000002, 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319630+0000", 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319630+0000", 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319630+0000", 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5999999999999999e-07 2024-09-17T16:20:45.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.319634+0000", 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1300000000000001e-06 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.774978+0000", 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.455344099 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.774982+0000", 2024-09-17T16:20:45.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7330000000000003e-06 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775696+0000", 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00071385899999999996 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775704+0000", 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6930000000000007e-06 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:11:32.775818+0000", 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114319 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:94077 2.16 e160/115 2:6e055a89:::benchmark_data_smithi173_55121_object17588:head v 160'3265, mlcod=160'3262)", 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:50.078811+0000", 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 475.53505888400002, 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28417171099999999, 2024-09-17T16:20:45.653 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.078811+0000", 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.078811+0000", 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.078812+0000", 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.559e-06 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.078821+0000", 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6219999999999998e-06 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.361567+0000", 2024-09-17T16:20:45.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28274618099999999 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.361573+0000", 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2890000000000003e-06 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.361664+0000", 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0236999999999997e-05 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.361685+0000", 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1540999999999999e-05 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.362952+0000", 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012664620000000001 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:50.362982+0000", 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.082e-05 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.657 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.658 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.658 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:150269 2.16 e196/115 2:6d6b4de7:::benchmark_data_smithi173_55121_object73780:head v 196'4962, mlcod=196'4958)", 2024-09-17T16:20:45.658 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:50.589009+0000", 2024-09-17T16:20:45.658 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 415.02486028700002, 2024-09-17T16:20:45.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29237028900000001, 2024-09-17T16:20:45.658 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.658 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.658 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.659 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:50.589009+0000", 2024-09-17T16:20:45.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:50.589009+0000", 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:50.589010+0000", 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3499999999999995e-07 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:50.589015+0000", 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0440000000000003e-06 2024-09-17T16:20:45.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:50.878735+0000", 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28972030100000001 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:50.878744+0000", 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2179999999999999e-06 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:50.878810+0000", 2024-09-17T16:20:45.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6029000000000001e-05 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:50.878827+0000", 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7467999999999999e-05 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:50.881347+0000", 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025200909999999999 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:50.881380+0000", 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2403000000000001e-05 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:152954 2.a 2:508c7837:::benchmark_data_smithi173_55121_object76465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:53.845194+0000", 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 411.76867569799998, 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61812484599999995, 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:45.663 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 152954 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.845194+0000", 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.845194+0000", 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.845195+0000", 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9900000000000009e-07 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.845200+0000", 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0919999999999997e-06 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.845212+0000", 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1802e-05 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.845219+0000", 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6629999999999993e-06 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.849012+0000", 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037925480000000002 2024-09-17T16:20:45.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.849135+0000", 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122798 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.849265+0000", 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000130569 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.853422+0000", 2024-09-17T16:20:45.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041561819999999996 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.200549+0000", 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34712759999999998 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.463256+0000", 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26270647600000002 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.463293+0000", 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7014000000000002e-05 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.463319+0000", 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6103000000000001e-05 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.669 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:152966 2.c 2:3484f1c7:::benchmark_data_smithi173_55121_object76477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:53.855816+0000", 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 411.758053991, 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58945345500000002, 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 152966 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.855816+0000", 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.855816+0000", 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.855817+0000", 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3149999999999999e-06 2024-09-17T16:20:45.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.855821+0000", 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9910000000000004e-06 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.855831+0000", 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8309999999999996e-06 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.855836+0000", 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5940000000000001e-06 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.855878+0000", 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1588999999999998e-05 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.855966+0000", 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8529999999999994e-05 2024-09-17T16:20:45.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.856032+0000", 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5141999999999994e-05 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.859453+0000", 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003421791 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.860315+0000", 2024-09-17T16:20:45.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086124599999999997 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.445197+0000", 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58488241399999996 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.445237+0000", 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9805999999999998e-05 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.445269+0000", 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2206000000000003e-05 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:152954 2.a e199/95 ondisk, result = 0)", 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:53.861520+0000", 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 411.75235009099998, 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60180385300000006, 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:45.676 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.861520+0000", 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.861520+0000", 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.861520+0000", 2024-09-17T16:20:45.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:53.861525+0000", 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4549999999999997e-06 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.463189+0000", 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60166468699999998 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.463195+0000", 2024-09-17T16:20:45.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3319999999999999e-06 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.463245+0000", 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0439000000000003e-05 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.463252+0000", 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6490000000000002e-06 2024-09-17T16:20:45.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:54.463323+0000", 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1581999999999993e-05 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:7343 3.0 3:04dbed09:::benchmark_data_smithi173_66272_object7342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:14:20.401942+0000", 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 385.211927398, 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.171982910000001, 2024-09-17T16:20:45.681 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 7343 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.401942+0000", 2024-09-17T16:20:45.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.401942+0000", 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.401944+0000", 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8729999999999999e-06 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.401985+0000", 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1192999999999997e-05 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.401994+0000", 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9930000000000001e-06 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.401999+0000", 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0440000000000003e-06 2024-09-17T16:20:45.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.402038+0000", 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8991e-05 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.402120+0000", 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1462e-05 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,9", 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.402205+0000", 2024-09-17T16:20:45.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5600999999999996e-05 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.402706+0000", 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000500392 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:20.402966+0000", 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025989200000000001 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:34.573849+0000", 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.170883460000001 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:34.573882+0000", 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2381e-05 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:14:34.573925+0000", 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3628000000000003e-05 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.687 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124280 3.0 3.53481990 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.935442+0000", 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 236.67842800099999, 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41915761200000001, 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124280 2024-09-17T16:20:45.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935442+0000", 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935442+0000", 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935443+0000", 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4270000000000001e-06 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935448+0000", 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2820000000000004e-06 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935460+0000", 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2091e-05 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935467+0000", 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4489999999999996e-06 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935529+0000", 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1877000000000004e-05 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935541+0000", 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2145e-05 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.354580+0000", 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41903870599999998 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.354599+0000", 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9635000000000001e-05 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.692 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25036.0:135118 3.e e281/273 3:7ce4ae2e:::benchmark_data_smithi173_66272_object63162:head v 281'8595, mlcod=281'8592)", 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:07.271192+0000", 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 218.34267765000001, 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34001416000000001, 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.271192+0000", 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.271192+0000", 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.271193+0000", 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6499999999999999e-07 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.271197+0000", 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6029999999999998e-06 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.607697+0000", 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33649947400000002 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.607701+0000", 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7400000000000004e-06 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.607746+0000", 2024-09-17T16:20:45.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4199999999999997e-05 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.607768+0000", 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2829000000000001e-05 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.611170+0000", 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034019469999999998 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.611206+0000", 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5701999999999998e-05 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(3.0 281/273 [PushOp(3:09b7de72:::benchmark_data_smithi173_66272_object64042:head, version: 245'4009, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:09b7de72:::benchmark_data_smithi173_66272_object64042:head@245'4009, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:07.277690+0000", 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 218.33617912700001, 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30499425000000002, 2024-09-17T16:20:45.697 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.277690+0000", 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.277690+0000", 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.277691+0000", 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3899999999999999e-07 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.277805+0000", 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114574 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.582375+0000", 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.304569592 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.582392+0000", 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6841999999999999e-05 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.582482+0000", 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0155999999999995e-05 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.582496+0000", 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4231e-05 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.582685+0000", 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018871599999999999 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.701 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e284,272 3:86b32a61:::benchmark_data_smithi173_66272_object3793:head,254'4797)", 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:09.831735+0000", 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 215.78213487299999, 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43025157200000003, 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:09.831735+0000", 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:09.831735+0000", 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:09.831736+0000", 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.581e-06 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:09.831743+0000", 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4030000000000004e-06 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:10.261847+0000", 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43010377999999999 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:10.261853+0000", 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.934e-06 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:10.261917+0000", 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4029999999999995e-05 2024-09-17T16:20:45.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:10.261940+0000", 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2642000000000001e-05 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:10.261986+0000", 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6202000000000003e-05 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:143877 3.0 3:010a2738:::benchmark_data_smithi173_66272_object71921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.450695+0000", 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 205.16317501699999, 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47867361000000003, 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 143877 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450695+0000", 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450695+0000", 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450696+0000", 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3540000000000001e-06 2024-09-17T16:20:45.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450701+0000", 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7269999999999998e-06 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450712+0000", 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1715999999999999e-05 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450720+0000", 2024-09-17T16:20:45.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5349999999999999e-06 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450766+0000", 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6044999999999997e-05 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450855+0000", 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9117999999999996e-05 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,9", 2024-09-17T16:20:45.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450938+0000", 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3379000000000002e-05 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.454980+0000", 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040412570000000004 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.455817+0000", 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083735500000000002 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.929284+0000", 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.473467058 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.929322+0000", 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7676000000000003e-05 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.929368+0000", 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6390000000000001e-05 2024-09-17T16:20:45.711 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25036.0:143908 3.e e290/273 3:70cae513:::benchmark_data_smithi173_66272_object71952:head v 290'9136, mlcod=290'9133)", 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.489614+0000", 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 205.12425566600001, 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35349719099999999, 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.489614+0000", 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.489614+0000", 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.489615+0000", 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0419999999999999e-06 2024-09-17T16:20:45.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.489620+0000", 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1490000000000002e-06 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.839741+0000", 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.350120926 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.839756+0000", 2024-09-17T16:20:45.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4943000000000001e-05 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.839921+0000", 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000165327 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.839942+0000", 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0633e-05 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.843071+0000", 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003129246 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.843111+0000", 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9925000000000001e-05 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e290,272 3:81df571b:::benchmark_data_smithi173_66272_object17841:head,261'5699)", 2024-09-17T16:20:45.716 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.667457+0000", 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 204.94641302400001, 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46443020699999998, 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.667457+0000", 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.667457+0000", 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.667457+0000", 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.82e-07 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.667461+0000", 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8600000000000003e-06 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.131740+0000", 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46427869199999999 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.131748+0000", 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3319999999999992e-06 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.131816+0000", 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7775999999999996e-05 2024-09-17T16:20:45.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.131826+0000", 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0421000000000001e-05 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.131887+0000", 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0544000000000002e-05 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:102161 4.9 e349/306 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head v 349'6286, mlcod=349'6284)", 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:19:11.509193+0000", 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 94.104677100999993, 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90182478799999999, 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:11.509193+0000", 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:11.509193+0000", 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:11.509193+0000", 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4300000000000002e-07 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:11.509286+0000", 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2286000000000004e-05 2024-09-17T16:20:45.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.408350+0000", 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89906420200000003 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.408359+0000", 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5310000000000007e-06 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.408445+0000", 2024-09-17T16:20:45.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6104000000000001e-05 2024-09-17T16:20:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:20:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.408464+0000", 2024-09-17T16:20:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8043e-05 2024-09-17T16:20:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:20:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.410981+0000", 2024-09-17T16:20:45.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002517278 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.411017+0000", 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6501000000000002e-05 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:20:45.725 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:20:45.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:45.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:45 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:45.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:45 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/699817872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:20:45.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:45 smithi017 ceph-mon[27571]: osdmap e385: 12 total, 12 up, 11 in 2024-09-17T16:20:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:45.760 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:45.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:45 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:45 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/699817872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:20:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:45 smithi071 ceph-mon[30002]: osdmap e385: 12 total, 12 up, 11 in 2024-09-17T16:20:45.862 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:45 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:45 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/699817872' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:20:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:45 smithi173 ceph-mon[30530]: osdmap e385: 12 total, 12 up, 11 in 2024-09-17T16:20:45.896 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:45.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:45.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:46.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:46.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:46.135 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:46.173 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:46.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:46.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:46.205 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:20:46.205 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:20:46.235 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:20:46.235 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:20:46.263 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:20:46.263 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:20:46.276 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:46.292 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:20:46.292 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:20:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:46.310 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:46.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:46.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:20:46.322 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:20:46.356 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:20:46.356 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:20:46.387 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:20:46.387 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:20:46.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:46.425 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:20:46.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:20:46.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:46.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:46.453 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:20:46.453 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:20:46.485 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:20:46.485 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:20:46.514 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:20:46.514 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:20:46.548 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:20:46.548 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:20:46.549 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:46 smithi017 ceph-mon[27571]: pgmap v883: 17 pgs: 5 active+remapped+backfilling, 3 peering, 3 active+remapped+backfill_toofull, 6 active+clean; 1.6 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 14544/78675 objects misplaced (18.486%); 15 MiB/s, 235 objects/s recovering 2024-09-17T16:20:46.549 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1021407270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:46.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:46.579 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:46.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:46.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:46.715 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:46.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:46.750 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:46 smithi173 ceph-mon[30530]: pgmap v883: 17 pgs: 5 active+remapped+backfilling, 3 peering, 3 active+remapped+backfill_toofull, 6 active+clean; 1.6 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 14544/78675 objects misplaced (18.486%); 15 MiB/s, 235 objects/s recovering 2024-09-17T16:20:46.750 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1021407270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:46.751 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:20:46.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:46 smithi071 ceph-mon[30002]: pgmap v883: 17 pgs: 5 active+remapped+backfilling, 3 peering, 3 active+remapped+backfill_toofull, 6 active+clean; 1.6 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 14544/78675 objects misplaced (18.486%); 15 MiB/s, 235 objects/s recovering 2024-09-17T16:20:46.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1021407270' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:46.817 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:46.850 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:46.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:46.952 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:46.980 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:46.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:47.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:47.126 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:47.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:47.170 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:47.228 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:47.257 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:47.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:47.359 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:47.394 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:47.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:47.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:20:47.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:47.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:47.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:47 smithi071 ceph-mon[30002]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:20:47.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:47.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:47.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:47.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:47 smithi017 ceph-mon[27571]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:20:47.754 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:47.767 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:47.767 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:47.791 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:47.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:47.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:47.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:47 smithi173 ceph-mon[30530]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:20:47.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:47.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:48.024 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:48.056 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:48.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:48.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:48.187 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:48.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:48.290 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:48.325 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:48.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:48.427 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:48.463 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:48.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:48.510 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:20:48.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:48.609 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:48.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:48.710 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:48.759 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:48.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:48 smithi071 ceph-mon[30002]: pgmap v884: 17 pgs: 6 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 6 active+clean; 1.5 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 62 MiB/s wr, 2.12k op/s; 19238/76020 objects misplaced (25.306%); 25 MiB/s, 402 objects/s recovering 2024-09-17T16:20:48.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:48.890 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:48.893 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:48.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:48.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:48 smithi173 ceph-mon[30530]: pgmap v884: 17 pgs: 6 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 6 active+clean; 1.5 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 62 MiB/s wr, 2.12k op/s; 19238/76020 objects misplaced (25.306%); 25 MiB/s, 402 objects/s recovering 2024-09-17T16:20:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:48 smithi017 ceph-mon[27571]: pgmap v884: 17 pgs: 6 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 6 active+clean; 1.5 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 62 MiB/s wr, 2.12k op/s; 19238/76020 objects misplaced (25.306%); 25 MiB/s, 402 objects/s recovering 2024-09-17T16:20:48.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:49.025 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:49.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:49.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:49.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:49.161 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:49.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:49.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:49.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:49.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:49.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:49.396 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:49.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:49.430 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:49.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:49.430 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:20:49.444 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:20:49.444 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:20:49.445 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-17T16:20:49.445 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-17T16:20:49.449 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:49.449 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:49.520 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:20:49.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:49.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:49.559 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:49 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:20:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:49 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:20:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:49 smithi071 ceph-mon[30002]: osdmap e386: 12 total, 12 up, 11 in 2024-09-17T16:20:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/844838568' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:49.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:49.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:49.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:49.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:49 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:20:49.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:49 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:20:49.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:49 smithi017 ceph-mon[27571]: osdmap e386: 12 total, 12 up, 11 in 2024-09-17T16:20:49.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/844838568' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:49.785 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:49.808 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:49.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:49 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:20:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:49 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:20:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:49 smithi173 ceph-mon[30530]: osdmap e386: 12 total, 12 up, 11 in 2024-09-17T16:20:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/844838568' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:20:49.910 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:49.936 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:49.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:50.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:50.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:50.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:50.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:20:50.161 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:50.185 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:50.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:50.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:50.287 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:50.310 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:50.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:50.394 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:20:50.396 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:50.411 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:50.434 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:50.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:50.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:50.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:50.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:50.662 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:50.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:50 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:20:50.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/844838568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:50.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:50 smithi017 ceph-mon[27571]: osdmap e387: 12 total, 12 up, 11 in 2024-09-17T16:20:50.682 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:50 smithi017 ceph-mon[27571]: pgmap v887: 17 pgs: 6 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 6 active+clean; 1.5 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 66 MiB/s wr, 2.39k op/s; 17874/72846 objects misplaced (24.537%); 34 MiB/s, 551 objects/s recovering 2024-09-17T16:20:50.704 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:50.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:50.802 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:50 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:20:50.803 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/844838568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:50.803 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:50 smithi071 ceph-mon[30002]: osdmap e387: 12 total, 12 up, 11 in 2024-09-17T16:20:50.803 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:50 smithi071 ceph-mon[30002]: pgmap v887: 17 pgs: 6 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 6 active+clean; 1.5 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 66 MiB/s wr, 2.39k op/s; 17874/72846 objects misplaced (24.537%); 34 MiB/s, 551 objects/s recovering 2024-09-17T16:20:50.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:50.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:50.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:50 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:20:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/844838568' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:20:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:50 smithi173 ceph-mon[30530]: osdmap e387: 12 total, 12 up, 11 in 2024-09-17T16:20:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:50 smithi173 ceph-mon[30530]: pgmap v887: 17 pgs: 6 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 6 active+clean; 1.5 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 66 MiB/s wr, 2.39k op/s; 17874/72846 objects misplaced (24.537%); 34 MiB/s, 551 objects/s recovering 2024-09-17T16:20:50.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:50.965 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:20:50.966 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:20:50.971 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:50.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:50.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:51.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:51.104 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:51.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:51.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:51.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:51.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:51.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:51.369 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:51.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:51.471 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:51.504 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:51.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:51.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:51.635 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:51.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:51.713 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:51 smithi017 ceph-mon[27571]: osdmap e388: 12 total, 12 up, 11 in 2024-09-17T16:20:51.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:51.739 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:51.752 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-17T16:20:51.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:51.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:51 smithi071 ceph-mon[30002]: osdmap e388: 12 total, 12 up, 11 in 2024-09-17T16:20:51.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:51 smithi173 ceph-mon[30530]: osdmap e388: 12 total, 12 up, 11 in 2024-09-17T16:20:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:51.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:51.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:52.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:52.026 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:52.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:52.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:52.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:52.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:52.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:52.312 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:52.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:52.354 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:20:52.355 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:20:52.363 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:52.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:20:52.386 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:20:52.395 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:20:52.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:52.428 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:20:52.428 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:20:52.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:52.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:52.456 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:20:52.457 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:20:52.494 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:20:52.495 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:20:52.527 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:20:52.527 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:20:52.546 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:52.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:20:52.561 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:20:52.562 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:52 smithi017 ceph-mon[27571]: 4.6 scrub starts 2024-09-17T16:20:52.562 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:52 smithi017 ceph-mon[27571]: 4.6 scrub ok 2024-09-17T16:20:52.562 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:52 smithi017 ceph-mon[27571]: pgmap v889: 17 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 7 active+clean; 1.4 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 13104/66780 objects misplaced (19.623%); 51 MiB/s, 811 objects/s recovering 2024-09-17T16:20:52.562 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4019134947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:52.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:52.586 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:52.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:52.605 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:20:52.605 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:20:52.617 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:20:52.640 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:20:52.640 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:20:52.679 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:20:52.680 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:20:52.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:52.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:52.715 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:20:52.716 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:20:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:52.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:52 smithi071 ceph-mon[30002]: 4.6 scrub starts 2024-09-17T16:20:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:52 smithi071 ceph-mon[30002]: 4.6 scrub ok 2024-09-17T16:20:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:52 smithi071 ceph-mon[30002]: pgmap v889: 17 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 7 active+clean; 1.4 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 13104/66780 objects misplaced (19.623%); 51 MiB/s, 811 objects/s recovering 2024-09-17T16:20:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4019134947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:52.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:52.852 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:52.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:52 smithi173 ceph-mon[30530]: 4.6 scrub starts 2024-09-17T16:20:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:52 smithi173 ceph-mon[30530]: 4.6 scrub ok 2024-09-17T16:20:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:52 smithi173 ceph-mon[30530]: pgmap v889: 17 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 7 active+clean; 1.4 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 13104/66780 objects misplaced (19.623%); 51 MiB/s, 811 objects/s recovering 2024-09-17T16:20:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4019134947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:20:52.954 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:52.969 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:20:52.969 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:20:52.984 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:52.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:53.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:53.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:53.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:53.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:53.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:53.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:53.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:53.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:53.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:53.380 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:53.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:53.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:53.516 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:53.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:53.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:53.547 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:53.547 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:53.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:53.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:53.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:53.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:53.769 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:53.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:53.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:53.871 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:53.893 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:53.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:53.995 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:54.019 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:54.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:54.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:54.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:54.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:54.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:54.271 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:54.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:54.372 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:20:54.395 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:54.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:20:54.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:54.531 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:54.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:54.632 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:54.665 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:54.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:54.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:54.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:54.799 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:54.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:54 smithi071 ceph-mon[30002]: osdmap e389: 12 total, 12 up, 11 in 2024-09-17T16:20:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:54 smithi071 ceph-mon[30002]: pgmap v891: 17 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 7 active+clean; 1.3 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 570 op/s; 12913/66207 objects misplaced (19.504%); 42 MiB/s, 666 objects/s recovering 2024-09-17T16:20:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:54 smithi071 ceph-mon[30002]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:20:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:54 smithi071 ceph-mon[30002]: osdmap e390: 12 total, 12 up, 11 in 2024-09-17T16:20:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:54 smithi173 ceph-mon[30530]: osdmap e389: 12 total, 12 up, 11 in 2024-09-17T16:20:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:54 smithi173 ceph-mon[30530]: pgmap v891: 17 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 7 active+clean; 1.3 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 570 op/s; 12913/66207 objects misplaced (19.504%); 42 MiB/s, 666 objects/s recovering 2024-09-17T16:20:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:54 smithi173 ceph-mon[30530]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:20:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:54 smithi173 ceph-mon[30530]: osdmap e390: 12 total, 12 up, 11 in 2024-09-17T16:20:54.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:54.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:54.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:54 smithi017 ceph-mon[27571]: osdmap e389: 12 total, 12 up, 11 in 2024-09-17T16:20:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:54 smithi017 ceph-mon[27571]: pgmap v891: 17 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 7 active+clean; 1.3 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 570 op/s; 12913/66207 objects misplaced (19.504%); 42 MiB/s, 666 objects/s recovering 2024-09-17T16:20:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:54 smithi017 ceph-mon[27571]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:20:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:54 smithi017 ceph-mon[27571]: osdmap e390: 12 total, 12 up, 11 in 2024-09-17T16:20:55.033 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:55.065 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:55.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:55.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:55.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:55.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:55.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:55.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:55.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:55.436 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:55.463 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:55.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:55.486 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-17T16:20:55.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:55.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:55.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:55.693 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:55.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:55.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:55.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:55.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:55 smithi017 ceph-mon[27571]: 4.4 scrub starts 2024-09-17T16:20:55.737 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:55 smithi017 ceph-mon[27571]: 4.4 scrub ok 2024-09-17T16:20:55.768 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:20:55.769 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:20:55.769 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:20:55.769 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:20:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:55 smithi071 ceph-mon[30002]: 4.4 scrub starts 2024-09-17T16:20:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:55 smithi071 ceph-mon[30002]: 4.4 scrub ok 2024-09-17T16:20:55.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:55.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:55.846 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:55.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:55 smithi173 ceph-mon[30530]: 4.4 scrub starts 2024-09-17T16:20:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:55 smithi173 ceph-mon[30530]: 4.4 scrub ok 2024-09-17T16:20:55.947 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:55.969 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:55.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:56.018 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:20:56.070 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:56.091 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:56.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:56.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:56.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:56.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:56.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:56.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:56.339 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:56.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:56.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:56.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:56.463 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:56.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:56.565 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:56.587 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:56.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:56.690 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:56.723 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:56.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:56.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:56 smithi017 ceph-mon[27571]: osdmap e391: 12 total, 12 up, 11 in 2024-09-17T16:20:56.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:56 smithi017 ceph-mon[27571]: pgmap v894: 17 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 8 active+clean; 1.2 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 748 op/s; 9836/60876 objects misplaced (16.157%); 54 MiB/s, 869 objects/s recovering 2024-09-17T16:20:56.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:56 smithi071 ceph-mon[30002]: osdmap e391: 12 total, 12 up, 11 in 2024-09-17T16:20:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:56 smithi071 ceph-mon[30002]: pgmap v894: 17 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 8 active+clean; 1.2 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 748 op/s; 9836/60876 objects misplaced (16.157%); 54 MiB/s, 869 objects/s recovering 2024-09-17T16:20:56.822 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:20:56.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:56.860 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:56.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:56.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:56 smithi173 ceph-mon[30530]: osdmap e391: 12 total, 12 up, 11 in 2024-09-17T16:20:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:56 smithi173 ceph-mon[30530]: pgmap v894: 17 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 8 active+clean; 1.2 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 748 op/s; 9836/60876 objects misplaced (16.157%); 54 MiB/s, 869 objects/s recovering 2024-09-17T16:20:56.962 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:57.002 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:57.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:57.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:57.105 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:57.138 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:57.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:57.168 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:57.235 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:57.240 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:20:57.249 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:20:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:57.286 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:57.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:20:57.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:57.415 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:57.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:57.517 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:20:57.544 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:57.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:20:57.569 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:20:57.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:57.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:57.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:57.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:57.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:20:57.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:57.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:20:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:57 smithi071 ceph-mon[30002]: 4.0 scrub starts 2024-09-17T16:20:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:57 smithi071 ceph-mon[30002]: 4.0 scrub ok 2024-09-17T16:20:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:57 smithi071 ceph-mon[30002]: osdmap e392: 12 total, 12 up, 11 in 2024-09-17T16:20:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2363751692' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:57 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:57.815 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:57 smithi017 ceph-mon[27571]: 4.0 scrub starts 2024-09-17T16:20:57.815 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:57 smithi017 ceph-mon[27571]: 4.0 scrub ok 2024-09-17T16:20:57.815 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:57 smithi017 ceph-mon[27571]: osdmap e392: 12 total, 12 up, 11 in 2024-09-17T16:20:57.816 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2363751692' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:57.816 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:57 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:57 smithi173 ceph-mon[30530]: 4.0 scrub starts 2024-09-17T16:20:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:57 smithi173 ceph-mon[30530]: 4.0 scrub ok 2024-09-17T16:20:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:57 smithi173 ceph-mon[30530]: osdmap e392: 12 total, 12 up, 11 in 2024-09-17T16:20:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2363751692' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:57 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:20:57.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:20:57.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:57.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:20:58.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:58.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:58.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:58.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:20:58.177 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:58.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:20:58.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:58.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:58.302 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:58.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:58.427 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:58.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:58.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:58.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:20:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:58.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:58.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:20:58.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:20:58.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:58.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:20:58.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:58.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:58.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:20:58.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:20:58.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:58 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:58.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:58 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:20:58.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:58 smithi071 ceph-mon[30002]: osdmap e393: 12 total, 12 up, 11 in 2024-09-17T16:20:58.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:58 smithi071 ceph-mon[30002]: pgmap v897: 17 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 9 active+clean; 1.2 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 639 op/s; 9063/58515 objects misplaced (15.488%); 34 MiB/s, 546 objects/s recovering 2024-09-17T16:20:58.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:58.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:20:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:20:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:58 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:58 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:20:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:58 smithi173 ceph-mon[30530]: osdmap e393: 12 total, 12 up, 11 in 2024-09-17T16:20:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:58 smithi173 ceph-mon[30530]: pgmap v897: 17 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 9 active+clean; 1.2 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 639 op/s; 9063/58515 objects misplaced (15.488%); 34 MiB/s, 546 objects/s recovering 2024-09-17T16:20:58.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:58.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:20:58.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:20:58.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:58 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:20:58.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:58 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:20:58.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:58 smithi017 ceph-mon[27571]: osdmap e393: 12 total, 12 up, 11 in 2024-09-17T16:20:58.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:58 smithi017 ceph-mon[27571]: pgmap v897: 17 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 9 active+clean; 1.2 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 639 op/s; 9063/58515 objects misplaced (15.488%); 34 MiB/s, 546 objects/s recovering 2024-09-17T16:20:58.948 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:58.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:59.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:59.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:59.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:59.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:59.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:59.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:59.317 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:20:59.341 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:20:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:59.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:20:59.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:59.464 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:59.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:59.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:20:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:59.597 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:59.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:20:59.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:20:59.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:20:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:59.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:20:59.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:59.846 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:59.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:20:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:59 smithi173 ceph-mon[30530]: osdmap e394: 12 total, 12 up, 11 in 2024-09-17T16:20:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:20:59 smithi173 ceph-mon[30530]: osdmap e395: 12 total, 12 up, 11 in 2024-09-17T16:20:59.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:59 smithi017 ceph-mon[27571]: osdmap e394: 12 total, 12 up, 11 in 2024-09-17T16:20:59.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:20:59 smithi017 ceph-mon[27571]: osdmap e395: 12 total, 12 up, 11 in 2024-09-17T16:20:59.948 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:20:59.969 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:20:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:20:59.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:00.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:59 smithi071 ceph-mon[30002]: osdmap e394: 12 total, 12 up, 11 in 2024-09-17T16:21:00.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:20:59 smithi071 ceph-mon[30002]: osdmap e395: 12 total, 12 up, 11 in 2024-09-17T16:21:00.071 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:00.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:00.094 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:00.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:00.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:00.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:00.217 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:00.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:00.319 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:00.341 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:00.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:00.443 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:00.466 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:00.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:00.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:00.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:00.602 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:00.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:00.613 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:21:00.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:00.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:00.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:00.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:00.857 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:00.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:00.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:00 smithi173 ceph-mon[30530]: pgmap v900: 17 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 9 active+clean; 1.1 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 411 op/s; 8713/55926 objects misplaced (15.580%); 4.5 MiB/s, 71 objects/s recovering 2024-09-17T16:21:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:00 smithi173 ceph-mon[30530]: osdmap e396: 12 total, 12 up, 11 in 2024-09-17T16:21:00.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:00.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:00.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:00 smithi017 ceph-mon[27571]: pgmap v900: 17 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 9 active+clean; 1.1 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 411 op/s; 8713/55926 objects misplaced (15.580%); 4.5 MiB/s, 71 objects/s recovering 2024-09-17T16:21:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:00 smithi017 ceph-mon[27571]: osdmap e396: 12 total, 12 up, 11 in 2024-09-17T16:21:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:00 smithi071 ceph-mon[30002]: pgmap v900: 17 pgs: 1 active+clean+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 9 active+clean; 1.1 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 411 op/s; 8713/55926 objects misplaced (15.580%); 4.5 MiB/s, 71 objects/s recovering 2024-09-17T16:21:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:00 smithi071 ceph-mon[30002]: osdmap e396: 12 total, 12 up, 11 in 2024-09-17T16:21:01.090 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:01.115 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:01.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:01.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:01.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:01.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:01.347 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-17T16:21:01.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:01.384 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:01.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:01.486 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:01.519 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:01.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:01.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:01.600 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:21:01.600 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:21:01.601 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:21:01.601 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:21:01.601 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:21:01.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:01.650 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:01.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:01.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:01.773 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:01.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:01.834 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:21:01.834 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:21:01.834 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:01 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:01.834 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:01 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:01.861 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:21:01.861 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:21:01.875 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:01.883 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:21:01.884 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:21:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:01 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:01 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:01.900 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:01.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:01.910 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:21:01.910 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:21:01.938 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:21:01.938 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:21:01.969 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:21:01.970 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:21:01.998 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:21:01.999 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:21:02.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:02.024 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:21:02.025 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:21:02.028 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:02.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:02.059 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:21:02.059 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:21:02.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:01 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:02.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:01 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:02.092 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:21:02.092 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:21:02.116 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:21:02.117 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:21:02.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:02.146 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:21:02.146 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:21:02.159 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:02.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:02.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:02.285 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:02.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:02.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:02.413 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:02.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:02.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:02.541 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:02.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:02.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:02.626 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:21:02.643 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:02.696 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:02.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2170837460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:02 smithi017 ceph-mon[27571]: from='client.16425 172.21.15.17:0/2170837460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:02.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:02 smithi017 ceph-mon[27571]: pgmap v902: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 1.0 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 7243/50661 objects misplaced (14.297%); 5.1 MiB/s, 81 objects/s recovering 2024-09-17T16:21:02.798 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:02.825 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:02.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2170837460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:02 smithi173 ceph-mon[30530]: from='client.16425 172.21.15.17:0/2170837460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:02 smithi173 ceph-mon[30530]: pgmap v902: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 1.0 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 7243/50661 objects misplaced (14.297%); 5.1 MiB/s, 81 objects/s recovering 2024-09-17T16:21:02.927 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:02.955 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:02.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:03.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:03.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2170837460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:02 smithi071 ceph-mon[30002]: from='client.16425 172.21.15.17:0/2170837460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:02 smithi071 ceph-mon[30002]: pgmap v902: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 1.0 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 7243/50661 objects misplaced (14.297%); 5.1 MiB/s, 81 objects/s recovering 2024-09-17T16:21:03.088 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:03.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:03.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:03.218 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:03.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:03.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:03.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:03.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:03.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:03.478 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:03.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:03.579 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:03.609 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:03.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:03.711 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:03.737 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:03.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:03.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:03.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:03.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:03.866 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:03.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:03.995 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:03.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:04.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:04.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:04.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:04.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:04.262 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:04.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:04.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:04.364 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:04.390 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:04.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:04.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:04.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:04.517 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:04.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:04.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:04.645 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:04.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:04.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:04.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:04.781 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:04.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:04.882 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:04.907 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:04.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:04.960 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:05.008 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:05.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:05.044 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:05.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:05.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:05.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:05.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:04 smithi017 ceph-mon[27571]: pgmap v903: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 1.0 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 7067/50133 objects misplaced (14.097%); 3.7 MiB/s, 58 objects/s recovering 2024-09-17T16:21:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:04 smithi017 ceph-mon[27571]: osdmap e397: 12 total, 12 up, 11 in 2024-09-17T16:21:05.271 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:05.295 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:05.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:05.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:05.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:04 smithi071 ceph-mon[30002]: pgmap v903: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 1.0 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 7067/50133 objects misplaced (14.097%); 3.7 MiB/s, 58 objects/s recovering 2024-09-17T16:21:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:04 smithi071 ceph-mon[30002]: osdmap e397: 12 total, 12 up, 11 in 2024-09-17T16:21:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:04 smithi173 ceph-mon[30530]: pgmap v903: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 1.0 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 7067/50133 objects misplaced (14.097%); 3.7 MiB/s, 58 objects/s recovering 2024-09-17T16:21:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:04 smithi173 ceph-mon[30530]: osdmap e397: 12 total, 12 up, 11 in 2024-09-17T16:21:05.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:05.422 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:05.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:05.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:05.559 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:05.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:05.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:05.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:05.694 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:05.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:05.782 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:05.795 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:05.828 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:05.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:05.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:05.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:05.961 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:05.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:06.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:06.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:06.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:06.188 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:06.214 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:06.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:06.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:06.316 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:06.345 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:06.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:06.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:06.474 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:06.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:06.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:06.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:06.603 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:06.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:06.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:06.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:06.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:06.832 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:21:06.832 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:21:06.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:06.867 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:06.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:06.933 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:21:06.969 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:06.999 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:07.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:07.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:21:07.085 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:06 smithi017 ceph-mon[27571]: pgmap v905: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 893 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 782 op/s; 6393/42840 objects misplaced (14.923%); 3.4 MiB/s, 54 objects/s recovering 2024-09-17T16:21:07.085 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3562295533' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:07.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:07.128 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:07.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:07.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:07.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:07.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:07.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:06 smithi071 ceph-mon[30002]: pgmap v905: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 893 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 782 op/s; 6393/42840 objects misplaced (14.923%); 3.4 MiB/s, 54 objects/s recovering 2024-09-17T16:21:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3562295533' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:07.356 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:07.384 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:07.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:07.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:06 smithi173 ceph-mon[30530]: pgmap v905: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 893 MiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 782 op/s; 6393/42840 objects misplaced (14.923%); 3.4 MiB/s, 54 objects/s recovering 2024-09-17T16:21:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3562295533' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:07.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:07.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:07.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:07.615 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:07.641 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:07.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:07.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:07.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:07.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:07.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:07.900 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:07.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:08.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:08.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:08.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:08.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:08.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:07 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:08.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3562295533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:08.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:07 smithi017 ceph-mon[27571]: osdmap e398: 12 total, 12 up, 11 in 2024-09-17T16:21:08.158 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:08.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:08.260 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:08.290 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:08.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:07 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3562295533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:07 smithi071 ceph-mon[30002]: osdmap e398: 12 total, 12 up, 11 in 2024-09-17T16:21:08.363 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-17T16:21:08.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:07 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3562295533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:07 smithi173 ceph-mon[30530]: osdmap e398: 12 total, 12 up, 11 in 2024-09-17T16:21:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:08.420 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:08.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:08.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:08.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:08.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:08.651 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:08.684 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:08.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:08.786 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:08.816 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:08.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:08.850 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:21:08.880 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:08.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:08.930 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:08.945 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:08.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:09.047 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:09.082 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:09.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:09.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:08 smithi017 ceph-mon[27571]: pgmap v907: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 819 MiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 5689/39279 objects misplaced (14.484%); 2.9 MiB/s, 47 objects/s recovering 2024-09-17T16:21:09.183 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:09.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:09.236 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:09.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:09.295 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:08 smithi173 ceph-mon[30530]: pgmap v907: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 819 MiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 5689/39279 objects misplaced (14.484%); 2.9 MiB/s, 47 objects/s recovering 2024-09-17T16:21:09.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:08 smithi071 ceph-mon[30002]: pgmap v907: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 819 MiB data, 5.8 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 5689/39279 objects misplaced (14.484%); 2.9 MiB/s, 47 objects/s recovering 2024-09-17T16:21:09.336 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:09.369 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:09.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:09.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:09.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:09.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:09.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:09.647 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:09.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:09.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:09.773 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:09.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:09.875 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:09.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:21:09.901 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:09.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:10.003 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:10.030 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:10.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:10.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:10.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:10.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:10.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:10.283 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:10.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:10.385 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:10.398 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:21:10.398 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:21:10.398 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:21:10.398 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:21:10.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:21:10.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:21:10.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:21:10.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:21:10.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:21:10.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:21:10.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:21:10.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:21:10.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:21:10.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:21:10.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:21:10.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:21:10.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:21:10.400 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:21:10.401 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:21:10.401 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:21:10.401 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:21:10.401 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:21:10.401 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:21:10.401 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:21:10.407 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:10.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:10.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:10.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:10.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:10.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:10.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:10.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:10.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:10.665 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:10.719 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:10 smithi017 ceph-mon[27571]: osdmap e399: 12 total, 12 up, 11 in 2024-09-17T16:21:10.719 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3609834410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:10.719 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:10 smithi017 ceph-mon[27571]: pgmap v909: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 742 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 5176/35607 objects misplaced (14.536%) 2024-09-17T16:21:10.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:10.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:10.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:10.794 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:10 smithi071 ceph-mon[30002]: osdmap e399: 12 total, 12 up, 11 in 2024-09-17T16:21:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3609834410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:10.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:10 smithi071 ceph-mon[30002]: pgmap v909: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 742 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 5176/35607 objects misplaced (14.536%) 2024-09-17T16:21:10.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:10 smithi173 ceph-mon[30530]: osdmap e399: 12 total, 12 up, 11 in 2024-09-17T16:21:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3609834410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:10 smithi173 ceph-mon[30530]: pgmap v909: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 742 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 5176/35607 objects misplaced (14.536%) 2024-09-17T16:21:10.895 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:10.922 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:10.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:11.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:11.057 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:11.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:11.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:11.159 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:11.191 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:11.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:11.197 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:11.292 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:11.321 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:11.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:11.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:11.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:11.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:11.555 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:11.581 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:11.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:11.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:11.691 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:11.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:11 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:11.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:11 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:11.697 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-17T16:21:11.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:11.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:11.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:11 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:11 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:11.839 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:11.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:11.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:11 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:11 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:11.941 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:11.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:11.967 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:11.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:12.069 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:12.093 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:12.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:12.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:12.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:12.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:12.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:12.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:12.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:12.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:12.473 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:12 smithi017 ceph-mon[27571]: osdmap e400: 12 total, 12 up, 11 in 2024-09-17T16:21:12.473 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:12 smithi017 ceph-mon[27571]: pgmap v911: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 563 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 879 op/s; 3894/27003 objects misplaced (14.421%) 2024-09-17T16:21:12.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:12.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:12.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:12.622 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:12.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:12.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:12.707 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:21:12.707 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:21:12.707 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:21:12.708 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:21:12.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:12.753 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:12.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:12 smithi071 ceph-mon[30002]: osdmap e400: 12 total, 12 up, 11 in 2024-09-17T16:21:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:12 smithi071 ceph-mon[30002]: pgmap v911: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 563 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 879 op/s; 3894/27003 objects misplaced (14.421%) 2024-09-17T16:21:12.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:12.887 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:12.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:12 smithi173 ceph-mon[30530]: osdmap e400: 12 total, 12 up, 11 in 2024-09-17T16:21:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:12 smithi173 ceph-mon[30530]: pgmap v911: 17 pgs: 5 active+remapped+backfill_toofull, 12 active+clean; 563 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 879 op/s; 3894/27003 objects misplaced (14.421%) 2024-09-17T16:21:12.989 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:13.016 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:13.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:13.118 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:13.145 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:13.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:13.193 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:21:13.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:13.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:13.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:13.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:13.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:13.405 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:13.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:13.457 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:21:13.475 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3922793341' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:21:13.475 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:13 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:21:13.507 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:13.533 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:13.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:13.542 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:21:13.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:13.659 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:13.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:13.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:13.783 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:13.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3922793341' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:21:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:13 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:21:13.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:13.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3922793341' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:21:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:13 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:21:13.909 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:13.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:14.011 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:14.035 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:14.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:14.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:14.158 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:14.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:14.260 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:14.289 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:14.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:14.303 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:14.364 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:14.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:14.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:14.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:14.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:14.538 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:14.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:14.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:14.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:14.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:14.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:14.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:14 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:14.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:14 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:21:14.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:14 smithi017 ceph-mon[27571]: osdmap e401: 12 total, 12 up, 11 in 2024-09-17T16:21:14.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:14 smithi017 ceph-mon[27571]: pgmap v913: 17 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 12 active+clean; 546 MiB data, 4.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 727 op/s; 3215/26175 objects misplaced (12.283%); 5.5 MiB/s, 88 objects/s recovering 2024-09-17T16:21:14.763 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:14.797 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:14.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:14 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:14 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:21:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:14 smithi071 ceph-mon[30002]: osdmap e401: 12 total, 12 up, 11 in 2024-09-17T16:21:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:14 smithi071 ceph-mon[30002]: pgmap v913: 17 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 12 active+clean; 546 MiB data, 4.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 727 op/s; 3215/26175 objects misplaced (12.283%); 5.5 MiB/s, 88 objects/s recovering 2024-09-17T16:21:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:14 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:14 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:21:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:14 smithi173 ceph-mon[30530]: osdmap e401: 12 total, 12 up, 11 in 2024-09-17T16:21:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:14 smithi173 ceph-mon[30530]: pgmap v913: 17 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 12 active+clean; 546 MiB data, 4.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 727 op/s; 3215/26175 objects misplaced (12.283%); 5.5 MiB/s, 88 objects/s recovering 2024-09-17T16:21:14.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:14.930 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:14.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:15.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:15.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:15.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:15.139 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:21:15.139 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:21:15.139 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:21:15.139 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:21:15.139 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:21:15.162 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:15.193 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:15.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:15.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:15.326 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:15.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:15.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:15.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:15.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:15.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:15.558 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:15.580 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:15.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:15.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:15 smithi017 ceph-mon[27571]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:21:15.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:15 smithi017 ceph-mon[27571]: osdmap e402: 12 total, 12 up, 11 in 2024-09-17T16:21:15.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/102937201' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:15.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:15.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:15.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:15.805 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:15 smithi071 ceph-mon[30002]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:21:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:15 smithi071 ceph-mon[30002]: osdmap e402: 12 total, 12 up, 11 in 2024-09-17T16:21:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/102937201' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:15.828 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:15.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:15.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:15 smithi173 ceph-mon[30530]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:21:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:15 smithi173 ceph-mon[30530]: osdmap e402: 12 total, 12 up, 11 in 2024-09-17T16:21:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/102937201' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:15.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:15.950 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:15.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:16.052 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:16.074 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:16.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:16.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:16.195 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:21:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:16.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:16.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:16.269 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:21:16.269 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:21:16.269 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:21:16.269 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:21:16.269 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:21:16.270 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:21:16.270 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:21:16.270 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:21:16.270 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:21:16.270 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:21:16.270 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:21:16.270 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:21:16.270 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:21:16.270 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:21:16.271 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:21:16.271 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:21:16.271 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:21:16.271 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:21:16.271 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:21:16.271 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:21:16.271 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:21:16.271 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:21:16.271 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:21:16.272 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:21:16.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:16.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:16.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:16.424 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:16.447 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:16.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:16.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:16.550 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:16.581 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:16.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:16.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:16.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:16.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:16.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:16 smithi017 ceph-mon[27571]: 4.3 scrub starts 2024-09-17T16:21:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:16 smithi017 ceph-mon[27571]: 4.3 scrub ok 2024-09-17T16:21:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:16 smithi017 ceph-mon[27571]: osdmap e403: 12 total, 12 up, 11 in 2024-09-17T16:21:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:16 smithi017 ceph-mon[27571]: pgmap v916: 17 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 12 active+clean; 418 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 822 op/s; 2150/20031 objects misplaced (10.733%); 11 MiB/s, 182 objects/s recovering 2024-09-17T16:21:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:16 smithi071 ceph-mon[30002]: 4.3 scrub starts 2024-09-17T16:21:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:16 smithi071 ceph-mon[30002]: 4.3 scrub ok 2024-09-17T16:21:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:16 smithi071 ceph-mon[30002]: osdmap e403: 12 total, 12 up, 11 in 2024-09-17T16:21:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:16 smithi071 ceph-mon[30002]: pgmap v916: 17 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 12 active+clean; 418 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 822 op/s; 2150/20031 objects misplaced (10.733%); 11 MiB/s, 182 objects/s recovering 2024-09-17T16:21:16.822 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:16.849 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:16.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:16 smithi173 ceph-mon[30530]: 4.3 scrub starts 2024-09-17T16:21:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:16 smithi173 ceph-mon[30530]: 4.3 scrub ok 2024-09-17T16:21:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:16 smithi173 ceph-mon[30530]: osdmap e403: 12 total, 12 up, 11 in 2024-09-17T16:21:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:16 smithi173 ceph-mon[30530]: pgmap v916: 17 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 12 active+clean; 418 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 822 op/s; 2150/20031 objects misplaced (10.733%); 11 MiB/s, 182 objects/s recovering 2024-09-17T16:21:16.952 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:16.980 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:16.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:17.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:17.110 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:17.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:17.116 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:21:17.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:17.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:17.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:17.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:17.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:17.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:17.462 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:17 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:17.462 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:17 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:17.463 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:17.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:17.489 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:17.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:17.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:17.613 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:17.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:17.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:17.736 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:17.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:17.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:17 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:17 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:17 smithi017 ceph-mon[27571]: osdmap e404: 12 total, 12 up, 11 in 2024-09-17T16:21:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:17 smithi071 ceph-mon[30002]: osdmap e404: 12 total, 12 up, 11 in 2024-09-17T16:21:17.838 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:17.859 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:17.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:17 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:17 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:17 smithi173 ceph-mon[30530]: osdmap e404: 12 total, 12 up, 11 in 2024-09-17T16:21:17.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:17.981 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:17.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:17.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:18.063 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:18.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:18.104 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:18.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:18.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:18.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:18.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:18.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:18.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:18.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:18.453 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:18.476 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:18.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:18.548 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-17T16:21:18.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:18.619 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:18.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:18.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:18.646 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:18.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:18.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:18 smithi017 ceph-mon[27571]: 4.f scrub starts 2024-09-17T16:21:18.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:18 smithi017 ceph-mon[27571]: 4.f scrub ok 2024-09-17T16:21:18.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:18 smithi017 ceph-mon[27571]: osdmap e405: 12 total, 12 up, 11 in 2024-09-17T16:21:18.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:18 smithi017 ceph-mon[27571]: pgmap v919: 17 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 12 active+clean; 367 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 1316/17589 objects misplaced (7.482%); 11 MiB/s, 182 objects/s recovering 2024-09-17T16:21:18.752 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:18.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:18.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:18 smithi071 ceph-mon[30002]: 4.f scrub starts 2024-09-17T16:21:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:18 smithi071 ceph-mon[30002]: 4.f scrub ok 2024-09-17T16:21:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:18 smithi071 ceph-mon[30002]: osdmap e405: 12 total, 12 up, 11 in 2024-09-17T16:21:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:18 smithi071 ceph-mon[30002]: pgmap v919: 17 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 12 active+clean; 367 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 1316/17589 objects misplaced (7.482%); 11 MiB/s, 182 objects/s recovering 2024-09-17T16:21:18.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:18.884 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:18.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:18.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:18 smithi173 ceph-mon[30530]: 4.f scrub starts 2024-09-17T16:21:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:18 smithi173 ceph-mon[30530]: 4.f scrub ok 2024-09-17T16:21:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:18 smithi173 ceph-mon[30530]: osdmap e405: 12 total, 12 up, 11 in 2024-09-17T16:21:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:18 smithi173 ceph-mon[30530]: pgmap v919: 17 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 12 active+clean; 367 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 1316/17589 objects misplaced (7.482%); 11 MiB/s, 182 objects/s recovering 2024-09-17T16:21:18.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:19.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:19.014 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:19.115 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:19.149 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:19.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:19.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:19.278 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:19.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:19.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:19.406 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:19.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:19.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:19.533 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:19.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:19.596 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:19 smithi017 ceph-mon[27571]: osdmap e406: 12 total, 12 up, 11 in 2024-09-17T16:21:19.596 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:19 smithi017 ceph-mon[27571]: osdmap e407: 12 total, 12 up, 11 in 2024-09-17T16:21:19.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:19.654 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:21:19.654 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:21:19.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:19.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:19.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:19.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:19.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:19.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:19 smithi071 ceph-mon[30002]: osdmap e406: 12 total, 12 up, 11 in 2024-09-17T16:21:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:19 smithi071 ceph-mon[30002]: osdmap e407: 12 total, 12 up, 11 in 2024-09-17T16:21:19.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:21:19.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:19.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:19 smithi173 ceph-mon[30530]: osdmap e406: 12 total, 12 up, 11 in 2024-09-17T16:21:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:19 smithi173 ceph-mon[30530]: osdmap e407: 12 total, 12 up, 11 in 2024-09-17T16:21:19.916 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:19.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:20.018 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:20.059 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:20.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:20.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:20.161 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:20.193 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:20.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:20.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:20.324 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:20.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:20.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:20.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:20.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:20.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:20.481 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:21:20.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:21:20.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:20.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:20.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:20.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1536181084' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:20 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:20 smithi173 ceph-mon[30530]: pgmap v922: 17 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 13 active+clean; 309 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 434 op/s; 1093/14817 objects misplaced (7.377%); 7.5 MiB/s, 120 objects/s recovering 2024-09-17T16:21:20.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:20.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:20.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1536181084' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:20 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:20 smithi017 ceph-mon[27571]: pgmap v922: 17 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 13 active+clean; 309 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 434 op/s; 1093/14817 objects misplaced (7.377%); 7.5 MiB/s, 120 objects/s recovering 2024-09-17T16:21:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1536181084' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:20 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:20 smithi071 ceph-mon[30002]: pgmap v922: 17 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 13 active+clean; 309 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 434 op/s; 1093/14817 objects misplaced (7.377%); 7.5 MiB/s, 120 objects/s recovering 2024-09-17T16:21:20.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:20.849 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:20.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:20.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:20.951 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:20.983 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:20.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:20.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:21.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:21.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:21.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:21.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:21.285 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:21.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:21.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:21.350 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:21.358 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:21.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:21.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:21.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:21.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:21 smithi017 ceph-mon[27571]: 4.7 scrub starts 2024-09-17T16:21:21.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:21 smithi017 ceph-mon[27571]: 4.7 scrub ok 2024-09-17T16:21:21.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:21 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:21:21.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:21 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:21.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:21 smithi017 ceph-mon[27571]: osdmap e408: 12 total, 12 up, 11 in 2024-09-17T16:21:21.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:21.540 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:21.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:21.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:21.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:21.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:21.774 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:21.799 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:21.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:21.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:21 smithi071 ceph-mon[30002]: 4.7 scrub starts 2024-09-17T16:21:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:21 smithi071 ceph-mon[30002]: 4.7 scrub ok 2024-09-17T16:21:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:21 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:21:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:21 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:21 smithi071 ceph-mon[30002]: osdmap e408: 12 total, 12 up, 11 in 2024-09-17T16:21:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:21 smithi173 ceph-mon[30530]: 4.7 scrub starts 2024-09-17T16:21:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:21 smithi173 ceph-mon[30530]: 4.7 scrub ok 2024-09-17T16:21:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:21 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:21:21.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:21 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:21.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:21 smithi173 ceph-mon[30530]: osdmap e408: 12 total, 12 up, 11 in 2024-09-17T16:21:21.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:21.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:21.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:22.027 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:22.049 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:22.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:22.151 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:22.179 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:22.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:22.281 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:22.305 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:22.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:22.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:22.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:22.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:22.512 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:21:22.513 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:21:22.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:22.544 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:21:22.544 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:21:22.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:22.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:22.570 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:21:22.570 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:21:22.601 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:21:22.601 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:21:22.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:22 smithi017 ceph-mon[27571]: 4.8 scrub starts 2024-09-17T16:21:22.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:22 smithi017 ceph-mon[27571]: 4.8 scrub ok 2024-09-17T16:21:22.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:22 smithi017 ceph-mon[27571]: pgmap v924: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 162 MiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 453/7743 objects misplaced (5.850%); 1.7 MiB/s, 27 objects/s recovering 2024-09-17T16:21:22.633 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:21:22.633 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:21:22.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:22.665 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:21:22.666 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:21:22.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:22.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:22.698 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:21:22.698 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:21:22.732 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:21:22.732 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:21:22.765 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:21:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:21:22.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:22.794 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:22.802 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:21:22.802 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:21:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:22 smithi071 ceph-mon[30002]: 4.8 scrub starts 2024-09-17T16:21:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:22 smithi071 ceph-mon[30002]: 4.8 scrub ok 2024-09-17T16:21:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:22 smithi071 ceph-mon[30002]: pgmap v924: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 162 MiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 453/7743 objects misplaced (5.850%); 1.7 MiB/s, 27 objects/s recovering 2024-09-17T16:21:22.818 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:22.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:22.843 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:21:22.843 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:21:22.877 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:21:22.878 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:21:22.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:22 smithi173 ceph-mon[30530]: 4.8 scrub starts 2024-09-17T16:21:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:22 smithi173 ceph-mon[30530]: 4.8 scrub ok 2024-09-17T16:21:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:22 smithi173 ceph-mon[30530]: pgmap v924: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 162 MiB data, 3.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 453/7743 objects misplaced (5.850%); 1.7 MiB/s, 27 objects/s recovering 2024-09-17T16:21:22.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:22.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:22.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:23.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:23.087 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:23.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:23.188 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:23.215 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:23.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:23.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:21:23.317 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:23.367 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:23.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:23.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:23.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:23.494 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:23.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:23.595 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:23.622 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:23.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:23.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:23.733 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2630298059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:23.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:23.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:23.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:23.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2630298059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:23.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:23.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:23.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:23.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2630298059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:23.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:23.992 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:23.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:24.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:24.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:24.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:24.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:24.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:24.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:24.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:24.244 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-09-17T16:21:24.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:24.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:24.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:24.384 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 102172 objects 2024-09-17T16:21:24.384 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :131.849 2024-09-17T16:21:24.465 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:24.487 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:24.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:24.589 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:24.612 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:24.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:24.700 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:21:24.701 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:21:24.701 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:21:24.701 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:21:24.702 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:24.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:24.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:24.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:24 smithi071 ceph-mon[30002]: 4.c scrub starts 2024-09-17T16:21:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:24 smithi071 ceph-mon[30002]: 4.c scrub ok 2024-09-17T16:21:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:24 smithi071 ceph-mon[30002]: pgmap v925: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 147 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 453/7017 objects misplaced (6.456%); 1.2 MiB/s, 20 objects/s recovering 2024-09-17T16:21:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:24 smithi071 ceph-mon[30002]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:21:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:24 smithi071 ceph-mon[30002]: osdmap e409: 12 total, 12 up, 11 in 2024-09-17T16:21:24.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:24.859 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:24.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:24.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:24 smithi173 ceph-mon[30530]: 4.c scrub starts 2024-09-17T16:21:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:24 smithi173 ceph-mon[30530]: 4.c scrub ok 2024-09-17T16:21:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:24 smithi173 ceph-mon[30530]: pgmap v925: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 147 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 453/7017 objects misplaced (6.456%); 1.2 MiB/s, 20 objects/s recovering 2024-09-17T16:21:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:24 smithi173 ceph-mon[30530]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:21:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:24 smithi173 ceph-mon[30530]: osdmap e409: 12 total, 12 up, 11 in 2024-09-17T16:21:24.961 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:24.983 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:24.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:24 smithi017 ceph-mon[27571]: 4.c scrub starts 2024-09-17T16:21:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:24 smithi017 ceph-mon[27571]: 4.c scrub ok 2024-09-17T16:21:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:24 smithi017 ceph-mon[27571]: pgmap v925: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 147 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 453/7017 objects misplaced (6.456%); 1.2 MiB/s, 20 objects/s recovering 2024-09-17T16:21:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:24 smithi017 ceph-mon[27571]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:21:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:24 smithi017 ceph-mon[27571]: osdmap e409: 12 total, 12 up, 11 in 2024-09-17T16:21:25.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:25.112 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:25.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:25.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:25.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:25.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:25.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:25.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:25.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:25.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:25.494 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:25.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:25.596 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:25.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:25.619 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:25.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:25.720 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:25.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:25.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:25.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:25.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:21:25.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:25.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:25.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:25.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:25.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:25.998 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:25.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:26.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:26.126 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:26.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:26.228 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:26.253 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:26.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:26.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:26.370 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:21:26.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:26.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:26.474 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:26.479 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:26.483 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:26.509 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:26.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:26.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:26 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:26.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:26 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:26.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:26 smithi017 ceph-mon[27571]: osdmap e410: 12 total, 12 up, 11 in 2024-09-17T16:21:26.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1453973399' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:21:26.511 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:26 smithi017 ceph-mon[27571]: pgmap v928: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 35 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 234/1650 objects misplaced (14.182%); 938 KiB/s, 14 objects/s recovering 2024-09-17T16:21:26.511 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:21:26.611 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:26.633 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:26.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:26.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:26.761 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:26.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:26 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:26 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:26 smithi071 ceph-mon[30002]: osdmap e410: 12 total, 12 up, 11 in 2024-09-17T16:21:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1453973399' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:21:26.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:26 smithi071 ceph-mon[30002]: pgmap v928: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 35 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 234/1650 objects misplaced (14.182%); 938 KiB/s, 14 objects/s recovering 2024-09-17T16:21:26.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:26.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:26.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:26.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:26 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:26.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:26 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:26.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:26 smithi173 ceph-mon[30530]: osdmap e410: 12 total, 12 up, 11 in 2024-09-17T16:21:26.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1453973399' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:21:26.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:26 smithi173 ceph-mon[30530]: pgmap v928: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 35 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 234/1650 objects misplaced (14.182%); 938 KiB/s, 14 objects/s recovering 2024-09-17T16:21:26.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:27.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:27.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:27.118 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:27.178 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:27.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:27.206 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-17T16:21:27.206 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-17T16:21:27.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:27.306 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:27.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:27.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:27.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:27.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:27.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:27.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:27.536 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:27.559 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:27.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:27.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:27.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:27.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:27.764 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:27 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:27.764 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1453973399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:21:27.764 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:27 smithi017 ceph-mon[27571]: osdmap e411: 12 total, 12 up, 11 in 2024-09-17T16:21:27.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:27.808 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:27.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:27.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:27 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1453973399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:21:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:27 smithi071 ceph-mon[30002]: osdmap e411: 12 total, 12 up, 11 in 2024-09-17T16:21:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:27 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1453973399' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:21:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:27 smithi173 ceph-mon[30530]: osdmap e411: 12 total, 12 up, 11 in 2024-09-17T16:21:27.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:27.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:27.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:28.034 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:28.056 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:28.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:28.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:28.173 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:21:28.173 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:21:28.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:28.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:28.188 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:21:28.189 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:21:28.209 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:21:28.210 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:21:28.230 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:21:28.230 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:21:28.252 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:21:28.252 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:21:28.274 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:21:28.274 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:21:28.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:28.295 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:21:28.296 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:21:28.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:28.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:28.317 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:21:28.317 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:21:28.338 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:21:28.339 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:21:28.360 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:21:28.360 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:21:28.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:21:28.387 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:21:28.404 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:28.408 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:21:28.408 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:21:28.426 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:28.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:28.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:28.554 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:28.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:28.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:28.678 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:28.679 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:21:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:28 smithi017 ceph-mon[27571]: pgmap v930: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 2.0 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 425 op/s 2024-09-17T16:21:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:21:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:21:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:28 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:21:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2614956119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:28.780 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:28.802 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:28.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:21:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:28 smithi071 ceph-mon[30002]: pgmap v930: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 2.0 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 425 op/s 2024-09-17T16:21:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:21:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:21:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:28 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:21:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2614956119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:21:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:28 smithi173 ceph-mon[30530]: pgmap v930: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 2.0 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 425 op/s 2024-09-17T16:21:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:21:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:21:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:28 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:21:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2614956119' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:28.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:28.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:28.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:29.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:29.052 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:29.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:29.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:29.177 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:29.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:29.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:29.302 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:29.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:29.404 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:29.430 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:29.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:29.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:29.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:29.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:29.591 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:29.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:29.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:29.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:29.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:29 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:29.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:29 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:29.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:29 smithi017 ceph-mon[27571]: osdmap e412: 12 total, 12 up, 11 in 2024-09-17T16:21:29.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:29 smithi017 ceph-mon[27571]: osdmap e413: 12 total, 12 up, 11 in 2024-09-17T16:21:29.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:29.793 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:21:29.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:29.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:29 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:29 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:29 smithi071 ceph-mon[30002]: osdmap e412: 12 total, 12 up, 11 in 2024-09-17T16:21:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:29 smithi071 ceph-mon[30002]: osdmap e413: 12 total, 12 up, 11 in 2024-09-17T16:21:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:29 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:29 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:29 smithi173 ceph-mon[30530]: osdmap e412: 12 total, 12 up, 11 in 2024-09-17T16:21:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:29 smithi173 ceph-mon[30530]: osdmap e413: 12 total, 12 up, 11 in 2024-09-17T16:21:29.904 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:29.926 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:29.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:30.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:30.055 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:30.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:30.156 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:30.167 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-09-17T16:21:30.181 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:30.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:30.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:30.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:30.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:30.411 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:30.435 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:30.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:30.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:30.563 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:30.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:30.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:30 smithi017 ceph-mon[27571]: pgmap v933: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 342 op/s 2024-09-17T16:21:30.665 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:30.689 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:21:30.689 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:21:30.689 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:30.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:30.791 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:30 smithi071 ceph-mon[30002]: pgmap v933: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 342 op/s 2024-09-17T16:21:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:30.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:30.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:30 smithi173 ceph-mon[30530]: pgmap v933: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 342 op/s 2024-09-17T16:21:30.916 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:30.941 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:30.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:31.044 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:31.069 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:31.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:31.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:21:31.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:31.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:31.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:31.299 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:31.321 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:31.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:31.423 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:31.450 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:31.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:31.552 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:31.575 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:31.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:31.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:31.703 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:31.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:31.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:31.832 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:31.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:31.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:31.959 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:31.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:31.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:32.061 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:32.087 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:32.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:32.175 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:32.177 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:21:32.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:32.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:32.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:32.266 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:32.266 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:32.266 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:32.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:32.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:32.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:32.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:32.444 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:32.467 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:32.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:32.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:32.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:32.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:32.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:32.720 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:32.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:32.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:32.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:32.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:32.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:32.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:32.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:32.972 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:32.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:33.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:33.101 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:33.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:33.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:33.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:33.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:33.230 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:32 smithi017 ceph-mon[27571]: pgmap v934: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 91 op/s 2024-09-17T16:21:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:32 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:32 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:32 smithi017 ceph-mon[27571]: osdmap e414: 12 total, 12 up, 11 in 2024-09-17T16:21:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:32 smithi071 ceph-mon[30002]: pgmap v934: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 91 op/s 2024-09-17T16:21:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:32 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:32 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:32 smithi071 ceph-mon[30002]: osdmap e414: 12 total, 12 up, 11 in 2024-09-17T16:21:33.331 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:33.356 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:33.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:32 smithi173 ceph-mon[30530]: pgmap v934: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 91 op/s 2024-09-17T16:21:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:32 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:32 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:32 smithi173 ceph-mon[30530]: osdmap e414: 12 total, 12 up, 11 in 2024-09-17T16:21:33.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:33.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:33.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:33.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:33.610 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:33.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:33.712 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:33.735 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:33.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:33.766 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-09-17T16:21:33.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:33.871 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:33.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:33.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:33.999 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:34.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:34.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:34.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:34.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:34.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:34.256 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:34.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:34.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:34.382 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:34.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:34.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:34.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:34.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:34.511 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:34.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:34.638 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:34.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:34.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:34.741 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:34.766 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:34.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:34.868 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:34.893 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:34.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:34.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:35.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:35.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:35.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:35.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:35.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:35.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:35.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:34 smithi017 ceph-mon[27571]: pgmap v936: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 3 op/s 2024-09-17T16:21:35.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:35.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:35.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2140631873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:35.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:34 smithi017 ceph-mon[27571]: osdmap e415: 12 total, 12 up, 11 in 2024-09-17T16:21:35.251 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:35.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:35.274 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:35.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:34 smithi071 ceph-mon[30002]: pgmap v936: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 3 op/s 2024-09-17T16:21:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2140631873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:34 smithi071 ceph-mon[30002]: osdmap e415: 12 total, 12 up, 11 in 2024-09-17T16:21:35.316 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:21:35.316 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:21:35.316 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:21:35.317 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:21:35.317 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:21:35.317 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:21:35.317 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:21:35.317 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:21:35.317 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:21:35.317 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:21:35.317 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:21:35.318 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:21:35.319 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:21:35.319 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:21:35.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:34 smithi173 ceph-mon[30530]: pgmap v936: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 3 op/s 2024-09-17T16:21:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2140631873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:34 smithi173 ceph-mon[30530]: osdmap e415: 12 total, 12 up, 11 in 2024-09-17T16:21:35.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:35.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:35.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:35.528 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:35.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:35.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:35.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:35.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:35.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:35.784 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:35.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:35.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:35.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:35.913 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:35.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:35.980 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:21:35.980 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_2' removed 2024-09-17T16:21:36.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:36.040 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:36.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:36.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:36.164 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:36.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:36.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3665906696' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:36.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2313587101' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:21:36.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:36.290 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:36.291 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:36.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3665906696' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:36.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2313587101' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:21:36.391 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:36.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3665906696' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:36.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2313587101' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:21:36.417 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:36.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:36.519 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:36.540 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:36.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:36.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:36.663 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:36.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:36.765 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:36.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:36.806 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:36.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:36.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:36.930 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:36.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:36.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:37.012 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T16:21:37.012 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T16:21:37.013 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-17T16:21:37.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-17T16:21:37.032 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:37.078 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:37.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:37.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:37.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:37.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:37.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:36 smithi017 ceph-mon[27571]: pgmap v938: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:37.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:36 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:37.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3665906696' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:37.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2313587101' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:21:37.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:36 smithi017 ceph-mon[27571]: osdmap e416: 12 total, 12 up, 11 in 2024-09-17T16:21:37.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:36 smithi071 ceph-mon[30002]: pgmap v938: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:36 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3665906696' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2313587101' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:21:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:36 smithi071 ceph-mon[30002]: osdmap e416: 12 total, 12 up, 11 in 2024-09-17T16:21:37.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:37.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:37.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:36 smithi173 ceph-mon[30530]: pgmap v938: 17 pgs: 2 active+remapped+backfill_toofull, 15 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:36 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3665906696' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2313587101' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:21:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:36 smithi173 ceph-mon[30530]: osdmap e416: 12 total, 12 up, 11 in 2024-09-17T16:21:37.428 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:37.452 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:37.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:37.486 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:37.486 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:37.500 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:37.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:37.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:37.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:37.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:37.705 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:37.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:37.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:37.829 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:37.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:37.931 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:37.953 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:37.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:38.045 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:21:38.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:38.078 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:38.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:38.085 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:21:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:21:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:21:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:21:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:29460 2.1 2:83aee601:::benchmark_data_smithi173_55121_object29459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 588.87025124299998, 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31850449400000003, 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 29460 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197662+0000", 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197664+0000", 2024-09-17T16:21:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1289999999999999e-06 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197740+0000", 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6361999999999998e-05 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197753+0000", 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3453e-05 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197759+0000", 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7610000000000004e-06 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197820+0000", 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0735e-05 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.197960+0000", 2024-09-17T16:21:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013987799999999999 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198120+0000", 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016034 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.198880+0000", 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075943099999999997 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.322818+0000", 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.123938883 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516108+0000", 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.193289187 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516141+0000", 2024-09-17T16:21:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3691e-05 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:11:49.516167+0000", 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5644e-05 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:129847 2.1 2:86a30a1b:::benchmark_data_smithi173_55121_object53358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-17T16:21:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 490.268871273, 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29781534999999998, 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 129847 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799042+0000", 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799048+0000", 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6289999999999998e-06 2024-09-17T16:21:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799056+0000", 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3809999999999999e-06 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799074+0000", 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7864e-05 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799085+0000", 2024-09-17T16:21:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1076e-05 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799177+0000", 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1402000000000003e-05 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799338+0000", 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000161263 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.799482+0000", 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000143638 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801720+0000", 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022381739999999999 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.901264+0000", 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.099544493999999997 2024-09-17T16:21:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096745+0000", 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.195480396 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096804+0000", 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8990000000000003e-05 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096858+0000", 2024-09-17T16:21:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4042999999999997e-05 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:129847 2.1 e184/104 ondisk, result = 0)", 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 490.26591444500002, 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29486439199999998, 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.801999+0000", 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.802000+0000", 2024-09-17T16:21:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1600000000000001e-07 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:27.802005+0000", 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9060000000000001e-06 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096600+0000", 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.294595249 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096606+0000", 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.733e-06 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096731+0000", 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000124735 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096739+0000", 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:28.096864+0000", 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012515399999999999 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(3.4 241/239 [PushOp(3:20540438:::benchmark_data_smithi173_66272_object5569:head, version: 213'343, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:20540438:::benchmark_data_smithi173_66272_object5569:head@213'343, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 380.51511975900002, 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56341482399999998, 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.103 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552796+0000", 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.888e-06 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552825+0000", 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9417999999999999e-05 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115364+0000", 2024-09-17T16:21:38.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56253852800000004 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115383+0000", 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9055999999999998e-05 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115470+0000", 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7520000000000002e-05 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115481+0000", 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0329000000000001e-05 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.116209+0000", 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072808499999999995 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25516.0:155234 4.6 e371/361 4:60535f2d:::benchmark_data_smithi173_77474_object53059:head v 371'9504, mlcod=371'9501)", 2024-09-17T16:21:38.107 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 85.478216075999995, 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30676646000000002, 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:21:38.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589704+0000", 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5330000000000002e-06 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.891997+0000", 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30229256300000001 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.892001+0000", 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6539999999999998e-06 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.892046+0000", 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5305000000000001e-05 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.892062+0000", 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5931999999999999e-05 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.896408+0000", 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043459869999999999 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.896464+0000", 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5717000000000002e-05 2024-09-17T16:21:38.111 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:185136 4.6 4:6e122307:::benchmark_data_smithi173_77474_object82961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:55.713056+0000", 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.354857758999998, 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30286027900000001, 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:38.112 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:38.113 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:38.113 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 185136 2024-09-17T16:21:38.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.113 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713056+0000", 2024-09-17T16:21:38.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713056+0000", 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713058+0000", 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1100000000000001e-06 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713068+0000", 2024-09-17T16:21:38.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7890000000000006e-06 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713085+0000", 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6660999999999999e-05 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713092+0000", 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5159999999999997e-06 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.714377+0000", 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012851379999999999 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.714505+0000", 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012745500000000001 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.714629+0000", 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012392699999999999 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.718974+0000", 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043457169999999998 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.718988+0000", 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.375e-05 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015789+0000", 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29680071600000002 2024-09-17T16:21:38.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015866+0000", 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7026000000000004e-05 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015916+0000", 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0473999999999998e-05 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:185136 4.6 e391/387 ondisk, result = 0)", 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:55.717118+0000", 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.350795542999997, 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29880589800000001, 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717118+0000", 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717118+0000", 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717120+0000", 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.392e-06 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717126+0000", 2024-09-17T16:21:38.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8279999999999997e-06 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015002+0000", 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.297875367 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015011+0000", 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1519999999999998e-06 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015769+0000", 2024-09-17T16:21:38.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075783199999999995 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015780+0000", 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.166e-05 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015924+0000", 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014366699999999999 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:185177 4.6 4:6ef442e2:::benchmark_data_smithi173_77474_object83002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:56.206993+0000", 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.860920382000003, 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39139948400000002, 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 185177 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.123 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.206993+0000", 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.206993+0000", 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.206995+0000", 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0619999999999999e-06 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207004+0000", 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5019999999999992e-06 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207022+0000", 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8365000000000001e-05 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207030+0000", 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5639999999999999e-06 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207096+0000", 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5938999999999993e-05 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207197+0000", 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100929 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207279+0000", 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2474999999999999e-05 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211157+0000", 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003878009 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.423507+0000", 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21234955799999999 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598152+0000", 2024-09-17T16:21:38.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.174645726 2024-09-17T16:21:38.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598251+0000", 2024-09-17T16:21:38.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8258999999999996e-05 2024-09-17T16:21:38.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598393+0000", 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142096 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:185177 4.6 e391/387 ondisk, result = 0)", 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:56.211354+0000", 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.856559830000002, 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38704429800000001, 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.129 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211354+0000", 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211354+0000", 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211355+0000", 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8899999999999998e-07 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211363+0000", 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4509999999999992e-06 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.597986+0000", 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38662227900000001 2024-09-17T16:21:38.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.597997+0000", 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.189e-05 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598126+0000", 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000128505 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598143+0000", 2024-09-17T16:21:38.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7405999999999999e-05 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598398+0000", 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025487799999999999 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:186301 4.4 4:28479395:::benchmark_data_smithi173_77474_object84126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:58.360185+0000", 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 39.707728649000003, 2024-09-17T16:21:38.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37107270399999998, 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186301 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360185+0000", 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360185+0000", 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360186+0000", 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.02e-06 2024-09-17T16:21:38.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360191+0000", 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6269999999999999e-06 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360201+0000", 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0583e-05 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360207+0000", 2024-09-17T16:21:38.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1989999999999997e-06 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360255+0000", 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8810999999999997e-05 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360347+0000", 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1436999999999999e-05 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:21:38.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360424+0000", 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7566999999999998e-05 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364029+0000", 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036048090000000001 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364399+0000", 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000370331 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731169+0000", 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36676913100000003 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731215+0000", 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6047000000000001e-05 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731258+0000", 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3142000000000001e-05 2024-09-17T16:21:38.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:186301 4.4 e393/306 ondisk, result = 0)", 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:58.364680+0000", 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 39.703233367000003, 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36658104899999999, 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364680+0000", 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364680+0000", 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364681+0000", 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6899999999999997e-07 2024-09-17T16:21:38.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364685+0000", 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1579999999999998e-06 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731114+0000", 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.366428431 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731117+0000", 2024-09-17T16:21:38.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5549999999999998e-06 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731158+0000", 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0655999999999998e-05 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731163+0000", 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1869999999999996e-06 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731261+0000", 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8393000000000005e-05 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:197926 4.6 4:6bb101f1:::benchmark_data_smithi173_77474_object95751:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:14.693991+0000", 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.373922784000001, 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.057340164999999998, 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.144 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 197926 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.693991+0000", 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.693991+0000", 2024-09-17T16:21:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.693992+0000", 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.376e-06 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.693998+0000", 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8209999999999998e-06 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.694014+0000", 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6279000000000001e-05 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.694022+0000", 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2910000000000002e-06 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.694076+0000", 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4719000000000002e-05 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.694197+0000", 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120495 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.694314+0000", 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116848 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.698913+0000", 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045990889999999998 2024-09-17T16:21:38.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.699273+0000", 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00035977099999999999 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.751217+0000", 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.051943967000000001 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.751277+0000", 2024-09-17T16:21:38.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0718999999999999e-05 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:14.751331+0000", 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3789999999999998e-05 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:203685 4.6 4:6b87bbab:::benchmark_data_smithi173_77474_object101510:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:22.773432+0000", 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 15.294481701, 2024-09-17T16:21:38.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.059340751999999997, 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 203685 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.773432+0000", 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.773432+0000", 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.773433+0000", 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3480000000000001e-06 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.773438+0000", 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7520000000000004e-06 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.773451+0000", 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2608e-05 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.773459+0000", 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0309999999999998e-06 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.776242+0000", 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002782891 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.776367+0000", 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012575999999999999 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.776484+0000", 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011686199999999999 2024-09-17T16:21:38.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.781921+0000", 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054364289999999996 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.787234+0000", 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053134539999999996 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.832582+0000", 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.045347767999999997 2024-09-17T16:21:38.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.832698+0000", 2024-09-17T16:21:38.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116197 2024-09-17T16:21:38.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:22.832773+0000", 2024-09-17T16:21:38.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4652000000000003e-05 2024-09-17T16:21:38.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.157 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.157 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204228 4.4 4:2e094692:::benchmark_data_smithi173_77474_object102053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:21:38.157 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.406771+0000", 2024-09-17T16:21:38.157 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.661142536, 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.118991899, 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204228 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.406771+0000", 2024-09-17T16:21:38.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.406771+0000", 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.406772+0000", 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2810000000000001e-06 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.406777+0000", 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.972e-06 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.406790+0000", 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2252000000000001e-05 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.406795+0000", 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6609999999999997e-06 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.406851+0000", 2024-09-17T16:21:38.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5893999999999997e-05 2024-09-17T16:21:38.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.406987+0000", 2024-09-17T16:21:38.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013612300000000001 2024-09-17T16:21:38.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.407112+0000", 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012424800000000001 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.411119+0000", 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040070389999999996 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:38.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.411168+0000", 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9787999999999998e-05 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.525691+0000", 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.114522517 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.525729+0000", 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7766999999999999e-05 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.525763+0000", 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4357000000000003e-05 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204281 4.4 4:24cd04e6:::benchmark_data_smithi173_77474_object102106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.517360+0000", 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.550553825, 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.082454983999999995, 2024-09-17T16:21:38.164 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204281 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.517360+0000", 2024-09-17T16:21:38.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.517360+0000", 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.517361+0000", 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.192e-06 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.517366+0000", 2024-09-17T16:21:38.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8879999999999996e-06 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.517377+0000", 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0750999999999999e-05 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.517384+0000", 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7149999999999998e-06 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.517444+0000", 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9336000000000002e-05 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.517534+0000", 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9999000000000003e-05 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.517614+0000", 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0525000000000006e-05 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.521376+0000", 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003761248 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.522231+0000", 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000855078 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.599721+0000", 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.077490092999999996 2024-09-17T16:21:38.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.599758+0000", 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7642000000000003e-05 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.599815+0000", 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6517000000000001e-05 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204346 4.4 4:22db2208:::benchmark_data_smithi173_77474_object102171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.625061+0000", 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.442852545999999, 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.67454315899999995, 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204346 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.171 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625061+0000", 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625061+0000", 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:38.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625062+0000", 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0300000000000001e-06 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625067+0000", 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4320000000000001e-06 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625077+0000", 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9189999999999994e-06 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625083+0000", 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9020000000000002e-06 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625140+0000", 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6677e-05 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625237+0000", 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6446999999999993e-05 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625308+0000", 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1585e-05 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.629047+0000", 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037388569999999999 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.630013+0000", 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096557000000000001 2024-09-17T16:21:38.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299522+0000", 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.66950939200000004 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299565+0000", 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.316e-05 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299604+0000", 2024-09-17T16:21:38.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9187999999999998e-05 2024-09-17T16:21:38.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.177 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:38.177 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:38.177 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:21:38.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:38.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:38.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:38.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:38 smithi017 ceph-mon[27571]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-17T16:21:38.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:38.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:38 smithi071 ceph-mon[30002]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-17T16:21:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:38.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:38.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:38 smithi173 ceph-mon[30530]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-17T16:21:38.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:38.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:38.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:38.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:38.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:38.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:38.692 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:38.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:38.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:38.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:38.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:38.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:38.890 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:21:38.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:38.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:38.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:38.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:39.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:39.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:39.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:39.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:39.222 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:39.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:39.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:39 smithi017 ceph-mon[27571]: pgmap v940: 1 pgs: 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:39.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:39 smithi173 ceph-mon[30530]: pgmap v940: 1 pgs: 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:39 smithi071 ceph-mon[30002]: pgmap v940: 1 pgs: 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:39.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:39.337 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:39.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:39.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:39.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:21:39.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:39.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:39.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:39.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:39.577 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:39.605 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:39.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:39.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:39.731 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:39.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:39.833 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:39.859 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:39.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:39.961 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:39.986 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:39.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:40.069 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-09-17T16:21:40.070 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:40.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:40.112 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:40.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:40.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:40.239 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:40.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:40.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:40.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:40.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:40.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:40.494 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:40.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:40.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:40.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:40.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:40.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:40.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:40.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:40 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:40.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:40 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:40.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:40 smithi017 ceph-mon[27571]: osdmap e417: 12 total, 12 up, 11 in 2024-09-17T16:21:40.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:40 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:40.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:40 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:40.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:40 smithi017 ceph-mon[27571]: pgmap v942: 1 pgs: 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:40.657 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:40.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:40.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:40.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:40 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:40 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:40 smithi071 ceph-mon[30002]: osdmap e417: 12 total, 12 up, 11 in 2024-09-17T16:21:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:40 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:40 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:40 smithi071 ceph-mon[30002]: pgmap v942: 1 pgs: 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:40.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:40.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:40.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:40 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:40 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:40 smithi173 ceph-mon[30530]: osdmap e417: 12 total, 12 up, 11 in 2024-09-17T16:21:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:40 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:40 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:40 smithi173 ceph-mon[30530]: pgmap v942: 1 pgs: 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:40.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:41.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:41.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:41.107 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:41.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:41.130 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:41.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:41.232 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:41.254 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:41.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:41.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:41.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:41.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:41.457 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_3' created 2024-09-17T16:21:41.483 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:41.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:41 smithi017 ceph-mon[27571]: osdmap e418: 12 total, 12 up, 11 in 2024-09-17T16:21:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2237673377' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-17T16:21:41.507 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:41.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:41.609 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:41.633 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:41.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:41.736 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:41.759 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:41.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:41 smithi071 ceph-mon[30002]: osdmap e418: 12 total, 12 up, 11 in 2024-09-17T16:21:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2237673377' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-17T16:21:41.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:41.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:41.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:41.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:41 smithi173 ceph-mon[30530]: osdmap e418: 12 total, 12 up, 11 in 2024-09-17T16:21:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2237673377' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-17T16:21:41.962 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:21:41.963 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:21:41.977 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:21:41.977 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:21:41.989 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:41.997 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:21:41.997 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:21:42.015 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:42.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:42.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:42.017 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:21:42.018 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:21:42.039 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:21:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:21:42.060 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:21:42.061 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:21:42.082 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:21:42.082 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:21:42.103 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:21:42.104 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:21:42.117 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:42.125 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:21:42.125 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:21:42.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:42.140 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:42.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:42.147 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:21:42.148 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:21:42.169 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:21:42.169 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:21:42.191 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:21:42.191 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:21:42.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:42.266 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:42.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:42.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:42.394 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:42.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:42.461 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:21:42.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:42.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:42.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:42.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:42.551 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:21:42.551 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:21:42.551 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:21:42.551 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:21:42.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:42.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:42.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:42.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2237673377' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-17T16:21:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:42 smithi017 ceph-mon[27571]: osdmap e419: 12 total, 12 up, 11 in 2024-09-17T16:21:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:42 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:42 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:42 smithi017 ceph-mon[27571]: pgmap v945: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1419761496' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:21:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2680162721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:42.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:42.775 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:42.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2237673377' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-17T16:21:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:42 smithi071 ceph-mon[30002]: osdmap e419: 12 total, 12 up, 11 in 2024-09-17T16:21:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:42 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:42 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:42 smithi071 ceph-mon[30002]: pgmap v945: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1419761496' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:21:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2680162721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:42.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2237673377' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-17T16:21:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:42 smithi173 ceph-mon[30530]: osdmap e419: 12 total, 12 up, 11 in 2024-09-17T16:21:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:42 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:42 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:42 smithi173 ceph-mon[30530]: pgmap v945: 17 pgs: 2 creating+peering, 14 unknown, 1 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1419761496' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:21:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2680162721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:42.902 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:42.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:42.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:43.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:43.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:43.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:43.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:43.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:43.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:43.257 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:43.263 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-17T16:21:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:43.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:43.385 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:43.408 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:43.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:43 smithi017 ceph-mon[27571]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:21:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:43 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1419761496' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:21:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:43 smithi017 ceph-mon[27571]: osdmap e420: 12 total, 12 up, 11 in 2024-09-17T16:21:43.508 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:43.530 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:43.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:43.632 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:43.654 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:43.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:43.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:43.782 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:43.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:43.803 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:21:43.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:43 smithi071 ceph-mon[30002]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:21:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:43 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1419761496' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:21:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:43 smithi071 ceph-mon[30002]: osdmap e420: 12 total, 12 up, 11 in 2024-09-17T16:21:43.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:43.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:43 smithi173 ceph-mon[30530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:21:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:43 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1419761496' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:21:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:43 smithi173 ceph-mon[30530]: osdmap e420: 12 total, 12 up, 11 in 2024-09-17T16:21:43.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:43.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:43.912 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:44.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:44.042 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:44.042 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:44.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:44.144 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:44.166 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:44.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:44.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:44.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:44.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:44.396 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:44.422 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:44.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:44.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:44.512 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:44.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:44.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:44.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:44.653 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:44.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:44.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:44 smithi017 ceph-mon[27571]: osdmap e421: 12 total, 12 up, 11 in 2024-09-17T16:21:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:44 smithi017 ceph-mon[27571]: pgmap v948: 17 pgs: 3 active+clean, 3 creating+peering, 11 unknown; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:44.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:44.801 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:44.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:44 smithi071 ceph-mon[30002]: osdmap e421: 12 total, 12 up, 11 in 2024-09-17T16:21:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:44 smithi071 ceph-mon[30002]: pgmap v948: 17 pgs: 3 active+clean, 3 creating+peering, 11 unknown; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:44 smithi173 ceph-mon[30530]: osdmap e421: 12 total, 12 up, 11 in 2024-09-17T16:21:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:44 smithi173 ceph-mon[30530]: pgmap v948: 17 pgs: 3 active+clean, 3 creating+peering, 11 unknown; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:44.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:44.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:44.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:45.014 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:45.029 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:45.052 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:45.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:45.155 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:45.179 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:45.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:45.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:45.302 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:45.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:45.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:45.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:45.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:45.533 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:45.555 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:45.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:45.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:45.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:45.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:45.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:45.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:45.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:45.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:45.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:45.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:45.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:46.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:46.065 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:46.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:46.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:46.195 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:46.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:46.297 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:46.322 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:46.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:46.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:46.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:46.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:46.466 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:21:46.466 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:21:46.480 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:21:46.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:46.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:46.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:46.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:46.702 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:46.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:46.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:46.827 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:46.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:46.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:46.955 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:46.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:47.049 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:46 smithi017 ceph-mon[27571]: pgmap v949: 17 pgs: 14 active+clean, 3 creating+peering; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:47.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:47.079 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:47.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:47.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:47.181 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:47.206 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:47.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:47.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:47.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:46 smithi071 ceph-mon[30002]: pgmap v949: 17 pgs: 14 active+clean, 3 creating+peering; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:47.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:47.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:47.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:46 smithi173 ceph-mon[30530]: pgmap v949: 17 pgs: 14 active+clean, 3 creating+peering; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:47.437 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:47.462 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:47.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:47.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:47.587 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:47.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:47.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:47.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:47.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:47.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:47.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:21:47.842 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:47.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:47.934 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-17T16:21:47.944 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:47.967 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:47.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:47.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:48.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:48.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:48.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:48.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:48.104 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2828895265' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:21:48.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:21:48.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:48.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:48.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2828895265' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:21:48.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:48.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:48.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:48.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2828895265' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:21:48.451 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:48.476 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:48.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:48.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:48.602 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:48.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:48.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:48.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:48.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:48.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:48.859 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:48.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:48.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:48.866 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:48.866 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:48.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:48.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:48.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:49.089 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:49.116 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:49.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:49.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:48 smithi017 ceph-mon[27571]: pgmap v950: 17 pgs: 17 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:49.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2828895265' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:21:49.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:48 smithi017 ceph-mon[27571]: osdmap e422: 12 total, 12 up, 11 in 2024-09-17T16:21:49.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:49.240 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:49.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:48 smithi071 ceph-mon[30002]: pgmap v950: 17 pgs: 17 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2828895265' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:21:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:48 smithi071 ceph-mon[30002]: osdmap e422: 12 total, 12 up, 11 in 2024-09-17T16:21:49.341 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:49.366 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:49.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:48 smithi173 ceph-mon[30530]: pgmap v950: 17 pgs: 17 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2828895265' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:21:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:48 smithi173 ceph-mon[30530]: osdmap e422: 12 total, 12 up, 11 in 2024-09-17T16:21:49.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:49.491 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:49.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:49.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:49.594 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:49.620 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:49.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:49.642 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:21:49.642 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:21:49.663 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:21:49.664 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:21:49.684 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:21:49.684 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:21:49.705 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:21:49.705 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:21:49.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:49.726 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:21:49.726 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:21:49.743 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:49.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:49.749 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:21:49.750 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:21:49.771 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:21:49.771 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:21:49.793 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:21:49.793 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:21:49.815 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:21:49.815 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:21:49.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:21:49.838 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:21:49.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:49.860 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:21:49.860 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:21:49.861 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:49.865 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-17T16:21:49.868 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T16:21:49.868 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T16:21:49.871 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:49.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:49.885 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:21:49.886 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:21:49.928 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-09-17T16:21:49.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T16:21:49.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_90735 2024-09-17T16:21:49.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:21:49.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-09-17T16:21:49.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:50.004 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:50.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:50.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:50.127 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:50.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:50.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:50.262 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:50.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:50.363 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:50.386 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:50.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:50.422 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:94007 2.19 2:9e42b13d:::benchmark_data_smithi173_55121_object17518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 540.53134192899995, 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.467128445, 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.423 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:21:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:21:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 94007 2024-09-17T16:21:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.424 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887692+0000", 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887693+0000", 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-17T16:21:50.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887700+0000", 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887711+0000", 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0739e-05 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887716+0000", 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2000000000000002e-06 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887759+0000", 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2898999999999997e-05 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887839+0000", 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9500999999999997e-05 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.887937+0000", 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8531999999999998e-05 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.889187+0000", 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012492580000000001 2024-09-17T16:21:50.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:49.889505+0000", 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031885699999999997 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354736+0000", 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46523019100000002 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354784+0000", 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8776999999999997e-05 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:12:50.354821+0000", 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6321e-05 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.430 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:105470 2.19 2:9b81cd1a:::benchmark_data_smithi173_55121_object28981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:02.872711+0000", 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 527.54632306799999, 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43773533999999997, 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 105470 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.431 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872711+0000", 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872711+0000", 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872713+0000", 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3689999999999999e-06 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872717+0000", 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9400000000000001e-06 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872729+0000", 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1994000000000001e-05 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872735+0000", 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.711e-06 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872777+0000", 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1894000000000002e-05 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872864+0000", 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7322000000000002e-05 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.872942+0000", 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7465000000000005e-05 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874265+0000", 2024-09-17T16:21:50.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0013235009999999999 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874550+0000", 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000285059 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310369+0000", 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43581851399999999 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310403+0000", 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3594999999999999e-05 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310447+0000", 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3976e-05 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.24550.0:105470 2.19 e170/114 ondisk, result = 0)", 2024-09-17T16:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:02.874849+0000", 2024-09-17T16:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 527.544185524, 2024-09-17T16:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43560358700000001, 2024-09-17T16:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:21:50.438 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874849+0000", 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874849+0000", 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874850+0000", 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2130000000000001e-06 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:02.874857+0000", 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9970000000000001e-06 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310301+0000", 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43544382999999998 2024-09-17T16:21:50.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310308+0000", 2024-09-17T16:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9210000000000003e-06 2024-09-17T16:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310360+0000", 2024-09-17T16:21:50.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2383000000000003e-05 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310366+0000", 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4410000000000001e-06 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:03.310452+0000", 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6802e-05 2024-09-17T16:21:50.442 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:152973 2.19 2:9e1f786b:::benchmark_data_smithi173_55121_object76484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 476.55199725900002, 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59801266799999997, 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.443 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:21:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:21:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 152973 2024-09-17T16:21:50.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867039+0000", 2024-09-17T16:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.581e-06 2024-09-17T16:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867044+0000", 2024-09-17T16:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0089999999999998e-06 2024-09-17T16:21:50.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867057+0000", 2024-09-17T16:21:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3658999999999999e-05 2024-09-17T16:21:50.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867064+0000", 2024-09-17T16:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2520000000000002e-06 2024-09-17T16:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867120+0000", 2024-09-17T16:21:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5937999999999999e-05 2024-09-17T16:21:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867194+0000", 2024-09-17T16:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3467000000000007e-05 2024-09-17T16:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:21:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867254+0000", 2024-09-17T16:21:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0510000000000002e-05 2024-09-17T16:21:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.869276+0000", 2024-09-17T16:21:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002021882 2024-09-17T16:21:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.869436+0000", 2024-09-17T16:21:50.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000159214 2024-09-17T16:21:50.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.464981+0000", 2024-09-17T16:21:50.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59554505000000002 2024-09-17T16:21:50.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.465020+0000", 2024-09-17T16:21:50.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9674e-05 2024-09-17T16:21:50.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.465050+0000", 2024-09-17T16:21:50.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9431999999999999e-05 2024-09-17T16:21:50.456 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.456 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.456 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.457 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(3.7 240/239 [PushReplyOp(3:e198b3f2:::benchmark_data_smithi173_66272_object478:head)])", 2024-09-17T16:21:50.457 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:21:50.457 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 395.40588096499999, 2024-09-17T16:21:50.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43133256199999997, 2024-09-17T16:21:50.458 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.458 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.458 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:21:50.458 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.459 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.459 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:21:50.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:21:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.460 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.461 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013155+0000", 2024-09-17T16:21:50.461 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-17T16:21:50.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.461 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.462 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013161+0000", 2024-09-17T16:21:50.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6830000000000002e-06 2024-09-17T16:21:50.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.462 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.462 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441372+0000", 2024-09-17T16:21:50.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42821048099999998 2024-09-17T16:21:50.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441385+0000", 2024-09-17T16:21:50.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.311e-05 2024-09-17T16:21:50.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441470+0000", 2024-09-17T16:21:50.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5185000000000005e-05 2024-09-17T16:21:50.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.444486+0000", 2024-09-17T16:21:50.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030157669999999999 2024-09-17T16:21:50.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.466 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.466 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124248 3.7 3:e0068a72:::benchmark_data_smithi173_66272_object52292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-09-17T16:21:50.467 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:21:50.467 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 301.52429367299999, 2024-09-17T16:21:50.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46227925399999997, 2024-09-17T16:21:50.467 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.467 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.468 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:21:50.468 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.468 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:21:50.468 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:21:50.468 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124248 2024-09-17T16:21:50.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.471 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:21:50.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.472 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.472 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:21:50.472 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894742+0000", 2024-09-17T16:21:50.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.392e-06 2024-09-17T16:21:50.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.474 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.474 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894747+0000", 2024-09-17T16:21:50.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4979999999999997e-06 2024-09-17T16:21:50.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894763+0000", 2024-09-17T16:21:50.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5095e-05 2024-09-17T16:21:50.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894769+0000", 2024-09-17T16:21:50.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7360000000000004e-06 2024-09-17T16:21:50.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.477 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894818+0000", 2024-09-17T16:21:50.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8551000000000003e-05 2024-09-17T16:21:50.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.478 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.478 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894928+0000", 2024-09-17T16:21:50.478 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110536 2024-09-17T16:21:50.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:21:50.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.895033+0000", 2024-09-17T16:21:50.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104589 2024-09-17T16:21:50.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.900743+0000", 2024-09-17T16:21:50.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057098299999999999 2024-09-17T16:21:50.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:50.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.901009+0000", 2024-09-17T16:21:50.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000265927 2024-09-17T16:21:50.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.927407+0000", 2024-09-17T16:21:50.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.026398688 2024-09-17T16:21:50.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:21:50.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.927419+0000", 2024-09-17T16:21:50.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1154e-05 2024-09-17T16:21:50.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.483 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.483 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:49.356994+0000", 2024-09-17T16:21:50.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.429575227 2024-09-17T16:21:50.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.484 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:49.357020+0000", 2024-09-17T16:21:50.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6030999999999999e-05 2024-09-17T16:21:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124248 3.7 3:e0068a72:::benchmark_data_smithi173_66272_object52292:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:21:50.486 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:21:50.486 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 300.04251296400003, 2024-09-17T16:21:50.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54499606199999995, 2024-09-17T16:21:50.486 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.486 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.487 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:21:50.487 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.487 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:21:50.487 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:21:50.487 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124248 2024-09-17T16:21:50.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.488 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.488 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.488 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:21:50.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:21:50.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376523+0000", 2024-09-17T16:21:50.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.42e-06 2024-09-17T16:21:50.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376528+0000", 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7200000000000003e-06 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376539+0000", 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.022e-05 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376546+0000", 2024-09-17T16:21:50.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8390000000000007e-06 2024-09-17T16:21:50.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376598+0000", 2024-09-17T16:21:50.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1953000000000002e-05 2024-09-17T16:21:50.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376611+0000", 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2320000000000001e-05 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.921322+0000", 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54471168999999997 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.921517+0000", 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019489999999999999 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_scan(digest 3.7 MIN-3:ec3efd19:::benchmark_data_smithi173_66272_object4297:head e 290/273)", 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 269.378876564, 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.427333666, 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:21:50.494 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:21:50.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040160+0000", 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0660000000000002e-06 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040256+0000", 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5756999999999998e-05 2024-09-17T16:21:50.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467140+0000", 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42688425000000002 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467149+0000", 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2289999999999992e-06 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467244+0000", 2024-09-17T16:21:50.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709000000000004e-05 2024-09-17T16:21:50.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467260+0000", 2024-09-17T16:21:50.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6232e-05 2024-09-17T16:21:50.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467491+0000", 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000231423 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDeleteReply(3.c e297,267 3:350956dd:::benchmark_data_smithi173_66272_object11629:head,259'5194)", 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 262.15202032500002, 2024-09-17T16:21:50.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.990895003, 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "queued for pg", 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:21:50.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267015+0000", 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0500000000000003e-07 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267021+0000", 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2550000000000003e-06 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257793+0000", 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.99077237299999998 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257810+0000", 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7005999999999999e-05 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257909+0000", 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8664000000000003e-05 2024-09-17T16:21:50.502 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.503 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.503 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.503 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:102161 4.9 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-17T16:21:50.503 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:21:50.503 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 158.910389702, 2024-09-17T16:21:50.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.902645841, 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 102161 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:21:50.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8700000000000005e-07 2024-09-17T16:21:50.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508680+0000", 2024-09-17T16:21:50.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4382999999999999e-05 2024-09-17T16:21:50.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508688+0000", 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5059999999999995e-06 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508699+0000", 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0781000000000001e-05 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508757+0000", 2024-09-17T16:21:50.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8026999999999999e-05 2024-09-17T16:21:50.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508920+0000", 2024-09-17T16:21:50.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163308 2024-09-17T16:21:50.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:21:50.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.509034+0000", 2024-09-17T16:21:50.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113961 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.512221+0000", 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003186805 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.515116+0000", 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002895299 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411212+0000", 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89609538399999999 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411258+0000", 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6217999999999997e-05 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411290+0000", 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2481999999999998e-05 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.510 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.511 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:184107 4.9 4:98a0eb83:::benchmark_data_smithi173_77474_object81932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-17T16:21:50.511 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:53.719117+0000", 2024-09-17T16:21:50.511 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.699917589000002, 2024-09-17T16:21:50.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.73240478200000003, 2024-09-17T16:21:50.511 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.511 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.512 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.512 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.512 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:50.512 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:50.512 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 184107 2024-09-17T16:21:50.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.512 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719117+0000", 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719117+0000", 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719120+0000", 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.88e-06 2024-09-17T16:21:50.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719133+0000", 2024-09-17T16:21:50.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.312e-05 2024-09-17T16:21:50.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719166+0000", 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3772999999999997e-05 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719183+0000", 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6668999999999999e-05 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719313+0000", 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129789 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719438+0000", 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012514700000000001 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719532+0000", 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4146000000000003e-05 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723223+0000", 2024-09-17T16:21:50.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036906310000000002 2024-09-17T16:21:50.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723276+0000", 2024-09-17T16:21:50.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2994000000000001e-05 2024-09-17T16:21:50.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:50.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723329+0000", 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2828000000000003e-05 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723485+0000", 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015637499999999999 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:54.451313+0000", 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.72782798500000001 2024-09-17T16:21:50.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:54.451383+0000", 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0465999999999999e-05 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:54.451521+0000", 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013797899999999999 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:185164 4.f 4:f8df73f0:::benchmark_data_smithi173_77474_object82989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:56.000100+0000", 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.418934045999997, 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.634535988, 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 185164 2024-09-17T16:21:50.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000100+0000", 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000100+0000", 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000102+0000", 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.404e-06 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000109+0000", 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9619999999999996e-06 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000119+0000", 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0558e-05 2024-09-17T16:21:50.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000124+0000", 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2170000000000002e-06 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000166+0000", 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2134e-05 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000270+0000", 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104005 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,9,11", 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000379+0000", 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010812 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.004004+0000", 2024-09-17T16:21:50.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036249329999999999 2024-09-17T16:21:50.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:50.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.004131+0000", 2024-09-17T16:21:50.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000127336 2024-09-17T16:21:50.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.020238+0000", 2024-09-17T16:21:50.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016107305999999998 2024-09-17T16:21:50.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.634511+0000", 2024-09-17T16:21:50.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61427235199999997 2024-09-17T16:21:50.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.634550+0000", 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9057000000000002e-05 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.634636+0000", 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6604e-05 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.527 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:186293 4.f 4:ffe1d3e0:::benchmark_data_smithi173_77474_object84118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:58.185546+0000", 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 52.233488133000002, 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42855515999999999, 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186293 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185546+0000", 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185546+0000", 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185547+0000", 2024-09-17T16:21:50.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.359e-06 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185553+0000", 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4349999999999996e-06 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185565+0000", 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2155e-05 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185571+0000", 2024-09-17T16:21:50.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6509999999999998e-06 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185619+0000", 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8696999999999998e-05 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185723+0000", 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010333 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,9,11", 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185816+0000", 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3335e-05 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.189433+0000", 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036167840000000001 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.190346+0000", 2024-09-17T16:21:50.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091320500000000005 2024-09-17T16:21:50.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.190429+0000", 2024-09-17T16:21:50.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3014000000000002e-05 2024-09-17T16:21:50.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.613994+0000", 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42356446800000003 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.614037+0000", 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3027e-05 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.614101+0000", 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4700000000000001e-05 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204335 4.9 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 26.812342956999998, 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77672725300000001, 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.535 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204335 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606692+0000", 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2600000000000002e-07 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606696+0000", 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7170000000000002e-06 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606707+0000", 2024-09-17T16:21:50.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1486e-05 2024-09-17T16:21:50.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606712+0000", 2024-09-17T16:21:50.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1560000000000003e-06 2024-09-17T16:21:50.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606769+0000", 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6836000000000003e-05 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606856+0000", 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6749999999999994e-05 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606942+0000", 2024-09-17T16:21:50.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6001999999999994e-05 2024-09-17T16:21:50.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611141+0000", 2024-09-17T16:21:50.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041987810000000004 2024-09-17T16:21:50.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611206+0000", 2024-09-17T16:21:50.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5581999999999997e-05 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611481+0000", 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027492400000000001 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611502+0000", 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0971999999999998e-05 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383271+0000", 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77176877600000005 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383334+0000", 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2928000000000005e-05 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383419+0000", 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4617000000000007e-05 2024-09-17T16:21:50.542 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204343 4.0 4:0b58e0a7:::benchmark_data_smithi173_77474_object102168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.621137+0000", 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 26.797896815000001, 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49750966600000002, 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:21:50.543 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204343 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621137+0000", 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621137+0000", 2024-09-17T16:21:50.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621139+0000", 2024-09-17T16:21:50.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.635e-06 2024-09-17T16:21:50.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621144+0000", 2024-09-17T16:21:50.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3170000000000001e-06 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621157+0000", 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2123999999999999e-05 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621164+0000", 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1679999999999998e-06 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621222+0000", 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8439000000000001e-05 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621307+0000", 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4940999999999999e-05 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621418+0000", 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110548 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626056+0000", 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046379489999999997 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095492+0000", 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46943599899999999 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:50.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118566+0000", 2024-09-17T16:21:50.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023074935000000001 2024-09-17T16:21:50.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118609+0000", 2024-09-17T16:21:50.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2106e-05 2024-09-17T16:21:50.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118647+0000", 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8504999999999998e-05 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:204343 4.0 e408/389 ondisk, result = 0)", 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.625102+0000", 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 26.793932393999999, 2024-09-17T16:21:50.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.493551564, 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625102+0000", 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625102+0000", 2024-09-17T16:21:50.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625103+0000", 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0100000000000004e-07 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625111+0000", 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9659999999999996e-06 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118509+0000", 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49339840800000001 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118514+0000", 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5589999999999996e-06 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118557+0000", 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2222000000000003e-05 2024-09-17T16:21:50.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118563+0000", 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2689999999999996e-06 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118653+0000", 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439000000000006e-05 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25516.0:204346 4.4 e408/306 4:22db2208:::benchmark_data_smithi173_77474_object102171:head v 408'12784, mlcod=408'12781)", 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.625376+0000", 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 26.79365803, 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.67400101199999995, 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625376+0000", 2024-09-17T16:21:50.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625376+0000", 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625377+0000", 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2699999999999998e-07 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625386+0000", 2024-09-17T16:21:50.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3260000000000002e-06 2024-09-17T16:21:50.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295220+0000", 2024-09-17T16:21:50.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.66983389199999999 2024-09-17T16:21:50.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295230+0000", 2024-09-17T16:21:50.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3090000000000001e-06 2024-09-17T16:21:50.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295298+0000", 2024-09-17T16:21:50.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8583000000000004e-05 2024-09-17T16:21:50.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295337+0000", 2024-09-17T16:21:50.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8992000000000002e-05 2024-09-17T16:21:50.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299315+0000", 2024-09-17T16:21:50.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039779949999999998 2024-09-17T16:21:50.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299377+0000", 2024-09-17T16:21:50.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2088000000000006e-05 2024-09-17T16:21:50.562 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.563 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.563 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:204343 4.0 e408/389 ondisk, result = 0)", 2024-09-17T16:21:50.563 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.626258+0000", 2024-09-17T16:21:50.563 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 26.792776383, 2024-09-17T16:21:50.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46924642799999999, 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626258+0000", 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626258+0000", 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626259+0000", 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0100000000000004e-07 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626262+0000", 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5300000000000001e-06 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095382+0000", 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46911975700000003 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095391+0000", 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4010000000000003e-06 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095466+0000", 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5087999999999997e-05 2024-09-17T16:21:50.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.567 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:21:50.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095481+0000", 2024-09-17T16:21:50.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4252e-05 2024-09-17T16:21:50.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095504+0000", 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3699000000000001e-05 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:9 5.1 e423/419 5:82b6797c:::benchmark_data_smithi173_90735_object8:head v 423'1, mlcod=0'0)", 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:49.934390+0000", 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.48464425100000003, 2024-09-17T16:21:50.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001126419, 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.934390+0000", 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.934390+0000", 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.934390+0000", 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2099999999999998e-07 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.934627+0000", 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023633899999999999 2024-09-17T16:21:50.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.934659+0000", 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2128999999999997e-05 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.934668+0000", 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6679999999999996e-06 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.934728+0000", 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9704000000000003e-05 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.934748+0000", 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9388000000000001e-05 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.935479+0000", 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073100400000000005 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.935516+0000", 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7866e-05 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:163 5.1 e423/419 5:83ba401f:::benchmark_data_smithi173_90735_object162:head v 423'9, mlcod=423'6)", 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:49.976129+0000", 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.44290516099999999, 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086246200000000004, 2024-09-17T16:21:50.573 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.976129+0000", 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.976129+0000", 2024-09-17T16:21:50.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:21:50.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:21:50.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.976130+0000", 2024-09-17T16:21:50.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-17T16:21:50.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:21:50.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.976305+0000", 2024-09-17T16:21:50.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000174931 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.976348+0000", 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.316e-05 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.976374+0000", 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5928e-05 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.976430+0000", 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6341000000000001e-05 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.976452+0000", 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1353e-05 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.976945+0000", 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000493499 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:50.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:21:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:49.976992+0000", 2024-09-17T16:21:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6477999999999998e-05 2024-09-17T16:21:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:50.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:50.578 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:21:50.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:50.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:50.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:50.708 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:50.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:50.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:50 smithi017 ceph-mon[27571]: osdmap e423: 12 total, 12 up, 11 in 2024-09-17T16:21:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/928244876' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:50 smithi017 ceph-mon[27571]: pgmap v953: 17 pgs: 17 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2805777766' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:50 smithi071 ceph-mon[30002]: osdmap e423: 12 total, 12 up, 11 in 2024-09-17T16:21:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/928244876' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:50 smithi071 ceph-mon[30002]: pgmap v953: 17 pgs: 17 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2805777766' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:50.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:50.857 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:50.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:50 smithi173 ceph-mon[30530]: osdmap e423: 12 total, 12 up, 11 in 2024-09-17T16:21:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/928244876' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:50 smithi173 ceph-mon[30530]: pgmap v953: 17 pgs: 17 active+clean; 577 KiB data, 3.0 GiB used, 980 GiB / 983 GiB avail 2024-09-17T16:21:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2805777766' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:21:50.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 1495 1479 92.4298 92.4375 0.00132488 0.0107155 2024-09-17T16:21:50.958 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:50.985 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:50.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:51.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:51.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:51.110 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:51.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:51.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:51.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:51.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:51.341 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:51.367 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:51.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:51.469 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:51.491 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:51.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:51.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:51.618 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:51.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:51.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:51 smithi017 ceph-mon[27571]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:21:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:51 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:21:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2805777766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:51 smithi017 ceph-mon[27571]: osdmap e424: 12 total, 12 up, 11 in 2024-09-17T16:21:51.740 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:51.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:51 smithi071 ceph-mon[30002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:21:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:51 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:21:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2805777766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:51 smithi071 ceph-mon[30002]: osdmap e424: 12 total, 12 up, 11 in 2024-09-17T16:21:51.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:51.866 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:51.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:51.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:51 smithi173 ceph-mon[30530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:21:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:51 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:21:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2805777766' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:21:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:51 smithi173 ceph-mon[30530]: osdmap e424: 12 total, 12 up, 11 in 2024-09-17T16:21:51.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 2494 2478 77.4313 62.4375 0.00142624 0.0128353 2024-09-17T16:21:51.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:51.995 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:51.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:52.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:52.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:52.132 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:52.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:52.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:52.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:52.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:52.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:52.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:52.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:52.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:52.396 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:21:52.396 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:21:52.396 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T16:21:52.396 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:21:52.490 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:52.517 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:52.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:52.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:52.648 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:52.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:52 smithi017 ceph-mon[27571]: pgmap v955: 17 pgs: 17 active+clean; 55 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 9.0 MiB/s wr, 144 op/s 2024-09-17T16:21:52.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:52.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:52.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:52.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:52 smithi071 ceph-mon[30002]: pgmap v955: 17 pgs: 17 active+clean; 55 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 9.0 MiB/s wr, 144 op/s 2024-09-17T16:21:52.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:52.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:52 smithi173 ceph-mon[30530]: pgmap v955: 17 pgs: 17 active+clean; 55 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 9.0 MiB/s wr, 144 op/s 2024-09-17T16:21:52.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:52.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:52.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 4594 4578 95.3675 131.25 0.00243141 0.0104579 2024-09-17T16:21:52.984 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:21:53.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:53.026 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:53.049 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:53.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:53.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:53.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:53.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:53.275 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:21:53.280 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:53.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:53.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:53.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:53.427 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:53.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:53.529 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:53.552 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:53.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:53.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:53.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:53.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:53.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:53.780 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:21:53.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:53.802 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:53.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:53.825 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:21:53.825 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 424, 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:21:50.443373+0000", 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:17:40.364450+0000", 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5799936, 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:21:53.826 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 5, 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "jewel", 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:21:53.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:21:53.828 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:21:53.829 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "21", 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:21:53.830 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:21:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:21:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:21:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:21:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:21:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:21:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:21:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:21:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:21:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:21:53.833 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 10, 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 10, 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.30000001192092896, 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.7735099196434021, 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-17T16:21:53.834 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.76999998092651367 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 5, 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_3", 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:21:40.832300+0000", 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:21:53.835 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:21:53.836 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:21:53.836 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:21:53.836 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:21:53.836 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:21:53.836 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:21:53.836 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:21:53.836 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:21:53.836 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 16, 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 16, 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 16, 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 16, 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 16, 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:21:53.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "422", 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:21:53.838 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:21:53.839 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:21:53.840 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 2.5, 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.5, 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1.1000000238418579, 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 2.75, 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 2.75, 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.7735099196434021, 2024-09-17T16:21:53.841 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.76999998092651367 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.842 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 295, 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:21:53.843 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:21:53.844 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.844 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.844 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.844 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.844 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.845 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:21:53.845 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:21:53.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.845 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.845 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:21:53.845 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:21:53.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.845 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.846 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.846 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.846 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.846 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:21:53.846 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:21:53.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.846 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.847 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.848 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 0, 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 301, 2024-09-17T16:21:53.849 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:21:53.850 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:21:53.851 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.852 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:21:53.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:21:53.854 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.855 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.858 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.859 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:21:53.860 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:21:53.861 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 403, 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.862 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.863 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:21:53.864 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:21:53.864 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.864 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.864 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.864 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:21:53.866 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:21:53.867 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:21:53.867 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:21:53.867 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.867 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.867 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.867 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.867 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:21:53.867 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.868 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.869 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:21:53.870 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:21:53.871 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.872 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.845672607421875, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.874 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.875 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:21:53.877 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.878 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.66790771484375, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.880 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.881 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.882 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:21:53.885 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.8061676025390625, 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.886 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.887 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.888 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:21:53.888 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:21:53.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.888 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:21:53.888 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.890 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.891 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:21:53.892 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 286, 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:21:53.893 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:21:53.894 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.896 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.897 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:21:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:21:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:21:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.2821197509765625, 2024-09-17T16:21:53.898 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.899 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:21:53.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:21:53.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.902 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:21:53.903 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:21:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.904 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:21:53.904 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:53.906 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:21:53.906 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.906 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.906 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:21:53.906 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:21:53.906 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:21:53.908 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:21:53.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.909 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.909 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:21:53.909 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:21:53.909 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.909 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:21:53.911 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:21:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:21:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:21:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:21:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:21:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.912 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.588836669921875, 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:21:53.913 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 291, 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:21:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:21:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:21:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.915 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:21:53.917 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.918 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:21:53.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:21:31.916187+0000", 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.920 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 65536, 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:20:57.117171+0000", 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.921 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:21:16.303303+0000", 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:21:41.705136+0000", 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.922 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:21:10.862851+0000", 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.923 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:21:39.426922+0000", 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:21:25.375931+0000", 2024-09-17T16:21:53.924 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:21:27.946194+0000", 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:21:53.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:20:47.792990+0000", 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:21:53.926 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:21:00.369254+0000", 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.927 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:21:39.476296+0000", 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:21:34.151677+0000", 2024-09-17T16:21:53.928 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [], 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:21:53.929 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:21:53.930 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:21:53.931 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:21:53.932 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:21:53.932 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:21:53.932 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:21:53.932 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:21:53.932 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:21:53.932 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:21:53.933 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 5735 5719 89.3523 71.3125 0.00966248 0.0111468 2024-09-17T16:21:53.956 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:53.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:54.018 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1456038227' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:21:54.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:54.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:54.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:5911 5.1 5:89a7775a:::benchmark_data_smithi173_90735_object5910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:54.094508+0000", 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.034912171999999998, 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.034935541000000001, 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:21:54.132 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:21:54.133 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:21:54.133 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:21:54.133 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 5911 2024-09-17T16:21:54.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.133 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:21:54.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:21:54.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.094508+0000", 2024-09-17T16:21:54.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:21:54.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.094508+0000", 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.094509+0000", 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5039999999999999e-06 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.094567+0000", 2024-09-17T16:21:54.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7209000000000001e-05 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.094580+0000", 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3689999999999999e-05 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.094586+0000", 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2500000000000003e-06 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.094636+0000", 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9234999999999999e-05 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.094727+0000", 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1675999999999994e-05 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.094823+0000", 2024-09-17T16:21:54.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5274999999999998e-05 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.095292+0000", 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000469505 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.095745+0000", 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00045304399999999999 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:21:54.137 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:5930 5.c 5:3cedb510:::benchmark_data_smithi173_90735_object5929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:54.125408+0000", 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.004011807, 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040814809999999997, 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:21:54.138 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 5930 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.125408+0000", 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.125408+0000", 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.139 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:21:54.140 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.125410+0000", 2024-09-17T16:21:54.140 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.339e-06 2024-09-17T16:21:54.140 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.140 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.140 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:21:54.140 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.125454+0000", 2024-09-17T16:21:54.140 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4737000000000002e-05 2024-09-17T16:21:54.140 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.140 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.140 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.125466+0000", 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1299e-05 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.125472+0000", 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1210000000000003e-06 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.125522+0000", 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0207000000000002e-05 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.125623+0000", 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010132900000000001 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.125710+0000", 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6373999999999997e-05 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.126181+0000", 2024-09-17T16:21:54.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000471068 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.126568+0000", 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00038747599999999999 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 2 2024-09-17T16:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:21:54.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1456038227' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:21:54.185 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:54.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1456038227' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:21:54.208 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:54.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:54.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:54.310 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:54.343 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:54.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:54.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:54.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:21:54.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:54.476 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:54.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:54.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:54.602 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:54.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:54.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:54.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:54.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:54.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:54.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:54.855 INFO:tasks.thrashosds.thrasher:j is {'epoch': 424, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:21:50.443373+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:17:40.364450+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.30000001192092896, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.7735099196434021, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.7699999809265137}}, {'pool': 5, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-17T16:21:40.832300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '422', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1.100000023841858, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.7735099196434021, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.7699999809265137}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 295, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 301, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 403, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 0.845672607421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 0.66790771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 0.8061676025390625, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 419, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 286, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0.2821197509765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 0.588836669921875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:21:31.916187+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-17T16:20:57.117171+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:21:16.303303+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:21:41.705136+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:21:10.862851+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:21:39.426922+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:21:25.375931+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:21:27.946194+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:20:47.792990+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:21:00.369254+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:21:39.476296+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:21:34.151677+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:21:54.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:21:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:54.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:54.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:54.908 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:54.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 7155 7139 89.23 88.75 0.149523 0.0110413 2024-09-17T16:21:54.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:54.979 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:54.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:54.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:55.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:54 smithi071 ceph-mon[30002]: pgmap v956: 17 pgs: 17 active+clean; 104 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 17 MiB/s wr, 276 op/s 2024-09-17T16:21:55.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:54 smithi071 ceph-mon[30002]: osdmap e425: 12 total, 12 up, 11 in 2024-09-17T16:21:55.081 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:55.087 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:54 smithi017 ceph-mon[27571]: pgmap v956: 17 pgs: 17 active+clean; 104 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 17 MiB/s wr, 276 op/s 2024-09-17T16:21:55.087 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:54 smithi017 ceph-mon[27571]: osdmap e425: 12 total, 12 up, 11 in 2024-09-17T16:21:55.111 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:55.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:55.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:55.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:54 smithi173 ceph-mon[30530]: pgmap v956: 17 pgs: 17 active+clean; 104 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 17 MiB/s wr, 276 op/s 2024-09-17T16:21:55.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:54 smithi173 ceph-mon[30530]: osdmap e425: 12 total, 12 up, 11 in 2024-09-17T16:21:55.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:55.239 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:55.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:55.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:55.341 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:55.346 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:21:55.364 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:55.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:55.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:55.489 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:55.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:55.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:55.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:55.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:55.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:55.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:55.615 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:21:55.714 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:55.738 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:55.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:55.840 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:55.871 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:55.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:55.881 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:21:55.917 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3044538582' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:21:55.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3044538582' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:21:55.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 8462 8446 87.9719 81.6875 0.00174347 0.0113546 2024-09-17T16:21:55.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:21:56.000 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:56.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:21:56.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:56.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:56.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:56.142 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:21:56.142 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:21:56.142 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:21:56.142 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:21:56.142 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:21:56.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3044538582' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:21:56.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:56.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:56.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:56.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:56.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:56.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:56.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:56.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:56.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:56.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:56.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:56.600 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:56.622 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:56.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:56.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:56.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:56.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:56.852 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:56.883 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:56.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:56.890 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:21:56.890 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:21:56.917 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:21:56.917 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:21:56.932 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 15 9518 9503 84.8412 66.0625 0.00171773 0.0117694 2024-09-17T16:21:56.942 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:21:56.943 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:21:56.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:21:56.973 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:21:56.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:56.999 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:21:56.999 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:21:57.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:57.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:57.024 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:21:57.024 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:21:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:56 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:56 smithi071 ceph-mon[30002]: pgmap v958: 17 pgs: 17 active+clean; 409 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-17T16:21:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3044538582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:21:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:56 smithi071 ceph-mon[30002]: osdmap e426: 12 total, 12 up, 11 in 2024-09-17T16:21:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/219171510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:57.095 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:21:57.095 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:21:57.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:21:57.116 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:21:57.117 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:21:57.117 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:56 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:57.117 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:56 smithi017 ceph-mon[27571]: pgmap v958: 17 pgs: 17 active+clean; 409 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-17T16:21:57.117 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3044538582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:21:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:56 smithi017 ceph-mon[27571]: osdmap e426: 12 total, 12 up, 11 in 2024-09-17T16:21:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/219171510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:57.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:57.132 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:21:57.138 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:21:57.139 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:21:57.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:56 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:21:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:56 smithi173 ceph-mon[30530]: pgmap v958: 17 pgs: 17 active+clean; 409 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-17T16:21:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3044538582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:21:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:56 smithi173 ceph-mon[30530]: osdmap e426: 12 total, 12 up, 11 in 2024-09-17T16:21:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/219171510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:21:57.176 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:21:57.176 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:21:57.197 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:21:57.197 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:21:57.227 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:21:57.227 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:21:57.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:57.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:57.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:57.270 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:21:57.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:57.381 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:57.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:57.436 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:21:57.436 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:21:57.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:57.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:57.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:57.526 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:21:57.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:21:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:57.632 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:57.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:21:57.734 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:57.761 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:57.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:57.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:57.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:57.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:57.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 10409 10393 81.1886 55.625 0.00277561 0.0122985 2024-09-17T16:21:57.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:58.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:58.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:58.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:58.104 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:21:58.104 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:21:58.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:58.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:58.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:58.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:58.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:21:58.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:58.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:21:58.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:58.402 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:58.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:58.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:58.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:58.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:21:58.632 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:21:58.662 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:58.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:58.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:21:58.764 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:21:58.791 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:58.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:21:58.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:21:58.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:58.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:58.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:21:58.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 12667 12651 87.8469 141.125 0.0062656 0.0113701 2024-09-17T16:21:58.965 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":959,"stamp":"2024-09-17T16:21:55.880343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":429326880,"num_objects":6544,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6544,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":6665,"num_write_kb":420838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6634,"ondisk_log_size":6634,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":4001444,"kb_used_data":1260160,"kb_used_omap":57,"kb_used_meta":2740998,"kb_avail":1027195228,"statfs":{"total":1055945392128,"available":1051847913472,"internally_reserved":0,"allocated":1290403840,"data_stored":1285136156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2806782686},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,1,1,0,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":428736512,"num_objects":6542,"num_object_clones":0,"num_object_copies":19626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6542,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.005711"},"pg_stats":[{"pgid":"5.f","version":"425'518","reported_seq":541,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:55.526000+0000","last_change":"2024-09-17T16:21:42.617220+0000","last_active":"2024-09-17T16:21:55.526000+0000","last_peered":"2024-09-17T16:21:55.526000+0000","last_clean":"2024-09-17T16:21:55.526000+0000","last_became_active":"2024-09-17T16:21:42.616065+0000","last_became_peered":"2024-09-17T16:21:42.616065+0000","last_unstale":"2024-09-17T16:21:55.526000+0000","last_undegraded":"2024-09-17T16:21:55.526000+0000","last_fullsized":"2024-09-17T16:21:55.526000+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:01.813525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33947648,"num_objects":518,"num_object_clones":0,"num_object_copies":1554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":518,"num_write_kb":33152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"424'201","reported_seq":221,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-17T16:21:52.326490+0000","last_change":"2024-09-17T16:21:42.615978+0000","last_active":"2024-09-17T16:21:52.326490+0000","last_peered":"2024-09-17T16:21:52.326490+0000","last_clean":"2024-09-17T16:21:52.326490+0000","last_became_active":"2024-09-17T16:21:42.615584+0000","last_became_peered":"2024-09-17T16:21:42.615584+0000","last_unstale":"2024-09-17T16:21:52.326490+0000","last_undegraded":"2024-09-17T16:21:52.326490+0000","last_fullsized":"2024-09-17T16:21:52.326490+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:08.519662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13172736,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":201,"num_write_kb":12864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"425'515","reported_seq":538,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:55.522188+0000","last_change":"2024-09-17T16:21:42.615849+0000","last_active":"2024-09-17T16:21:55.522188+0000","last_peered":"2024-09-17T16:21:55.522188+0000","last_clean":"2024-09-17T16:21:55.522188+0000","last_became_active":"2024-09-17T16:21:42.615522+0000","last_became_peered":"2024-09-17T16:21:42.615522+0000","last_unstale":"2024-09-17T16:21:55.522188+0000","last_undegraded":"2024-09-17T16:21:55.522188+0000","last_fullsized":"2024-09-17T16:21:55.522188+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:49.214195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":515,"num_object_clones":0,"num_object_copies":1545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":32960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"425'450","reported_seq":473,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:54.712178+0000","last_change":"2024-09-17T16:21:42.616080+0000","last_active":"2024-09-17T16:21:54.712178+0000","last_peered":"2024-09-17T16:21:54.712178+0000","last_clean":"2024-09-17T16:21:54.712178+0000","last_became_active":"2024-09-17T16:21:42.615761+0000","last_became_peered":"2024-09-17T16:21:42.615761+0000","last_unstale":"2024-09-17T16:21:54.712178+0000","last_undegraded":"2024-09-17T16:21:54.712178+0000","last_fullsized":"2024-09-17T16:21:54.712178+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:01.878455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29491200,"num_objects":450,"num_object_clones":0,"num_object_copies":1350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":450,"num_write_kb":28800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"424'304","reported_seq":325,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-17T16:21:53.006232+0000","last_change":"2024-09-17T16:21:42.616006+0000","last_active":"2024-09-17T16:21:53.006232+0000","last_peered":"2024-09-17T16:21:53.006232+0000","last_clean":"2024-09-17T16:21:53.006232+0000","last_became_active":"2024-09-17T16:21:42.615693+0000","last_became_peered":"2024-09-17T16:21:42.615693+0000","last_unstale":"2024-09-17T16:21:53.006232+0000","last_undegraded":"2024-09-17T16:21:53.006232+0000","last_fullsized":"2024-09-17T16:21:53.006232+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:08.008512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19922944,"num_objects":304,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":19456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"424'129","reported_seq":150,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-17T16:21:51.477475+0000","last_change":"2024-09-17T16:21:42.467809+0000","last_active":"2024-09-17T16:21:51.477475+0000","last_peered":"2024-09-17T16:21:51.477475+0000","last_clean":"2024-09-17T16:21:51.477475+0000","last_became_active":"2024-09-17T16:21:42.467536+0000","last_became_peered":"2024-09-17T16:21:42.467536+0000","last_unstale":"2024-09-17T16:21:51.477475+0000","last_undegraded":"2024-09-17T16:21:51.477475+0000","last_fullsized":"2024-09-17T16:21:51.477475+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:54.415573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454144,"num_objects":129,"num_object_clones":0,"num_object_copies":387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":129,"num_write_kb":8256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"424'288","reported_seq":309,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-17T16:21:53.014328+0000","last_change":"2024-09-17T16:21:42.615828+0000","last_active":"2024-09-17T16:21:53.014328+0000","last_peered":"2024-09-17T16:21:53.014328+0000","last_clean":"2024-09-17T16:21:53.014328+0000","last_became_active":"2024-09-17T16:21:42.615436+0000","last_became_peered":"2024-09-17T16:21:42.615436+0000","last_unstale":"2024-09-17T16:21:53.014328+0000","last_undegraded":"2024-09-17T16:21:53.014328+0000","last_fullsized":"2024-09-17T16:21:53.014328+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:07.562425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18874368,"num_objects":288,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":288,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"425'499","reported_seq":522,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:55.831401+0000","last_change":"2024-09-17T16:21:42.616143+0000","last_active":"2024-09-17T16:21:55.831401+0000","last_peered":"2024-09-17T16:21:55.831401+0000","last_clean":"2024-09-17T16:21:55.831401+0000","last_became_active":"2024-09-17T16:21:42.615778+0000","last_became_peered":"2024-09-17T16:21:42.615778+0000","last_unstale":"2024-09-17T16:21:55.831401+0000","last_undegraded":"2024-09-17T16:21:55.831401+0000","last_fullsized":"2024-09-17T16:21:55.831401+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:06.957578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32702464,"num_objects":499,"num_object_clones":0,"num_object_copies":1497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":499,"num_write_kb":31936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[10,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"425'577","reported_seq":599,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:55.836622+0000","last_change":"2024-09-17T16:21:42.468402+0000","last_active":"2024-09-17T16:21:55.836622+0000","last_peered":"2024-09-17T16:21:55.836622+0000","last_clean":"2024-09-17T16:21:55.836622+0000","last_became_active":"2024-09-17T16:21:42.467840+0000","last_became_peered":"2024-09-17T16:21:42.467840+0000","last_unstale":"2024-09-17T16:21:55.836622+0000","last_undegraded":"2024-09-17T16:21:55.836622+0000","last_fullsized":"2024-09-17T16:21:55.836622+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:44.683474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":577,"num_object_clones":0,"num_object_copies":1731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":577,"num_write_kb":36928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"424'139","reported_seq":160,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-17T16:21:51.497062+0000","last_change":"2024-09-17T16:21:42.472962+0000","last_active":"2024-09-17T16:21:51.497062+0000","last_peered":"2024-09-17T16:21:51.497062+0000","last_clean":"2024-09-17T16:21:51.497062+0000","last_became_active":"2024-09-17T16:21:42.472648+0000","last_became_peered":"2024-09-17T16:21:42.472648+0000","last_unstale":"2024-09-17T16:21:51.497062+0000","last_undegraded":"2024-09-17T16:21:51.497062+0000","last_fullsized":"2024-09-17T16:21:51.497062+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:00.127440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9109504,"num_objects":139,"num_object_clones":0,"num_object_copies":417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139,"num_write_kb":8896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"425'439","reported_seq":462,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:54.741674+0000","last_change":"2024-09-17T16:21:42.615917+0000","last_active":"2024-09-17T16:21:54.741674+0000","last_peered":"2024-09-17T16:21:54.741674+0000","last_clean":"2024-09-17T16:21:54.741674+0000","last_became_active":"2024-09-17T16:21:42.615595+0000","last_became_peered":"2024-09-17T16:21:42.615595+0000","last_unstale":"2024-09-17T16:21:54.741674+0000","last_undegraded":"2024-09-17T16:21:54.741674+0000","last_fullsized":"2024-09-17T16:21:54.741674+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":439,"log_dups_size":0,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:51.154511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28770304,"num_objects":439,"num_object_clones":0,"num_object_copies":1317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":439,"num_write_kb":28096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"425'519","reported_seq":542,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:55.716141+0000","last_change":"2024-09-17T16:21:42.616516+0000","last_active":"2024-09-17T16:21:55.716141+0000","last_peered":"2024-09-17T16:21:55.716141+0000","last_clean":"2024-09-17T16:21:55.716141+0000","last_became_active":"2024-09-17T16:21:42.615586+0000","last_became_peered":"2024-09-17T16:21:42.615586+0000","last_unstale":"2024-09-17T16:21:55.716141+0000","last_undegraded":"2024-09-17T16:21:55.716141+0000","last_fullsized":"2024-09-17T16:21:55.716141+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":519,"log_dups_size":0,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:48.998835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":519,"num_object_clones":0,"num_object_copies":1557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":519,"num_write_kb":33216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"425'504","reported_seq":527,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:55.594737+0000","last_change":"2024-09-17T16:21:42.617260+0000","last_active":"2024-09-17T16:21:55.594737+0000","last_peered":"2024-09-17T16:21:55.594737+0000","last_clean":"2024-09-17T16:21:55.594737+0000","last_became_active":"2024-09-17T16:21:42.615959+0000","last_became_peered":"2024-09-17T16:21:42.615959+0000","last_unstale":"2024-09-17T16:21:55.594737+0000","last_undegraded":"2024-09-17T16:21:55.594737+0000","last_fullsized":"2024-09-17T16:21:55.594737+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:09.798834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33030144,"num_objects":504,"num_object_clones":0,"num_object_copies":1512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":504,"num_write_kb":32256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"425'445","reported_seq":467,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:54.743116+0000","last_change":"2024-09-17T16:21:42.616857+0000","last_active":"2024-09-17T16:21:54.743116+0000","last_peered":"2024-09-17T16:21:54.743116+0000","last_clean":"2024-09-17T16:21:54.743116+0000","last_became_active":"2024-09-17T16:21:42.616534+0000","last_became_peered":"2024-09-17T16:21:42.616534+0000","last_unstale":"2024-09-17T16:21:54.743116+0000","last_undegraded":"2024-09-17T16:21:54.743116+0000","last_fullsized":"2024-09-17T16:21:54.743116+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:52.078506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29163520,"num_objects":445,"num_object_clones":0,"num_object_copies":1335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":445,"num_write_kb":28480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"425'540","reported_seq":563,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:55.838482+0000","last_change":"2024-09-17T16:21:42.615804+0000","last_active":"2024-09-17T16:21:55.838482+0000","last_peered":"2024-09-17T16:21:55.838482+0000","last_clean":"2024-09-17T16:21:55.838482+0000","last_became_active":"2024-09-17T16:21:42.615475+0000","last_became_peered":"2024-09-17T16:21:42.615475+0000","last_unstale":"2024-09-17T16:21:55.838482+0000","last_undegraded":"2024-09-17T16:21:55.838482+0000","last_fullsized":"2024-09-17T16:21:55.838482+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:53.339886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35389440,"num_objects":540,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":540,"num_write_kb":34560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"425'475","reported_seq":498,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-17T16:21:55.828637+0000","last_change":"2024-09-17T16:21:42.472814+0000","last_active":"2024-09-17T16:21:55.828637+0000","last_peered":"2024-09-17T16:21:55.828637+0000","last_clean":"2024-09-17T16:21:55.828637+0000","last_became_active":"2024-09-17T16:21:42.472512+0000","last_became_peered":"2024-09-17T16:21:42.472512+0000","last_unstale":"2024-09-17T16:21:55.828637+0000","last_undegraded":"2024-09-17T16:21:55.828637+0000","last_fullsized":"2024-09-17T16:21:55.828637+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:46.196379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":475,"num_write_kb":30400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":740,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-17T16:21:50.462121+0000","last_change":"2024-09-17T16:16:49.364827+0000","last_active":"2024-09-17T16:21:50.462121+0000","last_peered":"2024-09-17T16:21:50.462121+0000","last_clean":"2024-09-17T16:21:50.462121+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:21:50.462121+0000","last_undegraded":"2024-09-17T16:21:50.462121+0000","last_fullsized":"2024-09-17T16:21:50.462121+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:14:54.958088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:14:54.958088+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:59.929724+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":428736512,"num_objects":6542,"num_object_clones":0,"num_object_copies":19626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6542,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1072496640,"data_stored":1072496640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6542,"ondisk_log_size":6542,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285717,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":381328,"kb_used_data":83580,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93363824,"statfs":{"total":95995035648,"available":95604555776,"internally_reserved":0,"allocated":85585920,"data_stored":85101544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":304871876},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547354,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353736,"kb_used_data":192052,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93391416,"statfs":{"total":95995035648,"available":95632809984,"internally_reserved":0,"allocated":196661248,"data_stored":196180444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":165542336},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743582,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":423104,"kb_used_data":49000,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93322048,"statfs":{"total":95995035648,"available":95561777152,"internally_reserved":0,"allocated":50176000,"data_stored":49699744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":383056320},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193976,"kb_used_data":50344,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93551176,"statfs":{"total":95995035648,"available":95796404224,"internally_reserved":0,"allocated":51552256,"data_stored":51076000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963696,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":332096,"kb_used_data":72624,"kb_used_omap":41,"kb_used_meta":259414,"kb_avail":93413056,"statfs":{"total":95995035648,"available":95654969344,"internally_reserved":0,"allocated":74366976,"data_stored":73887168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":265640243},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495790,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":435212,"kb_used_data":155516,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93309940,"statfs":{"total":95995035648,"available":95549378560,"internally_reserved":0,"allocated":159248384,"data_stored":158764028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":286390726},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724721,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714504,"kb_used_data":448692,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":93030648,"statfs":{"total":95995035648,"available":95263383552,"internally_reserved":0,"allocated":459460608,"data_stored":458979804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":272169414},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986360,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":398840,"kb_used_data":133032,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":93346312,"statfs":{"total":95995035648,"available":95586623488,"internally_reserved":0,"allocated":136224768,"data_stored":135748512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305856,"kb_used_data":8104,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93439296,"statfs":{"total":95995035648,"available":95681839104,"internally_reserved":0,"allocated":8298496,"data_stored":7822240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":304871879},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411521,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175236,"kb_used_data":27432,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93569916,"statfs":{"total":95995035648,"available":95815593984,"internally_reserved":0,"allocated":28090368,"data_stored":27614112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575047,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738571,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":287556,"kb_used_data":39784,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93457596,"statfs":{"total":95995035648,"available":95700578304,"internally_reserved":0,"allocated":40738816,"data_stored":40262560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55508992,"data_stored":55508992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37748736,"data_stored":37748736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7012352,"data_stored":7012352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":148570112,"data_stored":148570112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":162988032,"data_stored":162988032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":165150720,"data_stored":165150720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":76414976,"data_stored":76414976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":52494336,"data_stored":52494336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":74973184,"data_stored":74973184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204668928,"data_stored":204668928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":86966272,"data_stored":86966272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:21:58.967 INFO:tasks.thrashosds.thrasher:Setting 5.5 to [10, 4, 3] 2024-09-17T16:21:58.967 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.5', '10', '4', '3'] 2024-09-17T16:21:58.967 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 5.5 10 4 3 2024-09-17T16:21:59.004 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:59.020 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:59.042 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:59.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:59.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:59.134 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:21:59.144 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:59.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:58 smithi173 ceph-mon[30530]: from='client.16599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:21:59.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:58 smithi173 ceph-mon[30530]: pgmap v960: 17 pgs: 17 active+clean; 487 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-17T16:21:59.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:21:59.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:21:59.156 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:58 smithi017 ceph-mon[27571]: from='client.16599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:21:59.157 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:58 smithi017 ceph-mon[27571]: pgmap v960: 17 pgs: 17 active+clean; 487 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-17T16:21:59.157 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:21:59.157 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:21:59.170 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:59.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:59.257 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:58 smithi071 ceph-mon[30002]: from='client.16599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:21:59.257 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:58 smithi071 ceph-mon[30002]: pgmap v960: 17 pgs: 17 active+clean; 487 MiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-17T16:21:59.257 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:21:59.257 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:21:59.272 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:21:59.293 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:59.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:21:59.396 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:21:59.421 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:59.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:21:59.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:21:59.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:59.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:21:59.614 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:21:59.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:21:59.679 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:21:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:59.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:21:59.733 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:21:59.733 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:21:59.780 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:21:59.810 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:21:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:59.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:21:59.894 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:21:59.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:21:59.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 14135 14119 88.2364 91.75 0.00451119 0.011317 2024-09-17T16:21:59.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:21:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:21:59.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:00.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:00.065 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:00.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:00.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:59 smithi173 ceph-mon[30530]: osdmap e427: 12 total, 12 up, 11 in 2024-09-17T16:22:00.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:21:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3017406805' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:00.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:22:00.167 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:00.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:59 smithi017 ceph-mon[27571]: osdmap e427: 12 total, 12 up, 11 in 2024-09-17T16:22:00.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:21:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3017406805' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:00.197 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:00.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:00.299 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:00.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:59 smithi071 ceph-mon[30002]: osdmap e427: 12 total, 12 up, 11 in 2024-09-17T16:22:00.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:21:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3017406805' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:00.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:00.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:00.370 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:22:00.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:00.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:00.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:00.546 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:00.568 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:00.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:00.638 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:22:00.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:00.697 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:00.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:00.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:00.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:00.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:00.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:00.924 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:00.929 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 14799 14783 83.9867 41.5 0.10346 0.0117657 2024-09-17T16:22:00.946 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:00.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:00 smithi017 ceph-mon[27571]: pgmap v962: 17 pgs: 17 active+clean; 606 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:22:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3017406805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:00 smithi017 ceph-mon[27571]: osdmap e428: 12 total, 12 up, 11 in 2024-09-17T16:22:01.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:00 smithi071 ceph-mon[30002]: pgmap v962: 17 pgs: 17 active+clean; 606 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:22:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3017406805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:00 smithi071 ceph-mon[30002]: osdmap e428: 12 total, 12 up, 11 in 2024-09-17T16:22:01.073 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:01.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:00 smithi173 ceph-mon[30530]: pgmap v962: 17 pgs: 17 active+clean; 606 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:22:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3017406805' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:00 smithi173 ceph-mon[30530]: osdmap e428: 12 total, 12 up, 11 in 2024-09-17T16:22:01.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:01.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:01.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:01.238 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:01.238 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:01.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:01.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:01.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:01.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:01.452 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:01.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:01.494 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:22:01.495 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:22:01.517 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:22:01.517 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:22:01.539 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:22:01.540 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:22:01.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:01.562 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:22:01.563 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:22:01.580 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:01.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:01.589 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:22:01.590 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:22:01.614 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:22:01.614 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:22:01.640 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:22:01.641 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:22:01.659 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:22:01.660 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:22:01.681 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:22:01.682 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:22:01.682 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:01.703 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:01.704 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:22:01.705 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:22:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:01.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:01.724 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:22:01.724 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:22:01.745 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:22:01.746 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:22:01.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:01.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:01.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:01.902 INFO:teuthology.orchestra.run.smithi017.stderr:set 5.5 pg_upmap mapping to [10,4,3] 2024-09-17T16:22:01.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:01.930 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 15792 15776 82.1592 62.0625 0.0017277 0.0121518 2024-09-17T16:22:01.933 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:22:01.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:01.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:01.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:02.053 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:02.075 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:02.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:02.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:02.188 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:22:02.201 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:02.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:02.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/492011514' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.5", "id": [10, 4, 3]} : dispatch 2024-09-17T16:22:02.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4177809960' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:02.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:02.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/492011514' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.5", "id": [10, 4, 3]} : dispatch 2024-09-17T16:22:02.309 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4177809960' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:02.324 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:02.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:02.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/492011514' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.5", "id": [10, 4, 3]} : dispatch 2024-09-17T16:22:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4177809960' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:02.426 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:02.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:02.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:02.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:02.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:02.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:02.674 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:02.694 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:02.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:02.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:02.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:22:02.796 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:02.818 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:02.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:02.869 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:22:02.869 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:22:02.869 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:22:02.869 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:22:02.869 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.869 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(2.3 158/155 [PushReplyOp(2:c6916317:::benchmark_data_smithi173_55121_object76243:head)])", 2024-09-17T16:22:02.869 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:22:02.869 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 554.13948047899999, 2024-09-17T16:22:02.869 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45253040700000002, 2024-09-17T16:22:02.869 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.870 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.870 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:22:02.870 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.870 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.870 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.870 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:22:02.870 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.870 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.870 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723047+0000", 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723048+0000", 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4300000000000003e-07 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:48.723052+0000", 2024-09-17T16:22:02.871 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9310000000000001e-06 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175226+0000", 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45217462899999999 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175233+0000", 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4099999999999996e-06 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.872 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175322+0000", 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9011000000000005e-05 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:49.175578+0000", 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025588299999999999 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.873 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:97425 2.3 2:c4562d4f:::benchmark_data_smithi173_55121_object20936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 549.01545650200001, 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29697053200000001, 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 97425 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.874 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847071+0000", 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.875 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847072+0000", 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1399999999999997e-07 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847074+0000", 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7329999999999999e-06 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847082+0000", 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6830000000000008e-06 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.847088+0000", 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0499999999999997e-06 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848028+0000", 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093989799999999999 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848133+0000", 2024-09-17T16:22:02.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000104874 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.848231+0000", 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8121999999999993e-05 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:53.849551+0000", 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00131985 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143633+0000", 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29408256900000002 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143940+0000", 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030690399999999999 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.143994+0000", 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3161999999999997e-05 2024-09-17T16:22:02.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.880 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.880 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.880 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:12:54.144042+0000", 2024-09-17T16:22:02.880 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8072999999999999e-05 2024-09-17T16:22:02.880 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.880 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.880 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.880 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.880 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.880 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:152907 2.3 2:c65c8e4e:::benchmark_data_smithi173_55121_object76418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 489.06510783800002, 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87037030500000001, 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152907 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.881 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797420+0000", 2024-09-17T16:22:02.882 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4099999999999999e-07 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797423+0000", 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797430+0000", 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1119999999999998e-06 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.883 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797436+0000", 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6189999999999999e-06 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797479+0000", 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3050999999999998e-05 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797597+0000", 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000117637 2024-09-17T16:22:02.884 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797696+0000", 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9079999999999993e-05 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.799730+0000", 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020345239999999998 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.885 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.801020+0000", 2024-09-17T16:22:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012897270000000001 2024-09-17T16:22:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667698+0000", 2024-09-17T16:22:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.866677794 2024-09-17T16:22:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.886 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667733+0000", 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4953999999999998e-05 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667790+0000", 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7265999999999997e-05 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.887 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDelete(3.9 e278,273 3:90341f46:::benchmark_data_smithi173_66272_object7671:head,254'4892)", 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 299.71104698800002, 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30541183, 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.888 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151481+0000", 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5000000000000001e-07 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.889 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.151489+0000", 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.52e-06 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456713+0000", 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.305224035 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456737+0000", 2024-09-17T16:22:02.890 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.429e-05 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456817+0000", 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9598000000000006e-05 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456828+0000", 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0740000000000001e-05 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.891 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.456892+0000", 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4796999999999996e-05 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:136335 3.5 3:aa93c047:::benchmark_data_smithi173_66272_object64379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:09.807143+0000", 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 293.05538485900001, 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37544051899999997, 2024-09-17T16:22:02.892 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 136335 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807143+0000", 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.893 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807143+0000", 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807144+0000", 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6899999999999999e-06 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.894 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807151+0000", 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6880000000000002e-06 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807166+0000", 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4883e-05 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807173+0000", 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1690000000000003e-06 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.895 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807229+0000", 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5337999999999998e-05 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807354+0000", 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000125007 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.807468+0000", 2024-09-17T16:22:02.896 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011404900000000001 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.811073+0000", 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036052879999999999 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:09.811379+0000", 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030657599999999997 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.897 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:10.182478+0000", 2024-09-17T16:22:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37109837699999998 2024-09-17T16:22:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:10.182538+0000", 2024-09-17T16:22:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9840000000000003e-05 2024-09-17T16:22:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.898 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:10.182583+0000", 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5614000000000001e-05 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:176912 4.2 4:44b32f3d:::benchmark_data_smithi173_77474_object74737:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e383)", 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:42.499251+0000", 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 80.363276674000005, 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34693342300000002, 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.899 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 176912 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499251+0000", 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.900 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499251+0000", 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499253+0000", 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.874e-06 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.901 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499260+0000", 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.187e-06 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499285+0000", 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5329000000000001e-05 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499301+0000", 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5590000000000002e-05 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499413+0000", 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011231 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499434+0000", 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0835e-05 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.846068+0000", 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34663389999999999 2024-09-17T16:22:02.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.846184+0000", 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011639800000000001 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:176918 4.2 4:4746b7f5:::benchmark_data_smithi173_77474_object74743:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e383)", 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:42.499325+0000", 2024-09-17T16:22:02.904 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 80.363202254000001, 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37077265799999998, 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 176918 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499325+0000", 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499325+0000", 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499326+0000", 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6400000000000002e-07 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499329+0000", 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0570000000000001e-06 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499346+0000", 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7367000000000002e-05 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499353+0000", 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6599999999999998e-06 2024-09-17T16:22:02.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499521+0000", 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016777900000000001 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499530+0000", 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7030000000000001e-06 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.846482+0000", 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34695168900000001 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.846605+0000", 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000122402 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9,11", 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.846744+0000", 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013938000000000001 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.858906+0000", 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.012162506999999999 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.867247+0000", 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0083402130000000008 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.867355+0000", 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000108434 2024-09-17T16:22:02.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.869979+0000", 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002623785 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.870021+0000", 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1816000000000001e-05 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.911 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.870098+0000", 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7302e-05 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(4.2 389/383 [PushReplyOp(4:4e55586c:::benchmark_data_smithi173_77474_object100589:head)])", 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:53.998631+0000", 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 68.863896143000005, 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44632730900000001, 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.912 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.998631+0000", 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.998631+0000", 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.913 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.998632+0000", 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.17e-07 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.998634+0000", 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9879999999999999e-06 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.442705+0000", 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44407097200000001 2024-09-17T16:22:02.914 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.915 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.915 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.916 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.442709+0000", 2024-09-17T16:22:02.916 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5210000000000004e-06 2024-09-17T16:22:02.916 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.916 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.916 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.916 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.444667+0000", 2024-09-17T16:22:02.916 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019580940000000001 2024-09-17T16:22:02.916 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.916 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.444959+0000", 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00029161699999999998 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(4.a 391/383 [PushOp(4:5f33903b:::benchmark_data_smithi173_77474_object97064:head, version: 347'6156, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:5f33903b:::benchmark_data_smithi173_77474_object97064:head@347'6156, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:55.758638+0000", 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 67.103889631000001, 2024-09-17T16:22:02.917 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70397542800000001, 2024-09-17T16:22:02.918 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.918 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.918 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:22:02.918 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.918 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.918 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.918 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:55.758638+0000", 2024-09-17T16:22:02.918 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.918 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:55.758638+0000", 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:55.758638+0000", 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.99e-07 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:55.758703+0000", 2024-09-17T16:22:02.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4610999999999995e-05 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:56.462316+0000", 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70361271599999997 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:56.462343+0000", 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7909999999999999e-05 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:56.462460+0000", 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000116815 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:56.462470+0000", 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5799999999999998e-06 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:56.462613+0000", 2024-09-17T16:22:02.921 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000143597 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:186256 4.2 4:400b56b9:::benchmark_data_smithi173_77474_object84081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:58.015996+0000", 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 64.846531471000006, 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31989559200000001, 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.922 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 186256 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.015996+0000", 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.923 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.015996+0000", 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.015998+0000", 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4589999999999999e-06 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.016013+0000", 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5048999999999999e-05 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.924 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.016030+0000", 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7227999999999998e-05 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.016038+0000", 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5189999999999992e-06 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.016103+0000", 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4522000000000003e-05 2024-09-17T16:22:02.925 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.016231+0000", 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000128169 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.016350+0000", 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011852400000000001 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.020050+0000", 2024-09-17T16:22:02.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0037007609999999999 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.020451+0000", 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00040046800000000002 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.335825+0000", 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31537411300000001 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.927 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.335862+0000", 2024-09-17T16:22:02.928 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7067000000000003e-05 2024-09-17T16:22:02.928 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.928 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.928 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.928 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.335892+0000", 2024-09-17T16:22:02.928 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9713000000000002e-05 2024-09-17T16:22:02.928 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.928 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.928 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:186256 4.2 e393/391 ondisk, result = 0)", 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:58.018782+0000", 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 64.843745163999998, 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31711379299999998, 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.018782+0000", 2024-09-17T16:22:02.929 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.018782+0000", 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.018783+0000", 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9500000000000001e-07 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.930 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.018787+0000", 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7740000000000002e-06 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.335765+0000", 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31697804299999999 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.335771+0000", 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.015e-06 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.335814+0000", 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3198000000000002e-05 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.335820+0000", 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6470000000000003e-06 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.335896+0000", 2024-09-17T16:22:02.932 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6321000000000003e-05 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:186293 4.f e393/383 4:ffe1d3e0:::benchmark_data_smithi173_77474_object84118:head v 393'11582, mlcod=393'11579)", 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:58.185863+0000", 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 64.676664373999998, 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.42666476800000003, 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.933 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.185863+0000", 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.185863+0000", 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.934 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.185865+0000", 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.277e-06 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.185873+0000", 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.371e-06 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.610471+0000", 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.42459858 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.610476+0000", 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7149999999999997e-06 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.610557+0000", 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1149000000000005e-05 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.610573+0000", 2024-09-17T16:22:02.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5951e-05 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.612500+0000", 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019271710000000001 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.612528+0000", 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7554000000000001e-05 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.937 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:197771 4.1 4:8a203159:::benchmark_data_smithi173_77474_object95596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:13.590237+0000", 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 49.272290966, 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32554565600000002, 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 197771 2024-09-17T16:22:02.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590237+0000", 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590237+0000", 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.939 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590238+0000", 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.271e-06 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590243+0000", 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1989999999999997e-06 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590254+0000", 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0638e-05 2024-09-17T16:22:02.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590259+0000", 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0069999999999997e-06 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590300+0000", 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1007999999999998e-05 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590385+0000", 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5731999999999998e-05 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590473+0000", 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7978000000000004e-05 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.594248+0000", 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0037743640000000001 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.594779+0000", 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00053105800000000005 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.915695+0000", 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32091583899999998 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.915732+0000", 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6918000000000001e-05 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.915782+0000", 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0643999999999998e-05 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:204340 4.a e408/393 4:583138b7:::benchmark_data_smithi173_77474_object102165:head v 408'12934, mlcod=408'12929)", 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:23.617934+0000", 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 39.244593936000001, 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57527535799999996, 2024-09-17T16:22:02.944 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:23.617934+0000", 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:23.617934+0000", 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.945 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:23.617934+0000", 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8299999999999996e-07 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:23.617941+0000", 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3990000000000001e-06 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.190571+0000", 2024-09-17T16:22:02.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57263039199999999 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.190579+0000", 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9230000000000002e-06 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.190659+0000", 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9571000000000001e-05 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.190686+0000", 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7177000000000001e-05 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.193171+0000", 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024847900000000002 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.193209+0000", 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8322999999999999e-05 2024-09-17T16:22:02.948 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:2247 5.d 5:b9fd46b8:::benchmark_data_smithi173_90735_object2246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:51.473532+0000", 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.388996064000001, 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29781685499999999, 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:22:02.949 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 2247 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.473532+0000", 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.473532+0000", 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.473533+0000", 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1489999999999999e-06 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.473584+0000", 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1369000000000002e-05 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.473596+0000", 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2305e-05 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.473602+0000", 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8259999999999998e-06 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.473643+0000", 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0972e-05 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.473725+0000", 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2087000000000002e-05 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.473821+0000", 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5310999999999996e-05 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.474166+0000", 2024-09-17T16:22:02.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000345776 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.478734+0000", 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0045676559999999998 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.771275+0000", 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29254132599999999 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.771308+0000", 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2926999999999998e-05 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.771348+0000", 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0151000000000002e-05 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:2250 5.d 5:b6725c8b:::benchmark_data_smithi173_90735_object2249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:51.477701+0000", 2024-09-17T16:22:02.955 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.384827078000001, 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.296583072, 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 2250 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.477701+0000", 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.477701+0000", 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.477701+0000", 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7100000000000011e-07 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.477728+0000", 2024-09-17T16:22:02.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6806999999999999e-05 2024-09-17T16:22:02.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.477739+0000", 2024-09-17T16:22:02.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0596e-05 2024-09-17T16:22:02.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.477756+0000", 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7598e-05 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.477798+0000", 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1835999999999997e-05 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.477872+0000", 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3679999999999999e-05 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.477956+0000", 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3881000000000005e-05 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.478364+0000", 2024-09-17T16:22:02.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00040803999999999998 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.503179+0000", 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.024814673999999998 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.774194+0000", 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27101582099999999 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.774253+0000", 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8748000000000001e-05 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.774284+0000", 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.042e-05 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:2279 5.f 5:f5772ab5:::benchmark_data_smithi173_90735_object2278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-09-17T16:22:02.962 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:51.502142+0000", 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.360386007000001, 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25316676799999999, 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 2279 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.502142+0000", 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.502142+0000", 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.502143+0000", 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.299e-06 2024-09-17T16:22:02.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.502197+0000", 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4128000000000001e-05 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.502208+0000", 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1406999999999999e-05 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.502214+0000", 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.553e-06 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.502256+0000", 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1644999999999999e-05 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.502338+0000", 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2824000000000005e-05 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,11", 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.502427+0000", 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8758999999999995e-05 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.502805+0000", 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00037788900000000001 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.503108+0000", 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030309400000000001 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.755206+0000", 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25209781999999997 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.755262+0000", 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5809e-05 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.755308+0000", 2024-09-17T16:22:02.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6541e-05 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.26158.0:7069 5.4 e425/419 5:2bdd4242:::benchmark_data_smithi173_90735_object7068:head v 425'402, mlcod=425'397)", 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:54.745526+0000", 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.1170019530000008, 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23819019799999999, 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.969 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.745526+0000", 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.745526+0000", 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.745526+0000", 2024-09-17T16:22:02.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.745618+0000", 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1179000000000002e-05 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.983028+0000", 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.237410695 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.983034+0000", 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0589999999999999e-06 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.983082+0000", 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8075000000000003e-05 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.983098+0000", 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5673999999999999e-05 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.983678+0000", 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058005899999999996 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.983716+0000", 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7688000000000002e-05 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.26158.0:7076 5.4 e425/419 5:2e3c82d7:::benchmark_data_smithi173_90735_object7075:head v 425'403, mlcod=425'398)", 2024-09-17T16:22:02.973 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:54.755076+0000", 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.1074511099999995, 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.231662599, 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.755076+0000", 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.755076+0000", 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.755077+0000", 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0200000000000002e-07 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.755161+0000", 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3467000000000006e-05 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.986161+0000", 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23100030599999999 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.986165+0000", 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7919999999999999e-06 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.986209+0000", 2024-09-17T16:22:02.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3962999999999999e-05 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.986238+0000", 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9360999999999999e-05 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.986700+0000", 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00046244199999999997 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:54.986739+0000", 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8665999999999999e-05 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:15891 5.f 5:f1ab552a:::benchmark_data_smithi173_90735_object15890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:02.189899+0000", 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.67262856400000004, 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23677706500000001, 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:02.978 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 15891 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.189899+0000", 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:02.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.189899+0000", 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.189901+0000", 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.556e-06 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.189944+0000", 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3711999999999998e-05 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.189957+0000", 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2992e-05 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.189964+0000", 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6579999999999996e-06 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.190013+0000", 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9314999999999998e-05 2024-09-17T16:22:02.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.190105+0000", 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1478999999999996e-05 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,11", 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.190194+0000", 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9160999999999996e-05 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:02.982 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.191030+0000", 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00083580700000000004 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.191428+0000", 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00039821600000000001 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.426600+0000", 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23517169399999999 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.426634+0000", 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4551000000000002e-05 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.426676+0000", 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1924e-05 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:02.984 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:22:02.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:02.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 15957 15941 76.6326 10.3125 0.00230152 0.0123113 2024-09-17T16:22:03.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:03.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:03.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:03.136 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:03.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:03.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:02 smithi017 ceph-mon[27571]: pgmap v964: 17 pgs: 17 active+clean; 867 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-17T16:22:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/492011514' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.5", "id": [10, 4, 3]}]': finished 2024-09-17T16:22:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:02 smithi017 ceph-mon[27571]: osdmap e429: 12 total, 12 up, 11 in 2024-09-17T16:22:03.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:03.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:03.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:02 smithi071 ceph-mon[30002]: pgmap v964: 17 pgs: 17 active+clean; 867 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-17T16:22:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/492011514' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.5", "id": [10, 4, 3]}]': finished 2024-09-17T16:22:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:02 smithi071 ceph-mon[30002]: osdmap e429: 12 total, 12 up, 11 in 2024-09-17T16:22:03.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:03.385 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:03.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:03.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:03.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:02 smithi173 ceph-mon[30530]: pgmap v964: 17 pgs: 17 active+clean; 867 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-17T16:22:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/492011514' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.5", "id": [10, 4, 3]}]': finished 2024-09-17T16:22:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:02 smithi173 ceph-mon[30530]: osdmap e429: 12 total, 12 up, 11 in 2024-09-17T16:22:03.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:03.516 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:03.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:03.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:03.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:03.640 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:03.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:03.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:03.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:03.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:03.870 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:03.892 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:03.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:03.930 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 15976 15960 71.2436 1.1875 0.00174559 0.0123488 2024-09-17T16:22:03.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:04.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:04.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:04.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:04.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:04.172 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:03 smithi071 ceph-mon[30002]: osdmap e430: 12 total, 12 up, 11 in 2024-09-17T16:22:04.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:22:04.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:03 smithi017 ceph-mon[27571]: osdmap e430: 12 total, 12 up, 11 in 2024-09-17T16:22:04.249 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:04.272 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:04.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:04.374 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:04.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:03 smithi173 ceph-mon[30530]: osdmap e430: 12 total, 12 up, 11 in 2024-09-17T16:22:04.396 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:04.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:04.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:04.440 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:22:04.498 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:04.521 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:04.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:04.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:04.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:04.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:04.751 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:04.779 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:04.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:04.881 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:04.916 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:04.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:04.923 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:22:04.930 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 16492 16476 68.6438 32.25 0.0108357 0.0145589 2024-09-17T16:22:04.943 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:22:04.943 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:22:04.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.944 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:16513 5.3 5:c9d5a066:::benchmark_data_smithi173_90735_object16512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-09-17T16:22:04.944 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:04.939167+0000", 2024-09-17T16:22:04.944 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.001070164, 2024-09-17T16:22:04.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010920319999999999, 2024-09-17T16:22:04.944 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:04.944 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:04.944 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:22:04.944 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:04.944 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 16513 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.939167+0000", 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.939167+0000", 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:04.945 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.939169+0000", 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.708e-06 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.939248+0000", 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9109000000000004e-05 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:04.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.939265+0000", 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6915999999999999e-05 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.939273+0000", 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2069999999999995e-06 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.939363+0000", 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0351000000000003e-05 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.939497+0000", 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013346199999999999 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.939646+0000", 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000149331 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:04.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.949 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:16499 5.3 5:c8394cd9:::benchmark_data_smithi173_90735_object16498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-09-17T16:22:04.949 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:04.932301+0000", 2024-09-17T16:22:04.949 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0079366360000000004, 2024-09-17T16:22:04.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0080031930000000005, 2024-09-17T16:22:04.949 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 16499 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.932301+0000", 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:04.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.932301+0000", 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.932302+0000", 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.637e-06 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:04.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.932363+0000", 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1147000000000003e-05 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.932383+0000", 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9514999999999999e-05 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.932394+0000", 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0737000000000001e-05 2024-09-17T16:22:04.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.932468+0000", 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4340999999999999e-05 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.932600+0000", 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000132099 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:22:04.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.932757+0000", 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015650399999999999 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.933738+0000", 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000981448 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.934145+0000", 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00040690700000000003 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:04.954 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:16503 5.2 5:4da02354:::benchmark_data_smithi173_90735_object16502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:04.934311+0000", 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0059257429999999998, 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0060640690000000001, 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 16503 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.955 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.934311+0000", 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.934311+0000", 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.934313+0000", 2024-09-17T16:22:04.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1829999999999999e-06 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.934369+0000", 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6781999999999999e-05 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.934383+0000", 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.312e-05 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:04.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.934389+0000", 2024-09-17T16:22:04.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0780000000000001e-06 2024-09-17T16:22:04.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:04.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.934442+0000", 2024-09-17T16:22:04.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3418999999999998e-05 2024-09-17T16:22:04.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:04.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.934564+0000", 2024-09-17T16:22:04.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000121585 2024-09-17T16:22:04.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:22:04.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.934708+0000", 2024-09-17T16:22:04.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000144139 2024-09-17T16:22:04.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:04.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:04.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:04.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:04.935479+0000", 2024-09-17T16:22:04.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00077123899999999995 2024-09-17T16:22:04.960 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:04.960 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:04.960 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:04.960 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:04.960 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:22:04.960 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 3 2024-09-17T16:22:04.960 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:22:05.018 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:05.044 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:05.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:05.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:05.173 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:05.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:04 smithi017 ceph-mon[27571]: pgmap v967: 17 pgs: 17 active+clean; 935 MiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-17T16:22:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:04 smithi017 ceph-mon[27571]: osdmap e431: 12 total, 12 up, 11 in 2024-09-17T16:22:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:04 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2149669487' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:05.275 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:05.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:22:05.302 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:05.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:05.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:04 smithi071 ceph-mon[30002]: pgmap v967: 17 pgs: 17 active+clean; 935 MiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-17T16:22:05.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:04 smithi071 ceph-mon[30002]: osdmap e431: 12 total, 12 up, 11 in 2024-09-17T16:22:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:04 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2149669487' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:04 smithi173 ceph-mon[30530]: pgmap v967: 17 pgs: 17 active+clean; 935 MiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-17T16:22:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:04 smithi173 ceph-mon[30530]: osdmap e431: 12 total, 12 up, 11 in 2024-09-17T16:22:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:04 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2149669487' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:05.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:05.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:05.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:05.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:05.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:05.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:05.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:05.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:05.669 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:22:05.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:05.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:05.706 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:05.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:05.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:05.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:05.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:05.930 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 18018 18002 70.314 95.375 0.00837834 0.0141416 2024-09-17T16:22:05.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:05.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:05.956 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:22:06.034 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:06.061 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:06.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:06.163 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:06.187 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:06.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:06.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:05 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:22:06.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:05 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:22:06.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2149669487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:22:06.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:05 smithi017 ceph-mon[27571]: osdmap e432: 12 total, 12 up, 11 in 2024-09-17T16:22:06.289 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:06.302 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:22:06.302 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:22:06.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:05 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:22:06.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:05 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:22:06.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2149669487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:22:06.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:05 smithi071 ceph-mon[30002]: osdmap e432: 12 total, 12 up, 11 in 2024-09-17T16:22:06.316 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:06.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:06.328 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:22:06.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:22:06.353 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:22:06.353 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:22:06.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:22:06.378 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:22:06.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:05 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:22:06.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:05 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:22:06.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2149669487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:22:06.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:05 smithi173 ceph-mon[30530]: osdmap e432: 12 total, 12 up, 11 in 2024-09-17T16:22:06.418 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:06.442 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:06.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:06.453 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:22:06.453 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:22:06.479 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:22:06.479 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:22:06.506 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:22:06.506 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:22:06.531 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:22:06.531 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:22:06.545 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:06.555 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:22:06.555 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:22:06.569 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:06.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:06.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:06.578 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:22:06.578 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:22:06.600 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:22:06.600 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:22:06.624 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:22:06.624 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:22:06.633 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:06.633 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:06.671 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:06.696 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:06.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:06.798 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:06.824 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:06.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:06.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:06.930 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 19562 19546 71.8539 96.5 0.00493398 0.0139081 2024-09-17T16:22:06.953 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:06.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:07.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:07.089 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:07.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:07.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:07.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:07.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:07.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:06 smithi017 ceph-mon[27571]: pgmap v970: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.0 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 47 MiB/s wr, 757 op/s; 1908/50685 objects misplaced (3.764%); 96 KiB/s, 1 objects/s recovering 2024-09-17T16:22:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/709541651' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:06 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:22:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:06 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:22:07.262 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:06 smithi071 ceph-mon[30002]: pgmap v970: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.0 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 47 MiB/s wr, 757 op/s; 1908/50685 objects misplaced (3.764%); 96 KiB/s, 1 objects/s recovering 2024-09-17T16:22:07.262 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/709541651' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:07.262 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:06 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:22:07.262 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:06 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:22:07.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:22:07.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:07.378 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:07.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:07.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:06 smithi173 ceph-mon[30530]: pgmap v970: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.0 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 47 MiB/s wr, 757 op/s; 1908/50685 objects misplaced (3.764%); 96 KiB/s, 1 objects/s recovering 2024-09-17T16:22:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/709541651' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:06 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:22:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:06 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:22:07.479 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:07.503 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:07.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:07.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:07.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:07.638 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:07.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:07.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:07.651 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:22:07.667 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:22:07.668 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:22:07.668 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-17T16:22:07.668 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T16:22:07.668 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T16:22:07.668 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-17T16:22:07.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:22:07.740 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:07.766 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:07.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:07.868 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:07.898 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:07.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:07.930 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 21191 21175 73.5178 101.812 0.0622036 0.0135907 2024-09-17T16:22:07.984 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:07.988 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:07.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:07 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:22:07.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:07 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:22:08.000 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:08.024 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:08.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:08.042 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:07 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:22:08.042 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:07 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:22:08.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:08.151 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:08.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:08.238 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:08.238 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:08.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:08.274 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:08.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:08.376 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:08.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:07 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:22:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:07 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:22:08.399 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:08.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:08.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:08.722 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:08.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:08.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:08.849 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:08.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:08.930 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 22388 22372 73.5856 74.8125 0.00176797 0.0135692 2024-09-17T16:22:08.939 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:22:08.951 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:08.977 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:08.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:09.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:08 smithi017 ceph-mon[27571]: pgmap v971: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.1 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 40 MiB/s wr, 632 op/s; 1908/52923 objects misplaced (3.605%); 64 KiB/s, 1 objects/s recovering 2024-09-17T16:22:09.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:08 smithi017 ceph-mon[27571]: osdmap e433: 12 total, 12 up, 11 in 2024-09-17T16:22:09.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2889296709' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:09.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:09.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:09.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:09.202 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:08 smithi071 ceph-mon[30002]: pgmap v971: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.1 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 40 MiB/s wr, 632 op/s; 1908/52923 objects misplaced (3.605%); 64 KiB/s, 1 objects/s recovering 2024-09-17T16:22:09.202 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:08 smithi071 ceph-mon[30002]: osdmap e433: 12 total, 12 up, 11 in 2024-09-17T16:22:09.202 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2889296709' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:09.205 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:09.232 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:09.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:09.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:22:09.271 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:08 smithi173 ceph-mon[30530]: pgmap v971: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.1 GiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 40 MiB/s wr, 632 op/s; 1908/52923 objects misplaced (3.605%); 64 KiB/s, 1 objects/s recovering 2024-09-17T16:22:09.272 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:08 smithi173 ceph-mon[30530]: osdmap e433: 12 total, 12 up, 11 in 2024-09-17T16:22:09.272 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2889296709' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:09.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:09.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:09.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:09.390 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:22:09.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:09.494 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:09.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:09.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:09.597 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:09.621 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:09.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:09.632 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:22:09.632 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T16:22:09.632 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:22:09.724 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:09.748 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:09.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:09.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:09.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:09.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:09.930 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:22:09.929497+0000 min lat: 0.00103987 max lat: 2.82824 avg lat: 0.0134364 2024-09-17T16:22:09.931 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:22:09.931 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 23815 23799 74.3654 89.1875 0.00211198 0.0134364 2024-09-17T16:22:09.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:09.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:09 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:22:09.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:09 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:22:09.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2889296709' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:09.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:09 smithi017 ceph-mon[27571]: osdmap e434: 12 total, 12 up, 11 in 2024-09-17T16:22:09.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:09 smithi017 ceph-mon[27571]: osdmap e435: 12 total, 12 up, 11 in 2024-09-17T16:22:10.002 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:10.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:10.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:10.135 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:10.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:10.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:09 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:22:10.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:09 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:22:10.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2889296709' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:10.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:09 smithi173 ceph-mon[30530]: osdmap e434: 12 total, 12 up, 11 in 2024-09-17T16:22:10.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:09 smithi173 ceph-mon[30530]: osdmap e435: 12 total, 12 up, 11 in 2024-09-17T16:22:10.237 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:10.262 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:10.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:10.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:09 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:22:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:09 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:22:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2889296709' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:09 smithi071 ceph-mon[30002]: osdmap e434: 12 total, 12 up, 11 in 2024-09-17T16:22:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:09 smithi071 ceph-mon[30002]: osdmap e435: 12 total, 12 up, 11 in 2024-09-17T16:22:10.365 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:10.393 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:10.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:10.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:10.519 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:10.519 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:10.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:10.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:10.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:10.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:10.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:10.747 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:10.770 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:10.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:10.872 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:10.897 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:10.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:10.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:10.931 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 25079 25063 74.5857 79 0.00402564 0.0133983 2024-09-17T16:22:10.990 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:10.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:11.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:11.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:11.048 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:10 smithi173 ceph-mon[30530]: pgmap v975: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.1 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 36 MiB/s wr, 582 op/s; 1908/54825 objects misplaced (3.480%) 2024-09-17T16:22:11.132 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:11.159 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:11.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:10 smithi017 ceph-mon[27571]: pgmap v975: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.1 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 36 MiB/s wr, 582 op/s; 1908/54825 objects misplaced (3.480%) 2024-09-17T16:22:11.261 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:11.287 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:11.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:11.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:22:11.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:10 smithi071 ceph-mon[30002]: pgmap v975: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.1 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 36 MiB/s wr, 582 op/s; 1908/54825 objects misplaced (3.480%) 2024-09-17T16:22:11.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:11.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:11.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:11.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:11.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:11.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:11.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:11.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:11.584 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:22:11.584 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:22:11.608 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:22:11.609 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:22:11.634 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:22:11.634 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:22:11.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:11.661 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:22:11.661 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:22:11.684 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:22:11.684 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:22:11.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:11.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:11.702 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:11.710 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:22:11.710 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:22:11.735 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:22:11.735 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:22:11.762 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:22:11.762 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:22:11.790 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:22:11.790 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:22:11.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:11.814 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:22:11.814 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:22:11.821 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:11.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:11.839 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:22:11.840 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:22:11.865 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:22:11.866 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:22:11.923 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:11.930 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 26698 26682 75.7945 101.188 0.0204878 0.0131876 2024-09-17T16:22:11.949 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:11.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:12.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:12.074 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:12.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:12.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/607256324' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:12.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:12.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:12.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:12.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/607256324' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:12.308 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:22:12.308 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:22:12.308 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:22:12.308 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:22:12.308 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:22:12.312 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:12.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/607256324' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:12.337 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:12.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:12.439 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:12.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:22:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:12.465 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:12.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:12.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:12.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:12.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:12.595 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:12.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:12.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:12.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:12.774 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:12.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:12.853 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:12.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:12.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:12.930 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 28241 28225 76.6917 96.4375 0.00428751 0.0130335 2024-09-17T16:22:12.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:12.984 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:12.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:13.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:12 smithi173 ceph-mon[30530]: pgmap v976: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.5 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s; 908/72144 objects misplaced (1.259%); 5.2 MiB/s, 83 objects/s recovering 2024-09-17T16:22:13.152 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:13.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:12 smithi017 ceph-mon[27571]: pgmap v976: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.5 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s; 908/72144 objects misplaced (1.259%); 5.2 MiB/s, 83 objects/s recovering 2024-09-17T16:22:13.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:22:13.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:13.281 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:13.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:13.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:12 smithi071 ceph-mon[30002]: pgmap v976: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.5 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s; 908/72144 objects misplaced (1.259%); 5.2 MiB/s, 83 objects/s recovering 2024-09-17T16:22:13.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:13.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:13.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:13.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:13.540 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:13.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:13.600 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:13.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:13.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:13.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:13.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:13.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:13.795 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:13.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:13.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:13.922 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:13.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:13.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 29866 29850 77.6728 101.562 0.00714974 0.0128589 2024-09-17T16:22:13.952 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:22:14.025 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:14.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:14.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:14.049 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2140673150' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:22:14.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:14.159 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:22:14.159 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:22:14.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:22:14.159 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:22:14.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24550.0:92960 2.b 2:d44e9bf2:::benchmark_data_smithi173_55121_object16471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.636820+0000", 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 565.51950958199996, 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63467018600000003, 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24550", 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 92960 2024-09-17T16:22:14.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636820+0000", 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636820+0000", 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636822+0000", 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1969999999999999e-06 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636830+0000", 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4490000000000002e-06 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636849+0000", 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9029999999999999e-05 2024-09-17T16:22:14.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636859+0000", 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0183999999999999e-05 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.636929+0000", 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0017000000000004e-05 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.637101+0000", 2024-09-17T16:22:14.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017157500000000001 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.637326+0000", 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022522300000000001 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640616+0000", 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032902130000000002 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004722+0000", 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36410627899999998 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269780+0000", 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.265057773 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269830+0000", 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9772999999999999e-05 2024-09-17T16:22:14.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.271490+0000", 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001660473 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:92960 2.b e158/155 ondisk, result = 0)", 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.640613+0000", 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 565.51571706100003, 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36412291800000002, 2024-09-17T16:22:14.166 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640613+0000", 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640613+0000", 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640614+0000", 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2500000000000004e-07 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640619+0000", 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5350000000000004e-06 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004492+0000", 2024-09-17T16:22:14.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36387301399999999 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004506+0000", 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4403000000000001e-05 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004662+0000", 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015578400000000001 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004694+0000", 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1482000000000001e-05 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.004736+0000", 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1774999999999999e-05 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24550.0:92960 2.b e158/155 ondisk, result = 0)", 2024-09-17T16:22:14.170 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:12:48.640951+0000", 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 565.51537819199996, 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.630555843, 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640951+0000", 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640951+0000", 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640952+0000", 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.54e-07 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:48.640960+0000", 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9570000000000002e-06 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.266220+0000", 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62525987900000002 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.266226+0000", 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9800000000000003e-06 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269764+0000", 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035381520000000001 2024-09-17T16:22:14.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.269773+0000", 2024-09-17T16:22:14.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5450000000000003e-06 2024-09-17T16:22:14.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:12:49.271507+0000", 2024-09-17T16:22:14.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001734576 2024-09-17T16:22:14.175 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.175 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.175 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24550.0:135561 2.6 e191/104 2:612899dc:::benchmark_data_smithi173_55121_object59072:head v 191'4677, mlcod=191'4675)", 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:13:35.921733+0000", 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 518.23459667300006, 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36610857200000002, 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921733+0000", 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921733+0000", 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921734+0000", 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:22:14.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:35.921739+0000", 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2630000000000003e-06 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.285858+0000", 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.364118357 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.285878+0000", 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0152999999999998e-05 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.286066+0000", 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000188016 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.286106+0000", 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0309999999999999e-05 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.287803+0000", 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016966290000000001 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:13:36.287842+0000", 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8945000000000001e-05 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.180 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(3.9 240/239 [PushReplyOp(3:9e32e3c9:::benchmark_data_smithi173_66272_object1144:head)])", 2024-09-17T16:22:14.181 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:22:14.181 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 419.07679211300001, 2024-09-17T16:22:14.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90253544399999996, 2024-09-17T16:22:14.181 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.181 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.181 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:22:14.181 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6799999999999999e-07 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079543+0000", 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6809999999999997e-06 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979238+0000", 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89969455399999998 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979247+0000", 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5659999999999996e-06 2024-09-17T16:22:14.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979307+0000", 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9338e-05 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.982073+0000", 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027664370000000001 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124278 3.9 3.d65c03a9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.930793+0000", 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 325.22553712500002, 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42595735800000001, 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124278 2024-09-17T16:22:14.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930793+0000", 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930793+0000", 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930794+0000", 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4890000000000001e-06 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930801+0000", 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5699999999999998e-06 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930816+0000", 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4953000000000001e-05 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930823+0000", 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6279999999999996e-06 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930874+0000", 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0333000000000001e-05 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930880+0000", 2024-09-17T16:22:14.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8090000000000004e-06 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.356716+0000", 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42583603799999997 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.356750+0000", 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3538000000000003e-05 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.189 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124282 3.9 3.c1fc19b9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.955157+0000", 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 325.20117250800001, 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.402016861, 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124282 2024-09-17T16:22:14.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955157+0000", 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955157+0000", 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955158+0000", 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.274e-06 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955164+0000", 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9240000000000004e-06 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955175+0000", 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0222e-05 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955181+0000", 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1709999999999999e-06 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955232+0000", 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0927999999999998e-05 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:22:14.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955240+0000", 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4039999999999995e-06 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.357130+0000", 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40189000200000002 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.357174+0000", 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3936000000000001e-05 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.194 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124278 3.9 3:95c03a6b:::benchmark_data_smithi173_66272_object52322:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:50.410673+0000", 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 323.74565653600001, 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47593780800000002, 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124278 2024-09-17T16:22:14.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410673+0000", 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410673+0000", 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410675+0000", 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6199999999999999e-06 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410689+0000", 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4664000000000001e-05 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410704+0000", 2024-09-17T16:22:14.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4174e-05 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410714+0000", 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0659000000000001e-05 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411252+0000", 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053726700000000002 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:22:14.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411263+0000", 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1297999999999999e-05 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.870528+0000", 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45926520500000001 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.870665+0000", 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013726100000000001 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7,11", 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.870758+0000", 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2767000000000001e-05 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875501+0000", 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047434310000000002 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875552+0000", 2024-09-17T16:22:14.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0096000000000003e-05 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875561+0000", 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0690000000000008e-06 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.885797+0000", 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010236413999999999 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.885869+0000", 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2149000000000003e-05 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.886611+0000", 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074173399999999997 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124282 3.9 3:9d983f83:::benchmark_data_smithi173_66272_object52326:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:50.410739+0000", 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 323.74559018899998, 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47645007299999997, 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124282 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410739+0000", 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410739+0000", 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410740+0000", 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1099999999999995e-07 2024-09-17T16:22:14.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410746+0000", 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0170000000000002e-06 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410754+0000", 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6499999999999996e-06 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410760+0000", 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5309999999999998e-06 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411348+0000", 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00058806500000000003 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411359+0000", 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0681e-05 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875583+0000", 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46422414299999998 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875678+0000", 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4757000000000001e-05 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7,11", 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875760+0000", 2024-09-17T16:22:14.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2109999999999998e-05 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.878759+0000", 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029989209999999999 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.879450+0000", 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069113699999999998 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:14.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.879505+0000", 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4765999999999997e-05 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.886692+0000", 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0071871299999999999 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.886735+0000", 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3312999999999997e-05 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.887190+0000", 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045414099999999997 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:198614 4.2 e402/391 4:455c645f:::benchmark_data_smithi173_77474_object96439:head v 402'12215, mlcod=402'12211)", 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:15.442755+0000", 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.713574635999997, 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078778648000000007, 2024-09-17T16:22:14.210 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:15.442755+0000", 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:15.442755+0000", 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:15.442756+0000", 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3960000000000001e-06 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:15.442768+0000", 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1952e-05 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:15.519017+0000", 2024-09-17T16:22:14.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.076248783000000001 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:15.519026+0000", 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.037e-06 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:15.519136+0000", 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010983 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:15.519160+0000", 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3595e-05 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:15.521496+0000", 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023364060000000001 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:15.521534+0000", 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7648999999999998e-05 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.214 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:5809 5.e e424/419 5:7a13a1a9:::benchmark_data_smithi173_90735_object5808:head v 424'349, mlcod=424'346)", 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:53.972354+0000", 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 20.183976000000001, 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041619595000000002, 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:53.972354+0000", 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:53.972354+0000", 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:53.972355+0000", 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4130000000000001e-06 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:53.972382+0000", 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6896000000000001e-05 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.013473+0000", 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041091210000000003 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.013480+0000", 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2180000000000002e-06 2024-09-17T16:22:14.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.013538+0000", 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7503999999999997e-05 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.013555+0000", 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6646e-05 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.013945+0000", 2024-09-17T16:22:14.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039040100000000002 2024-09-17T16:22:14.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.013973+0000", 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8306999999999999e-05 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:5876 5.e e424/419 5:79a77f38:::benchmark_data_smithi173_90735_object5875:head v 424'353, mlcod=424'350)", 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:54.061024+0000", 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 20.095305226000001, 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.066736820000000002, 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.220 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.061024+0000", 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.061024+0000", 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.061025+0000", 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3e-07 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.061044+0000", 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.857e-05 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.127223+0000", 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.066178925 2024-09-17T16:22:14.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.127230+0000", 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6830000000000008e-06 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.127283+0000", 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2673999999999997e-05 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.127305+0000", 2024-09-17T16:22:14.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1783000000000001e-05 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.127726+0000", 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042170700000000001 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.127761+0000", 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4848000000000002e-05 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.224 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:5916 5.a e424/419 5:5f1e5d8e:::benchmark_data_smithi173_90735_object5915:head v 424'364, mlcod=424'361)", 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:54.102014+0000", 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 20.054315293999998, 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.044236349000000001, 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.102014+0000", 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.102014+0000", 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.102015+0000", 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4000000000000001e-07 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.102111+0000", 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6323000000000006e-05 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.145732+0000", 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.043620561000000002 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.145747+0000", 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4973999999999999e-05 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.145811+0000", 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4412999999999995e-05 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.145826+0000", 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4141000000000001e-05 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.146223+0000", 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039765800000000001 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.146251+0000", 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7538999999999998e-05 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.229 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:5972 5.e e424/419 5:7a199ed2:::benchmark_data_smithi173_90735_object5971:head v 424'357, mlcod=424'354)", 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:54.166041+0000", 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 19.990288522, 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041146518999999999, 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.166041+0000", 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.166041+0000", 2024-09-17T16:22:14.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.166042+0000", 2024-09-17T16:22:14.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1399999999999997e-07 2024-09-17T16:22:14.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.166060+0000", 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8539999999999999e-05 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.206714+0000", 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.040653671000000002 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.206722+0000", 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7640000000000005e-06 2024-09-17T16:22:14.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.206762+0000", 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9965000000000001e-05 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.206776+0000", 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4314999999999999e-05 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.207156+0000", 2024-09-17T16:22:14.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037961800000000002 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.207188+0000", 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2032000000000001e-05 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.234 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:7082 5.e e425/419 5:77d55580:::benchmark_data_smithi173_90735_object7081:head v 425'429, mlcod=425'424)", 2024-09-17T16:22:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:14.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:14.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:14.236 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:54.759172+0000", 2024-09-17T16:22:14.236 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 19.397157274000001, 2024-09-17T16:22:14.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.057096304000000001, 2024-09-17T16:22:14.236 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.236 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.236 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.759172+0000", 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.759172+0000", 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.759173+0000", 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3900000000000004e-07 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.759194+0000", 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1056999999999998e-05 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.815786+0000", 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.056591394000000003 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.815792+0000", 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7530000000000004e-06 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.815837+0000", 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4897000000000001e-05 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.815851+0000", 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3998999999999999e-05 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.816243+0000", 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039166599999999999 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:54.816269+0000", 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5698999999999999e-05 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.240 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:8827 5.e e426/419 5:7dd96b7d:::benchmark_data_smithi173_90735_object8826:head v 426'529, mlcod=426'527)", 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:56.269417+0000", 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 17.886912878, 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12478228600000001, 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:56.269417+0000", 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:56.269417+0000", 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:56.269418+0000", 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4920000000000001e-06 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:56.269453+0000", 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4533999999999998e-05 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:56.393456+0000", 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.124003298 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:56.393463+0000", 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2919999999999998e-06 2024-09-17T16:22:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:56.393520+0000", 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6923999999999999e-05 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:56.393543+0000", 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2254000000000001e-05 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:56.394169+0000", 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062653299999999997 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:56.394199+0000", 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9958999999999999e-05 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:27566 5.e e435/419 5:7562f95e:::benchmark_data_smithi173_90735_object27565:head v 435'1721, mlcod=435'1718)", 2024-09-17T16:22:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:12.496921+0000", 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.6594091020000001, 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.070703796999999999, 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:12.496921+0000", 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:12.496921+0000", 2024-09-17T16:22:14.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:12.496922+0000", 2024-09-17T16:22:14.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.192e-06 2024-09-17T16:22:14.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:12.496945+0000", 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3223000000000001e-05 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:12.566432+0000", 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.069487336999999996 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:12.566441+0000", 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3999999999999992e-06 2024-09-17T16:22:14.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:12.566499+0000", 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8152000000000002e-05 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:12.566521+0000", 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2382e-05 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:12.567577+0000", 2024-09-17T16:22:14.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001056186 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:12.567624+0000", 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6925000000000002e-05 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:29433 5.a e435/419 5:59c78006:::benchmark_data_smithi173_90735_object29432:head v 435'1800, mlcod=435'1797)", 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:13.620820+0000", 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.535509499, 2024-09-17T16:22:14.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.067123927999999999, 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.620820+0000", 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.620820+0000", 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.620821+0000", 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8299999999999995e-07 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.620910+0000", 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8436000000000005e-05 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.686760+0000", 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.065850562000000001 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.686769+0000", 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9090000000000006e-06 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.686837+0000", 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7576000000000005e-05 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.686853+0000", 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5936000000000001e-05 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.687886+0000", 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001033813 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.687944+0000", 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7713000000000001e-05 2024-09-17T16:22:14.254 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:29556 5.a e435/419 5:5b3204bf:::benchmark_data_smithi173_90735_object29555:head v 435'1805, mlcod=435'1801)", 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:13.702115+0000", 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.45421439600000002, 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.045250125000000002, 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.702115+0000", 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.702115+0000", 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.702116+0000", 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0200000000000002e-07 2024-09-17T16:22:14.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.702225+0000", 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000109518 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.746058+0000", 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.043832208999999997 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.746063+0000", 2024-09-17T16:22:14.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4500000000000003e-06 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.746124+0000", 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1111000000000005e-05 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.746147+0000", 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2671e-05 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.747318+0000", 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011708389999999999 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.747365+0000", 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7725000000000001e-05 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.259 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:29818 5.a e435/419 5:5505d9da:::benchmark_data_smithi173_90735_object29817:head v 435'1826, mlcod=435'1822)", 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:13.903444+0000", 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.252885898, 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.069245952, 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.903444+0000", 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.903444+0000", 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.903445+0000", 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8599999999999997e-07 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.903549+0000", 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010422200000000001 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.971347+0000", 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.067797837 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.971356+0000", 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2900000000000008e-06 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.971491+0000", 2024-09-17T16:22:14.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013512800000000001 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.971528+0000", 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6751000000000001e-05 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.972645+0000", 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011171880000000001 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:22:14.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:13.972690+0000", 2024-09-17T16:22:14.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4650000000000001e-05 2024-09-17T16:22:14.264 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.264 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.264 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.264 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:14.264 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:14.264 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2140673150' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:22:14.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2140673150' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:22:14.337 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:14.361 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:14.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:14.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:14.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:14.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:14.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:14.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:14.615 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:14.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:14.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:14.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:22:14.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:14.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:14.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:14.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:14.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:14.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 30446 30430 76.0169 36.25 0.0016179 0.0130132 2024-09-17T16:22:14.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:14.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:14.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:14.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:14.990 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:15.090 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:15.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:15.113 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:15.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:14 smithi173 ceph-mon[30530]: pgmap v977: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.6 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 908/76293 objects misplaced (1.190%); 5.2 MiB/s, 83 objects/s recovering 2024-09-17T16:22:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:14 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2140673150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:22:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:14 smithi173 ceph-mon[30530]: osdmap e436: 12 total, 12 up, 11 in 2024-09-17T16:22:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:14 smithi173 ceph-mon[30530]: osdmap e437: 12 total, 12 up, 11 in 2024-09-17T16:22:15.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:15.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:15.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:14 smithi017 ceph-mon[27571]: pgmap v977: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.6 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 908/76293 objects misplaced (1.190%); 5.2 MiB/s, 83 objects/s recovering 2024-09-17T16:22:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:14 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2140673150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:22:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:14 smithi017 ceph-mon[27571]: osdmap e436: 12 total, 12 up, 11 in 2024-09-17T16:22:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:14 smithi017 ceph-mon[27571]: osdmap e437: 12 total, 12 up, 11 in 2024-09-17T16:22:15.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:22:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:14 smithi071 ceph-mon[30002]: pgmap v977: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.6 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s; 908/76293 objects misplaced (1.190%); 5.2 MiB/s, 83 objects/s recovering 2024-09-17T16:22:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:14 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2140673150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:22:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:14 smithi071 ceph-mon[30002]: osdmap e436: 12 total, 12 up, 11 in 2024-09-17T16:22:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:14 smithi071 ceph-mon[30002]: osdmap e437: 12 total, 12 up, 11 in 2024-09-17T16:22:15.337 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:15.359 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:15.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:15.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:15.491 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:15.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:15.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:22:15.560 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:22:15.586 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:22:15.586 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:22:15.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:15.611 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:22:15.611 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:22:15.619 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:15.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:15.621 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:15.638 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:22:15.639 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:22:15.666 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:22:15.666 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:22:15.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:22:15.694 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:22:15.719 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:22:15.719 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:22:15.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:15.745 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:22:15.745 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:22:15.746 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:15.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:15.773 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:22:15.774 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:22:15.798 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:22:15.799 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:22:15.825 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:22:15.825 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:22:15.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:15.850 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:22:15.850 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:22:15.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:15.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:15.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 31524 31508 75.6846 67.375 0.00835047 0.0132053 2024-09-17T16:22:15.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:15.992 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:22:15.992 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:22:15.993 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-09-17T16:22:15.993 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:22:16.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:16.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:16.108 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:16.132 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:16.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:16.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:16.235 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:16.261 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:16.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:16.290 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:16.290 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:16.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:16.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:16.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:16.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:16.517 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:16.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:16.619 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:16.642 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:16.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:16.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:16 smithi173 ceph-mon[30530]: osdmap e438: 12 total, 12 up, 11 in 2024-09-17T16:22:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2168964345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:16 smithi173 ceph-mon[30530]: pgmap v981: 17 pgs: 17 active+clean; 1.8 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 121 MiB/s wr, 1.93k op/s; 9.9 MiB/s, 158 objects/s recovering 2024-09-17T16:22:16.665 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:22:16.707 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:16.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:16 smithi017 ceph-mon[27571]: osdmap e438: 12 total, 12 up, 11 in 2024-09-17T16:22:16.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2168964345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:16.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:16 smithi017 ceph-mon[27571]: pgmap v981: 17 pgs: 17 active+clean; 1.8 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 121 MiB/s wr, 1.93k op/s; 9.9 MiB/s, 158 objects/s recovering 2024-09-17T16:22:16.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:16.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:16.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:16 smithi071 ceph-mon[30002]: osdmap e438: 12 total, 12 up, 11 in 2024-09-17T16:22:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2168964345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:16 smithi071 ceph-mon[30002]: pgmap v981: 17 pgs: 17 active+clean; 1.8 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 121 MiB/s wr, 1.93k op/s; 9.9 MiB/s, 158 objects/s recovering 2024-09-17T16:22:16.874 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:16.902 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:16.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:16.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 32755 32739 75.7308 76.9375 0.00927218 0.0131988 2024-09-17T16:22:17.004 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:17.029 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:17.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:17.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:22:17.131 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:17.152 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:17.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:17.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:17.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:17.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:17.336 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:17.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:17.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:17.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:17.441 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:17.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:17.528 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:17.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:17.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:17.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:17.641 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:22:17.642 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:22:17.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:17.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:17.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:17.784 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:17.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:17.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:17.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:17.917 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:17.917 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:17.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:17.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:17.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:17.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 33729 33713 75.2003 60.875 0.00945554 0.013292 2024-09-17T16:22:18.021 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:18.050 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:18.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:18.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:18.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:18.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:18.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:18.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:18.280 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:18.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:18.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:18.411 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:18.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:18.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:18.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:18.561 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:18.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:18.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:22:18.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:18.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:18.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:18.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:18.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:18.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:18.911 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:18.934 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:18.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:18.941 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:22:18.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 34813 34797 74.9434 67.75 0.00316556 0.0133316 2024-09-17T16:22:18.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:22:19.037 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:19.065 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:19.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:19.092 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":981,"stamp":"2024-09-17T16:22:15.879579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1956905504,"num_objects":29853,"num_object_clones":0,"num_object_copies":89559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29853,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":29974,"num_write_kb":1912614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":64501344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29943,"ondisk_log_size":29943,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":8665508,"kb_used_data":5784012,"kb_used_omap":57,"kb_used_meta":2881222,"kb_avail":1022531164,"statfs":{"total":1055945392128,"available":1047071911936,"internally_reserved":0,"allocated":5922828288,"data_stored":5916981598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":2950372062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":758775808,"num_objects":11578,"num_object_clones":0,"num_object_copies":34734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1908,"num_objects_unfound":0,"num_objects_dirty":11578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11578,"num_write_kb":740992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":62521344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001099"},"pg_stats":[{"pgid":"5.f","version":"438'1882","reported_seq":1931,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-17T16:22:15.536439+0000","last_change":"2024-09-17T16:21:42.617220+0000","last_active":"2024-09-17T16:22:15.536439+0000","last_peered":"2024-09-17T16:22:15.536439+0000","last_clean":"2024-09-17T16:22:15.536439+0000","last_became_active":"2024-09-17T16:21:42.616065+0000","last_became_peered":"2024-09-17T16:21:42.616065+0000","last_unstale":"2024-09-17T16:22:15.536439+0000","last_undegraded":"2024-09-17T16:22:15.536439+0000","last_fullsized":"2024-09-17T16:22:15.536439+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1882,"log_dups_size":0,"ondisk_log_size":1882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:01.813525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123338752,"num_objects":1882,"num_object_clones":0,"num_object_copies":5646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1882,"num_write_kb":120448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"435'1705","reported_seq":1748,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-17T16:22:12.337250+0000","last_change":"2024-09-17T16:21:42.615978+0000","last_active":"2024-09-17T16:22:12.337250+0000","last_peered":"2024-09-17T16:22:12.337250+0000","last_clean":"2024-09-17T16:22:12.337250+0000","last_became_active":"2024-09-17T16:21:42.615584+0000","last_became_peered":"2024-09-17T16:21:42.615584+0000","last_unstale":"2024-09-17T16:22:12.337250+0000","last_undegraded":"2024-09-17T16:22:12.337250+0000","last_fullsized":"2024-09-17T16:22:12.337250+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1705,"log_dups_size":0,"ondisk_log_size":1705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:08.519662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111738880,"num_objects":1705,"num_object_clones":0,"num_object_copies":5115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1705,"num_write_kb":109120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"438'1970","reported_seq":2019,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-17T16:22:15.529828+0000","last_change":"2024-09-17T16:21:42.615849+0000","last_active":"2024-09-17T16:22:15.529828+0000","last_peered":"2024-09-17T16:22:15.529828+0000","last_clean":"2024-09-17T16:22:15.529828+0000","last_became_active":"2024-09-17T16:21:42.615522+0000","last_became_peered":"2024-09-17T16:21:42.615522+0000","last_unstale":"2024-09-17T16:22:15.529828+0000","last_undegraded":"2024-09-17T16:22:15.529828+0000","last_fullsized":"2024-09-17T16:22:15.529828+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1970,"log_dups_size":0,"ondisk_log_size":1970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:49.214195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129105920,"num_objects":1970,"num_object_clones":0,"num_object_copies":5910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1970,"num_write_kb":126080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"437'1869","reported_seq":1916,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-09-17T16:22:14.757505+0000","last_change":"2024-09-17T16:21:42.616080+0000","last_active":"2024-09-17T16:22:14.757505+0000","last_peered":"2024-09-17T16:22:14.757505+0000","last_clean":"2024-09-17T16:22:14.757505+0000","last_became_active":"2024-09-17T16:21:42.615761+0000","last_became_peered":"2024-09-17T16:21:42.615761+0000","last_unstale":"2024-09-17T16:22:14.757505+0000","last_undegraded":"2024-09-17T16:22:14.757505+0000","last_fullsized":"2024-09-17T16:22:14.757505+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1869,"log_dups_size":0,"ondisk_log_size":1869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:01.878455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122486784,"num_objects":1869,"num_object_clones":0,"num_object_copies":5607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1869,"num_write_kb":119616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"435'1761","reported_seq":1803,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-17T16:22:13.010013+0000","last_change":"2024-09-17T16:21:42.616006+0000","last_active":"2024-09-17T16:22:13.010013+0000","last_peered":"2024-09-17T16:22:13.010013+0000","last_clean":"2024-09-17T16:22:13.010013+0000","last_became_active":"2024-09-17T16:21:42.615693+0000","last_became_peered":"2024-09-17T16:21:42.615693+0000","last_unstale":"2024-09-17T16:22:13.010013+0000","last_undegraded":"2024-09-17T16:22:13.010013+0000","last_fullsized":"2024-09-17T16:22:13.010013+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1761,"log_dups_size":0,"ondisk_log_size":1761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:08.008512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115408896,"num_objects":1761,"num_object_clones":0,"num_object_copies":5283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1761,"num_write_kb":112704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"435'1606","reported_seq":1647,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-17T16:22:11.770959+0000","last_change":"2024-09-17T16:21:42.467809+0000","last_active":"2024-09-17T16:22:11.770959+0000","last_peered":"2024-09-17T16:22:11.770959+0000","last_clean":"2024-09-17T16:22:11.770959+0000","last_became_active":"2024-09-17T16:21:42.467536+0000","last_became_peered":"2024-09-17T16:21:42.467536+0000","last_unstale":"2024-09-17T16:22:11.770959+0000","last_undegraded":"2024-09-17T16:22:11.770959+0000","last_fullsized":"2024-09-17T16:22:11.770959+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1606,"log_dups_size":0,"ondisk_log_size":1606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:54.415573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105250816,"num_objects":1606,"num_object_clones":0,"num_object_copies":4818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1606,"num_write_kb":102784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"435'1801","reported_seq":1844,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-17T16:22:13.012304+0000","last_change":"2024-09-17T16:21:42.615828+0000","last_active":"2024-09-17T16:22:13.012304+0000","last_peered":"2024-09-17T16:22:13.012304+0000","last_clean":"2024-09-17T16:22:13.012304+0000","last_became_active":"2024-09-17T16:21:42.615436+0000","last_became_peered":"2024-09-17T16:21:42.615436+0000","last_unstale":"2024-09-17T16:22:13.012304+0000","last_undegraded":"2024-09-17T16:22:13.012304+0000","last_fullsized":"2024-09-17T16:22:13.012304+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1801,"log_dups_size":0,"ondisk_log_size":1801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:07.562425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118030336,"num_objects":1801,"num_object_clones":0,"num_object_copies":5403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1801,"num_write_kb":115264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"438'1953","reported_seq":2002,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-17T16:22:15.827227+0000","last_change":"2024-09-17T16:21:42.616143+0000","last_active":"2024-09-17T16:22:15.827227+0000","last_peered":"2024-09-17T16:22:15.827227+0000","last_clean":"2024-09-17T16:22:15.827227+0000","last_became_active":"2024-09-17T16:21:42.615778+0000","last_became_peered":"2024-09-17T16:21:42.615778+0000","last_unstale":"2024-09-17T16:22:15.827227+0000","last_undegraded":"2024-09-17T16:22:15.827227+0000","last_fullsized":"2024-09-17T16:22:15.827227+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1953,"log_dups_size":0,"ondisk_log_size":1953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:06.957578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127991808,"num_objects":1953,"num_object_clones":0,"num_object_copies":5859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":124992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[10,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"438'2056","reported_seq":2104,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-17T16:22:15.846174+0000","last_change":"2024-09-17T16:21:42.468402+0000","last_active":"2024-09-17T16:22:15.846174+0000","last_peered":"2024-09-17T16:22:15.846174+0000","last_clean":"2024-09-17T16:22:15.846174+0000","last_became_active":"2024-09-17T16:21:42.467840+0000","last_became_peered":"2024-09-17T16:21:42.467840+0000","last_unstale":"2024-09-17T16:22:15.846174+0000","last_undegraded":"2024-09-17T16:22:15.846174+0000","last_fullsized":"2024-09-17T16:22:15.846174+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2056,"log_dups_size":0,"ondisk_log_size":2056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:44.683474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134742016,"num_objects":2056,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2056,"num_write_kb":131584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"435'1631","reported_seq":1674,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-17T16:22:11.758063+0000","last_change":"2024-09-17T16:21:42.472962+0000","last_active":"2024-09-17T16:22:11.758063+0000","last_peered":"2024-09-17T16:22:11.758063+0000","last_clean":"2024-09-17T16:22:11.758063+0000","last_became_active":"2024-09-17T16:21:42.472648+0000","last_became_peered":"2024-09-17T16:21:42.472648+0000","last_unstale":"2024-09-17T16:22:11.758063+0000","last_undegraded":"2024-09-17T16:22:11.758063+0000","last_fullsized":"2024-09-17T16:22:11.758063+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1631,"log_dups_size":0,"ondisk_log_size":1631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:00.127440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106889216,"num_objects":1631,"num_object_clones":0,"num_object_copies":4893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1631,"num_write_kb":104384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"438'1958","reported_seq":4899,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-17T16:22:15.851534+0000","last_change":"2024-09-17T16:22:15.448985+0000","last_active":"2024-09-17T16:22:15.851534+0000","last_peered":"2024-09-17T16:22:15.851534+0000","last_clean":"2024-09-17T16:22:15.851534+0000","last_became_active":"2024-09-17T16:22:15.445968+0000","last_became_peered":"2024-09-17T16:22:15.445968+0000","last_unstale":"2024-09-17T16:22:15.851534+0000","last_undegraded":"2024-09-17T16:22:15.851534+0000","last_fullsized":"2024-09-17T16:22:15.851534+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1958,"log_dups_size":0,"ondisk_log_size":1958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:52.945659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128319488,"num_objects":1958,"num_object_clones":0,"num_object_copies":5874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1958,"num_write_kb":125312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":960,"num_bytes_recovered":62914560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,3],"acting":[10,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"438'1923","reported_seq":1971,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-17T16:22:15.713304+0000","last_change":"2024-09-17T16:21:42.616516+0000","last_active":"2024-09-17T16:22:15.713304+0000","last_peered":"2024-09-17T16:22:15.713304+0000","last_clean":"2024-09-17T16:22:15.713304+0000","last_became_active":"2024-09-17T16:21:42.615586+0000","last_became_peered":"2024-09-17T16:21:42.615586+0000","last_unstale":"2024-09-17T16:22:15.713304+0000","last_undegraded":"2024-09-17T16:22:15.713304+0000","last_fullsized":"2024-09-17T16:22:15.713304+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1923,"log_dups_size":0,"ondisk_log_size":1923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:48.998835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126025728,"num_objects":1923,"num_object_clones":0,"num_object_copies":5769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1923,"num_write_kb":123072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"438'1935","reported_seq":1984,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-17T16:22:15.742791+0000","last_change":"2024-09-17T16:21:42.617260+0000","last_active":"2024-09-17T16:22:15.742791+0000","last_peered":"2024-09-17T16:22:15.742791+0000","last_clean":"2024-09-17T16:22:15.742791+0000","last_became_active":"2024-09-17T16:21:42.615959+0000","last_became_peered":"2024-09-17T16:21:42.615959+0000","last_unstale":"2024-09-17T16:22:15.742791+0000","last_undegraded":"2024-09-17T16:22:15.742791+0000","last_fullsized":"2024-09-17T16:22:15.742791+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1935,"log_dups_size":0,"ondisk_log_size":1935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:09.798834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126812160,"num_objects":1935,"num_object_clones":0,"num_object_copies":5805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1935,"num_write_kb":123840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"437'1915","reported_seq":1961,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-09-17T16:22:14.761451+0000","last_change":"2024-09-17T16:21:42.616857+0000","last_active":"2024-09-17T16:22:14.761451+0000","last_peered":"2024-09-17T16:22:14.761451+0000","last_clean":"2024-09-17T16:22:14.761451+0000","last_became_active":"2024-09-17T16:21:42.616534+0000","last_became_peered":"2024-09-17T16:21:42.616534+0000","last_unstale":"2024-09-17T16:22:14.761451+0000","last_undegraded":"2024-09-17T16:22:14.761451+0000","last_fullsized":"2024-09-17T16:22:14.761451+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:52.078506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125501440,"num_objects":1915,"num_object_clones":0,"num_object_copies":5745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1915,"num_write_kb":122560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"438'1968","reported_seq":2017,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-17T16:22:15.848191+0000","last_change":"2024-09-17T16:21:42.615804+0000","last_active":"2024-09-17T16:22:15.848191+0000","last_peered":"2024-09-17T16:22:15.848191+0000","last_clean":"2024-09-17T16:22:15.848191+0000","last_became_active":"2024-09-17T16:21:42.615475+0000","last_became_peered":"2024-09-17T16:21:42.615475+0000","last_unstale":"2024-09-17T16:22:15.848191+0000","last_undegraded":"2024-09-17T16:22:15.848191+0000","last_fullsized":"2024-09-17T16:22:15.848191+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1968,"log_dups_size":0,"ondisk_log_size":1968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:53.339886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128974848,"num_objects":1968,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1968,"num_write_kb":125952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"438'1918","reported_seq":1966,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-09-17T16:22:15.813523+0000","last_change":"2024-09-17T16:21:42.472814+0000","last_active":"2024-09-17T16:22:15.813523+0000","last_peered":"2024-09-17T16:22:15.813523+0000","last_clean":"2024-09-17T16:22:15.813523+0000","last_became_active":"2024-09-17T16:21:42.472512+0000","last_became_peered":"2024-09-17T16:21:42.472512+0000","last_unstale":"2024-09-17T16:22:15.813523+0000","last_undegraded":"2024-09-17T16:22:15.813523+0000","last_fullsized":"2024-09-17T16:22:15.813523+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":1918,"log_dups_size":0,"ondisk_log_size":1918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:46.196379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125698048,"num_objects":1918,"num_object_clones":0,"num_object_copies":5754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1918,"num_write_kb":122752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":770,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-17T16:22:09.448045+0000","last_change":"2024-09-17T16:22:06.807310+0000","last_active":"2024-09-17T16:22:09.448045+0000","last_peered":"2024-09-17T16:22:09.448045+0000","last_clean":"2024-09-17T16:22:09.448045+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:22:09.448045+0000","last_undegraded":"2024-09-17T16:22:09.448045+0000","last_fullsized":"2024-09-17T16:22:09.448045+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:22:06.807231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:22:06.807231+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:16.883658+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1956315136,"num_objects":29851,"num_object_clones":0,"num_object_copies":89553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29851,"num_write_kb":1910464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":960,"num_bytes_recovered":62914560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5774245888,"data_stored":5774245888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29851,"ondisk_log_size":29851,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285721,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":666276,"kb_used_data":368524,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93078876,"statfs":{"total":95995035648,"available":95312769024,"internally_reserved":0,"allocated":377368576,"data_stored":376830381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":304871876},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547358,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1059296,"kb_used_data":869840,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92685856,"statfs":{"total":95995035648,"available":94910316544,"internally_reserved":0,"allocated":890716160,"data_stored":890177030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":193984960},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743586,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924220,"kb_used_data":532076,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92820932,"statfs":{"total":95995035648,"available":95048634368,"internally_reserved":0,"allocated":544845824,"data_stored":544320173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":401537472},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005223,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":378188,"kb_used_data":238712,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93366964,"statfs":{"total":95995035648,"available":95607771136,"internally_reserved":0,"allocated":244441088,"data_stored":243910951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963700,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":701572,"kb_used_data":450420,"kb_used_omap":41,"kb_used_meta":251094,"kb_avail":93043580,"statfs":{"total":95995035648,"available":95276625920,"internally_reserved":0,"allocated":461230080,"data_stored":460700877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":257120563},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495794,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837668,"kb_used_data":549644,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92907484,"statfs":{"total":95995035648,"available":95137263616,"internally_reserved":0,"allocated":562835456,"data_stored":562297281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":294910406},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724725,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1120152,"kb_used_data":841860,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":92625000,"statfs":{"total":95995035648,"available":94848000000,"internally_reserved":0,"allocated":862064640,"data_stored":861530017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":284948934},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986364,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865628,"kb_used_data":603972,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92879524,"statfs":{"total":95995035648,"available":95108632576,"internally_reserved":0,"allocated":618467328,"data_stored":617932705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149886,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":482628,"kb_used_data":172396,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93262524,"statfs":{"total":95995035648,"available":95500824576,"internally_reserved":0,"allocated":176533504,"data_stored":176007853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":317651399},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411525,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560252,"kb_used_data":402732,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93184900,"statfs":{"total":95995035648,"available":95421337600,"internally_reserved":0,"allocated":412397568,"data_stored":411871917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738575,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1069628,"kb_used_data":753836,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92675524,"statfs":{"total":95995035648,"available":94899736576,"internally_reserved":0,"allocated":771928064,"data_stored":771402413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":786825216,"data_stored":786825216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":430768128,"data_stored":430768128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":188416000,"data_stored":188416000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":619642880,"data_stored":619642880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":624099328,"data_stored":624099328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":568852480,"data_stored":568852480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":467009536,"data_stored":467009536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":245104640,"data_stored":245104640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":568721408,"data_stored":568721408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":898236416,"data_stored":898236416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":376569856,"data_stored":376569856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:22:19.093 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-17T16:22:19.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_3 pg_num 26 2024-09-17T16:22:19.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:19.188 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:19.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:18 smithi017 ceph-mon[27571]: from='client.26264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:22:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:18 smithi017 ceph-mon[27571]: pgmap v982: 17 pgs: 17 active+clean; 1.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s; 4.7 MiB/s, 75 objects/s recovering 2024-09-17T16:22:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2486802651' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:19.273 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:19.290 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:19.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:18 smithi071 ceph-mon[30002]: from='client.26264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:22:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:18 smithi071 ceph-mon[30002]: pgmap v982: 17 pgs: 17 active+clean; 1.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s; 4.7 MiB/s, 75 objects/s recovering 2024-09-17T16:22:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2486802651' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:19.317 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:19.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:18 smithi173 ceph-mon[30530]: from='client.26264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:22:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:18 smithi173 ceph-mon[30530]: pgmap v982: 17 pgs: 17 active+clean; 1.9 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s; 4.7 MiB/s, 75 objects/s recovering 2024-09-17T16:22:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2486802651' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:19.419 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:19.444 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:19.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:19.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:19.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:19.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:19.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:19.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:19.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:19.706 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:19.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:19.835 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:19.835 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:19.836 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:19.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:19.859 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:19.862 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:19.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:19.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 36307 36291 75.5573 93.375 0.00529357 0.0132291 2024-09-17T16:22:19.965 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:19.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:19.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2486802651' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:19.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:19 smithi017 ceph-mon[27571]: osdmap e439: 12 total, 12 up, 11 in 2024-09-17T16:22:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:19 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:20.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:20.092 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:20.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:20.194 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:20.217 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:20.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2486802651' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:19 smithi071 ceph-mon[30002]: osdmap e439: 12 total, 12 up, 11 in 2024-09-17T16:22:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:19 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:20.317 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:20.361 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:20.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2486802651' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:19 smithi173 ceph-mon[30530]: osdmap e439: 12 total, 12 up, 11 in 2024-09-17T16:22:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:19 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:20.463 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:20.487 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:20.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:20.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:22:20.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:20.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:20.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:20.716 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:20.742 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:20.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:20.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:22:20.817 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:22:20.828 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:20.840 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:22:20.840 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:22:20.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:20.863 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:22:20.863 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:22:20.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:20.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:20.890 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:22:20.891 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:22:20.921 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:22:20.921 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:22:20.947 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:22:20.948 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:22:20.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 37515 37499 75.5552 75.5 0.003299 0.0132285 2024-09-17T16:22:20.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:22:20.973 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:22:20.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:20.997 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:22:20.997 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:22:21.003 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:21.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:21.022 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:22:21.022 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:22:21.050 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:22:21.050 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:22:21.077 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:22:21.077 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:22:21.078 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:20 smithi017 ceph-mon[27571]: pgmap v984: 17 pgs: 17 active+clean; 2.0 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 73 MiB/s wr, 1.18k op/s; 4.8 MiB/s, 76 objects/s recovering 2024-09-17T16:22:21.078 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4011079849' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:21.103 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:22:21.103 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:22:21.105 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:21.131 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:21.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:21.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:21.260 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:21.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:21.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:20 smithi173 ceph-mon[30530]: pgmap v984: 17 pgs: 17 active+clean; 2.0 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 73 MiB/s wr, 1.18k op/s; 4.8 MiB/s, 76 objects/s recovering 2024-09-17T16:22:21.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4011079849' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:20 smithi071 ceph-mon[30002]: pgmap v984: 17 pgs: 17 active+clean; 2.0 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 73 MiB/s wr, 1.18k op/s; 4.8 MiB/s, 76 objects/s recovering 2024-09-17T16:22:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4011079849' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:21.362 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:21.390 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:21.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:21.416 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:22:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:22:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:38180 5.8 5:1101e739:::benchmark_data_smithi173_90735_object38179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-17T16:22:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:21.405579+0000", 2024-09-17T16:22:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0079626829999999999, 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0079929890000000007, 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 38180 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.405579+0000", 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.405579+0000", 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.405580+0000", 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3200000000000003e-07 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.405602+0000", 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1990000000000001e-05 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.405613+0000", 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1079e-05 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.405618+0000", 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8330000000000001e-06 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.405665+0000", 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6934999999999997e-05 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.405793+0000", 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012793400000000001 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.405939+0000", 2024-09-17T16:22:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014605900000000001 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.407445+0000", 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015056939999999999 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.408292+0000", 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00084684400000000003 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:38189 5.4 5:22b944ec:::benchmark_data_smithi173_90735_object38188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:21.410147+0000", 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0033945970000000001, 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035071439999999998, 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 38189 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.410147+0000", 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.410147+0000", 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.410149+0000", 2024-09-17T16:22:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7829999999999999e-06 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.410175+0000", 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5998000000000001e-05 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.410189+0000", 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3794999999999999e-05 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.410196+0000", 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7669999999999998e-06 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.410249+0000", 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3096000000000001e-05 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.410379+0000", 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012989100000000001 2024-09-17T16:22:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,5", 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.410531+0000", 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000151989 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.411814+0000", 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001283258 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.412677+0000", 2024-09-17T16:22:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086263599999999996 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:38193 5.0 5:0d9ebedb:::benchmark_data_smithi173_90735_object38192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:21.411839+0000", 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.001703027, 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018579989999999999, 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:22:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 38193 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.411839+0000", 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.411839+0000", 2024-09-17T16:22:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.411840+0000", 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7900000000000007e-07 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.411859+0000", 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8581e-05 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.411869+0000", 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0509999999999999e-05 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.411876+0000", 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.17e-06 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.411932+0000", 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5566000000000003e-05 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.412010+0000", 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8053999999999995e-05 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.412090+0000", 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0266999999999996e-05 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.413397+0000", 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013068299999999999 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26158.0:38180 5.8 e439/419 ondisk, result = 0)", 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:21.413589+0000", 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4294967295.9999528, 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000142977, 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:22:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.413589+0000", 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.413589+0000", 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.413594+0000", 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8779999999999997e-06 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.413598+0000", 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0130000000000004e-06 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.413613+0000", 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4541e-05 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.413618+0000", 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9930000000000003e-06 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.413662+0000", 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4456000000000003e-05 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:21.413667+0000", 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2970000000000003e-06 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 4 2024-09-17T16:22:21.435 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:22:21.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:21.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:21.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:21.625 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:21.647 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:21.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:21.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:21.749 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:21.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:22:21.774 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:21.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:21.876 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:21.901 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:21.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:21.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 38964 38948 76.0237 90.5625 0.00625937 0.0131445 2024-09-17T16:22:21.953 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 5 pg_num to 26 2024-09-17T16:22:22.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:22.030 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:22.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:22.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:22.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3716566964' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:22:22.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:22.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:22.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:22:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3716566964' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:22:22.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:22.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:22.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:22.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3716566964' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:22:22.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:22.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:22.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:22.520 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:22.539 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:22.549 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:22.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:22.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:22.678 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:22.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:22.780 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:22.808 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:22.811 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:22.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:22.912 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:22.938 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:22.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:22.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:22.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 40526 40510 76.6778 97.625 0.0168477 0.013036 2024-09-17T16:22:23.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:23.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:23.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:23.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:22 smithi017 ceph-mon[27571]: pgmap v985: 17 pgs: 17 active+clean; 2.2 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-09-17T16:22:23.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3716566964' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:22:23.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:22 smithi017 ceph-mon[27571]: osdmap e440: 12 total, 12 up, 11 in 2024-09-17T16:22:23.101 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:22:23.101 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:22:23.101 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:22:23.101 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:22:23.101 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:22:23.169 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:23.193 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:23.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:23.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:22 smithi071 ceph-mon[30002]: pgmap v985: 17 pgs: 17 active+clean; 2.2 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-09-17T16:22:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3716566964' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:22:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:22 smithi071 ceph-mon[30002]: osdmap e440: 12 total, 12 up, 11 in 2024-09-17T16:22:23.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:23.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:23.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:22 smithi173 ceph-mon[30530]: pgmap v985: 17 pgs: 17 active+clean; 2.2 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-09-17T16:22:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3716566964' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:22:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:22 smithi173 ceph-mon[30530]: osdmap e440: 12 total, 12 up, 11 in 2024-09-17T16:22:23.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:23.449 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:23.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:23.551 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:23.574 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:23.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:23.676 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:23.698 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:23.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:23.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:23.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:22:23.800 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:23.825 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:23.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:23.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:23.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:23.948 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:22:23.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 41595 41579 76.3876 66.8125 0.00276907 0.0130836 2024-09-17T16:22:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:23.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:23.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:23.962 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:22:24.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:24.080 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:24.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:24.106 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2437681129' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:22:24.181 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:24.204 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:24.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:24.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2437681129' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:24.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:22:24.305 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2437681129' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:22:24.327 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:24.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:24.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:24.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:24.454 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:24.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:24.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:24.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:24.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:24.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:24.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:24.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:24.779 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:24.809 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:24.831 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:24.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:24.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:24.933 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:24.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 41619 41603 74.2489 1.5 0.00321616 0.0131241 2024-09-17T16:22:24.955 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:24.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:24.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:25.057 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:25.084 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:25.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:25.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:25.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:25.209 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:25.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:24 smithi017 ceph-mon[27571]: pgmap v987: 17 pgs: 17 active+clean; 2.3 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:22:25.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2437681129' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:22:25.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:24 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:22:25.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:24 smithi017 ceph-mon[27571]: osdmap e441: 12 total, 12 up, 11 in 2024-09-17T16:22:25.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:24 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:25.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:25.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:24 smithi071 ceph-mon[30002]: pgmap v987: 17 pgs: 17 active+clean; 2.3 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:22:25.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2437681129' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:22:25.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:24 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:22:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:24 smithi071 ceph-mon[30002]: osdmap e441: 12 total, 12 up, 11 in 2024-09-17T16:22:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:24 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:25.330 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:25.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:25.334 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:22:25.335 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:22:25.356 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:22:25.356 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:22:25.376 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:22:25.377 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:22:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:24 smithi173 ceph-mon[30530]: pgmap v987: 17 pgs: 17 active+clean; 2.3 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:22:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2437681129' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:22:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:24 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:22:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:24 smithi173 ceph-mon[30530]: osdmap e441: 12 total, 12 up, 11 in 2024-09-17T16:22:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:24 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:25.397 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:22:25.397 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:22:25.419 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:22:25.419 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:22:25.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:25.441 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:22:25.441 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:22:25.454 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:25.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:25.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:25.462 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:22:25.462 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:22:25.484 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:22:25.485 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:22:25.506 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:22:25.506 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:22:25.527 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:22:25.528 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:22:25.550 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:22:25.550 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:22:25.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:25.573 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:22:25.573 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:22:25.579 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:25.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:25.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:25.703 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:25.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:25.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:25.827 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:25.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:25.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:25.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:25.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 41914 41898 72.6992 18.4375 0.0015601 0.0134765 2024-09-17T16:22:25.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:25.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:26.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:26.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:26.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:26.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:26.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:26.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:26.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:26.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:25 smithi173 ceph-mon[30530]: osdmap e442: 12 total, 12 up, 11 in 2024-09-17T16:22:26.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:25 smithi173 ceph-mon[30530]: 5.6 scrub starts 2024-09-17T16:22:26.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:25 smithi173 ceph-mon[30530]: 5.3 scrub starts 2024-09-17T16:22:26.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:25 smithi173 ceph-mon[30530]: 5.6 scrub ok 2024-09-17T16:22:26.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:25 smithi173 ceph-mon[30530]: 5.3 scrub ok 2024-09-17T16:22:26.223 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3255978001' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:26.223 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:22:26.223 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:26.223 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:25 smithi017 ceph-mon[27571]: osdmap e442: 12 total, 12 up, 11 in 2024-09-17T16:22:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:25 smithi017 ceph-mon[27571]: 5.6 scrub starts 2024-09-17T16:22:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:25 smithi017 ceph-mon[27571]: 5.3 scrub starts 2024-09-17T16:22:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:25 smithi017 ceph-mon[27571]: 5.6 scrub ok 2024-09-17T16:22:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:25 smithi017 ceph-mon[27571]: 5.3 scrub ok 2024-09-17T16:22:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3255978001' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:22:26.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:25 smithi071 ceph-mon[30002]: osdmap e442: 12 total, 12 up, 11 in 2024-09-17T16:22:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:25 smithi071 ceph-mon[30002]: 5.6 scrub starts 2024-09-17T16:22:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:25 smithi071 ceph-mon[30002]: 5.3 scrub starts 2024-09-17T16:22:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:25 smithi071 ceph-mon[30002]: 5.6 scrub ok 2024-09-17T16:22:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:25 smithi071 ceph-mon[30002]: 5.3 scrub ok 2024-09-17T16:22:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3255978001' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:22:26.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:26.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:26.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:26.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:26.451 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:26.552 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:26.576 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:26.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:26.678 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:26.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:26.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:22:26.805 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:26.828 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:26.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:26.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:22:26.930 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:26.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 41917 41901 70.7403 0.1875 0.00989733 0.0134858 2024-09-17T16:22:26.952 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:26.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:27.054 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:27.077 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:27.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:27.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:27.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: 5.1 scrub starts 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: 5.1 scrub ok 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: 5.5 scrub starts 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: 5.5 scrub ok 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: pgmap v990: 27 pgs: 1 peering, 4 unknown, 22 active+clean; 2.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: 5.9 scrub starts 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: osdmap e443: 12 total, 12 up, 11 in 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: 5.2 scrub starts 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: 5.2 scrub ok 2024-09-17T16:22:27.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:26 smithi017 ceph-mon[27571]: 5.9 scrub ok 2024-09-17T16:22:27.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: 5.1 scrub starts 2024-09-17T16:22:27.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: 5.1 scrub ok 2024-09-17T16:22:27.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: 5.5 scrub starts 2024-09-17T16:22:27.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: 5.5 scrub ok 2024-09-17T16:22:27.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: pgmap v990: 27 pgs: 1 peering, 4 unknown, 22 active+clean; 2.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-17T16:22:27.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: 5.9 scrub starts 2024-09-17T16:22:27.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:22:27.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: osdmap e443: 12 total, 12 up, 11 in 2024-09-17T16:22:27.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: 5.2 scrub starts 2024-09-17T16:22:27.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:22:27.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:22:27.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: 5.2 scrub ok 2024-09-17T16:22:27.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:26 smithi173 ceph-mon[30530]: 5.9 scrub ok 2024-09-17T16:22:27.167 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:27.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:27.202 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:27.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:27.302 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: 5.1 scrub starts 2024-09-17T16:22:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: 5.1 scrub ok 2024-09-17T16:22:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: 5.5 scrub starts 2024-09-17T16:22:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: 5.5 scrub ok 2024-09-17T16:22:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: pgmap v990: 27 pgs: 1 peering, 4 unknown, 22 active+clean; 2.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-17T16:22:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: 5.9 scrub starts 2024-09-17T16:22:27.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:22:27.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: osdmap e443: 12 total, 12 up, 11 in 2024-09-17T16:22:27.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: 5.2 scrub starts 2024-09-17T16:22:27.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:22:27.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:22:27.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: 5.2 scrub ok 2024-09-17T16:22:27.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:26 smithi071 ceph-mon[30002]: 5.9 scrub ok 2024-09-17T16:22:27.325 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:27.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:27.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:27.427 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:27.453 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:27.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:27.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:27.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:27.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:27.698 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:27.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:27.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:27.794 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:27.795 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:27.810 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:22:27.810 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:22:27.811 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-17T16:22:27.811 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:22:27.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:27.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:27.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:27.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:27.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 41917 41901 68.8795 0 - 0.0134858 2024-09-17T16:22:27.967 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:27.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:27.974 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:22:28.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:28 smithi173 ceph-mon[30530]: 5.8 scrub starts 2024-09-17T16:22:28.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:28 smithi173 ceph-mon[30530]: 5.8 scrub ok 2024-09-17T16:22:28.004 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:28 smithi173 ceph-mon[30530]: osdmap e444: 12 total, 12 up, 11 in 2024-09-17T16:22:28.005 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2122985707' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:28.005 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:28 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:22:28.005 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:28 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:22:28.005 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2122985707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:28.005 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:28 smithi173 ceph-mon[30530]: osdmap e445: 12 total, 12 up, 11 in 2024-09-17T16:22:28.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:28.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:28.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:28.197 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:28.234 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:28.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:28.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:28.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:28 smithi017 ceph-mon[27571]: 5.8 scrub starts 2024-09-17T16:22:28.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:28 smithi017 ceph-mon[27571]: 5.8 scrub ok 2024-09-17T16:22:28.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:28 smithi017 ceph-mon[27571]: osdmap e444: 12 total, 12 up, 11 in 2024-09-17T16:22:28.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2122985707' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:28.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:28 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:22:28.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:28 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:22:28.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2122985707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:28.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:28 smithi017 ceph-mon[27571]: osdmap e445: 12 total, 12 up, 11 in 2024-09-17T16:22:28.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:28 smithi071 ceph-mon[30002]: 5.8 scrub starts 2024-09-17T16:22:28.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:28 smithi071 ceph-mon[30002]: 5.8 scrub ok 2024-09-17T16:22:28.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:28 smithi071 ceph-mon[30002]: osdmap e444: 12 total, 12 up, 11 in 2024-09-17T16:22:28.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2122985707' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:28.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:28 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:22:28.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:28 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:22:28.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2122985707' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:28.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:28 smithi071 ceph-mon[30002]: osdmap e445: 12 total, 12 up, 11 in 2024-09-17T16:22:28.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:28.370 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:28.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:28.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:28.511 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:28.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:28.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:22:28.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:28.643 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:28.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:28.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:28.745 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:28.776 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:28.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:28.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:28.846 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:28.868 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:22:28.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:28.888 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:28.949 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:28.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:28.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 42738 42722 68.4292 25.6562 0.0362074 0.0145998 2024-09-17T16:22:29.049 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:29.082 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:29.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:29.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:29 smithi017 ceph-mon[27571]: 5.0 scrub starts 2024-09-17T16:22:29.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:29 smithi017 ceph-mon[27571]: 5.0 scrub ok 2024-09-17T16:22:29.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:29 smithi017 ceph-mon[27571]: pgmap v993: 27 pgs: 1 peering, 6 unknown, 20 active+clean; 2.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:22:29.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:29 smithi017 ceph-mon[27571]: 5.15 starting backfill to osd.0 from (0'0,428'959] MIN to 440'2624 2024-09-17T16:22:29.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:29 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:22:29.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:29.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:29.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:29.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:29 smithi071 ceph-mon[30002]: 5.0 scrub starts 2024-09-17T16:22:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:29 smithi071 ceph-mon[30002]: 5.0 scrub ok 2024-09-17T16:22:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:29 smithi071 ceph-mon[30002]: pgmap v993: 27 pgs: 1 peering, 6 unknown, 20 active+clean; 2.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:22:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:29 smithi071 ceph-mon[30002]: 5.15 starting backfill to osd.0 from (0'0,428'959] MIN to 440'2624 2024-09-17T16:22:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:29 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:22:29.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:29.342 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:29.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:29 smithi173 ceph-mon[30530]: 5.0 scrub starts 2024-09-17T16:22:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:29 smithi173 ceph-mon[30530]: 5.0 scrub ok 2024-09-17T16:22:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:29 smithi173 ceph-mon[30530]: pgmap v993: 27 pgs: 1 peering, 6 unknown, 20 active+clean; 2.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:22:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:29 smithi173 ceph-mon[30530]: 5.15 starting backfill to osd.0 from (0'0,428'959] MIN to 440'2624 2024-09-17T16:22:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:29 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:22:29.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:29.477 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:29.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:29.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:29.585 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:29.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:29.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:29.706 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:29.706 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:29.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:29.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:29.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:22:29.789 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:22:29.840 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:29.872 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:29.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:29.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:29.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:22:29.948095+0000 min lat: 0.00103987 max lat: 3.10335 avg lat: 0.0146298 2024-09-17T16:22:29.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:22:29.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 43765 43749 68.3231 64.1875 0.0645224 0.0146298 2024-09-17T16:22:29.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:30.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:30.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:30.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:30.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:30.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:30.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:30.231 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:30.262 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:30.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:30.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:30.288 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:30 smithi017 ceph-mon[27571]: 5.7 scrub starts 2024-09-17T16:22:30.288 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:22:30.288 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:22:30.289 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:22:30.289 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:30 smithi017 ceph-mon[27571]: 5.7 scrub ok 2024-09-17T16:22:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:30 smithi071 ceph-mon[30002]: 5.7 scrub starts 2024-09-17T16:22:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:22:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:22:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:22:30.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:30 smithi071 ceph-mon[30002]: 5.7 scrub ok 2024-09-17T16:22:30.341 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:30 smithi173 ceph-mon[30530]: 5.7 scrub starts 2024-09-17T16:22:30.341 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:22:30.342 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:22:30.342 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:22:30.342 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:30 smithi173 ceph-mon[30530]: 5.7 scrub ok 2024-09-17T16:22:30.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:30.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:22:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:30.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:30.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:30.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:30.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:30.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:30.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:30.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:30.665 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:30.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:30.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:30.765 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":994,"stamp":"2024-09-17T16:22:27.973696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2477228576,"num_objects":37792,"num_object_clones":0,"num_object_copies":113376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37792,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":33956,"num_write_kb":2167494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":33109600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57419,"ondisk_log_size":57419,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":11362520,"kb_used_data":8303348,"kb_used_omap":57,"kb_used_meta":3058886,"kb_avail":1019834152,"statfs":{"total":1055945392128,"available":1044310171648,"internally_reserved":0,"allocated":8502628352,"data_stored":8496510459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58658,"internal_metadata":3132299998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,0,0,4,1,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5275648,"num_objects":-81,"num_object_clones":0,"num_object_copies":-243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-4038,"num_write_kb":-258400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-479,"num_bytes_recovered":-31391744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013238"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:22:23.952922+0000","last_change":"2024-09-17T16:22:23.952922+0000","last_active":"2024-09-17T16:22:23.952922+0000","last_peered":"2024-09-17T16:22:23.952922+0000","last_clean":"2024-09-17T16:22:23.952922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:22:23.952922+0000","last_undegraded":"2024-09-17T16:22:23.952922+0000","last_fullsized":"2024-09-17T16:22:23.952922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:22:23.952922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:22:23.952922+0000","last_clean_scrub_stamp":"2024-09-17T16:22:23.952922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":443,"state":"unknown","last_fresh":"2024-09-17T16:22:25.973457+0000","last_change":"2024-09-17T16:22:26.143724+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:22:25.973457+0000","last_undegraded":"2024-09-17T16:22:25.973457+0000","last_fullsized":"2024-09-17T16:22:25.973457+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,11],"acting":[3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:22:23.952922+0000","last_change":"2024-09-17T16:22:23.952922+0000","last_active":"2024-09-17T16:22:23.952922+0000","last_peered":"2024-09-17T16:22:23.952922+0000","last_clean":"2024-09-17T16:22:23.952922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:22:23.952922+0000","last_undegraded":"2024-09-17T16:22:23.952922+0000","last_fullsized":"2024-09-17T16:22:23.952922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:22:23.952922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:22:23.952922+0000","last_clean_scrub_stamp":"2024-09-17T16:22:23.952922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"440'2624","reported_seq":5570,"reported_epoch":440,"state":"unknown","last_fresh":"2024-09-17T16:22:23.927975+0000","last_change":"2024-09-17T16:22:24.791886+0000","last_active":"2024-09-17T16:22:24.791886+0000","last_peered":"2024-09-17T16:22:23.927975+0000","last_clean":"2024-09-17T16:22:23.927975+0000","last_became_active":"2024-09-17T16:22:15.445968+0000","last_became_peered":"2024-09-17T16:22:15.445968+0000","last_unstale":"2024-09-17T16:22:23.927975+0000","last_undegraded":"2024-09-17T16:22:23.927975+0000","last_fullsized":"2024-09-17T16:22:23.927975+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2624,"log_dups_size":0,"ondisk_log_size":2624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85983232,"num_objects":1312,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1312,"num_write_kb":83968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":480,"num_bytes_recovered":31457280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":443,"state":"unknown","last_fresh":"2024-09-17T16:22:25.973426+0000","last_change":"2024-09-17T16:22:26.143706+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:22:25.973426+0000","last_undegraded":"2024-09-17T16:22:25.973426+0000","last_fullsized":"2024-09-17T16:22:25.973426+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"442'2655","reported_seq":2721,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.109591+0000","last_change":"2024-09-17T16:22:24.987002+0000","last_active":"2024-09-17T16:22:25.109591+0000","last_peered":"2024-09-17T16:22:25.109591+0000","last_clean":"2024-09-17T16:22:25.109591+0000","last_became_active":"2024-09-17T16:22:24.985498+0000","last_became_peered":"2024-09-17T16:22:24.985498+0000","last_unstale":"2024-09-17T16:22:25.109591+0000","last_undegraded":"2024-09-17T16:22:25.109591+0000","last_fullsized":"2024-09-17T16:22:25.109591+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2655,"log_dups_size":0,"ondisk_log_size":2655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87195648,"num_objects":1331,"num_object_clones":0,"num_object_copies":3993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1331,"num_write_kb":85152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"442'2594","reported_seq":2660,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.108383+0000","last_change":"2024-09-17T16:22:24.988084+0000","last_active":"2024-09-17T16:22:25.108383+0000","last_peered":"2024-09-17T16:22:25.108383+0000","last_clean":"2024-09-17T16:22:25.108383+0000","last_became_active":"2024-09-17T16:22:24.986677+0000","last_became_peered":"2024-09-17T16:22:24.986677+0000","last_unstale":"2024-09-17T16:22:25.108383+0000","last_undegraded":"2024-09-17T16:22:25.108383+0000","last_fullsized":"2024-09-17T16:22:25.108383+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2594,"log_dups_size":0,"ondisk_log_size":2594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85295104,"num_objects":1302,"num_object_clones":0,"num_object_copies":3906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1302,"num_write_kb":83296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"440'2617","reported_seq":2670,"reported_epoch":440,"state":"unknown","last_fresh":"2024-09-17T16:22:23.934675+0000","last_change":"2024-09-17T16:22:24.791863+0000","last_active":"2024-09-17T16:22:24.791863+0000","last_peered":"2024-09-17T16:22:23.934675+0000","last_clean":"2024-09-17T16:22:23.934675+0000","last_became_active":"2024-09-17T16:21:42.616534+0000","last_became_peered":"2024-09-17T16:21:42.616534+0000","last_unstale":"2024-09-17T16:22:23.934675+0000","last_undegraded":"2024-09-17T16:22:23.934675+0000","last_fullsized":"2024-09-17T16:22:23.934675+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85753856,"num_objects":1309,"num_object_clones":0,"num_object_copies":3927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1309,"num_write_kb":83744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"442'2626","reported_seq":2692,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.204008+0000","last_change":"2024-09-17T16:22:24.999741+0000","last_active":"2024-09-17T16:22:25.204008+0000","last_peered":"2024-09-17T16:22:25.204008+0000","last_clean":"2024-09-17T16:22:25.204008+0000","last_became_active":"2024-09-17T16:22:24.996969+0000","last_became_peered":"2024-09-17T16:22:24.996969+0000","last_unstale":"2024-09-17T16:22:25.204008+0000","last_undegraded":"2024-09-17T16:22:25.204008+0000","last_fullsized":"2024-09-17T16:22:25.204008+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2626,"log_dups_size":0,"ondisk_log_size":2626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86409216,"num_objects":1319,"num_object_clones":0,"num_object_copies":3957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1319,"num_write_kb":84384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.f","version":"442'2568","reported_seq":2627,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.212376+0000","last_change":"2024-09-17T16:21:42.617220+0000","last_active":"2024-09-17T16:22:25.212376+0000","last_peered":"2024-09-17T16:22:25.212376+0000","last_clean":"2024-09-17T16:22:25.212376+0000","last_became_active":"2024-09-17T16:21:42.616065+0000","last_became_peered":"2024-09-17T16:21:42.616065+0000","last_unstale":"2024-09-17T16:22:25.212376+0000","last_undegraded":"2024-09-17T16:22:25.212376+0000","last_fullsized":"2024-09-17T16:22:25.212376+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:44.069995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168296448,"num_objects":2568,"num_object_clones":0,"num_object_copies":7704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2568,"num_write_kb":164352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,11],"acting":[5,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"442'2597","reported_seq":2661,"reported_epoch":444,"state":"active+clean","last_fresh":"2024-09-17T16:22:26.977608+0000","last_change":"2024-09-17T16:21:42.615978+0000","last_active":"2024-09-17T16:22:26.977608+0000","last_peered":"2024-09-17T16:22:26.977608+0000","last_clean":"2024-09-17T16:22:26.977608+0000","last_became_active":"2024-09-17T16:21:42.615584+0000","last_became_peered":"2024-09-17T16:21:42.615584+0000","last_unstale":"2024-09-17T16:22:26.977608+0000","last_undegraded":"2024-09-17T16:22:26.977608+0000","last_fullsized":"2024-09-17T16:22:26.977608+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2597,"log_dups_size":0,"ondisk_log_size":2597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:51.566657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170196992,"num_objects":2597,"num_object_clones":0,"num_object_copies":7791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2597,"num_write_kb":166208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"442'2662","reported_seq":2720,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.208384+0000","last_change":"2024-09-17T16:21:42.615849+0000","last_active":"2024-09-17T16:22:25.208384+0000","last_peered":"2024-09-17T16:22:25.208384+0000","last_clean":"2024-09-17T16:22:25.208384+0000","last_became_active":"2024-09-17T16:21:42.615522+0000","last_became_peered":"2024-09-17T16:21:42.615522+0000","last_unstale":"2024-09-17T16:22:25.208384+0000","last_undegraded":"2024-09-17T16:22:25.208384+0000","last_fullsized":"2024-09-17T16:22:25.208384+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2662,"log_dups_size":0,"ondisk_log_size":2662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:52.289060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174456832,"num_objects":2662,"num_object_clones":0,"num_object_copies":7986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"440'2576","reported_seq":2633,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-17T16:22:23.964204+0000","last_change":"2024-09-17T16:21:42.616080+0000","last_active":"2024-09-17T16:22:23.964204+0000","last_peered":"2024-09-17T16:22:23.964204+0000","last_clean":"2024-09-17T16:22:23.964204+0000","last_became_active":"2024-09-17T16:21:42.615761+0000","last_became_peered":"2024-09-17T16:21:42.615761+0000","last_unstale":"2024-09-17T16:22:23.964204+0000","last_undegraded":"2024-09-17T16:22:23.964204+0000","last_fullsized":"2024-09-17T16:22:23.964204+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2576,"log_dups_size":0,"ondisk_log_size":2576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:51.242156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168820736,"num_objects":2576,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":164864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"440'2577","reported_seq":2631,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T16:22:23.028597+0000","last_change":"2024-09-17T16:21:42.616006+0000","last_active":"2024-09-17T16:22:23.028597+0000","last_peered":"2024-09-17T16:22:23.028597+0000","last_clean":"2024-09-17T16:22:23.028597+0000","last_became_active":"2024-09-17T16:21:42.615693+0000","last_became_peered":"2024-09-17T16:21:42.615693+0000","last_unstale":"2024-09-17T16:22:23.028597+0000","last_undegraded":"2024-09-17T16:22:23.028597+0000","last_fullsized":"2024-09-17T16:22:23.028597+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2577,"log_dups_size":0,"ondisk_log_size":2577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:00.448756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168886272,"num_objects":2577,"num_object_clones":0,"num_object_copies":7731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":164928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"442'2550","reported_seq":2612,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.973623+0000","last_change":"2024-09-17T16:21:42.467809+0000","last_active":"2024-09-17T16:22:25.973623+0000","last_peered":"2024-09-17T16:22:25.973623+0000","last_clean":"2024-09-17T16:22:25.973623+0000","last_became_active":"2024-09-17T16:21:42.467536+0000","last_became_peered":"2024-09-17T16:21:42.467536+0000","last_unstale":"2024-09-17T16:22:25.973623+0000","last_undegraded":"2024-09-17T16:22:25.973623+0000","last_fullsized":"2024-09-17T16:22:25.973623+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:55.542588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167116800,"num_objects":2550,"num_object_clones":0,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":163200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"440'2604","reported_seq":2656,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T16:22:23.021072+0000","last_change":"2024-09-17T16:21:42.615828+0000","last_active":"2024-09-17T16:22:23.021072+0000","last_peered":"2024-09-17T16:22:23.021072+0000","last_clean":"2024-09-17T16:22:23.021072+0000","last_became_active":"2024-09-17T16:21:42.615436+0000","last_became_peered":"2024-09-17T16:21:42.615436+0000","last_unstale":"2024-09-17T16:22:23.021072+0000","last_undegraded":"2024-09-17T16:22:23.021072+0000","last_fullsized":"2024-09-17T16:22:23.021072+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2604,"log_dups_size":0,"ondisk_log_size":2604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:07.668412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170655744,"num_objects":2604,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":166656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"442'2579","reported_seq":2644,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.076670+0000","last_change":"2024-09-17T16:22:24.989180+0000","last_active":"2024-09-17T16:22:25.076670+0000","last_peered":"2024-09-17T16:22:25.076670+0000","last_clean":"2024-09-17T16:22:25.076670+0000","last_became_active":"2024-09-17T16:22:24.987655+0000","last_became_peered":"2024-09-17T16:22:24.987655+0000","last_unstale":"2024-09-17T16:22:25.076670+0000","last_undegraded":"2024-09-17T16:22:25.076670+0000","last_fullsized":"2024-09-17T16:22:25.076670+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2579,"log_dups_size":0,"ondisk_log_size":2579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84705280,"num_objects":1292,"num_object_clones":0,"num_object_copies":3876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1292,"num_write_kb":82720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[10,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"442'2652","reported_seq":2718,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.119120+0000","last_change":"2024-09-17T16:22:24.977505+0000","last_active":"2024-09-17T16:22:25.119120+0000","last_peered":"2024-09-17T16:22:25.119120+0000","last_clean":"2024-09-17T16:22:25.119120+0000","last_became_active":"2024-09-17T16:22:24.973401+0000","last_became_peered":"2024-09-17T16:22:24.973401+0000","last_unstale":"2024-09-17T16:22:25.119120+0000","last_undegraded":"2024-09-17T16:22:25.119120+0000","last_fullsized":"2024-09-17T16:22:25.119120+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87523328,"num_objects":1335,"num_object_clones":0,"num_object_copies":4005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1335,"num_write_kb":85472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"442'2594","reported_seq":2683,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.975559+0000","last_change":"2024-09-17T16:22:25.246767+0000","last_active":"2024-09-17T16:22:25.975559+0000","last_peered":"2024-09-17T16:22:25.975559+0000","last_clean":"2024-09-17T16:22:25.975559+0000","last_became_active":"2024-09-17T16:22:24.972997+0000","last_became_peered":"2024-09-17T16:22:24.972997+0000","last_unstale":"2024-09-17T16:22:25.975559+0000","last_undegraded":"2024-09-17T16:22:25.975559+0000","last_fullsized":"2024-09-17T16:22:25.975559+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'2594","last_scrub_stamp":"2024-09-17T16:22:25.246644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:22:25.246644+0000","objects_scrubbed":1289,"log_size":2594,"log_dups_size":0,"ondisk_log_size":2594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:52.129565+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84475904,"num_objects":1289,"num_object_clones":0,"num_object_copies":3867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"442'2580","reported_seq":2649,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.214052+0000","last_change":"2024-09-17T16:22:25.006691+0000","last_active":"2024-09-17T16:22:25.214052+0000","last_peered":"2024-09-17T16:22:25.214052+0000","last_clean":"2024-09-17T16:22:25.214052+0000","last_became_active":"2024-09-17T16:22:24.999864+0000","last_became_peered":"2024-09-17T16:22:24.999864+0000","last_unstale":"2024-09-17T16:22:25.214052+0000","last_undegraded":"2024-09-17T16:22:25.214052+0000","last_fullsized":"2024-09-17T16:22:25.214052+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2580,"log_dups_size":0,"ondisk_log_size":2580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84770816,"num_objects":1294,"num_object_clones":0,"num_object_copies":3882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1294,"num_write_kb":82784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[10,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.5","version":"442'2631","reported_seq":5610,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.774421+0000","last_change":"2024-09-17T16:22:25.774421+0000","last_active":"2024-09-17T16:22:25.774421+0000","last_peered":"2024-09-17T16:22:25.774421+0000","last_clean":"2024-09-17T16:22:25.774421+0000","last_became_active":"2024-09-17T16:22:24.972827+0000","last_became_peered":"2024-09-17T16:22:24.972827+0000","last_unstale":"2024-09-17T16:22:25.774421+0000","last_undegraded":"2024-09-17T16:22:25.774421+0000","last_fullsized":"2024-09-17T16:22:25.774421+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'2631","last_scrub_stamp":"2024-09-17T16:22:25.774345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:22:25.774345+0000","objects_scrubbed":1342,"log_size":2631,"log_dups_size":0,"ondisk_log_size":2631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:30.606526+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88014848,"num_objects":1343,"num_object_clones":0,"num_object_copies":4029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,3],"acting":[10,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"442'2657","reported_seq":2744,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.287740+0000","last_change":"2024-09-17T16:22:25.287688+0000","last_active":"2024-09-17T16:22:25.287740+0000","last_peered":"2024-09-17T16:22:25.287740+0000","last_clean":"2024-09-17T16:22:25.287740+0000","last_became_active":"2024-09-17T16:22:24.971506+0000","last_became_peered":"2024-09-17T16:22:24.971506+0000","last_unstale":"2024-09-17T16:22:25.287740+0000","last_undegraded":"2024-09-17T16:22:25.287740+0000","last_fullsized":"2024-09-17T16:22:25.287740+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'2656","last_scrub_stamp":"2024-09-17T16:22:25.287621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:22:25.287621+0000","objects_scrubbed":1326,"log_size":2657,"log_dups_size":0,"ondisk_log_size":2657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:37.736150+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86966272,"num_objects":1327,"num_object_clones":0,"num_object_copies":3981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"442'2595","reported_seq":2661,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.154808+0000","last_change":"2024-09-17T16:22:24.974521+0000","last_active":"2024-09-17T16:22:25.154808+0000","last_peered":"2024-09-17T16:22:25.154808+0000","last_clean":"2024-09-17T16:22:25.154808+0000","last_became_active":"2024-09-17T16:22:24.971915+0000","last_became_peered":"2024-09-17T16:22:24.971915+0000","last_unstale":"2024-09-17T16:22:25.154808+0000","last_undegraded":"2024-09-17T16:22:25.154808+0000","last_fullsized":"2024-09-17T16:22:25.154808+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2595,"log_dups_size":0,"ondisk_log_size":2595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85360640,"num_objects":1302,"num_object_clones":0,"num_object_copies":3906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1302,"num_write_kb":83360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"440'2617","reported_seq":2671,"reported_epoch":441,"state":"peering","last_fresh":"2024-09-17T16:22:23.987261+0000","last_change":"2024-09-17T16:22:23.987261+0000","last_active":"2024-09-17T16:22:23.987261+0000","last_peered":"2024-09-17T16:22:23.934675+0000","last_clean":"2024-09-17T16:22:23.934675+0000","last_became_active":"2024-09-17T16:21:42.616534+0000","last_became_peered":"2024-09-17T16:21:42.616534+0000","last_unstale":"2024-09-17T16:22:23.987261+0000","last_undegraded":"2024-09-17T16:22:23.987261+0000","last_fullsized":"2024-09-17T16:22:23.987261+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85753856,"num_objects":1308,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1308,"num_write_kb":83744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,9],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"442'2622","reported_seq":2688,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.076885+0000","last_change":"2024-09-17T16:22:24.978494+0000","last_active":"2024-09-17T16:22:25.076885+0000","last_peered":"2024-09-17T16:22:25.076885+0000","last_clean":"2024-09-17T16:22:25.076885+0000","last_became_active":"2024-09-17T16:22:24.976710+0000","last_became_peered":"2024-09-17T16:22:24.976710+0000","last_unstale":"2024-09-17T16:22:25.076885+0000","last_undegraded":"2024-09-17T16:22:25.076885+0000","last_fullsized":"2024-09-17T16:22:25.076885+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86147072,"num_objects":1314,"num_object_clones":0,"num_object_copies":3942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1314,"num_write_kb":84128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"442'2550","reported_seq":2615,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-17T16:22:25.178361+0000","last_change":"2024-09-17T16:22:24.986703+0000","last_active":"2024-09-17T16:22:25.178361+0000","last_peered":"2024-09-17T16:22:25.178361+0000","last_clean":"2024-09-17T16:22:25.178361+0000","last_became_active":"2024-09-17T16:22:24.984304+0000","last_became_peered":"2024-09-17T16:22:24.984304+0000","last_unstale":"2024-09-17T16:22:25.178361+0000","last_undegraded":"2024-09-17T16:22:25.178361+0000","last_fullsized":"2024-09-17T16:22:25.178361+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83853312,"num_objects":1279,"num_object_clones":0,"num_object_copies":3837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1279,"num_write_kb":81888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":780,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-17T16:22:21.956059+0000","last_change":"2024-09-17T16:22:06.807310+0000","last_active":"2024-09-17T16:22:21.956059+0000","last_peered":"2024-09-17T16:22:21.956059+0000","last_clean":"2024-09-17T16:22:21.956059+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:22:21.956059+0000","last_undegraded":"2024-09-17T16:22:21.956059+0000","last_fullsized":"2024-09-17T16:22:21.956059+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:22:06.807231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:22:06.807231+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:23:16.883658+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":26,"stat_sum":{"num_bytes":2476638208,"num_objects":37790,"num_object_clones":0,"num_object_copies":113370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33833,"num_write_kb":2165344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":31522816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8286961664,"data_stored":8286961664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57327,"ondisk_log_size":57327,"up":72,"acting":72,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285723,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846992,"kb_used_data":504828,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92898160,"statfs":{"total":95995035648,"available":95127715840,"internally_reserved":0,"allocated":516943872,"data_stored":516388313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":350353860},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547360,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1361044,"kb_used_data":1167424,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92384108,"statfs":{"total":95995035648,"available":94601326592,"internally_reserved":0,"allocated":1195442176,"data_stored":1194885714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":198244800},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743589,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1234720,"kb_used_data":842572,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92510432,"statfs":{"total":95995035648,"available":94730682368,"internally_reserved":0,"allocated":862793728,"data_stored":862233165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":401537472},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005225,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":476104,"kb_used_data":328308,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93269048,"statfs":{"total":95995035648,"available":95507505152,"internally_reserved":0,"allocated":336187392,"data_stored":335635405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963702,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920260,"kb_used_data":655216,"kb_used_omap":41,"kb_used_meta":264982,"kb_avail":92824892,"statfs":{"total":95995035648,"available":95052689408,"internally_reserved":0,"allocated":670941184,"data_stored":670389812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":271341875},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495796,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1038492,"kb_used_data":718536,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92706660,"statfs":{"total":95995035648,"available":94931619840,"internally_reserved":0,"allocated":735780864,"data_stored":735220850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724727,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1325016,"kb_used_data":1032832,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92420136,"statfs":{"total":95995035648,"available":94638219264,"internally_reserved":0,"allocated":1057619968,"data_stored":1057063506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986366,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1146696,"kb_used_data":832308,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":92598456,"statfs":{"total":95995035648,"available":94820818944,"internally_reserved":0,"allocated":852283392,"data_stored":851731405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":321911237},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149889,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":640344,"kb_used_data":338432,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93104808,"statfs":{"total":95995035648,"available":95339323392,"internally_reserved":0,"allocated":346554368,"data_stored":345997906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411528,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822756,"kb_used_data":665228,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92922396,"statfs":{"total":95995035648,"available":95152533504,"internally_reserved":0,"allocated":681193472,"data_stored":680632909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738578,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1550096,"kb_used_data":1217664,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":92195056,"statfs":{"total":95995035648,"available":94407737344,"internally_reserved":0,"allocated":1246887936,"data_stored":1246331474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":340392389},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1243807744,"data_stored":1243807744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":679673856,"data_stored":679673856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":345112576,"data_stored":345112576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":850788352,"data_stored":850788352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":855375872,"data_stored":855375872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":734461952,"data_stored":734461952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":672858112,"data_stored":672858112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":334692352,"data_stored":334692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":861274112,"data_stored":861274112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1194000384,"data_stored":1194000384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":514916352,"data_stored":514916352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:22:30.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-17T16:22:30.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:30.772 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:22:30.798 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:30.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:30.825 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:22:30.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:30.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:30.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:30.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 45038 45022 68.5969 79.5625 0.0041292 0.0145711 2024-09-17T16:22:31.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:31.060 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:31.061 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:22:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:31.061 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:22:31.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:31.097 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:22:31.097 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:22:31.135 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:22:31.136 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:22:31.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:31.175 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:22:31.175 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:22:31.180 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:31 smithi173 ceph-mon[30530]: 5.4 scrub starts 2024-09-17T16:22:31.180 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:31 smithi173 ceph-mon[30530]: from='client.16701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:22:31.180 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:31 smithi173 ceph-mon[30530]: pgmap v995: 27 pgs: 1 active+recovering+undersized+remapped, 5 unknown, 21 active+clean; 2.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 3450/110727 objects misplaced (3.116%) 2024-09-17T16:22:31.180 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:31 smithi173 ceph-mon[30530]: 5.4 scrub ok 2024-09-17T16:22:31.191 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:31.192 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:31.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:31.211 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:22:31.211 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:22:31.247 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:22:31.247 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:22:31.286 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:22:31.286 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:22:31.287 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:31 smithi017 ceph-mon[27571]: 5.4 scrub starts 2024-09-17T16:22:31.287 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:31 smithi017 ceph-mon[27571]: from='client.16701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:22:31.287 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:31 smithi017 ceph-mon[27571]: pgmap v995: 27 pgs: 1 active+recovering+undersized+remapped, 5 unknown, 21 active+clean; 2.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 3450/110727 objects misplaced (3.116%) 2024-09-17T16:22:31.287 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:31 smithi017 ceph-mon[27571]: 5.4 scrub ok 2024-09-17T16:22:31.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:31.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:31 smithi071 ceph-mon[30002]: 5.4 scrub starts 2024-09-17T16:22:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:31 smithi071 ceph-mon[30002]: from='client.16701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:22:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:31 smithi071 ceph-mon[30002]: pgmap v995: 27 pgs: 1 active+recovering+undersized+remapped, 5 unknown, 21 active+clean; 2.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 3450/110727 objects misplaced (3.116%) 2024-09-17T16:22:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:31 smithi071 ceph-mon[30002]: 5.4 scrub ok 2024-09-17T16:22:31.320 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:31.321 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:31.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:31.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:31.434 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:31.465 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:31.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:31.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:31.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:31.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:31.696 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:31.729 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:31.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:31.753 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:31.831 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:31.860 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:31.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:31.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:31.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 46321 46305 68.8726 80.1875 0.00995927 0.0145112 2024-09-17T16:22:31.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:31.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:31.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:31.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:32.092 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:32.119 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:32.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:32.129 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:22:32.129 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:22:32.162 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:22:32.162 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:22:32.196 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:22:32.196 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:22:32.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1941744045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:32.221 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:32.231 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:22:32.231 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:22:32.261 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:32.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:32.268 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:22:32.269 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:22:32.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1941744045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:32.319 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1941744045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:32.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:32.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:22:32.395 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:32.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:32.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:32.526 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:32.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:32.627 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:32.659 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:32.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:32.707 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:22:32.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:32.793 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:32.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:32.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:32.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:32.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:32.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:32.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 47478 47462 68.9524 72.3125 0.00696651 0.0144959 2024-09-17T16:22:33.020 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:22:33.020 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 1 pgp_num to 1 2024-09-17T16:22:33.025 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:33.059 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:33.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:33.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:33 smithi071 ceph-mon[30002]: pgmap v996: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 2.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 7908/133497 objects degraded (5.924%); 19498/133497 objects misplaced (14.606%); 17 MiB/s, 272 objects/s recovering 2024-09-17T16:22:33.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:33 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 7908/133497 objects degraded (5.924%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:33.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:33 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:22:33.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:33 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:22:33.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/753620263' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-17T16:22:33.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:33 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-17T16:22:33.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/546442216' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:22:33.141 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:33 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:22:33.161 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:33.196 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:33.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:33.298 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:33.310 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:47864 5.9 5:96e0e482:::benchmark_data_smithi173_90735_object47863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:33.301779+0000", 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0056716340000000001, 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.005707073, 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:33.311 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 47864 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.301779+0000", 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.301779+0000", 2024-09-17T16:22:33.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.301781+0000", 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.787e-06 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.301817+0000", 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6050000000000002e-05 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.313 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.301833+0000", 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5685e-05 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.301841+0000", 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9910000000000002e-06 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.301894+0000", 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3737999999999999e-05 2024-09-17T16:22:33.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.301993+0000", 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8469999999999997e-05 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.302084+0000", 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1068000000000003e-05 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:33.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.303002+0000", 2024-09-17T16:22:33.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00091764500000000003 2024-09-17T16:22:33.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:33.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.304027+0000", 2024-09-17T16:22:33.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001025492 2024-09-17T16:22:33.316 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:47858 5.b 5:d1d3e601:::benchmark_data_smithi173_90735_object47857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-09-17T16:22:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:33.292889+0000", 2024-09-17T16:22:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.014561596, 2024-09-17T16:22:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.014685805999999999, 2024-09-17T16:22:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 47858 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.292889+0000", 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.292889+0000", 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.292890+0000", 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.269e-06 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.292921+0000", 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0137999999999998e-05 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.292934+0000", 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3879999999999999e-05 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.292941+0000", 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7129999999999999e-06 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.294716+0000", 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017745 2024-09-17T16:22:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.294807+0000", 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1352000000000002e-05 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.294895+0000", 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8407999999999998e-05 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.296042+0000", 2024-09-17T16:22:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001146853 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:33.296158+0000", 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011562599999999999 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 2 2024-09-17T16:22:33.322 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:22:33.327 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:33.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:33.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:33 smithi017 ceph-mon[27571]: pgmap v996: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 2.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 7908/133497 objects degraded (5.924%); 19498/133497 objects misplaced (14.606%); 17 MiB/s, 272 objects/s recovering 2024-09-17T16:22:33.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:33 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 7908/133497 objects degraded (5.924%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:33.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:33 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:22:33.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:33 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:22:33.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/753620263' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-17T16:22:33.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:33 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-17T16:22:33.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/546442216' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:22:33.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:33 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:22:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:33 smithi173 ceph-mon[30530]: pgmap v996: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 2.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 7908/133497 objects degraded (5.924%); 19498/133497 objects misplaced (14.606%); 17 MiB/s, 272 objects/s recovering 2024-09-17T16:22:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:33 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 7908/133497 objects degraded (5.924%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:33 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:22:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:33 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:22:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/753620263' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-17T16:22:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:33 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-17T16:22:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/546442216' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:22:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:33 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:22:33.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:33.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:33.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:33.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:33.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:33.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:33.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:33.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:33.715 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:33.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:33.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:33.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:33.852 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:33.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:33.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 48614 48598 68.9988 71 0.00905752 0.0144845 2024-09-17T16:22:33.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:33.979 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:33.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:34.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:34.113 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:34.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:34.139 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:34 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:34.139 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:34 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-17T16:22:34.139 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:34 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:22:34.139 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:34 smithi071 ceph-mon[30002]: osdmap e446: 12 total, 12 up, 11 in 2024-09-17T16:22:34.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:22:34.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:34.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:34.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:34.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:34 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:34.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:34 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-17T16:22:34.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:34 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:22:34.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:34 smithi017 ceph-mon[27571]: osdmap e446: 12 total, 12 up, 11 in 2024-09-17T16:22:34.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:34.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:34.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:34.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:34 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:34.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:34 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-17T16:22:34.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:34 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:22:34.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:34 smithi173 ceph-mon[30530]: osdmap e446: 12 total, 12 up, 11 in 2024-09-17T16:22:34.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:34.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:34.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:34.582 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:22:34.612 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:34.636 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:34.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:34.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:34.768 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:34.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:34.870 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:34.894 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:34.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:34.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 49479 49463 68.6669 54.0625 0.0848912 0.0145522 2024-09-17T16:22:34.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:35.023 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:35.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:35.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:35.129 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:22:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:22:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:22:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:22:35.129 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:22:35.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:35.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:35.180 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:22:35.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:35.284 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:35.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:35.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:35 smithi017 ceph-mon[27571]: pgmap v998: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 2.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 7908/135915 objects degraded (5.818%); 19498/135915 objects misplaced (14.346%); 15 MiB/s, 236 objects/s recovering 2024-09-17T16:22:35.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:35 smithi017 ceph-mon[27571]: osdmap e447: 12 total, 12 up, 11 in 2024-09-17T16:22:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:35 smithi071 ceph-mon[30002]: pgmap v998: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 2.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 7908/135915 objects degraded (5.818%); 19498/135915 objects misplaced (14.346%); 15 MiB/s, 236 objects/s recovering 2024-09-17T16:22:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:35 smithi071 ceph-mon[30002]: osdmap e447: 12 total, 12 up, 11 in 2024-09-17T16:22:35.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:35.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:35 smithi173 ceph-mon[30530]: pgmap v998: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 2.8 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 7908/135915 objects degraded (5.818%); 19498/135915 objects misplaced (14.346%); 15 MiB/s, 236 objects/s recovering 2024-09-17T16:22:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:35 smithi173 ceph-mon[30530]: osdmap e447: 12 total, 12 up, 11 in 2024-09-17T16:22:35.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:35.411 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:35.512 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:35.543 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:35.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:35.645 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:35.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:35.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:35.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:35.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:35.806 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:35.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:35.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:22:35.908 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:35.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 50564 50548 68.6483 67.8125 0.0100146 0.0145609 2024-09-17T16:22:35.955 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:35.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:36.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:36.092 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:36.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:36.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:36.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:36.220 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:36.227 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:22:36.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:36.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:36.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:36.448 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:36.474 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:36.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:36.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:36.608 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:36.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:36.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:36.733 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:36.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:36.780 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:22:36.780 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:22:36.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:22:36.780 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:22:36.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.780 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.24550.0:152907 2.3 e199/162 2:c65c8e4e:::benchmark_data_smithi173_55121_object76418:head v 199'5280, mlcod=199'5277)", 2024-09-17T16:22:36.780 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.797893+0000", 2024-09-17T16:22:36.780 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 522.97780887399995, 2024-09-17T16:22:36.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86957243200000001, 2024-09-17T16:22:36.780 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797893+0000", 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797893+0000", 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797902+0000", 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9670000000000007e-06 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.797908+0000", 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8279999999999999e-06 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.665615+0000", 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86770716699999995 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.665626+0000", 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1146e-05 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.665718+0000", 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1854999999999994e-05 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.665739+0000", 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0394000000000002e-05 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667418+0000", 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016790710000000001 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.667466+0000", 2024-09-17T16:22:36.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8004000000000003e-05 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.24550.0:152967 2.8 2:178f648d:::benchmark_data_smithi173_55121_object76478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.857328+0000", 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 522.91837379399999, 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.592092548, 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:36.785 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.24550", 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152967 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857328+0000", 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857328+0000", 2024-09-17T16:22:36.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857329+0000", 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0559999999999999e-06 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857334+0000", 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4900000000000002e-06 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857344+0000", 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6830000000000003e-06 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857349+0000", 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4110000000000004e-06 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857387+0000", 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8015000000000001e-05 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857468+0000", 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1210000000000003e-05 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.857534+0000", 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5752000000000003e-05 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859065+0000", 2024-09-17T16:22:36.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001530942 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859971+0000", 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00090643500000000003 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449347+0000", 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58937566600000002 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:36.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449387+0000", 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9771999999999998e-05 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449421+0000", 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4116000000000003e-05 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.24550.0:152967 2.8 e199/174 ondisk, result = 0)", 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:13:53.859730+0000", 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 522.91597246900005, 2024-09-17T16:22:36.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58969649800000001, 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859730+0000", 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859730+0000", 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859730+0000", 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:53.859734+0000", 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7869999999999999e-06 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449259+0000", 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58952536 2024-09-17T16:22:36.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.794 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449283+0000", 2024-09-17T16:22:36.794 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3918999999999999e-05 2024-09-17T16:22:36.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449335+0000", 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1146000000000001e-05 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449343+0000", 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3240000000000003e-06 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:13:54.449426+0000", 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3239999999999996e-05 2024-09-17T16:22:36.795 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(3.f 240/239 [PushReplyOp(3:f5f4cfa1:::benchmark_data_smithi173_66272_object320:head)])", 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:14.963777+0000", 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 441.811924992, 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58111184800000004, 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963777+0000", 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963777+0000", 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963778+0000", 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1500000000000003e-07 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963783+0000", 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9540000000000003e-06 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.542775+0000", 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57899182000000005 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.542785+0000", 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7559999999999994e-06 2024-09-17T16:22:36.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.542840+0000", 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5593000000000001e-05 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.544889+0000", 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020488099999999999 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(3.f 241/239 [PushReplyOp(3:f282ca6b:::benchmark_data_smithi173_66272_object3185:head)])", 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:17.042771+0000", 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 439.732931415, 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86596120499999996, 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.042771+0000", 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.042771+0000", 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.042771+0000", 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5700000000000002e-07 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.042776+0000", 2024-09-17T16:22:36.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2239999999999997e-06 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.906632+0000", 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86385691099999995 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.906655+0000", 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2901999999999999e-05 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.906737+0000", 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1955999999999999e-05 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.908732+0000", 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019945549999999999 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:124271 3.f 3.7598f19f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:48.913907+0000", 2024-09-17T16:22:36.803 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 347.86179535899998, 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44303875500000001, 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 124271 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913907+0000", 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913907+0000", 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913908+0000", 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.761e-06 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913917+0000", 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9660000000000002e-06 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913932+0000", 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4611e-05 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913942+0000", 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5200000000000003e-06 2024-09-17T16:22:36.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.914003+0000", 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1857000000000001e-05 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.914010+0000", 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7939999999999997e-06 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.922734+0000", 2024-09-17T16:22:36.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0087233880000000003 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.922745+0000", 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1299e-05 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:49.356907+0000", 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43416214800000003 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:49.356945+0000", 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8411000000000002e-05 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:124274 3.f 3.5367ea5f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:48.919228+0000", 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 347.85647378300001, 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43814494599999998, 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.809 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 124274 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919228+0000", 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919228+0000", 2024-09-17T16:22:36.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919229+0000", 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0729999999999999e-06 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919239+0000", 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1460000000000002e-06 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919249+0000", 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0900000000000001e-05 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919257+0000", 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3350000000000001e-06 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.922889+0000", 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036324040000000001 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.922897+0000", 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6490000000000008e-06 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:49.357363+0000", 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43446637799999999 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:49.357373+0000", 2024-09-17T16:22:36.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0061000000000001e-05 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:124271 3.f 3:f98f19ae:::benchmark_data_smithi173_66272_object52315:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:50.373045+0000", 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 346.40265677600001, 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53754313499999995, 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:36.814 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 124271 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373045+0000", 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373045+0000", 2024-09-17T16:22:36.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373047+0000", 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1019999999999999e-06 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373074+0000", 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6693999999999999e-05 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.816 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373094+0000", 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9723000000000001e-05 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373119+0000", 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5072e-05 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373201+0000", 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1788999999999999e-05 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373225+0000", 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4304e-05 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.893185+0000", 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51995968400000003 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.893329+0000", 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014477399999999999 2024-09-17T16:22:36.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9,11", 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.893431+0000", 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010175600000000001 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.901505+0000", 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0080735990000000007 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.901549+0000", 2024-09-17T16:22:36.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4211000000000001e-05 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.908942+0000", 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0073930949999999997 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910371+0000", 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001429156 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:36.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910454+0000", 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3088000000000001e-05 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910588+0000", 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013408800000000001 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:124274 3.f 3:fa57e6ca:::benchmark_data_smithi173_66272_object52318:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:50.373143+0000", 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 346.40255929300002, 2024-09-17T16:22:36.821 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53763810700000003, 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 124274 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373143+0000", 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.823 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.823 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.823 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373143+0000", 2024-09-17T16:22:36.823 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373144+0000", 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4e-07 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373169+0000", 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5327e-05 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373179+0000", 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0419e-05 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373187+0000", 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3209999999999999e-06 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373273+0000", 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6007000000000005e-05 2024-09-17T16:22:36.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373287+0000", 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4559000000000001e-05 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.896946+0000", 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52365888000000005 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.897031+0000", 2024-09-17T16:22:36.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4889999999999995e-05 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9,11", 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.897106+0000", 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5129000000000005e-05 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.903625+0000", 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0065190990000000004 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.827 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.904361+0000", 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00073532600000000001 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.909012+0000", 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0046517360000000001 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910655+0000", 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016429190000000001 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910703+0000", 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7757999999999999e-05 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910781+0000", 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7897000000000003e-05 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.829 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:124292 3.f 3:f8d3325b:::benchmark_data_smithi173_66272_object52336:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:50.373224+0000", 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 346.40247768299997, 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53778705599999999, 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 124292 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.830 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373224+0000", 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373224+0000", 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373225+0000", 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8700000000000005e-07 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.831 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373229+0000", 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0790000000000002e-06 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373238+0000", 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8130000000000002e-06 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373366+0000", 2024-09-17T16:22:36.832 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012767799999999999 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373644+0000", 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027804400000000002 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373651+0000", 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9020000000000002e-06 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.833 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.898880+0000", 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52522909900000003 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.898963+0000", 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3573999999999996e-05 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9,11", 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.899041+0000", 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7885000000000004e-05 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.903635+0000", 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0045940720000000003 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.908105+0000", 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0044698480000000002 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.909071+0000", 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00096629500000000004 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.835 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910837+0000", 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017652670000000001 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910896+0000", 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8987999999999998e-05 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.911011+0000", 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011572499999999999 2024-09-17T16:22:36.836 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:132738 3.f 3:f3b4f8b2:::benchmark_data_smithi173_66272_object60782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:03.472328+0000", 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 333.303373884, 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.452172817, 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:22:36.837 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 132738 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472328+0000", 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472328+0000", 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472330+0000", 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9810000000000002e-06 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472338+0000", 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5530000000000004e-06 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472356+0000", 2024-09-17T16:22:36.839 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7867e-05 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472363+0000", 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.678e-06 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472414+0000", 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0523999999999999e-05 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.840 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472487+0000", 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2974999999999998e-05 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9,11", 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472600+0000", 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000112915 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.477969+0000", 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0053689280000000002 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.841 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.478014+0000", 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5596000000000002e-05 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.478619+0000", 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00060512900000000004 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924344+0000", 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44572508900000002 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.842 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924388+0000", 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3427999999999998e-05 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924501+0000", 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011315399999999999 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.843 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25036.0:132738 3.f e278/273 ondisk, result = 0)", 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:03.476714+0000", 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 333.298988345, 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44779243800000001, 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.476714+0000", 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.844 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.476714+0000", 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.476714+0000", 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4900000000000001e-07 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.476717+0000", 2024-09-17T16:22:36.845 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0359999999999998e-06 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924244+0000", 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.447526912 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924251+0000", 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9249999999999998e-06 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924318+0000", 2024-09-17T16:22:36.846 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7082999999999995e-05 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924337+0000", 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8723999999999999e-05 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924506+0000", 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000169309 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.847 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(3.f 278/273 [PushReplyOp(3:f48b3bb8:::benchmark_data_smithi173_66272_object68210:head)])", 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:03.478530+0000", 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 333.29717197999997, 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30847184100000002, 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.848 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.478530+0000", 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.478530+0000", 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.478530+0000", 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.43e-07 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.849 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.478533+0000", 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8480000000000001e-06 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.786776+0000", 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30824315800000002 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.786783+0000", 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2300000000000002e-06 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.850 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.786838+0000", 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4299999999999998e-05 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.787002+0000", 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000164162 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.851 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:17181 4.8 4:144293bb:::benchmark_data_smithi173_77474_object17180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:51.573649+0000", 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 285.20205339799998, 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.417298059, 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 17181 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.852 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.573649+0000", 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.573649+0000", 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.573651+0000", 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.928e-06 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.853 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.573697+0000", 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6514000000000002e-05 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.573711+0000", 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.428e-05 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.573722+0000", 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0499000000000001e-05 2024-09-17T16:22:36.854 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.573789+0000", 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7586e-05 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.573903+0000", 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011390799999999999 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.574030+0000", 2024-09-17T16:22:36.855 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000127033 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.574894+0000", 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00086319999999999995 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.576938+0000", 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020438869999999999 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.856 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.990880+0000", 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.413942647 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.990918+0000", 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8099000000000002e-05 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.990947+0000", 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8478000000000001e-05 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.857 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:17181 4.8 e310/301 ondisk, result = 0)", 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:51.615322+0000", 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 285.16037959900001, 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37563017900000001, 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.615322+0000", 2024-09-17T16:22:36.858 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.615322+0000", 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.615323+0000", 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2600000000000002e-07 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.859 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.615327+0000", 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8079999999999998e-06 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.990775+0000", 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37544852499999998 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.990782+0000", 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0990000000000004e-06 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.860 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.990870+0000", 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7973999999999995e-05 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.990876+0000", 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9090000000000003e-06 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:51.990953+0000", 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6237999999999998e-05 2024-09-17T16:22:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:76054 4.3 4:c3ed8a0e:::benchmark_data_smithi173_77474_object76053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:18:39.803227+0000", 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 236.972474869, 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60180096699999996, 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:22:36.862 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 76054 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803227+0000", 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803227+0000", 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.863 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803229+0000", 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.902e-06 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803254+0000", 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5031000000000001e-05 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803268+0000", 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4273e-05 2024-09-17T16:22:36.864 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803276+0000", 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0960000000000003e-06 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803323+0000", 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7163000000000002e-05 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.865 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:36.867 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.867 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803421+0000", 2024-09-17T16:22:36.867 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8788000000000004e-05 2024-09-17T16:22:36.867 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.867 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.867 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:22:36.867 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803511+0000", 2024-09-17T16:22:36.867 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9868000000000001e-05 2024-09-17T16:22:36.867 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.867 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.805487+0000", 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001975405 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.806491+0000", 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00100475 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404816+0000", 2024-09-17T16:22:36.868 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59832433699999998 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404935+0000", 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011914200000000001 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.405028+0000", 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3212000000000001e-05 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.869 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:76054 4.3 e335/306 ondisk, result = 0)", 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:18:39.805871+0000", 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 236.969831511, 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59919401100000003, 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.805871+0000", 2024-09-17T16:22:36.870 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.805871+0000", 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.805871+0000", 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9500000000000002e-07 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.871 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.805875+0000", 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.827e-06 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404595+0000", 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59872037099999997 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404608+0000", 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2846e-05 2024-09-17T16:22:36.872 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404771+0000", 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016244100000000001 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404797+0000", 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6092e-05 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.405065+0000", 2024-09-17T16:22:36.873 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026773900000000002 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:140458 4.3 4:c40a4f04:::benchmark_data_smithi173_77474_object38283:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:54.079732+0000", 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 162.69597001299999, 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50211494899999998, 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:22:36.874 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 140458 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079732+0000", 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079732+0000", 2024-09-17T16:22:36.875 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079734+0000", 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5e-06 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079741+0000", 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7340000000000008e-06 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.876 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079764+0000", 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2946999999999999e-05 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079773+0000", 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7430000000000008e-06 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079834+0000", 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1323999999999998e-05 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079922+0000", 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8029999999999996e-05 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079993+0000", 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0492000000000002e-05 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.083444+0000", 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0034509950000000001 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.084011+0000", 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00056759900000000001 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581774+0000", 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49776288600000002 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581815+0000", 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1185e-05 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.879 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581847+0000", 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1513999999999997e-05 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:140458 4.3 e361/306 ondisk, result = 0)", 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:54.084283+0000", 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 162.69141945699999, 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49756956800000002, 2024-09-17T16:22:36.880 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.084283+0000", 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.084283+0000", 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.881 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.084283+0000", 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8999999999999996e-07 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.084290+0000", 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3049999999999998e-06 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581691+0000", 2024-09-17T16:22:36.882 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49740152500000001 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581699+0000", 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3429999999999996e-06 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581763+0000", 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3686e-05 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.883 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581770+0000", 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0840000000000003e-06 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581852+0000", 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2034999999999996e-05 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:153098 4.6 e368/361 4:68123cde:::benchmark_data_smithi173_77474_object50923:head v 368'9358, mlcod=368'9355)", 2024-09-17T16:22:36.884 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:09.619552+0000", 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 147.156150123, 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45652909000000003, 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.619552+0000", 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.885 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.619552+0000", 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.619553+0000", 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2600000000000001e-07 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.619560+0000", 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2180000000000002e-06 2024-09-17T16:22:36.886 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.072256+0000", 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45269624400000003 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.072262+0000", 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3079999999999998e-06 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.072331+0000", 2024-09-17T16:22:36.887 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9907999999999995e-05 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.072355+0000", 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3960000000000001e-05 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.076033+0000", 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036773629999999999 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:22:36.888 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:22:36.889 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.076081+0000", 2024-09-17T16:22:36.889 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8263000000000002e-05 2024-09-17T16:22:36.889 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.889 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.889 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.889 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:22:36.889 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:22:36.889 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:22:36.895 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:36.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:36.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:36.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 51522 51506 68.4615 59.875 0.00820299 0.014599 2024-09-17T16:22:36.997 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:37.023 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:37.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:37.125 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:37.154 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:37.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:37.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:37.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:37.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:37 smithi071 ceph-mon[30002]: pgmap v1000: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 9.5 MiB/s wr, 152 op/s; 7908/144897 objects degraded (5.458%); 17173/144897 objects misplaced (11.852%); 21 MiB/s, 331 objects/s recovering 2024-09-17T16:22:37.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:37.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:37 smithi173 ceph-mon[30530]: pgmap v1000: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 9.5 MiB/s wr, 152 op/s; 7908/144897 objects degraded (5.458%); 17173/144897 objects misplaced (11.852%); 21 MiB/s, 331 objects/s recovering 2024-09-17T16:22:37.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:37.413 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:37 smithi017 ceph-mon[27571]: pgmap v1000: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 9.5 MiB/s wr, 152 op/s; 7908/144897 objects degraded (5.458%); 17173/144897 objects misplaced (11.852%); 21 MiB/s, 331 objects/s recovering 2024-09-17T16:22:37.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:37.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:37.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:37.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:37.669 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:37.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:37.771 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:37.805 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:37.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:37.907 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:37.928 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:37.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:37.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 52566 52550 68.3946 65.25 0.0532673 0.0146133 2024-09-17T16:22:38.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:38.060 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:38.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:38.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:38.195 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:38.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:38.198 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:38 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:22:38.198 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:38 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:22:38.198 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:38 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:22:38.198 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:38 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:22:38.198 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:38 smithi071 ceph-mon[30002]: osdmap e448: 12 total, 12 up, 11 in 2024-09-17T16:22:38.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:22:38.291 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:38 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:22:38.291 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:38 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:22:38.291 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:38 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:22:38.291 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:38 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:22:38.291 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:38 smithi017 ceph-mon[27571]: osdmap e448: 12 total, 12 up, 11 in 2024-09-17T16:22:38.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:38.332 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:38.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:38 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:22:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:38 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:22:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:38 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:22:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:38 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:22:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:38 smithi173 ceph-mon[30530]: osdmap e448: 12 total, 12 up, 11 in 2024-09-17T16:22:38.434 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:38.466 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:38.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:38.568 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:38.598 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:38.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:38.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:38.732 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:38.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:38.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:38.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:38.873 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:38.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:38.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 53930 53914 68.7383 85.25 0.00245388 0.0145423 2024-09-17T16:22:38.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:39.005 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:39.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:39.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:39.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:39.134 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:39.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:39.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:39.259 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:39.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:39.295 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:39 smithi173 ceph-mon[30530]: pgmap v1002: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 3.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s; 7908/150786 objects degraded (5.245%); 14206/150786 objects misplaced (9.421%); 30 MiB/s, 484 objects/s recovering 2024-09-17T16:22:39.295 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:39 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7908/150786 objects degraded (5.245%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:39.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:39 smithi071 ceph-mon[30002]: pgmap v1002: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 3.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s; 7908/150786 objects degraded (5.245%); 14206/150786 objects misplaced (9.421%); 30 MiB/s, 484 objects/s recovering 2024-09-17T16:22:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:39 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7908/150786 objects degraded (5.245%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:39.353 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:22:39.353 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:22:39.353 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-17T16:22:39.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd out 5 2024-09-17T16:22:39.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:39.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:39 smithi017 ceph-mon[27571]: pgmap v1002: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 3.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s; 7908/150786 objects degraded (5.245%); 14206/150786 objects misplaced (9.421%); 30 MiB/s, 484 objects/s recovering 2024-09-17T16:22:39.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:39 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7908/150786 objects degraded (5.245%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:39.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:39.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:39.488 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:39.519 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:39.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:39.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:39.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:39.648 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:39.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:39.750 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:39.774 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:39.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:39.876 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:39.899 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:39.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:39.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 54653 54637 68.2673 45.1875 0.00250176 0.0145624 2024-09-17T16:22:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:40.001 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:40.023 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:40.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:40.125 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:40.146 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:40.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:40.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:40.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:40 smithi017 ceph-mon[27571]: osdmap e449: 12 total, 12 up, 11 in 2024-09-17T16:22:40.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:40 smithi017 ceph-mon[27571]: osdmap e450: 12 total, 12 up, 11 in 2024-09-17T16:22:40.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:40 smithi017 ceph-mon[27571]: 5.14 scrub starts 2024-09-17T16:22:40.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:40.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:40.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:40 smithi071 ceph-mon[30002]: osdmap e449: 12 total, 12 up, 11 in 2024-09-17T16:22:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:40 smithi071 ceph-mon[30002]: osdmap e450: 12 total, 12 up, 11 in 2024-09-17T16:22:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:40 smithi071 ceph-mon[30002]: 5.14 scrub starts 2024-09-17T16:22:40.370 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:40.394 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:40.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:40 smithi173 ceph-mon[30530]: osdmap e449: 12 total, 12 up, 11 in 2024-09-17T16:22:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:40 smithi173 ceph-mon[30530]: osdmap e450: 12 total, 12 up, 11 in 2024-09-17T16:22:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:40 smithi173 ceph-mon[30530]: 5.14 scrub starts 2024-09-17T16:22:40.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:40.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:40.532 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:40.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:40.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:40.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:40.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:40.790 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:40.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:40.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:40.924 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:40.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:40.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 55395 55379 67.8381 46.375 0.00340178 0.0147344 2024-09-17T16:22:41.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:41.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:41.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:41 smithi173 ceph-mon[30530]: pgmap v1005: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 3.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 7908/154194 objects degraded (5.129%); 13195/154194 objects misplaced (8.557%); 37 MiB/s, 596 objects/s recovering 2024-09-17T16:22:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:41 smithi173 ceph-mon[30530]: osdmap e451: 12 total, 12 up, 11 in 2024-09-17T16:22:41.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:41.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:41.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:41.302 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:41.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:41.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:41 smithi071 ceph-mon[30002]: pgmap v1005: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 3.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 7908/154194 objects degraded (5.129%); 13195/154194 objects misplaced (8.557%); 37 MiB/s, 596 objects/s recovering 2024-09-17T16:22:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:41 smithi071 ceph-mon[30002]: osdmap e451: 12 total, 12 up, 11 in 2024-09-17T16:22:41.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:41.435 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:41.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:41 smithi017 ceph-mon[27571]: pgmap v1005: 27 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 17 active+clean; 3.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 7908/154194 objects degraded (5.129%); 13195/154194 objects misplaced (8.557%); 37 MiB/s, 596 objects/s recovering 2024-09-17T16:22:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:41 smithi017 ceph-mon[27571]: osdmap e451: 12 total, 12 up, 11 in 2024-09-17T16:22:41.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:41.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:41.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:41.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:41.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:41.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:41.728 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:41.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:41.814 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:41.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:41.915 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:22:41.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:41.939 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:41.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:41.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 56538 56522 67.9072 71.4375 0.00877288 0.0146965 2024-09-17T16:22:41.983 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:22:41.984 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:22:42.002 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:22:42.002 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:22:42.026 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:22:42.026 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:22:42.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:42.052 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:22:42.052 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:22:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:42.067 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:42.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:42.074 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:22:42.074 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:22:42.095 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:22:42.095 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:22:42.116 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:22:42.117 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:22:42.138 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:22:42.138 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:22:42.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:22:42.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:22:42.168 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:42.190 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:22:42.190 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:22:42.197 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:42.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:42.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:42.213 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:22:42.214 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:22:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:42 smithi017 ceph-mon[27571]: 5.11 scrub starts 2024-09-17T16:22:42.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1357780624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:42.242 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:22:42.242 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:22:42.299 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:42.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:42 smithi071 ceph-mon[30002]: 5.11 scrub starts 2024-09-17T16:22:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1357780624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:42.324 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:42.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:42.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:42 smithi173 ceph-mon[30530]: 5.11 scrub starts 2024-09-17T16:22:42.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1357780624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:42.426 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:42.456 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:42.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:42.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:42.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:42.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:42.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:42.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:42.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:42.822 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:42.849 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:42.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:42.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:42.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 57624 57608 67.9065 67.875 0.0220484 0.0147083 2024-09-17T16:22:42.951 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:42.984 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:42.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:43.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:43.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:43.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:43.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:43.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:43.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:43 smithi071 ceph-mon[30002]: pgmap v1007: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s; 7908/164697 objects degraded (4.802%); 9480/164697 objects misplaced (5.756%); 30 MiB/s, 475 objects/s recovering 2024-09-17T16:22:43.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:43.372 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:43.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:43.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:43 smithi173 ceph-mon[30530]: pgmap v1007: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s; 7908/164697 objects degraded (4.802%); 9480/164697 objects misplaced (5.756%); 30 MiB/s, 475 objects/s recovering 2024-09-17T16:22:43.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:43 smithi017 ceph-mon[27571]: pgmap v1007: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s; 7908/164697 objects degraded (4.802%); 9480/164697 objects misplaced (5.756%); 30 MiB/s, 475 objects/s recovering 2024-09-17T16:22:43.502 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:43.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:43.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:43.604 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:43.638 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:43.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:43.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:43.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:43.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:43.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:43.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:43.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:43.899 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:43.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 58769 58753 67.9741 71.5625 0.0299327 0.014706 2024-09-17T16:22:44.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:44.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:44.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:44.123 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:44.154 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:44.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:44.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:44.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:44.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:44.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:44 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:22:44.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:44 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:22:44.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:44 smithi071 ceph-mon[30002]: osdmap e452: 12 total, 12 up, 11 in 2024-09-17T16:22:44.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:44.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:44 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:22:44.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:44 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:22:44.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:44 smithi173 ceph-mon[30530]: osdmap e452: 12 total, 12 up, 11 in 2024-09-17T16:22:44.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:44.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:44.417 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:44 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:22:44.417 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:44 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:22:44.417 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:44 smithi017 ceph-mon[27571]: osdmap e452: 12 total, 12 up, 11 in 2024-09-17T16:22:44.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:44.530 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:44.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:44.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:44.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:44.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:44.755 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:44.779 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:44.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:44.827 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:44.828 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:44.882 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:44.910 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:44.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:44.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 59255 59239 67.2907 30.375 0.00555856 0.0147415 2024-09-17T16:22:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:44 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:22:44.733+0000 7f086a07d640 -1 mon.a@0(leader).osd e453 definitely_dead 0 2024-09-17T16:22:45.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:45.034 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:45.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:45.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:45.067 INFO:teuthology.orchestra.run.smithi017.stderr:marked out osd.5. 2024-09-17T16:22:45.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:45.158 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:45.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:45.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:45.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:45.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:45 smithi071 ceph-mon[30002]: pgmap v1009: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 7908/166968 objects degraded (4.736%); 9480/166968 objects misplaced (5.678%); 30 MiB/s, 477 objects/s recovering 2024-09-17T16:22:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:45 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7908/166968 objects degraded (4.736%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:45 smithi071 ceph-mon[30002]: osdmap e453: 12 total, 12 up, 11 in 2024-09-17T16:22:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:45 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2847073651' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-17T16:22:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:45 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-17T16:22:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:45 smithi071 ceph-mon[30002]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-17T16:22:45.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:45 smithi017 ceph-mon[27571]: pgmap v1009: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 7908/166968 objects degraded (4.736%); 9480/166968 objects misplaced (5.678%); 30 MiB/s, 477 objects/s recovering 2024-09-17T16:22:45.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:45 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7908/166968 objects degraded (4.736%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:45.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:45 smithi017 ceph-mon[27571]: osdmap e453: 12 total, 12 up, 11 in 2024-09-17T16:22:45.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:45 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2847073651' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-17T16:22:45.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:45 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-17T16:22:45.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:45 smithi017 ceph-mon[27571]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-17T16:22:45.384 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:45 smithi173 ceph-mon[30530]: pgmap v1009: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 7908/166968 objects degraded (4.736%); 9480/166968 objects misplaced (5.678%); 30 MiB/s, 477 objects/s recovering 2024-09-17T16:22:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:45 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7908/166968 objects degraded (4.736%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:45 smithi173 ceph-mon[30530]: osdmap e453: 12 total, 12 up, 11 in 2024-09-17T16:22:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:45 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2847073651' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-17T16:22:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:45 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-17T16:22:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:45 smithi173 ceph-mon[30530]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-17T16:22:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:45.407 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:45.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:45.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:45.531 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:45.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:45.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:45.655 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:45.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:45.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:45.780 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:45.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:45.882 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:45.906 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:45.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:45.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 59342 59326 66.1864 5.4375 0.00621559 0.0148717 2024-09-17T16:22:46.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:46.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:46.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:46.037 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:22:46.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:46.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:46.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:46.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:46.227 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:46.258 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:46.280 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:46.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:46.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:46 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-17T16:22:46.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:46 smithi017 ceph-mon[27571]: osdmap e454: 12 total, 12 up, 10 in 2024-09-17T16:22:46.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:46 smithi017 ceph-mon[27571]: 5.13 scrub starts 2024-09-17T16:22:46.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:46 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-17T16:22:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:46 smithi173 ceph-mon[30530]: osdmap e454: 12 total, 12 up, 10 in 2024-09-17T16:22:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:46 smithi173 ceph-mon[30530]: 5.13 scrub starts 2024-09-17T16:22:46.403 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:46.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:46.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:46.528 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:46.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:46 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-17T16:22:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:46 smithi071 ceph-mon[30002]: osdmap e454: 12 total, 12 up, 10 in 2024-09-17T16:22:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:46 smithi071 ceph-mon[30002]: 5.13 scrub starts 2024-09-17T16:22:46.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:46.652 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:46.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:46.754 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:46.776 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:46.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:46.878 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:46.902 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:46.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:46.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 59342 59326 65.0255 0 - 0.0148717 2024-09-17T16:22:47.003 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:47.028 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:47.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:47.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:47.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:47.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:47.259 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:47.273 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:47 smithi017 ceph-mon[27571]: pgmap v1012: 27 pgs: 2 remapped+peering, 3 peering, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped, 1 active+recovering+undersized+remapped, 17 active+clean; 3.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 72 MiB/s wr, 1.16k op/s; 7908/173037 objects degraded (4.570%); 5484/173037 objects misplaced (3.169%); 39 MiB/s, 628 objects/s recovering 2024-09-17T16:22:47.274 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:47 smithi017 ceph-mon[27571]: osdmap e455: 12 total, 12 up, 10 in 2024-09-17T16:22:47.304 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:47.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:47.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:47.332 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:47.367 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:22:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:47 smithi173 ceph-mon[30530]: pgmap v1012: 27 pgs: 2 remapped+peering, 3 peering, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped, 1 active+recovering+undersized+remapped, 17 active+clean; 3.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 72 MiB/s wr, 1.16k op/s; 7908/173037 objects degraded (4.570%); 5484/173037 objects misplaced (3.169%); 39 MiB/s, 628 objects/s recovering 2024-09-17T16:22:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:47 smithi173 ceph-mon[30530]: osdmap e455: 12 total, 12 up, 10 in 2024-09-17T16:22:47.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:47.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:47.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:47.550 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:47 smithi071 ceph-mon[30002]: pgmap v1012: 27 pgs: 2 remapped+peering, 3 peering, 3 active+recovery_wait+undersized+degraded+remapped, 1 remapped, 1 active+recovering+undersized+remapped, 17 active+clean; 3.5 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 72 MiB/s wr, 1.16k op/s; 7908/173037 objects degraded (4.570%); 5484/173037 objects misplaced (3.169%); 39 MiB/s, 628 objects/s recovering 2024-09-17T16:22:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:47 smithi071 ceph-mon[30002]: osdmap e455: 12 total, 12 up, 10 in 2024-09-17T16:22:47.574 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:47.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:47.676 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:47.702 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:47.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:47.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:47.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:47.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:47.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:47.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:47.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:47.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 59409 59393 63.9768 2.09375 0.00454915 0.015306 2024-09-17T16:22:47.989 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:22:47.989 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:22:48.050 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:48.073 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:48.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:48.101 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:22:48.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:48 smithi017 ceph-mon[27571]: 5.15 continuing backfill to osd.0 from (0'0,453'3178] 5:a839abad:::benchmark_data_smithi173_90735_object3618:head to 453'3178 2024-09-17T16:22:48.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:48 smithi017 ceph-mon[27571]: osdmap e456: 12 total, 12 up, 10 in 2024-09-17T16:22:48.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1714903028' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:48.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:48 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:48.176 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:48.199 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:48.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:48.301 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:48 smithi071 ceph-mon[30002]: 5.15 continuing backfill to osd.0 from (0'0,453'3178] 5:a839abad:::benchmark_data_smithi173_90735_object3618:head to 453'3178 2024-09-17T16:22:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:48 smithi071 ceph-mon[30002]: osdmap e456: 12 total, 12 up, 10 in 2024-09-17T16:22:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1714903028' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:48 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:48.333 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:48.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:48.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:48 smithi173 ceph-mon[30530]: 5.15 continuing backfill to osd.0 from (0'0,453'3178] 5:a839abad:::benchmark_data_smithi173_90735_object3618:head to 453'3178 2024-09-17T16:22:48.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:48 smithi173 ceph-mon[30530]: osdmap e456: 12 total, 12 up, 10 in 2024-09-17T16:22:48.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1714903028' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:48.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:48 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:22:48.435 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:48.464 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:48.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:48.566 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:48.590 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:48.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:48.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:48.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:48.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:48.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:48.843 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:48.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:48.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:48.945 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:48.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 59434 59418 62.9193 1.5625 0.00397598 0.0153482 2024-09-17T16:22:48.972 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:48.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:49.019 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:49.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:49.099 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:49.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:49.201 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:49.223 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:49.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:49 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:22:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:49 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:22:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:49 smithi017 ceph-mon[27571]: pgmap v1015: 27 pgs: 3 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped, 3 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 45 MiB/s wr, 724 op/s; 2584/175662 objects degraded (1.471%); 9588/175662 objects misplaced (5.458%); 29 MiB/s, 461 objects/s recovering 2024-09-17T16:22:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:49 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:49 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:49 smithi017 ceph-mon[27571]: osdmap e457: 12 total, 12 up, 10 in 2024-09-17T16:22:49.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:49.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:49.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:49 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:22:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:49 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:22:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:49 smithi173 ceph-mon[30530]: pgmap v1015: 27 pgs: 3 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped, 3 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 45 MiB/s wr, 724 op/s; 2584/175662 objects degraded (1.471%); 9588/175662 objects misplaced (5.458%); 29 MiB/s, 461 objects/s recovering 2024-09-17T16:22:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:49 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:49 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:49 smithi173 ceph-mon[30530]: osdmap e457: 12 total, 12 up, 10 in 2024-09-17T16:22:49.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:49.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:49.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:49 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:22:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:49 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:22:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:49 smithi071 ceph-mon[30002]: pgmap v1015: 27 pgs: 3 remapped+peering, 3 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped, 3 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 45 MiB/s wr, 724 op/s; 2584/175662 objects degraded (1.471%); 9588/175662 objects misplaced (5.458%); 29 MiB/s, 461 objects/s recovering 2024-09-17T16:22:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:49 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:49 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:22:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:49 smithi071 ceph-mon[30002]: osdmap e457: 12 total, 12 up, 10 in 2024-09-17T16:22:49.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:49.596 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:49.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:49.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:22:49.697 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:49.728 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:49.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:49.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:49.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:49.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:49.856 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:49.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:22:49.949887+0000 min lat: 0.00103987 max lat: 4.71965 avg lat: 0.0156897 2024-09-17T16:22:49.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:22:49.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 59488 59472 61.9271 3.375 0.0185179 0.0156897 2024-09-17T16:22:49.957 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:50.012 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:50.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:50.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:50.139 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:50.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:50.193 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:50 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2584/175662 objects degraded (1.471%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:22:50.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:50.252 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:50.268 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:22:50.269 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:22:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:50.270 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:50.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:50.306 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:22:50.306 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:22:50.338 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:22:50.338 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:22:50.371 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:50.372 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:22:50.373 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:22:50.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:50 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2584/175662 objects degraded (1.471%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:22:50.406 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:22:50.406 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:22:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:50.410 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:50.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:50.437 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:22:50.438 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:22:50.473 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:22:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:22:50.502 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:22:50.503 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:22:50.511 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:50.530 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:22:50.530 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:22:50.541 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:50.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:50.558 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:22:50.559 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:22:50.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:50 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2584/175662 objects degraded (1.471%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:22:50.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:22:50.591 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:22:50.630 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:22:50.630 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:22:50.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:50.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:50.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:50.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:50.799 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:50.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:50.900 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:50.926 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:50.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:50.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 60041 60025 61.4786 34.5625 0.0421141 0.0162319 2024-09-17T16:22:51.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:22:51.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:51.052 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:51.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:51.107 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:22:51.107 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:22:51.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:22:51.107 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:22:51.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.107 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:152973 2.19 e199/114 2:9e1f786b:::benchmark_data_smithi173_55121_object76484:head v 199'5124, mlcod=199'5121)", 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.867324+0000", 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 537.01129460100003, 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59753164299999995, 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867324+0000", 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867324+0000", 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867325+0000", 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.78e-07 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867331+0000", 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6239999999999996e-06 2024-09-17T16:22:51.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462706+0000", 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59537525499999999 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462713+0000", 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4019999999999999e-06 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462789+0000", 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6006000000000004e-05 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.462816+0000", 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7087e-05 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.464821+0000", 2024-09-17T16:22:51.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002004731 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.464855+0000", 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4560000000000001e-05 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:8110 3.b e220/208 3:de8160e2:::benchmark_data_smithi173_66272_object8109:head v 220'485, mlcod=220'481)", 2024-09-17T16:22:51.112 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:14:34.926435+0000", 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 495.95218339199999, 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.293198399, 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926435+0000", 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926435+0000", 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926436+0000", 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.049e-06 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:34.926456+0000", 2024-09-17T16:22:51.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0636e-05 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219071+0000", 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29261461100000002 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219077+0000", 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9880000000000001e-06 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219126+0000", 2024-09-17T16:22:51.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8807000000000002e-05 2024-09-17T16:22:51.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219142+0000", 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5846e-05 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219593+0000", 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045157699999999999 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:14:35.219633+0000", 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9885000000000002e-05 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.118 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124293 3.e 3:74c76eb5:::benchmark_data_smithi173_66272_object52337:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:22:51.118 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:22:51.118 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 360.50670313000001, 2024-09-17T16:22:51.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.55229465499999997, 2024-09-17T16:22:51.118 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.118 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124293 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371917+0000", 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1430000000000001e-06 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371923+0000", 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1709999999999999e-06 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371936+0000", 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2947e-05 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371951+0000", 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4547e-05 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.372152+0000", 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020128999999999999 2024-09-17T16:22:51.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.372168+0000", 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6218999999999998e-05 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.916603+0000", 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54443496499999999 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.919612+0000", 2024-09-17T16:22:51.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030085120000000001 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.919736+0000", 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012435899999999999 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.923884+0000", 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041481460000000001 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.923928+0000", 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4159999999999997e-05 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924005+0000", 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6178000000000001e-05 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924024+0000", 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9318e-05 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924094+0000", 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9605999999999997e-05 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924210+0000", 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116094 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.125 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:132741 3.e 3:729d5b9e:::benchmark_data_smithi173_66272_object60785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-17T16:22:51.126 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:22:51.126 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 347.24669442700002, 2024-09-17T16:22:51.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43202703199999998, 2024-09-17T16:22:51.126 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.126 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.126 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:22:51.126 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 132741 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:22:51.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631925+0000", 2024-09-17T16:22:51.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3239999999999999e-06 2024-09-17T16:22:51.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631930+0000", 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6700000000000002e-06 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631942+0000", 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1857e-05 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631947+0000", 2024-09-17T16:22:51.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8980000000000001e-06 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631998+0000", 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0365000000000003e-05 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.632091+0000", 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3592999999999997e-05 2024-09-17T16:22:51.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:22:51.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.632178+0000", 2024-09-17T16:22:51.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6648000000000001e-05 2024-09-17T16:22:51.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:22:51.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635422+0000", 2024-09-17T16:22:51.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003244289 2024-09-17T16:22:51.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.636256+0000", 2024-09-17T16:22:51.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083370299999999998 2024-09-17T16:22:51.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.636486+0000", 2024-09-17T16:22:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022978699999999999 2024-09-17T16:22:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063733+0000", 2024-09-17T16:22:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42724667900000002 2024-09-17T16:22:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063826+0000", 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3579999999999995e-05 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063951+0000", 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012463900000000001 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25036.0:132741 3.e e278/273 ondisk, result = 0)", 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 347.24268305700002, 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42802421600000001, 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:22:51.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:22:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635936+0000", 2024-09-17T16:22:51.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.023e-06 2024-09-17T16:22:51.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635941+0000", 2024-09-17T16:22:51.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5000000000000001e-06 2024-09-17T16:22:51.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.060074+0000", 2024-09-17T16:22:51.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42413389000000001 2024-09-17T16:22:51.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.060083+0000", 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6670000000000002e-06 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063717+0000", 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036338640000000001 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063726+0000", 2024-09-17T16:22:51.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.789e-06 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063959+0000", 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023348299999999999 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.140 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:134243 3.1 e280/272 3:8c1e41ff:::benchmark_data_smithi173_66272_object62287:head v 280'8469, mlcod=247'4427)", 2024-09-17T16:22:51.141 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:05.911868+0000", 2024-09-17T16:22:51.141 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 344.96675032000002, 2024-09-17T16:22:51.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26671571799999999, 2024-09-17T16:22:51.141 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.141 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.141 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.141 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911868+0000", 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911868+0000", 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911869+0000", 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911874+0000", 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4380000000000003e-06 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169900+0000", 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25802566399999999 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169906+0000", 2024-09-17T16:22:51.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4180000000000002e-06 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169967+0000", 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1247000000000005e-05 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169991+0000", 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4151999999999998e-05 2024-09-17T16:22:51.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.178494+0000", 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0085021780000000009 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.178584+0000", 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9865999999999996e-05 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.145 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:135118 3.e 3:7ce4ae2e:::benchmark_data_smithi173_66272_object63162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 343.60820969399998, 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34112115999999998, 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 135118 2024-09-17T16:22:51.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270411+0000", 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3520000000000001e-06 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270420+0000", 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270443+0000", 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2789000000000002e-05 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270453+0000", 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0322e-05 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270692+0000", 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023867100000000001 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270833+0000", 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014147500000000001 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.271004+0000", 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017102299999999999 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279153+0000", 2024-09-17T16:22:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081486939999999997 2024-09-17T16:22:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279218+0000", 2024-09-17T16:22:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5320999999999993e-05 2024-09-17T16:22:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:22:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279235+0000", 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6402999999999999e-05 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611337+0000", 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33210225300000001 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611397+0000", 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9757999999999999e-05 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611530+0000", 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013292299999999999 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:136353 3.e 3:75c6ba64:::benchmark_data_smithi173_66272_object64397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 340.92982548100002, 2024-09-17T16:22:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34635525499999997, 2024-09-17T16:22:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:22:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:22:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 136353 2024-09-17T16:22:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948795+0000", 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4269999999999998e-06 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948804+0000", 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948823+0000", 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8892000000000001e-05 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948832+0000", 2024-09-17T16:22:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0629999999999995e-06 2024-09-17T16:22:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948903+0000", 2024-09-17T16:22:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0796000000000005e-05 2024-09-17T16:22:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.949058+0000", 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015499099999999999 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.949219+0000", 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016127900000000001 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953488+0000", 2024-09-17T16:22:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042689440000000002 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953702+0000", 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021406499999999999 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953721+0000", 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8936999999999998e-05 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294938+0000", 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.341217465 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295013+0000", 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4389999999999998e-05 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295148+0000", 2024-09-17T16:22:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000135148 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25036.0:136353 3.e e284/273 ondisk, result = 0)", 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 340.92615316199999, 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.342690457, 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:22:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:22:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952466+0000", 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0389999999999999e-06 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952472+0000", 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6910000000000002e-06 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294851+0000", 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34237905899999999 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294856+0000", 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3499999999999996e-06 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294916+0000", 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9580000000000002e-05 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294928+0000", 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2571999999999999e-05 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295155+0000", 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000227166 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:143899 3.1 e290/272 3:8d15b72a:::benchmark_data_smithi173_66272_object71943:head v 290'9096, mlcod=261'5689)", 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 330.39949858099999, 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37496784399999999, 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:22:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9699999999999996e-07 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479128+0000", 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850526+0000", 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37139821699999997 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850534+0000", 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1760000000000004e-06 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850583+0000", 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9013e-05 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850601+0000", 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8916e-05 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.854050+0000", 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034480639999999998 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.854087+0000", 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7846000000000003e-05 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:143908 3.e 3:70cae513:::benchmark_data_smithi173_66272_object71952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 330.38944741799997, 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35428895300000002, 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 143908 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489172+0000", 2024-09-17T16:22:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6700000000000001e-06 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489177+0000", 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.651e-06 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489191+0000", 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3497e-05 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489197+0000", 2024-09-17T16:22:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0719999999999996e-06 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489244+0000", 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7037999999999999e-05 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489341+0000", 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7664000000000005e-05 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489424+0000", 2024-09-17T16:22:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2916000000000004e-05 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493180+0000", 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037554379999999998 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493210+0000", 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0076000000000001e-05 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.494016+0000", 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080592900000000004 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843310+0000", 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34929427400000002 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843362+0000", 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2549000000000001e-05 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843460+0000", 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7178999999999999e-05 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(4.6 383/382 [PushOp(4:63469528:::benchmark_data_smithi173_77474_object91937:head, version: 345'5625, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:63469528:::benchmark_data_smithi173_77474_object91937:head@345'5625, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 128.48218701799999, 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29598169099999999, 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396432+0000", 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9999999999999997e-07 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396467+0000", 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5184000000000001e-05 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692000+0000", 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29553331999999999 2024-09-17T16:22:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692039+0000", 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8491999999999997e-05 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692173+0000", 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134411 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692186+0000", 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3118e-05 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692413+0000", 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022656600000000001 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_backfill(progress 4.6 e 383/383 lb 4:6341a44b:::benchmark_data_smithi173_77474_object88529:head)", 2024-09-17T16:22:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 128.48212719200001, 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29619305299999998, 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396495+0000", 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9450000000000003e-06 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692562+0000", 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29606637800000002 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692584+0000", 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2526000000000001e-05 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692645+0000", 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1309999999999994e-05 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692657+0000", 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1798e-05 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692684+0000", 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6821e-05 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:2268 5.2 e424/419 5:448d2a5f:::benchmark_data_smithi173_90735_object2267:head v 424'138, mlcod=424'135)", 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:51.493252+0000", 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 59.385365970000002, 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26039381099999998, 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.493252+0000", 2024-09-17T16:22:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.493252+0000", 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.493253+0000", 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.06e-07 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.493347+0000", 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4554000000000003e-05 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.752984+0000", 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.2596366 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.752992+0000", 2024-09-17T16:22:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5879999999999998e-06 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.753063+0000", 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0337000000000001e-05 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.753090+0000", 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7562000000000001e-05 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.753603+0000", 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051267200000000004 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.753646+0000", 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2991999999999998e-05 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(5.13 450/444 [PushReplyOp(5:ce04d783:::benchmark_data_smithi173_90735_object32464:head)])", 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:39.936908+0000", 2024-09-17T16:22:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 10.941710519000001, 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.258692325, 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.936908+0000", 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.936908+0000", 2024-09-17T16:22:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.936908+0000", 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7700000000000004e-07 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.936912+0000", 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9160000000000003e-06 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.195065+0000", 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25815273900000002 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.195073+0000", 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9100000000000005e-06 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.195133+0000", 2024-09-17T16:22:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0081999999999998e-05 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.195600+0000", 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046710099999999999 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:54653 5.18 5:189bdb07:::benchmark_data_smithi173_90735_object54652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:39.943343+0000", 2024-09-17T16:22:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 10.935275451999999, 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25819365399999999, 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54653 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.943343+0000", 2024-09-17T16:22:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.943343+0000", 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.943344+0000", 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.387e-06 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.943412+0000", 2024-09-17T16:22:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7786000000000005e-05 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.943424+0000", 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2241e-05 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.943430+0000", 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6180000000000003e-06 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.943494+0000", 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4190999999999996e-05 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.943708+0000", 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021455000000000001 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,6,7", 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.943910+0000", 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020145499999999999 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:22:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.945224+0000", 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001314155 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.945969+0000", 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00074458200000000003 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.147173+0000", 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.20120450000000001 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.201137+0000", 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.053964067999999997 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.201212+0000", 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4785999999999999e-05 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.201536+0000", 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032433499999999998 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(5.18 450/444 [PushReplyOp(5:1bb1cd69:::benchmark_data_smithi173_90735_object21638:head)])", 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:39.945259+0000", 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 10.93335929, 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25582289499999999, 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.945259+0000", 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.945259+0000", 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.945259+0000", 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2799999999999996e-07 2024-09-17T16:22:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.945267+0000", 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4170000000000003e-06 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.200374+0000", 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25510751599999998 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.200380+0000", 2024-09-17T16:22:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3419999999999998e-06 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.200424+0000", 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4434999999999997e-05 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.201082+0000", 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065765699999999997 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:59843 5.c e458/419 5:33ae617c:::benchmark_data_smithi173_90735_object59842:head v 458'3698, mlcod=457'3695)", 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:50.490873+0000", 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.38774493700000001, 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.116679832, 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.490873+0000", 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.490873+0000", 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.490874+0000", 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8099999999999998e-07 2024-09-17T16:22:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.491004+0000", 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013056499999999999 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.605519+0000", 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.11451439300000001 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.605527+0000", 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3659999999999992e-06 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.605596+0000", 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9123999999999998e-05 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.605622+0000", 2024-09-17T16:22:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6282999999999999e-05 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.607510+0000", 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018875649999999999 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.607553+0000", 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3155000000000003e-05 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:59864 5.c e458/419 5:355c4c9f:::benchmark_data_smithi173_90735_object59863:head v 458'3699, mlcod=457'3695)", 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:50.555591+0000", 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.32302758599999998, 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.105836969, 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.555591+0000", 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.555591+0000", 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.555591+0000", 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9400000000000004e-07 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.555664+0000", 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2564999999999994e-05 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.659365+0000", 2024-09-17T16:22:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.103700708 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.659372+0000", 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4410000000000004e-06 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.659426+0000", 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3913999999999999e-05 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.659448+0000", 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1554000000000001e-05 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.661367+0000", 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019195359999999999 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.661428+0000", 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0457000000000001e-05 2024-09-17T16:22:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:59870 5.3 e458/455 5:c5924d15:::benchmark_data_smithi173_90735_object59869:head v 458'3220, mlcod=425'485)", 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:50.581654+0000", 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.29696397200000002, 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.12736665, 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:22:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.581654+0000", 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.581654+0000", 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:22:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.581665+0000", 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0859999999999999e-05 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.581734+0000", 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8546000000000004e-05 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.708036+0000", 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.12630270299999999 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.708041+0000", 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7480000000000001e-06 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.708082+0000", 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.121e-05 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.708098+0000", 2024-09-17T16:22:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5735000000000001e-05 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.708995+0000", 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000897433 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:50.709021+0000", 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5415e-05 2024-09-17T16:22:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:22:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:22:51.220 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:22:51.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:51.251 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:51.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:51.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:51.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:51.377 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:51.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:51.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:51 smithi173 ceph-mon[30530]: pgmap v1017: 27 pgs: 1 active+undersized+degraded+remapped, 2 remapped+peering, 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 36 MiB/s wr, 569 op/s; 6516/177324 objects degraded (3.675%); 12388/177324 objects misplaced (6.986%); 20 MiB/s, 325 objects/s recovering 2024-09-17T16:22:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2630298947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:51 smithi173 ceph-mon[30530]: osdmap e458: 12 total, 12 up, 10 in 2024-09-17T16:22:51.480 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:51 smithi017 ceph-mon[27571]: pgmap v1017: 27 pgs: 1 active+undersized+degraded+remapped, 2 remapped+peering, 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 36 MiB/s wr, 569 op/s; 6516/177324 objects degraded (3.675%); 12388/177324 objects misplaced (6.986%); 20 MiB/s, 325 objects/s recovering 2024-09-17T16:22:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2630298947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:51 smithi017 ceph-mon[27571]: osdmap e458: 12 total, 12 up, 10 in 2024-09-17T16:22:51.515 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:51.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:51 smithi071 ceph-mon[30002]: pgmap v1017: 27 pgs: 1 active+undersized+degraded+remapped, 2 remapped+peering, 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 36 MiB/s wr, 569 op/s; 6516/177324 objects degraded (3.675%); 12388/177324 objects misplaced (6.986%); 20 MiB/s, 325 objects/s recovering 2024-09-17T16:22:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2630298947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:51 smithi071 ceph-mon[30002]: osdmap e458: 12 total, 12 up, 10 in 2024-09-17T16:22:51.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:51.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:51.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:51.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:51.745 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:51.768 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:22:51.768 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:22:51.769 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-17T16:22:51.769 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-17T16:22:51.776 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:51.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:51.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:51.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:51.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:51.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 15 60601 60586 61.0526 35.0625 0.0124683 0.0163508 2024-09-17T16:22:52.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:52.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:52.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:52.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:52.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:52.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:52.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:22:52.258 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:52.293 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:52.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:52.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:52.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:52.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:52 smithi017 ceph-mon[27571]: osdmap e459: 12 total, 12 up, 10 in 2024-09-17T16:22:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:52 smithi017 ceph-mon[27571]: 5.18 scrub starts 2024-09-17T16:22:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:52 smithi017 ceph-mon[27571]: pgmap v1020: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 15 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 18 MiB/s wr, 284 op/s; 6517/179274 objects degraded (3.635%); 20932/179274 objects misplaced (11.676%); 29 MiB/s, 456 objects/s recovering 2024-09-17T16:22:52.520 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:52.545 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:52.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:52.648 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:52.674 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:52.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:52.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:52.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:52.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:52 smithi071 ceph-mon[30002]: osdmap e459: 12 total, 12 up, 10 in 2024-09-17T16:22:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:52 smithi071 ceph-mon[30002]: 5.18 scrub starts 2024-09-17T16:22:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:52 smithi071 ceph-mon[30002]: pgmap v1020: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 15 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 18 MiB/s wr, 284 op/s; 6517/179274 objects degraded (3.635%); 20932/179274 objects misplaced (11.676%); 29 MiB/s, 456 objects/s recovering 2024-09-17T16:22:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:52 smithi173 ceph-mon[30530]: osdmap e459: 12 total, 12 up, 10 in 2024-09-17T16:22:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:52 smithi173 ceph-mon[30530]: 5.18 scrub starts 2024-09-17T16:22:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:52 smithi173 ceph-mon[30530]: pgmap v1020: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 15 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 18 MiB/s wr, 284 op/s; 6517/179274 objects degraded (3.635%); 20932/179274 objects misplaced (11.676%); 29 MiB/s, 456 objects/s recovering 2024-09-17T16:22:52.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:52.918 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:52.918 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:52.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 61114 61098 60.5915 32 0.0272891 0.0164952 2024-09-17T16:22:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:52.967 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:52.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:53.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:53.095 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:53.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:53.198 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:53.230 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:53.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:53.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:53.364 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:53.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:53.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:53.494 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:53.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:53.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:53.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:53.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:53.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:53.728 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:53.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:53 smithi017 ceph-mon[27571]: osdmap e460: 12 total, 12 up, 10 in 2024-09-17T16:22:53.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:53 smithi017 ceph-mon[27571]: 5.15 continuing backfill to osd.0 from (0'0,458'3210] 5:a8690a71:::benchmark_data_smithi173_90735_object52034:head to 458'3210 2024-09-17T16:22:53.753 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:53.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:53 smithi071 ceph-mon[30002]: osdmap e460: 12 total, 12 up, 10 in 2024-09-17T16:22:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:53 smithi071 ceph-mon[30002]: 5.15 continuing backfill to osd.0 from (0'0,458'3210] 5:a8690a71:::benchmark_data_smithi173_90735_object52034:head to 458'3210 2024-09-17T16:22:53.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:53.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:53.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:53.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:53 smithi173 ceph-mon[30530]: osdmap e460: 12 total, 12 up, 10 in 2024-09-17T16:22:53.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:53 smithi173 ceph-mon[30530]: 5.15 continuing backfill to osd.0 from (0'0,458'3210] 5:a8690a71:::benchmark_data_smithi173_90735_object52034:head to 458'3210 2024-09-17T16:22:53.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 62012 61996 60.5217 56.125 0.00626224 0.0165165 2024-09-17T16:22:53.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:54.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:54.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:54.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:54.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:54.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:54.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:54.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:54.268 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:54.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:54.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:54.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:54.400 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:54.462 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:22:54.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:54.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:54.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:54.633 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:54.661 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:54.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:54 smithi017 ceph-mon[27571]: pgmap v1022: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 15 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 13 MiB/s wr, 214 op/s; 6517/180618 objects degraded (3.608%); 20571/180618 objects misplaced (11.389%); 27 MiB/s, 430 objects/s recovering 2024-09-17T16:22:54.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3622640401' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:54.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:54 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:54.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:54 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 6517/179274 objects degraded (3.635%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:54.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:54 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:54.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:54 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:22:54.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:54 smithi017 ceph-mon[27571]: osdmap e461: 12 total, 12 up, 10 in 2024-09-17T16:22:54.744 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-17T16:22:54.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:54 smithi071 ceph-mon[30002]: pgmap v1022: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 15 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 13 MiB/s wr, 214 op/s; 6517/180618 objects degraded (3.608%); 20571/180618 objects misplaced (11.389%); 27 MiB/s, 430 objects/s recovering 2024-09-17T16:22:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3622640401' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:54 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:54 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 6517/179274 objects degraded (3.635%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:54 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:54 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:22:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:54 smithi071 ceph-mon[30002]: osdmap e461: 12 total, 12 up, 10 in 2024-09-17T16:22:54.823 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:54.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:54 smithi173 ceph-mon[30530]: pgmap v1022: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 15 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 13 MiB/s wr, 214 op/s; 6517/180618 objects degraded (3.608%); 20571/180618 objects misplaced (11.389%); 27 MiB/s, 430 objects/s recovering 2024-09-17T16:22:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3622640401' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:54 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:22:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:54 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 6517/179274 objects degraded (3.635%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:22:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:54 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:22:54.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:54 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:22:54.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:54 smithi173 ceph-mon[30530]: osdmap e461: 12 total, 12 up, 10 in 2024-09-17T16:22:54.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:22:54.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:54.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 62761 62745 60.3108 46.8125 0.0112155 0.0165719 2024-09-17T16:22:54.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:54.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:55.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:22:55.080 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:55.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:22:55.183 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:55.223 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:55.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:55.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:55.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:55.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:55.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:55.456 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:55.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:55.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:55.517 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:55.540 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:55.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:55.616 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:55.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:55.718 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:55.745 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:55.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:55.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:55.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:55.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:55.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 63436 63420 60.0362 42.1875 0.0153903 0.0166465 2024-09-17T16:22:55.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:56.000 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:56.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:56.106 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:56.143 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:56.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:56.244 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:56.253 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:22:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:22:56.253 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:22:56.253 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:22:56.279 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:56.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:56.381 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:22:56.413 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:56.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:56.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:22:56.480 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:56.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:56.544 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:56.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:56.646 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:56.682 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:56.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:56.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:56.815 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:56.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:56.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:56.951 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:56.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 64506 64490 60.1382 66.875 0.0258348 0.0166193 2024-09-17T16:22:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:56.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:56 smithi017 ceph-mon[27571]: pgmap v1024: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 29 MiB/s wr, 471 op/s; 2173/185925 objects degraded (1.169%); 24025/185925 objects misplaced (12.922%); 45 MiB/s, 714 objects/s recovering 2024-09-17T16:22:57.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:22:57.078 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:57.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:22:57.085 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:22:57.179 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:57.212 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:57.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:56 smithi071 ceph-mon[30002]: pgmap v1024: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 29 MiB/s wr, 471 op/s; 2173/185925 objects degraded (1.169%); 24025/185925 objects misplaced (12.922%); 45 MiB/s, 714 objects/s recovering 2024-09-17T16:22:57.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:57.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:57.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:57.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:56 smithi173 ceph-mon[30530]: pgmap v1024: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 29 MiB/s wr, 471 op/s; 2173/185925 objects degraded (1.169%); 24025/185925 objects misplaced (12.922%); 45 MiB/s, 714 objects/s recovering 2024-09-17T16:22:57.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:57.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:57.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:57.510 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:22:57.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:57.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:57.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:57.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:57.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:57.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:57.839 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:57.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:57.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:57.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 65357 65341 60.0359 53.1875 0.00971498 0.0166501 2024-09-17T16:22:57.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:58.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:58.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:58.102 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:22:58.134 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:58.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:22:58.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:22:58.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:58.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:22:58.367 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:58.406 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:58.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:58.507 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:22:58.541 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:58.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:22:58.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:58.673 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:58.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:58.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:58.684 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:22:58.684 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:22:58.722 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:22:58.722 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:22:58.757 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:22:58.757 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:22:58.775 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:58.788 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:22:58.789 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:22:58.806 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:58.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:22:58.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:22:58.822 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:22:58.857 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:22:58.857 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:22:58.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:22:58.942 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:58.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:22:58.951 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 66464 66448 60.1684 69.1875 0.0139115 0.0166139 2024-09-17T16:22:59.044 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:22:59.076 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:22:59.076 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:58 smithi017 ceph-mon[27571]: pgmap v1025: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 33 MiB/s wr, 531 op/s; 2173/189105 objects degraded (1.149%); 21678/189105 objects misplaced (11.463%); 41 MiB/s, 659 objects/s recovering 2024-09-17T16:22:59.076 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/150448832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:59.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:22:59.096 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:22:59.096 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:22:59.099 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:59.105 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:22:59.134 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:22:59.134 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:22:59.135 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-17T16:22:59.172 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:22:59.172 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:22:59.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:59.209 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:22:59.209 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:22:59.210 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:59.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:59.249 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:22:59.250 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:22:59.292 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:22:59.292 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:22:59.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:59.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:58 smithi071 ceph-mon[30002]: pgmap v1025: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 33 MiB/s wr, 531 op/s; 2173/189105 objects degraded (1.149%); 21678/189105 objects misplaced (11.463%); 41 MiB/s, 659 objects/s recovering 2024-09-17T16:22:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/150448832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:59.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:59.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:58 smithi173 ceph-mon[30530]: pgmap v1025: 27 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 33 MiB/s wr, 531 op/s; 2173/189105 objects degraded (1.149%); 21678/189105 objects misplaced (11.463%); 41 MiB/s, 659 objects/s recovering 2024-09-17T16:22:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/150448832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:22:59.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:22:59.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:59.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:22:59.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:59.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:59.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:59.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:22:59.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:59.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:59.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:22:59.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:22:59.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:22:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:59.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:22:59.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 67402 67386 60.1463 58.625 0.00577079 0.0166195 2024-09-17T16:22:59.956 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:22:59.985 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:22:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:22:59.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:00.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:00.115 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:00.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:00.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:00.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:00.220 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:23:00.220 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:23:00.220 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:22:59 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2173/189105 objects degraded (1.149%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:23:00.244 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:00.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:00.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:22:59 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2173/189105 objects degraded (1.149%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:23:00.347 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:00.376 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:00.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:00.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:22:59 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2173/189105 objects degraded (1.149%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:23:00.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:00.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:00.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:00.611 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:00.643 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:00.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:00.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:00.772 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:00.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:00.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:00.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:00.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:00.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:00.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 68334 68318 60.1195 58.25 0.0106635 0.0166218 2024-09-17T16:23:00.959 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:23:01.004 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:01.038 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:01.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:01.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:01.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:01.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:01.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:01.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:00 smithi017 ceph-mon[27571]: pgmap v1026: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 16 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 42 MiB/s wr, 671 op/s; 2173/195402 objects degraded (1.112%); 20242/195402 objects misplaced (10.359%); 38 MiB/s, 604 objects/s recovering 2024-09-17T16:23:01.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1689469770' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:01.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:00 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:01.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:01.273 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:23:01.295 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:01.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:00 smithi071 ceph-mon[30002]: pgmap v1026: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 16 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 42 MiB/s wr, 671 op/s; 2173/195402 objects degraded (1.112%); 20242/195402 objects misplaced (10.359%); 38 MiB/s, 604 objects/s recovering 2024-09-17T16:23:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1689469770' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:00 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:00 smithi173 ceph-mon[30530]: pgmap v1026: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 16 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 42 MiB/s wr, 671 op/s; 2173/195402 objects degraded (1.112%); 20242/195402 objects misplaced (10.359%); 38 MiB/s, 604 objects/s recovering 2024-09-17T16:23:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1689469770' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:00 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:01.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:01.423 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:01.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:01.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:01.552 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:01.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:01.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:01.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:01.680 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:01.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:01.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:01.811 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:01.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:01.913 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:01.940 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:01.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:01.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 68655 68639 59.5633 20.0625 0.00957567 0.0166641 2024-09-17T16:23:02.042 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:02.069 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:02.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:02.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:02.171 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:02.201 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:02.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:02.221 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:01 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:23:02.221 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:01 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:02.221 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:01 smithi017 ceph-mon[27571]: osdmap e462: 12 total, 12 up, 10 in 2024-09-17T16:23:02.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:02.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:01 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:23:02.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:01 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:02.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:01 smithi071 ceph-mon[30002]: osdmap e462: 12 total, 12 up, 10 in 2024-09-17T16:23:02.330 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:02.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:01 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:23:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:01 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:01 smithi173 ceph-mon[30530]: osdmap e462: 12 total, 12 up, 10 in 2024-09-17T16:23:02.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:02.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:02.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:02.559 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:02.584 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:02.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:02.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:02.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:02.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:02.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:02.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:02.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:02.937 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:02.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 69039 69023 59.0762 24 0.00955451 0.016782 2024-09-17T16:23:02.958 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:02.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:02.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:03.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:03.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:03.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:03.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:03.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:03.214 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:02 smithi017 ceph-mon[27571]: pgmap v1028: 27 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+recovering+undersized+remapped, 16 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 55 MiB/s wr, 887 op/s; 2173/201933 objects degraded (1.076%); 16268/201933 objects misplaced (8.056%); 59 MiB/s, 938 objects/s recovering 2024-09-17T16:23:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:02 smithi017 ceph-mon[27571]: osdmap e463: 12 total, 12 up, 10 in 2024-09-17T16:23:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:02 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:23:03.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:02 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:23:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:02 smithi071 ceph-mon[30002]: pgmap v1028: 27 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+recovering+undersized+remapped, 16 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 55 MiB/s wr, 887 op/s; 2173/201933 objects degraded (1.076%); 16268/201933 objects misplaced (8.056%); 59 MiB/s, 938 objects/s recovering 2024-09-17T16:23:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:02 smithi071 ceph-mon[30002]: osdmap e463: 12 total, 12 up, 10 in 2024-09-17T16:23:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:02 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:23:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:02 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:23:03.314 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:03.340 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:03.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:02 smithi173 ceph-mon[30530]: pgmap v1028: 27 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+recovering+undersized+remapped, 16 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 55 MiB/s wr, 887 op/s; 2173/201933 objects degraded (1.076%); 16268/201933 objects misplaced (8.056%); 59 MiB/s, 938 objects/s recovering 2024-09-17T16:23:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:02 smithi173 ceph-mon[30530]: osdmap e463: 12 total, 12 up, 10 in 2024-09-17T16:23:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:02 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:23:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:02 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:23:03.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:03.471 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:03.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:03.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:03.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:03.595 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:03.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:03.698 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:03.729 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:03.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:03.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:03.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:03.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:03.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:23:03.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 69971 69955 59.065 58.25 0.00288095 0.0169235 2024-09-17T16:23:03.965 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:04.026 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:04.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:04.128 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:04.153 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:04.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:04.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:03 smithi017 ceph-mon[27571]: osdmap e464: 12 total, 12 up, 10 in 2024-09-17T16:23:04.255 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:04.282 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:04.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:04.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:03 smithi071 ceph-mon[30002]: osdmap e464: 12 total, 12 up, 10 in 2024-09-17T16:23:04.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:04.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:03 smithi173 ceph-mon[30530]: osdmap e464: 12 total, 12 up, 10 in 2024-09-17T16:23:04.409 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:04.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:04.512 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:04.527 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:04.546 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:04.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:04.648 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:04.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:04.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:04.692 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:04.755 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:04.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:04.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:04.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:04.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:04.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:04.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:04.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:04.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 70509 70493 58.7258 33.625 0.00512482 0.0169902 2024-09-17T16:23:05.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:05.049 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:05.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:05.117 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:04 smithi017 ceph-mon[27571]: pgmap v1031: 27 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 7 active+recovering+undersized+remapped, 16 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 54 MiB/s wr, 864 op/s; 1/204675 objects degraded (0.000%); 17834/204675 objects misplaced (8.713%); 50 MiB/s, 794 objects/s recovering 2024-09-17T16:23:05.117 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:04 smithi017 ceph-mon[27571]: osdmap e465: 12 total, 12 up, 10 in 2024-09-17T16:23:05.117 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:04 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/204675 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:05.117 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:04 smithi017 ceph-mon[27571]: osdmap e466: 12 total, 12 up, 10 in 2024-09-17T16:23:05.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:05.177 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:05.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:05.278 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:05.305 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:05.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:05.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:04 smithi071 ceph-mon[30002]: pgmap v1031: 27 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 7 active+recovering+undersized+remapped, 16 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 54 MiB/s wr, 864 op/s; 1/204675 objects degraded (0.000%); 17834/204675 objects misplaced (8.713%); 50 MiB/s, 794 objects/s recovering 2024-09-17T16:23:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:04 smithi071 ceph-mon[30002]: osdmap e465: 12 total, 12 up, 10 in 2024-09-17T16:23:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:04 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/204675 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:04 smithi071 ceph-mon[30002]: osdmap e466: 12 total, 12 up, 10 in 2024-09-17T16:23:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:04 smithi173 ceph-mon[30530]: pgmap v1031: 27 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 7 active+recovering+undersized+remapped, 16 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 54 MiB/s wr, 864 op/s; 1/204675 objects degraded (0.000%); 17834/204675 objects misplaced (8.713%); 50 MiB/s, 794 objects/s recovering 2024-09-17T16:23:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:04 smithi173 ceph-mon[30530]: osdmap e465: 12 total, 12 up, 10 in 2024-09-17T16:23:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:04 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/204675 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:04 smithi173 ceph-mon[30530]: osdmap e466: 12 total, 12 up, 10 in 2024-09-17T16:23:05.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:05.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:05.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:05.535 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:05.566 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:05.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:05.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:05.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:05.702 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:05.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:05.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:05.828 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:05.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:05.930 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:05.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 15 71154 71139 58.4844 40.375 0.00269085 0.0170916 2024-09-17T16:23:05.959 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:05.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:06.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:06.094 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:06.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:06.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:06.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:06.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:06.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:06.350 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:06.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:06.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:06.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:06.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:06.579 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:06.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:23:06.608 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:06.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:06.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:06.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:06.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:06 smithi017 ceph-mon[27571]: 5.15 scrub starts 2024-09-17T16:23:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:06 smithi017 ceph-mon[27571]: osdmap e467: 12 total, 12 up, 10 in 2024-09-17T16:23:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:06 smithi017 ceph-mon[27571]: 5.15 scrub ok 2024-09-17T16:23:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:06 smithi017 ceph-mon[27571]: pgmap v1035: 27 pgs: 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 19 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 40 MiB/s wr, 645 op/s; 15856/209532 objects misplaced (7.567%); 34 MiB/s, 539 objects/s recovering 2024-09-17T16:23:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:06 smithi071 ceph-mon[30002]: 5.15 scrub starts 2024-09-17T16:23:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:06 smithi071 ceph-mon[30002]: osdmap e467: 12 total, 12 up, 10 in 2024-09-17T16:23:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:06 smithi071 ceph-mon[30002]: 5.15 scrub ok 2024-09-17T16:23:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:06 smithi071 ceph-mon[30002]: pgmap v1035: 27 pgs: 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 19 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 40 MiB/s wr, 645 op/s; 15856/209532 objects misplaced (7.567%); 34 MiB/s, 539 objects/s recovering 2024-09-17T16:23:06.836 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:06.864 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:06.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:06 smithi173 ceph-mon[30530]: 5.15 scrub starts 2024-09-17T16:23:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:06 smithi173 ceph-mon[30530]: osdmap e467: 12 total, 12 up, 10 in 2024-09-17T16:23:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:06 smithi173 ceph-mon[30530]: 5.15 scrub ok 2024-09-17T16:23:06.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:06 smithi173 ceph-mon[30530]: pgmap v1035: 27 pgs: 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 19 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 40 MiB/s wr, 645 op/s; 15856/209532 objects misplaced (7.567%); 34 MiB/s, 539 objects/s recovering 2024-09-17T16:23:06.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 71845 71829 58.2849 43.125 0.00944378 0.0171517 2024-09-17T16:23:06.966 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:06.998 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:06.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:07.100 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:07.129 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:07.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:07.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:07.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:07.259 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:07.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:07.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:07.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:07.493 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:07.517 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:07.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:07.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:07.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:07.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:07.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:07 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/204675 objects degraded (0.000%), 1 pg degraded) 2024-09-17T16:23:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:07 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:23:07.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:07.781 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:07.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:07 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/204675 objects degraded (0.000%), 1 pg degraded) 2024-09-17T16:23:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:07 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:23:07.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:07.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:07 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/204675 objects degraded (0.000%), 1 pg degraded) 2024-09-17T16:23:07.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:07 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:23:07.914 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:07.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:07.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:07.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 72714 72698 58.234 54.3125 0.00318437 0.0171645 2024-09-17T16:23:08.016 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:08.040 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:08.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:08.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:08.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:08.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:08.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:08.290 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:08.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:08.393 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:08.421 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:08.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:08.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:08.548 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:08.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:08.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:08.680 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:08.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:08 smithi017 ceph-mon[27571]: pgmap v1036: 27 pgs: 7 active+recovering+undersized+remapped, 20 active+clean; 4.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 40 MiB/s wr, 642 op/s; 13565/212181 objects misplaced (6.393%); 41 MiB/s, 654 objects/s recovering 2024-09-17T16:23:08.781 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:08.807 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:08.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:08 smithi071 ceph-mon[30002]: pgmap v1036: 27 pgs: 7 active+recovering+undersized+remapped, 20 active+clean; 4.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 40 MiB/s wr, 642 op/s; 13565/212181 objects misplaced (6.393%); 41 MiB/s, 654 objects/s recovering 2024-09-17T16:23:08.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:08 smithi173 ceph-mon[30530]: pgmap v1036: 27 pgs: 7 active+recovering+undersized+remapped, 20 active+clean; 4.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 40 MiB/s wr, 642 op/s; 13565/212181 objects misplaced (6.393%); 41 MiB/s, 654 objects/s recovering 2024-09-17T16:23:08.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:08.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:08.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:08.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:08.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 73598 73582 58.1961 55.25 0.00222304 0.0171746 2024-09-17T16:23:09.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:09.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:09.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:09.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:09.198 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:09.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:09.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:09.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:09.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:09.423 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:09.454 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:09.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:09.556 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:09.580 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:09.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:09.682 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:09.714 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:09.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:09.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:09 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:23:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:09 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:23:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:09 smithi017 ceph-mon[27571]: osdmap e468: 12 total, 12 up, 10 in 2024-09-17T16:23:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:09 smithi017 ceph-mon[27571]: osdmap e469: 12 total, 12 up, 10 in 2024-09-17T16:23:09.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:09 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:23:09.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:09 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:23:09.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:09 smithi173 ceph-mon[30530]: osdmap e468: 12 total, 12 up, 10 in 2024-09-17T16:23:09.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:09 smithi173 ceph-mon[30530]: osdmap e469: 12 total, 12 up, 10 in 2024-09-17T16:23:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:09 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:23:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:09 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:23:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:09 smithi071 ceph-mon[30002]: osdmap e468: 12 total, 12 up, 10 in 2024-09-17T16:23:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:09 smithi071 ceph-mon[30002]: osdmap e469: 12 total, 12 up, 10 in 2024-09-17T16:23:09.816 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:09.839 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:09.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:09.941 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:09.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:23:09.951547+0000 min lat: 0.00103987 max lat: 4.96371 avg lat: 0.0171791 2024-09-17T16:23:09.952 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:23:09.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 74505 74489 58.1772 56.6875 0.00449321 0.0171791 2024-09-17T16:23:09.966 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:09.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:10.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:10.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:10.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:10.197 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:10.226 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:10.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:10.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:10.355 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:10.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:10.457 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:10.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:10.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:10.612 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:10.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:10 smithi173 ceph-mon[30530]: pgmap v1039: 27 pgs: 7 active+recovering+undersized+remapped, 20 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 24 MiB/s wr, 386 op/s; 12112/217500 objects misplaced (5.569%); 542 KiB/s, 8 objects/s recovering 2024-09-17T16:23:10.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:10.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:10.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:10.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:10 smithi017 ceph-mon[27571]: pgmap v1039: 27 pgs: 7 active+recovering+undersized+remapped, 20 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 24 MiB/s wr, 386 op/s; 12112/217500 objects misplaced (5.569%); 542 KiB/s, 8 objects/s recovering 2024-09-17T16:23:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:10 smithi071 ceph-mon[30002]: pgmap v1039: 27 pgs: 7 active+recovering+undersized+remapped, 20 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 24 MiB/s wr, 386 op/s; 12112/217500 objects misplaced (5.569%); 542 KiB/s, 8 objects/s recovering 2024-09-17T16:23:10.838 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:10.863 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:10.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:10.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 75073 75057 57.8973 35.5 0.0845662 0.0172226 2024-09-17T16:23:10.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:10.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:10.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:11.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:11.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:11.110 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:11.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:11.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:11.236 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:11.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:11.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:11.364 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:11.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:11.466 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:11.492 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:11.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:11.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:11.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:11.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:11.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:11 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:23:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:11 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:23:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:11 smithi173 ceph-mon[30530]: 5.14 scrub starts 2024-09-17T16:23:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:11 smithi173 ceph-mon[30530]: osdmap e470: 12 total, 12 up, 10 in 2024-09-17T16:23:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1393272584' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:11 smithi173 ceph-mon[30530]: 5.14 scrub ok 2024-09-17T16:23:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:11 smithi173 ceph-mon[30530]: 5.11 scrub starts 2024-09-17T16:23:11.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:23:11.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:23:11.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:23:11.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:23:11.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:23:11.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:23:11.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:23:11.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:23:11.695 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:23:11.696 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:23:11.696 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:23:11.696 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:23:11.696 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:23:11.696 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:23:11.724 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:11.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:11 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:23:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:11 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:23:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:11 smithi017 ceph-mon[27571]: 5.14 scrub starts 2024-09-17T16:23:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:11 smithi017 ceph-mon[27571]: osdmap e470: 12 total, 12 up, 10 in 2024-09-17T16:23:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1393272584' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:11 smithi017 ceph-mon[27571]: 5.14 scrub ok 2024-09-17T16:23:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:11 smithi017 ceph-mon[27571]: 5.11 scrub starts 2024-09-17T16:23:11.750 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:11.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:11 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:23:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:11 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:23:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:11 smithi071 ceph-mon[30002]: 5.14 scrub starts 2024-09-17T16:23:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:11 smithi071 ceph-mon[30002]: osdmap e470: 12 total, 12 up, 10 in 2024-09-17T16:23:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1393272584' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:11 smithi071 ceph-mon[30002]: 5.14 scrub ok 2024-09-17T16:23:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:11 smithi071 ceph-mon[30002]: 5.11 scrub starts 2024-09-17T16:23:11.852 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:11.875 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:11.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:11.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 75505 75489 57.5205 27 0.0112294 0.0173774 2024-09-17T16:23:11.977 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:12.000 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:12.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:12.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:12.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:12.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:12.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:12.234 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:12.262 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:12.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:12.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:12.393 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:12.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:12.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:12.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:12.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:12.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:12.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:12.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:12.754 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:12.779 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:12.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:12 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:23:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:12 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:23:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:12 smithi071 ceph-mon[30002]: osdmap e471: 12 total, 12 up, 10 in 2024-09-17T16:23:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:12 smithi071 ceph-mon[30002]: pgmap v1042: 27 pgs: 1 peering, 6 active+recovering+undersized+remapped, 20 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 10642/222507 objects misplaced (4.783%) 2024-09-17T16:23:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:12 smithi071 ceph-mon[30002]: 5.11 scrub ok 2024-09-17T16:23:12.881 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:12.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:12 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:23:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:12 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:23:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:12 smithi173 ceph-mon[30530]: osdmap e471: 12 total, 12 up, 10 in 2024-09-17T16:23:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:12 smithi173 ceph-mon[30530]: pgmap v1042: 27 pgs: 1 peering, 6 active+recovering+undersized+remapped, 20 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 10642/222507 objects misplaced (4.783%) 2024-09-17T16:23:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:12 smithi173 ceph-mon[30530]: 5.11 scrub ok 2024-09-17T16:23:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:12.905 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:12.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:12.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 76381 76365 57.4871 54.75 0.04047 0.0173764 2024-09-17T16:23:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:12 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:23:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:12 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:23:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:12 smithi017 ceph-mon[27571]: osdmap e471: 12 total, 12 up, 10 in 2024-09-17T16:23:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:12 smithi017 ceph-mon[27571]: pgmap v1042: 27 pgs: 1 peering, 6 active+recovering+undersized+remapped, 20 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 10642/222507 objects misplaced (4.783%) 2024-09-17T16:23:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:12 smithi017 ceph-mon[27571]: 5.11 scrub ok 2024-09-17T16:23:13.006 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:13.035 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:13.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:13.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:13.162 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:13.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:13.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:13.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:13.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:13.297 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:13.399 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:13.425 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:13.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:13.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:13.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:13.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:13.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:13.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:13.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:13.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:13.783 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:13.805 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:13.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:13.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:13 smithi071 ceph-mon[30002]: 5.10 scrub starts 2024-09-17T16:23:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:13 smithi071 ceph-mon[30002]: 5.17 scrub starts 2024-09-17T16:23:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:13 smithi071 ceph-mon[30002]: 5.10 scrub ok 2024-09-17T16:23:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:13 smithi071 ceph-mon[30002]: 5.17 scrub ok 2024-09-17T16:23:13.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:13 smithi173 ceph-mon[30530]: 5.10 scrub starts 2024-09-17T16:23:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:13 smithi173 ceph-mon[30530]: 5.17 scrub starts 2024-09-17T16:23:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:13 smithi173 ceph-mon[30530]: 5.10 scrub ok 2024-09-17T16:23:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:13 smithi173 ceph-mon[30530]: 5.17 scrub ok 2024-09-17T16:23:13.907 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:13.930 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:13.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:13.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:13.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 77300 77284 57.4865 57.4375 0.00420422 0.01736 2024-09-17T16:23:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:13 smithi017 ceph-mon[27571]: 5.10 scrub starts 2024-09-17T16:23:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:13 smithi017 ceph-mon[27571]: 5.17 scrub starts 2024-09-17T16:23:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:13 smithi017 ceph-mon[27571]: 5.10 scrub ok 2024-09-17T16:23:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:13 smithi017 ceph-mon[27571]: 5.17 scrub ok 2024-09-17T16:23:14.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:14.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:14.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:14.154 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:14.181 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:14.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:14.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:14.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:14.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:14.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:14.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:14.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:14.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:14.562 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:14.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:14.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:14.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:14.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:14.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:14 smithi071 ceph-mon[30002]: osdmap e472: 12 total, 12 up, 10 in 2024-09-17T16:23:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:14 smithi071 ceph-mon[30002]: pgmap v1044: 27 pgs: 1 peering, 5 active+recovering+undersized+remapped, 21 active+clean; 4.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 9018/225801 objects misplaced (3.994%) 2024-09-17T16:23:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:14 smithi071 ceph-mon[30002]: osdmap e473: 12 total, 12 up, 10 in 2024-09-17T16:23:14.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:14.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:14.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:14 smithi173 ceph-mon[30530]: osdmap e472: 12 total, 12 up, 10 in 2024-09-17T16:23:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:14 smithi173 ceph-mon[30530]: pgmap v1044: 27 pgs: 1 peering, 5 active+recovering+undersized+remapped, 21 active+clean; 4.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 9018/225801 objects misplaced (3.994%) 2024-09-17T16:23:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:14 smithi173 ceph-mon[30530]: osdmap e473: 12 total, 12 up, 10 in 2024-09-17T16:23:14.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:14.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 77928 77912 57.2719 39.25 0.00366302 0.017454 2024-09-17T16:23:14.961 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:14.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:14 smithi017 ceph-mon[27571]: osdmap e472: 12 total, 12 up, 10 in 2024-09-17T16:23:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:14 smithi017 ceph-mon[27571]: pgmap v1044: 27 pgs: 1 peering, 5 active+recovering+undersized+remapped, 21 active+clean; 4.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 9018/225801 objects misplaced (3.994%) 2024-09-17T16:23:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:14 smithi017 ceph-mon[27571]: osdmap e473: 12 total, 12 up, 10 in 2024-09-17T16:23:15.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:15.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:15.095 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:15.196 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:15.226 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:15.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:15.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:15.350 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:15.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:15.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:15.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:15.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:15.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:15.606 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:15.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:15.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:15.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:15.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:15.835 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:15.859 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:15.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:15.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 78767 78751 57.2157 52.4375 0.0279912 0.0174687 2024-09-17T16:23:15.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:15.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:15.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:16.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:16.113 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:16.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:16.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:16.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:15 smithi017 ceph-mon[27571]: 5.16 scrub starts 2024-09-17T16:23:16.243 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:16.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:16.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:15 smithi071 ceph-mon[30002]: 5.16 scrub starts 2024-09-17T16:23:16.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:16.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:16.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:15 smithi173 ceph-mon[30530]: 5.16 scrub starts 2024-09-17T16:23:16.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:16.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:16.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:16.607 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:16.630 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:16.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:16.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:16.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:16.760 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:16.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:16.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:16.888 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:16.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:16.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 79686 79670 57.2182 57.4375 0.0109385 0.0174709 2024-09-17T16:23:16.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:17.017 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:17.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:17.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:17.145 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:17.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:17.219 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:16 smithi017 ceph-mon[27571]: pgmap v1046: 27 pgs: 1 peering, 5 active+recovering+undersized+remapped, 21 active+clean; 4.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 8349/231363 objects misplaced (3.609%) 2024-09-17T16:23:17.219 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:16 smithi017 ceph-mon[27571]: 5.13 scrub starts 2024-09-17T16:23:17.219 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:16 smithi017 ceph-mon[27571]: 5.13 scrub ok 2024-09-17T16:23:17.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:17.248 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:23:17.248 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:23:17.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:23:17.249 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:23:17.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.249 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:135561 2.6 2:612899dc:::benchmark_data_smithi173_55121_object59072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-17T16:23:17.249 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:23:17.249 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 581.32253286000002, 2024-09-17T16:23:17.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36676542899999998, 2024-09-17T16:23:17.249 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.249 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 135561 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921376+0000", 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921378+0000", 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.897e-06 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921386+0000", 2024-09-17T16:23:17.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.17e-06 2024-09-17T16:23:17.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921398+0000", 2024-09-17T16:23:17.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2408999999999999e-05 2024-09-17T16:23:17.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921406+0000", 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6179999999999997e-06 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921457+0000", 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1743999999999998e-05 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921550+0000", 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2318000000000006e-05 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,8", 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.921623+0000", 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3337999999999995e-05 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.923435+0000", 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001811654 2024-09-17T16:23:17.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:35.923947+0000", 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051277600000000005 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288065+0000", 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36411789 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:17.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288107+0000", 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1479e-05 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:36.288142+0000", 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5135999999999997e-05 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.24550.0:152966 2.c e199/95 2:3484f1c7:::benchmark_data_smithi173_55121_object76477:head v 199'9438, mlcod=199'9436)", 2024-09-17T16:23:17.256 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 563.38778303399999, 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58877864599999996, 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856126+0000", 2024-09-17T16:23:17.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856128+0000", 2024-09-17T16:23:17.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3740000000000001e-06 2024-09-17T16:23:17.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.856136+0000", 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6950000000000006e-06 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441396+0000", 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58525992800000004 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441405+0000", 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9770000000000006e-06 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441463+0000", 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8025000000000001e-05 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.441483+0000", 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9678999999999999e-05 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.444869+0000", 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033864070000000001 2024-09-17T16:23:17.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.444905+0000", 2024-09-17T16:23:17.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5561000000000001e-05 2024-09-17T16:23:17.261 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.261 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.261 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.261 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(3.0 256/239 [PushOp(3:0ca39f28:::benchmark_data_smithi173_66272_object41987:head, version: 231'2635, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0ca39f28:::benchmark_data_smithi173_66272_object41987:head@231'2635, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 440.20324567099999, 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59916758699999995, 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040665+0000", 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0100000000000003e-07 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040719+0000", 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4487000000000002e-05 2024-09-17T16:23:17.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638102+0000", 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.597383255 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638121+0000", 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8595999999999999e-05 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638205+0000", 2024-09-17T16:23:17.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4042999999999995e-05 2024-09-17T16:23:17.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638216+0000", 2024-09-17T16:23:17.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1299e-05 2024-09-17T16:23:17.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.639831+0000", 2024-09-17T16:23:17.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001615006 2024-09-17T16:23:17.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.266 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.267 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124280 3.0 3:099812ca:::benchmark_data_smithi173_66272_object52324:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:23:17.267 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377919+0000", 2024-09-17T16:23:17.267 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.865990361, 2024-09-17T16:23:17.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63194678999999998, 2024-09-17T16:23:17.267 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.267 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.267 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:17.268 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:17.268 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:17.268 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:17.268 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124280 2024-09-17T16:23:17.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.268 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377919+0000", 2024-09-17T16:23:17.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377919+0000", 2024-09-17T16:23:17.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377921+0000", 2024-09-17T16:23:17.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.793e-06 2024-09-17T16:23:17.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377925+0000", 2024-09-17T16:23:17.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5859999999999998e-06 2024-09-17T16:23:17.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377937+0000", 2024-09-17T16:23:17.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1168999999999999e-05 2024-09-17T16:23:17.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377944+0000", 2024-09-17T16:23:17.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5209999999999997e-06 2024-09-17T16:23:17.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391301+0000", 2024-09-17T16:23:17.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013356989999999999 2024-09-17T16:23:17.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:17.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391307+0000", 2024-09-17T16:23:17.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0939999999999996e-06 2024-09-17T16:23:17.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.983230+0000", 2024-09-17T16:23:17.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59192265799999999 2024-09-17T16:23:17.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.983309+0000", 2024-09-17T16:23:17.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9083000000000002e-05 2024-09-17T16:23:17.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:23:17.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.983395+0000", 2024-09-17T16:23:17.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6595999999999996e-05 2024-09-17T16:23:17.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:17.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991593+0000", 2024-09-17T16:23:17.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081975050000000008 2024-09-17T16:23:17.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991633+0000", 2024-09-17T16:23:17.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9579000000000002e-05 2024-09-17T16:23:17.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991675+0000", 2024-09-17T16:23:17.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2407999999999998e-05 2024-09-17T16:23:17.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009712+0000", 2024-09-17T16:23:17.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.018037553000000001 2024-09-17T16:23:17.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:17.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009777+0000", 2024-09-17T16:23:17.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4436000000000004e-05 2024-09-17T16:23:17.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009866+0000", 2024-09-17T16:23:17.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8819000000000005e-05 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124284 3.0 3:04f57090:::benchmark_data_smithi173_66272_object52328:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377957+0000", 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.86595193300002, 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63216316299999997, 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.282 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.283 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:17.283 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:17.283 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:17.283 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:17.283 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124284 2024-09-17T16:23:17.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.283 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377957+0000", 2024-09-17T16:23:17.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377957+0000", 2024-09-17T16:23:17.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377958+0000", 2024-09-17T16:23:17.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:23:17.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377960+0000", 2024-09-17T16:23:17.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9209999999999999e-06 2024-09-17T16:23:17.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377964+0000", 2024-09-17T16:23:17.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6569999999999996e-06 2024-09-17T16:23:17.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377970+0000", 2024-09-17T16:23:17.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3290000000000001e-06 2024-09-17T16:23:17.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391358+0000", 2024-09-17T16:23:17.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013388304 2024-09-17T16:23:17.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:17.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391368+0000", 2024-09-17T16:23:17.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0570999999999999e-05 2024-09-17T16:23:17.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.986075+0000", 2024-09-17T16:23:17.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.594706819 2024-09-17T16:23:17.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.986194+0000", 2024-09-17T16:23:17.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118577 2024-09-17T16:23:17.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:23:17.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.986314+0000", 2024-09-17T16:23:17.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000119782 2024-09-17T16:23:17.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:17.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991687+0000", 2024-09-17T16:23:17.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053729240000000003 2024-09-17T16:23:17.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996674+0000", 2024-09-17T16:23:17.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00498761 2024-09-17T16:23:17.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.000224+0000", 2024-09-17T16:23:17.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035494630000000001 2024-09-17T16:23:17.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009919+0000", 2024-09-17T16:23:17.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096958189999999996 2024-09-17T16:23:17.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:17.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009978+0000", 2024-09-17T16:23:17.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8242000000000002e-05 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.010121+0000", 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014286999999999999 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.298 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124286 3.0 3:01b0fd2d:::benchmark_data_smithi173_66272_object52330:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:23:17.299 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377976+0000", 2024-09-17T16:23:17.299 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.86593303799998, 2024-09-17T16:23:17.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63695855000000001, 2024-09-17T16:23:17.299 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.299 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.299 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124286 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377976+0000", 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377976+0000", 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377977+0000", 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4699999999999998e-07 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377978+0000", 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.342e-06 2024-09-17T16:23:17.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377982+0000", 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6389999999999998e-06 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377985+0000", 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2789999999999999e-06 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391399+0000", 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013414272 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391403+0000", 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.799e-06 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.989045+0000", 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59764173499999995 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.989160+0000", 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000115654 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.989270+0000", 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000109574 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996735+0000", 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074655440000000002 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996747+0000", 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2125e-05 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.010180+0000", 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013432908 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.014804+0000", 2024-09-17T16:23:17.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046233350000000001 2024-09-17T16:23:17.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:17.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.014845+0000", 2024-09-17T16:23:17.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1390999999999997e-05 2024-09-17T16:23:17.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.014935+0000", 2024-09-17T16:23:17.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9705999999999998e-05 2024-09-17T16:23:17.309 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.309 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.309 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.310 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124287 3.0 3:022daafd:::benchmark_data_smithi173_66272_object52331:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:23:17.310 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:23:17.310 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.86592068300001, 2024-09-17T16:23:17.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63722835200000005, 2024-09-17T16:23:17.310 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.310 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124287 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:23:17.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:23:17.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:23:17.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2399999999999999e-07 2024-09-17T16:23:17.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377990+0000", 2024-09-17T16:23:17.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1489999999999999e-06 2024-09-17T16:23:17.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377993+0000", 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.19e-06 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377996+0000", 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8289999999999999e-06 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391438+0000", 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013442154 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391448+0000", 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9159999999999996e-06 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991732+0000", 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.600283593 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991823+0000", 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1145999999999998e-05 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991905+0000", 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1664999999999998e-05 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996755+0000", 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048505700000000002 2024-09-17T16:23:17.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.000183+0000", 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034277359999999998 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.010393+0000", 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010210558 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015049+0000", 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046556740000000003 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015106+0000", 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7339000000000001e-05 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015217+0000", 2024-09-17T16:23:17.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110609 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124291 3.0 3:0f520599:::benchmark_data_smithi173_66272_object52335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.86590848899999, 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.64518043599999997, 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:17.319 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:17.320 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:17.320 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124291 2024-09-17T16:23:17.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.320 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.320 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.320 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:23:17.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0800000000000001e-07 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378002+0000", 2024-09-17T16:23:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1620000000000001e-06 2024-09-17T16:23:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378006+0000", 2024-09-17T16:23:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9910000000000004e-06 2024-09-17T16:23:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378009+0000", 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.785e-06 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391464+0000", 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013454729 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391467+0000", 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4750000000000002e-06 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.994266+0000", 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.602799058 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.994361+0000", 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4487999999999994e-05 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.994432+0000", 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1518000000000002e-05 2024-09-17T16:23:17.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.002826+0000", 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083936540000000004 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.002845+0000", 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9128999999999999e-05 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015269+0000", 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012424269 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.022862+0000", 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075930640000000001 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.022977+0000", 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114539 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.023181+0000", 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000204367 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124263 3.4 3:2eabebde:::benchmark_data_smithi173_66272_object52307:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382419+0000", 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.86148988600002, 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.499722689, 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.327 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124263 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382419+0000", 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382419+0000", 2024-09-17T16:23:17.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382421+0000", 2024-09-17T16:23:17.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.536e-06 2024-09-17T16:23:17.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382427+0000", 2024-09-17T16:23:17.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6500000000000001e-06 2024-09-17T16:23:17.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382441+0000", 2024-09-17T16:23:17.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4368e-05 2024-09-17T16:23:17.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382449+0000", 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3680000000000002e-06 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387414+0000", 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049646159999999998 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:17.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387421+0000", 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5889999999999999e-06 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882026+0000", 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49460579599999999 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882142+0000", 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000115766 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.332 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124265 3.4 3:2dc750d7:::benchmark_data_smithi173_66272_object52309:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.861420786, 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49990343999999998, 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:17.333 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124265 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:23:17.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:23:17.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5300000000000004e-07 2024-09-17T16:23:17.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382492+0000", 2024-09-17T16:23:17.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2370000000000001e-06 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382502+0000", 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0037000000000001e-05 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382509+0000", 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9149999999999999e-06 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.337 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387622+0000", 2024-09-17T16:23:17.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051124600000000001 2024-09-17T16:23:17.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:17.337 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387630+0000", 2024-09-17T16:23:17.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9510000000000006e-06 2024-09-17T16:23:17.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882341+0000", 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49471140899999999 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882392+0000", 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0878000000000003e-05 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.338 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124266 3.4 3:2de8446c:::benchmark_data_smithi173_66272_object52310:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.861390465, 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.51201529199999996, 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124266 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2899999999999999e-07 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382521+0000", 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8959999999999999e-06 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382527+0000", 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9950000000000002e-06 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382532+0000", 2024-09-17T16:23:17.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1749999999999996e-06 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387669+0000", 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051369980000000003 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387680+0000", 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1e-05 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882422+0000", 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49474148299999998 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882507+0000", 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4802999999999994e-05 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,5,11", 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882611+0000", 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104527 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.890862+0000", 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082508949999999994 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893785+0000", 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029228700000000002 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893832+0000", 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7620999999999997e-05 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.894360+0000", 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000527605 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.894418+0000", 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8057000000000003e-05 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.894534+0000", 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116038 2024-09-17T16:23:17.345 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124283 3.4 3:2189d34a:::benchmark_data_smithi173_66272_object52327:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382540+0000", 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 386.86136915200001, 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.52263484000000004, 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:17.346 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:17.347 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124283 2024-09-17T16:23:17.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.347 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382540+0000", 2024-09-17T16:23:17.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382540+0000", 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382541+0000", 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6899999999999998e-07 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382543+0000", 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0530000000000001e-06 2024-09-17T16:23:17.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382549+0000", 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0580000000000003e-06 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382554+0000", 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0300000000000001e-06 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387707+0000", 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051537110000000001 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387715+0000", 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8220000000000007e-06 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.886356+0000", 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49864073799999997 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.886497+0000", 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014137500000000001 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,5,11", 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.886624+0000", 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126551 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893848+0000", 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007223648 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893892+0000", 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4084e-05 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.896419+0000", 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002527502 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.904985+0000", 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0085655720000000005 2024-09-17T16:23:17.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.905056+0000", 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1730999999999995e-05 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.905175+0000", 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118596 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(3.c e290,267 3:3e4663e2:::benchmark_data_smithi173_66272_object807:head,251'4519)", 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.493506+0000", 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 356.750403457, 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62512154200000003, 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493506+0000", 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493506+0000", 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493507+0000", 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.164e-06 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493514+0000", 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3549999999999999e-06 2024-09-17T16:23:17.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118475+0000", 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62496048599999998 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118483+0000", 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6769999999999995e-06 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118554+0000", 2024-09-17T16:23:17.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1210000000000004e-05 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118561+0000", 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.029e-06 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118627+0000", 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6620999999999998e-05 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.357 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:17.358 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:17.358 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:23:17.358 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:16 smithi071 ceph-mon[30002]: pgmap v1046: 27 pgs: 1 peering, 5 active+recovering+undersized+remapped, 21 active+clean; 4.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 8349/231363 objects misplaced (3.609%) 2024-09-17T16:23:17.358 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:16 smithi071 ceph-mon[30002]: 5.13 scrub starts 2024-09-17T16:23:17.359 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:16 smithi071 ceph-mon[30002]: 5.13 scrub ok 2024-09-17T16:23:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:17.360 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:17.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:16 smithi173 ceph-mon[30530]: pgmap v1046: 27 pgs: 1 peering, 5 active+recovering+undersized+remapped, 21 active+clean; 4.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 8349/231363 objects misplaced (3.609%) 2024-09-17T16:23:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:16 smithi173 ceph-mon[30530]: 5.13 scrub starts 2024-09-17T16:23:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:16 smithi173 ceph-mon[30530]: 5.13 scrub ok 2024-09-17T16:23:17.460 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:17.485 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:17.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:17.587 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:17.616 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:17.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:17.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:17.748 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:17.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:17.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:17.878 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:17.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:17.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 80711 80695 57.2959 64.0625 0.0432874 0.0174442 2024-09-17T16:23:17.966 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-17T16:23:17.978 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:18.006 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:18.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:18.108 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:18.138 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:18.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:17 smithi017 ceph-mon[27571]: 5.16 scrub ok 2024-09-17T16:23:18.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:18.263 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:18.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:18.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:17 smithi071 ceph-mon[30002]: 5.16 scrub ok 2024-09-17T16:23:18.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:18.390 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:18.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:18.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:17 smithi173 ceph-mon[30530]: 5.16 scrub ok 2024-09-17T16:23:18.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:18.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:18.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:18.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:18.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:18.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:18.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:18.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:18.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:18.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:18.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:18.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:18.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 81842 81826 57.4462 70.6875 0.0166117 0.017402 2024-09-17T16:23:19.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:19.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:19.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:19.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:19.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:19.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:19.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:18 smithi017 ceph-mon[27571]: pgmap v1047: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 22 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 8102/234576 objects misplaced (3.454%) 2024-09-17T16:23:19.284 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:19.310 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:19.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:19.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:18 smithi071 ceph-mon[30002]: pgmap v1047: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 22 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 8102/234576 objects misplaced (3.454%) 2024-09-17T16:23:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:18 smithi173 ceph-mon[30530]: pgmap v1047: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 22 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 8102/234576 objects misplaced (3.454%) 2024-09-17T16:23:19.411 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:19.430 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:19.438 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:19.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:19.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:23:19.540 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:19.567 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:19.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:19.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:19.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:19.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:19.798 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:19.823 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:19.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:19.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:19.949 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:19.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:19.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 15 82768 82753 57.4516 57.9375 0.0151128 0.017393 2024-09-17T16:23:20.027 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:20.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:20.074 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:20.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:20.176 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:20.198 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:20.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:20.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:19 smithi017 ceph-mon[27571]: osdmap e474: 12 total, 12 up, 10 in 2024-09-17T16:23:20.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:19 smithi017 ceph-mon[27571]: osdmap e475: 12 total, 12 up, 10 in 2024-09-17T16:23:20.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:20.609 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:19 smithi071 ceph-mon[30002]: osdmap e474: 12 total, 12 up, 10 in 2024-09-17T16:23:20.609 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:19 smithi071 ceph-mon[30002]: osdmap e475: 12 total, 12 up, 10 in 2024-09-17T16:23:20.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:19 smithi173 ceph-mon[30530]: osdmap e474: 12 total, 12 up, 10 in 2024-09-17T16:23:20.610 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:19 smithi173 ceph-mon[30530]: osdmap e475: 12 total, 12 up, 10 in 2024-09-17T16:23:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:20.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:20.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:20.650 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:23:20.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:20.756 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:20.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:20.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:20.880 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:20.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:20.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.8296 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 82768 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 56.9528 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 29.9221 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 141.125 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 911 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 478.77 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2258 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 0 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0174613 2024-09-17T16:23:20.954 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.084239 2024-09-17T16:23:20.955 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 4.96371 2024-09-17T16:23:20.955 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00103987 2024-09-17T16:23:20.961 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T16:23:20.968 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:23:20.982 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:21.010 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:21.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:21.112 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:21.139 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:21.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:21.228 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:20 smithi017 ceph-mon[27571]: pgmap v1050: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 22 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 6212/241482 objects misplaced (2.572%) 2024-09-17T16:23:21.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:20 smithi017 ceph-mon[27571]: 5.19 scrub starts 2024-09-17T16:23:21.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1167071425' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:23:21.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:20 smithi017 ceph-mon[27571]: 5.19 scrub ok 2024-09-17T16:23:21.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:21.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:21.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:21.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:20 smithi071 ceph-mon[30002]: pgmap v1050: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 22 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 6212/241482 objects misplaced (2.572%) 2024-09-17T16:23:21.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:20 smithi071 ceph-mon[30002]: 5.19 scrub starts 2024-09-17T16:23:21.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1167071425' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:23:21.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:20 smithi071 ceph-mon[30002]: 5.19 scrub ok 2024-09-17T16:23:21.348 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:20 smithi173 ceph-mon[30530]: pgmap v1050: 27 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+remapped, 22 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 6212/241482 objects misplaced (2.572%) 2024-09-17T16:23:21.348 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:20 smithi173 ceph-mon[30530]: 5.19 scrub starts 2024-09-17T16:23:21.349 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1167071425' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:23:21.349 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:20 smithi173 ceph-mon[30530]: 5.19 scrub ok 2024-09-17T16:23:21.370 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:21.397 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:21.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:21.498 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:21.521 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:21.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:21.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:21.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:21.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:21.751 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:21.775 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:21.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:21.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:21.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:21.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:22.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:22.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:22.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:22.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:22.154 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:22.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:22.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:21 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1167071425' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:23:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:21 smithi017 ceph-mon[27571]: osdmap e476: 12 total, 12 up, 10 in 2024-09-17T16:23:22.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:22.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:22.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:22.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:21 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:22.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1167071425' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:23:22.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:21 smithi071 ceph-mon[30002]: osdmap e476: 12 total, 12 up, 10 in 2024-09-17T16:23:22.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:21 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1167071425' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:23:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:21 smithi173 ceph-mon[30530]: osdmap e476: 12 total, 12 up, 10 in 2024-09-17T16:23:22.414 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:22.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:22.516 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:22.539 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:22.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:22.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:22.669 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:22.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:22.771 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:22.795 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:22.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:22.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:22.923 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:22.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:22.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:23.025 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:23.050 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:23.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:23.152 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:23.174 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:23.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:23.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:22 smithi017 ceph-mon[27571]: 5.18 scrub starts 2024-09-17T16:23:23.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:22 smithi017 ceph-mon[27571]: pgmap v1052: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 6212/246060 objects misplaced (2.525%) 2024-09-17T16:23:23.276 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:23.298 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:23.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:22 smithi071 ceph-mon[30002]: 5.18 scrub starts 2024-09-17T16:23:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:22 smithi071 ceph-mon[30002]: pgmap v1052: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 6212/246060 objects misplaced (2.525%) 2024-09-17T16:23:23.360 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-17T16:23:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:22 smithi173 ceph-mon[30530]: 5.18 scrub starts 2024-09-17T16:23:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:22 smithi173 ceph-mon[30530]: pgmap v1052: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 6212/246060 objects misplaced (2.525%) 2024-09-17T16:23:23.400 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:23.423 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:23.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:23.433 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:23.479 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:23.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:23.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:23.549 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:23.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:23.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:23.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:23.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:23.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:23.777 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:23.798 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:23.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:23.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:23.931 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:23.932 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:24.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:24.059 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:24.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:24.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:24.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:24.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:24.288 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:24.319 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:24.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:24.421 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:24.448 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:24.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:24.471 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:23:24.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:23:24.500 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:23:24.500 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:23:24.530 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:23:24.530 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:23:24.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:24.555 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:23:24.555 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:23:24.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:24.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:24.584 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:23:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:23:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:86653 5.a 5:51839755:::benchmark_data_smithi173_90735_object3882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-17T16:23:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:24.345766+0000", 2024-09-17T16:23:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.0036977889999999999, 2024-09-17T16:23:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037295869999999999, 2024-09-17T16:23:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:23:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:23:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 86653 2024-09-17T16:23:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.345766+0000", 2024-09-17T16:23:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.345766+0000", 2024-09-17T16:23:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.345768+0000", 2024-09-17T16:23:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3370000000000001e-06 2024-09-17T16:23:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.345774+0000", 2024-09-17T16:23:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9660000000000001e-06 2024-09-17T16:23:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.345786+0000", 2024-09-17T16:23:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2565000000000001e-05 2024-09-17T16:23:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.345792+0000", 2024-09-17T16:23:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8579999999999996e-06 2024-09-17T16:23:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.345851+0000", 2024-09-17T16:23:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9234999999999998e-05 2024-09-17T16:23:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.345938+0000", 2024-09-17T16:23:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6617000000000001e-05 2024-09-17T16:23:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,10", 2024-09-17T16:23:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.346019+0000", 2024-09-17T16:23:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1292999999999996e-05 2024-09-17T16:23:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.348579+0000", 2024-09-17T16:23:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002559988 2024-09-17T16:23:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.349285+0000", 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070523099999999996 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:86652 5.f e476/455 5:fe86949f:::benchmark_data_smithi173_90735_object3881:head v 476'5351, mlcod=445'2889)", 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:24.346655+0000", 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.002809644, 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029316450000000001, 2024-09-17T16:23:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.346655+0000", 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.346655+0000", 2024-09-17T16:23:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.346656+0000", 2024-09-17T16:23:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:23:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.346661+0000", 2024-09-17T16:23:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3600000000000004e-06 2024-09-17T16:23:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.346687+0000", 2024-09-17T16:23:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6601000000000002e-05 2024-09-17T16:23:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.346694+0000", 2024-09-17T16:23:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7299999999999999e-06 2024-09-17T16:23:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.346742+0000", 2024-09-17T16:23:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7626000000000001e-05 2024-09-17T16:23:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.346765+0000", 2024-09-17T16:23:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3008999999999999e-05 2024-09-17T16:23:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:24.349548+0000", 2024-09-17T16:23:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002783256 2024-09-17T16:23:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:23:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 2 2024-09-17T16:23:24.603 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:23:24.604 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:23:24.605 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:23:24.614 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:23:24.614 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:23:24.640 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:23:24.640 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:23:24.668 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:23:24.668 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:23:24.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:24.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:23:24.694 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:23:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:24.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:24.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:24.716 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:23:24.716 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:23:24.743 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:23:24.743 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:23:24.766 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:23:24.766 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:23:24.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:24.829 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:24.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:24.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:24.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:24.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:25.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:25.085 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:25.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:25.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:25.213 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:25.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:25.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:23:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:24 smithi017 ceph-mon[27571]: pgmap v1053: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 170 B/s rd, 0 op/s; 5764/245862 objects misplaced (2.344%) 2024-09-17T16:23:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/826845151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:24 smithi017 ceph-mon[27571]: osdmap e477: 12 total, 12 up, 10 in 2024-09-17T16:23:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:24 smithi071 ceph-mon[30002]: pgmap v1053: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 170 B/s rd, 0 op/s; 5764/245862 objects misplaced (2.344%) 2024-09-17T16:23:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/826845151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:24 smithi071 ceph-mon[30002]: osdmap e477: 12 total, 12 up, 10 in 2024-09-17T16:23:25.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:25.343 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:25.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:24 smithi173 ceph-mon[30530]: pgmap v1053: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 170 B/s rd, 0 op/s; 5764/245862 objects misplaced (2.344%) 2024-09-17T16:23:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/826845151' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:24 smithi173 ceph-mon[30530]: osdmap e477: 12 total, 12 up, 10 in 2024-09-17T16:23:25.400 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:23:25.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:25.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:25.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:25.579 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:25.602 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:25.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:25.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:25.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:25.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:25.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:25.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:25.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:25.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:25.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:25.986 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:25.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:26.088 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:26.116 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:26.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:26.125 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:26.219 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:26.244 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:26.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:26.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:26 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:23:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:26 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:23:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:26 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 34/237855 objects degraded (0.014%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:26 smithi071 ceph-mon[30002]: osdmap e478: 12 total, 12 up, 10 in 2024-09-17T16:23:26.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:26 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:23:26.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:26 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:23:26.332 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:26 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 34/237855 objects degraded (0.014%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:26.332 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:26 smithi017 ceph-mon[27571]: osdmap e478: 12 total, 12 up, 10 in 2024-09-17T16:23:26.346 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:26.374 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:26.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:26.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:26 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:23:26.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:26 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:23:26.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:26 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 34/237855 objects degraded (0.014%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:26.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:26 smithi173 ceph-mon[30530]: osdmap e478: 12 total, 12 up, 10 in 2024-09-17T16:23:26.477 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:26.502 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:26.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:26.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:26.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:26.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:26.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:26.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:26.756 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:26.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:26.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:26.882 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:26.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:26.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:27.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:27.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:27.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:27.139 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:27.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:27.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:27.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:27.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:27.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:27 smithi071 ceph-mon[30002]: pgmap v1055: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 159 B/s rd, 0 op/s; 34/237855 objects degraded (0.014%); 5034/237855 objects misplaced (2.116%) 2024-09-17T16:23:27.374 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:27 smithi173 ceph-mon[30530]: pgmap v1055: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 159 B/s rd, 0 op/s; 34/237855 objects degraded (0.014%); 5034/237855 objects misplaced (2.116%) 2024-09-17T16:23:27.402 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:27.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:27 smithi017 ceph-mon[27571]: pgmap v1055: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 159 B/s rd, 0 op/s; 34/237855 objects degraded (0.014%); 5034/237855 objects misplaced (2.116%) 2024-09-17T16:23:27.503 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:27.532 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:27.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:27.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:27.631 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-17T16:23:27.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:27.650 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:27.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:27.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:27.688 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:27.775 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:27.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:27.799 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:27.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:27.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:27.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:27.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:28.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:28.067 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:28.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:28.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:23:28.169 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:28.195 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:28.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:28.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:28.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:28.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:28.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:28.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:28.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:28.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:28.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:28.556 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:28.585 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:28.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:28.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:28.712 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:28.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:28.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:28.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:28.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:28.860 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:23:28.861 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:23:28.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:28.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:28.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:29.049 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:23:29.060 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:29.083 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:29.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:29.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:29.185 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:29.211 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:29.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:29.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:29 smithi071 ceph-mon[30002]: pgmap v1057: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 147 B/s rd, 0 B/s wr, 152 op/s; 34/233703 objects degraded (0.015%); 5034/233703 objects misplaced (2.154%) 2024-09-17T16:23:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1591153333' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:29 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:29.326 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:29.326 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:29.343 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:29.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:29.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:29.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:29 smithi173 ceph-mon[30530]: pgmap v1057: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 147 B/s rd, 0 B/s wr, 152 op/s; 34/233703 objects degraded (0.015%); 5034/233703 objects misplaced (2.154%) 2024-09-17T16:23:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1591153333' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:29 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:29.427 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:29 smithi017 ceph-mon[27571]: pgmap v1057: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 147 B/s rd, 0 B/s wr, 152 op/s; 34/233703 objects degraded (0.015%); 5034/233703 objects misplaced (2.154%) 2024-09-17T16:23:29.428 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1591153333' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:29.428 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:29 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:29.445 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:29.476 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:29.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:29.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:29.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:29.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:29.708 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:29.739 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:29.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:29.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:29.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:29.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:29.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:29.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:29.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:30.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:30.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:30.124 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:30.225 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:30.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:30 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:30.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:30 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:30.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:30 smithi017 ceph-mon[27571]: osdmap e479: 12 total, 12 up, 10 in 2024-09-17T16:23:30.229 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:23:30.253 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:30.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:30.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:30.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:30 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:30 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:30 smithi071 ceph-mon[30002]: osdmap e479: 12 total, 12 up, 10 in 2024-09-17T16:23:30.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:23:30.355 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:30.378 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:30.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:30.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:30 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:30 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:30 smithi173 ceph-mon[30530]: osdmap e479: 12 total, 12 up, 10 in 2024-09-17T16:23:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:23:30.480 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:30.503 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:30.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:30.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:30.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:30.725 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:23:30.725 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:23:30.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:30.760 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:23:30.760 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:23:30.761 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:30.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:30.791 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:23:30.791 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:23:30.818 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:23:30.818 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:23:30.849 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:23:30.849 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:23:30.863 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:30.876 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:23:30.876 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:23:30.891 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:30.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:30.902 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:23:30.903 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:23:30.930 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:23:30.930 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:23:30.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:23:30.960 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:23:30.993 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:30.998 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:23:30.998 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:23:31.023 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:31.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:31.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:31.026 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:23:31.026 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:23:31.058 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:23:31.058 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:23:31.125 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:31.151 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:31.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:31.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:31.275 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:31.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:31 smithi071 ceph-mon[30002]: pgmap v1059: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 245 KiB/s wr, 1.16k op/s; 34/225072 objects degraded (0.015%); 3919/225072 objects misplaced (1.741%); 17 MiB/s, 264 objects/s recovering 2024-09-17T16:23:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:23:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:23:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:23:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1303681821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:31.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:31.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:31 smithi173 ceph-mon[30530]: pgmap v1059: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 245 KiB/s wr, 1.16k op/s; 34/225072 objects degraded (0.015%); 3919/225072 objects misplaced (1.741%); 17 MiB/s, 264 objects/s recovering 2024-09-17T16:23:31.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:23:31.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:23:31.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:23:31.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1303681821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:31.400 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:31 smithi017 ceph-mon[27571]: pgmap v1059: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 245 KiB/s wr, 1.16k op/s; 34/225072 objects degraded (0.015%); 3919/225072 objects misplaced (1.741%); 17 MiB/s, 264 objects/s recovering 2024-09-17T16:23:31.401 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:23:31.401 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:23:31.401 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:23:31.401 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1303681821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:31.401 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:31.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:31.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:31.527 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:31.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:31.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:31.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:31.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:31.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:31.774 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:31.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:31.876 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:31.899 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:31.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:31.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:31.912 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:23:32.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:32.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:32.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:32.125 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:32.147 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:32.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:32.249 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:32.275 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:32.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:32.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:32.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:32.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:32.503 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-09-17T16:23:32.507 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:32.528 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:32.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:32.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:32.652 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:32.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:32.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:32.718 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:32 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:23:32.718 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:32 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:23:32.719 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:32 smithi017 ceph-mon[27571]: osdmap e480: 12 total, 12 up, 10 in 2024-09-17T16:23:32.719 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:32 smithi017 ceph-mon[27571]: pgmap v1061: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 34/218028 objects degraded (0.016%); 3919/218028 objects misplaced (1.797%); 10 MiB/s, 166 objects/s recovering 2024-09-17T16:23:32.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:23:32.721 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:32.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:32.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:32.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:32 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:23:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:32 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:23:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:32 smithi071 ceph-mon[30002]: osdmap e480: 12 total, 12 up, 10 in 2024-09-17T16:23:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:32 smithi071 ceph-mon[30002]: pgmap v1061: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 34/218028 objects degraded (0.016%); 3919/218028 objects misplaced (1.797%); 10 MiB/s, 166 objects/s recovering 2024-09-17T16:23:32.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:32 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:23:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:32 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:23:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:32 smithi173 ceph-mon[30530]: osdmap e480: 12 total, 12 up, 10 in 2024-09-17T16:23:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:32 smithi173 ceph-mon[30530]: pgmap v1061: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 24 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 34/218028 objects degraded (0.016%); 3919/218028 objects misplaced (1.797%); 10 MiB/s, 166 objects/s recovering 2024-09-17T16:23:32.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:32.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:33.004 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:33.026 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:33.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:33.128 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:33.150 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:33.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:33.251 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:33.275 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:33.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:33.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:33.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:33.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:33.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:33.499 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:33.526 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:33.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:33.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:33.648 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:33.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:33 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 34/218028 objects degraded (0.016%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:33 smithi017 ceph-mon[27571]: osdmap e481: 12 total, 12 up, 10 in 2024-09-17T16:23:33.750 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:33.772 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:33.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:33 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 34/218028 objects degraded (0.016%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:33 smithi071 ceph-mon[30002]: osdmap e481: 12 total, 12 up, 10 in 2024-09-17T16:23:33.848 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:33.848 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:33.874 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:33 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 34/218028 objects degraded (0.016%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:33 smithi173 ceph-mon[30530]: osdmap e481: 12 total, 12 up, 10 in 2024-09-17T16:23:33.898 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:33.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:34.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:34.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:34.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:34.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:34.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:34.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:34.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:34.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:34.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:34.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:34.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:34.396 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:34.466 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:23:34.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:34.517 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:34.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:34.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:34.650 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:34.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:34.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:34 smithi017 ceph-mon[27571]: osdmap e482: 12 total, 12 up, 10 in 2024-09-17T16:23:34.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:34 smithi017 ceph-mon[27571]: pgmap v1064: 27 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 24 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 34/214431 objects degraded (0.016%); 3301/214431 objects misplaced (1.539%); 14 MiB/s, 226 objects/s recovering 2024-09-17T16:23:34.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/36407346' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:23:34.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:34 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:34.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/36407346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:23:34.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:34 smithi017 ceph-mon[27571]: osdmap e483: 12 total, 12 up, 10 in 2024-09-17T16:23:34.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:34.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:34.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:34.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:34 smithi071 ceph-mon[30002]: osdmap e482: 12 total, 12 up, 10 in 2024-09-17T16:23:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:34 smithi071 ceph-mon[30002]: pgmap v1064: 27 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 24 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 34/214431 objects degraded (0.016%); 3301/214431 objects misplaced (1.539%); 14 MiB/s, 226 objects/s recovering 2024-09-17T16:23:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/36407346' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:23:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:34 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/36407346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:23:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:34 smithi071 ceph-mon[30002]: osdmap e483: 12 total, 12 up, 10 in 2024-09-17T16:23:34.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:34 smithi173 ceph-mon[30530]: osdmap e482: 12 total, 12 up, 10 in 2024-09-17T16:23:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:34 smithi173 ceph-mon[30530]: pgmap v1064: 27 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 24 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s; 34/214431 objects degraded (0.016%); 3301/214431 objects misplaced (1.539%); 14 MiB/s, 226 objects/s recovering 2024-09-17T16:23:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/36407346' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:23:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:34 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:34.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/36407346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:23:34.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:34 smithi173 ceph-mon[30530]: osdmap e483: 12 total, 12 up, 10 in 2024-09-17T16:23:34.907 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:34.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:35.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:35.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:35.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:35.110 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:23:35.110 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:23:35.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:23:35.110 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:23:35.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.110 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(3.4 241/239 [PushOp(3:20540438:::benchmark_data_smithi173_66272_object5569:head, version: 213'343, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:20540438:::benchmark_data_smithi173_66272_object5569:head@213'343, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:23:35.110 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 497.54617631600001, 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56341482399999998, 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552794+0000", 2024-09-17T16:23:35.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552796+0000", 2024-09-17T16:23:35.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.888e-06 2024-09-17T16:23:35.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:17.552825+0000", 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9417999999999999e-05 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115364+0000", 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56253852800000004 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115383+0000", 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9055999999999998e-05 2024-09-17T16:23:35.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115470+0000", 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7520000000000002e-05 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.115481+0000", 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0329000000000001e-05 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:18.116209+0000", 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072808499999999995 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25516.0:155234 4.6 e371/361 4:60535f2d:::benchmark_data_smithi173_77474_object53059:head v 371'9504, mlcod=371'9501)", 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 202.50927263299999, 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30676646000000002, 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.115 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589704+0000", 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5330000000000002e-06 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.891997+0000", 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30229256300000001 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.892001+0000", 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6539999999999998e-06 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.892046+0000", 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5305000000000001e-05 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.892062+0000", 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5931999999999999e-05 2024-09-17T16:23:35.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.896408+0000", 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043459869999999999 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.896464+0000", 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5717000000000002e-05 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.119 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:185136 4.6 4:6e122307:::benchmark_data_smithi173_77474_object82961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:55.713056+0000", 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 159.385914316, 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30286027900000001, 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 185136 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713056+0000", 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713056+0000", 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713058+0000", 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1100000000000001e-06 2024-09-17T16:23:35.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713068+0000", 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7890000000000006e-06 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713085+0000", 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6660999999999999e-05 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713092+0000", 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5159999999999997e-06 2024-09-17T16:23:35.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.714377+0000", 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012851379999999999 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.714505+0000", 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012745500000000001 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:23:35.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.714629+0000", 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012392699999999999 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.718974+0000", 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043457169999999998 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.718988+0000", 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.375e-05 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015789+0000", 2024-09-17T16:23:35.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29680071600000002 2024-09-17T16:23:35.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015866+0000", 2024-09-17T16:23:35.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7026000000000004e-05 2024-09-17T16:23:35.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015916+0000", 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0473999999999998e-05 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:185136 4.6 e391/387 ondisk, result = 0)", 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:55.717118+0000", 2024-09-17T16:23:35.129 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 159.3818521, 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29880589800000001, 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717118+0000", 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717118+0000", 2024-09-17T16:23:35.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717120+0000", 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.392e-06 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717126+0000", 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8279999999999997e-06 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015002+0000", 2024-09-17T16:23:35.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.297875367 2024-09-17T16:23:35.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015011+0000", 2024-09-17T16:23:35.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1519999999999998e-06 2024-09-17T16:23:35.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015769+0000", 2024-09-17T16:23:35.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075783199999999995 2024-09-17T16:23:35.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015780+0000", 2024-09-17T16:23:35.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.166e-05 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015924+0000", 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014366699999999999 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:185177 4.6 4:6ef442e2:::benchmark_data_smithi173_77474_object83002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:56.206993+0000", 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 158.89197693899999, 2024-09-17T16:23:35.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39139948400000002, 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 185177 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.206993+0000", 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.206993+0000", 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.206995+0000", 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0619999999999999e-06 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207004+0000", 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5019999999999992e-06 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207022+0000", 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8365000000000001e-05 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207030+0000", 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5639999999999999e-06 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207096+0000", 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5938999999999993e-05 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207197+0000", 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100929 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207279+0000", 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2474999999999999e-05 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211157+0000", 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003878009 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.423507+0000", 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21234955799999999 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598152+0000", 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.174645726 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598251+0000", 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8258999999999996e-05 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598393+0000", 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142096 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:185177 4.6 e391/387 ondisk, result = 0)", 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:56.211354+0000", 2024-09-17T16:23:35.140 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 158.88761638700001, 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38704429800000001, 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211354+0000", 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211354+0000", 2024-09-17T16:23:35.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211355+0000", 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8899999999999998e-07 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211363+0000", 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4509999999999992e-06 2024-09-17T16:23:35.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.597986+0000", 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38662227900000001 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.597997+0000", 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.189e-05 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598126+0000", 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000128505 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598143+0000", 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7405999999999999e-05 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598398+0000", 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025487799999999999 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:186301 4.4 4:28479395:::benchmark_data_smithi173_77474_object84126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:58.360185+0000", 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 156.73878520599999, 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37107270399999998, 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186301 2024-09-17T16:23:35.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360185+0000", 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360185+0000", 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360186+0000", 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.02e-06 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360191+0000", 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6269999999999999e-06 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360201+0000", 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0583e-05 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360207+0000", 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1989999999999997e-06 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360255+0000", 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8810999999999997e-05 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360347+0000", 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1436999999999999e-05 2024-09-17T16:23:35.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360424+0000", 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7566999999999998e-05 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364029+0000", 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036048090000000001 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364399+0000", 2024-09-17T16:23:35.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000370331 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731169+0000", 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36676913100000003 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731215+0000", 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6047000000000001e-05 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731258+0000", 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3142000000000001e-05 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:186301 4.4 e393/306 ondisk, result = 0)", 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:58.364680+0000", 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 156.734289924, 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36658104899999999, 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:35.151 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364680+0000", 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364680+0000", 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364681+0000", 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6899999999999997e-07 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364685+0000", 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1579999999999998e-06 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731114+0000", 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.366428431 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731117+0000", 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5549999999999998e-06 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731158+0000", 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0655999999999998e-05 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731163+0000", 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1869999999999996e-06 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731261+0000", 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8393000000000005e-05 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204346 4.4 4:22db2208:::benchmark_data_smithi173_77474_object102171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.625061+0000", 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 131.47390910300001, 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.67454315899999995, 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.155 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.156 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:35.156 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:23:35.156 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:23:35.156 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204346 2024-09-17T16:23:35.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.156 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:35.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625061+0000", 2024-09-17T16:23:35.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625061+0000", 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625062+0000", 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0300000000000001e-06 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625067+0000", 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4320000000000001e-06 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625077+0000", 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9189999999999994e-06 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625083+0000", 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9020000000000002e-06 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625140+0000", 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6677e-05 2024-09-17T16:23:35.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625237+0000", 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6446999999999993e-05 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625308+0000", 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1585e-05 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.629047+0000", 2024-09-17T16:23:35.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037388569999999999 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.630013+0000", 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096557000000000001 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299522+0000", 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.66950939200000004 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299565+0000", 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.316e-05 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299604+0000", 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9187999999999998e-05 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:2256 5.7 e424/419 5:e0aee0a5:::benchmark_data_smithi173_90735_object2255:head v 424'163, mlcod=424'160)", 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:51.481567+0000", 2024-09-17T16:23:35.163 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 103.617402833, 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27652685100000002, 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.481567+0000", 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.481567+0000", 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.481568+0000", 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6300000000000004e-07 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.481611+0000", 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2704000000000003e-05 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.757551+0000", 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27593994300000002 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.757561+0000", 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0052e-05 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.757619+0000", 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8112000000000002e-05 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.757636+0000", 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6789000000000001e-05 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.758061+0000", 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042476699999999999 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.758094+0000", 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3720999999999998e-05 2024-09-17T16:23:35.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:30436 5.a 5:5adf2834:::benchmark_data_smithi173_90735_object30435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:14.766561+0000", 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 80.332408786000002, 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38050945800000002, 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:23:35.168 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 30436 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766561+0000", 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766561+0000", 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766564+0000", 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1490000000000001e-06 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766601+0000", 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7072e-05 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766613+0000", 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2714000000000001e-05 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766619+0000", 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6210000000000001e-06 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766677+0000", 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7572999999999999e-05 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766839+0000", 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016253700000000001 2024-09-17T16:23:35.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,10", 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766995+0000", 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015596800000000001 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.768067+0000", 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010722049999999999 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.768388+0000", 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032046199999999998 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:15.146963+0000", 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37857484499999999 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:15.147022+0000", 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9108999999999999e-05 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:15.147071+0000", 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9203000000000003e-05 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:41583 5.a 5:52d43247:::benchmark_data_smithi173_90735_object41582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:23.939462+0000", 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.159507805999993, 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27797282600000001, 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.174 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41583 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939462+0000", 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939462+0000", 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939463+0000", 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9100000000000002e-07 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939505+0000", 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1600999999999997e-05 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939515+0000", 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0448999999999999e-05 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939526+0000", 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.027e-05 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939586+0000", 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0689000000000001e-05 2024-09-17T16:23:35.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939714+0000", 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012727999999999999 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,10", 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939864+0000", 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000150419 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:35.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.941306+0000", 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001441459 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.948800+0000", 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074941019999999999 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217291+0000", 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.268491539 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217377+0000", 2024-09-17T16:23:35.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5650999999999997e-05 2024-09-17T16:23:35.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217435+0000", 2024-09-17T16:23:35.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8476000000000001e-05 2024-09-17T16:23:35.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.180 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26158.0:41583 5.a e440/419 ondisk, result = 0)", 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:23.941903+0000", 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.157066985, 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27554273499999998, 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.941903+0000", 2024-09-17T16:23:35.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.941903+0000", 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.941904+0000", 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8400000000000004e-07 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.941909+0000", 2024-09-17T16:23:35.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0599999999999998e-06 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217044+0000", 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27513484599999999 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217061+0000", 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7237000000000001e-05 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217271+0000", 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021042399999999999 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217284+0000", 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2237e-05 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217446+0000", 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000162347 2024-09-17T16:23:35.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:41594 5.a 5:5f46650c:::benchmark_data_smithi173_90735_object41593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:23.946995+0000", 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.151975022000002, 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62805596699999999, 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.185 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41594 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.946995+0000", 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.946995+0000", 2024-09-17T16:23:35.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.946999+0000", 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7570000000000002e-06 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947087+0000", 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8115999999999995e-05 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947127+0000", 2024-09-17T16:23:35.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9746000000000002e-05 2024-09-17T16:23:35.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947149+0000", 2024-09-17T16:23:35.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1704000000000001e-05 2024-09-17T16:23:35.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947258+0000", 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010904399999999999 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947407+0000", 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014971 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,10", 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947573+0000", 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165639 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.948902+0000", 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001329326 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.952783+0000", 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038808900000000001 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574883+0000", 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62209975299999998 2024-09-17T16:23:35.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574982+0000", 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8795999999999994e-05 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.575051+0000", 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9486000000000005e-05 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.191 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26158.0:41594 5.a e440/419 ondisk, result = 0)", 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:23.949386+0000", 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.149583786999997, 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62567850599999997, 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.949386+0000", 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.949386+0000", 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.949387+0000", 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6199999999999998e-07 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.949390+0000", 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3620000000000002e-06 2024-09-17T16:23:35.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574683+0000", 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62529263099999999 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574708+0000", 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5130000000000002e-05 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574856+0000", 2024-09-17T16:23:35.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147732 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574873+0000", 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7430000000000001e-05 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.575065+0000", 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000191759 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.196 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:54616 5.17 5:eee108da:::benchmark_data_smithi173_90735_object54615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-17T16:23:35.196 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:39.739821+0000", 2024-09-17T16:23:35.196 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.359149031999998, 2024-09-17T16:23:35.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58859298699999996, 2024-09-17T16:23:35.196 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.196 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.196 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.196 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:35.196 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:23:35.196 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54616 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739821+0000", 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739821+0000", 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739822+0000", 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.02e-07 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739877+0000", 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5259000000000001e-05 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739889+0000", 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1405e-05 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739894+0000", 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.135e-06 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739944+0000", 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0217999999999999e-05 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.740046+0000", 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000101471 2024-09-17T16:23:35.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,8,11", 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.740149+0000", 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103126 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.741188+0000", 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001039138 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.742452+0000", 2024-09-17T16:23:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012636100000000001 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.742555+0000", 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103479 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.327594+0000", 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58503854899999996 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.327627+0000", 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3605000000000001e-05 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.328414+0000", 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078708999999999995 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:82768 5.6 5:63886666:::benchmark_data_smithi173_90735_object82767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:19.925436+0000", 2024-09-17T16:23:35.202 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 15.173534122, 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.83184638399999999, 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 82768 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925436+0000", 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925436+0000", 2024-09-17T16:23:35.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925438+0000", 2024-09-17T16:23:35.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-17T16:23:35.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925521+0000", 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3159999999999997e-05 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925532+0000", 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1735e-05 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925538+0000", 2024-09-17T16:23:35.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4870000000000002e-06 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925599+0000", 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1247999999999994e-05 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925711+0000", 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112236 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:23:35.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925799+0000", 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7453000000000004e-05 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.927073+0000", 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012737180000000001 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.927421+0000", 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00034853299999999999 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.757202+0000", 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.829780505 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.757247+0000", 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5207999999999998e-05 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.757283+0000", 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5734e-05 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.208 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(5.f 475/455 [PushOp(5:ff17ea6b:::benchmark_data_smithi173_90735_object41658:head, version: 442'2553, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ff17ea6b:::benchmark_data_smithi173_90735_object41658:head@442'2553, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:20.334257+0000", 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.764713638, 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.573612715, 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.334257+0000", 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.334257+0000", 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.334257+0000", 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6700000000000003e-07 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.334304+0000", 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6513e-05 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.906366+0000", 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.572062183 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.906385+0000", 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9215000000000002e-05 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.906471+0000", 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.906481+0000", 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7990000000000005e-06 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.907869+0000", 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001388249 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:95258 5.a 5:5d45a7a2:::benchmark_data_smithi173_90735_object12487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:32.503633+0000", 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 2.5953371550000002, 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.214840057, 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 95258 2024-09-17T16:23:35.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.503633+0000", 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.503633+0000", 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.503635+0000", 2024-09-17T16:23:35.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4529999999999999e-06 2024-09-17T16:23:35.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.503639+0000", 2024-09-17T16:23:35.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8810000000000004e-06 2024-09-17T16:23:35.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.503652+0000", 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2233e-05 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.503660+0000", 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7169999999999997e-06 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.503710+0000", 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9400999999999997e-05 2024-09-17T16:23:35.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.503803+0000", 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3356999999999994e-05 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,10", 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.503881+0000", 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8122999999999997e-05 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.506457+0000", 2024-09-17T16:23:35.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0025755589999999998 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.535468+0000", 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.029011067000000001 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.718398+0000", 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.18292983800000001 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:35.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.718440+0000", 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2327999999999999e-05 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.718473+0000", 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3099999999999998e-05 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26158.0:95258 5.a e481/419 ondisk, result = 0)", 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:32.522142+0000", 2024-09-17T16:23:35.219 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 2.5768285280000001, 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.196337659, 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.522142+0000", 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.522142+0000", 2024-09-17T16:23:35.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.522143+0000", 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3659999999999999e-06 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.522151+0000", 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1990000000000006e-06 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:35.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.718314+0000", 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.19616244699999999 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.718333+0000", 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.878e-05 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.718388+0000", 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5473000000000002e-05 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:35.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.718394+0000", 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0979999999999999e-06 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:32.718479+0000", 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5296000000000005e-05 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:35.223 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:23:35.224 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:23:35.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:35.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:35.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:35.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:35.351 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:35.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:35.453 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:35.478 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:35.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:35.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:35.603 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:35.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:35.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:35.705 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:35.730 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:35.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:35.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:35.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:35.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:35.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:35.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:35.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:35.987 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:35.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:35.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:35 smithi017 ceph-mon[27571]: 5.6 scrub starts 2024-09-17T16:23:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:35 smithi071 ceph-mon[30002]: 5.6 scrub starts 2024-09-17T16:23:36.088 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:36.113 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:36.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:35 smithi173 ceph-mon[30530]: 5.6 scrub starts 2024-09-17T16:23:36.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:36.238 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:36.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:36.340 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:36.367 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:36.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:36.468 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:36.494 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:36.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:36.596 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:36.622 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:36.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:36.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:36.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:36.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:36 smithi071 ceph-mon[30002]: pgmap v1066: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 25 active+clean; 4.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 739 op/s; 34/209658 objects degraded (0.016%); 2759/209658 objects misplaced (1.316%); 14 MiB/s, 220 objects/s recovering 2024-09-17T16:23:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:36 smithi071 ceph-mon[30002]: 5.6 scrub ok 2024-09-17T16:23:36.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-09-17T16:23:36.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:36.884 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:36.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:36.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:36 smithi017 ceph-mon[27571]: pgmap v1066: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 25 active+clean; 4.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 739 op/s; 34/209658 objects degraded (0.016%); 2759/209658 objects misplaced (1.316%); 14 MiB/s, 220 objects/s recovering 2024-09-17T16:23:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:36 smithi017 ceph-mon[27571]: 5.6 scrub ok 2024-09-17T16:23:37.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:37.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:37.014 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:37.115 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:37.144 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:37.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:36 smithi173 ceph-mon[30530]: pgmap v1066: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 25 active+clean; 4.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 739 op/s; 34/209658 objects degraded (0.016%); 2759/209658 objects misplaced (1.316%); 14 MiB/s, 220 objects/s recovering 2024-09-17T16:23:37.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:36 smithi173 ceph-mon[30530]: 5.6 scrub ok 2024-09-17T16:23:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:37.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:37.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:37.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:37.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:37.298 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:23:37.306 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:37.354 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:37.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:37.409 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:37.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:37.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:37.540 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:37.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:37.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:37.670 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:37.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:37.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:37.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:37.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:37.800 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:37.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:37.929 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:37.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:37.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:37.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:37 smithi017 ceph-mon[27571]: 5.f scrub starts 2024-09-17T16:23:38.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:38.055 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:38.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:38.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:38.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:37 smithi071 ceph-mon[30002]: 5.f scrub starts 2024-09-17T16:23:38.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:37 smithi173 ceph-mon[30530]: 5.f scrub starts 2024-09-17T16:23:38.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:38.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:38.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:38.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:38.281 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:38.307 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:38.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:38.367 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:23:38.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:38.455 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:38.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:38.485 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:23:38.486 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:23:38.514 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:23:38.514 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:23:38.546 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:23:38.546 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:23:38.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:38.575 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:23:38.575 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:23:38.581 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:38.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:38.605 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:23:38.608 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:23:38.639 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:23:38.639 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:23:38.665 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:23:38.665 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:23:38.683 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:38.697 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:23:38.697 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:23:38.709 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:38.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:38.733 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:23:38.733 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:23:38.764 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:23:38.765 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:23:38.796 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:23:38.796 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:23:38.811 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:38.824 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:23:38.824 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:23:38.835 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:38.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:38.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:38.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:38 smithi017 ceph-mon[27571]: 5.f scrub ok 2024-09-17T16:23:38.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:38 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:23:38.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:38 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:23:38.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:38 smithi017 ceph-mon[27571]: osdmap e484: 12 total, 12 up, 10 in 2024-09-17T16:23:38.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:38 smithi017 ceph-mon[27571]: pgmap v1068: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 25 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 615 op/s; 34/207135 objects degraded (0.016%); 2759/207135 objects misplaced (1.332%); 11 MiB/s, 179 objects/s recovering 2024-09-17T16:23:38.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1305579873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:38.936 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:38.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:39.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:38 smithi173 ceph-mon[30530]: 5.f scrub ok 2024-09-17T16:23:39.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:38 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:23:39.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:38 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:23:39.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:38 smithi173 ceph-mon[30530]: osdmap e484: 12 total, 12 up, 10 in 2024-09-17T16:23:39.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:38 smithi173 ceph-mon[30530]: pgmap v1068: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 25 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 615 op/s; 34/207135 objects degraded (0.016%); 2759/207135 objects misplaced (1.332%); 11 MiB/s, 179 objects/s recovering 2024-09-17T16:23:39.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1305579873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:39.063 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:38 smithi071 ceph-mon[30002]: 5.f scrub ok 2024-09-17T16:23:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:38 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:23:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:38 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:23:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:38 smithi071 ceph-mon[30002]: osdmap e484: 12 total, 12 up, 10 in 2024-09-17T16:23:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:38 smithi071 ceph-mon[30002]: pgmap v1068: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 25 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 615 op/s; 34/207135 objects degraded (0.016%); 2759/207135 objects misplaced (1.332%); 11 MiB/s, 179 objects/s recovering 2024-09-17T16:23:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1305579873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:39.088 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:39.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:39.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:39.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:39.213 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:39.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:39.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:39.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:39.340 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:39.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:39.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:39.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:39.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:39.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:39.595 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:39.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:39.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:39.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:39.726 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:39.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:39.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:39.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:39.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:39.982 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:39.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:39.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:39 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 34/207135 objects degraded (0.016%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:39 smithi017 ceph-mon[27571]: osdmap e485: 12 total, 12 up, 10 in 2024-09-17T16:23:40.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:39 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 34/207135 objects degraded (0.016%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:40.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:39 smithi071 ceph-mon[30002]: osdmap e485: 12 total, 12 up, 10 in 2024-09-17T16:23:40.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:40.109 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:40.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:40.113 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:40.114 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:39 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 34/207135 objects degraded (0.016%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:39 smithi173 ceph-mon[30530]: osdmap e485: 12 total, 12 up, 10 in 2024-09-17T16:23:40.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:40.235 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:40.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:40.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:40.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:40.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:40.466 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:40.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:40.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:40.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:40.616 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:40.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:40.678 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-09-17T16:23:40.718 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:40.740 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:40.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:40.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:40.843 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:40.870 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:40.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:40.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:40 smithi017 ceph-mon[27571]: pgmap v1070: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 849 op/s; 34/199131 objects degraded (0.017%); 2567/199131 objects misplaced (1.289%); 4.4 MiB/s, 70 objects/s recovering 2024-09-17T16:23:40.972 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:40.994 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:40.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:40 smithi071 ceph-mon[30002]: pgmap v1070: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 849 op/s; 34/199131 objects degraded (0.017%); 2567/199131 objects misplaced (1.289%); 4.4 MiB/s, 70 objects/s recovering 2024-09-17T16:23:41.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:41.124 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:41.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:40 smithi173 ceph-mon[30530]: pgmap v1070: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 849 op/s; 34/199131 objects degraded (0.017%); 2567/199131 objects misplaced (1.289%); 4.4 MiB/s, 70 objects/s recovering 2024-09-17T16:23:41.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:41.251 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:41.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:41.352 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:41.380 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:41.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:41.481 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:41.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:41.505 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:41.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:41.607 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:41.629 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:41.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:41.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:41.754 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:41.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:41.856 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:41.881 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:41.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:41.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:41.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:41 smithi017 ceph-mon[27571]: 5.1 scrub starts 2024-09-17T16:23:41.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:41 smithi017 ceph-mon[27571]: 5.1 scrub ok 2024-09-17T16:23:42.004 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:23:42.004 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:23:42.011 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:42.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:42.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:42.055 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:23:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:41 smithi071 ceph-mon[30002]: 5.1 scrub starts 2024-09-17T16:23:42.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:41 smithi071 ceph-mon[30002]: 5.1 scrub ok 2024-09-17T16:23:42.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:42.141 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:42.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:42.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:41 smithi173 ceph-mon[30530]: 5.1 scrub starts 2024-09-17T16:23:42.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:41 smithi173 ceph-mon[30530]: 5.1 scrub ok 2024-09-17T16:23:42.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:42.265 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:42.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:42.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:42.374 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:42.398 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:42.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:42.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:42.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:42.529 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:42.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:42.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:42.655 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:42.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:42.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:42.755 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:23:42.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:42.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:42.784 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:42.884 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:42.907 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:42.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:42.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:43.010 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:43.033 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:43.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:43.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:43.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:42 smithi071 ceph-mon[30002]: pgmap v1071: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1k op/s; 34/191349 objects degraded (0.018%); 2567/191349 objects misplaced (1.342%); 2.8 MiB/s, 44 objects/s recovering 2024-09-17T16:23:43.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1321788988' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:43.113 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:42 smithi017 ceph-mon[27571]: pgmap v1071: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1k op/s; 34/191349 objects degraded (0.018%); 2567/191349 objects misplaced (1.342%); 2.8 MiB/s, 44 objects/s recovering 2024-09-17T16:23:43.116 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1321788988' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:43.135 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:43.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:42 smithi173 ceph-mon[30530]: pgmap v1071: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1k op/s; 34/191349 objects degraded (0.018%); 2567/191349 objects misplaced (1.342%); 2.8 MiB/s, 44 objects/s recovering 2024-09-17T16:23:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1321788988' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:43.148 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:23:43.157 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:43.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:43.259 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:43.283 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:43.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:43.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:43.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:43.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:43.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:43.511 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:43.535 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:43.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:43.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:43.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:43.665 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:43.766 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:43.793 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:43.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:43.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:43.895 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:43.924 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:43.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:44.025 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:44.050 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:44.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:44.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:43 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:23:44.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1321788988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:44.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:43 smithi071 ceph-mon[30002]: osdmap e486: 12 total, 12 up, 10 in 2024-09-17T16:23:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:43 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:23:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1321788988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:43 smithi173 ceph-mon[30530]: osdmap e486: 12 total, 12 up, 10 in 2024-09-17T16:23:44.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:44.178 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:44.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:44.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:43 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:23:44.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1321788988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:44.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:43 smithi017 ceph-mon[27571]: osdmap e486: 12 total, 12 up, 10 in 2024-09-17T16:23:44.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:44.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:44.303 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:44.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:44.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:44.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:44.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:44.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:44.438 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:44.475 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:44.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:44.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:44.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:44.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:44.653 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:44.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:44.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:44.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:44.783 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:44.804 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:44.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:44.906 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:44.932 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:44.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:44 smithi017 ceph-mon[27571]: pgmap v1073: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 34/186852 objects degraded (0.018%); 2567/186852 objects misplaced (1.374%); 665 KiB/s, 10 objects/s recovering 2024-09-17T16:23:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:44 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 34/191349 objects degraded (0.018%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:44 smithi017 ceph-mon[27571]: osdmap e487: 12 total, 12 up, 10 in 2024-09-17T16:23:45.034 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:45.057 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:45.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:44 smithi071 ceph-mon[30002]: pgmap v1073: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 34/186852 objects degraded (0.018%); 2567/186852 objects misplaced (1.374%); 665 KiB/s, 10 objects/s recovering 2024-09-17T16:23:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:44 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 34/191349 objects degraded (0.018%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:44 smithi071 ceph-mon[30002]: osdmap e487: 12 total, 12 up, 10 in 2024-09-17T16:23:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:44 smithi173 ceph-mon[30530]: pgmap v1073: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 34/186852 objects degraded (0.018%); 2567/186852 objects misplaced (1.374%); 665 KiB/s, 10 objects/s recovering 2024-09-17T16:23:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:44 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 34/191349 objects degraded (0.018%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:44 smithi173 ceph-mon[30530]: osdmap e487: 12 total, 12 up, 10 in 2024-09-17T16:23:45.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:45.180 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:45.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:45.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:45.308 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:45.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:45.410 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:45.433 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:45.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:45.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-09-17T16:23:45.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:45.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:45.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:45.664 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:45.687 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:45.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:45.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:45.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:45.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:45.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:45.942 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:45.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:45.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:46.045 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:46.071 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:46.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:46.173 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:46.202 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:46.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:46.236 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:23:46.236 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:23:46.260 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:23:46.261 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:23:46.283 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:23:46.283 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:23:46.302 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:23:46.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:46.305 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:23:46.305 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:23:46.329 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:23:46.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:23:46.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:46.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:46.357 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:23:46.357 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:23:46.385 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:23:46.386 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:23:46.417 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:23:46.417 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:23:46.422 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:23:46.422 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:23:46.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:46.442 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:23:46.442 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:23:46.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:46.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:46.472 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:23:46.472 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:23:46.496 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:23:46.497 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:23:46.522 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:23:46.523 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:23:46.561 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:46.584 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:46.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:46.686 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:46.711 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:46.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:46.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:46.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:46.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:46.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:46.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:46.960 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:46.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:46.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:46 smithi017 ceph-mon[27571]: pgmap v1075: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 34/179622 objects degraded (0.019%); 2297/179622 objects misplaced (1.279%); 887 KiB/s, 13 objects/s recovering 2024-09-17T16:23:46.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/514082780' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:47.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:47.085 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:47.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:47.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:47.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:47.215 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:47.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:47.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:46 smithi071 ceph-mon[30002]: pgmap v1075: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 34/179622 objects degraded (0.019%); 2297/179622 objects misplaced (1.279%); 887 KiB/s, 13 objects/s recovering 2024-09-17T16:23:47.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/514082780' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:47.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:47.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:47.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:46 smithi173 ceph-mon[30530]: pgmap v1075: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 34/179622 objects degraded (0.019%); 2297/179622 objects misplaced (1.279%); 887 KiB/s, 13 objects/s recovering 2024-09-17T16:23:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/514082780' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:47.439 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:47.462 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:47.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:47.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:47.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:47.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:47.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:47.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:47.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:47.814 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:23:47.825 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:47.848 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:47.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:47.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:47.977 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:47.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:48.079 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:48.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:48.115 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:48.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:48.116 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:48.117 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:48.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:47 smithi017 ceph-mon[27571]: 5.0 scrub starts 2024-09-17T16:23:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:48.240 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:48.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:47 smithi071 ceph-mon[30002]: 5.0 scrub starts 2024-09-17T16:23:48.341 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:48.364 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:48.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:48.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:47 smithi173 ceph-mon[30530]: 5.0 scrub starts 2024-09-17T16:23:48.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:48.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:48.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:48.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:48.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:48.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:48.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:48.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:48.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:48.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:48.867 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:48.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:48.969 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:48.991 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:48.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:49.094 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:49.124 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:49.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:49.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:48 smithi017 ceph-mon[27571]: 5.0 scrub ok 2024-09-17T16:23:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:48 smithi017 ceph-mon[27571]: pgmap v1076: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 950 op/s; 34/176310 objects degraded (0.019%); 2297/176310 objects misplaced (1.303%); 712 KiB/s, 11 objects/s recovering 2024-09-17T16:23:49.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:49.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:49.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:48 smithi071 ceph-mon[30002]: 5.0 scrub ok 2024-09-17T16:23:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:48 smithi071 ceph-mon[30002]: pgmap v1076: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 950 op/s; 34/176310 objects degraded (0.019%); 2297/176310 objects misplaced (1.303%); 712 KiB/s, 11 objects/s recovering 2024-09-17T16:23:49.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:49.375 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:49.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:49.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:48 smithi173 ceph-mon[30530]: 5.0 scrub ok 2024-09-17T16:23:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:48 smithi173 ceph-mon[30530]: pgmap v1076: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 950 op/s; 34/176310 objects degraded (0.019%); 2297/176310 objects misplaced (1.303%); 712 KiB/s, 11 objects/s recovering 2024-09-17T16:23:49.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:49.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:49.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:49.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:49.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:49.635 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:49.736 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:49.761 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:49.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:49.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:49.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:49.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:49.992 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:50.014 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:50.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:50.117 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:50.143 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:50.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:50.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:49 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 34/176310 objects degraded (0.019%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:50.244 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:50.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:50.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:49 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 34/176310 objects degraded (0.019%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:50.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:50.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:49 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 34/176310 objects degraded (0.019%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:23:50.403 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:50.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:50.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:50.529 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:50.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:50.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:50.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:50.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:50.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:50.776 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:50.780 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:50.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:50.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:50.904 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:50.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:50.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:50.971 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:23:50.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:50 smithi017 ceph-mon[27571]: pgmap v1077: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 890 op/s; 34/169965 objects degraded (0.020%); 1920/169965 objects misplaced (1.130%); 1.7 MiB/s, 26 objects/s recovering 2024-09-17T16:23:50.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2418413581' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:23:50.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:50 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:23:50.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:50 smithi017 ceph-mon[27571]: 5.3 scrub starts 2024-09-17T16:23:51.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:51.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:51.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:51.134 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:51.215 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:51.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:51.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:50 smithi071 ceph-mon[30002]: pgmap v1077: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 890 op/s; 34/169965 objects degraded (0.020%); 1920/169965 objects misplaced (1.130%); 1.7 MiB/s, 26 objects/s recovering 2024-09-17T16:23:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2418413581' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:23:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:50 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:23:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:50 smithi071 ceph-mon[30002]: 5.3 scrub starts 2024-09-17T16:23:51.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:51.341 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:51.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:50 smithi173 ceph-mon[30530]: pgmap v1077: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 890 op/s; 34/169965 objects degraded (0.020%); 1920/169965 objects misplaced (1.130%); 1.7 MiB/s, 26 objects/s recovering 2024-09-17T16:23:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2418413581' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:23:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:50 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:23:51.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:50 smithi173 ceph-mon[30530]: 5.3 scrub starts 2024-09-17T16:23:51.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:51.466 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:51.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:51.569 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:51.593 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:51.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:51.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:51.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:51.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:51.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:51.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:51.844 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:51.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:51.947 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:51.973 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:51.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:52.068 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:23:52.068 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:23:52.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:23:52.068 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:23:52.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.24550.0:152973 2.19 2:9e1f786b:::benchmark_data_smithi173_55121_object76484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 598.19793874899995, 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59801266799999997, 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.24550", 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1408758765", 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 152973 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.069 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867037+0000", 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867039+0000", 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.581e-06 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867044+0000", 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0089999999999998e-06 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867057+0000", 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3658999999999999e-05 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867064+0000", 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2520000000000002e-06 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867120+0000", 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5937999999999999e-05 2024-09-17T16:23:52.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867194+0000", 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3467000000000007e-05 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,6", 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.867254+0000", 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0510000000000002e-05 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.869276+0000", 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002021882 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:53.869436+0000", 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000159214 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.464981+0000", 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59554505000000002 2024-09-17T16:23:52.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.465020+0000", 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9674e-05 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:13:54.465050+0000", 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9431999999999999e-05 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(3.7 240/239 [PushReplyOp(3:e198b3f2:::benchmark_data_smithi173_66272_object478:head)])", 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 517.05182245499998, 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43133256199999997, 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:23:52.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013153+0000", 2024-09-17T16:23:52.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013155+0000", 2024-09-17T16:23:52.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.013161+0000", 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6830000000000002e-06 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441372+0000", 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42821048099999998 2024-09-17T16:23:52.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441385+0000", 2024-09-17T16:23:52.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.311e-05 2024-09-17T16:23:52.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.441470+0000", 2024-09-17T16:23:52.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5185000000000005e-05 2024-09-17T16:23:52.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:15.444486+0000", 2024-09-17T16:23:52.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030157669999999999 2024-09-17T16:23:52.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.080 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.080 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124248 3.7 3:e0068a72:::benchmark_data_smithi173_66272_object52292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 423.17023516299997, 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46227925399999997, 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124248 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.081 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894742+0000", 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.392e-06 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894747+0000", 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4979999999999997e-06 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894763+0000", 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5095e-05 2024-09-17T16:23:52.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894769+0000", 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7360000000000004e-06 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894818+0000", 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8551000000000003e-05 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894928+0000", 2024-09-17T16:23:52.085 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:51 smithi017 ceph-mon[27571]: 5.3 scrub ok 2024-09-17T16:23:52.085 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:51 smithi017 ceph-mon[27571]: 5.4 scrub starts 2024-09-17T16:23:52.085 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:51 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:52.085 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:51 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:23:52.085 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:51 smithi017 ceph-mon[27571]: osdmap e488: 12 total, 12 up, 10 in 2024-09-17T16:23:52.085 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:51 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:23:52.085 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:51 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:23:52.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110536 2024-09-17T16:23:52.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:23:52.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.895033+0000", 2024-09-17T16:23:52.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104589 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.900743+0000", 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057098299999999999 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.901009+0000", 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000265927 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.927407+0000", 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.026398688 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.927419+0000", 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1154e-05 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:49.356994+0000", 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.429575227 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:49.357020+0000", 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6030999999999999e-05 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124248 3.7 3:e0068a72:::benchmark_data_smithi173_66272_object52292:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 421.68845445400001, 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54499606199999995, 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.089 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124248 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376523+0000", 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.42e-06 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376528+0000", 2024-09-17T16:23:52.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7200000000000003e-06 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376539+0000", 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.022e-05 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376546+0000", 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8390000000000007e-06 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376598+0000", 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1953000000000002e-05 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376611+0000", 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2320000000000001e-05 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.921322+0000", 2024-09-17T16:23:52.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54471168999999997 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.921517+0000", 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019489999999999999 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_scan(digest 3.7 MIN-3:ec3efd19:::benchmark_data_smithi173_66272_object4297:head e 290/273)", 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:23:52.094 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 391.02481805399998, 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.427333666, 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040160+0000", 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0660000000000002e-06 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040256+0000", 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5756999999999998e-05 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467140+0000", 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42688425000000002 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467149+0000", 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2289999999999992e-06 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467244+0000", 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709000000000004e-05 2024-09-17T16:23:52.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467260+0000", 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6232e-05 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467491+0000", 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000231423 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.098 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDeleteReply(3.c e297,267 3:350956dd:::benchmark_data_smithi173_66272_object11629:head,259'5194)", 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 383.79796181500001, 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.990895003, 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "queued for pg", 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:23:52.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267015+0000", 2024-09-17T16:23:52.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0500000000000003e-07 2024-09-17T16:23:52.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267021+0000", 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2550000000000003e-06 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257793+0000", 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.99077237299999998 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257810+0000", 2024-09-17T16:23:52.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7005999999999999e-05 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257909+0000", 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8664000000000003e-05 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:102161 4.9 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:23:52.102 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 280.55633119200002, 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.902645841, 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 102161 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8700000000000005e-07 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508680+0000", 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4382999999999999e-05 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508688+0000", 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5059999999999995e-06 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508699+0000", 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0781000000000001e-05 2024-09-17T16:23:52.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508757+0000", 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8026999999999999e-05 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508920+0000", 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163308 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.509034+0000", 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113961 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.512221+0000", 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003186805 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.515116+0000", 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002895299 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411212+0000", 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89609538399999999 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411258+0000", 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6217999999999997e-05 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411290+0000", 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2481999999999998e-05 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:184107 4.9 4:98a0eb83:::benchmark_data_smithi173_77474_object81932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:53.719117+0000", 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 178.34585907900001, 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.73240478200000003, 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.109 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 184107 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719117+0000", 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719117+0000", 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719120+0000", 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.88e-06 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719133+0000", 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.312e-05 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719166+0000", 2024-09-17T16:23:52.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3772999999999997e-05 2024-09-17T16:23:52.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719183+0000", 2024-09-17T16:23:52.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6668999999999999e-05 2024-09-17T16:23:52.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719313+0000", 2024-09-17T16:23:52.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129789 2024-09-17T16:23:52.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719438+0000", 2024-09-17T16:23:52.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012514700000000001 2024-09-17T16:23:52.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-17T16:23:52.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719532+0000", 2024-09-17T16:23:52.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4146000000000003e-05 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723223+0000", 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036906310000000002 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723276+0000", 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2994000000000001e-05 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:52.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723329+0000", 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2828000000000003e-05 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723485+0000", 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015637499999999999 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:54.451313+0000", 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.72782798500000001 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:54.451383+0000", 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0465999999999999e-05 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:54.451521+0000", 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013797899999999999 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:185164 4.f 4:f8df73f0:::benchmark_data_smithi173_77474_object82989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:56.000100+0000", 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 176.06487553599999, 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.634535988, 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 185164 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000100+0000", 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000100+0000", 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000102+0000", 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.404e-06 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000109+0000", 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9619999999999996e-06 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000119+0000", 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0558e-05 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000124+0000", 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2170000000000002e-06 2024-09-17T16:23:52.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000166+0000", 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2134e-05 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000270+0000", 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104005 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,9,11", 2024-09-17T16:23:52.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000379+0000", 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010812 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.004004+0000", 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036249329999999999 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.004131+0000", 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000127336 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.020238+0000", 2024-09-17T16:23:52.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016107305999999998 2024-09-17T16:23:52.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.634511+0000", 2024-09-17T16:23:52.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61427235199999997 2024-09-17T16:23:52.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.634550+0000", 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9057000000000002e-05 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.634636+0000", 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6604e-05 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.124 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:186293 4.f 4:ffe1d3e0:::benchmark_data_smithi173_77474_object84118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:58.185546+0000", 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 173.87942962299999, 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42855515999999999, 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:23:52.125 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186293 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185546+0000", 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185546+0000", 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185547+0000", 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.359e-06 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185553+0000", 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4349999999999996e-06 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185565+0000", 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2155e-05 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185571+0000", 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6509999999999998e-06 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185619+0000", 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8696999999999998e-05 2024-09-17T16:23:52.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185723+0000", 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010333 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,9,11", 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185816+0000", 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3335e-05 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.189433+0000", 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036167840000000001 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.190346+0000", 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091320500000000005 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.190429+0000", 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3014000000000002e-05 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.613994+0000", 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42356446800000003 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.614037+0000", 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3027e-05 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.614101+0000", 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4700000000000001e-05 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204335 4.9 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 148.45828444700001, 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77672725300000001, 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.132 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204335 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606692+0000", 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2600000000000002e-07 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606696+0000", 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7170000000000002e-06 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606707+0000", 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1486e-05 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606712+0000", 2024-09-17T16:23:52.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1560000000000003e-06 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606769+0000", 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6836000000000003e-05 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606856+0000", 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6749999999999994e-05 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-17T16:23:52.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606942+0000", 2024-09-17T16:23:52.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6001999999999994e-05 2024-09-17T16:23:52.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611141+0000", 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041987810000000004 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611206+0000", 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5581999999999997e-05 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611481+0000", 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027492400000000001 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611502+0000", 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0971999999999998e-05 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383271+0000", 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77176877600000005 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383334+0000", 2024-09-17T16:23:52.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2928000000000005e-05 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383419+0000", 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4617000000000007e-05 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.143 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204343 4.0 4:0b58e0a7:::benchmark_data_smithi173_77474_object102168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.621137+0000", 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 148.44383830500001, 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49750966600000002, 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204343 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621137+0000", 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621137+0000", 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621139+0000", 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.635e-06 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621144+0000", 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3170000000000001e-06 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621157+0000", 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2123999999999999e-05 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621164+0000", 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1679999999999998e-06 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621222+0000", 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8439000000000001e-05 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621307+0000", 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4940999999999999e-05 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621418+0000", 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110548 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626056+0000", 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046379489999999997 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095492+0000", 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46943599899999999 2024-09-17T16:23:52.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118566+0000", 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023074935000000001 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118609+0000", 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2106e-05 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118647+0000", 2024-09-17T16:23:52.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8504999999999998e-05 2024-09-17T16:23:52.150 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.150 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.150 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.150 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:204343 4.0 e408/389 ondisk, result = 0)", 2024-09-17T16:23:52.150 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.625102+0000", 2024-09-17T16:23:52.150 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 148.43987388400001, 2024-09-17T16:23:52.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:52.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.493551564, 2024-09-17T16:23:52.152 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.152 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.152 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:52.152 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625102+0000", 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625102+0000", 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625103+0000", 2024-09-17T16:23:52.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0100000000000004e-07 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625111+0000", 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9659999999999996e-06 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118509+0000", 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49339840800000001 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118514+0000", 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5589999999999996e-06 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118557+0000", 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2222000000000003e-05 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118563+0000", 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2689999999999996e-06 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118653+0000", 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439000000000006e-05 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25516.0:204346 4.4 e408/306 4:22db2208:::benchmark_data_smithi173_77474_object102171:head v 408'12784, mlcod=408'12781)", 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.625376+0000", 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 148.43959952, 2024-09-17T16:23:52.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.67400101199999995, 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625376+0000", 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625376+0000", 2024-09-17T16:23:52.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625377+0000", 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2699999999999998e-07 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625386+0000", 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3260000000000002e-06 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295220+0000", 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.66983389199999999 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295230+0000", 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3090000000000001e-06 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295298+0000", 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8583000000000004e-05 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295337+0000", 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8992000000000002e-05 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299315+0000", 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039779949999999998 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299377+0000", 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2088000000000006e-05 2024-09-17T16:23:52.160 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:204343 4.0 e408/389 ondisk, result = 0)", 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.626258+0000", 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 148.438717873, 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46924642799999999, 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626258+0000", 2024-09-17T16:23:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626258+0000", 2024-09-17T16:23:52.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626259+0000", 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0100000000000004e-07 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626262+0000", 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5300000000000001e-06 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095382+0000", 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46911975700000003 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095391+0000", 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4010000000000003e-06 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095466+0000", 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5087999999999997e-05 2024-09-17T16:23:52.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095481+0000", 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4252e-05 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095504+0000", 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3699000000000001e-05 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(5.16 450/444 [PushOp(5:6f5bd1ca:::benchmark_data_smithi173_90735_object25509:head, version: 435'1582, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:6f5bd1ca:::benchmark_data_smithi173_90735_object25509:head@435'1582, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:39.916010+0000", 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.148965606000004, 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29690697999999999, 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.916010+0000", 2024-09-17T16:23:52.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.916010+0000", 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.916012+0000", 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6509999999999999e-06 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.916060+0000", 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8584000000000001e-05 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.212082+0000", 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29602187099999999 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.212094+0000", 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.181e-05 2024-09-17T16:23:52.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.212192+0000", 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8441000000000002e-05 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.212202+0000", 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7680000000000003e-06 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.212917+0000", 2024-09-17T16:23:52.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00071485500000000003 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(5.16 470/455 [PushReplyOp(5:6e1c4d97:::benchmark_data_smithi173_90735_object50923:head)])", 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:10.891412+0000", 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.173564042999999, 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.40973801100000001, 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:10.891412+0000", 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:10.891412+0000", 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:10.891412+0000", 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3900000000000004e-07 2024-09-17T16:23:52.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:10.891417+0000", 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3590000000000001e-06 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:11.300664+0000", 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.40924686399999999 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:11.300669+0000", 2024-09-17T16:23:52.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.215e-06 2024-09-17T16:23:52.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:11.300732+0000", 2024-09-17T16:23:52.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3565000000000006e-05 2024-09-17T16:23:52.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:11.301150+0000", 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00041736900000000001 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:77257 5.17 5:ee127e34:::benchmark_data_smithi173_90735_object77256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:13.811418+0000", 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 38.253557755999999, 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26312511300000002, 2024-09-17T16:23:52.174 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.175 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.175 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:52.175 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.175 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:23:52.175 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:52.175 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 77257 2024-09-17T16:23:52.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.175 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.811418+0000", 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.811418+0000", 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.811419+0000", 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4799999999999998e-07 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.811493+0000", 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4027000000000001e-05 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.811501+0000", 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.67e-06 2024-09-17T16:23:52.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.811505+0000", 2024-09-17T16:23:52.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0600000000000001e-06 2024-09-17T16:23:52.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.811538+0000", 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2564000000000002e-05 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.811612+0000", 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4188000000000002e-05 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,11", 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.811692+0000", 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9691999999999995e-05 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.812443+0000", 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075136300000000001 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:13.813060+0000", 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061667000000000004 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:14.074432+0000", 2024-09-17T16:23:52.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26137216200000002 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:14.074509+0000", 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.682e-05 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:14.074543+0000", 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4248999999999997e-05 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:83695 5.14 5:2f856cc4:::benchmark_data_smithi173_90735_object924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:21.705024+0000", 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.359951970000001, 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21715516900000001, 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 83695 2024-09-17T16:23:52.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.705024+0000", 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.705024+0000", 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.705026+0000", 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.812e-06 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.705033+0000", 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1829999999999997e-06 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.705050+0000", 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7631000000000001e-05 2024-09-17T16:23:52.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.705059+0000", 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.884e-06 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.705123+0000", 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3555999999999999e-05 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.705251+0000", 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012771899999999999 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,10", 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.705360+0000", 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000109715 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.706878+0000", 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015181369999999999 2024-09-17T16:23:52.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.707070+0000", 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019149299999999999 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.922100+0000", 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21502979799999999 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:23:52.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.922146+0000", 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6150999999999999e-05 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.922179+0000", 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3090000000000003e-05 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.188 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26158.0:83695 5.14 e476/449 ondisk, result = 0)", 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:21.706890+0000", 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.35808609, 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.215294126, 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.706890+0000", 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.706890+0000", 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.706891+0000", 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.706895+0000", 2024-09-17T16:23:52.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.797e-06 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.922042+0000", 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21514667900000001 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.922049+0000", 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.584e-06 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.922087+0000", 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8019000000000003e-05 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.922094+0000", 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3669999999999999e-06 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:21.922184+0000", 2024-09-17T16:23:52.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9809999999999996e-05 2024-09-17T16:23:52.193 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.193 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.193 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.193 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:52.193 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:52.193 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:23:52.218 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:52.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:51 smithi071 ceph-mon[30002]: 5.3 scrub ok 2024-09-17T16:23:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:51 smithi071 ceph-mon[30002]: 5.4 scrub starts 2024-09-17T16:23:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:51 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:51 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:23:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:51 smithi071 ceph-mon[30002]: osdmap e488: 12 total, 12 up, 10 in 2024-09-17T16:23:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:51 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:23:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:51 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:23:52.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:52.346 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:52.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:51 smithi173 ceph-mon[30530]: 5.3 scrub ok 2024-09-17T16:23:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:51 smithi173 ceph-mon[30530]: 5.4 scrub starts 2024-09-17T16:23:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:51 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:51 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:23:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:51 smithi173 ceph-mon[30530]: osdmap e488: 12 total, 12 up, 10 in 2024-09-17T16:23:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:51 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:23:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:51 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:23:52.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:52.471 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:52.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:52.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:52.598 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:52.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:52.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:52.723 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:52.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:52.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:52.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:52.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:52.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:52.951 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:52.976 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:52.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:53.078 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:53.102 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:53.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:53.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:53.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:53.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:53.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:52 smithi017 ceph-mon[27571]: 5.18 scrub starts 2024-09-17T16:23:53.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:52 smithi017 ceph-mon[27571]: pgmap v1079: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 935 op/s; 34/164403 objects degraded (0.021%); 1920/164403 objects misplaced (1.168%); 1.7 MiB/s, 26 objects/s recovering 2024-09-17T16:23:53.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:52 smithi017 ceph-mon[27571]: osdmap e489: 12 total, 12 up, 10 in 2024-09-17T16:23:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:52 smithi071 ceph-mon[30002]: 5.18 scrub starts 2024-09-17T16:23:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:52 smithi071 ceph-mon[30002]: pgmap v1079: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 935 op/s; 34/164403 objects degraded (0.021%); 1920/164403 objects misplaced (1.168%); 1.7 MiB/s, 26 objects/s recovering 2024-09-17T16:23:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:52 smithi071 ceph-mon[30002]: osdmap e489: 12 total, 12 up, 10 in 2024-09-17T16:23:53.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:53.358 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:53.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:53.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:52 smithi173 ceph-mon[30530]: 5.18 scrub starts 2024-09-17T16:23:53.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:52 smithi173 ceph-mon[30530]: pgmap v1079: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 935 op/s; 34/164403 objects degraded (0.021%); 1920/164403 objects misplaced (1.168%); 1.7 MiB/s, 26 objects/s recovering 2024-09-17T16:23:53.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:52 smithi173 ceph-mon[30530]: osdmap e489: 12 total, 12 up, 10 in 2024-09-17T16:23:53.460 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:53.482 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:23:53.485 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:53.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:53.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:53.611 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:53.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:53.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:53.737 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:53.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:53.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:53.839 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:53.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:53.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:53.964 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:53.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:53.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:54.093 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:54.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:23:54.114 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:54.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:54.193 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:54.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:54.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:54.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:54.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:53 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:23:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:53 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:23:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:53 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:23:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:53 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:23:54.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:54.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:54.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:54.384 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:23:54.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:53 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:23:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:53 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:23:54.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:54.492 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:54.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:54.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:54.594 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:54.622 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:54.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:54.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:54.725 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:54.754 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:54.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:54.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:54.884 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:54.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:54.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:23:54.986 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:54.988 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:23:54.988 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:23:54.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.989 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:115245 5.f 5:ffe5a3b7:::benchmark_data_smithi173_90735_object32474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-09-17T16:23:54.989 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:54.968644+0000", 2024-09-17T16:23:54.989 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.015912803, 2024-09-17T16:23:54.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.015942713000000001, 2024-09-17T16:23:54.990 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:23:54.990 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:23:54.990 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:23:54.990 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:23:54.990 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:23:54.990 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:54.991 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 115245 2024-09-17T16:23:54.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.991 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:23:54.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:23:54.992 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968644+0000", 2024-09-17T16:23:54.992 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:23:54.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.992 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.992 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:23:54.993 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968644+0000", 2024-09-17T16:23:54.993 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:23:54.993 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.993 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.993 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:23:54.993 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968645+0000", 2024-09-17T16:23:54.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5540000000000001e-06 2024-09-17T16:23:54.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.994 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.994 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:23:54.994 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968651+0000", 2024-09-17T16:23:54.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6690000000000003e-06 2024-09-17T16:23:54.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.994 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.994 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:23:54.995 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968668+0000", 2024-09-17T16:23:54.995 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6433999999999999e-05 2024-09-17T16:23:54.995 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.995 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.995 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:23:54.995 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968676+0000", 2024-09-17T16:23:54.995 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4859999999999997e-06 2024-09-17T16:23:54.995 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.996 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.996 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:23:54.996 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968734+0000", 2024-09-17T16:23:54.996 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7942999999999997e-05 2024-09-17T16:23:54.996 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.996 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.996 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:23:54.996 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968835+0000", 2024-09-17T16:23:54.997 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100521 2024-09-17T16:23:54.997 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.997 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.997 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,11", 2024-09-17T16:23:54.997 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968928+0000", 2024-09-17T16:23:54.997 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3105999999999999e-05 2024-09-17T16:23:54.998 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.998 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.998 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:23:54.998 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.972157+0000", 2024-09-17T16:23:54.998 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0032294950000000002 2024-09-17T16:23:54.998 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:23:54.998 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:23:54.999 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:23:54.999 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:54.999 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:54.999 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:115248 5.1 5:82dec3e9:::benchmark_data_smithi173_90735_object32477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-09-17T16:23:54.999 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:54.968793+0000", 2024-09-17T16:23:54.999 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.015764033, 2024-09-17T16:23:55.000 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.015871336999999999, 2024-09-17T16:23:55.000 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:23:55.000 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:23:55.000 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:23:55.000 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:23:55.000 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:23:55.001 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:55.001 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 115248 2024-09-17T16:23:55.001 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.001 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:23:55.001 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.002 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:23:55.002 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968793+0000", 2024-09-17T16:23:55.002 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:23:55.002 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.002 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.002 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:23:55.002 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968793+0000", 2024-09-17T16:23:55.002 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:23:55.002 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.002 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968794+0000", 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968801+0000", 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2150000000000004e-06 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.003 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968810+0000", 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1400000000000006e-06 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968817+0000", 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7390000000000002e-06 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968887+0000", 2024-09-17T16:23:55.004 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0220000000000002e-05 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.968988+0000", 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010121800000000001 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.969084+0000", 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5657999999999998e-05 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.005 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.970893+0000", 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018089549999999999 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.971776+0000", 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00088365099999999995 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:115272 5.f 5:f98a74f3:::benchmark_data_smithi173_90735_object32501:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-09-17T16:23:55.006 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:54.982082+0000", 2024-09-17T16:23:55.007 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0024748209999999999, 2024-09-17T16:23:55.007 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0026372779999999998, 2024-09-17T16:23:55.007 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:23:55.007 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:23:55.007 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:23:55.007 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:23:55.007 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:23:55.008 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:23:55.008 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 115272 2024-09-17T16:23:55.008 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.008 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.982082+0000", 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.982082+0000", 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.009 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.982083+0000", 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.082e-06 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.982087+0000", 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2220000000000004e-06 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.982098+0000", 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0549e-05 2024-09-17T16:23:55.010 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.982103+0000", 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9790000000000001e-06 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.982155+0000", 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2466999999999998e-05 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.011 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.012 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:23:55.012 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.982234+0000", 2024-09-17T16:23:55.012 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8506999999999999e-05 2024-09-17T16:23:55.012 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:23:55.012 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:23:55.012 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,11", 2024-09-17T16:23:55.012 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.982324+0000", 2024-09-17T16:23:55.012 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9957999999999995e-05 2024-09-17T16:23:55.013 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:23:55.013 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:23:55.013 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:23:55.013 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:23:55.013 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:23:55.014 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 3 2024-09-17T16:23:55.014 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:23:55.017 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:55.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:55.119 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:55.143 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:55.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:55.228 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:54 smithi017 ceph-mon[27571]: pgmap v1081: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 791 op/s; 34/160632 objects degraded (0.021%); 1920/160632 objects misplaced (1.195%); 1000 KiB/s, 15 objects/s recovering 2024-09-17T16:23:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:54 smithi017 ceph-mon[27571]: osdmap e490: 12 total, 12 up, 10 in 2024-09-17T16:23:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:54 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 34/160632 objects degraded (0.021%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:23:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:54 smithi017 ceph-mon[27571]: osdmap e491: 12 total, 12 up, 10 in 2024-09-17T16:23:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3618119213' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:55.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:55.266 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:55.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:54 smithi071 ceph-mon[30002]: pgmap v1081: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 791 op/s; 34/160632 objects degraded (0.021%); 1920/160632 objects misplaced (1.195%); 1000 KiB/s, 15 objects/s recovering 2024-09-17T16:23:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:54 smithi071 ceph-mon[30002]: osdmap e490: 12 total, 12 up, 10 in 2024-09-17T16:23:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:54 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 34/160632 objects degraded (0.021%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:23:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:54 smithi071 ceph-mon[30002]: osdmap e491: 12 total, 12 up, 10 in 2024-09-17T16:23:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3618119213' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:55.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:55.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:55.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:55.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:54 smithi173 ceph-mon[30530]: pgmap v1081: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 25 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 791 op/s; 34/160632 objects degraded (0.021%); 1920/160632 objects misplaced (1.195%); 1000 KiB/s, 15 objects/s recovering 2024-09-17T16:23:55.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:54 smithi173 ceph-mon[30530]: osdmap e490: 12 total, 12 up, 10 in 2024-09-17T16:23:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:54 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 34/160632 objects degraded (0.021%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:23:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:54 smithi173 ceph-mon[30530]: osdmap e491: 12 total, 12 up, 10 in 2024-09-17T16:23:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3618119213' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:23:55.494 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:55.519 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:55.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:55.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:55.622 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:23:55.622 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:23:55.643 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:23:55.643 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:23:55.644 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:55.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:55.671 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:23:55.672 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:23:55.696 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:23:55.696 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:23:55.720 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:23:55.720 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:23:55.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:23:55.744 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:23:55.746 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:55.750 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:23:55.751 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:23:55.751 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T16:23:55.751 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:23:55.770 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:55.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:55.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:23:55.872 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:55.899 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:55.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:56.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:56.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:56.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:56.054 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:23:56.105 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:23:56.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:56.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:56.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:56.256 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:56.279 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:56.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:56.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:56.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:56.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:56.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:56.534 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:56.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:56.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:56.619 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:23:56.619 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:23:56.636 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:56.663 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:56.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:56.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:56.783 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:23:56.783 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:23:56.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:56.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:56.812 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:23:56.812 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:23:56.842 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:23:56.843 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:23:56.862 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:23:56.863 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:23:56.863 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 492, 2024-09-17T16:23:56.863 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:23:56.863 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:23:56.864 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:23:55.988420+0000", 2024-09-17T16:23:56.864 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:23:56.864 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:22:45.057702+0000", 2024-09-17T16:23:56.864 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:23:56.864 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5801984, 2024-09-17T16:23:56.864 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:23:56.864 INFO:teuthology.orchestra.run.smithi017.stdout: "noscrub", 2024-09-17T16:23:56.864 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:23:56.864 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:23:56.864 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 5, 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-17T16:23:56.865 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:23:56.866 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:23:56.867 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.868 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "446", 2024-09-17T16:23:56.869 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:23:56.869 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:23:56.869 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:23:56.869 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:23:56.869 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:23:56.869 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:23:56.869 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:23:56.869 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:23:56.870 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:23:56.871 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.872 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 10, 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 10, 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.30000001192092896, 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.7735099196434021, 2024-09-17T16:23:56.873 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.76999998092651367 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 5, 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_3", 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:21:40.832300+0000", 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:23:56.874 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 26, 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 26, 2024-09-17T16:23:56.875 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 26, 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 26, 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 26, 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.876 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "443", 2024-09-17T16:23:56.877 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:23:56.877 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:23:56.877 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "441", 2024-09-17T16:23:56.877 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:23:56.877 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:23:56.877 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:23:56.877 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:23:56.877 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:23:56.878 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:23:56.879 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:23:56.880 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:23:56.880 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:23:56.880 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:23:56.880 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:23:56.880 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:23:56.880 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:23:56.880 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:23:56.880 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:23:56.880 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 1.9199999570846558, 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 1.9199999570846558, 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1, 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 1.9199999570846558, 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 1.9199999570846558, 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.7735099196434021, 2024-09-17T16:23:56.881 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.76999998092651367 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:23:56.882 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 469, 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.885 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.885 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:23:56.885 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:23:56.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.886 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.887 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:23:56.887 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:23:56.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.887 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.887 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:23:56.887 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:23:56.887 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.887 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:23:56.888 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 0, 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:23:56.890 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 301, 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.891 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:23:56.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.893 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.894 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:23:56.895 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.896 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.896 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:23:56.896 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:23:56.896 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:23:56.896 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:23:56.896 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.896 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.896 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.896 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.897 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 469, 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.898 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:23:56.899 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:23:56.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.900 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.900 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:23:56.900 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:23:56.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.900 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.900 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.900 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.901 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:23:56.902 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 472, 2024-09-17T16:23:56.904 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:23:56.905 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.907 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:23:56.907 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:23:56.907 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.907 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.907 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.907 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.907 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.907 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:23:56.908 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:23:56.908 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:23:56.909 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:23:56.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.909 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.909 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:23:56.909 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:23:56.909 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.910 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.910 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.910 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.910 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.910 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:23:56.910 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:23:56.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.911 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.912 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 481, 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.914 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.914 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:23:56.914 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:23:56.914 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.914 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.914 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.914 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.914 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.915 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.916 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.917 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.918 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 0, 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.919 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 454, 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.920 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:23:56.921 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.922 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:23:56.925 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.66790771484375, 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 466, 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.926 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:23:56.928 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.929 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:23:56.931 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:23:56.932 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.8061676025390625, 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 482, 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.933 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:23:56.935 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:23:56.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.937 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.939 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 465, 2024-09-17T16:23:56.940 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.941 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:23:56.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:23:56.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.2821197509765625, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.947 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:23:56.948 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:23:56.950 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.951 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:23:56.952 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.953 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 462, 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.954 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.955 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.956 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.957 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:23:56.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:23:56.959 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.588836669921875, 2024-09-17T16:23:56.960 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 291, 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.961 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.963 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.964 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:23:56.965 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.966 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:23:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:23:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:23:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:23:56.967 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:22:42.842035+0000", 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 65536, 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:25.058618+0000", 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:51.474083+0000", 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.969 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:10.344304+0000", 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.970 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:53.204639+0000", 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.971 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 55422, 2024-09-17T16:23:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:08.375353+0000", 2024-09-17T16:23:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:23:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.972 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:55.626986+0000", 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.973 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:23:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:37.531895+0000", 2024-09-17T16:23:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:23:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:23:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.974 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:23:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:10.867524+0000", 2024-09-17T16:23:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:23:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.975 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.975 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:23:56.976 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:23:56.976 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.976 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.976 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:23:56.976 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:30.923568+0000", 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:22:47.836398+0000", 2024-09-17T16:23:56.977 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:02.282366+0000", 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:23:56.978 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [ 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "5.5", 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: 10, 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: 4, 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: 3 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:23:56.979 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [ 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "5.12", 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: 8, 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: 3 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:23:56.980 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:23:56.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:23:56.982 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:23:56.983 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:23:56.984 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:23:56.984 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:23:56.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:57.001 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:23:57.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:57.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:57.115 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:57.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:56 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:23:57.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:56 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:23:57.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:56 smithi017 ceph-mon[27571]: pgmap v1084: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 869 op/s; 34/153921 objects degraded (0.022%); 1519/153921 objects misplaced (0.987%); 1.7 MiB/s, 27 objects/s recovering 2024-09-17T16:23:57.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:56 smithi017 ceph-mon[27571]: osdmap e492: 12 total, 12 up, 10 in 2024-09-17T16:23:57.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1269504631' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:57.120 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2330757896' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:23:57.143 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:57.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:57.245 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:23:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:57.274 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:57.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:23:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:56 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:23:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:56 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:23:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:56 smithi071 ceph-mon[30002]: pgmap v1084: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 869 op/s; 34/153921 objects degraded (0.022%); 1519/153921 objects misplaced (0.987%); 1.7 MiB/s, 27 objects/s recovering 2024-09-17T16:23:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:56 smithi071 ceph-mon[30002]: osdmap e492: 12 total, 12 up, 10 in 2024-09-17T16:23:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1269504631' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2330757896' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:23:57.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:56 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:23:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:56 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:23:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:56 smithi173 ceph-mon[30530]: pgmap v1084: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 869 op/s; 34/153921 objects degraded (0.022%); 1519/153921 objects misplaced (0.987%); 1.7 MiB/s, 27 objects/s recovering 2024-09-17T16:23:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:56 smithi173 ceph-mon[30530]: osdmap e492: 12 total, 12 up, 10 in 2024-09-17T16:23:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1269504631' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:23:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2330757896' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:23:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:57.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:57.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:57.441 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:23:57.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:23:57.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:57.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:23:57.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:57.657 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:57.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:57.751 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:23:57.759 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:57.789 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:57.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:57.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:23:57.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:57.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:23:58.019 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:58.045 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:58.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:58.098 INFO:tasks.thrashosds.thrasher:j is {'epoch': 492, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:23:55.988420+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:22:45.057702+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '446', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.30000001192092896, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.7735099196434021, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.7699999809265137}}, {'pool': 5, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-17T16:21:40.832300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '443', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '441', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9199999570846558, 'score_stable': 1.9199999570846558, 'optimal_score': 1, 'raw_score_acting': 1.9199999570846558, 'raw_score_stable': 1.9199999570846558, 'primary_affinity_weighted': 0.7735099196434021, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.7699999809265137}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 469, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 301, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 469, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 472, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 481, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 454, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 0.66790771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 466, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 0.8061676025390625, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 482, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 465, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0.2821197509765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 462, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 0.588836669921875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:22:42.842035+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-17T16:23:25.058618+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:51.474083+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:10.344304+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:53.204639+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 55422, 'last_purged_snaps_scrub': '2024-09-17T16:23:08.375353+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:55.626986+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:37.531895+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:10.867524+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:30.923568+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:22:47.836398+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:02.282366+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.5', 'osds': [10, 4, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '5.12', 'osds': [8, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:23:58.098 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-17T16:23:58.147 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:58.172 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:58.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:23:58.182 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:57 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:58.182 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1269504631' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:58.182 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:57 smithi071 ceph-mon[30002]: osdmap e493: 12 total, 12 up, 10 in 2024-09-17T16:23:58.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:58.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:58.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:58.314 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:23:58.314 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:23:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:57 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1269504631' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:57 smithi173 ceph-mon[30530]: osdmap e493: 12 total, 12 up, 10 in 2024-09-17T16:23:58.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:58.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:58.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:57 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:23:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1269504631' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:23:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:57 smithi017 ceph-mon[27571]: osdmap e493: 12 total, 12 up, 10 in 2024-09-17T16:23:58.530 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:23:58.556 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:58.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:23:58.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:23:58.681 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:58.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:23:58.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:23:58.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:58.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:23:58.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:58.934 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:58.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:58.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:23:59.036 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:23:59.084 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:59.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:23:59.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:23:59.214 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:59.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:23:59.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:23:59 smithi017 ceph-mon[27571]: pgmap v1087: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 34/150741 objects degraded (0.023%); 1519/150741 objects misplaced (1.008%); 2.1 MiB/s, 33 objects/s recovering 2024-09-17T16:23:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:23:59 smithi071 ceph-mon[30002]: pgmap v1087: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 34/150741 objects degraded (0.023%); 1519/150741 objects misplaced (1.008%); 2.1 MiB/s, 33 objects/s recovering 2024-09-17T16:23:59.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:59.322 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:23:59.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:59.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:23:59 smithi173 ceph-mon[30530]: pgmap v1087: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 34/150741 objects degraded (0.023%); 1519/150741 objects misplaced (1.008%); 2.1 MiB/s, 33 objects/s recovering 2024-09-17T16:23:59.444 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:23:59.468 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:59.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:23:59.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:59.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:59.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:59.700 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:23:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:59.726 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:23:59.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:23:59.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:23:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:59.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:23:59.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:23:59.868 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:23:59.868 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:23:59.960 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:23:59.984 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:23:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:23:59.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:00.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:00.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:00.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:00.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:00.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:00.239 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:00.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:00 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:24:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:00 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:24:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:00 smithi071 ceph-mon[30002]: osdmap e494: 12 total, 12 up, 10 in 2024-09-17T16:24:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:00 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 34/150741 objects degraded (0.023%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:00 smithi071 ceph-mon[30002]: osdmap e495: 12 total, 12 up, 10 in 2024-09-17T16:24:00.340 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:00.363 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:00.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:00.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:00 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:24:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:00 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:24:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:00 smithi173 ceph-mon[30530]: osdmap e494: 12 total, 12 up, 10 in 2024-09-17T16:24:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:00 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 34/150741 objects degraded (0.023%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:00 smithi173 ceph-mon[30530]: osdmap e495: 12 total, 12 up, 10 in 2024-09-17T16:24:00.465 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:00.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:00 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:24:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:00 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:24:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:00 smithi017 ceph-mon[27571]: osdmap e494: 12 total, 12 up, 10 in 2024-09-17T16:24:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:00 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 34/150741 objects degraded (0.023%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:00 smithi017 ceph-mon[27571]: osdmap e495: 12 total, 12 up, 10 in 2024-09-17T16:24:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:00.490 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:00.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:00.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:24:00.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:00.630 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:00.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:00.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:24:00.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:00.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:00.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:00.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:00.888 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:00.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:00.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:00.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:01.015 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:24:01.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:01.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:01.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:01.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:01.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:01.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:01.278 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:01.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:01 smithi071 ceph-mon[30002]: pgmap v1090: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 853 op/s; 34/143682 objects degraded (0.024%); 1086/143682 objects misplaced (0.756%); 2.2 MiB/s, 35 objects/s recovering 2024-09-17T16:24:01.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:01 smithi017 ceph-mon[27571]: pgmap v1090: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 853 op/s; 34/143682 objects degraded (0.024%); 1086/143682 objects misplaced (0.756%); 2.2 MiB/s, 35 objects/s recovering 2024-09-17T16:24:01.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:01.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:01 smithi173 ceph-mon[30530]: pgmap v1090: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 853 op/s; 34/143682 objects degraded (0.024%); 1086/143682 objects misplaced (0.756%); 2.2 MiB/s, 35 objects/s recovering 2024-09-17T16:24:01.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:01.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:01.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:01.537 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:01.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:01.547 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:01.547 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:01.584 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:24:01.639 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:01.663 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:01.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:01.765 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:01.789 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:01.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:01.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:01.886 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:01.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:01.916 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:01.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:02.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:02.024 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:24:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:02.043 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:02.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:02.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:02.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:02.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:02.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:24:02.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:02.294 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:02.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:02.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:02.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4167384250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4167384250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:02.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:02.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:02.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:02.477 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:24:02.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4167384250' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:02.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:02.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:02.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:02.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:02.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:02.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:02.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:02.783 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:02.811 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:02.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:02.913 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:02.941 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:02.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:02.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:03.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:03.059 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:03.059 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:03.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:03.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:03.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:03.098 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:24:03.099 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:24:03.099 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-09-17T16:24:03.099 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T16:24:03.099 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T16:24:03.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-09-17T16:24:03.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:03.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:03.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:03.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:03 smithi017 ceph-mon[27571]: pgmap v1091: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 966 op/s; 34/136794 objects degraded (0.025%); 1086/136794 objects misplaced (0.794%); 1.5 MiB/s, 24 objects/s recovering 2024-09-17T16:24:03.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:03 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:24:03.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:03 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:24:03.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:03 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:03.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4167384250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:24:03.218 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:03 smithi017 ceph-mon[27571]: osdmap e496: 12 total, 12 up, 10 in 2024-09-17T16:24:03.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:03.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:03 smithi071 ceph-mon[30002]: pgmap v1091: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 966 op/s; 34/136794 objects degraded (0.025%); 1086/136794 objects misplaced (0.794%); 1.5 MiB/s, 24 objects/s recovering 2024-09-17T16:24:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:03 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:24:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:03 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:24:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:03 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4167384250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:24:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:03 smithi071 ceph-mon[30002]: osdmap e496: 12 total, 12 up, 10 in 2024-09-17T16:24:03.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:03.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:03 smithi173 ceph-mon[30530]: pgmap v1091: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 966 op/s; 34/136794 objects degraded (0.025%); 1086/136794 objects misplaced (0.794%); 1.5 MiB/s, 24 objects/s recovering 2024-09-17T16:24:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:03 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:24:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:03 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:24:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:03 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4167384250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:24:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:03 smithi173 ceph-mon[30530]: osdmap e496: 12 total, 12 up, 10 in 2024-09-17T16:24:03.404 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:03.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:03.458 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:03.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:03.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:03.584 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:03.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:03.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:03.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:24:03.750 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:03.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:03.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:03.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:03.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:03.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:03.942 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:24:03.942 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T16:24:03.942 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:24:03.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:03.989 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:24:03.999 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:03.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:03.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:04.027 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3242904917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:04.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:24:04.041 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:24:04.069 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:24:04.069 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:24:04.095 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:24:04.095 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:24:04.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:04.123 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:24:04.123 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:24:04.135 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:04.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3242904917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:04.150 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:24:04.151 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:24:04.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:24:04.177 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:24:04.203 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:24:04.204 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:24:04.229 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:24:04.229 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:24:04.237 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:04.261 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:24:04.261 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:24:04.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:04.265 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:04.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:04.288 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:24:04.289 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:24:04.289 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3242904917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:04.312 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:24:04.312 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:24:04.335 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:24:04.335 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:24:04.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:04.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:04.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:04.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:04.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:04.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:04.564 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.26158.0:123991 5.c e497/419 5:303d2aed:::benchmark_data_smithi173_90735_object41220:head v 497'7721, mlcod=497'7718)", 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:24:04.560185+0000", 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.001977131, 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020029539999999999, 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:04.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:04.560185+0000", 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:04.560185+0000", 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:04.560185+0000", 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1200000000000002e-07 2024-09-17T16:24:04.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:04.560190+0000", 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1229999999999999e-06 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:04.560221+0000", 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0391000000000001e-05 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:04.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:04.560228+0000", 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7259999999999996e-06 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:04.560276+0000", 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7942000000000003e-05 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:04.560293+0000", 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7958e-05 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:04.568 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:24:04.569 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 1 2024-09-17T16:24:04.569 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:24:04.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:04.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:04.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:04.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:04.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:04.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:04.890 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:04.916 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:04.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:04.962 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:24:05.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:05.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:05.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:05.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:05.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:05.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:05 smithi017 ceph-mon[27571]: pgmap v1093: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 956 op/s; 34/133527 objects degraded (0.025%); 1086/133527 objects misplaced (0.813%); 1.5 MiB/s, 23 objects/s recovering 2024-09-17T16:24:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:05 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 34/133527 objects degraded (0.025%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:05 smithi017 ceph-mon[27571]: osdmap e497: 12 total, 12 up, 10 in 2024-09-17T16:24:05.277 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:05.289 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:05 smithi071 ceph-mon[30002]: pgmap v1093: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 956 op/s; 34/133527 objects degraded (0.025%); 1086/133527 objects misplaced (0.813%); 1.5 MiB/s, 23 objects/s recovering 2024-09-17T16:24:05.289 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:05 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 34/133527 objects degraded (0.025%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:05.289 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:05 smithi071 ceph-mon[30002]: osdmap e497: 12 total, 12 up, 10 in 2024-09-17T16:24:05.292 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:05.309 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:05.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:05 smithi173 ceph-mon[30530]: pgmap v1093: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 956 op/s; 34/133527 objects degraded (0.025%); 1086/133527 objects misplaced (0.813%); 1.5 MiB/s, 23 objects/s recovering 2024-09-17T16:24:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:05 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 34/133527 objects degraded (0.025%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:05 smithi173 ceph-mon[30530]: osdmap e497: 12 total, 12 up, 10 in 2024-09-17T16:24:05.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:05.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:05.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:05.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:05.566 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:05.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:05.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:05.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:24:05.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:05.720 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:05.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:05.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:05.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:05.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:05.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:05.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:05.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:05.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:05.979 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:24:06.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:06.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:06.103 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:06.205 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:06.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:06.231 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:06.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:06.333 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:06.356 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:06.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:06.458 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:06.480 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:06.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:06.582 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:06.613 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:06.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:06.716 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:06.739 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:06.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:06.840 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:06.871 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:06.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:06.899 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:24:06.899 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:24:06.899 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:24:06.899 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:24:06.899 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:24:06.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:06.998 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:06.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:07.035 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:24:07.100 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:07.123 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:07.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:07.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:24:07.225 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:07.253 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:07.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:07.288 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:07 smithi071 ceph-mon[30002]: pgmap v1095: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s; 34/126081 objects degraded (0.027%); 619/126081 objects misplaced (0.491%); 1.5 MiB/s, 24 objects/s recovering 2024-09-17T16:24:07.289 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/711310404' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:07.355 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:07.383 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:07.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:07 smithi173 ceph-mon[30530]: pgmap v1095: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s; 34/126081 objects degraded (0.027%); 619/126081 objects misplaced (0.491%); 1.5 MiB/s, 24 objects/s recovering 2024-09-17T16:24:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/711310404' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:07.424 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:07 smithi017 ceph-mon[27571]: pgmap v1095: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.05k op/s; 34/126081 objects degraded (0.027%); 619/126081 objects misplaced (0.491%); 1.5 MiB/s, 24 objects/s recovering 2024-09-17T16:24:07.424 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/711310404' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:07.483 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:07.513 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:07.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:07.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:24:07.615 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:07.640 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:07.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:07.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:07.767 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:07.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:07.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:07.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:07.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:07.931 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:24:07.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:08.022 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:08.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:08.035 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:08.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:08 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:24:08.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/711310404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:08.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:08 smithi071 ceph-mon[30002]: osdmap e498: 12 total, 12 up, 10 in 2024-09-17T16:24:08.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:08.151 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:08.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:08.163 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:08 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:24:08.163 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/711310404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:08.163 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:08 smithi017 ceph-mon[27571]: osdmap e498: 12 total, 12 up, 10 in 2024-09-17T16:24:08.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:08.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:08.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:08.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:08 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:24:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/711310404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:08 smithi173 ceph-mon[30530]: osdmap e498: 12 total, 12 up, 10 in 2024-09-17T16:24:08.409 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:08.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:08.508 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:24:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:24:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:24:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:24:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25036.0:8122 3.2 e220/208 3:4938ef6e:::benchmark_data_smithi173_66272_object8121:head v 220'496, mlcod=220'493)", 2024-09-17T16:24:08.508 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:14:34.934266+0000", 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 573.56965689399999, 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28873317999999998, 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:34.934266+0000", 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.509 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:34.934266+0000", 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:34.934267+0000", 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9900000000000009e-07 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:34.934362+0000", 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5050000000000006e-05 2024-09-17T16:24:08.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222243+0000", 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28788133799999999 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222250+0000", 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0310000000000001e-06 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222296+0000", 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6381999999999997e-05 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222317+0000", 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0401e-05 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222944+0000", 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00062735000000000004 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:14:35.222999+0000", 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4629000000000002e-05 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:76068 4.5 4:a01bffc8:::benchmark_data_smithi173_77474_object76067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:18:40.015930+0000", 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 328.48799242699999, 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39555464400000001, 2024-09-17T16:24:08.513 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 76068 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015930+0000", 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015930+0000", 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015931+0000", 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.44e-06 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015982+0000", 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0076e-05 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015997+0000", 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5531000000000001e-05 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016004+0000", 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6930000000000001e-06 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016063+0000", 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9621999999999999e-05 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016194+0000", 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013074899999999999 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016340+0000", 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000145601 2024-09-17T16:24:08.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.018468+0000", 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021282110000000001 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.018797+0000", 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032921399999999998 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.411385+0000", 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39258796099999999 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.411442+0000", 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6997000000000003e-05 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.411485+0000", 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2549000000000002e-05 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.519 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:102166 4.a 4:583138b7:::benchmark_data_smithi173_77474_object102165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:11.515734+0000", 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 296.98818853199998, 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31708639999999999, 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:24:08.520 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 102166 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515734+0000", 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515734+0000", 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515736+0000", 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.903e-06 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515777+0000", 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1304000000000003e-05 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515791+0000", 2024-09-17T16:24:08.523 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:08.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3648e-05 2024-09-17T16:24:08.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515798+0000", 2024-09-17T16:24:08.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4139999999999997e-06 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515850+0000", 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2157000000000002e-05 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515951+0000", 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100321 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.516061+0000", 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000110636 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.518923+0000", 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028621470000000002 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519104+0000", 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018084500000000001 2024-09-17T16:24:08.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832702+0000", 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31359804299999999 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832773+0000", 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0612999999999996e-05 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832820+0000", 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7369e-05 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:102166 4.a e349/306 ondisk, result = 0)", 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:11.519407+0000", 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 296.98451499499998, 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31341929800000001, 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.528 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519407+0000", 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519407+0000", 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519408+0000", 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5300000000000003e-07 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519417+0000", 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7660000000000004e-06 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832605+0000", 2024-09-17T16:24:08.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31318752500000002 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832629+0000", 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4039000000000001e-05 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832686+0000", 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7722000000000001e-05 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832696+0000", 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3610000000000007e-06 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832827+0000", 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000131132 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.532 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:144445 4.5 4:a72f5a89:::benchmark_data_smithi173_77474_object42270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:59.250222+0000", 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 249.253700116, 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32863163499999998, 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 144445 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250222+0000", 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250222+0000", 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250224+0000", 2024-09-17T16:24:08.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2240000000000001e-06 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250229+0000", 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4720000000000003e-06 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250242+0000", 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2508e-05 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250251+0000", 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0049999999999993e-06 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250311+0000", 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0441e-05 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250442+0000", 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013124299999999999 2024-09-17T16:24:08.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250564+0000", 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012152200000000001 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254897+0000", 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0043328849999999999 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.259207+0000", 2024-09-17T16:24:08.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0043099599999999998 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578783+0000", 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31957687099999998 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578813+0000", 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9745000000000001e-05 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578854+0000", 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0759000000000001e-05 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:144445 4.5 e363/301 ondisk, result = 0)", 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:59.254966+0000", 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 249.24895647100001, 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32389465200000001, 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.539 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254966+0000", 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254966+0000", 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254967+0000", 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6899999999999996e-07 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254972+0000", 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9259999999999999e-06 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578690+0000", 2024-09-17T16:24:08.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32371775800000002 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578694+0000", 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3189999999999996e-06 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578774+0000", 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0020999999999998e-05 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578779+0000", 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0669999999999999e-06 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578861+0000", 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1692000000000003e-05 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:184107 4.9 e389/383 4:98a0eb83:::benchmark_data_smithi173_77474_object81932:head v 389'11321, mlcod=389'11318)", 2024-09-17T16:24:08.543 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:53.719658+0000", 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 194.78426456599999, 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.73134149999999998, 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.719658+0000", 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.719658+0000", 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.719659+0000", 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8400000000000004e-07 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.719663+0000", 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.831e-06 2024-09-17T16:24:08.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.446826+0000", 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.72716291899999996 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.446836+0000", 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3319999999999998e-06 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.446925+0000", 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9407000000000006e-05 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.446953+0000", 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8351e-05 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.450963+0000", 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040093389999999998 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.450999+0000", 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6637000000000002e-05 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:197771 4.1 e401/301 4:8a203159:::benchmark_data_smithi173_77474_object95596:head v 401'12502, mlcod=401'12499)", 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:13.590532+0000", 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 174.91339053900001, 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.325052122, 2024-09-17T16:24:08.548 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590532+0000", 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590532+0000", 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590533+0000", 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5700000000000002e-07 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590540+0000", 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.939e-06 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.911473+0000", 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32093297300000001 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.911477+0000", 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6759999999999998e-06 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.911514+0000", 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7191999999999999e-05 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.911533+0000", 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8754999999999999e-05 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.915546+0000", 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040127770000000004 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.915584+0000", 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8053000000000003e-05 2024-09-17T16:24:08.552 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(4.8 401/383 [PushReplyOp(4:16ec698f:::benchmark_data_smithi173_77474_object98014:head)])", 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:14.329740+0000", 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 174.17418213900001, 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31203499200000001, 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.329740+0000", 2024-09-17T16:24:08.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.329740+0000", 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.329740+0000", 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.31e-07 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.329744+0000", 2024-09-17T16:24:08.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0979999999999998e-06 2024-09-17T16:24:08.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.640369+0000", 2024-09-17T16:24:08.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31062495299999998 2024-09-17T16:24:08.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.640377+0000", 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7800000000000006e-06 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.641583+0000", 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012057770000000001 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.641775+0000", 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000192253 2024-09-17T16:24:08.559 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:325 5.3 5:c6aa875f:::benchmark_data_smithi173_90735_object324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:50.018074+0000", 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 138.485848197, 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21758697599999999, 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.560 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 325 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.018074+0000", 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.018074+0000", 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.018075+0000", 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5300000000000002e-07 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.018293+0000", 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00021800500000000001 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.018305+0000", 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1708e-05 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.018312+0000", 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.063e-06 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.018400+0000", 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8188000000000003e-05 2024-09-17T16:24:08.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.018518+0000", 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011824000000000001 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.018648+0000", 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000130023 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.019196+0000", 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000547643 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.020983+0000", 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017866410000000001 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.235592+0000", 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21460924200000001 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.235630+0000", 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7880999999999999e-05 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:50.235661+0000", 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1389000000000001e-05 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:2268 5.2 5:448d2a5f:::benchmark_data_smithi173_90735_object2267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:51.492843+0000", 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 137.01107967999999, 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26093462899999997, 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 2268 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.567 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492843+0000", 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492843+0000", 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492844+0000", 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2899999999999999e-06 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492888+0000", 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3458999999999998e-05 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492901+0000", 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3258000000000001e-05 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492906+0000", 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4620000000000004e-06 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492949+0000", 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2256000000000003e-05 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.493035+0000", 2024-09-17T16:24:08.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6211999999999994e-05 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.493123+0000", 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7849999999999994e-05 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.493462+0000", 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00033951800000000002 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:08.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.494234+0000", 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00077194099999999995 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.753697+0000", 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25946257700000003 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.753735+0000", 2024-09-17T16:24:08.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8816000000000002e-05 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.753777+0000", 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1990000000000003e-05 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:2278 5.2 5:4f171031:::benchmark_data_smithi173_90735_object2277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:51.500929+0000", 2024-09-17T16:24:08.575 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 137.002993839, 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25564988700000002, 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 2278 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.500929+0000", 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.500929+0000", 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.500930+0000", 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8899999999999998e-07 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.500963+0000", 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3757000000000002e-05 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.500975+0000", 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1919999999999999e-05 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.500980+0000", 2024-09-17T16:24:08.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0660000000000003e-06 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.501019+0000", 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.888e-05 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.501104+0000", 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4745000000000003e-05 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.501186+0000", 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1709e-05 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.501486+0000", 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00029989200000000001 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.501875+0000", 2024-09-17T16:24:08.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000389825 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.756522+0000", 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25464635400000002 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.756553+0000", 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1226999999999998e-05 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.756579+0000", 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5522999999999999e-05 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:54626 5.19 5:9d4a6242:::benchmark_data_smithi173_90735_object54625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:39.758709+0000", 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 88.745213562000004, 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.414626938, 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.582 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 54626 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758709+0000", 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758709+0000", 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758710+0000", 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2389999999999999e-06 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758803+0000", 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3035000000000006e-05 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758813+0000", 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6900000000000004e-06 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758818+0000", 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8080000000000004e-06 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758950+0000", 2024-09-17T16:24:08.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000132719 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.759057+0000", 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000106562 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 5,7,8", 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.759170+0000", 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011260999999999999 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.760775+0000", 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001605605 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.761105+0000", 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032938999999999999 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.763632+0000", 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0025276539999999998 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172496+0000", 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40886421000000001 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172702+0000", 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000206011 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.173336+0000", 2024-09-17T16:24:08.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00063340499999999997 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26158.0:54626 5.19 e450/444 ondisk, result = 0)", 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:39.760626+0000", 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 88.743296037999997, 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.412716058, 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:24:08.589 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.760626+0000", 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.760626+0000", 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.760627+0000", 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9500000000000002e-07 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.760632+0000", 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4689999999999999e-06 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172298+0000", 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41166605099999998 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172309+0000", 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1286e-05 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172438+0000", 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012923200000000001 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172470+0000", 2024-09-17T16:24:08.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1585000000000003e-05 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.173342+0000", 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00087274 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(5.19 450/444 [PushReplyOp(5:99cc975c:::benchmark_data_smithi173_90735_object22010:head)])", 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:39.919313+0000", 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 88.584609365000006, 2024-09-17T16:24:08.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26979225699999998, 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.919313+0000", 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.919313+0000", 2024-09-17T16:24:08.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.919314+0000", 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7999999999999996e-07 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.919317+0000", 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.517e-06 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.188344+0000", 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26902674500000001 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.188349+0000", 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0100000000000003e-06 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.188388+0000", 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9285000000000001e-05 2024-09-17T16:24:08.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.189105+0000", 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00071721999999999997 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:59957 5.11 5:8aea6b7a:::benchmark_data_smithi173_90735_object59956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e458)", 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:50.799372+0000", 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 77.704550181000002, 2024-09-17T16:24:08.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24227959399999999, 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 59957 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.799372+0000", 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.799372+0000", 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.799374+0000", 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5540000000000001e-06 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.799420+0000", 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6511000000000002e-05 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.799434+0000", 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3671e-05 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.799441+0000", 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2459999999999997e-06 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.799493+0000", 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1561000000000003e-05 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.799600+0000", 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010702899999999999 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.799695+0000", 2024-09-17T16:24:08.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5487000000000002e-05 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.800571+0000", 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00087532300000000001 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:50.800878+0000", 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030772800000000001 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:51.041539+0000", 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24066100900000001 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:51.041603+0000", 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3400999999999998e-05 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:51.041652+0000", 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9073999999999998e-05 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.603 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:62813 5.2 5:43e619ba:::benchmark_data_smithi173_90735_object62812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e461)", 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:54.999396+0000", 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 73.504526420999994, 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22538602399999999, 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:08.604 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 62813 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:54.999396+0000", 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:54.999396+0000", 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:54.999398+0000", 2024-09-17T16:24:08.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.455e-06 2024-09-17T16:24:08.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:54.999451+0000", 2024-09-17T16:24:08.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3847999999999997e-05 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:54.999464+0000", 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3106e-05 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:54.999470+0000", 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6529999999999999e-06 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:54.999527+0000", 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6419999999999999e-05 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:54.999676+0000", 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000149529 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:54.999849+0000", 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017248900000000001 2024-09-17T16:24:08.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:55.001180+0000", 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013311180000000001 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:55.001880+0000", 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00069993699999999998 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:55.224708+0000", 2024-09-17T16:24:08.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22282792400000001 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:55.224741+0000", 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3339e-05 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:55.224782+0000", 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1205999999999998e-05 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPull(5.12 e472/455 cost 1049576)", 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:13.988499+0000", 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 54.515423634999998, 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28772111500000003, 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:13.988499+0000", 2024-09-17T16:24:08.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:13.988499+0000", 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:13.988500+0000", 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2309999999999999e-06 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:13.988513+0000", 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2965999999999999e-05 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:14.275869+0000", 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.287355468 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:14.275874+0000", 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9059999999999999e-06 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:14.275915+0000", 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0499e-05 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:14.276220+0000", 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030504499999999999 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.614 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:114350 5.11 5:8df9aac6:::benchmark_data_smithi173_90735_object31579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:54.080092+0000", 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 14.423830145, 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22322766499999999, 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 114350 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.080092+0000", 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.080092+0000", 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.080094+0000", 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9400000000000001e-06 2024-09-17T16:24:08.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.080104+0000", 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8099999999999992e-06 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.080132+0000", 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8209e-05 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.080146+0000", 2024-09-17T16:24:08.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3696999999999999e-05 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.080288+0000", 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014235499999999999 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.080450+0000", 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016200000000000001 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.080561+0000", 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000111098 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.082014+0000", 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014526470000000001 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.082270+0000", 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025553400000000002 2024-09-17T16:24:08.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.303223+0000", 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.220952905 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.303274+0000", 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1251000000000001e-05 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.620 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.303320+0000", 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6218999999999999e-05 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26158.0:114350 5.11 e490/450 ondisk, result = 0)", 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:54.082356+0000", 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 14.421566534, 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22096954499999999, 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:08.621 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.082356+0000", 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.082356+0000", 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:08.622 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.082356+0000", 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3700000000000001e-07 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.082362+0000", 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6960000000000002e-06 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.303172+0000", 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.220809532 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.623 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.303177+0000", 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7420000000000003e-06 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.303212+0000", 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5095000000000002e-05 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.303218+0000", 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.48e-06 2024-09-17T16:24:08.624 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:08.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:08.625 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:08.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.303325+0000", 2024-09-17T16:24:08.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000107563 2024-09-17T16:24:08.625 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.625 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.625 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.625 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:08.625 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:08.625 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:24:08.626 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:08.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:08.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:08.753 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:08.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:08.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:08.880 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:08.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:08.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:24:08.899 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:24:08.925 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:24:08.925 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:24:08.950 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:24:08.950 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:24:08.981 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:08.985 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:24:08.985 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:24:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:09.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:09.015 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:24:09.015 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:24:09.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:24:09.042 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:24:09.065 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:24:09.066 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:24:09.089 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:24:09.089 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:24:09.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:09.115 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:24:09.116 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:24:09.135 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:09.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:09.141 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:24:09.141 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:24:09.172 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:24:09.172 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:24:09.197 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:24:09.197 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:24:09.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:09.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:09.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:09.295 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:09 smithi173 ceph-mon[30530]: pgmap v1097: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 792 op/s; 34/122529 objects degraded (0.028%); 619/122529 objects misplaced (0.505%); 1.6 MiB/s, 26 objects/s recovering 2024-09-17T16:24:09.295 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/772609488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:09.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:09 smithi071 ceph-mon[30002]: pgmap v1097: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 792 op/s; 34/122529 objects degraded (0.028%); 619/122529 objects misplaced (0.505%); 1.6 MiB/s, 26 objects/s recovering 2024-09-17T16:24:09.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/772609488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:09.366 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:09.392 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:09.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:09 smithi017 ceph-mon[27571]: pgmap v1097: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 792 op/s; 34/122529 objects degraded (0.028%); 619/122529 objects misplaced (0.505%); 1.6 MiB/s, 26 objects/s recovering 2024-09-17T16:24:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/772609488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:09.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:09.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:24:09.521 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:09.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:09.623 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:09.647 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:09.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:09.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:09.773 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:09.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:09.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:09.899 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:09.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:24:09.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:09.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:10.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:10.059 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:10.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:10.089 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:24:10.090 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:24:10.090 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T16:24:10.090 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-09-17T16:24:10.090 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-17T16:24:10.161 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:10.192 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:10.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:10.199 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 34/122529 objects degraded (0.028%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:10.199 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:10 smithi173 ceph-mon[30530]: osdmap e499: 12 total, 12 up, 10 in 2024-09-17T16:24:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 34/122529 objects degraded (0.028%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:10 smithi017 ceph-mon[27571]: osdmap e499: 12 total, 12 up, 10 in 2024-09-17T16:24:10.295 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:10.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 34/122529 objects degraded (0.028%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:10 smithi071 ceph-mon[30002]: osdmap e499: 12 total, 12 up, 10 in 2024-09-17T16:24:10.323 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:10.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:10.426 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:10.454 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:10.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:10.556 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:10.578 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:10.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:10.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:10.701 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:10.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:10.804 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:10.825 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:10.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:10.875 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:10.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:10.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:10.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:11.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:11.073 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:11.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:11.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:11 smithi017 ceph-mon[27571]: pgmap v1099: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 22/114279 objects degraded (0.019%); 7.5 MiB/s, 122 objects/s recovering 2024-09-17T16:24:11.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:11 smithi017 ceph-mon[27571]: osdmap e500: 12 total, 12 up, 10 in 2024-09-17T16:24:11.175 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:11.197 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:11.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:11.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:11.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:11 smithi071 ceph-mon[30002]: pgmap v1099: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 22/114279 objects degraded (0.019%); 7.5 MiB/s, 122 objects/s recovering 2024-09-17T16:24:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:11 smithi071 ceph-mon[30002]: osdmap e500: 12 total, 12 up, 10 in 2024-09-17T16:24:11.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:11.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:11.379 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:11 smithi173 ceph-mon[30530]: pgmap v1099: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s; 22/114279 objects degraded (0.019%); 7.5 MiB/s, 122 objects/s recovering 2024-09-17T16:24:11.380 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:11 smithi173 ceph-mon[30530]: osdmap e500: 12 total, 12 up, 10 in 2024-09-17T16:24:11.383 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:11.383 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:11.424 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:11.434 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:11.446 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:11.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:11.548 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:11.576 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:11.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:11.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:11.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:11.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:11.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:11.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:11.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:11.838 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:11.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:11.966 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:11.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:11.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:12.068 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:12.097 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:12.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:12.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:24:12.198 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:12.228 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:12.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:12.330 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:12.357 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:12.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:12.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:12.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:12.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:12.496 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:24:12.497 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-17T16:24:12.506 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:12.594 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:12.616 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:12.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:12.718 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:12.740 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:12.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:12 smithi071 ceph-mon[30002]: 5.d scrub starts 2024-09-17T16:24:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:12 smithi071 ceph-mon[30002]: osdmap e501: 12 total, 12 up, 10 in 2024-09-17T16:24:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:12 smithi071 ceph-mon[30002]: 5.12 scrub starts 2024-09-17T16:24:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3468039171' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:24:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:12 smithi071 ceph-mon[30002]: 5.d scrub ok 2024-09-17T16:24:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:12 smithi071 ceph-mon[30002]: pgmap v1102: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s; 22/106215 objects degraded (0.021%); 7.2 MiB/s, 118 objects/s recovering 2024-09-17T16:24:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:12 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:24:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:12 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:24:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3954271658' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:24:12.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:12.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:24:12.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:12.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:12.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:12 smithi017 ceph-mon[27571]: 5.d scrub starts 2024-09-17T16:24:12.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:12 smithi017 ceph-mon[27571]: osdmap e501: 12 total, 12 up, 10 in 2024-09-17T16:24:12.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:12 smithi017 ceph-mon[27571]: 5.12 scrub starts 2024-09-17T16:24:12.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3468039171' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:24:12.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:12 smithi017 ceph-mon[27571]: 5.d scrub ok 2024-09-17T16:24:12.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:12 smithi017 ceph-mon[27571]: pgmap v1102: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s; 22/106215 objects degraded (0.021%); 7.2 MiB/s, 118 objects/s recovering 2024-09-17T16:24:12.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:12 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:24:12.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:12 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:24:12.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3954271658' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:24:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:12 smithi173 ceph-mon[30530]: 5.d scrub starts 2024-09-17T16:24:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:12 smithi173 ceph-mon[30530]: osdmap e501: 12 total, 12 up, 10 in 2024-09-17T16:24:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:12 smithi173 ceph-mon[30530]: 5.12 scrub starts 2024-09-17T16:24:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3468039171' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:24:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:12 smithi173 ceph-mon[30530]: 5.d scrub ok 2024-09-17T16:24:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:12 smithi173 ceph-mon[30530]: pgmap v1102: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 1.12k op/s; 22/106215 objects degraded (0.021%); 7.2 MiB/s, 118 objects/s recovering 2024-09-17T16:24:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:12 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:24:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:12 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:24:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3954271658' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:24:12.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:12.988 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:12.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:13.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:13.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:13.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:13.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:13.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:13.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:13.241 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:13.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:13.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:13.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:13.369 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:13.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:13.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:13.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:13.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:13.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:13.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:13.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:13.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:13.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:13.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:13.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:13 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:24:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:13 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:24:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:13 smithi071 ceph-mon[30002]: 5.12 scrub ok 2024-09-17T16:24:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:13 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3468039171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:24:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3954271658' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:24:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:13 smithi071 ceph-mon[30002]: osdmap e502: 12 total, 12 up, 10 in 2024-09-17T16:24:13.860 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:13.887 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:13.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:13 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:24:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:13 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:24:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:13 smithi173 ceph-mon[30530]: 5.12 scrub ok 2024-09-17T16:24:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:13 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3468039171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:24:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3954271658' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:24:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:13 smithi173 ceph-mon[30530]: osdmap e502: 12 total, 12 up, 10 in 2024-09-17T16:24:13.988 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:13 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:24:13.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:13 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:24:13.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:13 smithi017 ceph-mon[27571]: 5.12 scrub ok 2024-09-17T16:24:13.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:13 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:13.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3468039171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:24:13.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3954271658' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:24:13.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:13 smithi017 ceph-mon[27571]: osdmap e502: 12 total, 12 up, 10 in 2024-09-17T16:24:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:14.013 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:14.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:14.115 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:14.141 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:14.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:14.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:14.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:14.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:14.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:14.381 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:14.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:14.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:14.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:14.493 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:14.519 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:14.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:14.621 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:14.647 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:14.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:14.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:14.750 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:14.750 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:14.773 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:14.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:14.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:14.806 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:14 smithi017 ceph-mon[27571]: osdmap e503: 12 total, 12 up, 10 in 2024-09-17T16:24:14.807 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:14 smithi017 ceph-mon[27571]: pgmap v1105: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 854 op/s; 22/104025 objects degraded (0.021%) 2024-09-17T16:24:14.807 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:14 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 22/106215 objects degraded (0.021%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:14 smithi071 ceph-mon[30002]: osdmap e503: 12 total, 12 up, 10 in 2024-09-17T16:24:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:14 smithi071 ceph-mon[30002]: pgmap v1105: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 854 op/s; 22/104025 objects degraded (0.021%) 2024-09-17T16:24:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:14 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 22/106215 objects degraded (0.021%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:14.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:14 smithi173 ceph-mon[30530]: osdmap e503: 12 total, 12 up, 10 in 2024-09-17T16:24:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:14 smithi173 ceph-mon[30530]: pgmap v1105: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 2.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 854 op/s; 22/104025 objects degraded (0.021%) 2024-09-17T16:24:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:14 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 22/106215 objects degraded (0.021%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-17T16:24:14.901 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:14.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:14.942 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:24:14.943 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:24:14.969 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:24:14.969 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:24:14.993 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:24:14.993 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:24:15.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:15.016 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:24:15.016 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:24:15.024 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:15.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:15.038 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:24:15.038 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:24:15.065 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:24:15.065 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:24:15.091 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:24:15.092 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:24:15.116 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:24:15.117 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:24:15.125 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:15.145 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:24:15.146 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:24:15.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:15.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:15.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:15.174 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:24:15.174 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:24:15.199 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:24:15.200 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:24:15.223 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:24:15.223 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:24:15.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:15.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:15.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:15.382 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:15.408 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:15.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:15.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:24:15.510 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:15.540 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:15.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:15.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:15.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:15.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:15.777 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:15.804 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:15.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3998902693' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:15.852 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:15.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3998902693' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:15.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:15.934 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:15.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:15.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:15.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3998902693' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:16.037 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:16.060 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:16.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:16.162 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:16.185 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:16.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:16.276 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:24:16.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:16.350 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:16.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:16.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:16.483 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:16.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:16.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:16.595 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:24:16.595 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(3.9 240/239 [PushReplyOp(3:9e32e3c9:::benchmark_data_smithi173_66272_object1144:head)])", 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 541.41702734499995, 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90253544399999996, 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079538+0000", 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6799999999999999e-07 2024-09-17T16:24:16.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.079543+0000", 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6809999999999997e-06 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979238+0000", 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89969455399999998 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979247+0000", 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5659999999999996e-06 2024-09-17T16:24:16.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.979307+0000", 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9338e-05 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:15:15.982073+0000", 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027664370000000001 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.599 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124278 3.9 3.d65c03a9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.930793+0000", 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 447.56577235700001, 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42595735800000001, 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124278 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930793+0000", 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930793+0000", 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930794+0000", 2024-09-17T16:24:16.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4890000000000001e-06 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930801+0000", 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5699999999999998e-06 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930816+0000", 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4953000000000001e-05 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930823+0000", 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6279999999999996e-06 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930874+0000", 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0333000000000001e-05 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.930880+0000", 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8090000000000004e-06 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.356716+0000", 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42583603799999997 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.356750+0000", 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3538000000000003e-05 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.604 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124282 3.9 3.c1fc19b9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.955157+0000", 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 447.54140774000001, 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.402016861, 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124282 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955157+0000", 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955157+0000", 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955158+0000", 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.274e-06 2024-09-17T16:24:16.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955164+0000", 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9240000000000004e-06 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955175+0000", 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0222e-05 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955181+0000", 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1709999999999999e-06 2024-09-17T16:24:16.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955232+0000", 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0927999999999998e-05 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.955240+0000", 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4039999999999995e-06 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.357130+0000", 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40189000200000002 2024-09-17T16:24:16.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.357174+0000", 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3936000000000001e-05 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124278 3.9 3:95c03a6b:::benchmark_data_smithi173_66272_object52322:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:50.410673+0000", 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.08589176800001, 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47593780800000002, 2024-09-17T16:24:16.609 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124278 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410673+0000", 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410673+0000", 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410675+0000", 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6199999999999999e-06 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410689+0000", 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4664000000000001e-05 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410704+0000", 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4174e-05 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410714+0000", 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0659000000000001e-05 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411252+0000", 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053726700000000002 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411263+0000", 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1297999999999999e-05 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.870528+0000", 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45926520500000001 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.870665+0000", 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013726100000000001 2024-09-17T16:24:16.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7,11", 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.870758+0000", 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2767000000000001e-05 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875501+0000", 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047434310000000002 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:16.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875552+0000", 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0096000000000003e-05 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875561+0000", 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0690000000000008e-06 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.885797+0000", 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010236413999999999 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.885869+0000", 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2149000000000003e-05 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.886611+0000", 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074173399999999997 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124282 3.9 3:9d983f83:::benchmark_data_smithi173_66272_object52326:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:50.410739+0000", 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.08582542099998, 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47645007299999997, 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124282 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410739+0000", 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410739+0000", 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410740+0000", 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1099999999999995e-07 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410746+0000", 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0170000000000002e-06 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410754+0000", 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6499999999999996e-06 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.410760+0000", 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5309999999999998e-06 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411348+0000", 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00058806500000000003 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.411359+0000", 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0681e-05 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875583+0000", 2024-09-17T16:24:16.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46422414299999998 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875678+0000", 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4757000000000001e-05 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7,11", 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.875760+0000", 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2109999999999998e-05 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:16.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.878759+0000", 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029989209999999999 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.879450+0000", 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069113699999999998 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.879505+0000", 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4765999999999997e-05 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:16.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.886692+0000", 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0071871299999999999 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.886735+0000", 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3312999999999997e-05 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:50.887190+0000", 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045414099999999997 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.623 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:29960 5.a e436/419 5:53e23a48:::benchmark_data_smithi173_90735_object29959:head v 436'1832, mlcod=436'1829)", 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:14.025136+0000", 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 122.47142887299999, 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23739592400000001, 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.025136+0000", 2024-09-17T16:24:16.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.025136+0000", 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.025137+0000", 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1999999999999999e-06 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.025235+0000", 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7558000000000003e-05 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.261558+0000", 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23632298600000001 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.261564+0000", 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.251e-06 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.261616+0000", 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1742e-05 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.261629+0000", 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3072e-05 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.262500+0000", 2024-09-17T16:24:16.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00087088300000000003 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.262532+0000", 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2231999999999999e-05 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:30436 5.a e437/419 5:5adf2834:::benchmark_data_smithi173_90735_object30435:head v 437'1857, mlcod=437'1855)", 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:14.767059+0000", 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.72950571, 2024-09-17T16:24:16.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37971130600000003, 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.767059+0000", 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.767059+0000", 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.767060+0000", 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:14.767185+0000", 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012498699999999999 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:15.145664+0000", 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37847908499999999 2024-09-17T16:24:16.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:15.145670+0000", 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.84e-06 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:15.145718+0000", 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8374000000000001e-05 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:15.145741+0000", 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2274999999999999e-05 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:15.146736+0000", 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00099567399999999999 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:15.146770+0000", 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4272e-05 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.632 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.12 460/455 [PushOp(5:48c2ef22:::benchmark_data_smithi173_90735_object1746:head, version: 424'104, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:48c2ef22:::benchmark_data_smithi173_90735_object1746:head@424'104, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:52.531363+0000", 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 83.965201635, 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38257142900000002, 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:52.531363+0000", 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:52.531363+0000", 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:52.531364+0000", 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1699999999999998e-07 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:52.531444+0000", 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0252000000000003e-05 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:52.904011+0000", 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37256663800000001 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:52.904021+0000", 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0601000000000001e-05 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:52.904492+0000", 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047083299999999998 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:52.904791+0000", 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029902200000000002 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:52.913935+0000", 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0091435660000000005 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:75081 5.e e470/419 5:7e559e6f:::benchmark_data_smithi173_90735_object75080:head v 470'4698, mlcod=470'4696)", 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:11.118730+0000", 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 65.377834950999997, 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38462375199999999, 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118730+0000", 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118730+0000", 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118731+0000", 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2600000000000002e-07 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118755+0000", 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4680000000000001e-05 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.498232+0000", 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37947653599999998 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.498240+0000", 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9880000000000004e-06 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.498377+0000", 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013760899999999999 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.498404+0000", 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7109000000000001e-05 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.503320+0000", 2024-09-17T16:24:16.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049152409999999999 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.503354+0000", 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3963e-05 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:77290 5.a e472/419 5:59243417:::benchmark_data_smithi173_90735_object77289:head v 472'4770, mlcod=472'4767)", 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:13.894908+0000", 2024-09-17T16:24:16.641 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 62.601657322000001, 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24265737100000001, 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.894908+0000", 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.894908+0000", 2024-09-17T16:24:16.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.894908+0000", 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3900000000000004e-07 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.894999+0000", 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0921000000000006e-05 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.135543+0000", 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24054373500000001 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.135550+0000", 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7469999999999999e-06 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.135605+0000", 2024-09-17T16:24:16.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4982999999999999e-05 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.135628+0000", 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3079e-05 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.137517+0000", 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001888898 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.137565+0000", 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8368999999999997e-05 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:82763 5.13 5:cd63b9f8:::benchmark_data_smithi173_90735_object82762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.917697+0000", 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.578867660999997, 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.621673853, 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:16.646 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 82763 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917697+0000", 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917697+0000", 2024-09-17T16:24:16.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917698+0000", 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.215e-06 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917717+0000", 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8207999999999999e-05 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917727+0000", 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0699e-05 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917733+0000", 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5169999999999999e-06 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917785+0000", 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2389000000000003e-05 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917916+0000", 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013031800000000001 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.918058+0000", 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014283000000000001 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.919174+0000", 2024-09-17T16:24:16.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001115603 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.483081+0000", 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.56390697300000003 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.539298+0000", 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.056217271999999999 2024-09-17T16:24:16.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.539342+0000", 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3949000000000002e-05 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.539371+0000", 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8880000000000001e-05 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "replica_scrub(pg: 5.19,from:0'0,to:473'3919,epoch:475/474,start:5:98000000::::0,end:5:985b9da8:::benchmark_data_smithi173_90735_object72057:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.947931+0000", 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.548633596999998, 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48217526300000002, 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.947931+0000", 2024-09-17T16:24:16.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.947931+0000", 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.947933+0000", 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6300000000000001e-06 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.947944+0000", 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1119e-05 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.429987+0000", 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48204317899999999 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.430011+0000", 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4256999999999998e-05 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.430069+0000", 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7262999999999997e-05 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.430078+0000", 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1009999999999998e-06 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.430107+0000", 2024-09-17T16:24:16.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8714e-05 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.12 483/455 [PushOp(5:496af286:::benchmark_data_smithi173_90735_object18457:head, version: 432'1154, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:496af286:::benchmark_data_smithi173_90735_object18457:head@432'1154, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:34.656554+0000", 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 41.840011124999997, 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225000747, 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:16.657 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:34.656554+0000", 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:34.656554+0000", 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:34.656555+0000", 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:34.656628+0000", 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3416000000000003e-05 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:34.656656+0000", 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8065999999999999e-05 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:34.656714+0000", 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7623e-05 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:34.874460+0000", 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21774614 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:34.874618+0000", 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015851099999999999 2024-09-17T16:24:16.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:34.881555+0000", 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069362 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.12 487/455 [PushOp(5:4ccfc3d8:::benchmark_data_smithi173_90735_object25468:head, version: 435'1570, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:4ccfc3d8:::benchmark_data_smithi173_90735_object25468:head@435'1570, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:46.793951+0000", 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.702613986999999, 2024-09-17T16:24:16.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31247714500000001, 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.793951+0000", 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.793951+0000", 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.793952+0000", 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4300000000000002e-07 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.793976+0000", 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.408e-05 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.793996+0000", 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0103999999999999e-05 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.794007+0000", 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1549000000000001e-05 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.099431+0000", 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30542397999999998 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.099558+0000", 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012641099999999999 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.106428+0000", 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0068702779999999996 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.12 487/455 [PushReplyOp(5:4f2d6763:::benchmark_data_smithi173_90735_object25303:head)])", 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:46.862524+0000", 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.634040420000002, 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26042854999999998, 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.862524+0000", 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.862524+0000", 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.862525+0000", 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5799999999999998e-07 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.862534+0000", 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3899999999999993e-06 2024-09-17T16:24:16.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.869635+0000", 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0071014859999999997 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.869643+0000", 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0330000000000008e-06 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.115517+0000", 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24587421000000001 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.122953+0000", 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.007435573 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.12 487/455 [PushReplyOp(5:4f2d6763:::benchmark_data_smithi173_90735_object25303:head)])", 2024-09-17T16:24:16.669 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:46.868259+0000", 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.628305414, 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24525148799999999, 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.868259+0000", 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.868259+0000", 2024-09-17T16:24:16.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.868260+0000", 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0500000000000002e-07 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.868268+0000", 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3030000000000002e-06 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.868282+0000", 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4849e-05 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:46.868290+0000", 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5020000000000003e-06 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.109478+0000", 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24118838400000001 2024-09-17T16:24:16.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.113511+0000", 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040326449999999996 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.12 487/455 [PushOp(5:492b1b12:::benchmark_data_smithi173_90735_object25692:head, version: 435'1584, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:492b1b12:::benchmark_data_smithi173_90735_object25692:head@435'1584, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:47.109500+0000", 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.387065010000001, 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26063943499999997, 2024-09-17T16:24:16.673 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.109500+0000", 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.109500+0000", 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.109501+0000", 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6299999999999993e-07 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.109543+0000", 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2061999999999998e-05 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.109565+0000", 2024-09-17T16:24:16.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1832e-05 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.109575+0000", 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0686999999999999e-05 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.363203+0000", 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25362720900000002 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.363333+0000", 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013026699999999999 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:47.370139+0000", 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0068064149999999997 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.677 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:114326 5.a e490/419 5:55b0fcfc:::benchmark_data_smithi173_90735_object31555:head v 490'7037, mlcod=489'7034)", 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:54.016971+0000", 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 22.479594066000001, 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27998295299999998, 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.016971+0000", 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.016971+0000", 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.016971+0000", 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8800000000000002e-07 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.016976+0000", 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7929999999999997e-06 2024-09-17T16:24:16.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.293506+0000", 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27652974899999999 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.293518+0000", 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.202e-05 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.293611+0000", 2024-09-17T16:24:16.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3091999999999996e-05 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.293630+0000", 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8626000000000001e-05 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.296902+0000", 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003272802 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.296954+0000", 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1282999999999997e-05 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.12 490/455 [PushReplyOp(5:4cdaf9d1:::benchmark_data_smithi173_90735_object32062:head)])", 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:54.051288+0000", 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 22.445277177000001, 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.267691334, 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.682 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.051288+0000", 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.051288+0000", 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.051289+0000", 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6799999999999999e-07 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.051293+0000", 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1010000000000002e-06 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.312509+0000", 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.261216687 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.312517+0000", 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3119999999999996e-06 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.312988+0000", 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047120899999999999 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.318979+0000", 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0059911570000000004 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.685 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:114359 5.10 e490/470 5:0e1ae67b:::benchmark_data_smithi173_90735_object31588:head v 490'4909, mlcod=489'4906)", 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:54.098359+0000", 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 22.39820542, 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21600043499999999, 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:16.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.098359+0000", 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.098359+0000", 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.098360+0000", 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1200000000000001e-07 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.098383+0000", 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2450999999999999e-05 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.312544+0000", 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21416079800000001 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.312548+0000", 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7190000000000001e-06 2024-09-17T16:24:16.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.312617+0000", 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8373000000000004e-05 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.312632+0000", 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5316999999999999e-05 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.314317+0000", 2024-09-17T16:24:16.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001684959 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.314360+0000", 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2905999999999999e-05 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:16.690 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:24:16.691 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:16 smithi173 ceph-mon[30530]: pgmap v1106: 27 pgs: 27 active+clean; 2.0 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 216 op/s 2024-09-17T16:24:16.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:16.692 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:16.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:16.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:16 smithi017 ceph-mon[27571]: pgmap v1106: 27 pgs: 27 active+clean; 2.0 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 216 op/s 2024-09-17T16:24:16.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:16.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:16 smithi071 ceph-mon[30002]: pgmap v1106: 27 pgs: 27 active+clean; 2.0 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 216 op/s 2024-09-17T16:24:16.816 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:16.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:16.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:16.849 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:16.918 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:16.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:17.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:17.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:17.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:17.181 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:17.207 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:17.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:17.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:17.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:17.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:17.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:17.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:17.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:17.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:17.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:17.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:17.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:17.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:24:17.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:17.716 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:17.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:17 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/104025 objects degraded (0.021%), 1 pg degraded, 1 pg undersized) 2024-09-17T16:24:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:17 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/104025 objects degraded (0.021%), 1 pg degraded, 1 pg undersized) 2024-09-17T16:24:17.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:17.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:17.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:17 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/104025 objects degraded (0.021%), 1 pg degraded, 1 pg undersized) 2024-09-17T16:24:17.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:17.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:17.968 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:17.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:17.975 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:18.069 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:18.091 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:18.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:18.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:18.216 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:18.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:18.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:18.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:18.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:18.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:18.473 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:18.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:18.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:18.532 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:137710 5.e 5:7c417fed:::benchmark_data_smithi173_90735_object54939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:18.505565+0000", 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.024197964999999998, 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024229250000000001, 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:18.533 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 137710 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.505565+0000", 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.505565+0000", 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:18.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.505567+0000", 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1019999999999999e-06 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.505575+0000", 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0369999999999995e-06 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.505594+0000", 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9558000000000001e-05 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.505604+0000", 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5670000000000001e-06 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.505671+0000", 2024-09-17T16:24:18.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7354000000000006e-05 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.505812+0000", 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014026500000000001 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.505943+0000", 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013112399999999999 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.510113+0000", 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041707209999999996 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.510329+0000", 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021547499999999999 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:137724 5.e 5:78bf29d9:::benchmark_data_smithi173_90735_object54953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:18.516587+0000", 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.013175958999999999, 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013282950999999999, 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 137724 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:18.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.516587+0000", 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.516587+0000", 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.516588+0000", 2024-09-17T16:24:18.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7039999999999999e-06 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.516594+0000", 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7350000000000001e-06 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.516609+0000", 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.45e-05 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.516616+0000", 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6179999999999997e-06 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.516668+0000", 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1566000000000001e-05 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:18.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.516756+0000", 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8321999999999999e-05 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.516825+0000", 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9296999999999997e-05 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.520472+0000", 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036468899999999999 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.521085+0000", 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00061273499999999999 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:137729 5.e 5:71a52833:::benchmark_data_smithi173_90735_object54958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:18.526211+0000", 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0035511940000000001, 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037171560000000001, 2024-09-17T16:24:18.544 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 137729 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.526211+0000", 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.526211+0000", 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.526213+0000", 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.294e-06 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.526218+0000", 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2549999999999997e-06 2024-09-17T16:24:18.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.526232+0000", 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3633e-05 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.526238+0000", 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2999999999999998e-06 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:18.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.526295+0000", 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7467999999999999e-05 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.526427+0000", 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013163000000000001 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.526550+0000", 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012270200000000001 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.529824+0000", 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003273888 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:18.529909+0000", 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5597000000000001e-05 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:24:18.549 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 3 2024-09-17T16:24:18.550 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:24:18.550 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:24:18.550 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:24:18.550 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T16:24:18.550 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:24:18.551 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:24:18.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:18.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:18.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3477767988' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:18 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:18 smithi173 ceph-mon[30530]: pgmap v1107: 27 pgs: 27 active+clean; 1.9 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 125 op/s 2024-09-17T16:24:18.703 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:18.732 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:18.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:18.796 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:24:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3477767988' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:18 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:18 smithi071 ceph-mon[30002]: pgmap v1107: 27 pgs: 27 active+clean; 1.9 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 125 op/s 2024-09-17T16:24:18.828 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3477767988' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:18.828 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:18 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:18.828 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:18 smithi017 ceph-mon[27571]: pgmap v1107: 27 pgs: 27 active+clean; 1.9 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 125 op/s 2024-09-17T16:24:18.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:18.860 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:18.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:18.962 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:18.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:18.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:19.093 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:19.116 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:19.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:19.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:19.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:19.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:19.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:19.377 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:19.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:19.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:24:19.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:19.515 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:19.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:19.527 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:19.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:19 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:24:19.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:19 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:24:19.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:19 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:19.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:19 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:19.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:19 smithi017 ceph-mon[27571]: osdmap e504: 12 total, 12 up, 10 in 2024-09-17T16:24:19.549 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:19 smithi017 ceph-mon[27571]: osdmap e505: 12 total, 12 up, 10 in 2024-09-17T16:24:19.578 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:19.617 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:19.642 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:19.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:19 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:24:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:19 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:24:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:19 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:19 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:19 smithi173 ceph-mon[30530]: osdmap e504: 12 total, 12 up, 10 in 2024-09-17T16:24:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:19 smithi173 ceph-mon[30530]: osdmap e505: 12 total, 12 up, 10 in 2024-09-17T16:24:19.744 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:19.755 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:19.772 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:19.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:19 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:24:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:19 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:24:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:19 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:19 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:19 smithi071 ceph-mon[30002]: osdmap e504: 12 total, 12 up, 10 in 2024-09-17T16:24:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:19 smithi071 ceph-mon[30002]: osdmap e505: 12 total, 12 up, 10 in 2024-09-17T16:24:19.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:19.903 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:19.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:20.005 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:20.028 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:20.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:20.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:20.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:20.157 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:20.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:20.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:20.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:20.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:20.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:20.415 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:20.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:20.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:20.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:20.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:20.625 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:24:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:24:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:24:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:24:20.626 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:24:20.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:20.679 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:20.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:20.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:20.808 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:20.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:20 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:24:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:20 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:24:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:20 smithi071 ceph-mon[30002]: pgmap v1110: 27 pgs: 27 active+clean; 1.8 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 279 op/s 2024-09-17T16:24:20.865 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:20 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:24:20.866 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:20 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:24:20.866 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:20 smithi017 ceph-mon[27571]: pgmap v1110: 27 pgs: 27 active+clean; 1.8 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 279 op/s 2024-09-17T16:24:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:20 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:24:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:20 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:24:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:20 smithi173 ceph-mon[30530]: pgmap v1110: 27 pgs: 27 active+clean; 1.8 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 279 op/s 2024-09-17T16:24:20.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:20.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:20.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:20.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:21.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:21.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:21.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:21.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:21.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:21.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:21.220 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 506, 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:24:20.537999+0000", 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:22:45.057702+0000", 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5806080, 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:24:21.221 INFO:teuthology.orchestra.run.smithi017.stdout: "nodeep-scrub", 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "noscrub", 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 5, 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:24:21.222 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:24:21.223 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:24:21.224 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "446", 2024-09-17T16:24:21.225 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:24:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:24:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:24:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:24:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:24:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:24:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:24:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:24:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:24:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:24:21.226 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:24:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:24:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:24:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:24:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:24:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:24:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:24:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:24:21.227 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:24:21.228 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:24:21.228 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:24:21.228 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:24:21.228 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:24:21.228 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:24:21.228 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:24:21.229 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:24:21.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.230 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:24:21.230 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:24:21.230 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:24:21.230 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:24:21.230 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:24:21.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:24:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:24:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:24:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:24:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:24:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:24:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:24:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.85684323310852051, 2024-09-17T16:24:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-17T16:24:21.232 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 5, 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_3", 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:21:40.832300+0000", 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:24:21.233 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 26, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 26, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 26, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 26, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 26, 2024-09-17T16:24:21.234 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "443", 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "441", 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:24:21.235 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:24:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:24:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:24:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:24:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:24:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:24:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:24:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:24:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:24:21.236 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:24:21.237 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:24:21.237 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:24:21.237 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:24:21.237 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:24:21.237 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:24:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:24:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:24:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:24:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:24:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:24:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:24:21.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:24:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:24:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:24:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:24:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:24:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:24:21.239 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:24:21.240 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:24:21.240 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:24:21.240 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:24:21.240 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:24:21.240 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:24:21.240 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:24:21.240 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:24:21.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.241 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:24:21.241 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:24:21.241 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 1.690000057220459, 2024-09-17T16:24:21.241 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 1.690000057220459, 2024-09-17T16:24:21.241 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.9100000262260437, 2024-09-17T16:24:21.241 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 1.5399999618530273, 2024-09-17T16:24:21.241 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 1.5399999618530273, 2024-09-17T16:24:21.242 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.85684323310852051, 2024-09-17T16:24:21.242 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.93000000715255737, 2024-09-17T16:24:21.242 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-17T16:24:21.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.242 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:24:21.243 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:24:21.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.243 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:24:21.243 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:24:21.243 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.243 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 502, 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.245 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.245 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:24:21.245 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:21.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.245 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.245 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:24:21.245 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:21.245 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.245 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.246 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.246 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.246 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.247 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.248 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.250 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:24:21.250 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:24:21.250 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:24:21.250 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:24:21.250 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.250 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.250 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.252 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.252 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:24:21.252 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:24:21.252 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.252 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 0, 2024-09-17T16:24:21.252 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-17T16:24:21.252 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:21.253 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.253 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.253 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:24:21.253 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 301, 2024-09-17T16:24:21.253 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.253 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.253 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.253 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.253 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.254 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:24:21.254 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:21.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.254 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.254 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:24:21.254 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:21.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.254 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.255 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.255 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.255 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.255 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:24:21.255 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:21.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.256 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.256 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:24:21.256 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:21.256 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.256 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.256 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.256 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.257 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.257 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:24:21.257 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:21.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.257 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.257 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:24:21.257 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:24:21.258 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 469, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.260 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.263 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:21.264 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.265 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 472, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.266 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.267 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.270 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:24:21.271 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 481, 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.273 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.276 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:24:21.277 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 0, 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 454, 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.278 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:24:21.279 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.281 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:24:21.282 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.66790771484375, 2024-09-17T16:24:21.283 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:21.284 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.284 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.284 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:24:21.284 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 466, 2024-09-17T16:24:21.284 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.284 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.284 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.284 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.285 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.286 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:21.288 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.289 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.8061676025390625, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 482, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.291 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.293 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:24:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:24:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 500, 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.299 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.300 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:21.301 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.2821197509765625, 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:21.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:21.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:24:21.304 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:21.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:24:21.306 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.307 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:21.308 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 462, 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:21.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:21.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.312 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:24:21.313 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.588836669921875, 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:21.314 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 291, 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.315 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:21.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:21.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:21.319 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:01.931904+0000", 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 65536, 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:25.058618+0000", 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.321 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:51.474083+0000", 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:19.641962+0000", 2024-09-17T16:24:21.322 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:53.204639+0000", 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.323 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 55422, 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:18.369564+0000", 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:21.324 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:55.626986+0000", 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:37.531895+0000", 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:11.761032+0000", 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.326 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:30.923568+0000", 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:21.327 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:58.985462+0000", 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:12.252165+0000", 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.328 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [ 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "5.5", 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: 10, 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: 4, 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: 3 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-17T16:24:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.330 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:24:21.331 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:24:21.332 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:24:21.332 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:24:21.332 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:24:21.332 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:24:21.332 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:24:21.332 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:24:21.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:21.332 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:24:21.354 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:21.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:21.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:24:21.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:21.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:24:21.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:21.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:21.521 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:24:21.521 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:24:21.549 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:24:21.549 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:24:21.576 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:24:21.576 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:24:21.587 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:21.609 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:24:21.610 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:24:21.618 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:21.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:21.638 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:24:21.638 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:24:21.676 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:24:21.677 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:24:21.709 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:21.714 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:24:21.714 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:24:21.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:21 smithi017 ceph-mon[27571]: osdmap e506: 12 total, 12 up, 10 in 2024-09-17T16:24:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3468904507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2924953934' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:24:21.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:24:21.745 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:24:21.750 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:21.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:21.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:24:21.768 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:24:21.794 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:24:21.795 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:24:21.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:21 smithi071 ceph-mon[30002]: osdmap e506: 12 total, 12 up, 10 in 2024-09-17T16:24:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3468904507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2924953934' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:24:21.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:24:21.821 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:24:21.837 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:21 smithi173 ceph-mon[30530]: osdmap e506: 12 total, 12 up, 10 in 2024-09-17T16:24:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3468904507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:21.857 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2924953934' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:24:21.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:21.859 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:24:21.859 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:24:21.885 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:21.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:21.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:22.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:22.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:22.115 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:22.140 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:22.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:22.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:22.266 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:22.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:22.273 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:22.273 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:22.367 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:22.395 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:22.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:22.497 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:22.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:22.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:22.628 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:22.662 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:22.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:22.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:22.673 INFO:tasks.thrashosds.thrasher:j is {'epoch': 506, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:24:20.537999+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:22:45.057702+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '446', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8568432331085205, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 5, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-17T16:21:40.832300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '443', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '441', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 0.9100000262260437, 'raw_score_acting': 1.5399999618530273, 'raw_score_stable': 1.5399999618530273, 'primary_affinity_weighted': 0.8568432331085205, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.8600000143051147}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 502, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 301, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 469, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 472, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 481, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 454, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 0.66790771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 466, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 0.8061676025390625, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 482, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 500, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0.2821197509765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 462, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 0.588836669921875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:24:01.931904+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-17T16:23:25.058618+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:51.474083+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:24:19.641962+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:53.204639+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 55422, 'last_purged_snaps_scrub': '2024-09-17T16:24:18.369564+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:55.626986+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:37.531895+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:24:11.761032+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:30.923568+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:58.985462+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:24:12.252165+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.5', 'osds': [10, 4, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:24:22.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:24:22.764 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:22.796 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:22.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:22.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:22 smithi071 ceph-mon[30002]: pgmap v1112: 27 pgs: 27 active+clean; 1.6 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 0 B/s, 0 objects/s recovering 2024-09-17T16:24:22.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:22 smithi173 ceph-mon[30530]: pgmap v1112: 27 pgs: 27 active+clean; 1.6 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 0 B/s, 0 objects/s recovering 2024-09-17T16:24:22.897 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:22.921 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:22.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:22 smithi017 ceph-mon[27571]: pgmap v1112: 27 pgs: 27 active+clean; 1.6 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.19k op/s; 0 B/s, 0 objects/s recovering 2024-09-17T16:24:23.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:23.048 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:23.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:23.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:24:23.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:23.175 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:23.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:23.277 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:23.302 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:23.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:23.404 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:23.418 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:23.430 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:23.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:23.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:23.510 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:23.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:23.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:23.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:23.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:23.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:23.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:23.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:23 smithi017 ceph-mon[27571]: 5.18 scrub starts 2024-09-17T16:24:23.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:23.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:23 smithi071 ceph-mon[30002]: 5.18 scrub starts 2024-09-17T16:24:23.818 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:23.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:23.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:23.872 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:23 smithi173 ceph-mon[30530]: 5.18 scrub starts 2024-09-17T16:24:23.920 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:23.943 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:23.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:24.001 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:24.002 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:24.045 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:24.069 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:24.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:24.101 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:24.170 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:24.197 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:24.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:24.299 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:24.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:24.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:24.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:24.457 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:24.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:24.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:24.483 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:24:24.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:24.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:24.585 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:24.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:24.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:24.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:24.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:24.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:24:24.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:24 smithi071 ceph-mon[30002]: pgmap v1113: 27 pgs: 27 active+clean; 1.5 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-17T16:24:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2466437267' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:24 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2466437267' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:24:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:24 smithi071 ceph-mon[30002]: osdmap e507: 12 total, 12 up, 10 in 2024-09-17T16:24:24.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:24.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:24.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:24 smithi173 ceph-mon[30530]: pgmap v1113: 27 pgs: 27 active+clean; 1.5 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-17T16:24:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2466437267' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:24 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2466437267' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:24:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:24 smithi173 ceph-mon[30530]: osdmap e507: 12 total, 12 up, 10 in 2024-09-17T16:24:24.937 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:24:24.937 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:24 smithi017 ceph-mon[27571]: pgmap v1113: 27 pgs: 27 active+clean; 1.5 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-17T16:24:24.937 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2466437267' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:24.937 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:24 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:24.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2466437267' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:24:24.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:24 smithi017 ceph-mon[27571]: osdmap e507: 12 total, 12 up, 10 in 2024-09-17T16:24:24.939 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:24:24.960 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:24.988 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:24.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:25.090 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:25.112 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:25.114 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:25.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:25.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:25.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:25.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:25.341 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:25.362 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:25.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:25.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:25.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:25.481 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:24:25.489 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:25.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:25.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:25.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:25.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:25.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:25.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:25.719 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:25.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:25 smithi017 ceph-mon[27571]: from='client.26533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:24:25.746 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:25.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:25.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:25 smithi173 ceph-mon[30530]: from='client.26533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:24:25.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:25.872 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:25.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:25.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:25.958 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:25.959 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:25.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:26.001 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:26.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:26.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:26.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:25 smithi071 ceph-mon[30002]: from='client.26533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:24:26.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:26.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:26.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:26.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:26.138 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1114,"stamp":"2024-09-17T16:24:24.481814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1590002232,"num_objects":24256,"num_object_clones":0,"num_object_copies":72768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24256,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":104770,"num_write_kb":3052039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14221,"num_bytes_recovered":933115488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167449,"ondisk_log_size":167449,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":8448500,"kb_used_data":5014508,"kb_used_omap":55,"kb_used_meta":3431752,"kb_avail":929003020,"statfs":{"total":959950356480,"available":951299092480,"internally_reserved":0,"allocated":5134856192,"data_stored":5128268692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57064,"internal_metadata":3514114328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-394526720,"num_objects":-6020,"num_object_clones":0,"num_object_copies":-18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6020,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.944815"},"pg_stats":[{"pgid":"5.19","version":"506'5857","reported_seq":13201,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:23.040580+0000","last_change":"2024-09-17T16:23:20.695328+0000","last_active":"2024-09-17T16:24:23.040580+0000","last_peered":"2024-09-17T16:24:23.040580+0000","last_clean":"2024-09-17T16:24:23.040580+0000","last_became_active":"2024-09-17T16:23:19.479765+0000","last_became_peered":"2024-09-17T16:23:19.479765+0000","last_unstale":"2024-09-17T16:24:23.040580+0000","last_undegraded":"2024-09-17T16:24:23.040580+0000","last_fullsized":"2024-09-17T16:24:23.040580+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":5,"last_scrub":"475'3952","last_scrub_stamp":"2024-09-17T16:23:20.695212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:20.695212+0000","objects_scrubbed":2599,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:43.454939+0000","scrub_duration":747,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45481984,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1905,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,9],"acting":[7,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"506'5761","reported_seq":11366,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:21.196343+0000","last_change":"2024-09-17T16:23:12.834238+0000","last_active":"2024-09-17T16:24:21.196343+0000","last_peered":"2024-09-17T16:24:21.196343+0000","last_clean":"2024-09-17T16:24:21.196343+0000","last_became_active":"2024-09-17T16:23:11.481470+0000","last_became_peered":"2024-09-17T16:23:11.481470+0000","last_unstale":"2024-09-17T16:24:21.196343+0000","last_undegraded":"2024-09-17T16:24:21.196343+0000","last_fullsized":"2024-09-17T16:24:21.196343+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":5,"last_scrub":"471'3717","last_scrub_stamp":"2024-09-17T16:23:12.834164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:12.834164+0000","objects_scrubbed":2407,"log_size":5761,"log_dups_size":0,"ondisk_log_size":5761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:16.100430+0000","scrub_duration":375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52101120,"num_objects":795,"num_object_clones":0,"num_object_copies":2385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2056,"num_write_kb":14208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,11],"acting":[3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"506'5681","reported_seq":13497,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:21.156051+0000","last_change":"2024-09-17T16:23:16.286018+0000","last_active":"2024-09-17T16:24:21.156051+0000","last_peered":"2024-09-17T16:24:21.156051+0000","last_clean":"2024-09-17T16:24:21.156051+0000","last_became_active":"2024-09-17T16:23:14.463250+0000","last_became_peered":"2024-09-17T16:23:14.463250+0000","last_unstale":"2024-09-17T16:24:21.156051+0000","last_undegraded":"2024-09-17T16:24:21.156051+0000","last_fullsized":"2024-09-17T16:24:21.156051+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'3757","last_scrub_stamp":"2024-09-17T16:23:16.285798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:16.285798+0000","objects_scrubbed":2469,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:29.440605+0000","scrub_duration":869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49938432,"num_objects":762,"num_object_clones":0,"num_object_copies":2286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1931,"num_write_kb":7168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,6],"acting":[3,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"506'5804","reported_seq":16996,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:24.384662+0000","last_change":"2024-09-17T16:23:05.463059+0000","last_active":"2024-09-17T16:24:24.384662+0000","last_peered":"2024-09-17T16:24:24.384662+0000","last_clean":"2024-09-17T16:24:24.384662+0000","last_became_active":"2024-09-17T16:23:04.506933+0000","last_became_peered":"2024-09-17T16:23:04.506933+0000","last_unstale":"2024-09-17T16:24:24.384662+0000","last_undegraded":"2024-09-17T16:24:24.384662+0000","last_fullsized":"2024-09-17T16:24:24.384662+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"466'3509","last_scrub_stamp":"2024-09-17T16:23:05.462957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:05.462957+0000","objects_scrubbed":2172,"log_size":5804,"log_dups_size":0,"ondisk_log_size":5804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:19.514469+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41811968,"num_objects":638,"num_object_clones":0,"num_object_copies":1914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2296,"num_write_kb":24384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.14","version":"506'5563","reported_seq":8351,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:21.052884+0000","last_change":"2024-09-17T16:23:10.736490+0000","last_active":"2024-09-17T16:24:21.052884+0000","last_peered":"2024-09-17T16:24:21.052884+0000","last_clean":"2024-09-17T16:24:21.052884+0000","last_became_active":"2024-09-17T16:22:39.452456+0000","last_became_peered":"2024-09-17T16:22:39.452456+0000","last_unstale":"2024-09-17T16:24:21.052884+0000","last_undegraded":"2024-09-17T16:24:21.052884+0000","last_fullsized":"2024-09-17T16:24:21.052884+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":5,"last_scrub":"470'3570","last_scrub_stamp":"2024-09-17T16:23:10.736312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:10.736312+0000","objects_scrubbed":2295,"log_size":5563,"log_dups_size":0,"ondisk_log_size":5563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:21.572267+0000","scrub_duration":336,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50987008,"num_objects":778,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1993,"num_write_kb":15232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"506'5993","reported_seq":10245,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:24.255921+0000","last_change":"2024-09-17T16:23:16.579844+0000","last_active":"2024-09-17T16:24:24.255921+0000","last_peered":"2024-09-17T16:24:24.255921+0000","last_clean":"2024-09-17T16:24:24.255921+0000","last_became_active":"2024-09-17T16:22:45.067204+0000","last_became_peered":"2024-09-17T16:22:45.067204+0000","last_unstale":"2024-09-17T16:24:24.255921+0000","last_undegraded":"2024-09-17T16:24:24.255921+0000","last_fullsized":"2024-09-17T16:24:24.255921+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":5,"last_scrub":"473'3905","last_scrub_stamp":"2024-09-17T16:23:16.579631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:16.579631+0000","objects_scrubbed":2579,"log_size":5993,"log_dups_size":0,"ondisk_log_size":5993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:20.536887+0000","scrub_duration":520,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46006272,"num_objects":702,"num_object_clones":0,"num_object_copies":2106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,10],"acting":[8,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.12","version":"506'5803","reported_seq":12344,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:24.284393+0000","last_change":"2024-09-17T16:24:12.090610+0000","last_active":"2024-09-17T16:24:24.284393+0000","last_peered":"2024-09-17T16:24:24.284393+0000","last_clean":"2024-09-17T16:24:24.284393+0000","last_became_active":"2024-09-17T16:24:11.504911+0000","last_became_peered":"2024-09-17T16:24:11.504911+0000","last_unstale":"2024-09-17T16:24:24.284393+0000","last_undegraded":"2024-09-17T16:24:24.284393+0000","last_fullsized":"2024-09-17T16:24:24.284393+0000","mapping_epoch":500,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":501,"parent":"0.0","parent_split_bits":5,"last_scrub":"501'5399","last_scrub_stamp":"2024-09-17T16:24:12.090381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:24:12.090381+0000","objects_scrubbed":1070,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:21.010219+0000","scrub_duration":467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43253760,"num_objects":660,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":410,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,7],"acting":[8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.11","version":"506'5692","reported_seq":8519,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:20.735339+0000","last_change":"2024-09-17T16:23:11.994555+0000","last_active":"2024-09-17T16:24:20.735339+0000","last_peered":"2024-09-17T16:24:20.735339+0000","last_clean":"2024-09-17T16:24:20.735339+0000","last_became_active":"2024-09-17T16:22:40.460789+0000","last_became_peered":"2024-09-17T16:22:40.460789+0000","last_unstale":"2024-09-17T16:24:20.735339+0000","last_undegraded":"2024-09-17T16:24:20.735339+0000","last_fullsized":"2024-09-17T16:24:20.735339+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"471'3684","last_scrub_stamp":"2024-09-17T16:23:11.994378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:11.994378+0000","objects_scrubbed":2339,"log_size":5692,"log_dups_size":0,"ondisk_log_size":5692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:20.140833+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50724864,"num_objects":774,"num_object_clones":0,"num_object_copies":2322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2019,"num_write_kb":14528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"506'5790","reported_seq":11310,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:22.893264+0000","last_change":"2024-09-17T16:23:12.692764+0000","last_active":"2024-09-17T16:24:22.893264+0000","last_peered":"2024-09-17T16:24:22.893264+0000","last_clean":"2024-09-17T16:24:22.893264+0000","last_became_active":"2024-09-17T16:23:11.485206+0000","last_became_peered":"2024-09-17T16:23:11.485206+0000","last_unstale":"2024-09-17T16:24:22.893264+0000","last_undegraded":"2024-09-17T16:24:22.893264+0000","last_fullsized":"2024-09-17T16:24:22.893264+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":5,"last_scrub":"471'3718","last_scrub_stamp":"2024-09-17T16:23:12.692445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:12.692445+0000","objects_scrubbed":2401,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:29.573114+0000","scrub_duration":658,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":740,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2083,"num_write_kb":13504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,6],"acting":[7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"505'8589","reported_seq":16167,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-17T16:24:19.769933+0000","last_change":"2024-09-17T16:23:37.492144+0000","last_active":"2024-09-17T16:24:19.769933+0000","last_peered":"2024-09-17T16:24:19.769933+0000","last_clean":"2024-09-17T16:24:19.769933+0000","last_became_active":"2024-09-17T16:23:33.481067+0000","last_became_peered":"2024-09-17T16:23:33.481067+0000","last_unstale":"2024-09-17T16:24:19.769933+0000","last_undegraded":"2024-09-17T16:24:19.769933+0000","last_fullsized":"2024-09-17T16:24:19.769933+0000","mapping_epoch":481,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'6097","last_scrub_stamp":"2024-09-17T16:23:37.491931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:37.491931+0000","objects_scrubbed":4135,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:53.441397+0000","scrub_duration":729,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105578496,"num_objects":1611,"num_object_clones":0,"num_object_copies":4833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8589,"num_write_kb":326400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3666,"num_bytes_recovered":240254976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,2],"acting":[4,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"506'8853","reported_seq":9042,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:21.293095+0000","last_change":"2024-09-17T16:21:42.615978+0000","last_active":"2024-09-17T16:24:21.293095+0000","last_peered":"2024-09-17T16:24:21.293095+0000","last_clean":"2024-09-17T16:24:21.293095+0000","last_became_active":"2024-09-17T16:21:42.615584+0000","last_became_peered":"2024-09-17T16:21:42.615584+0000","last_unstale":"2024-09-17T16:24:21.293095+0000","last_undegraded":"2024-09-17T16:24:21.293095+0000","last_fullsized":"2024-09-17T16:24:21.293095+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":8853,"log_dups_size":0,"ondisk_log_size":8853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:52.557187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103088128,"num_objects":1573,"num_object_clones":0,"num_object_copies":4719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8853,"num_write_kb":333632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"506'8954","reported_seq":16607,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:23.115774+0000","last_change":"2024-09-17T16:24:11.718487+0000","last_active":"2024-09-17T16:24:23.115774+0000","last_peered":"2024-09-17T16:24:23.115774+0000","last_clean":"2024-09-17T16:24:23.115774+0000","last_became_active":"2024-09-17T16:23:34.565964+0000","last_became_peered":"2024-09-17T16:23:34.565964+0000","last_unstale":"2024-09-17T16:24:23.115774+0000","last_undegraded":"2024-09-17T16:24:23.115774+0000","last_fullsized":"2024-09-17T16:24:23.115774+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"501'8298","last_scrub_stamp":"2024-09-17T16:24:11.718300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:24:11.718300+0000","objects_scrubbed":2083,"log_size":8954,"log_dups_size":0,"ondisk_log_size":8954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:32.763015+0000","scrub_duration":576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92602392,"num_objects":1414,"num_object_clones":0,"num_object_copies":4242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8954,"num_write_kb":331713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3712,"num_bytes_recovered":243269632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"505'8676","reported_seq":8862,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-17T16:24:19.818968+0000","last_change":"2024-09-17T16:21:42.616080+0000","last_active":"2024-09-17T16:24:19.818968+0000","last_peered":"2024-09-17T16:24:19.818968+0000","last_clean":"2024-09-17T16:24:19.818968+0000","last_became_active":"2024-09-17T16:21:42.615761+0000","last_became_peered":"2024-09-17T16:21:42.615761+0000","last_unstale":"2024-09-17T16:24:19.818968+0000","last_undegraded":"2024-09-17T16:24:19.818968+0000","last_fullsized":"2024-09-17T16:24:19.818968+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":8676,"log_dups_size":0,"ondisk_log_size":8676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:32.728532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110362624,"num_objects":1684,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":331520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"506'8809","reported_seq":9004,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:21.107999+0000","last_change":"2024-09-17T16:24:13.556638+0000","last_active":"2024-09-17T16:24:21.107999+0000","last_peered":"2024-09-17T16:24:21.107999+0000","last_clean":"2024-09-17T16:24:21.107999+0000","last_became_active":"2024-09-17T16:24:13.525528+0000","last_became_peered":"2024-09-17T16:24:13.525528+0000","last_unstale":"2024-09-17T16:24:21.107999+0000","last_undegraded":"2024-09-17T16:24:21.107999+0000","last_fullsized":"2024-09-17T16:24:21.107999+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":8809,"log_dups_size":0,"ondisk_log_size":8809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98238464,"num_objects":1499,"num_object_clones":0,"num_object_copies":4497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8809,"num_write_kb":329856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,7],"acting":[0,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"506'8705","reported_seq":8894,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:21.775860+0000","last_change":"2024-09-17T16:21:42.467809+0000","last_active":"2024-09-17T16:24:21.775860+0000","last_peered":"2024-09-17T16:24:21.775860+0000","last_clean":"2024-09-17T16:24:21.775860+0000","last_became_active":"2024-09-17T16:21:42.467536+0000","last_became_peered":"2024-09-17T16:21:42.467536+0000","last_unstale":"2024-09-17T16:24:21.775860+0000","last_undegraded":"2024-09-17T16:24:21.775860+0000","last_fullsized":"2024-09-17T16:24:21.775860+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":8705,"log_dups_size":0,"ondisk_log_size":8705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:34.062509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99155968,"num_objects":1513,"num_object_clones":0,"num_object_copies":4539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":326976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"506'5816","reported_seq":6032,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:23.054116+0000","last_change":"2024-09-17T16:22:26.417071+0000","last_active":"2024-09-17T16:24:23.054116+0000","last_peered":"2024-09-17T16:24:23.054116+0000","last_clean":"2024-09-17T16:24:23.054116+0000","last_became_active":"2024-09-17T16:22:24.973556+0000","last_became_peered":"2024-09-17T16:22:24.973556+0000","last_unstale":"2024-09-17T16:24:23.054116+0000","last_undegraded":"2024-09-17T16:24:23.054116+0000","last_fullsized":"2024-09-17T16:24:23.054116+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'2674","last_scrub_stamp":"2024-09-17T16:22:26.416985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:22:26.416985+0000","objects_scrubbed":1363,"log_size":5816,"log_dups_size":0,"ondisk_log_size":5816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:16.061765+0000","scrub_duration":488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45809664,"num_objects":699,"num_object_clones":0,"num_object_copies":2097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3142,"num_write_kb":79296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"506'5734","reported_seq":5956,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:21.133373+0000","last_change":"2024-09-17T16:24:13.542510+0000","last_active":"2024-09-17T16:24:21.133373+0000","last_peered":"2024-09-17T16:24:21.133373+0000","last_clean":"2024-09-17T16:24:21.133373+0000","last_became_active":"2024-09-17T16:24:13.527094+0000","last_became_peered":"2024-09-17T16:24:13.527094+0000","last_unstale":"2024-09-17T16:24:21.133373+0000","last_undegraded":"2024-09-17T16:24:21.133373+0000","last_fullsized":"2024-09-17T16:24:21.133373+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'2579","last_scrub_stamp":"2024-09-17T16:22:26.788807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:22:26.788807+0000","objects_scrubbed":1278,"log_size":5734,"log_dups_size":0,"ondisk_log_size":5734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48431104,"num_objects":739,"num_object_clones":0,"num_object_copies":2217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3155,"num_write_kb":83712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,9],"acting":[0,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"506'5780","reported_seq":5997,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:20.845730+0000","last_change":"2024-09-17T16:22:29.367096+0000","last_active":"2024-09-17T16:24:20.845730+0000","last_peered":"2024-09-17T16:24:20.845730+0000","last_clean":"2024-09-17T16:24:20.845730+0000","last_became_active":"2024-09-17T16:22:24.973401+0000","last_became_peered":"2024-09-17T16:22:24.973401+0000","last_unstale":"2024-09-17T16:24:20.845730+0000","last_undegraded":"2024-09-17T16:24:20.845730+0000","last_fullsized":"2024-09-17T16:24:20.845730+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'2690","last_scrub_stamp":"2024-09-17T16:22:29.366875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:22:29.366875+0000","objects_scrubbed":1351,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:33.965458+0000","scrub_duration":686,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51052544,"num_objects":779,"num_object_clones":0,"num_object_copies":2337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3094,"num_write_kb":80704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"506'5625","reported_seq":9543,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:21.809559+0000","last_change":"2024-09-17T16:23:36.047157+0000","last_active":"2024-09-17T16:24:21.809559+0000","last_peered":"2024-09-17T16:24:21.809559+0000","last_clean":"2024-09-17T16:24:21.809559+0000","last_became_active":"2024-09-17T16:23:02.974040+0000","last_became_peered":"2024-09-17T16:23:02.974040+0000","last_unstale":"2024-09-17T16:24:21.809559+0000","last_undegraded":"2024-09-17T16:24:21.809559+0000","last_fullsized":"2024-09-17T16:24:21.809559+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'4254","last_scrub_stamp":"2024-09-17T16:23:36.047054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:36.047054+0000","objects_scrubbed":2094,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:51.718675+0000","scrub_duration":1014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46399488,"num_objects":708,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3031,"num_write_kb":78400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1830,"num_bytes_recovered":119930880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,10],"acting":[2,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"505'5627","reported_seq":11073,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-17T16:24:19.754544+0000","last_change":"2024-09-17T16:23:51.981344+0000","last_active":"2024-09-17T16:24:19.754544+0000","last_peered":"2024-09-17T16:24:19.754544+0000","last_clean":"2024-09-17T16:24:19.754544+0000","last_became_active":"2024-09-17T16:22:51.468995+0000","last_became_peered":"2024-09-17T16:22:51.468995+0000","last_unstale":"2024-09-17T16:24:19.754544+0000","last_undegraded":"2024-09-17T16:24:19.754544+0000","last_fullsized":"2024-09-17T16:24:19.754544+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":441,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:21:41.448260+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:24:21.729233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53510144,"num_objects":817,"num_object_clones":0,"num_object_copies":2451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4341,"num_write_kb":165024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1309,"num_bytes_recovered":85786624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"506'5747","reported_seq":8856,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:20.876169+0000","last_change":"2024-09-17T16:22:25.774421+0000","last_active":"2024-09-17T16:24:20.876169+0000","last_peered":"2024-09-17T16:24:20.876169+0000","last_clean":"2024-09-17T16:24:20.876169+0000","last_became_active":"2024-09-17T16:22:24.972827+0000","last_became_peered":"2024-09-17T16:22:24.972827+0000","last_unstale":"2024-09-17T16:24:20.876169+0000","last_undegraded":"2024-09-17T16:24:20.876169+0000","last_fullsized":"2024-09-17T16:24:20.876169+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'2631","last_scrub_stamp":"2024-09-17T16:22:25.774345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:22:25.774345+0000","objects_scrubbed":1342,"log_size":5747,"log_dups_size":0,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:30.054287+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49610752,"num_objects":757,"num_object_clones":0,"num_object_copies":2271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3117,"num_write_kb":81024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,3],"acting":[10,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"506'5755","reported_seq":9748,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:21.097604+0000","last_change":"2024-09-17T16:24:13.607826+0000","last_active":"2024-09-17T16:24:21.097604+0000","last_peered":"2024-09-17T16:24:21.097604+0000","last_clean":"2024-09-17T16:24:21.097604+0000","last_became_active":"2024-09-17T16:24:13.525183+0000","last_became_peered":"2024-09-17T16:24:13.525183+0000","last_unstale":"2024-09-17T16:24:21.097604+0000","last_undegraded":"2024-09-17T16:24:21.097604+0000","last_fullsized":"2024-09-17T16:24:21.097604+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'4788","last_scrub_stamp":"2024-09-17T16:23:50.558144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:50.558144+0000","objects_scrubbed":1727,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:07.018011+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49610752,"num_objects":757,"num_object_clones":0,"num_object_copies":2271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3099,"num_write_kb":80960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1867,"num_bytes_recovered":122290176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"506'5671","reported_seq":5888,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:20.704870+0000","last_change":"2024-09-17T16:22:26.234684+0000","last_active":"2024-09-17T16:24:20.704870+0000","last_peered":"2024-09-17T16:24:20.704870+0000","last_clean":"2024-09-17T16:24:20.704870+0000","last_became_active":"2024-09-17T16:22:24.971915+0000","last_became_peered":"2024-09-17T16:22:24.971915+0000","last_unstale":"2024-09-17T16:24:20.704870+0000","last_undegraded":"2024-09-17T16:24:20.704870+0000","last_fullsized":"2024-09-17T16:24:20.704870+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'2596","last_scrub_stamp":"2024-09-17T16:22:26.234613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:22:26.234613+0000","objects_scrubbed":1314,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:51.262186+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51052544,"num_objects":779,"num_object_clones":0,"num_object_copies":2337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3075,"num_write_kb":81280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"505'5702","reported_seq":5943,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-17T16:24:19.756474+0000","last_change":"2024-09-17T16:23:41.003174+0000","last_active":"2024-09-17T16:24:19.756474+0000","last_peered":"2024-09-17T16:24:19.756474+0000","last_clean":"2024-09-17T16:24:19.756474+0000","last_became_active":"2024-09-17T16:22:24.971393+0000","last_became_peered":"2024-09-17T16:22:24.971393+0000","last_unstale":"2024-09-17T16:24:19.756474+0000","last_undegraded":"2024-09-17T16:24:19.756474+0000","last_fullsized":"2024-09-17T16:24:19.756474+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'4551","last_scrub_stamp":"2024-09-17T16:23:41.003076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:41.003076+0000","objects_scrubbed":1967,"log_size":5702,"log_dups_size":0,"ondisk_log_size":5702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:04.117476+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53018624,"num_objects":809,"num_object_clones":0,"num_object_copies":2427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":81472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"506'5748","reported_seq":5993,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:20.848562+0000","last_change":"2024-09-17T16:23:47.791775+0000","last_active":"2024-09-17T16:24:20.848562+0000","last_peered":"2024-09-17T16:24:20.848562+0000","last_clean":"2024-09-17T16:24:20.848562+0000","last_became_active":"2024-09-17T16:22:24.976710+0000","last_became_peered":"2024-09-17T16:22:24.976710+0000","last_unstale":"2024-09-17T16:24:20.848562+0000","last_undegraded":"2024-09-17T16:24:20.848562+0000","last_fullsized":"2024-09-17T16:24:20.848562+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'4762","last_scrub_stamp":"2024-09-17T16:23:47.791690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:23:47.791690+0000","objects_scrubbed":1783,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:12.891929+0000","scrub_duration":973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51183616,"num_objects":781,"num_object_clones":0,"num_object_copies":2343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3126,"num_write_kb":83008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"506'5622","reported_seq":9521,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:20.798242+0000","last_change":"2024-09-17T16:23:51.011333+0000","last_active":"2024-09-17T16:24:20.798242+0000","last_peered":"2024-09-17T16:24:20.798242+0000","last_clean":"2024-09-17T16:24:20.798242+0000","last_became_active":"2024-09-17T16:23:01.967728+0000","last_became_peered":"2024-09-17T16:23:01.967728+0000","last_unstale":"2024-09-17T16:24:20.798242+0000","last_undegraded":"2024-09-17T16:24:20.798242+0000","last_fullsized":"2024-09-17T16:24:20.798242+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":419,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'2613","last_scrub_stamp":"2024-09-17T16:22:29.950043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:21:41.448260+0000","last_clean_scrub_stamp":"2024-09-17T16:22:29.950043+0000","objects_scrubbed":699,"log_size":5622,"log_dups_size":0,"ondisk_log_size":5622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:20.983721+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51904512,"num_objects":792,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3015,"num_write_kb":78912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1831,"num_bytes_recovered":119996416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":914,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-17T16:24:20.554345+0000","last_change":"2024-09-17T16:22:06.807310+0000","last_active":"2024-09-17T16:24:20.554345+0000","last_peered":"2024-09-17T16:24:20.554345+0000","last_clean":"2024-09-17T16:24:20.554345+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:24:20.554345+0000","last_undegraded":"2024-09-17T16:24:20.554345+0000","last_fullsized":"2024-09-17T16:24:20.554345+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:22:06.807231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:22:06.807231+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:14.054861+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":26,"stat_sum":{"num_bytes":1589411864,"num_objects":24254,"num_object_clones":0,"num_object_copies":72762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104647,"num_write_kb":3049889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14215,"num_bytes_recovered":931528704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5340934144,"data_stored":5340921928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167357,"ondisk_log_size":167357,"up":78,"acting":78,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285746,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":698264,"kb_used_data":319640,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93046888,"statfs":{"total":95995035648,"available":95280013312,"internally_reserved":0,"allocated":327311360,"data_stored":326609682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":387316164},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547383,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":983588,"kb_used_data":663268,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92761564,"statfs":{"total":95995035648,"available":94987841536,"internally_reserved":0,"allocated":679186432,"data_stored":678555020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743612,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":884652,"kb_used_data":452068,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92860500,"statfs":{"total":95995035648,"available":95089152000,"internally_reserved":0,"allocated":462917632,"data_stored":462286220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005250,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":731124,"kb_used_data":497252,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":93014028,"statfs":{"total":95995035648,"available":95246364672,"internally_reserved":0,"allocated":509186048,"data_stored":508554636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963726,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837760,"kb_used_data":451952,"kb_used_omap":41,"kb_used_meta":385750,"kb_avail":92907392,"statfs":{"total":95995035648,"available":95137169408,"internally_reserved":0,"allocated":462798848,"data_stored":462159812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":395008307},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495819,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893916,"kb_used_data":537684,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92851236,"statfs":{"total":95995035648,"available":95079665664,"internally_reserved":0,"allocated":550588416,"data_stored":549890830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724750,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986389,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":810508,"kb_used_data":476300,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92934644,"statfs":{"total":95995035648,"available":95165075456,"internally_reserved":0,"allocated":487731200,"data_stored":487037166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149912,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":766892,"kb_used_data":396772,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92978260,"statfs":{"total":95995035648,"available":95209738240,"internally_reserved":0,"allocated":406294528,"data_stored":405663116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":378796487},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411551,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":600548,"kb_used_data":388516,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93144604,"statfs":{"total":95995035648,"available":95380074496,"internally_reserved":0,"allocated":397840384,"data_stored":397208972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738601,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241248,"kb_used_data":831056,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92503904,"statfs":{"total":95995035648,"available":94723997696,"internally_reserved":0,"allocated":851001344,"data_stored":850303238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":825364480,"data_stored":825360408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":390922240,"data_stored":390922240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":402849792,"data_stored":402849792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":482869248,"data_stored":482869248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":318570496,"data_stored":318570496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":529465344,"data_stored":529465344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":458690560,"data_stored":458686488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":486866944,"data_stored":486866944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":447348736,"data_stored":447348736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":675414016,"data_stored":675414016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":322572288,"data_stored":322568216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:24:26.139 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [3, 1, 8] 2024-09-17T16:24:26.140 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '3', '1', '8'] 2024-09-17T16:24:26.140 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 1.0 3 1 8 2024-09-17T16:24:26.205 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:26.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:26.275 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:26.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:26.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:26.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:26.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:26.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:26.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:26.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:26.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:26.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:24:26.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:26.661 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:26.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:26.672 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:26 smithi017 ceph-mon[27571]: pgmap v1115: 27 pgs: 27 active+clean; 1.4 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-17T16:24:26.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:26.792 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:26.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:26.814 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:24:26.814 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:24:26.844 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:24:26.844 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:24:26.870 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:24:26.870 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:24:26.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:26.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:26 smithi173 ceph-mon[30530]: pgmap v1115: 27 pgs: 27 active+clean; 1.4 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-17T16:24:26.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:24:26.898 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:24:26.918 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:26.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:26.931 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:26.933 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:24:26.934 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:24:26.963 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:24:26.963 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:24:26.979 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:26.990 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:24:26.991 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:24:27.017 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:24:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:24:27.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:27.049 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:24:27.050 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:24:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:27.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:27.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:27.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:26 smithi071 ceph-mon[30002]: pgmap v1115: 27 pgs: 27 active+clean; 1.4 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-17T16:24:27.083 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:24:27.083 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:24:27.108 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:24:27.109 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:24:27.137 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:24:27.137 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:24:27.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:27.175 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:27.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:27.277 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:27.303 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:27.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:27.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:27.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:27.430 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:27.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:24:27.530 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:27.558 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:24:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:24:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:24:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:24:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:143899 3.1 3:8d15b72a:::benchmark_data_smithi173_66272_object71943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-17T16:24:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.475047+0000", 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 427.07898261100001, 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37967805900000001, 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 143899 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475047+0000", 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475047+0000", 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475048+0000", 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.065e-06 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475053+0000", 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8330000000000001e-06 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475067+0000", 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.416e-05 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475075+0000", 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7049999999999999e-06 2024-09-17T16:24:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.478661+0000", 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00358662 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.478828+0000", 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016719200000000001 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.478988+0000", 2024-09-17T16:24:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015920299999999999 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.487257+0000", 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0082690630000000001 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.498341+0000", 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011084053999999999 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.854410+0000", 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35606905799999999 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.854451+0000", 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.155e-05 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.854725+0000", 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00027355599999999999 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1 e290,272 3:8b678a21:::benchmark_data_smithi173_66272_object17829:head,261'5698)", 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.665092+0000", 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 426.88893736199998, 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46414988299999999, 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665092+0000", 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665092+0000", 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665093+0000", 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9599999999999999e-07 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665097+0000", 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2830000000000003e-06 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.128735+0000", 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46363770199999998 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.128747+0000", 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2779000000000001e-05 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.128815+0000", 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7749999999999993e-05 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.129242+0000", 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042677299999999997 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:76068 4.5 e335/301 4:a01bffc8:::benchmark_data_smithi173_77474_object76067:head v 335'4877, mlcod=335'4875)", 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:18:40.016492+0000", 2024-09-17T16:24:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 347.53753740100001, 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39477241400000002, 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016492+0000", 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016492+0000", 2024-09-17T16:24:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016493+0000", 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3659999999999999e-06 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016567+0000", 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3520000000000001e-05 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409280+0000", 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39271265799999999 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409289+0000", 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8789999999999992e-06 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409338+0000", 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9029000000000001e-05 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409355+0000", 2024-09-17T16:24:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7727000000000002e-05 2024-09-17T16:24:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.411210+0000", 2024-09-17T16:24:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018547570000000001 2024-09-17T16:24:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.411265+0000", 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4478000000000003e-05 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(4.a 389/383 [PushReplyOp(4:5cbc523f:::benchmark_data_smithi173_77474_object91105:head)])", 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:20:53.654477+0000", 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 213.89955241600001, 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46374295300000001, 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.654477+0000", 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.654477+0000", 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.654477+0000", 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.55e-07 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.654480+0000", 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.021e-06 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.117956+0000", 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46347616600000002 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.117966+0000", 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4749999999999999e-06 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.118056+0000", 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9608e-05 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.118220+0000", 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016452799999999999 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(4.5 389/383 [PushReplyOp(4:aaa668bb:::benchmark_data_smithi173_77474_object91191:head)])", 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:20:53.787848+0000", 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 213.76618157300001, 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.677856495, 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.787848+0000", 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.787848+0000", 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.787848+0000", 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.06e-07 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.787850+0000", 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3010000000000001e-06 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.465463+0000", 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.677612942 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.465468+0000", 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2089999999999996e-06 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.465525+0000", 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6440999999999997e-05 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.465704+0000", 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017949599999999999 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25516.0:204338 4.7 4:e82f76a0:::benchmark_data_smithi173_77474_object102163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.614506+0000", 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.93952351300001, 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59349127700000004, 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25516", 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 204338 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614506+0000", 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614506+0000", 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614508+0000", 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.745e-06 2024-09-17T16:24:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614514+0000", 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6170000000000003e-06 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614528+0000", 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3977e-05 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614536+0000", 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1030000000000002e-06 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614584+0000", 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7456000000000001e-05 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614674+0000", 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0142000000000005e-05 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614753+0000", 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8603999999999995e-05 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.618482+0000", 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037292520000000002 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.622334+0000", 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038525199999999999 2024-09-17T16:24:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207921+0000", 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58558639499999998 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207962+0000", 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1196999999999999e-05 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207997+0000", 2024-09-17T16:24:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5268999999999997e-05 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25516.0:204340 4.a 4:583138b7:::benchmark_data_smithi173_77474_object102165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.617529+0000", 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.9365004, 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61758273900000005, 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25516", 2024-09-17T16:24:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:24:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 204340 2024-09-17T16:24:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617529+0000", 2024-09-17T16:24:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617529+0000", 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617530+0000", 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617539+0000", 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7830000000000004e-06 2024-09-17T16:24:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617554+0000", 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4467999999999999e-05 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617562+0000", 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8709999999999997e-06 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617616+0000", 2024-09-17T16:24:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4682000000000003e-05 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617738+0000", 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121927 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617861+0000", 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123084 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.621672+0000", 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038110869999999999 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.193552+0000", 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57187970399999999 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235037+0000", 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041484961000000001 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235078+0000", 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0890000000000003e-05 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235112+0000", 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3945000000000001e-05 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25516.0:204338 4.7 e408/306 ondisk, result = 0)", 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.619785+0000", 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.934244936, 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58821733300000001, 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.619785+0000", 2024-09-17T16:24:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.619785+0000", 2024-09-17T16:24:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.619785+0000", 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4799999999999998e-07 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.619788+0000", 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2909999999999999e-06 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207799+0000", 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58801086199999997 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207811+0000", 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1856e-05 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207912+0000", 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100862 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207917+0000", 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1919999999999996e-06 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.208002+0000", 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4722000000000006e-05 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25516.0:204340 4.a e408/393 ondisk, result = 0)", 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.622256+0000", 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.931773518, 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61286083199999997, 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.622256+0000", 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.622256+0000", 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.622257+0000", 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0500000000000003e-07 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.622261+0000", 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3850000000000004e-06 2024-09-17T16:24:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.234958+0000", 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61269643900000004 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.234966+0000", 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2940000000000006e-06 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235024+0000", 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8069000000000003e-05 2024-09-17T16:24:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235032+0000", 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8709999999999997e-06 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235117+0000", 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5068999999999995e-05 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:41596 5.17 5:edc74594:::benchmark_data_smithi173_90735_object41595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e441)", 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:23.961409+0000", 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 123.592620387, 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0288790649999999, 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 41596 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961409+0000", 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961409+0000", 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961410+0000", 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:24:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961444+0000", 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3828000000000002e-05 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961454+0000", 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9760000000000007e-06 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961460+0000", 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9720000000000003e-06 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.032985+0000", 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.071524061 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.032994+0000", 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3030000000000005e-06 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.990202+0000", 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95720833599999999 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.990288+0000", 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5996999999999997e-05 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:41597 5.14 5:2fd5d8a7:::benchmark_data_smithi173_90735_object41596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e441)", 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:23.961480+0000", 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 123.592549758, 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0299093850000001, 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 41597 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961480+0000", 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961480+0000", 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961480+0000", 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.68e-07 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961501+0000", 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0988999999999998e-05 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961509+0000", 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2830000000000004e-06 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961515+0000", 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0390000000000001e-06 2024-09-17T16:24:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.022776+0000", 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.061260904999999997 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.022785+0000", 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2969999999999992e-06 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.991313+0000", 2024-09-17T16:24:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96852803600000004 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.991389+0000", 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6267999999999996e-05 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:41603 5.18 5:1fe2546e:::benchmark_data_smithi173_90735_object41602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e441)", 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:23.961531+0000", 2024-09-17T16:24:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 123.59249853199999, 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.040207801, 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 41603 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961531+0000", 2024-09-17T16:24:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961531+0000", 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961531+0000", 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9100000000000004e-07 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961552+0000", 2024-09-17T16:24:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0920999999999998e-05 2024-09-17T16:24:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961560+0000", 2024-09-17T16:24:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9119999999999998e-06 2024-09-17T16:24:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961566+0000", 2024-09-17T16:24:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:27.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1580000000000002e-06 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036249+0000", 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.074682889000000002 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036257+0000", 2024-09-17T16:24:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7170000000000008e-06 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001493+0000", 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96523577699999996 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001739+0000", 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024593599999999998 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:41610 5.18 5:1c8bbb85:::benchmark_data_smithi173_90735_object41609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e441)", 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:23.961621+0000", 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 123.592408874, 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.040819079, 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 41610 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961621+0000", 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961621+0000", 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961621+0000", 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8100000000000003e-07 2024-09-17T16:24:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961654+0000", 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2790999999999998e-05 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961662+0000", 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7700000000000001e-06 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961668+0000", 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9159999999999998e-06 2024-09-17T16:24:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036392+0000", 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.074724225000000005 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036398+0000", 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2720000000000003e-06 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001792+0000", 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96539429499999996 2024-09-17T16:24:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.002440+0000", 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00064732899999999998 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:41615 5.10 5:09bf280c:::benchmark_data_smithi173_90735_object41614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:24.035550+0000", 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 123.518479027, 2024-09-17T16:24:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96589714400000004, 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 41615 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035550+0000", 2024-09-17T16:24:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035550+0000", 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035553+0000", 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2239999999999998e-06 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035582+0000", 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9641999999999998e-05 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035596+0000", 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3346000000000001e-05 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035606+0000", 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0032e-05 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036421+0000", 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081489300000000002 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036429+0000", 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.799e-06 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.998358+0000", 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.961928597 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.998483+0000", 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012492100000000001 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.998643+0000", 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015998499999999999 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001077+0000", 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002434322 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001309+0000", 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023223699999999999 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001357+0000", 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7206999999999997e-05 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001416+0000", 2024-09-17T16:24:27.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9110000000000002e-05 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001448+0000", 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1829000000000003e-05 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:82755 5.7 5:e29d80c7:::benchmark_data_smithi173_90735_object82754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.908516+0000", 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 67.645513043999998, 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.660800991, 2024-09-17T16:24:27.631 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 82755 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908516+0000", 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908516+0000", 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908518+0000", 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6819999999999999e-06 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908541+0000", 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2518999999999999e-05 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908556+0000", 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5136e-05 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908562+0000", 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1909999999999997e-06 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908608+0000", 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5762e-05 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908709+0000", 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101384 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908805+0000", 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6137999999999993e-05 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.910192+0000", 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013868350000000001 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.914165+0000", 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039727699999999996 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569146+0000", 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65498125500000004 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569247+0000", 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010063799999999999 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569317+0000", 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0680999999999996e-05 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26158.0:82755 5.7 e475/441 ondisk, result = 0)", 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.910504+0000", 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 67.643525147999995, 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65882479199999999, 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.637 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.910504+0000", 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.910504+0000", 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.910505+0000", 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4200000000000005e-07 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.910510+0000", 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4909999999999999e-06 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569010+0000", 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65850046799999995 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569022+0000", 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2257e-05 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569117+0000", 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4108000000000001e-05 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569136+0000", 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9956999999999999e-05 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569329+0000", 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019266899999999999 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:82763 5.13 e475/453 5:cd63b9f8:::benchmark_data_smithi173_90735_object82762:head v 475'4007, mlcod=475'4003)", 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.918146+0000", 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 67.635883481999997, 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62102570099999999, 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.641 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.918146+0000", 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.918146+0000", 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.918147+0000", 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4799999999999998e-07 2024-09-17T16:24:27.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.918166+0000", 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9106999999999998e-05 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.537841+0000", 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61967572500000001 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.537850+0000", 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9560000000000003e-06 2024-09-17T16:24:27.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.537915+0000", 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4109999999999994e-05 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.537939+0000", 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4128999999999999e-05 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.539124+0000", 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011852399999999999 2024-09-17T16:24:27.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.539172+0000", 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7786e-05 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:82768 5.6 e475/463 5:63886666:::benchmark_data_smithi173_90735_object82767:head v 475'3819, mlcod=475'3817)", 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.925908+0000", 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 67.628121026000002, 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83103535299999998, 2024-09-17T16:24:27.645 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.925908+0000", 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.925908+0000", 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.925909+0000", 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4600000000000004e-07 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.925968+0000", 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8801000000000002e-05 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.755595+0000", 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82962728200000002 2024-09-17T16:24:27.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.755603+0000", 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2389999999999996e-06 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.755677+0000", 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4873e-05 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.755695+0000", 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7167999999999999e-05 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.756901+0000", 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012064560000000001 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.756944+0000", 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2787999999999998e-05 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.649 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:95335 5.7 5:e4eaf7db:::benchmark_data_smithi173_90735_object12564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:32.800982+0000", 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 54.753047625000001, 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44480024899999998, 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:24:27.650 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 95335 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.800982+0000", 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.800982+0000", 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.800983+0000", 2024-09-17T16:24:27.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.181e-06 2024-09-17T16:24:27.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.800988+0000", 2024-09-17T16:24:27.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9849999999999997e-06 2024-09-17T16:24:27.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.800999+0000", 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1228e-05 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.801006+0000", 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1679999999999998e-06 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.801055+0000", 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8681999999999999e-05 2024-09-17T16:24:27.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.801174+0000", 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118672 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.801331+0000", 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015736600000000001 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.802689+0000", 2024-09-17T16:24:27.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001357886 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.803076+0000", 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038712499999999999 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245680+0000", 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44260339799999998 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:24:27.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245736+0000", 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6468000000000002e-05 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245782+0000", 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6090000000000001e-05 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26158.0:95335 5.7 e481/441 ondisk, result = 0)", 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:32.803052+0000", 2024-09-17T16:24:27.656 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 54.750977695000003, 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44273757400000002, 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.803052+0000", 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.803052+0000", 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:27.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.803052+0000", 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7899999999999998e-07 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.803056+0000", 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.489e-06 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:27.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245613+0000", 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.442557058 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245618+0000", 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9930000000000003e-06 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245665+0000", 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6489000000000002e-05 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:27.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245674+0000", 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1230000000000006e-06 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245789+0000", 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011574300000000001 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:27.660 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:24:27.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:27.744 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:27.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:27.840 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/255650784' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:27.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:27.870 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:27.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:27.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/255650784' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:27.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:27.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:27.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:28.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/255650784' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:28.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:28.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:28.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:28.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:28.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:28.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:28.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:28.322 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:28.356 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:28.381 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:28.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:28.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:24:28.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:28.511 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:28.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:28.598 INFO:teuthology.orchestra.run.smithi017.stderr:set 1.0 pg_upmap mapping to [3,1,8] 2024-09-17T16:24:28.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:28.648 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:28.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:28.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:28.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:28.781 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:28.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:28.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:28.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:28.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:28.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:28 smithi173 ceph-mon[30530]: pgmap v1116: 27 pgs: 27 active+clean; 1.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-17T16:24:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1759247485' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.0", "id": [3, 1, 8]} : dispatch 2024-09-17T16:24:28.907 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:28.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:28.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:28.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:28 smithi017 ceph-mon[27571]: pgmap v1116: 27 pgs: 27 active+clean; 1.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-17T16:24:28.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1759247485' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.0", "id": [3, 1, 8]} : dispatch 2024-09-17T16:24:29.009 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:29.032 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:29.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:29.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:28 smithi071 ceph-mon[30002]: pgmap v1116: 27 pgs: 27 active+clean; 1.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-17T16:24:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1759247485' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.0", "id": [3, 1, 8]} : dispatch 2024-09-17T16:24:29.135 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:29.157 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:29.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:29.258 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:29.288 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:29.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:29.371 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:24:29.371 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:24:29.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.371 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:149441 5.d e508/482 5:bfc1fd92:::benchmark_data_smithi173_90735_object66670:head v 508'9377, mlcod=508'9374)", 2024-09-17T16:24:29.371 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:29.356532+0000", 2024-09-17T16:24:29.371 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.011689928, 2024-09-17T16:24:29.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011713408999999999, 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.356532+0000", 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.356532+0000", 2024-09-17T16:24:29.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.356533+0000", 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0589999999999999e-06 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.356539+0000", 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5300000000000002e-06 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.356569+0000", 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0025000000000001e-05 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.356576+0000", 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7920000000000004e-06 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:29.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.357668+0000", 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010922169999999999 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.357687+0000", 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8513999999999999e-05 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:149442 5.d e508/482 5:bf242c43:::benchmark_data_smithi173_90735_object66671:head v 508'9378, mlcod=508'9374)", 2024-09-17T16:24:29.375 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:29.359008+0000", 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0092137720000000003, 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0092795959999999993, 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.359008+0000", 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.359008+0000", 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.359009+0000", 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0100000000000003e-07 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.359016+0000", 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8340000000000001e-06 2024-09-17T16:24:29.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.359045+0000", 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8994000000000001e-05 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.359052+0000", 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0430000000000002e-06 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.361688+0000", 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026360849999999998 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.361706+0000", 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7873e-05 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:149443 5.d e508/482 5:b4f22520:::benchmark_data_smithi173_90735_object66672:head v 508'9379, mlcod=508'9374)", 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:29.361671+0000", 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0065503389999999996, 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0066510850000000002, 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.361671+0000", 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.361671+0000", 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.361672+0000", 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3600000000000003e-07 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.361678+0000", 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3739999999999997e-06 2024-09-17T16:24:29.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.361699+0000", 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1610000000000001e-05 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.361705+0000", 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9599999999999997e-06 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.365014+0000", 2024-09-17T16:24:29.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033085380000000002 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.365033+0000", 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.959e-05 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:149448 5.d e508/482)", 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:29.364400+0000", 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.003821404, 2024-09-17T16:24:29.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039661899999999996, 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.364400+0000", 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.364400+0000", 2024-09-17T16:24:29.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.364401+0000", 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9000000000000005e-07 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.364409+0000", 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4250000000000001e-06 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.364440+0000", 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1390999999999998e-05 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:29.364448+0000", 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7600000000000002e-06 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:24:29.386 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 4 2024-09-17T16:24:29.387 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:24:29.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:29.414 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:29.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:29.488 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:24:29.516 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:29.546 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:29.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:29.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:29.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:29.673 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1759247485' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [3, 1, 8]}]': finished 2024-09-17T16:24:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:29 smithi017 ceph-mon[27571]: osdmap e508: 12 total, 12 up, 10 in 2024-09-17T16:24:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/982910702' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:29 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:29 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:24:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:29 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:24:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:29 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:29 smithi017 ceph-mon[27571]: osdmap e509: 12 total, 12 up, 10 in 2024-09-17T16:24:29.774 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:29.799 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:29.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1759247485' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [3, 1, 8]}]': finished 2024-09-17T16:24:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:29 smithi173 ceph-mon[30530]: osdmap e508: 12 total, 12 up, 10 in 2024-09-17T16:24:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/982910702' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:29 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:29 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:24:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:29 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:24:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:29 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:29.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:29 smithi173 ceph-mon[30530]: osdmap e509: 12 total, 12 up, 10 in 2024-09-17T16:24:29.901 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:29.925 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:29.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:29.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:30.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:30.054 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:30.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1759247485' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [3, 1, 8]}]': finished 2024-09-17T16:24:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:29 smithi071 ceph-mon[30002]: osdmap e508: 12 total, 12 up, 10 in 2024-09-17T16:24:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/982910702' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:29 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:29 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:24:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:29 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:24:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:29 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:29 smithi071 ceph-mon[30002]: osdmap e509: 12 total, 12 up, 10 in 2024-09-17T16:24:30.156 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:30.177 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:30.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:30.272 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:24:30.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:30.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:24:30.350 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:30.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:30.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:30.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:30.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:30.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:30.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:30 smithi173 ceph-mon[30530]: pgmap v1119: 27 pgs: 27 active+clean; 1.1 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-09-17T16:24:30.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:24:30.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:24:30.609 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:24:30.617 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:30.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:30.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:30.698 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:30.723 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:30.758 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:30.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:30.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:30.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:30.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:30.902 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:30 smithi017 ceph-mon[27571]: pgmap v1119: 27 pgs: 27 active+clean; 1.1 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-09-17T16:24:30.902 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:24:30.902 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:24:30.902 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:24:30.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:31.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:31.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:31.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:30 smithi071 ceph-mon[30002]: pgmap v1119: 27 pgs: 27 active+clean; 1.1 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-09-17T16:24:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:24:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:24:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:24:31.138 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:31.167 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:31.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:31.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:31.236 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:24:31.237 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:24:31.257 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:24:31.257 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:24:31.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:31.280 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:24:31.280 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:24:31.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:31.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:31.309 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:24:31.309 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:24:31.338 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:24:31.338 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:24:31.369 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:24:31.370 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:24:31.399 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:24:31.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:24:31.402 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:31.430 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:24:31.430 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:24:31.433 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:31.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:31.452 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:24:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:24:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:24:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:24:31.453 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:24:31.460 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:24:31.460 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:24:31.491 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:24:31.492 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:24:31.522 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:24:31.522 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:24:31.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:31.549 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:24:31.549 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:24:31.560 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:31.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:31.662 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:31.686 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:31.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:24:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:24:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:24:31.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1731009366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:31.788 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:31.812 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:31.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:31.865 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:24:31.865 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:24:31.865 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:24:31.865 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1731009366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:31.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:31.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:31.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:32.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:32.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:24:32.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:24:32.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:24:32.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1731009366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:32.071 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:32.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:32.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:24:32.173 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:32.194 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:32.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:32.297 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:32.322 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:32.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:32.419 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:24:32.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:32.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:32.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:32.471 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:24:32.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:32.589 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:32.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:32.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:32.691 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:32.720 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:32.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:32.758 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:32.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:32.844 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:32.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:32.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:32.881 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:32 smithi017 ceph-mon[27571]: 5.5 scrub starts 2024-09-17T16:24:32.881 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:32 smithi017 ceph-mon[27571]: 5.5 scrub ok 2024-09-17T16:24:32.881 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:32 smithi017 ceph-mon[27571]: pgmap v1120: 27 pgs: 27 active+clean; 953 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-09-17T16:24:32.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:32 smithi173 ceph-mon[30530]: 5.5 scrub starts 2024-09-17T16:24:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:32 smithi173 ceph-mon[30530]: 5.5 scrub ok 2024-09-17T16:24:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:32 smithi173 ceph-mon[30530]: pgmap v1120: 27 pgs: 27 active+clean; 953 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-09-17T16:24:32.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:32.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:32.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:32 smithi071 ceph-mon[30002]: 5.5 scrub starts 2024-09-17T16:24:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:32 smithi071 ceph-mon[30002]: 5.5 scrub ok 2024-09-17T16:24:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:32 smithi071 ceph-mon[30002]: pgmap v1120: 27 pgs: 27 active+clean; 953 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.21k op/s 2024-09-17T16:24:33.074 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:33.074 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:33.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:33.098 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:33.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:33.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:33.222 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:33.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:33.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:33.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:33.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:33.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:33.487 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:33.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:33.589 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:33.612 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:33.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:33.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:33.620 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:24:33.714 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:33.739 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:33.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:33.778 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:33 smithi173 ceph-mon[30530]: 5.10 scrub starts 2024-09-17T16:24:33.778 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:33 smithi173 ceph-mon[30530]: 5.10 scrub ok 2024-09-17T16:24:33.778 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3339286866' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:24:33.778 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:33 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:24:33.820 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:24:33.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:33.867 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:33.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:33.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:33 smithi017 ceph-mon[27571]: 5.10 scrub starts 2024-09-17T16:24:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:33 smithi017 ceph-mon[27571]: 5.10 scrub ok 2024-09-17T16:24:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3339286866' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:24:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:33 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:24:33.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:33.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:34.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:33 smithi071 ceph-mon[30002]: 5.10 scrub starts 2024-09-17T16:24:34.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:33 smithi071 ceph-mon[30002]: 5.10 scrub ok 2024-09-17T16:24:34.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3339286866' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:24:34.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:33 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:24:34.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:34.098 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:24:34.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:34.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:34.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:34.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:34.242 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:34.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:34.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:34.373 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:34.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:34.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:34.506 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:34.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:34.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:34.619 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:24:34.620 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:24:34.620 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T16:24:34.620 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-09-17T16:24:34.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-17T16:24:34.636 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:34.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:34.725 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:24:34.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:34 smithi017 ceph-mon[27571]: 5.16 scrub starts 2024-09-17T16:24:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:34 smithi017 ceph-mon[27571]: 5.16 scrub ok 2024-09-17T16:24:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:34 smithi017 ceph-mon[27571]: 5.c scrub starts 2024-09-17T16:24:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:34 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:34 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:24:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:34 smithi017 ceph-mon[27571]: osdmap e510: 12 total, 12 up, 10 in 2024-09-17T16:24:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:34 smithi017 ceph-mon[27571]: pgmap v1122: 27 pgs: 27 active+clean; 903 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 960 op/s 2024-09-17T16:24:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:34 smithi017 ceph-mon[27571]: osdmap e511: 12 total, 12 up, 10 in 2024-09-17T16:24:34.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:34.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:34.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:34 smithi173 ceph-mon[30530]: 5.16 scrub starts 2024-09-17T16:24:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:34 smithi173 ceph-mon[30530]: 5.16 scrub ok 2024-09-17T16:24:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:34 smithi173 ceph-mon[30530]: 5.c scrub starts 2024-09-17T16:24:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:34 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:34 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:24:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:34 smithi173 ceph-mon[30530]: osdmap e510: 12 total, 12 up, 10 in 2024-09-17T16:24:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:34 smithi173 ceph-mon[30530]: pgmap v1122: 27 pgs: 27 active+clean; 903 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 960 op/s 2024-09-17T16:24:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:34 smithi173 ceph-mon[30530]: osdmap e511: 12 total, 12 up, 10 in 2024-09-17T16:24:34.904 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:34.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:34.929 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:34 smithi071 ceph-mon[30002]: 5.16 scrub starts 2024-09-17T16:24:34.929 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:34 smithi071 ceph-mon[30002]: 5.16 scrub ok 2024-09-17T16:24:34.930 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:34 smithi071 ceph-mon[30002]: 5.c scrub starts 2024-09-17T16:24:34.930 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:34 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:34.930 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:34 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:24:34.930 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:34 smithi071 ceph-mon[30002]: osdmap e510: 12 total, 12 up, 10 in 2024-09-17T16:24:34.930 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:34 smithi071 ceph-mon[30002]: pgmap v1122: 27 pgs: 27 active+clean; 903 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 960 op/s 2024-09-17T16:24:34.930 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:34 smithi071 ceph-mon[30002]: osdmap e511: 12 total, 12 up, 10 in 2024-09-17T16:24:35.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:35.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:35.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:35.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:35.142 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:35.165 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:35.172 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:35.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:35.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:35.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:35.301 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:35.302 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:35.338 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:35.338 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:35.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:35.431 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:35.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:35.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:35.559 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:35.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:35.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:35.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:35.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:35.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:35.788 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:35.815 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:35.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:35.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:35.917 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:35.946 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:35.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:36.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:24:36.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:36.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:36.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:36.175 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:36.201 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:36.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:36.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:36.334 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:36.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:36.337 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:24:36.358 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:24:36.358 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:24:36.387 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:24:36.388 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:24:36.415 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:24:36.415 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:24:36.437 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:36.443 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:24:36.443 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:24:36.464 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:36.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:36.472 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:24:36.472 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:24:36.496 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:24:36.496 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:24:36.525 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:24:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:24:36.534 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:24:36.555 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:24:36.555 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:24:36.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:36.585 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:24:36.585 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:24:36.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:36.597 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:36.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:36.617 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:24:36.617 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:24:36.648 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:24:36.648 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:24:36.677 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:24:36.677 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:24:36.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:36.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:36.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:36.828 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:36.860 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:36.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:36.904 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:24:36.904 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:24:36.904 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:24:36.905 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:24:36.905 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:24:36.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:36.976 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-17T16:24:36.989 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:36.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:37.091 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:37.115 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:37.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:37.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:36 smithi071 ceph-mon[30002]: pgmap v1124: 27 pgs: 27 active+clean; 760 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-17T16:24:37.117 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1217341561' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:24:37.117 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:36 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:24:37.117 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1588257100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:37.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:37.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:36 smithi017 ceph-mon[27571]: pgmap v1124: 27 pgs: 27 active+clean; 760 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-17T16:24:37.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1217341561' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:24:37.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:36 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:24:37.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1588257100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:37.239 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:37.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:37.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:37.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:37.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:37.372 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:36 smithi173 ceph-mon[30530]: pgmap v1124: 27 pgs: 27 active+clean; 760 MiB data, 5.9 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-17T16:24:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1217341561' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:24:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:36 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:24:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1588257100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:37.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:37.499 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:37.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:37.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:37.620 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:24:37.627 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:37.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:37.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:37.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:37.757 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:37.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:37.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:37.859 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:37.884 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:37.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:37.930 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:24:37.986 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:38.008 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:38.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:38.110 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:38.135 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:38.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:38.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:38.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:37 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:24:38.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:37 smithi017 ceph-mon[27571]: osdmap e512: 12 total, 12 up, 10 in 2024-09-17T16:24:38.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:38.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:38.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:37 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:24:38.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:37 smithi071 ceph-mon[30002]: osdmap e512: 12 total, 12 up, 10 in 2024-09-17T16:24:38.363 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:38.390 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:38.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:38.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:37 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:24:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:37 smithi173 ceph-mon[30530]: osdmap e512: 12 total, 12 up, 10 in 2024-09-17T16:24:38.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:38.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:38.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:38.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:38.649 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:38.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:38.699 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:24:38.699 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:24:38.699 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:24:38.699 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:24:38.699 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(3.f 240/239 [PushReplyOp(3:f5f4cfa1:::benchmark_data_smithi173_66272_object320:head)])", 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:14.963777+0000", 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 563.73249694200001, 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58111184800000004, 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963777+0000", 2024-09-17T16:24:38.700 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963777+0000", 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963778+0000", 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1500000000000003e-07 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.701 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:14.963783+0000", 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9540000000000003e-06 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.542775+0000", 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57899182000000005 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.542785+0000", 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7559999999999994e-06 2024-09-17T16:24:38.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.542840+0000", 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5593000000000001e-05 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:15.544889+0000", 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020488099999999999 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.703 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(3.f 241/239 [PushReplyOp(3:f282ca6b:::benchmark_data_smithi173_66272_object3185:head)])", 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:15:17.042771+0000", 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 561.65350336500001, 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86596120499999996, 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.704 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.042771+0000", 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.042771+0000", 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.042771+0000", 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5700000000000002e-07 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.042776+0000", 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2239999999999997e-06 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.906632+0000", 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86385691099999995 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.906655+0000", 2024-09-17T16:24:38.706 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2901999999999999e-05 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.906737+0000", 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1955999999999999e-05 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:15:17.908732+0000", 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019945549999999999 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.707 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:124271 3.f 3.7598f19f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:48.913907+0000", 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 469.78236730899999, 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44303875500000001, 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:38.708 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 124271 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913907+0000", 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913907+0000", 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.710 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.710 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913908+0000", 2024-09-17T16:24:38.710 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.761e-06 2024-09-17T16:24:38.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.710 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.710 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913917+0000", 2024-09-17T16:24:38.710 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9660000000000002e-06 2024-09-17T16:24:38.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913932+0000", 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4611e-05 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.913942+0000", 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5200000000000003e-06 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.914003+0000", 2024-09-17T16:24:38.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1857000000000001e-05 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.914010+0000", 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7939999999999997e-06 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.922734+0000", 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0087233880000000003 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.712 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.922745+0000", 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1299e-05 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:49.356907+0000", 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43416214800000003 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:49.356945+0000", 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8411000000000002e-05 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:124274 3.f 3.5367ea5f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:48.919228+0000", 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 469.77704573300002, 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43814494599999998, 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.714 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:24:38.715 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.715 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:24:38.715 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:38.715 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 124274 2024-09-17T16:24:38.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919228+0000", 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919228+0000", 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.716 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919229+0000", 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0729999999999999e-06 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919239+0000", 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1460000000000002e-06 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.717 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919249+0000", 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0900000000000001e-05 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.919257+0000", 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3350000000000001e-06 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.922889+0000", 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036324040000000001 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:48.922897+0000", 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6490000000000008e-06 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:49.357363+0000", 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43446637799999999 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:49.357373+0000", 2024-09-17T16:24:38.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0061000000000001e-05 2024-09-17T16:24:38.720 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.720 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.720 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.720 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:124271 3.f 3:f98f19ae:::benchmark_data_smithi173_66272_object52315:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:24:38.720 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:50.373045+0000", 2024-09-17T16:24:38.720 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 468.32322872600002, 2024-09-17T16:24:38.720 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53754313499999995, 2024-09-17T16:24:38.720 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 124271 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373045+0000", 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.721 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373045+0000", 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373047+0000", 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1019999999999999e-06 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373074+0000", 2024-09-17T16:24:38.722 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6693999999999999e-05 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373094+0000", 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9723000000000001e-05 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373119+0000", 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5072e-05 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373201+0000", 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1788999999999999e-05 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373225+0000", 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4304e-05 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.893185+0000", 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51995968400000003 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.893329+0000", 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014477399999999999 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9,11", 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.893431+0000", 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010175600000000001 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.901505+0000", 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0080735990000000007 2024-09-17T16:24:38.725 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.901549+0000", 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4211000000000001e-05 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.908942+0000", 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0073930949999999997 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910371+0000", 2024-09-17T16:24:38.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001429156 2024-09-17T16:24:38.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910454+0000", 2024-09-17T16:24:38.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3088000000000001e-05 2024-09-17T16:24:38.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910588+0000", 2024-09-17T16:24:38.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013408800000000001 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:124274 3.f 3:fa57e6ca:::benchmark_data_smithi173_66272_object52318:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:50.373143+0000", 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 468.32313124299998, 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53763810700000003, 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.728 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 124274 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373143+0000", 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.729 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373143+0000", 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373144+0000", 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4e-07 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373169+0000", 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5327e-05 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.730 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373179+0000", 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0419e-05 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373187+0000", 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3209999999999999e-06 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373273+0000", 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6007000000000005e-05 2024-09-17T16:24:38.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373287+0000", 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4559000000000001e-05 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.896946+0000", 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52365888000000005 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.897031+0000", 2024-09-17T16:24:38.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4889999999999995e-05 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9,11", 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.897106+0000", 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5129000000000005e-05 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.903625+0000", 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0065190990000000004 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.904361+0000", 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00073532600000000001 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.909012+0000", 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0046517360000000001 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910655+0000", 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016429190000000001 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910703+0000", 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7757999999999999e-05 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910781+0000", 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7897000000000003e-05 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.735 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:124292 3.f 3:f8d3325b:::benchmark_data_smithi173_66272_object52336:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:16:50.373224+0000", 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 468.32304963299998, 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53778705599999999, 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 124292 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373224+0000", 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373224+0000", 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373225+0000", 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8700000000000005e-07 2024-09-17T16:24:38.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373229+0000", 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0790000000000002e-06 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373238+0000", 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8130000000000002e-06 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373366+0000", 2024-09-17T16:24:38.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012767799999999999 2024-09-17T16:24:38.739 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.739 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.739 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373644+0000", 2024-09-17T16:24:38.739 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027804400000000002 2024-09-17T16:24:38.739 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.373651+0000", 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9020000000000002e-06 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.898880+0000", 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52522909900000003 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.740 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.898963+0000", 2024-09-17T16:24:38.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3573999999999996e-05 2024-09-17T16:24:38.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9,11", 2024-09-17T16:24:38.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.899041+0000", 2024-09-17T16:24:38.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7885000000000004e-05 2024-09-17T16:24:38.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.903635+0000", 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0045940720000000003 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.908105+0000", 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0044698480000000002 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.909071+0000", 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00096629500000000004 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910837+0000", 2024-09-17T16:24:38.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017652670000000001 2024-09-17T16:24:38.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.910896+0000", 2024-09-17T16:24:38.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8987999999999998e-05 2024-09-17T16:24:38.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:16:50.911011+0000", 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011572499999999999 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25036.0:132738 3.f 3:f3b4f8b2:::benchmark_data_smithi173_66272_object60782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:03.472328+0000", 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 455.22394583400001, 2024-09-17T16:24:38.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.452172817, 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25036", 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 132738 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472328+0000", 2024-09-17T16:24:38.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472328+0000", 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472330+0000", 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9810000000000002e-06 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472338+0000", 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5530000000000004e-06 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472356+0000", 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7867e-05 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472363+0000", 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.678e-06 2024-09-17T16:24:38.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472414+0000", 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0523999999999999e-05 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472487+0000", 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2974999999999998e-05 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9,11", 2024-09-17T16:24:38.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.472600+0000", 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000112915 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.477969+0000", 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0053689280000000002 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.478014+0000", 2024-09-17T16:24:38.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5596000000000002e-05 2024-09-17T16:24:38.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.478619+0000", 2024-09-17T16:24:38.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00060512900000000004 2024-09-17T16:24:38.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924344+0000", 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44572508900000002 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924388+0000", 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3427999999999998e-05 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924501+0000", 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011315399999999999 2024-09-17T16:24:38.751 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25036.0:132738 3.f e278/273 ondisk, result = 0)", 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:17:03.476714+0000", 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 455.21956029500001, 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44779243800000001, 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.476714+0000", 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.476714+0000", 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.476714+0000", 2024-09-17T16:24:38.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4900000000000001e-07 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.476717+0000", 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0359999999999998e-06 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924244+0000", 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.447526912 2024-09-17T16:24:38.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924251+0000", 2024-09-17T16:24:38.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9249999999999998e-06 2024-09-17T16:24:38.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924318+0000", 2024-09-17T16:24:38.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7082999999999995e-05 2024-09-17T16:24:38.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924337+0000", 2024-09-17T16:24:38.758 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8723999999999999e-05 2024-09-17T16:24:38.758 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.758 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.758 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:17:03.924506+0000", 2024-09-17T16:24:38.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000169309 2024-09-17T16:24:38.759 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.759 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.759 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.760 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:76054 4.3 4:c3ed8a0e:::benchmark_data_smithi173_77474_object76053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-17T16:24:38.760 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:18:39.803227+0000", 2024-09-17T16:24:38.760 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 358.89304681900001, 2024-09-17T16:24:38.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60180096699999996, 2024-09-17T16:24:38.760 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.761 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.761 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.761 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.761 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:24:38.761 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:24:38.762 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 76054 2024-09-17T16:24:38.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.762 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803227+0000", 2024-09-17T16:24:38.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803227+0000", 2024-09-17T16:24:38.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803229+0000", 2024-09-17T16:24:38.765 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.902e-06 2024-09-17T16:24:38.765 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.766 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803254+0000", 2024-09-17T16:24:38.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5031000000000001e-05 2024-09-17T16:24:38.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803268+0000", 2024-09-17T16:24:38.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4273e-05 2024-09-17T16:24:38.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.767 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803276+0000", 2024-09-17T16:24:38.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0960000000000003e-06 2024-09-17T16:24:38.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803323+0000", 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7163000000000002e-05 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803421+0000", 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8788000000000004e-05 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.803511+0000", 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9868000000000001e-05 2024-09-17T16:24:38.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.770 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.770 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.770 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.805487+0000", 2024-09-17T16:24:38.770 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:38.771 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001975405 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.806491+0000", 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00100475 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404816+0000", 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59832433699999998 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404935+0000", 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011914200000000001 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.405028+0000", 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3212000000000001e-05 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:76054 4.3 e335/306 ondisk, result = 0)", 2024-09-17T16:24:38.773 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:18:39.805871+0000", 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 358.89040346100001, 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59919401100000003, 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.805871+0000", 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.805871+0000", 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.805871+0000", 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9500000000000002e-07 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:39.805875+0000", 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.827e-06 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404595+0000", 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59872037099999997 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404608+0000", 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2846e-05 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404771+0000", 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016244100000000001 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.404797+0000", 2024-09-17T16:24:38.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6092e-05 2024-09-17T16:24:38.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.405065+0000", 2024-09-17T16:24:38.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026773900000000002 2024-09-17T16:24:38.777 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.777 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:140458 4.3 4:c40a4f04:::benchmark_data_smithi173_77474_object38283:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:54.079732+0000", 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 284.61654196299997, 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50211494899999998, 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:24:38.778 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 140458 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079732+0000", 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079732+0000", 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079734+0000", 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5e-06 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079741+0000", 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7340000000000008e-06 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079764+0000", 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2946999999999999e-05 2024-09-17T16:24:38.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079773+0000", 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7430000000000008e-06 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079834+0000", 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1323999999999998e-05 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079922+0000", 2024-09-17T16:24:38.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8029999999999996e-05 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.079993+0000", 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0492000000000002e-05 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.083444+0000", 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0034509950000000001 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.084011+0000", 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00056759900000000001 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581774+0000", 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49776288600000002 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581815+0000", 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1185e-05 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581847+0000", 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1513999999999997e-05 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:140458 4.3 e361/306 ondisk, result = 0)", 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:54.084283+0000", 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 284.611991407, 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49756956800000002, 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:24:38.784 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.084283+0000", 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.084283+0000", 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.084283+0000", 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8999999999999996e-07 2024-09-17T16:24:38.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.084290+0000", 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3049999999999998e-06 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581691+0000", 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49740152500000001 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581699+0000", 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3429999999999996e-06 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581763+0000", 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3686e-05 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581770+0000", 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0840000000000003e-06 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:54.581852+0000", 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2034999999999996e-05 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:153098 4.6 e368/361 4:68123cde:::benchmark_data_smithi173_77474_object50923:head v 368'9358, mlcod=368'9355)", 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:09.619552+0000", 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 269.07672207299998, 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45652909000000003, 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.788 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.619552+0000", 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.619552+0000", 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.619553+0000", 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2600000000000001e-07 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:09.619560+0000", 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2180000000000002e-06 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.072256+0000", 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45269624400000003 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.072262+0000", 2024-09-17T16:24:38.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3079999999999998e-06 2024-09-17T16:24:38.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.072331+0000", 2024-09-17T16:24:38.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9907999999999995e-05 2024-09-17T16:24:38.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.072355+0000", 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3960000000000001e-05 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.076033+0000", 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036773629999999999 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:10.076081+0000", 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8263000000000002e-05 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:82754 5.d 5:becaa051:::benchmark_data_smithi173_90735_object82753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:19.907024+0000", 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 78.789250331999995, 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.581173257, 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.793 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 82754 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.907024+0000", 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.907024+0000", 2024-09-17T16:24:38.794 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.907025+0000", 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.689e-06 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.907134+0000", 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010814099999999999 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.907144+0000", 2024-09-17T16:24:38.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0913999999999999e-05 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.907150+0000", 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3170000000000001e-06 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.907204+0000", 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4163999999999998e-05 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.907359+0000", 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015478599999999999 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.907535+0000", 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017618399999999999 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.909658+0000", 2024-09-17T16:24:38.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00212276 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.916741+0000", 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0070829159999999999 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.487590+0000", 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.570849351 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.487638+0000", 2024-09-17T16:24:38.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7791999999999999e-05 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.488197+0000", 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055924299999999996 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:82762 5.d 5:bc524506:::benchmark_data_smithi173_90735_object82761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:19.915720+0000", 2024-09-17T16:24:38.799 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 78.780553949999998, 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83921830799999997, 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 82762 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.915720+0000", 2024-09-17T16:24:38.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.915720+0000", 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.915721+0000", 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1540000000000001e-06 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.915785+0000", 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3551000000000002e-05 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.915796+0000", 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0879e-05 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.915801+0000", 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9699999999999998e-06 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.916779+0000", 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00097800500000000011 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.916925+0000", 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000146501 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.917092+0000", 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000167262 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.919546+0000", 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002453733 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.922030+0000", 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024840190000000001 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.754033+0000", 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83200266300000003 2024-09-17T16:24:38.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.754099+0000", 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6612000000000005e-05 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.754938+0000", 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00083895900000000002 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26158.0:82754 5.d e475/455 ondisk, result = 0)", 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:19.919619+0000", 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 78.776655460000001, 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56858865400000003, 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.919619+0000", 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.919619+0000", 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.919619+0000", 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8199999999999999e-07 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:19.919623+0000", 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7529999999999999e-06 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.487533+0000", 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56791025699999997 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.487539+0000", 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2419999999999997e-06 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.487580+0000", 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0030999999999997e-05 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.487586+0000", 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1260000000000003e-06 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:20.488207+0000", 2024-09-17T16:24:38.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000621563 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:114282 5.d 5:bfd83ddc:::benchmark_data_smithi173_90735_object31511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e489)", 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:53.959573+0000", 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 44.736701230000001, 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33682278199999999, 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.810 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 114282 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.959573+0000", 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.959573+0000", 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.959574+0000", 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.266e-06 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.959579+0000", 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.724e-06 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.959592+0000", 2024-09-17T16:24:38.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3220999999999999e-05 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.959599+0000", 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0550000000000002e-06 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.959652+0000", 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2655999999999999e-05 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.959761+0000", 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010953199999999999 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.959855+0000", 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4085000000000005e-05 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.962997+0000", 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0031420850000000002 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:53.963054+0000", 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6804e-05 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.296299+0000", 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33324458000000001 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.296351+0000", 2024-09-17T16:24:38.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2463000000000002e-05 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.296396+0000", 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4311000000000003e-05 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:114359 5.10 5:0e1ae67b:::benchmark_data_smithi173_90735_object31588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:54.097959+0000", 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 44.598315059000001, 2024-09-17T16:24:38.816 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21669380799999999, 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 114359 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.097959+0000", 2024-09-17T16:24:38.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.097959+0000", 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.097961+0000", 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8870000000000001e-06 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.097969+0000", 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0239999999999997e-06 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.097989+0000", 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9673999999999998e-05 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.097998+0000", 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6719999999999999e-06 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.098056+0000", 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8133000000000001e-05 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.098153+0000", 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6368999999999998e-05 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,8", 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.098226+0000", 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3621999999999994e-05 2024-09-17T16:24:38.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.821 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.821 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.821 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.099969+0000", 2024-09-17T16:24:38.821 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017427 2024-09-17T16:24:38.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.821 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.821 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.821 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.310682+0000", 2024-09-17T16:24:38.821 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21071340899999999 2024-09-17T16:24:38.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.314524+0000", 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038418079999999999 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.314585+0000", 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0726000000000001e-05 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:54.314653+0000", 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7784e-05 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:150237 5.d 5:b701c5ae:::benchmark_data_smithi173_90735_object67466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e509)", 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:24:30.120668+0000", 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.5756064629999997, 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.189497106, 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:38.823 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 150237 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.120668+0000", 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.120668+0000", 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.120669+0000", 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3799999999999999e-06 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.120674+0000", 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7890000000000002e-06 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.120685+0000", 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0862e-05 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.120692+0000", 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5109999999999997e-06 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.120737+0000", 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4431999999999997e-05 2024-09-17T16:24:38.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.120824+0000", 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7782999999999996e-05 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.120889+0000", 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4573000000000007e-05 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.124336+0000", 2024-09-17T16:24:38.827 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0034471279999999998 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.124637+0000", 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000300838 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.310076+0000", 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.185438719 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.310123+0000", 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6968000000000002e-05 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:30.310165+0000", 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2123000000000003e-05 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:24:38.829 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:24:38.851 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:38.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:38.938 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:38.952 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:38.975 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:38.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:39.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:39.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:39.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:39.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:39.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:39.227 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:38 smithi017 ceph-mon[27571]: pgmap v1126: 27 pgs: 27 active+clean; 650 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 806 op/s 2024-09-17T16:24:39.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:39.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:39.295 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:38 smithi173 ceph-mon[30530]: pgmap v1126: 27 pgs: 27 active+clean; 650 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 806 op/s 2024-09-17T16:24:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:38 smithi071 ceph-mon[30002]: pgmap v1126: 27 pgs: 27 active+clean; 650 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 806 op/s 2024-09-17T16:24:39.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:39.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:39.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:39.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:39.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:39.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:39.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:39.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:39.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:24:39.630 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:39.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:39.646 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:24:39.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:39.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:39.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:39.866 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:39.902 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:39.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:39.989 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:40.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:40.026 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:40.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:40.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:40.128 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:40.155 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:40.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:40.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:40.284 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:40.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:40.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:40.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:40.410 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:40.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:40.497 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:24:40.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:40.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:40.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:40.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:24:40.591 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:24:40.616 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:24:40.616 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:24:40.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:40.642 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:24:40.642 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:24:40.665 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:40.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:40.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:40.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:24:40.669 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:24:40.697 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:24:40.697 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:24:40.722 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:24:40.722 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:24:40.722 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:40 smithi017 ceph-mon[27571]: osdmap e513: 12 total, 12 up, 10 in 2024-09-17T16:24:40.722 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3170600245' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:40.722 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:40 smithi017 ceph-mon[27571]: pgmap v1128: 27 pgs: 27 active+clean; 514 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-17T16:24:40.749 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:24:40.750 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:24:40.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:40.792 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:40.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:40.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:40 smithi071 ceph-mon[30002]: osdmap e513: 12 total, 12 up, 10 in 2024-09-17T16:24:40.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3170600245' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:40 smithi071 ceph-mon[30002]: pgmap v1128: 27 pgs: 27 active+clean; 514 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-17T16:24:40.879 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:24:40.879 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:24:40.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:40 smithi173 ceph-mon[30530]: osdmap e513: 12 total, 12 up, 10 in 2024-09-17T16:24:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3170600245' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:40 smithi173 ceph-mon[30530]: pgmap v1128: 27 pgs: 27 active+clean; 514 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-17T16:24:40.905 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:24:40.905 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:24:40.919 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:40.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:40.934 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:24:40.934 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:24:40.957 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:24:40.958 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:24:40.982 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:24:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:24:41.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:41.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:41.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:41.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:41.176 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:41.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:41.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:41.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:41.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:41.396 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:41.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:41.432 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:41.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:41.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:41.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:41.562 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:41.663 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:41.686 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:41.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:41.788 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:41.813 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:41.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:41.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:41 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3170600245' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:41 smithi071 ceph-mon[30002]: osdmap e514: 12 total, 12 up, 10 in 2024-09-17T16:24:41.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2501947377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:41 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3170600245' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:41 smithi173 ceph-mon[30530]: osdmap e514: 12 total, 12 up, 10 in 2024-09-17T16:24:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2501947377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:41.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:41.936 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:41.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:41.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:41 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:41.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3170600245' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:41.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:41 smithi017 ceph-mon[27571]: osdmap e514: 12 total, 12 up, 10 in 2024-09-17T16:24:41.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2501947377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:42.037 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:42.061 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:42.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:42.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:42.183 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:42.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:42.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:42.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:42.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:42.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:42.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:42.506 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:42.530 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:42.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:42.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:42.655 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:42.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:42.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:42.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:42.767 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: pgmap v1130: 27 pgs: 27 active+clean; 347 MiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-17T16:24:42.767 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=53,logm=3} crc {auth=3208584350,config=1059521453,health=179246142,kv=2641251154,logm=1949879301}) 2024-09-17T16:24:42.767 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1127649056}) 2024-09-17T16:24:42.767 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=199043636}) 2024-09-17T16:24:42.767 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247098686}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4149124979}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3000189615}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=48144009}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2977400650}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814352666}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1937147402}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3521005173}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3997911498}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=436516106}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3208278443}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=810497528}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3567929109}) 2024-09-17T16:24:42.768 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417916045}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4125508323}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3221565618}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3117088085}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=582781061}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743064733}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3147451828}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=648416419}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830814128}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2270489636}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833227794}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=640463276}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576497232}) 2024-09-17T16:24:42.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=853810201}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4170883373}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887247600}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2980387970}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4000467653}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2959921682}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564790666}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3205343609}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592430276}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4147036856}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3247308318}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1797474315}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949380070}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=148125914}) 2024-09-17T16:24:42.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3391883447}) 2024-09-17T16:24:42.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=29,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=33} crc {logm=657648321,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=208369040,mon_config_key=2258063202}) 2024-09-17T16:24:42.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=719589365}) 2024-09-17T16:24:42.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2250919368}) 2024-09-17T16:24:42.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=52,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=27} crc {mon_config_key=3803891733,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=543506709,osdmap=317275749}) 2024-09-17T16:24:42.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1956319345}) 2024-09-17T16:24:42.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=3885517342}) 2024-09-17T16:24:42.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:42.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: pgmap v1130: 27 pgs: 27 active+clean; 347 MiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-17T16:24:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=53,logm=3} crc {auth=3208584350,config=1059521453,health=179246142,kv=2641251154,logm=1949879301}) 2024-09-17T16:24:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1127649056}) 2024-09-17T16:24:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=199043636}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247098686}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4149124979}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3000189615}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=48144009}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2977400650}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814352666}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1937147402}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3521005173}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3997911498}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=436516106}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3208278443}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=810497528}) 2024-09-17T16:24:42.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3567929109}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417916045}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4125508323}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3221565618}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3117088085}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=582781061}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743064733}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3147451828}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=648416419}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830814128}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2270489636}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833227794}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=640463276}) 2024-09-17T16:24:42.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576497232}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=853810201}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4170883373}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887247600}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2980387970}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4000467653}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2959921682}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564790666}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3205343609}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592430276}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4147036856}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3247308318}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1797474315}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949380070}) 2024-09-17T16:24:42.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=148125914}) 2024-09-17T16:24:42.818 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3391883447}) 2024-09-17T16:24:42.818 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=29,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=33} crc {logm=657648321,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=208369040,mon_config_key=2258063202}) 2024-09-17T16:24:42.818 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=719589365}) 2024-09-17T16:24:42.818 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2250919368}) 2024-09-17T16:24:42.818 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=52,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=27} crc {mon_config_key=3803891733,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=543506709,osdmap=317275749}) 2024-09-17T16:24:42.818 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1956319345}) 2024-09-17T16:24:42.818 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=3885517342}) 2024-09-17T16:24:42.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: pgmap v1130: 27 pgs: 27 active+clean; 347 MiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=53,logm=3} crc {auth=3208584350,config=1059521453,health=179246142,kv=2641251154,logm=1949879301}) 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1127649056}) 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=199043636}) 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247098686}) 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4149124979}) 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3000189615}) 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=48144009}) 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2977400650}) 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1814352666}) 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1937147402}) 2024-09-17T16:24:42.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3521005173}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3997911498}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=436516106}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3208278443}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=810497528}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3567929109}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=417916045}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4125508323}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3221565618}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3117088085}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=582781061}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743064733}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3147451828}) 2024-09-17T16:24:42.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=648416419}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830814128}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2270489636}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1833227794}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=640463276}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576497232}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=853810201}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4170883373}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=887247600}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2980387970}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4000467653}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2959921682}) 2024-09-17T16:24:42.898 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=564790666}) 2024-09-17T16:24:42.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3205343609}) 2024-09-17T16:24:42.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3592430276}) 2024-09-17T16:24:42.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4147036856}) 2024-09-17T16:24:42.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3247308318}) 2024-09-17T16:24:42.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1797474315}) 2024-09-17T16:24:42.899 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1949380070}) 2024-09-17T16:24:42.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=148125914}) 2024-09-17T16:24:42.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3391883447}) 2024-09-17T16:24:42.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=29,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=33} crc {logm=657648321,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=208369040,mon_config_key=2258063202}) 2024-09-17T16:24:42.900 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=719589365}) 2024-09-17T16:24:42.901 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2250919368}) 2024-09-17T16:24:42.901 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=52,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=27} crc {mon_config_key=3803891733,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=543506709,osdmap=317275749}) 2024-09-17T16:24:42.901 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1956319345}) 2024-09-17T16:24:42.901 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=46} crc {osdmap=3885517342}) 2024-09-17T16:24:42.913 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:42.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:43.015 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:43.040 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:43.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:43.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:43.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:43.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:43.173 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:43.173 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:43.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:43.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:43.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:43.396 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:43.421 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:43.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:43.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:43.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:43.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:43.650 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:43.674 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:43.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:43.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:43.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:43 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:24:43.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:43 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:24:43.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:43 smithi017 ceph-mon[27571]: osdmap e515: 12 total, 12 up, 10 in 2024-09-17T16:24:43.776 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:43.798 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:43.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:43.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:43 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:24:43.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:43 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:24:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:43 smithi071 ceph-mon[30002]: osdmap e515: 12 total, 12 up, 10 in 2024-09-17T16:24:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:43 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:24:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:43 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:24:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:43 smithi173 ceph-mon[30530]: osdmap e515: 12 total, 12 up, 10 in 2024-09-17T16:24:43.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:43.921 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:24:43.921 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:24:43.921 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T16:24:43.922 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.734927 2024-09-17T16:24:43.922 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 0.7349273213347179 2024-09-17T16:24:43.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:43.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:43.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:43.993 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:24:44.029 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:44.074 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:44.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:44.175 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:44.200 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:44.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:44.302 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:44.332 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:44.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:44.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:44.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:44.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:44.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:44.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:24:44.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:44.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:24:44.586 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:44.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:44.615 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:44.626 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:44.687 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:44.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:44.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:44 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:24:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:44 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:24:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:44 smithi071 ceph-mon[30002]: osdmap e516: 12 total, 12 up, 10 in 2024-09-17T16:24:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:44 smithi071 ceph-mon[30002]: pgmap v1133: 27 pgs: 27 active+clean; 303 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 982 op/s 2024-09-17T16:24:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:44 smithi071 ceph-mon[30002]: osdmap e517: 12 total, 12 up, 10 in 2024-09-17T16:24:44.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:44.827 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:44 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:24:44.827 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:44 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:24:44.827 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:44 smithi017 ceph-mon[27571]: osdmap e516: 12 total, 12 up, 10 in 2024-09-17T16:24:44.827 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:44 smithi017 ceph-mon[27571]: pgmap v1133: 27 pgs: 27 active+clean; 303 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 982 op/s 2024-09-17T16:24:44.827 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:44 smithi017 ceph-mon[27571]: osdmap e517: 12 total, 12 up, 10 in 2024-09-17T16:24:44.846 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:44.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:44 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:24:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:44 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:24:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:44 smithi173 ceph-mon[30530]: osdmap e516: 12 total, 12 up, 10 in 2024-09-17T16:24:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:44 smithi173 ceph-mon[30530]: pgmap v1133: 27 pgs: 27 active+clean; 303 MiB data, 4.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 982 op/s 2024-09-17T16:24:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:44 smithi173 ceph-mon[30530]: osdmap e517: 12 total, 12 up, 10 in 2024-09-17T16:24:44.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:44.973 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:44.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:44.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:45.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:45.100 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:45.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:45.202 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:45.225 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:45.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:45.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:45.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:45.354 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:45.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:45.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:45.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:45.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:45.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:45.606 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:45.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:45.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:45.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:45.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:45.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:45 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:24:45.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:45 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:24:45.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:45 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:24:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:45 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:24:45.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:45.858 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:45.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:45 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:24:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:45 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:24:45.899 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:45.904 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:45.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:45.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:45.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:46.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:46.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:46.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:46.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:46.236 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:46.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:46.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:46.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:46.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:46.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:46.462 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:46.463 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:46.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:46.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:46.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:46.525 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.7 primary-affinity to 0.734927 (8481642) 2024-09-17T16:24:46.527 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:24:46.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:46.606 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:46.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:46.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:46.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:46.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:46 smithi017 ceph-mon[27571]: osdmap e518: 12 total, 12 up, 10 in 2024-09-17T16:24:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/159438739' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:46 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:46 smithi017 ceph-mon[27571]: pgmap v1136: 27 pgs: 27 active+clean; 166 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 725 op/s 2024-09-17T16:24:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2681808621' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7349273213347179} : dispatch 2024-09-17T16:24:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:46 smithi071 ceph-mon[30002]: osdmap e518: 12 total, 12 up, 10 in 2024-09-17T16:24:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/159438739' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:46 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:46 smithi071 ceph-mon[30002]: pgmap v1136: 27 pgs: 27 active+clean; 166 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 725 op/s 2024-09-17T16:24:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2681808621' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7349273213347179} : dispatch 2024-09-17T16:24:46.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:46.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:46.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:46 smithi173 ceph-mon[30530]: osdmap e518: 12 total, 12 up, 10 in 2024-09-17T16:24:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/159438739' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:46 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:24:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:46 smithi173 ceph-mon[30530]: pgmap v1136: 27 pgs: 27 active+clean; 166 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 725 op/s 2024-09-17T16:24:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2681808621' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.7349273213347179} : dispatch 2024-09-17T16:24:46.954 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:46.976 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:46.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:46.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:47.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:47.101 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:47.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:47.134 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:24:47.135 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:24:47.157 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:24:47.157 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:24:47.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:24:47.177 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:24:47.197 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:24:47.197 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:24:47.203 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:47.219 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:24:47.219 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:24:47.225 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:47.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:47.241 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:24:47.241 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:24:47.262 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:24:47.262 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:24:47.283 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:24:47.284 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:24:47.305 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:24:47.306 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:24:47.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:47.327 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:24:47.327 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:24:47.348 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:47.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:47.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:47.475 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:47.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:47.552 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:24:47.553 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:24:47.559 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 82768 objects 2024-09-17T16:24:47.559 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :86.5975 2024-09-17T16:24:47.574 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:24:47.574 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:24:47.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:47.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:47.598 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:47.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:47.699 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:47.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:24:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:47.721 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:47.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:47.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:47 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:47.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:47 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:24:47.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2681808621' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7349273213347179}]': finished 2024-09-17T16:24:47.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:47 smithi071 ceph-mon[30002]: osdmap e519: 12 total, 12 up, 10 in 2024-09-17T16:24:47.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1072903935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:47.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:47.830 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:47 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:47.830 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:47 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:24:47.830 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2681808621' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7349273213347179}]': finished 2024-09-17T16:24:47.831 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:47 smithi017 ceph-mon[27571]: osdmap e519: 12 total, 12 up, 10 in 2024-09-17T16:24:47.831 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1072903935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:47.843 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:47.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:47 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:24:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:47 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:24:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2681808621' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7349273213347179}]': finished 2024-09-17T16:24:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:47 smithi173 ceph-mon[30530]: osdmap e519: 12 total, 12 up, 10 in 2024-09-17T16:24:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1072903935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:47.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:47.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:47.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:48.073 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:48.094 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:48.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:48.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:48.196 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:48.218 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:48.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:48.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:48.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:48.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:48.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:48.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:48.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:48.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:48.596 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:48.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:48.652 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:48 smithi017 ceph-mon[27571]: 5.19 scrub starts 2024-09-17T16:24:48.652 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:48 smithi017 ceph-mon[27571]: 5.19 scrub ok 2024-09-17T16:24:48.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:48 smithi017 ceph-mon[27571]: osdmap e520: 12 total, 12 up, 10 in 2024-09-17T16:24:48.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:48 smithi017 ceph-mon[27571]: pgmap v1139: 27 pgs: 27 active+clean; 58 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 980 op/s 2024-09-17T16:24:48.697 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:48.719 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:48.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:48 smithi071 ceph-mon[30002]: 5.19 scrub starts 2024-09-17T16:24:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:48 smithi071 ceph-mon[30002]: 5.19 scrub ok 2024-09-17T16:24:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:48 smithi071 ceph-mon[30002]: osdmap e520: 12 total, 12 up, 10 in 2024-09-17T16:24:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:48 smithi071 ceph-mon[30002]: pgmap v1139: 27 pgs: 27 active+clean; 58 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 980 op/s 2024-09-17T16:24:48.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:48.846 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:48.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:48 smithi173 ceph-mon[30530]: 5.19 scrub starts 2024-09-17T16:24:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:48 smithi173 ceph-mon[30530]: 5.19 scrub ok 2024-09-17T16:24:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:48 smithi173 ceph-mon[30530]: osdmap e520: 12 total, 12 up, 10 in 2024-09-17T16:24:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:48 smithi173 ceph-mon[30530]: pgmap v1139: 27 pgs: 27 active+clean; 58 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 980 op/s 2024-09-17T16:24:48.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:48.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:48.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:49.079 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:49.103 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:49.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:49.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:49.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:49.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:49.333 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:49.360 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:49.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:49.462 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:49.488 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:49.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:49.589 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:24:49.615 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:49.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:24:49.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:49.733 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:49.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:49.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:49.848 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:49.874 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:49.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:49.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:50.001 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:50.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:50.102 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:50.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:50.127 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:50.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:50.228 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:50.254 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:50.255 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:24:50.255 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:24:50.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:24:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:24:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124293 3.e 3:74c76eb5:::benchmark_data_smithi173_66272_object52337:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:24:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:24:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 479.866001751, 2024-09-17T16:24:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.55229465499999997, 2024-09-17T16:24:50.256 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124293 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:24:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:24:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371917+0000", 2024-09-17T16:24:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1430000000000001e-06 2024-09-17T16:24:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371923+0000", 2024-09-17T16:24:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1709999999999999e-06 2024-09-17T16:24:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371936+0000", 2024-09-17T16:24:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2947e-05 2024-09-17T16:24:50.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371951+0000", 2024-09-17T16:24:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4547e-05 2024-09-17T16:24:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.372152+0000", 2024-09-17T16:24:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020128999999999999 2024-09-17T16:24:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:24:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.372168+0000", 2024-09-17T16:24:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6218999999999998e-05 2024-09-17T16:24:50.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.916603+0000", 2024-09-17T16:24:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54443496499999999 2024-09-17T16:24:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.919612+0000", 2024-09-17T16:24:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030085120000000001 2024-09-17T16:24:50.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.919736+0000", 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012435899999999999 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.923884+0000", 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041481460000000001 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.923928+0000", 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4159999999999997e-05 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924005+0000", 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6178000000000001e-05 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924024+0000", 2024-09-17T16:24:50.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9318e-05 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924094+0000", 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9605999999999997e-05 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924210+0000", 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116094 2024-09-17T16:24:50.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:132741 3.e 3:729d5b9e:::benchmark_data_smithi173_66272_object60785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 466.60599304800002, 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43202703199999998, 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:24:50.267 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 132741 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:24:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631925+0000", 2024-09-17T16:24:50.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3239999999999999e-06 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631930+0000", 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6700000000000002e-06 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631942+0000", 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1857e-05 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631947+0000", 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8980000000000001e-06 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631998+0000", 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0365000000000003e-05 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.632091+0000", 2024-09-17T16:24:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3592999999999997e-05 2024-09-17T16:24:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:24:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.632178+0000", 2024-09-17T16:24:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6648000000000001e-05 2024-09-17T16:24:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:24:50.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635422+0000", 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003244289 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.636256+0000", 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083370299999999998 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.636486+0000", 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022978699999999999 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063733+0000", 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42724667900000002 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063826+0000", 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3579999999999995e-05 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063951+0000", 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012463900000000001 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.274 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25036.0:132741 3.e e278/273 ondisk, result = 0)", 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 466.60198167800002, 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42802421600000001, 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635936+0000", 2024-09-17T16:24:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.023e-06 2024-09-17T16:24:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635941+0000", 2024-09-17T16:24:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5000000000000001e-06 2024-09-17T16:24:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.060074+0000", 2024-09-17T16:24:50.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42413389000000001 2024-09-17T16:24:50.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.060083+0000", 2024-09-17T16:24:50.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6670000000000002e-06 2024-09-17T16:24:50.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063717+0000", 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036338640000000001 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063726+0000", 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.789e-06 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063959+0000", 2024-09-17T16:24:50.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023348299999999999 2024-09-17T16:24:50.280 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.280 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.280 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.280 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:134243 3.1 e280/272 3:8c1e41ff:::benchmark_data_smithi173_66272_object62287:head v 280'8469, mlcod=247'4427)", 2024-09-17T16:24:50.280 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:05.911868+0000", 2024-09-17T16:24:50.280 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 464.32604894100001, 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26671571799999999, 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911868+0000", 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911868+0000", 2024-09-17T16:24:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911869+0000", 2024-09-17T16:24:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:24:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:05.911874+0000", 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4380000000000003e-06 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169900+0000", 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25802566399999999 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169906+0000", 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4180000000000002e-06 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169967+0000", 2024-09-17T16:24:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1247000000000005e-05 2024-09-17T16:24:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.169991+0000", 2024-09-17T16:24:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4151999999999998e-05 2024-09-17T16:24:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.178494+0000", 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0085021780000000009 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:06.178584+0000", 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9865999999999996e-05 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:135118 3.e 3:7ce4ae2e:::benchmark_data_smithi173_66272_object63162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-17T16:24:50.285 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 462.96750831499998, 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34112115999999998, 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 135118 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:24:50.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:24:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270411+0000", 2024-09-17T16:24:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3520000000000001e-06 2024-09-17T16:24:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270420+0000", 2024-09-17T16:24:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:24:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270443+0000", 2024-09-17T16:24:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2789000000000002e-05 2024-09-17T16:24:50.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270453+0000", 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0322e-05 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270692+0000", 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023867100000000001 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270833+0000", 2024-09-17T16:24:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014147500000000001 2024-09-17T16:24:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:24:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.271004+0000", 2024-09-17T16:24:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017102299999999999 2024-09-17T16:24:50.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279153+0000", 2024-09-17T16:24:50.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081486939999999997 2024-09-17T16:24:50.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279218+0000", 2024-09-17T16:24:50.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5320999999999993e-05 2024-09-17T16:24:50.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279235+0000", 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6402999999999999e-05 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611337+0000", 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33210225300000001 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611397+0000", 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9757999999999999e-05 2024-09-17T16:24:50.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611530+0000", 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013292299999999999 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:136353 3.e 3:75c6ba64:::benchmark_data_smithi173_66272_object64397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 460.28912410200002, 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34635525499999997, 2024-09-17T16:24:50.295 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 136353 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:24:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948795+0000", 2024-09-17T16:24:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4269999999999998e-06 2024-09-17T16:24:50.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948804+0000", 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948823+0000", 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8892000000000001e-05 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948832+0000", 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0629999999999995e-06 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948903+0000", 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0796000000000005e-05 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.949058+0000", 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015499099999999999 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.949219+0000", 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016127900000000001 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953488+0000", 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042689440000000002 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953702+0000", 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021406499999999999 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953721+0000", 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8936999999999998e-05 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294938+0000", 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.341217465 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295013+0000", 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4389999999999998e-05 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295148+0000", 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000135148 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.302 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25036.0:136353 3.e e284/273 ondisk, result = 0)", 2024-09-17T16:24:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:24:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 460.28545178299998, 2024-09-17T16:24:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.342690457, 2024-09-17T16:24:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:24:50.303 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952466+0000", 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0389999999999999e-06 2024-09-17T16:24:50.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952472+0000", 2024-09-17T16:24:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6910000000000002e-06 2024-09-17T16:24:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294851+0000", 2024-09-17T16:24:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34237905899999999 2024-09-17T16:24:50.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294856+0000", 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3499999999999996e-06 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294916+0000", 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9580000000000002e-05 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294928+0000", 2024-09-17T16:24:50.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2571999999999999e-05 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295155+0000", 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000227166 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:143899 3.1 e290/272 3:8d15b72a:::benchmark_data_smithi173_66272_object71943:head v 290'9096, mlcod=261'5689)", 2024-09-17T16:24:50.307 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 449.75879720199998, 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37496784399999999, 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:24:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:24:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9699999999999996e-07 2024-09-17T16:24:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479128+0000", 2024-09-17T16:24:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-17T16:24:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850526+0000", 2024-09-17T16:24:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37139821699999997 2024-09-17T16:24:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850534+0000", 2024-09-17T16:24:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1760000000000004e-06 2024-09-17T16:24:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850583+0000", 2024-09-17T16:24:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9013e-05 2024-09-17T16:24:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850601+0000", 2024-09-17T16:24:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8916e-05 2024-09-17T16:24:50.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.854050+0000", 2024-09-17T16:24:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034480639999999998 2024-09-17T16:24:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.854087+0000", 2024-09-17T16:24:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7846000000000003e-05 2024-09-17T16:24:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.314 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:143908 3.e 3:70cae513:::benchmark_data_smithi173_66272_object71952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-17T16:24:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:24:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 449.74874603900003, 2024-09-17T16:24:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35428895300000002, 2024-09-17T16:24:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.315 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:24:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:24:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:24:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 143908 2024-09-17T16:24:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.316 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:24:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:24:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489172+0000", 2024-09-17T16:24:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6700000000000001e-06 2024-09-17T16:24:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489177+0000", 2024-09-17T16:24:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.651e-06 2024-09-17T16:24:50.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489191+0000", 2024-09-17T16:24:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3497e-05 2024-09-17T16:24:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489197+0000", 2024-09-17T16:24:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0719999999999996e-06 2024-09-17T16:24:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489244+0000", 2024-09-17T16:24:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7037999999999999e-05 2024-09-17T16:24:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489341+0000", 2024-09-17T16:24:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7664000000000005e-05 2024-09-17T16:24:50.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:24:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489424+0000", 2024-09-17T16:24:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2916000000000004e-05 2024-09-17T16:24:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:24:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493180+0000", 2024-09-17T16:24:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037554379999999998 2024-09-17T16:24:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493210+0000", 2024-09-17T16:24:50.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0076000000000001e-05 2024-09-17T16:24:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.494016+0000", 2024-09-17T16:24:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080592900000000004 2024-09-17T16:24:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:24:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843310+0000", 2024-09-17T16:24:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34929427400000002 2024-09-17T16:24:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843362+0000", 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2549000000000001e-05 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843460+0000", 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7178999999999999e-05 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(4.6 383/382 [PushOp(4:63469528:::benchmark_data_smithi173_77474_object91937:head, version: 345'5625, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:63469528:::benchmark_data_smithi173_77474_object91937:head@345'5625, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 247.84148563900001, 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29598169099999999, 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:24:50.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396432+0000", 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9999999999999997e-07 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396467+0000", 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5184000000000001e-05 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692000+0000", 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29553331999999999 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692039+0000", 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8491999999999997e-05 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692173+0000", 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134411 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692186+0000", 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3118e-05 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692413+0000", 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022656600000000001 2024-09-17T16:24:50.331 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_backfill(progress 4.6 e 383/383 lb 4:6341a44b:::benchmark_data_smithi173_77474_object88529:head)", 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 247.841425813, 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29619305299999998, 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:24:50.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396495+0000", 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9450000000000003e-06 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692562+0000", 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29606637800000002 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692584+0000", 2024-09-17T16:24:50.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2526000000000001e-05 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692645+0000", 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1309999999999994e-05 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692657+0000", 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1798e-05 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692684+0000", 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6821e-05 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:2268 5.2 e424/419 5:448d2a5f:::benchmark_data_smithi173_90735_object2267:head v 424'138, mlcod=424'135)", 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:51.493252+0000", 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 178.744664591, 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26039381099999998, 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.336 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.493252+0000", 2024-09-17T16:24:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.493252+0000", 2024-09-17T16:24:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.493253+0000", 2024-09-17T16:24:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.06e-07 2024-09-17T16:24:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.493347+0000", 2024-09-17T16:24:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4554000000000003e-05 2024-09-17T16:24:50.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.752984+0000", 2024-09-17T16:24:50.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.2596366 2024-09-17T16:24:50.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.752992+0000", 2024-09-17T16:24:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5879999999999998e-06 2024-09-17T16:24:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.753063+0000", 2024-09-17T16:24:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0337000000000001e-05 2024-09-17T16:24:50.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.753090+0000", 2024-09-17T16:24:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7562000000000001e-05 2024-09-17T16:24:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.753603+0000", 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051267200000000004 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.753646+0000", 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2991999999999998e-05 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(5.13 450/444 [PushReplyOp(5:ce04d783:::benchmark_data_smithi173_90735_object32464:head)])", 2024-09-17T16:24:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:39.936908+0000", 2024-09-17T16:24:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 130.30100913999999, 2024-09-17T16:24:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.258692325, 2024-09-17T16:24:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:24:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.936908+0000", 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.936908+0000", 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.936908+0000", 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7700000000000004e-07 2024-09-17T16:24:50.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.936912+0000", 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9160000000000003e-06 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.195065+0000", 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25815273900000002 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.195073+0000", 2024-09-17T16:24:50.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9100000000000005e-06 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.195133+0000", 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0081999999999998e-05 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.195600+0000", 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046710099999999999 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:82763 5.13 e475/453 5:cd63b9f8:::benchmark_data_smithi173_90735_object82762:head v 475'4007, mlcod=475'4003)", 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 90.319717558999997, 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56469166699999995, 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:24:50.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:50.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:24:50.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:24:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918200+0000", 2024-09-17T16:24:50.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1500000000000004e-07 2024-09-17T16:24:50.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918245+0000", 2024-09-17T16:24:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5034000000000003e-05 2024-09-17T16:24:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481583+0000", 2024-09-17T16:24:50.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56333854699999997 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481588+0000", 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5800000000000002e-06 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481652+0000", 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4309999999999999e-05 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481669+0000", 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6880000000000001e-05 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.482864+0000", 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011944200000000001 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.482891+0000", 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7180999999999999e-05 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.355 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPull(5.12 e475/455 cost 1049576)", 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 90.278457990999996, 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54846142799999997, 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959460+0000", 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5099999999999998e-07 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959470+0000", 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9380000000000004e-06 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507615+0000", 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54814523100000001 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507626+0000", 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1484000000000001e-05 2024-09-17T16:24:50.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507695+0000", 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8990000000000002e-05 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507920+0000", 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022483400000000001 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(5.d 475/455 [PushOp(5:b9f39261:::benchmark_data_smithi173_90735_object37371:head, version: 439'2383, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:b9f39261:::benchmark_data_smithi173_90735_object37371:head@439'2383, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 89.757540340000006, 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45551481399999999, 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:24:50.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480378+0000", 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6470000000000001e-06 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480434+0000", 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6162999999999997e-05 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934248+0000", 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45381411300000002 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934260+0000", 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1382e-05 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934362+0000", 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000102681 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934379+0000", 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6498e-05 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.935891+0000", 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015123300000000001 2024-09-17T16:24:50.363 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:95580 5.b e482/419 5:d4bc1ca6:::benchmark_data_smithi173_90735_object12809:head v 482'5935, mlcod=482'5933)", 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 76.505449526000007, 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39904708700000002, 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:24:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:24:50.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732468+0000", 2024-09-17T16:24:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-17T16:24:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732477+0000", 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4880000000000007e-06 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128932+0000", 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.396455369 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128940+0000", 2024-09-17T16:24:50.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5089999999999996e-06 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128990+0000", 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0068999999999998e-05 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.129012+0000", 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2218e-05 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.131477+0000", 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002464843 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.131514+0000", 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7515000000000003e-05 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:95584 5.b e482/419 5:dc96c928:::benchmark_data_smithi173_90735_object12813:head v 482'5936, mlcod=482'5933)", 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:24:50.369 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 76.353802332000001, 2024-09-17T16:24:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39023148299999999, 2024-09-17T16:24:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:24:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:24:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884116+0000", 2024-09-17T16:24:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9319999999999999e-06 2024-09-17T16:24:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884127+0000", 2024-09-17T16:24:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0198e-05 2024-09-17T16:24:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272085+0000", 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38795868300000003 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272089+0000", 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1729999999999997e-06 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272129+0000", 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9937000000000001e-05 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272145+0000", 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5863e-05 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.274307+0000", 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021616040000000001 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.274346+0000", 2024-09-17T16:24:50.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9093e-05 2024-09-17T16:24:50.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.375 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.375 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:150237 5.d e509/482 5:b701c5ae:::benchmark_data_smithi173_90735_object67466:head v 509'9426, mlcod=509'9424)", 2024-09-17T16:24:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:24:30.121032+0000", 2024-09-17T16:24:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 20.116884906999999, 2024-09-17T16:24:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.188520244, 2024-09-17T16:24:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:30.121032+0000", 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:30.121032+0000", 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:30.121032+0000", 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4199999999999995e-07 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:30.121037+0000", 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7129999999999996e-06 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:30.305879+0000", 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.18484140900000001 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:30.305884+0000", 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2229999999999998e-06 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:30.305935+0000", 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1530999999999998e-05 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:30.305964+0000", 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9023e-05 2024-09-17T16:24:50.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:30.309497+0000", 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003532758 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:30.309552+0000", 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4945000000000003e-05 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:158731 5.d e512/482 5:bd0a9746:::benchmark_data_smithi173_90735_object75960:head v 512'9909, mlcod=512'9906)", 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:24:38.876003+0000", 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 11.361913895000001, 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.163930674, 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:38.876003+0000", 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:38.876003+0000", 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:38.876004+0000", 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:38.876009+0000", 2024-09-17T16:24:50.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3449999999999996e-06 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:39.036725+0000", 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.16071574899999999 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:39.036736+0000", 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1656e-05 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:39.036829+0000", 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2896e-05 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:39.036871+0000", 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1322999999999997e-05 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:39.039894+0000", 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030234519999999998 2024-09-17T16:24:50.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:24:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:24:39.039934+0000", 2024-09-17T16:24:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9382999999999999e-05 2024-09-17T16:24:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:50.385 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:50.385 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:24:50.450 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:50.474 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:50.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:50.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:50.597 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:50.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:50.699 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:50.726 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:50.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:50 smithi017 ceph-mon[27571]: osdmap e521: 12 total, 12 up, 10 in 2024-09-17T16:24:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:50 smithi017 ceph-mon[27571]: pgmap v1141: 27 pgs: 27 active+clean; 17 MiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 745 op/s 2024-09-17T16:24:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:50 smithi071 ceph-mon[30002]: osdmap e521: 12 total, 12 up, 10 in 2024-09-17T16:24:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:50 smithi071 ceph-mon[30002]: pgmap v1141: 27 pgs: 27 active+clean; 17 MiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 745 op/s 2024-09-17T16:24:50.827 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:50.852 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:50.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:50 smithi173 ceph-mon[30530]: osdmap e521: 12 total, 12 up, 10 in 2024-09-17T16:24:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:50 smithi173 ceph-mon[30530]: pgmap v1141: 27 pgs: 27 active+clean; 17 MiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 745 op/s 2024-09-17T16:24:50.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:50.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:50.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:51.082 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:51.106 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:51.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:51.166 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:24:51.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:51.209 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:51.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:51.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:51.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:51.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:51.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:51.457 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:51.484 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:51.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:51.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:51.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:51.610 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:51.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:51.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:51.725 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:24:51.736 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:51.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:51.838 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:51.862 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:51.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:51.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:51.988 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:51.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:52.058 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:24:52.058 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:24:52.079 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:24:52.079 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:24:52.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:52.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:24:52.100 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:24:52.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:52.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:52.124 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:24:52.124 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:24:52.154 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:24:52.154 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:24:52.171 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:52.172 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:52.180 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:24:52.180 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:24:52.203 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:24:52.204 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:24:52.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:52.225 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:24:52.225 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:24:52.240 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:52.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:52.247 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:24:52.248 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:24:52.269 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:24:52.270 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:24:52.291 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:24:52.292 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:24:52.313 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:24:52.313 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:24:52.341 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:52.367 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:52.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:52.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:52.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:52.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:52.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:52.623 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:52.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:52.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:52.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:52.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:52.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:52.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:52.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:52.905 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-17T16:24:52.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-17T16:24:52.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:53.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:53.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:53.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:53.108 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:53.134 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:53.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:53.236 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:53.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:52 smithi017 ceph-mon[27571]: pgmap v1142: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 442 op/s 2024-09-17T16:24:53.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/387240677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:53.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:53.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:52 smithi071 ceph-mon[30002]: pgmap v1142: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 442 op/s 2024-09-17T16:24:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/387240677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:53.363 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:53.386 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:53.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:53.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:52 smithi173 ceph-mon[30530]: pgmap v1142: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 442 op/s 2024-09-17T16:24:53.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/387240677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:24:53.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:53.513 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:53.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:53.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:53.641 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:53.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:53.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:53.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:53.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:53.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:53.817 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:53.817 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:53.871 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:53.897 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:53.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:53.970 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:24:53.988 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:24:53.988 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:24:53.988 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T16:24:53.988 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:24:53.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:54.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:54.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:54.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:54.148 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:54.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:53 smithi017 ceph-mon[27571]: 5.18 scrub starts 2024-09-17T16:24:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:53 smithi017 ceph-mon[27571]: 5.18 scrub ok 2024-09-17T16:24:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:53 smithi017 ceph-mon[27571]: 5.6 scrub starts 2024-09-17T16:24:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:53 smithi017 ceph-mon[27571]: 5.6 scrub ok 2024-09-17T16:24:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:53 smithi017 ceph-mon[27571]: 5.e scrub starts 2024-09-17T16:24:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:53 smithi017 ceph-mon[27571]: 5.e scrub ok 2024-09-17T16:24:54.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1061884405' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:54.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:54.276 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:54.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:53 smithi071 ceph-mon[30002]: 5.18 scrub starts 2024-09-17T16:24:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:53 smithi071 ceph-mon[30002]: 5.18 scrub ok 2024-09-17T16:24:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:53 smithi071 ceph-mon[30002]: 5.6 scrub starts 2024-09-17T16:24:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:53 smithi071 ceph-mon[30002]: 5.6 scrub ok 2024-09-17T16:24:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:53 smithi071 ceph-mon[30002]: 5.e scrub starts 2024-09-17T16:24:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:53 smithi071 ceph-mon[30002]: 5.e scrub ok 2024-09-17T16:24:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1061884405' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:54.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:53 smithi173 ceph-mon[30530]: 5.18 scrub starts 2024-09-17T16:24:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:53 smithi173 ceph-mon[30530]: 5.18 scrub ok 2024-09-17T16:24:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:53 smithi173 ceph-mon[30530]: 5.6 scrub starts 2024-09-17T16:24:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:53 smithi173 ceph-mon[30530]: 5.6 scrub ok 2024-09-17T16:24:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:53 smithi173 ceph-mon[30530]: 5.e scrub starts 2024-09-17T16:24:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:53 smithi173 ceph-mon[30530]: 5.e scrub ok 2024-09-17T16:24:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1061884405' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:24:54.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:54.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:54.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:54.531 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:54.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:54.633 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:54.658 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:54.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:54.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:54.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:54.785 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:54.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:54.886 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:54.910 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:54.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:54.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:55.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:55.035 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:55.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:55.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:55.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:55.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:54 smithi017 ceph-mon[27571]: pgmap v1143: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 177 op/s 2024-09-17T16:24:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:54 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:24:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:54 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:24:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1061884405' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:54 smithi017 ceph-mon[27571]: osdmap e522: 12 total, 12 up, 10 in 2024-09-17T16:24:55.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:54 smithi017 ceph-mon[27571]: osdmap e523: 12 total, 12 up, 10 in 2024-09-17T16:24:55.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:55.290 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:55.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:55.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:54 smithi071 ceph-mon[30002]: pgmap v1143: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 177 op/s 2024-09-17T16:24:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:54 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:24:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:54 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:24:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1061884405' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:54 smithi071 ceph-mon[30002]: osdmap e522: 12 total, 12 up, 10 in 2024-09-17T16:24:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:54 smithi071 ceph-mon[30002]: osdmap e523: 12 total, 12 up, 10 in 2024-09-17T16:24:55.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:24:55.346 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:55.356 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:55.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:54 smithi173 ceph-mon[30530]: pgmap v1143: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 177 op/s 2024-09-17T16:24:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:54 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:24:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:54 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:24:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1061884405' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:24:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:54 smithi173 ceph-mon[30530]: osdmap e522: 12 total, 12 up, 10 in 2024-09-17T16:24:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:54 smithi173 ceph-mon[30530]: osdmap e523: 12 total, 12 up, 10 in 2024-09-17T16:24:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:55.416 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:55.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:55.517 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:55.540 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:55.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:55.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:55.670 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:55.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:55.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:55.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:55.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:55.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:55.922 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:55.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:55.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:56.024 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:56.046 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:56.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:56.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:56.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:56.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:24:56.260 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:24:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 523, 2024-09-17T16:24:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:24:56.260 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:24:54.491026+0000", 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:22:45.057702+0000", 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5799936, 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:24:56.261 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 5, 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:24:56.262 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:24:56.263 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:24:56.264 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "446", 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:24:56.265 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:24:56.266 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:24:56.267 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:24:56.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.267 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:24:56.267 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:24:56.267 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:24:56.267 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:24:56.267 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:24:56.267 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:24:56.267 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:24:56.267 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:24:56.268 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 5, 2024-09-17T16:24:56.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_3", 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:21:40.832300+0000", 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:24:56.270 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 26, 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 26, 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 26, 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 26, 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 26, 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:24:56.271 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "443", 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "441", 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:24:56.272 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:24:56.273 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:24:56.274 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 1.690000057220459, 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 1.690000057220459, 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.9100000262260437, 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 1.5399999618530273, 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 1.5399999618530273, 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:24:56.275 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 519, 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:24:56.278 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:56.278 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.278 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.278 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.279 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:24:56.280 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:24:56.281 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 0, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 301, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.283 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.287 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:24:56.288 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 469, 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.289 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:24:56.293 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:56.294 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 472, 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:56.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:56.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:24:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.300 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.301 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:24:56.301 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 481, 2024-09-17T16:24:56.301 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.301 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.301 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.301 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.301 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.301 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:24:56.301 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:56.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.302 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.302 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:24:56.302 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:56.302 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.302 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.302 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.302 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.303 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:24:56.306 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 0, 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 454, 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:24:56.307 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.308 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:24:56.311 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.66790771484375, 2024-09-17T16:24:56.312 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 466, 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.314 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.314 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:24:56.314 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:24:56.315 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.316 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.317 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.318 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.8061676025390625, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.73492431640625, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 482, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:24:56.319 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.320 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:56.321 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:56.322 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.323 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.323 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.323 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.323 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.323 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:24:56.323 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:56.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:24:56.324 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:24:56.325 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.325 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.325 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.325 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.325 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:24:56.325 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:24:56.325 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.325 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:56.326 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:56.326 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:56.326 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.326 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.326 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:24:56.326 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 500, 2024-09-17T16:24:56.326 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.326 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.326 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.326 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.327 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.327 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:24:56.327 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:56.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.328 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.328 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:24:56.328 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:24:56.329 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.330 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.331 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.2821197509765625, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.333 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.334 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.335 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:24:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.336 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.337 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.338 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:56.339 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 462, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.341 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.344 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.345 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.345 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:24:56.346 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:24:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:24:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:24:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.347 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.588836669921875, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 291, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:24:56.348 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.350 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:24:56.350 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.350 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.350 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:24:56.350 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:56.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.350 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.350 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:24:56.352 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:24:56.353 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.354 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:01.931904+0000", 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 65536, 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:42.178867+0000", 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:51.474083+0000", 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:24:56.356 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.357 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.357 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.357 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.357 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:56.357 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:19.641962+0000", 2024-09-17T16:24:56.357 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.357 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:53.204639+0000", 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.358 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 55422, 2024-09-17T16:24:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:18.369564+0000", 2024-09-17T16:24:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:24:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.359 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.360 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.360 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.360 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:56.360 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:55.626986+0000", 2024-09-17T16:24:56.361 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.361 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:24:56.361 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.361 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.361 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.361 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.361 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:56.361 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:44.839194+0000", 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:11.761032+0000", 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:43.347262+0000", 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.363 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:23:58.985462+0000", 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:24:56.364 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:24:12.252165+0000", 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [ 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "1.0", 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: 3, 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: 8 2024-09-17T16:24:56.365 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "5.5", 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: 10, 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: 4, 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: 3 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-17T16:24:56.366 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:24:56.367 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:24:56.367 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:24:56.367 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:24:56.367 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:24:56.367 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:24:56.367 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:24:56.367 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:24:56.367 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:24:56.367 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:24:56.369 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:24:56.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:56.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:56.497 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:56.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:56.520 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:56.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:56.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:56.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:56.647 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:56.743 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:24:56.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:56.800 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:24:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:56.810 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:56.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:56.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:56.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:56.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:56.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:56.991 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_3' removed 2024-09-17T16:24:57.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:57.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:57.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:57.088 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:56 smithi017 ceph-mon[27571]: pgmap v1146: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:24:57.088 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:56 smithi017 ceph-mon[27571]: 5.2 scrub starts 2024-09-17T16:24:57.089 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:56 smithi017 ceph-mon[27571]: 5.2 scrub ok 2024-09-17T16:24:57.089 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1604974191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:24:57.089 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3477651987' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:24:57.089 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:56 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:24:57.124 INFO:tasks.thrashosds.thrasher:j is {'epoch': 523, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:24:54.491026+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:22:45.057702+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '446', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 5, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-17T16:21:40.832300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '443', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '441', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 0.9100000262260437, 'raw_score_acting': 1.5399999618530273, 'raw_score_stable': 1.5399999618530273, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 519, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 301, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 469, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 472, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 481, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 454, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 0.66790771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 466, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 0.8061676025390625, 'primary_affinity': 0.73492431640625, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 482, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 500, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0.2821197509765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 462, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 0.588836669921875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:24:01.931904+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-17T16:24:42.178867+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:51.474083+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:24:19.641962+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:53.204639+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 55422, 'last_purged_snaps_scrub': '2024-09-17T16:24:18.369564+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:55.626986+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:24:44.839194+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:24:11.761032+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:24:43.347262+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:23:58.985462+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:24:12.252165+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [3, 1, 8]}, {'pgid': '5.5', 'osds': [10, 4, 3]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:24:57.125 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:24:57.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:57.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:57.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:57.287 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:57.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:57.313 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:57.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:56 smithi071 ceph-mon[30002]: pgmap v1146: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:24:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:56 smithi071 ceph-mon[30002]: 5.2 scrub starts 2024-09-17T16:24:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:56 smithi071 ceph-mon[30002]: 5.2 scrub ok 2024-09-17T16:24:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1604974191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:24:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3477651987' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:24:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:56 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:24:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:56 smithi173 ceph-mon[30530]: pgmap v1146: 27 pgs: 27 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:24:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:56 smithi173 ceph-mon[30530]: 5.2 scrub starts 2024-09-17T16:24:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:56 smithi173 ceph-mon[30530]: 5.2 scrub ok 2024-09-17T16:24:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1604974191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:24:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3477651987' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:24:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:56 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:24:57.414 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:24:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:57.437 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:57.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:24:57.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:57.564 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:57.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:57.666 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:57.690 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:57.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:57.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:57.819 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:57.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:57.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:57.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:57.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:58.045 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:24:58.067 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:58.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:24:58.114 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T16:24:58.115 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T16:24:58.115 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-17T16:24:58.116 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-17T16:24:58.138 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:58.156 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:58.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:58.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:58.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:58.267 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:57 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:24:58.267 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:57 smithi017 ceph-mon[27571]: osdmap e524: 12 total, 12 up, 10 in 2024-09-17T16:24:58.293 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:24:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:57 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:24:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:57 smithi071 ceph-mon[30002]: osdmap e524: 12 total, 12 up, 10 in 2024-09-17T16:24:58.317 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:58.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:24:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:57 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:24:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:57 smithi173 ceph-mon[30530]: osdmap e524: 12 total, 12 up, 10 in 2024-09-17T16:24:58.419 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:58.443 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:58.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:58.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:58.545 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:24:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:58.570 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:58.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:24:58.672 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:24:58.696 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:58.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:24:58.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:24:58.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:58.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:24:58.901 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:24:58.902 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:24:58.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:58.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:58.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:59.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:59.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:59.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:59.093 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:58 smithi017 ceph-mon[27571]: pgmap v1148: 1 pgs: 1 active+clean; 577 KiB data, 3.4 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:24:59.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:24:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:59.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:59.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:24:59.292 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:59.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:58 smithi071 ceph-mon[30002]: pgmap v1148: 1 pgs: 1 active+clean; 577 KiB data, 3.4 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:24:59.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:59.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:59 smithi173 ceph-mon[30530]: pgmap v1148: 1 pgs: 1 active+clean; 577 KiB data, 3.4 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:24:59.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:59.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:59.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:24:59.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:24:59.586 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:24:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:59.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:24:59.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:24:59.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:59.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:24:59.725 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:24:59.757 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:24:59.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:24:59.844 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:24:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:59.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:24:59.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:24:59.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:24:59.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:24:59.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:00.007 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:25:00.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:00.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:00.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:00.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:00.229 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:00.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:59 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:59 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:59 smithi017 ceph-mon[27571]: osdmap e525: 12 total, 12 up, 10 in 2024-09-17T16:25:00.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:24:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1459947822' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:25:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:59 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:59 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:59 smithi071 ceph-mon[30002]: osdmap e525: 12 total, 12 up, 10 in 2024-09-17T16:25:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:24:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1459947822' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:25:00.330 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:00.356 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:00.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:59 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:59 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:59 smithi173 ceph-mon[30530]: osdmap e525: 12 total, 12 up, 10 in 2024-09-17T16:25:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:24:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1459947822' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:25:00.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:00.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:00.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:00.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:00.611 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:00.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:00.714 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:00.739 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:00.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:00.841 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:00.866 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:00.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:00.910 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:25:00.911 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:25:00.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:00.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:00.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:01.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:01.122 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:01.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:01.224 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:01.248 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:01.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:01.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:01 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:01.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:01 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:01.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:01 smithi017 ceph-mon[27571]: pgmap v1150: 1 pgs: 1 active+clean; 577 KiB data, 3.4 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:01.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:01 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:01.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1459947822' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:25:01.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:01 smithi017 ceph-mon[27571]: osdmap e526: 12 total, 12 up, 10 in 2024-09-17T16:25:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:01 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:01.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:01 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:01.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:01 smithi071 ceph-mon[30002]: pgmap v1150: 1 pgs: 1 active+clean; 577 KiB data, 3.4 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:01.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:01 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:01.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1459947822' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:25:01.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:01 smithi071 ceph-mon[30002]: osdmap e526: 12 total, 12 up, 10 in 2024-09-17T16:25:01.350 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:01.375 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:01.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:01.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:01 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:01 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:01 smithi173 ceph-mon[30530]: pgmap v1150: 1 pgs: 1 active+clean; 577 KiB data, 3.4 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:01 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1459947822' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:25:01.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:01 smithi173 ceph-mon[30530]: osdmap e526: 12 total, 12 up, 10 in 2024-09-17T16:25:01.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:01.499 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:01.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:01.601 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:01.623 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:01.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:01.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:01.748 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:01.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:01.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:01.779 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:01.852 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:01.875 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:01.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:01.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:02.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:02.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:02.007 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_4' created 2024-09-17T16:25:02.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:02.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:02.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:02.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:02.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:02 smithi017 ceph-mon[27571]: from='client.17070 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:25:02.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2608890646' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-17T16:25:02.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:02.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:02.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:02 smithi071 ceph-mon[30002]: from='client.17070 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:25:02.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2608890646' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-17T16:25:02.358 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:02.379 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:02.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:02 smithi173 ceph-mon[30530]: from='client.17070 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:25:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2608890646' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-17T16:25:02.481 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:02.506 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:02.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:02.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:02.631 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:02.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:02.717 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-17T16:25:02.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:02.758 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:02.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:02.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:02.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:02.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:02.988 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:03.010 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:03.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:03.112 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:03.134 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:03.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:03.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:03.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:03 smithi017 ceph-mon[27571]: pgmap v1152: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:03.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2608890646' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-17T16:25:03.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:03 smithi017 ceph-mon[27571]: osdmap e527: 12 total, 12 up, 10 in 2024-09-17T16:25:03.236 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:03.239 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1151,"stamp":"2024-09-17T16:25:00.006010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":937451520,"kb_used":3610216,"kb_used_data":44956,"kb_used_omap":55,"kb_used_meta":3565000,"kb_avail":933841304,"statfs":{"total":959950356480,"available":956253495296,"internally_reserved":0,"allocated":46034944,"data_stored":41256470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57064,"internal_metadata":3650560280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-128915,"num_write_kb":-3049889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-14215,"num_bytes_recovered":-931528704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.513344"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":951,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-17T16:24:57.755521+0000","last_change":"2024-09-17T16:22:06.807310+0000","last_active":"2024-09-17T16:24:57.755521+0000","last_peered":"2024-09-17T16:24:57.755521+0000","last_clean":"2024-09-17T16:24:57.755521+0000","last_became_active":"2024-09-17T16:16:49.359652+0000","last_became_peered":"2024-09-17T16:16:49.359652+0000","last_unstale":"2024-09-17T16:24:57.755521+0000","last_undegraded":"2024-09-17T16:24:57.755521+0000","last_fullsized":"2024-09-17T16:24:57.755521+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:22:06.807231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:22:06.807231+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:25:14.054861+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285754,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":384224,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93360928,"statfs":{"total":95995035648,"available":95601590272,"internally_reserved":0,"allocated":1847296,"data_stored":1361417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":391576004},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547390,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325384,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93419768,"statfs":{"total":95995035648,"available":95661842432,"internally_reserved":0,"allocated":1298432,"data_stored":824711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743619,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":429512,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93315640,"statfs":{"total":95995035648,"available":95555215360,"internally_reserved":0,"allocated":1298432,"data_stored":824711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":438499776},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005257,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":249048,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93496104,"statfs":{"total":95995035648,"available":95740010496,"internally_reserved":0,"allocated":1314816,"data_stored":832509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963733,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401492,"kb_used_data":1788,"kb_used_omap":41,"kb_used_meta":399638,"kb_avail":93343660,"statfs":{"total":95995035648,"available":95583907840,"internally_reserved":0,"allocated":1830912,"data_stored":1353639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":409229619},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495826,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":393936,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93351216,"statfs":{"total":95995035648,"available":95591645184,"internally_reserved":0,"allocated":1830912,"data_stored":1353639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":167503724757,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986397,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":351780,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":350462,"kb_avail":93393372,"statfs":{"total":95995035648,"available":95634812928,"internally_reserved":0,"allocated":1327104,"data_stored":839967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":358873541},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149919,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":379528,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93365624,"statfs":{"total":95995035648,"available":95606398976,"internally_reserved":0,"allocated":1298432,"data_stored":824711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":387316167},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411558,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":239304,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93505848,"statfs":{"total":95995035648,"available":95749988352,"internally_reserved":0,"allocated":1298432,"data_stored":824711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738608,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":456008,"kb_used_data":31924,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93289144,"statfs":{"total":95995035648,"available":95528083456,"internally_reserved":0,"allocated":32690176,"data_stored":32216455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:25:03.239 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [4, 6, 8] 2024-09-17T16:25:03.239 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '4', '6', '8'] 2024-09-17T16:25:03.240 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap 1.0 4 6 8 2024-09-17T16:25:03.262 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:03.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:03.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:03 smithi071 ceph-mon[30002]: pgmap v1152: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2608890646' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-17T16:25:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:03 smithi071 ceph-mon[30002]: osdmap e527: 12 total, 12 up, 10 in 2024-09-17T16:25:03.327 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:25:03.328 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:25:03.328 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:25:03.328 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:25:03.328 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:25:03.328 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:25:03.328 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:25:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:25:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:25:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:25:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:25:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:25:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:25:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:25:03.329 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:25:03.355 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:25:03.356 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:25:03.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:03.377 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:25:03.378 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:25:03.389 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:03.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:03 smithi173 ceph-mon[30530]: pgmap v1152: 1 pgs: 1 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2608890646' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-17T16:25:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:03 smithi173 ceph-mon[30530]: osdmap e527: 12 total, 12 up, 10 in 2024-09-17T16:25:03.399 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:25:03.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:25:03.422 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:25:03.422 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:25:03.445 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:25:03.445 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:25:03.491 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:03.516 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:03.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:03.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:03.645 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:03.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:03.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:03.772 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:03.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:03.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:03.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:03.902 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:04.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:04.029 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:04.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:04.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:04.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:04.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:04.259 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:04.286 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:04.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:04.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:04 smithi071 ceph-mon[30002]: osdmap e528: 12 total, 12 up, 10 in 2024-09-17T16:25:04.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3479008838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:04.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:04.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:04 smithi173 ceph-mon[30530]: osdmap e528: 12 total, 12 up, 10 in 2024-09-17T16:25:04.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3479008838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:04.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:04.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:04 smithi017 ceph-mon[27571]: osdmap e528: 12 total, 12 up, 10 in 2024-09-17T16:25:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3479008838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:04.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:04.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:04.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:04.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:04.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:04.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:04.771 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:04.797 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:04.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:04.899 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:04.923 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:04.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:05.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:05.050 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:05.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:05.084 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:05.097 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(3.0 256/239 [PushOp(3:0ca39f28:::benchmark_data_smithi173_66272_object41987:head, version: 231'2635, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0ca39f28:::benchmark_data_smithi173_66272_object41987:head@231'2635, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 548.05224266699997, 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59916758699999995, 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040664+0000", 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040665+0000", 2024-09-17T16:25:05.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0100000000000003e-07 2024-09-17T16:25:05.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.040719+0000", 2024-09-17T16:25:05.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4487000000000002e-05 2024-09-17T16:25:05.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638102+0000", 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.597383255 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638121+0000", 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8595999999999999e-05 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638205+0000", 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4042999999999995e-05 2024-09-17T16:25:05.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.638216+0000", 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1299e-05 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:15:57.639831+0000", 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001615006 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124280 3.0 3:099812ca:::benchmark_data_smithi173_66272_object52324:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377919+0000", 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 494.71498735699998, 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63194678999999998, 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124280 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.103 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377919+0000", 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377919+0000", 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377921+0000", 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.793e-06 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377925+0000", 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5859999999999998e-06 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377937+0000", 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1168999999999999e-05 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377944+0000", 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5209999999999997e-06 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391301+0000", 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013356989999999999 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391307+0000", 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0939999999999996e-06 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.983230+0000", 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59192265799999999 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.983309+0000", 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9083000000000002e-05 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.983395+0000", 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6595999999999996e-05 2024-09-17T16:25:05.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991593+0000", 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081975050000000008 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991633+0000", 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9579000000000002e-05 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991675+0000", 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2407999999999998e-05 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009712+0000", 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.018037553000000001 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009777+0000", 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4436000000000004e-05 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009866+0000", 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8819000000000005e-05 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124284 3.0 3:04f57090:::benchmark_data_smithi173_66272_object52328:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377957+0000", 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 494.714948929, 2024-09-17T16:25:05.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63216316299999997, 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124284 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377957+0000", 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377957+0000", 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377958+0000", 2024-09-17T16:25:05.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377960+0000", 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9209999999999999e-06 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377964+0000", 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6569999999999996e-06 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377970+0000", 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3290000000000001e-06 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391358+0000", 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013388304 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391368+0000", 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0570999999999999e-05 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.986075+0000", 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.594706819 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.986194+0000", 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118577 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.986314+0000", 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000119782 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991687+0000", 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053729240000000003 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996674+0000", 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00498761 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.000224+0000", 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035494630000000001 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009919+0000", 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096958189999999996 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.009978+0000", 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8242000000000002e-05 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.010121+0000", 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014286999999999999 2024-09-17T16:25:05.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124286 3.0 3:01b0fd2d:::benchmark_data_smithi173_66272_object52330:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377976+0000", 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 494.71493003400002, 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63695855000000001, 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:05.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124286 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377976+0000", 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377976+0000", 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377977+0000", 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4699999999999998e-07 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377978+0000", 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.342e-06 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377982+0000", 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6389999999999998e-06 2024-09-17T16:25:05.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377985+0000", 2024-09-17T16:25:05.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2789999999999999e-06 2024-09-17T16:25:05.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391399+0000", 2024-09-17T16:25:05.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013414272 2024-09-17T16:25:05.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391403+0000", 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.799e-06 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.989045+0000", 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.59764173499999995 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:05.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.989160+0000", 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000115654 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.989270+0000", 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000109574 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996735+0000", 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074655440000000002 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:05.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996747+0000", 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2125e-05 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.010180+0000", 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013432908 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.014804+0000", 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046233350000000001 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:05.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.014845+0000", 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1390999999999997e-05 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.014935+0000", 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9705999999999998e-05 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124287 3.0 3:022daafd:::benchmark_data_smithi173_66272_object52331:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:25:05.125 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 494.714917679, 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.63722835200000005, 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124287 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:25:05.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377989+0000", 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2399999999999999e-07 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377990+0000", 2024-09-17T16:25:05.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1489999999999999e-06 2024-09-17T16:25:05.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377993+0000", 2024-09-17T16:25:05.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.19e-06 2024-09-17T16:25:05.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.377996+0000", 2024-09-17T16:25:05.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8289999999999999e-06 2024-09-17T16:25:05.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391438+0000", 2024-09-17T16:25:05.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013442154 2024-09-17T16:25:05.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:05.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391448+0000", 2024-09-17T16:25:05.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9159999999999996e-06 2024-09-17T16:25:05.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991732+0000", 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.600283593 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991823+0000", 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1145999999999998e-05 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.991905+0000", 2024-09-17T16:25:05.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1664999999999998e-05 2024-09-17T16:25:05.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:05.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.996755+0000", 2024-09-17T16:25:05.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048505700000000002 2024-09-17T16:25:05.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.000183+0000", 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034277359999999998 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.010393+0000", 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010210558 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015049+0000", 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046556740000000003 2024-09-17T16:25:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015106+0000", 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7339000000000001e-05 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015217+0000", 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110609 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124291 3.0 3:0f520599:::benchmark_data_smithi173_66272_object52335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 494.71490548499997, 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.64518043599999997, 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124291 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378001+0000", 2024-09-17T16:25:05.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0800000000000001e-07 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378002+0000", 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1620000000000001e-06 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378006+0000", 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9910000000000004e-06 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.378009+0000", 2024-09-17T16:25:05.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.785e-06 2024-09-17T16:25:05.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391464+0000", 2024-09-17T16:25:05.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013454729 2024-09-17T16:25:05.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:05.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.391467+0000", 2024-09-17T16:25:05.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4750000000000002e-06 2024-09-17T16:25:05.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.994266+0000", 2024-09-17T16:25:05.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.602799058 2024-09-17T16:25:05.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:05.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.994361+0000", 2024-09-17T16:25:05.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4487999999999994e-05 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,9,11", 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.994432+0000", 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1518000000000002e-05 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.002826+0000", 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083936540000000004 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:05.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.002845+0000", 2024-09-17T16:25:05.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9128999999999999e-05 2024-09-17T16:25:05.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.015269+0000", 2024-09-17T16:25:05.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012424269 2024-09-17T16:25:05.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.022862+0000", 2024-09-17T16:25:05.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075930640000000001 2024-09-17T16:25:05.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:05.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.022977+0000", 2024-09-17T16:25:05.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114539 2024-09-17T16:25:05.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:51.023181+0000", 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000204367 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124263 3.4 3:2eabebde:::benchmark_data_smithi173_66272_object52307:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382419+0000", 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 494.710486882, 2024-09-17T16:25:05.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.499722689, 2024-09-17T16:25:05.145 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.145 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.145 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:25:05.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:05.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:05.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:05.145 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124263 2024-09-17T16:25:05.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.145 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382419+0000", 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382419+0000", 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382421+0000", 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.536e-06 2024-09-17T16:25:05.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382427+0000", 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6500000000000001e-06 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382441+0000", 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4368e-05 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382449+0000", 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3680000000000002e-06 2024-09-17T16:25:05.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387414+0000", 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049646159999999998 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387421+0000", 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5889999999999999e-06 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882026+0000", 2024-09-17T16:25:05.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49460579599999999 2024-09-17T16:25:05.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882142+0000", 2024-09-17T16:25:05.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000115766 2024-09-17T16:25:05.149 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.149 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.149 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.150 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124265 3.4 3:2dc750d7:::benchmark_data_smithi173_66272_object52309:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:25:05.150 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:25:05.150 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 494.71041778199998, 2024-09-17T16:25:05.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49990343999999998, 2024-09-17T16:25:05.150 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.150 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.150 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:25:05.150 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:05.150 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:05.150 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124265 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382489+0000", 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5300000000000004e-07 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382492+0000", 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2370000000000001e-06 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382502+0000", 2024-09-17T16:25:05.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0037000000000001e-05 2024-09-17T16:25:05.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382509+0000", 2024-09-17T16:25:05.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9149999999999999e-06 2024-09-17T16:25:05.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387622+0000", 2024-09-17T16:25:05.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051124600000000001 2024-09-17T16:25:05.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:05.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387630+0000", 2024-09-17T16:25:05.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9510000000000006e-06 2024-09-17T16:25:05.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882341+0000", 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49471140899999999 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882392+0000", 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0878000000000003e-05 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.155 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.156 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124266 3.4 3:2de8446c:::benchmark_data_smithi173_66272_object52310:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:25:05.156 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:25:05.156 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 494.71038746099998, 2024-09-17T16:25:05.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.51201529199999996, 2024-09-17T16:25:05.157 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.157 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.157 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:05.157 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:05.157 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:05.157 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124266 2024-09-17T16:25:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:25:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:25:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382519+0000", 2024-09-17T16:25:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2899999999999999e-07 2024-09-17T16:25:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382521+0000", 2024-09-17T16:25:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8959999999999999e-06 2024-09-17T16:25:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382527+0000", 2024-09-17T16:25:05.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9950000000000002e-06 2024-09-17T16:25:05.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382532+0000", 2024-09-17T16:25:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1749999999999996e-06 2024-09-17T16:25:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387669+0000", 2024-09-17T16:25:05.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051369980000000003 2024-09-17T16:25:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387680+0000", 2024-09-17T16:25:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1e-05 2024-09-17T16:25:05.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882422+0000", 2024-09-17T16:25:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49474148299999998 2024-09-17T16:25:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882507+0000", 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4802999999999994e-05 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,5,11", 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.882611+0000", 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104527 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.890862+0000", 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082508949999999994 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893785+0000", 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029228700000000002 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893832+0000", 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7620999999999997e-05 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.894360+0000", 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000527605 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.894418+0000", 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8057000000000003e-05 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.894534+0000", 2024-09-17T16:25:05.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116038 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124283 3.4 3:2189d34a:::benchmark_data_smithi173_66272_object52327:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.382540+0000", 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 494.71036614799999, 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.52263484000000004, 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:05.168 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124283 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382540+0000", 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382540+0000", 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382541+0000", 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6899999999999998e-07 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382543+0000", 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0530000000000001e-06 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382549+0000", 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0580000000000003e-06 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.382554+0000", 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0300000000000001e-06 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387707+0000", 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051537110000000001 2024-09-17T16:25:05.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.387715+0000", 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8220000000000007e-06 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.886356+0000", 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49864073799999997 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:05.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.886497+0000", 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014137500000000001 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,5,11", 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.886624+0000", 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126551 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893848+0000", 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007223648 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.893892+0000", 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4084e-05 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.896419+0000", 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002527502 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.904985+0000", 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0085655720000000005 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.905056+0000", 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1730999999999995e-05 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.905175+0000", 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118596 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(3.c e290,267 3:3e4663e2:::benchmark_data_smithi173_66272_object807:head,251'4519)", 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.493506+0000", 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 464.59940045299999, 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62512154200000003, 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:05.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493506+0000", 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493506+0000", 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493507+0000", 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.164e-06 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493514+0000", 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3549999999999999e-06 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118475+0000", 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62496048599999998 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118483+0000", 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6769999999999995e-06 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:05.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118554+0000", 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1210000000000004e-05 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118561+0000", 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.029e-06 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.118627+0000", 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6620999999999998e-05 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:05.179 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:05.180 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:25:05.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:05.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:05.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:05 smithi017 ceph-mon[27571]: pgmap v1155: 17 pgs: 3 active+clean, 14 unknown; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:05 smithi017 ceph-mon[27571]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:25:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:05 smithi017 ceph-mon[27571]: osdmap e529: 12 total, 12 up, 10 in 2024-09-17T16:25:05.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:05.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:05 smithi071 ceph-mon[30002]: pgmap v1155: 17 pgs: 3 active+clean, 14 unknown; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:05.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:05 smithi071 ceph-mon[30002]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:25:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:05 smithi071 ceph-mon[30002]: osdmap e529: 12 total, 12 up, 10 in 2024-09-17T16:25:05.334 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:05.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:05.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:05 smithi173 ceph-mon[30530]: pgmap v1155: 17 pgs: 3 active+clean, 14 unknown; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:05 smithi173 ceph-mon[30530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:25:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:05 smithi173 ceph-mon[30530]: osdmap e529: 12 total, 12 up, 10 in 2024-09-17T16:25:05.436 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:05.461 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:05.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:05.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:05.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:05.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:05.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:05.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:05.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:05.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:05.845 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:05.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:05.947 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:05.974 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:05.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:05.992 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:25:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:06.057 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:06.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:06.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:06.101 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:06.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:06.202 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:06.226 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:06.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:06.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:06.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:06.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:06.354 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:06.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:06.455 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:06.479 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:06.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:06.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:06.604 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:06.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:06.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:25:06.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:06.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:06.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:06.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:06.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:06.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:06.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:06.987 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:06.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:06.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:07.045 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-17T16:25:07.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:07.091 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:07 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:07.091 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:07 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:07.091 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:07 smithi017 ceph-mon[27571]: pgmap v1157: 17 pgs: 8 active+clean, 9 unknown; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:07.092 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:07 smithi017 ceph-mon[27571]: osdmap e530: 12 total, 12 up, 10 in 2024-09-17T16:25:07.092 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2814484671' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:25:07.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:07.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:07.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:07.240 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:07.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:07 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:07 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:07 smithi071 ceph-mon[30002]: pgmap v1157: 17 pgs: 8 active+clean, 9 unknown; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:07 smithi071 ceph-mon[30002]: osdmap e530: 12 total, 12 up, 10 in 2024-09-17T16:25:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2814484671' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:25:07.343 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:07.364 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:07.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:07 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:07 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:07 smithi173 ceph-mon[30530]: pgmap v1157: 17 pgs: 8 active+clean, 9 unknown; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:07.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:07 smithi173 ceph-mon[30530]: osdmap e530: 12 total, 12 up, 10 in 2024-09-17T16:25:07.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2814484671' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:25:07.465 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:07.489 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:07.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:07.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:07.614 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:07.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:07.716 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:07.738 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:07.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:07.840 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:07.862 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:07.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:07.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:08.050 INFO:teuthology.orchestra.run.smithi017.stderr:set 1.0 pg_upmap mapping to [4,6,8] 2024-09-17T16:25:08.060 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:08.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:08.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:08.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:08.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2814484671' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:25:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:08 smithi017 ceph-mon[27571]: osdmap e531: 12 total, 12 up, 10 in 2024-09-17T16:25:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/681443331' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.0", "id": [4, 6, 8]} : dispatch 2024-09-17T16:25:08.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:08.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2814484671' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:25:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:08 smithi071 ceph-mon[30002]: osdmap e531: 12 total, 12 up, 10 in 2024-09-17T16:25:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/681443331' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.0", "id": [4, 6, 8]} : dispatch 2024-09-17T16:25:08.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:08.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2814484671' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:25:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:08 smithi173 ceph-mon[30530]: osdmap e531: 12 total, 12 up, 10 in 2024-09-17T16:25:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/681443331' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.0", "id": [4, 6, 8]} : dispatch 2024-09-17T16:25:08.418 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:08.442 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:08.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:08.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:08.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:08.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:08.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:08.671 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:08.695 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:08.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:08.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:08.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:08.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:08.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:08.949 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:08.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:09.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:09.055 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:25:09.075 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:09.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:09.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:09.201 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:09.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:09.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:09 smithi017 ceph-mon[27571]: pgmap v1160: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:09.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:09 smithi017 ceph-mon[27571]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:25:09.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/681443331' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [4, 6, 8]}]': finished 2024-09-17T16:25:09.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:09 smithi017 ceph-mon[27571]: osdmap e532: 12 total, 12 up, 10 in 2024-09-17T16:25:09.227 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/262262863' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:09.303 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:09 smithi071 ceph-mon[30002]: pgmap v1160: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:09 smithi071 ceph-mon[30002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:25:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/681443331' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [4, 6, 8]}]': finished 2024-09-17T16:25:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:09 smithi071 ceph-mon[30002]: osdmap e532: 12 total, 12 up, 10 in 2024-09-17T16:25:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/262262863' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:09.315 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:09 smithi173 ceph-mon[30530]: pgmap v1160: 17 pgs: 17 active+clean; 577 KiB data, 3.5 GiB used, 891 GiB / 894 GiB avail 2024-09-17T16:25:09.315 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:09 smithi173 ceph-mon[30530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:25:09.315 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/681443331' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [4, 6, 8]}]': finished 2024-09-17T16:25:09.315 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:09 smithi173 ceph-mon[30530]: osdmap e532: 12 total, 12 up, 10 in 2024-09-17T16:25:09.315 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/262262863' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:09.327 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:09.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:09.429 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:09.454 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:09.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:09.457 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:09.476 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-17T16:25:09.478 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T16:25:09.479 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T16:25:09.496 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:09.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-09-17T16:25:09.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T16:25:09.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_102017 2024-09-17T16:25:09.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:25:09.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-09-17T16:25:09.556 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:09.585 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:09.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:09.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:09.712 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:09.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:09.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:09.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:09.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:09.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:09.946 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:09.982 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:09.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:10.085 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:10.113 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:10.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:10.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:10.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:10 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:10.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/262262863' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:10.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:10 smithi173 ceph-mon[30530]: osdmap e533: 12 total, 12 up, 10 in 2024-09-17T16:25:10.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:10 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/262262863' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:10 smithi017 ceph-mon[27571]: osdmap e533: 12 total, 12 up, 10 in 2024-09-17T16:25:10.243 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:10.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:10 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/262262863' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:10 smithi071 ceph-mon[30002]: osdmap e533: 12 total, 12 up, 10 in 2024-09-17T16:25:10.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:10.370 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:10.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:10.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:10.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:10.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:10.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:10.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:10.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 2218 2202 137.617 137.625 0.0116891 0.00683632 2024-09-17T16:25:10.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:10.620 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:10.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:10.675 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:25:10.676 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:25:10.701 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:25:10.702 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:25:10.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:10.722 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:25:10.723 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:25:10.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:25:10.744 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:25:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:10.745 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:10.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:10.766 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:25:10.766 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:25:10.787 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:25:10.787 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:25:10.810 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:25:10.810 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:25:10.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:10.869 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:10.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:10.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:10.918 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:25:10.919 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:25:10.931 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:25:10.931 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:25:10.957 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:25:10.958 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:25:10.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:10.981 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:25:10.982 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:25:10.992 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:10.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:10.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:11.007 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:25:11.007 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:25:11.032 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:25:11.032 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:25:11.094 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:11.116 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:11.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:11.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:11.218 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:11.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:11 smithi017 ceph-mon[27571]: pgmap v1163: 17 pgs: 17 active+clean; 6.2 MiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 1.4 MiB/s wr, 22 op/s 2024-09-17T16:25:11.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:11 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:11.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:11 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:11.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1448997914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:11.242 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:11.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:11 smithi071 ceph-mon[30002]: pgmap v1163: 17 pgs: 17 active+clean; 6.2 MiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 1.4 MiB/s wr, 22 op/s 2024-09-17T16:25:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:11 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:11 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1448997914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:11.343 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:11.366 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:11.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:11 smithi173 ceph-mon[30530]: pgmap v1163: 17 pgs: 17 active+clean; 6.2 MiB data, 3.5 GiB used, 891 GiB / 894 GiB avail; 1.4 MiB/s wr, 22 op/s 2024-09-17T16:25:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:11 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:11 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1448997914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:11.468 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:11.499 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:11.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:11.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 3284 3268 102.118 66.625 0.00549757 0.00976734 2024-09-17T16:25:11.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:11.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:11.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:11.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:11.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:11.760 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:11.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:11.862 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:11.889 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:11.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:11.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:11.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:12.021 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:12.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:12.123 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:12.144 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:12.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:12.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:12.274 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:12.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:12.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:12.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:12.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:12.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:12.436 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:25:12.504 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:12.528 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:12.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:12.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 5475 5459 113.721 136.938 0.00306575 0.00875869 2024-09-17T16:25:12.630 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:12.657 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:12.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:12 smithi017 ceph-mon[27571]: osdmap e534: 12 total, 12 up, 10 in 2024-09-17T16:25:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:12 smithi017 ceph-mon[27571]: pgmap v1165: 17 pgs: 17 active+clean; 125 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 25 MiB/s wr, 407 op/s 2024-09-17T16:25:12.744 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:25:12.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:12.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:12.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:12.783 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:12.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:12 smithi071 ceph-mon[30002]: osdmap e534: 12 total, 12 up, 10 in 2024-09-17T16:25:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:12 smithi071 ceph-mon[30002]: pgmap v1165: 17 pgs: 17 active+clean; 125 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 25 MiB/s wr, 407 op/s 2024-09-17T16:25:12.883 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:12 smithi173 ceph-mon[30530]: osdmap e534: 12 total, 12 up, 10 in 2024-09-17T16:25:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:12 smithi173 ceph-mon[30530]: pgmap v1165: 17 pgs: 17 active+clean; 125 MiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 25 MiB/s wr, 407 op/s 2024-09-17T16:25:12.906 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:12.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:13.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:13.034 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:13.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:13.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:13.161 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:13.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:13.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:13.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:13.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:13.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:13.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:13.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:13.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:13.416 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:13.516 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:13.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 6650 6634 103.649 73.4375 0.00652132 0.00960801 2024-09-17T16:25:13.539 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:13.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:13.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:13.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:13.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:13.665 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:13.685 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:25:13.766 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:13.795 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:13.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:13.897 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:13.918 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:13.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:13.954 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:25:13.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2883435852' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:25:14.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:14.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:14.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:14.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2883435852' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:25:14.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2883435852' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:25:14.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:14.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:14.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:14.202 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:25:14.202 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:25:14.259 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] out_osds: [1, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:25:14.260 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:25:14.260 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-17T16:25:14.260 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd in 1 2024-09-17T16:25:14.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:14.298 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:14.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:14.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:14.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:14.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:14.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:14.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:14.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:14.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 8028 8012 100.143 86.125 0.00187252 0.0099723 2024-09-17T16:25:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:14.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:14.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:14.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:14.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:14.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:14.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:14.787 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:14.814 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:14.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:14.916 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:14.938 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:14.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2883435852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:25:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:14 smithi017 ceph-mon[27571]: osdmap e535: 12 total, 12 up, 10 in 2024-09-17T16:25:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:14 smithi017 ceph-mon[27571]: pgmap v1167: 17 pgs: 17 active+clean; 173 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 29 MiB/s wr, 469 op/s 2024-09-17T16:25:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:14 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:15.038 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:15.062 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:15.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:15.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2883435852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:25:15.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:14 smithi071 ceph-mon[30002]: osdmap e535: 12 total, 12 up, 10 in 2024-09-17T16:25:15.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:14 smithi071 ceph-mon[30002]: pgmap v1167: 17 pgs: 17 active+clean; 173 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 29 MiB/s wr, 469 op/s 2024-09-17T16:25:15.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:14 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2883435852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:25:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:14 smithi173 ceph-mon[30530]: osdmap e535: 12 total, 12 up, 10 in 2024-09-17T16:25:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:14 smithi173 ceph-mon[30530]: pgmap v1167: 17 pgs: 17 active+clean; 173 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 29 MiB/s wr, 469 op/s 2024-09-17T16:25:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:14 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:15.163 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:15.187 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:15.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:15.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:15.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:15.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:15.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:15.418 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:15.430 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:15.453 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:15.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:15.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:15.459 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:25:15.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 9587 9571 99.6904 97.4375 0.00810902 0.0100056 2024-09-17T16:25:15.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:15.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:15.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:15.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:15.710 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:15.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:15.812 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:15.835 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:15.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:15.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:15.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:15.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:15.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:16.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:16.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:16.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:16.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:16.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:16.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:16.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:16.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:16.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:16.382 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:25:16.383 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:25:16.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:25:16.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:25:16.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:25:16.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:25:16.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:25:16.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:25:16.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:25:16.384 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:25:16.384 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:25:16.384 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:25:16.385 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:25:16.385 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:25:16.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:16.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:16.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:16.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 11125 11109 99.1798 96.125 0.0333515 0.0100559 2024-09-17T16:25:16.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:16.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:16.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:16.663 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:16.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:16.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:16.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:16.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:16.773 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:25:16.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:16.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:16.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:16.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:16.950 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:16.973 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:16.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:17.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:17.101 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:17.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:17.203 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:17.226 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:17.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:17.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:16 smithi017 ceph-mon[27571]: pgmap v1168: 17 pgs: 17 active+clean; 326 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 47 MiB/s wr, 759 op/s 2024-09-17T16:25:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2786436828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:17.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:16 smithi071 ceph-mon[30002]: pgmap v1168: 17 pgs: 17 active+clean; 326 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 47 MiB/s wr, 759 op/s 2024-09-17T16:25:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2786436828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:17.328 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:17.330 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:17.353 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:17.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:17.360 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:16 smithi173 ceph-mon[30530]: pgmap v1168: 17 pgs: 17 active+clean; 326 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 47 MiB/s wr, 759 op/s 2024-09-17T16:25:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2786436828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:17.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:17.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:17.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:17.537 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 12825 12809 100.062 106.25 0.0261408 0.00996993 2024-09-17T16:25:17.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:17.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:17.604 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:17.705 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:17.728 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:17.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:17.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:17.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:17.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:17.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:17.974 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:17.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:18.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:18.108 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:18.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:18.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:18.237 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:18.238 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:25:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:25:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:25:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:25:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25516.0:155234 4.6 e371/361 4:60535f2d:::benchmark_data_smithi173_77474_object53059:head v 371'9504, mlcod=371'9501)", 2024-09-17T16:25:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:25:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 305.56060049299998, 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30676646000000002, 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589698+0000", 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.589704+0000", 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5330000000000002e-06 2024-09-17T16:25:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.891997+0000", 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30229256300000001 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.892001+0000", 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6539999999999998e-06 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.892046+0000", 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5305000000000001e-05 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.892062+0000", 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5931999999999999e-05 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.896408+0000", 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043459869999999999 2024-09-17T16:25:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:12.896464+0000", 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5717000000000002e-05 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:185136 4.6 4:6e122307:::benchmark_data_smithi173_77474_object82961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:55.713056+0000", 2024-09-17T16:25:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 262.43724217599998, 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30286027900000001, 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 185136 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713056+0000", 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713056+0000", 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713058+0000", 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1100000000000001e-06 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713068+0000", 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7890000000000006e-06 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713085+0000", 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6660999999999999e-05 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.713092+0000", 2024-09-17T16:25:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5159999999999997e-06 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.714377+0000", 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012851379999999999 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.714505+0000", 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012745500000000001 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:25:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.714629+0000", 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012392699999999999 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.718974+0000", 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043457169999999998 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.718988+0000", 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.375e-05 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015789+0000", 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29680071600000002 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015866+0000", 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7026000000000004e-05 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015916+0000", 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0473999999999998e-05 2024-09-17T16:25:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:185136 4.6 e391/387 ondisk, result = 0)", 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:55.717118+0000", 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 262.43317996000002, 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29880589800000001, 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717118+0000", 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717118+0000", 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717120+0000", 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.392e-06 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:55.717126+0000", 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8279999999999997e-06 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015002+0000", 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.297875367 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015011+0000", 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1519999999999998e-06 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015769+0000", 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075783199999999995 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015780+0000", 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.166e-05 2024-09-17T16:25:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.015924+0000", 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014366699999999999 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:185177 4.6 4:6ef442e2:::benchmark_data_smithi173_77474_object83002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:56.206993+0000", 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 261.94330479899997, 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39139948400000002, 2024-09-17T16:25:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 185177 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.206993+0000", 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.206993+0000", 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.206995+0000", 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0619999999999999e-06 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207004+0000", 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5019999999999992e-06 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207022+0000", 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8365000000000001e-05 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207030+0000", 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5639999999999999e-06 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207096+0000", 2024-09-17T16:25:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5938999999999993e-05 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207197+0000", 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100929 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.207279+0000", 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2474999999999999e-05 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211157+0000", 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003878009 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.423507+0000", 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21234955799999999 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598152+0000", 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.174645726 2024-09-17T16:25:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598251+0000", 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8258999999999996e-05 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598393+0000", 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142096 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:185177 4.6 e391/387 ondisk, result = 0)", 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:56.211354+0000", 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 261.93894424699999, 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38704429800000001, 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211354+0000", 2024-09-17T16:25:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211354+0000", 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211355+0000", 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8899999999999998e-07 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.211363+0000", 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4509999999999992e-06 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.597986+0000", 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38662227900000001 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.597997+0000", 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.189e-05 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598126+0000", 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000128505 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598143+0000", 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7405999999999999e-05 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.598398+0000", 2024-09-17T16:25:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025487799999999999 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:186301 4.4 4:28479395:::benchmark_data_smithi173_77474_object84126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:58.360185+0000", 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 259.790113066, 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37107270399999998, 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186301 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360185+0000", 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360185+0000", 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360186+0000", 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.02e-06 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360191+0000", 2024-09-17T16:25:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6269999999999999e-06 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360201+0000", 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0583e-05 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360207+0000", 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1989999999999997e-06 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360255+0000", 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8810999999999997e-05 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360347+0000", 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1436999999999999e-05 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:25:18.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.360424+0000", 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7566999999999998e-05 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364029+0000", 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036048090000000001 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364399+0000", 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000370331 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731169+0000", 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36676913100000003 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731215+0000", 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6047000000000001e-05 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731258+0000", 2024-09-17T16:25:18.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3142000000000001e-05 2024-09-17T16:25:18.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.272 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.272 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:186301 4.4 e393/306 ondisk, result = 0)", 2024-09-17T16:25:18.272 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:58.364680+0000", 2024-09-17T16:25:18.272 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 259.78561778400001, 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36658104899999999, 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364680+0000", 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364680+0000", 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364681+0000", 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6899999999999997e-07 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.364685+0000", 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1579999999999998e-06 2024-09-17T16:25:18.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731114+0000", 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.366428431 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731117+0000", 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5549999999999998e-06 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731158+0000", 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0655999999999998e-05 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731163+0000", 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1869999999999996e-06 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.731261+0000", 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8393000000000005e-05 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204346 4.4 4:22db2208:::benchmark_data_smithi173_77474_object102171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.625061+0000", 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 234.525236963, 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.67454315899999995, 2024-09-17T16:25:18.277 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204346 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625061+0000", 2024-09-17T16:25:18.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625061+0000", 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625062+0000", 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0300000000000001e-06 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625067+0000", 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4320000000000001e-06 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625077+0000", 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9189999999999994e-06 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625083+0000", 2024-09-17T16:25:18.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9020000000000002e-06 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625140+0000", 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6677e-05 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625237+0000", 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6446999999999993e-05 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625308+0000", 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1585e-05 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.629047+0000", 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037388569999999999 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.630013+0000", 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096557000000000001 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299522+0000", 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.66950939200000004 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299565+0000", 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.316e-05 2024-09-17T16:25:18.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299604+0000", 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9187999999999998e-05 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:2256 5.7 e424/419 5:e0aee0a5:::benchmark_data_smithi173_90735_object2255:head v 424'163, mlcod=424'160)", 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:51.481567+0000", 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 206.66873069299999, 2024-09-17T16:25:18.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27652685100000002, 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.481567+0000", 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.481567+0000", 2024-09-17T16:25:18.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.481568+0000", 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6300000000000004e-07 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.481611+0000", 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2704000000000003e-05 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.757551+0000", 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27593994300000002 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.757561+0000", 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0052e-05 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.757619+0000", 2024-09-17T16:25:18.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8112000000000002e-05 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.757636+0000", 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6789000000000001e-05 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.758061+0000", 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042476699999999999 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:51.758094+0000", 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3720999999999998e-05 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:30436 5.a 5:5adf2834:::benchmark_data_smithi173_90735_object30435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:14.766561+0000", 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 183.38373664599999, 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38050945800000002, 2024-09-17T16:25:18.289 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 30436 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766561+0000", 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766561+0000", 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766564+0000", 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1490000000000001e-06 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766601+0000", 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7072e-05 2024-09-17T16:25:18.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766613+0000", 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2714000000000001e-05 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766619+0000", 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6210000000000001e-06 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766677+0000", 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7572999999999999e-05 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766839+0000", 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016253700000000001 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,10", 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.766995+0000", 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015596800000000001 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.768067+0000", 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010722049999999999 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:14.768388+0000", 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032046199999999998 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:15.146963+0000", 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37857484499999999 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:15.147022+0000", 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9108999999999999e-05 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:15.147071+0000", 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9203000000000003e-05 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:41583 5.a 5:52d43247:::benchmark_data_smithi173_90735_object41582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-17T16:25:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:23.939462+0000", 2024-09-17T16:25:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 174.21083566600001, 2024-09-17T16:25:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27797282600000001, 2024-09-17T16:25:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.296 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41583 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939462+0000", 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939462+0000", 2024-09-17T16:25:18.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939463+0000", 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9100000000000002e-07 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939505+0000", 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1600999999999997e-05 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:18.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939515+0000", 2024-09-17T16:25:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0448999999999999e-05 2024-09-17T16:25:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939526+0000", 2024-09-17T16:25:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.027e-05 2024-09-17T16:25:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939586+0000", 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0689000000000001e-05 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939714+0000", 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012727999999999999 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,10", 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.939864+0000", 2024-09-17T16:25:18.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000150419 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.941306+0000", 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001441459 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.948800+0000", 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074941019999999999 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217291+0000", 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.268491539 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217377+0000", 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5650999999999997e-05 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217435+0000", 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8476000000000001e-05 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.303 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26158.0:41583 5.a e440/419 ondisk, result = 0)", 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:23.941903+0000", 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 174.20839484499999, 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27554273499999998, 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.941903+0000", 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.941903+0000", 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.941904+0000", 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8400000000000004e-07 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.941909+0000", 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0599999999999998e-06 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217044+0000", 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27513484599999999 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217061+0000", 2024-09-17T16:25:18.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7237000000000001e-05 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217271+0000", 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021042399999999999 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217284+0000", 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2237e-05 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.217446+0000", 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000162347 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:41594 5.a 5:5f46650c:::benchmark_data_smithi173_90735_object41593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:23.946995+0000", 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 174.203302882, 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62805596699999999, 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.308 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41594 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.946995+0000", 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.946995+0000", 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.946999+0000", 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7570000000000002e-06 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947087+0000", 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8115999999999995e-05 2024-09-17T16:25:18.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947127+0000", 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9746000000000002e-05 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947149+0000", 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1704000000000001e-05 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947258+0000", 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010904399999999999 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947407+0000", 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014971 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,10", 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.947573+0000", 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165639 2024-09-17T16:25:18.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.948902+0000", 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001329326 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.952783+0000", 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038808900000000001 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574883+0000", 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62209975299999998 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574982+0000", 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8795999999999994e-05 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.575051+0000", 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9486000000000005e-05 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.314 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26158.0:41594 5.a e440/419 ondisk, result = 0)", 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:23.949386+0000", 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 174.200911647, 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62567850599999997, 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.949386+0000", 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.949386+0000", 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.949387+0000", 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6199999999999998e-07 2024-09-17T16:25:18.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:23.949390+0000", 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3620000000000002e-06 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574683+0000", 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62529263099999999 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574708+0000", 2024-09-17T16:25:18.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5130000000000002e-05 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574856+0000", 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147732 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.574873+0000", 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7430000000000001e-05 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:24.575065+0000", 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000191759 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:54616 5.17 5:eee108da:::benchmark_data_smithi173_90735_object54615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:22:39.739821+0000", 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 158.41047689199999, 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58859298699999996, 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.319 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54616 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739821+0000", 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739821+0000", 2024-09-17T16:25:18.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739822+0000", 2024-09-17T16:25:18.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.02e-07 2024-09-17T16:25:18.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739877+0000", 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5259000000000001e-05 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739889+0000", 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1405e-05 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739894+0000", 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.135e-06 2024-09-17T16:25:18.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.739944+0000", 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0217999999999999e-05 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.740046+0000", 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000101471 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,8,11", 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.740149+0000", 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103126 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.741188+0000", 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001039138 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.742452+0000", 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012636100000000001 2024-09-17T16:25:18.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:39.742555+0000", 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103479 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.327594+0000", 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58503854899999996 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.327627+0000", 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3605000000000001e-05 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:22:40.328414+0000", 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078708999999999995 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:82768 5.6 5:63886666:::benchmark_data_smithi173_90735_object82767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:19.925436+0000", 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 118.22486198199999, 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.83184638399999999, 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 82768 2024-09-17T16:25:18.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925436+0000", 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925436+0000", 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925438+0000", 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925521+0000", 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3159999999999997e-05 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925532+0000", 2024-09-17T16:25:18.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1735e-05 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925538+0000", 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4870000000000002e-06 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925599+0000", 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1247999999999994e-05 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925711+0000", 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112236 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.925799+0000", 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7453000000000004e-05 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.927073+0000", 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012737180000000001 2024-09-17T16:25:18.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.927421+0000", 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00034853299999999999 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.757202+0000", 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.829780505 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.757247+0000", 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5207999999999998e-05 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.757283+0000", 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5734e-05 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(5.f 475/455 [PushOp(5:ff17ea6b:::benchmark_data_smithi173_90735_object41658:head, version: 442'2553, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ff17ea6b:::benchmark_data_smithi173_90735_object41658:head@442'2553, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:25:18.333 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:20.334257+0000", 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 117.816041498, 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.573612715, 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.334257+0000", 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.334257+0000", 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.334257+0000", 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6700000000000003e-07 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.334304+0000", 2024-09-17T16:25:18.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6513e-05 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.906366+0000", 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.572062183 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.906385+0000", 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9215000000000002e-05 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.906471+0000", 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.906481+0000", 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7990000000000005e-06 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.907869+0000", 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001388249 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.337 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:114326 5.a 5:55b0fcfc:::benchmark_data_smithi173_90735_object31555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:54.016678+0000", 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 84.133620280000002, 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28172619999999998, 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:18.338 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 114326 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.016678+0000", 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.016678+0000", 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.016679+0000", 2024-09-17T16:25:18.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.023e-06 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.016684+0000", 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0259999999999998e-06 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.016695+0000", 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.016702+0000", 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5050000000000001e-06 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.016758+0000", 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5433999999999998e-05 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.016846+0000", 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7880000000000006e-05 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,10", 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.016915+0000", 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9282000000000005e-05 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.019504+0000", 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0025894109999999998 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.297152+0000", 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27764728999999999 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.298333+0000", 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011812050000000001 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.298365+0000", 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1794000000000001e-05 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.298404+0000", 2024-09-17T16:25:18.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9455000000000001e-05 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26158.0:114346 5.6 5:620fc355:::benchmark_data_smithi173_90735_object31575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:54.067911+0000", 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 84.082387377000003, 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.227755285, 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.344 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26158", 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 114346 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.067911+0000", 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.067911+0000", 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.067913+0000", 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9719999999999999e-06 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.067921+0000", 2024-09-17T16:25:18.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1319999999999994e-06 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.067940+0000", 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9630999999999999e-05 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.067950+0000", 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3940000000000002e-06 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.068012+0000", 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1978999999999997e-05 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.068106+0000", 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3966000000000001e-05 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.068179+0000", 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2846e-05 2024-09-17T16:25:18.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.069666+0000", 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0014869810000000001 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.069983+0000", 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031726299999999997 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:18.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.295563+0000", 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22558025600000001 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.295620+0000", 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6907000000000003e-05 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:54.295666+0000", 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5958000000000003e-05 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.350 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26665.0:12937 6.5 e535/527 6:ad850d77:::benchmark_data_smithi173_102017_object12936:head v 535'795, mlcod=535'792)", 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:17.847719+0000", 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.302578925, 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.13493596799999999, 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:17.847719+0000", 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:17.847719+0000", 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:17.847720+0000", 2024-09-17T16:25:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7400000000000003e-07 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:17.847792+0000", 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2222000000000001e-05 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:17.981902+0000", 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.134109797 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:18.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:17.981910+0000", 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3210000000000005e-06 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:17.981965+0000", 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5191000000000001e-05 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:17.981978+0000", 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2232000000000001e-05 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:17.982607+0000", 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062962300000000005 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:17.982655+0000", 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7908000000000003e-05 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:18.355 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:25:18.400 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:18.425 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:18.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:18.527 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:18.538 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 13523 13507 93.7902 43.625 0.00437974 0.0105751 2024-09-17T16:25:18.548 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:18.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:18.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:18.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:18.672 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:18.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:18.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:18.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:18.902 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:18.929 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:18.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:18.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:18 smithi017 ceph-mon[27571]: pgmap v1169: 17 pgs: 17 active+clean; 606 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-17T16:25:18.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1939581689' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:19.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:19.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:19.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:19.069 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:25:19.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:18 smithi173 ceph-mon[30530]: pgmap v1169: 17 pgs: 17 active+clean; 606 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-17T16:25:19.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1939581689' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:19.154 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:19.181 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:19.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:19.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:19.306 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:19.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:19.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:18 smithi071 ceph-mon[30002]: pgmap v1169: 17 pgs: 17 active+clean; 606 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-17T16:25:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1939581689' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:19.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:25:19.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:19.430 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:19.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:19.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:19.538 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 14487 14471 90.4352 60.25 0.0191566 0.0110166 2024-09-17T16:25:19.554 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:19.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:19.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:19.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:19.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:19.781 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:19.802 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:19.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:19.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:19.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:19.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:20.027 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:20.050 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:20.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:20.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:20.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:20.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:20.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:20.222 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:25:20.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:19 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:25:20.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:19 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:25:20.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1939581689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:20.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:19 smithi017 ceph-mon[27571]: osdmap e536: 12 total, 12 up, 10 in 2024-09-17T16:25:20.277 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:20.303 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:20.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:20.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:19 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:25:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:19 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:25:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1939581689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:19 smithi071 ceph-mon[30002]: osdmap e536: 12 total, 12 up, 10 in 2024-09-17T16:25:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:19 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:25:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:19 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:25:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1939581689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:19 smithi173 ceph-mon[30530]: osdmap e536: 12 total, 12 up, 10 in 2024-09-17T16:25:20.405 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:20.430 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:20.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:20.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:20.538 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 15518 15502 88.0713 64.4375 0.0267619 0.0113425 2024-09-17T16:25:20.555 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:20.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:20.656 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:20.683 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:20.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:20.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:20.809 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:20.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:20.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:20.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:20.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:20.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:20.934 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:21.035 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:21.058 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:21.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:21.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:20 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:20 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:20 smithi173 ceph-mon[30530]: pgmap v1171: 17 pgs: 17 active+clean; 736 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-17T16:25:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:20 smithi173 ceph-mon[30530]: osdmap e537: 12 total, 12 up, 10 in 2024-09-17T16:25:21.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:21.179 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:21.180 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:21.183 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:21.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:21.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:21.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:20 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:21.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:20 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:21.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:20 smithi017 ceph-mon[27571]: pgmap v1171: 17 pgs: 17 active+clean; 736 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-17T16:25:21.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:20 smithi017 ceph-mon[27571]: osdmap e537: 12 total, 12 up, 10 in 2024-09-17T16:25:21.285 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:21.311 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:21.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:21.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:20 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:20 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:20 smithi071 ceph-mon[30002]: pgmap v1171: 17 pgs: 17 active+clean; 736 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-17T16:25:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:20 smithi071 ceph-mon[30002]: osdmap e537: 12 total, 12 up, 10 in 2024-09-17T16:25:21.412 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:21.434 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:21.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:21.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:21.538 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 16752 16736 87.1583 77.125 0.0499455 0.0114024 2024-09-17T16:25:21.558 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:21.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:21.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:21.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:21.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:21.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:21.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:21 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:25:21.302+0000 7f086a07d640 -1 mon.a@0(leader).osd e537 definitely_dead 0 2024-09-17T16:25:21.793 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:21.822 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:21.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:21.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:21.952 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:21.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:21.984 INFO:teuthology.orchestra.run.smithi017.stderr:marked in osd.1. 2024-09-17T16:25:22.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:22.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:22.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:22.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:22.199 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:22.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:22.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:22.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4266753421' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-17T16:25:22.301 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:22.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4266753421' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-17T16:25:22.321 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:22.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:22.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4266753421' entity='client.admin' cmd={"prefix": "osd in", "ids": ["1"]} : dispatch 2024-09-17T16:25:22.422 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:22.442 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:25:22.442 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:25:22.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.442 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17629 6.8 6:196f18af:::benchmark_data_smithi173_102017_object17628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:25:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:21.997934+0000", 2024-09-17T16:25:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.44225774699999998, 2024-09-17T16:25:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44229275299999998, 2024-09-17T16:25:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:25:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:22.443 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17629 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997934+0000", 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997934+0000", 2024-09-17T16:25:22.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997936+0000", 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3859999999999999e-06 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998411+0000", 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000475902 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998433+0000", 2024-09-17T16:25:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.141e-05 2024-09-17T16:25:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998452+0000", 2024-09-17T16:25:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9619e-05 2024-09-17T16:25:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998531+0000", 2024-09-17T16:25:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8924000000000005e-05 2024-09-17T16:25:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998705+0000", 2024-09-17T16:25:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017413300000000001 2024-09-17T16:25:22.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:25:22.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998883+0000", 2024-09-17T16:25:22.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017767 2024-09-17T16:25:22.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000058+0000", 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001174701 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000188+0000", 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013036300000000001 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17641 6.7 6.aff5e7f7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:25:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:25:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.31883671800000002, 2024-09-17T16:25:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31891909299999999, 2024-09-17T16:25:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for peered", 2024-09-17T16:25:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:25:22.451 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17641 2024-09-17T16:25:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:25:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:25:22.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121357+0000", 2024-09-17T16:25:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4020000000000001e-06 2024-09-17T16:25:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:22.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121396+0000", 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9561999999999998e-05 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121410+0000", 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3645000000000001e-05 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121415+0000", 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6169999999999998e-06 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121465+0000", 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9648000000000003e-05 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121474+0000", 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.568e-06 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17642 6.7 6.5a462497 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.246051088, 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24616421599999999, 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for peered", 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17642 2024-09-17T16:25:22.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:22.458 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194142+0000", 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2399999999999997e-07 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194163+0000", 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1841e-05 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194175+0000", 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1487e-05 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194180+0000", 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2639999999999999e-06 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194227+0000", 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6675000000000003e-05 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194233+0000", 2024-09-17T16:25:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1229999999999997e-06 2024-09-17T16:25:22.461 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:22.461 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:22.461 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:22.461 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:22.461 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:25:22.461 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 3 2024-09-17T16:25:22.461 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:25:22.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:22.462 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:22.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:22.538 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 17648 17632 84.7606 56 0.40246 0.0114637 2024-09-17T16:25:22.563 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:22.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:22.586 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:22.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:22.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:22.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:22.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:22.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:22.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:22.820 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:25:22.832 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:22.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:22.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:22.957 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:22.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:23.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:23.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:23.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:23.182 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:23.206 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:23.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:23.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4266753421' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: osdmap e538: 12 total, 12 up, 11 in 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:25:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:25:23.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4266753421' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: osdmap e538: 12 total, 12 up, 11 in 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:25:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:25:23.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:23.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:23.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4266753421' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-17T16:25:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: osdmap e538: 12 total, 12 up, 11 in 2024-09-17T16:25:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:25:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:25:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:25:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:25:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:25:23.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:25:23.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:23.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:23.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:23.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:23.456 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:23.538 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 17661 17645 78.7641 0.8125 0.00844491 0.0115751 2024-09-17T16:25:23.541 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:25:23.541 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:25:23.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:23.562 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:25:23.562 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:25:23.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:23.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:23.582 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:25:23.582 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:25:23.604 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:25:23.604 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:25:23.627 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:25:23.627 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:25:23.649 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:25:23.649 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:25:23.671 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:25:23.671 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:25:23.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:23.692 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:25:23.693 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:25:23.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:23.706 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:23.713 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:25:23.714 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:25:23.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:25:23.737 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:25:23.756 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:25:23.756 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:25:23.777 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:25:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:25:23.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:23.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:23.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:23.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:23.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:23.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:24.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:24.078 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:24.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:24.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:24.196 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-17T16:25:24.204 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:24.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:24.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:23 smithi071 ceph-mon[30002]: osdmap e539: 12 total, 12 up, 11 in 2024-09-17T16:25:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:25:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:25:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:25:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:25:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:25:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:25:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/451772764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:24.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:23 smithi071 ceph-mon[30002]: osdmap e540: 12 total, 12 up, 11 in 2024-09-17T16:25:24.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:24.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:23 smithi173 ceph-mon[30530]: osdmap e539: 12 total, 12 up, 11 in 2024-09-17T16:25:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:25:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:25:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:25:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:25:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:25:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:25:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:24.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/451772764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:24.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:23 smithi173 ceph-mon[30530]: osdmap e540: 12 total, 12 up, 11 in 2024-09-17T16:25:24.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:24.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:24.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:24.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:23 smithi017 ceph-mon[27571]: osdmap e539: 12 total, 12 up, 11 in 2024-09-17T16:25:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:25:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:25:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:25:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:25:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:25:24.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:25:24.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:25:24.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/451772764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:24.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:23 smithi017 ceph-mon[27571]: osdmap e540: 12 total, 12 up, 11 in 2024-09-17T16:25:24.538 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 17998 17982 74.9172 21.0625 0.00205346 0.0132715 2024-09-17T16:25:24.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:24.597 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:24.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:24.698 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:24.720 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:24.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:24.822 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:24.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:24.959 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:24.982 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:24.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:25.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:25.110 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:25.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:25.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:25.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:25 smithi017 ceph-mon[27571]: pgmap v1176: 17 pgs: 17 active+clean; 1012 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-17T16:25:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:25 smithi017 ceph-mon[27571]: osdmap e541: 12 total, 12 up, 11 in 2024-09-17T16:25:25.240 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:25.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:25.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:25 smithi071 ceph-mon[30002]: pgmap v1176: 17 pgs: 17 active+clean; 1012 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-17T16:25:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:25 smithi071 ceph-mon[30002]: osdmap e541: 12 total, 12 up, 11 in 2024-09-17T16:25:25.342 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:25.364 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:25.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:25.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:25 smithi173 ceph-mon[30530]: pgmap v1176: 17 pgs: 17 active+clean; 1012 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-17T16:25:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:25 smithi173 ceph-mon[30530]: osdmap e541: 12 total, 12 up, 11 in 2024-09-17T16:25:25.466 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:25.497 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:25.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:25.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 19216 19200 74.9923 76.125 0.00423899 0.0133194 2024-09-17T16:25:25.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:25.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:25.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:25.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:25.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:25.745 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:25.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:25.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:25.870 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:25.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:25.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:25.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:25.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:25.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:26.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:26.121 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:26.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:26.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:26.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:26.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:26.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:26.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:26 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:26 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:26.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:26.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:26.372 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:26.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:26 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:26.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:26 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:26.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:26 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:26 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:26.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:26.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:26.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 20462 20446 75.1613 77.875 0.00252713 0.0132703 2024-09-17T16:25:26.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:26.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:26.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:26.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:26.748 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:26.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:26.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:26.872 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:26.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:26.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:26.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:26.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:26.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:27.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:25:27.099 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:27.134 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:27.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:27.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:27.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:27.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:27 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:27 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:27 smithi071 ceph-mon[30002]: pgmap v1179: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 1.0 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 23 MiB/s wr, 366 op/s; 1080/51381 objects degraded (2.102%) 2024-09-17T16:25:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:27 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 1080/51381 objects degraded (2.102%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:25:27.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:27 smithi071 ceph-mon[30002]: osdmap e542: 12 total, 12 up, 11 in 2024-09-17T16:25:27.361 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:27.383 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:27.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:27.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:27 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:27 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:27 smithi173 ceph-mon[30530]: pgmap v1179: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 1.0 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 23 MiB/s wr, 366 op/s; 1080/51381 objects degraded (2.102%) 2024-09-17T16:25:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:27 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 1080/51381 objects degraded (2.102%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:25:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:27 smithi173 ceph-mon[30530]: osdmap e542: 12 total, 12 up, 11 in 2024-09-17T16:25:27.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:27.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:27 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:27 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:27 smithi017 ceph-mon[27571]: pgmap v1179: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 1.0 GiB data, 6.6 GiB used, 887 GiB / 894 GiB avail; 23 MiB/s wr, 366 op/s; 1080/51381 objects degraded (2.102%) 2024-09-17T16:25:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:27 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 1080/51381 objects degraded (2.102%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:25:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:27 smithi017 ceph-mon[27571]: osdmap e542: 12 total, 12 up, 11 in 2024-09-17T16:25:27.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:27.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:27.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:27.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 21352 21336 74.0754 55.625 0.0926524 0.0134139 2024-09-17T16:25:27.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:27.644 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:27.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:27.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:27.746 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:27.768 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:27.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:27.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:27.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:27.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:27.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:28.019 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:28.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:28.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:28.154 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:28.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:28.256 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:28.289 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:28.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:28.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:28.482 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:28.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:28.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 22602 22586 74.2882 78.125 0.00511143 0.0134497 2024-09-17T16:25:28.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:28.609 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:28.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:28.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:28.694 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:28.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:28.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:28.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:28.737 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:25:28.836 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:28.857 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:28.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:28.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:28.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:28.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:29.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:29.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:29.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:29.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:29.196 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:25:29.197 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:25:29.197 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-17T16:25:29.197 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T16:25:29.197 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T16:25:29.197 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-17T16:25:29.220 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:29.243 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:29.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:29.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:29 smithi071 ceph-mon[30002]: pgmap v1181: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.2 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 48 MiB/s wr, 760 op/s; 2177/58242 objects degraded (3.738%); 2609/58242 objects misplaced (4.480%); 8.4 MiB/s, 134 objects/s recovering 2024-09-17T16:25:29.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:29.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:29.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:29.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:29 smithi173 ceph-mon[30530]: pgmap v1181: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.2 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 48 MiB/s wr, 760 op/s; 2177/58242 objects degraded (3.738%); 2609/58242 objects misplaced (4.480%); 8.4 MiB/s, 134 objects/s recovering 2024-09-17T16:25:29.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:29.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:29 smithi017 ceph-mon[27571]: pgmap v1181: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.2 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 48 MiB/s wr, 760 op/s; 2177/58242 objects degraded (3.738%); 2609/58242 objects misplaced (4.480%); 8.4 MiB/s, 134 objects/s recovering 2024-09-17T16:25:29.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:29.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:29.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:25:29.538080+0000 min lat: 0.00108766 max lat: 2.21118 avg lat: 0.013441 2024-09-17T16:25:29.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:25:29.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 23788 23772 74.2797 74.125 0.00856502 0.013441 2024-09-17T16:25:29.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:29.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:29.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:29.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:29.623 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:29.636 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:29.653 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:29.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:29.743 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:29.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:29.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:29.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:29.872 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:29.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:29.995 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:29.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:30.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:30.119 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:30.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:30.221 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:30.254 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:30.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:30.356 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:30.391 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:30.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:30.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:30.530 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:25:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:30.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:30.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 24945 24929 74.1858 72.3125 0.0115837 0.0134665 2024-09-17T16:25:30.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:30 smithi017 ceph-mon[27571]: osdmap e543: 12 total, 12 up, 11 in 2024-09-17T16:25:30.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/867262942' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:25:30.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:30 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:25:30.551 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:30 smithi017 ceph-mon[27571]: pgmap v1183: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.3 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 51 MiB/s wr, 821 op/s; 2177/63183 objects degraded (3.446%); 2609/63183 objects misplaced (4.129%); 7.0 MiB/s, 111 objects/s recovering 2024-09-17T16:25:30.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:30.657 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:30.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:30.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:30.664 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:25:30.664 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:25:30.691 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:25:30.692 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:25:30.723 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:25:30.723 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:25:30.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:30.760 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:25:30.760 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:25:30.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:30.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:30.795 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:25:30.795 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:25:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:30 smithi071 ceph-mon[30002]: osdmap e543: 12 total, 12 up, 11 in 2024-09-17T16:25:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/867262942' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:25:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:30 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:25:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:30 smithi071 ceph-mon[30002]: pgmap v1183: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.3 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 51 MiB/s wr, 821 op/s; 2177/63183 objects degraded (3.446%); 2609/63183 objects misplaced (4.129%); 7.0 MiB/s, 111 objects/s recovering 2024-09-17T16:25:30.832 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:25:30.832 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:25:30.870 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:25:30.870 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:25:30.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:30.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:30 smithi173 ceph-mon[30530]: osdmap e543: 12 total, 12 up, 11 in 2024-09-17T16:25:30.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/867262942' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:25:30.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:30 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:25:30.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:30 smithi173 ceph-mon[30530]: pgmap v1183: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.3 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 51 MiB/s wr, 821 op/s; 2177/63183 objects degraded (3.446%); 2609/63183 objects misplaced (4.129%); 7.0 MiB/s, 111 objects/s recovering 2024-09-17T16:25:30.903 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:25:30.903 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T16:25:30.903 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:25:30.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:30.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:31.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:25:31.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:25:31.010 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:31.050 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:31.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:31.052 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:25:31.052 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:25:31.087 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:25:31.087 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:25:31.127 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:25:31.128 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:25:31.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:31.162 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:25:31.162 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:25:31.162 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:25:31.162 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:25:31.162 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:25:31.166 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:25:31.167 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:25:31.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:31.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:31.278 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:31.311 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:31.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:31.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:31.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:31.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:31.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:31.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 26247 26231 74.5122 81.375 0.00235141 0.0134103 2024-09-17T16:25:31.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:31.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:31.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:31.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:31.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:31.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:31 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:31 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:25:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:31 smithi017 ceph-mon[27571]: osdmap e544: 12 total, 12 up, 11 in 2024-09-17T16:25:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1883716773' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:31.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:31.814 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:31.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:31.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:31 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:31.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:31 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:25:31.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:31 smithi071 ceph-mon[30002]: osdmap e544: 12 total, 12 up, 11 in 2024-09-17T16:25:31.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1883716773' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:31 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:31 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:25:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:31 smithi173 ceph-mon[30530]: osdmap e544: 12 total, 12 up, 11 in 2024-09-17T16:25:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1883716773' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:31.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:31.939 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:31.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:32.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:32.074 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:32.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:32.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:32.201 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:32.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:32.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:32.303 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:32.341 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:32.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:32.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:32.467 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:32.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:32.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 27571 27555 74.8701 82.75 0.00469984 0.0133413 2024-09-17T16:25:32.569 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:32.599 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:32.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:32.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:32.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:32.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:32.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:32 smithi017 ceph-mon[27571]: pgmap v1185: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.5 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s; 2177/73968 objects degraded (2.943%); 930/73968 objects misplaced (1.257%); 24 MiB/s, 390 objects/s recovering 2024-09-17T16:25:32.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:32 smithi071 ceph-mon[30002]: pgmap v1185: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.5 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s; 2177/73968 objects degraded (2.943%); 930/73968 objects misplaced (1.257%); 24 MiB/s, 390 objects/s recovering 2024-09-17T16:25:32.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:32.852 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:32.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:32 smithi173 ceph-mon[30530]: pgmap v1185: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.5 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s; 2177/73968 objects degraded (2.943%); 930/73968 objects misplaced (1.257%); 24 MiB/s, 390 objects/s recovering 2024-09-17T16:25:32.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:32.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:32.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:33.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:33.099 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:33.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:33.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:33.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:33.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:33.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:33.348 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:33.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:33.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:33.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:33.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:33.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 27731 27715 72.1673 10 0.989452 0.0136075 2024-09-17T16:25:33.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:33.597 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:33.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:33.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:33.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:33.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:33.800 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:33 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:33.800 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:33 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:33.800 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:33 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2177/73968 objects degraded (2.943%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:25:33.800 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:33 smithi017 ceph-mon[27571]: osdmap e545: 12 total, 12 up, 11 in 2024-09-17T16:25:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:33 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:33 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:33 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2177/73968 objects degraded (2.943%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:25:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:33 smithi071 ceph-mon[30002]: osdmap e545: 12 total, 12 up, 11 in 2024-09-17T16:25:33.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:33.850 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:33.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:33 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:33 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:33 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2177/73968 objects degraded (2.943%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:25:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:33 smithi173 ceph-mon[30530]: osdmap e545: 12 total, 12 up, 11 in 2024-09-17T16:25:33.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:33.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:33.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:34.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:34.098 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:34.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:34.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:34.221 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:34.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:34.324 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:34.342 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:25:34.345 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:34.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:34.405 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:25:34.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:34.470 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:34.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:34.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 27840 27824 69.553 6.8125 0.00548471 0.0138418 2024-09-17T16:25:34.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:34.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:34.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:34.676 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:34.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:34 smithi017 ceph-mon[27571]: osdmap e546: 12 total, 12 up, 11 in 2024-09-17T16:25:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:34.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:34 smithi017 ceph-mon[27571]: pgmap v1188: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s; 2177/77109 objects degraded (2.823%); 930/77109 objects misplaced (1.206%); 24 MiB/s, 380 objects/s recovering 2024-09-17T16:25:34.743 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:34.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:34 smithi071 ceph-mon[30002]: osdmap e546: 12 total, 12 up, 11 in 2024-09-17T16:25:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:34 smithi071 ceph-mon[30002]: pgmap v1188: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s; 2177/77109 objects degraded (2.823%); 930/77109 objects misplaced (1.206%); 24 MiB/s, 380 objects/s recovering 2024-09-17T16:25:34.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:34.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:34.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:34.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:34.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:34 smithi173 ceph-mon[30530]: osdmap e546: 12 total, 12 up, 11 in 2024-09-17T16:25:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:34 smithi173 ceph-mon[30530]: pgmap v1188: 17 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s; 2177/77109 objects degraded (2.823%); 930/77109 objects misplaced (1.206%); 24 MiB/s, 380 objects/s recovering 2024-09-17T16:25:34.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:34.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:34.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:35.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:35.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:35.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:35.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:35.159 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:25:35.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:35.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:35.252 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:35.315 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:35.338 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:25:35.338 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:25:35.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:25:35.338 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124248 3.7 3:e0068a72:::benchmark_data_smithi173_66272_object52292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 526.44023607600002, 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46227925399999997, 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:35.339 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124248 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894741+0000", 2024-09-17T16:25:35.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894742+0000", 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.392e-06 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894747+0000", 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4979999999999997e-06 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894763+0000", 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5095e-05 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894769+0000", 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7360000000000004e-06 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894818+0000", 2024-09-17T16:25:35.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8551000000000003e-05 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.894928+0000", 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110536 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.895033+0000", 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104589 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.900743+0000", 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057098299999999999 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.901009+0000", 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000265927 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.927407+0000", 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.026398688 2024-09-17T16:25:35.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:48.927419+0000", 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1154e-05 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:49.356994+0000", 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.429575227 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:49.357020+0000", 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6030999999999999e-05 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124248 3.7 3:e0068a72:::benchmark_data_smithi173_66272_object52292:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 524.95845536700006, 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54499606199999995, 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.346 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124248 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376521+0000", 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376523+0000", 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.42e-06 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376528+0000", 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7200000000000003e-06 2024-09-17T16:25:35.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376539+0000", 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.022e-05 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376546+0000", 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8390000000000007e-06 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376598+0000", 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1953000000000002e-05 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.376611+0000", 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2320000000000001e-05 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.921322+0000", 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54471168999999997 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.921517+0000", 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019489999999999999 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_scan(digest 3.7 MIN-3:ec3efd19:::benchmark_data_smithi173_66272_object4297:head e 290/273)", 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 494.29481896700003, 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.427333666, 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.351 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040158+0000", 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040160+0000", 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0660000000000002e-06 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.040256+0000", 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5756999999999998e-05 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467140+0000", 2024-09-17T16:25:35.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:35.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42688425000000002 2024-09-17T16:25:35.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467149+0000", 2024-09-17T16:25:35.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2289999999999992e-06 2024-09-17T16:25:35.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467244+0000", 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709000000000004e-05 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467260+0000", 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6232e-05 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:21.467491+0000", 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000231423 2024-09-17T16:25:35.356 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDeleteReply(3.c e297,267 3:350956dd:::benchmark_data_smithi173_66272_object11629:head,259'5194)", 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 487.067962728, 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.990895003, 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "queued for pg", 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267014+0000", 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267015+0000", 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0500000000000003e-07 2024-09-17T16:25:35.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:28.267021+0000", 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2550000000000003e-06 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257793+0000", 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.99077237299999998 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257810+0000", 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7005999999999999e-05 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:29.257909+0000", 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8664000000000003e-05 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:102161 4.9 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-17T16:25:35.360 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 383.82633210500001, 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.902645841, 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 102161 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8700000000000005e-07 2024-09-17T16:25:35.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508680+0000", 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4382999999999999e-05 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508688+0000", 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5059999999999995e-06 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508699+0000", 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0781000000000001e-05 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508757+0000", 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8026999999999999e-05 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508920+0000", 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163308 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.509034+0000", 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113961 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.512221+0000", 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003186805 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.515116+0000", 2024-09-17T16:25:35.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002895299 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411212+0000", 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89609538399999999 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411258+0000", 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6217999999999997e-05 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411290+0000", 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2481999999999998e-05 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:184107 4.9 4:98a0eb83:::benchmark_data_smithi173_77474_object81932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:53.719117+0000", 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 281.61585999200003, 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.73240478200000003, 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.367 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 184107 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719117+0000", 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719117+0000", 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719120+0000", 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.88e-06 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719133+0000", 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.312e-05 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719166+0000", 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3772999999999997e-05 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719183+0000", 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6668999999999999e-05 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719313+0000", 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129789 2024-09-17T16:25:35.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719438+0000", 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012514700000000001 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.719532+0000", 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4146000000000003e-05 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723223+0000", 2024-09-17T16:25:35.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036906310000000002 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723276+0000", 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2994000000000001e-05 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723329+0000", 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2828000000000003e-05 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:53.723485+0000", 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015637499999999999 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:54.451313+0000", 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.72782798500000001 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:54.451383+0000", 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0465999999999999e-05 2024-09-17T16:25:35.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:54.451521+0000", 2024-09-17T16:25:35.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013797899999999999 2024-09-17T16:25:35.374 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.374 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.374 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:185164 4.f 4:f8df73f0:::benchmark_data_smithi173_77474_object82989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:56.000100+0000", 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 279.33487644899998, 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.634535988, 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 185164 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000100+0000", 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000100+0000", 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000102+0000", 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.404e-06 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000109+0000", 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9619999999999996e-06 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000119+0000", 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0558e-05 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000124+0000", 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2170000000000002e-06 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000166+0000", 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2134e-05 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000270+0000", 2024-09-17T16:25:35.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104005 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,9,11", 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.000379+0000", 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010812 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.004004+0000", 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036249329999999999 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.004131+0000", 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000127336 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.020238+0000", 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016107305999999998 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.634511+0000", 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61427235199999997 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.634550+0000", 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9057000000000002e-05 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:56.634636+0000", 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6604e-05 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.381 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:186293 4.f 4:ffe1d3e0:::benchmark_data_smithi173_77474_object84118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:58.185546+0000", 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 277.14943053600001, 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42855515999999999, 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186293 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185546+0000", 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185546+0000", 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185547+0000", 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.359e-06 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185553+0000", 2024-09-17T16:25:35.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4349999999999996e-06 2024-09-17T16:25:35.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185565+0000", 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2155e-05 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185571+0000", 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6509999999999998e-06 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185619+0000", 2024-09-17T16:25:35.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8696999999999998e-05 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185723+0000", 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010333 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,9,11", 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.185816+0000", 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3335e-05 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.189433+0000", 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036167840000000001 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.190346+0000", 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091320500000000005 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.190429+0000", 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3014000000000002e-05 2024-09-17T16:25:35.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.613994+0000", 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42356446800000003 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.614037+0000", 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3027e-05 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:58.614101+0000", 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4700000000000001e-05 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204335 4.9 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 251.72828536, 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77672725300000001, 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.390 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204335 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606692+0000", 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2600000000000002e-07 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606696+0000", 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7170000000000002e-06 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606707+0000", 2024-09-17T16:25:35.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1486e-05 2024-09-17T16:25:35.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606712+0000", 2024-09-17T16:25:35.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1560000000000003e-06 2024-09-17T16:25:35.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606769+0000", 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6836000000000003e-05 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606856+0000", 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6749999999999994e-05 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-17T16:25:35.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606942+0000", 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6001999999999994e-05 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611141+0000", 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041987810000000004 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611206+0000", 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5581999999999997e-05 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611481+0000", 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027492400000000001 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611502+0000", 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0971999999999998e-05 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383271+0000", 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77176877600000005 2024-09-17T16:25:35.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383334+0000", 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2928000000000005e-05 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383419+0000", 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4617000000000007e-05 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.397 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204343 4.0 4:0b58e0a7:::benchmark_data_smithi173_77474_object102168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.621137+0000", 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 251.713839218, 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49750966600000002, 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:35.401 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204343 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621137+0000", 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621137+0000", 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621139+0000", 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.635e-06 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621144+0000", 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3170000000000001e-06 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621157+0000", 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2123999999999999e-05 2024-09-17T16:25:35.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621164+0000", 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1679999999999998e-06 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621222+0000", 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8439000000000001e-05 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621307+0000", 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4940999999999999e-05 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.621418+0000", 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110548 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626056+0000", 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046379489999999997 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095492+0000", 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46943599899999999 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118566+0000", 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023074935000000001 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118609+0000", 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2106e-05 2024-09-17T16:25:35.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118647+0000", 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8504999999999998e-05 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:204343 4.0 e408/389 ondisk, result = 0)", 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.625102+0000", 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 251.709874797, 2024-09-17T16:25:35.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.493551564, 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625102+0000", 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625102+0000", 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625103+0000", 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0100000000000004e-07 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625111+0000", 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9659999999999996e-06 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118509+0000", 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49339840800000001 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118514+0000", 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5589999999999996e-06 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118557+0000", 2024-09-17T16:25:35.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2222000000000003e-05 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118563+0000", 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2689999999999996e-06 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.118653+0000", 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439000000000006e-05 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.411 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25516.0:204346 4.4 e408/306 4:22db2208:::benchmark_data_smithi173_77474_object102171:head v 408'12784, mlcod=408'12781)", 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.625376+0000", 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 251.70960043299999, 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.67400101199999995, 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.412 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625376+0000", 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625376+0000", 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625377+0000", 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2699999999999998e-07 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.625386+0000", 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3260000000000002e-06 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295220+0000", 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.66983389199999999 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295230+0000", 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3090000000000001e-06 2024-09-17T16:25:35.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295298+0000", 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8583000000000004e-05 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.295337+0000", 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8992000000000002e-05 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299315+0000", 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039779949999999998 2024-09-17T16:25:35.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.299377+0000", 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2088000000000006e-05 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25516.0:204343 4.0 e408/389 ondisk, result = 0)", 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.626258+0000", 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 251.70871878599999, 2024-09-17T16:25:35.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46924642799999999, 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626258+0000", 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626258+0000", 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626259+0000", 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0100000000000004e-07 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.626262+0000", 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5300000000000001e-06 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095382+0000", 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46911975700000003 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095391+0000", 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4010000000000003e-06 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095466+0000", 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5087999999999997e-05 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.419 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095481+0000", 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4252e-05 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.095504+0000", 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3699000000000001e-05 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.420 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(5.16 470/455 [PushReplyOp(5:6e1c4d97:::benchmark_data_smithi173_90735_object50923:head)])", 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:10.891412+0000", 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 144.44356495599999, 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.40973801100000001, 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:10.891412+0000", 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:10.891412+0000", 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:10.891412+0000", 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3900000000000004e-07 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:10.891417+0000", 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3590000000000001e-06 2024-09-17T16:25:35.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:11.300664+0000", 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.40924686399999999 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:11.300669+0000", 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.215e-06 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:11.300732+0000", 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3565000000000006e-05 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:11.301150+0000", 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00041736900000000001 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17629 6.8 6:196f18af:::benchmark_data_smithi173_102017_object17628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:25:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:21.997934+0000", 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.337042535, 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56117388700000004, 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17629 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997934+0000", 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997934+0000", 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997936+0000", 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3859999999999999e-06 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998411+0000", 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000475902 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998433+0000", 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.141e-05 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998452+0000", 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9619e-05 2024-09-17T16:25:35.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998531+0000", 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8924000000000005e-05 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998705+0000", 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017413300000000001 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:25:35.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998883+0000", 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017767 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000058+0000", 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001174701 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000188+0000", 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013036300000000001 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.558942+0000", 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.55875353999999999 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.559036+0000", 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4048999999999993e-05 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.559108+0000", 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2189999999999998e-05 2024-09-17T16:25:35.430 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26665.0:17629 6.8 e538/527 ondisk, result = 0)", 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.000308+0000", 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.334668442, 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.55881296400000002, 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.431 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000308+0000", 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000308+0000", 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000309+0000", 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.066e-06 2024-09-17T16:25:35.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:35.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:35.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:35.434 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000344+0000", 2024-09-17T16:25:35.434 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4366999999999998e-05 2024-09-17T16:25:35.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.558795+0000", 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.55845097700000002 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.558808+0000", 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3295999999999999e-05 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.558911+0000", 2024-09-17T16:25:35.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103183 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.558932+0000", 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0743000000000001e-05 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.559121+0000", 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018933200000000001 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.436 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17641 6.7 6.aff5e7f7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.213621506000001, 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86676560000000002, 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:35.437 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17641 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121357+0000", 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4020000000000001e-06 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121396+0000", 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9561999999999998e-05 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121410+0000", 2024-09-17T16:25:35.439 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3645000000000001e-05 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121415+0000", 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6169999999999998e-06 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121465+0000", 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9648000000000003e-05 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121474+0000", 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.568e-06 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988093+0000", 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86661935700000003 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988121+0000", 2024-09-17T16:25:35.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7801e-05 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17642 6.7 6.5a462497 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.140835876000001, 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79401586899999999, 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:25:35.442 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17642 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.443 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194142+0000", 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2399999999999997e-07 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194163+0000", 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1841e-05 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194175+0000", 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1487e-05 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194180+0000", 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2639999999999999e-06 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194227+0000", 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6675000000000003e-05 2024-09-17T16:25:35.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194233+0000", 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1229999999999997e-06 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988148+0000", 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79391546800000001 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988157+0000", 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1869999999999997e-06 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17647 6.7 6.ae4066d7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.502124+0000", 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 12.83285236, 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48608792200000001, 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:25:35.447 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17647 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.502124+0000", 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.502124+0000", 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.502125+0000", 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8800000000000001e-07 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.502165+0000", 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9551999999999997e-05 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.502177+0000", 2024-09-17T16:25:35.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1877e-05 2024-09-17T16:25:35.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.502182+0000", 2024-09-17T16:25:35.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2320000000000001e-06 2024-09-17T16:25:35.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.502244+0000", 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1768999999999998e-05 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.502251+0000", 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.002e-06 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988200+0000", 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48594971100000001 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988212+0000", 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1891e-05 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17652 6.7 6.d517fea7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:25:35.452 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.565366+0000", 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 12.769610475, 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42288178599999998, 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17652 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.565366+0000", 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.565366+0000", 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.565367+0000", 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0299999999999997e-07 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.565407+0000", 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0098999999999997e-05 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.565419+0000", 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1697e-05 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.565424+0000", 2024-09-17T16:25:35.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.994e-06 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.565468+0000", 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4286000000000003e-05 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.565478+0000", 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6809999999999993e-06 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:25:35.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988240+0000", 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.422762366 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988248+0000", 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7600000000000002e-06 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:25:35.457 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:25:35.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:35.539 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 29341 29325 70.4858 93.8125 0.0157424 0.0141781 2024-09-17T16:25:35.559 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:35.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:35.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:35.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:35.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:35.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:35 smithi017 ceph-mon[27571]: osdmap e547: 12 total, 12 up, 11 in 2024-09-17T16:25:35.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:35.806 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:35.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:35 smithi071 ceph-mon[30002]: osdmap e547: 12 total, 12 up, 11 in 2024-09-17T16:25:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:35 smithi173 ceph-mon[30530]: osdmap e547: 12 total, 12 up, 11 in 2024-09-17T16:25:35.907 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:35.934 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:35.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:36.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:36.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:36.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:36.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:36.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:36.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:36.287 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:36.310 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:36.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:36.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:36.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:36.448 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:36.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:36.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 30494 30478 70.544 72.0625 0.006283 0.0141697 2024-09-17T16:25:36.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:36.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:36.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:36.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:36.704 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:36.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:36.775 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:25:36.802 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:36 smithi071 ceph-mon[30002]: pgmap v1190: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s; 1097/80184 objects degraded (1.368%); 1554/80184 objects misplaced (1.938%); 25 MiB/s, 395 objects/s recovering 2024-09-17T16:25:36.806 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:36.832 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:36.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:36 smithi173 ceph-mon[30530]: pgmap v1190: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s; 1097/80184 objects degraded (1.368%); 1554/80184 objects misplaced (1.938%); 25 MiB/s, 395 objects/s recovering 2024-09-17T16:25:36.913 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:36 smithi017 ceph-mon[27571]: pgmap v1190: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 1.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s; 1097/80184 objects degraded (1.368%); 1554/80184 objects misplaced (1.938%); 25 MiB/s, 395 objects/s recovering 2024-09-17T16:25:36.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:36.951 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:36.961 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:36.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:37.063 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:37.097 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:37.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:37.164 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:25:37.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:37.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:37.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:37.369 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:37.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:37.471 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:37.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:37.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:37.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 31808 31792 70.9574 82.125 0.00639192 0.0140855 2024-09-17T16:25:37.545 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:25:37.554 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:25:37.554 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:25:37.576 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:25:37.576 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:25:37.597 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:25:37.598 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:25:37.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:37.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:37.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:37.622 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:37.625 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:25:37.625 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:25:37.649 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:25:37.650 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:25:37.681 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:25:37.682 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:25:37.712 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:25:37.712 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:25:37.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:37.740 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:25:37.741 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:25:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:37.745 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:37.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:37.769 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:25:37.769 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:25:37.769 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1548314026' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:37.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:37 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:37.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2273471668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:37.798 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:25:37.798 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:25:37.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1548314026' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:37 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2273471668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:37.825 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:25:37.825 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:25:37.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:37.854 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:25:37.854 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:25:37.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:37.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1548314026' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:37 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2273471668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:37.972 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:37.998 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:37.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:38.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:38.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:38.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:38.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:38.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:38.249 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:38.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:38.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:38.383 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:38.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:38.423 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:25:38.423 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:25:38.486 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:38.509 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:38.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:38.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 32653 32637 70.3315 52.8125 0.00399697 0.0142052 2024-09-17T16:25:38.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:38.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:38.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:38.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:38 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:38 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:38 smithi017 ceph-mon[27571]: osdmap e548: 12 total, 12 up, 11 in 2024-09-17T16:25:38.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:38 smithi017 ceph-mon[27571]: pgmap v1192: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 1.8 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 52 MiB/s wr, 825 op/s; 1265/88443 objects misplaced (1.430%); 21 MiB/s, 340 objects/s recovering 2024-09-17T16:25:38.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:38.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:38 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:38 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:38 smithi071 ceph-mon[30002]: osdmap e548: 12 total, 12 up, 11 in 2024-09-17T16:25:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:38 smithi071 ceph-mon[30002]: pgmap v1192: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 1.8 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 52 MiB/s wr, 825 op/s; 1265/88443 objects misplaced (1.430%); 21 MiB/s, 340 objects/s recovering 2024-09-17T16:25:38.834 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:25:38.835 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:25:38.835 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-17T16:25:38.835 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-17T16:25:38.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:38 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:38 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:38 smithi173 ceph-mon[30530]: osdmap e548: 12 total, 12 up, 11 in 2024-09-17T16:25:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:38 smithi173 ceph-mon[30530]: pgmap v1192: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 1.8 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 52 MiB/s wr, 825 op/s; 1265/88443 objects misplaced (1.430%); 21 MiB/s, 340 objects/s recovering 2024-09-17T16:25:38.917 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:38.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:39.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:39.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:39.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:39.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:39.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:39.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:39.272 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:39.299 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:39.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:39.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:39.429 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:39.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:39.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:39.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 33915 33899 70.616 78.875 0.037578 0.0141399 2024-09-17T16:25:39.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:39.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:39.654 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:39.688 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:39.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:39.790 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:39 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1097/80184 objects degraded (1.368%), 1 pg degraded) 2024-09-17T16:25:39.818 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:39.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:39.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:39 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1097/80184 objects degraded (1.368%), 1 pg degraded) 2024-09-17T16:25:39.919 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:39.946 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:39.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:39 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1097/80184 objects degraded (1.368%), 1 pg degraded) 2024-09-17T16:25:40.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:40.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:40.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:40.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:40.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:40.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:40.302 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:40.326 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:40.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:40.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:25:40.420 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:25:40.429 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:40.468 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:40.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:40.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 35274 35258 71.0778 84.9375 0.0995969 0.0140555 2024-09-17T16:25:40.569 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:40.593 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:40.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:40.683 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:25:40.695 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:40.718 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:40.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:40.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:40 smithi017 ceph-mon[27571]: pgmap v1193: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 1.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 54 MiB/s wr, 864 op/s; 666/93708 objects misplaced (0.711%); 24 MiB/s, 381 objects/s recovering 2024-09-17T16:25:40.805 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:40 smithi071 ceph-mon[30002]: pgmap v1193: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 1.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 54 MiB/s wr, 864 op/s; 666/93708 objects misplaced (0.711%); 24 MiB/s, 381 objects/s recovering 2024-09-17T16:25:40.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:40.842 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:40.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:40 smithi173 ceph-mon[30530]: pgmap v1193: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 1.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 54 MiB/s wr, 864 op/s; 666/93708 objects misplaced (0.711%); 24 MiB/s, 381 objects/s recovering 2024-09-17T16:25:40.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:40.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:40.967 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:40.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:41.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:41.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:41.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:41.174 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:25:41.174 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:25:41.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:41.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:41.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:41.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:41.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:41.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:41.439 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:41.460 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:41.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:41.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 35540 35524 69.3761 16.625 0.266733 0.0141285 2024-09-17T16:25:41.563 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:41.596 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:41.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:41.699 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:41.722 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:41.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:41.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:41.810 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:25:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:41 smithi071 ceph-mon[30002]: osdmap e549: 12 total, 12 up, 11 in 2024-09-17T16:25:41.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:41.845 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:41.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:41.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:41 smithi173 ceph-mon[30530]: osdmap e549: 12 total, 12 up, 11 in 2024-09-17T16:25:41.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:41.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:41.968 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:41.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:41.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:41 smithi017 ceph-mon[27571]: osdmap e549: 12 total, 12 up, 11 in 2024-09-17T16:25:42.049 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:42.065 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:42.069 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:42.069 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:25:42.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:42.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:42.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:42.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:42.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:42.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:42.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:42.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:42.338 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:42.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:42.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:42.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:42.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 35831 35815 67.825 18.1875 0.00210168 0.0144354 2024-09-17T16:25:42.563 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:42.570 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:42 smithi017 ceph-mon[27571]: osdmap e550: 12 total, 12 up, 11 in 2024-09-17T16:25:42.570 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:42 smithi017 ceph-mon[27571]: pgmap v1196: 17 pgs: 1 creating+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s; 25/104682 objects misplaced (0.024%); 27 MiB/s, 437 objects/s recovering 2024-09-17T16:25:42.578 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:25:42.578 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:25:42.586 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:42.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:42.688 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:42.726 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:42.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:42 smithi071 ceph-mon[30002]: osdmap e550: 12 total, 12 up, 11 in 2024-09-17T16:25:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:42 smithi071 ceph-mon[30002]: pgmap v1196: 17 pgs: 1 creating+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s; 25/104682 objects misplaced (0.024%); 27 MiB/s, 437 objects/s recovering 2024-09-17T16:25:42.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:42.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:42.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:42 smithi173 ceph-mon[30530]: osdmap e550: 12 total, 12 up, 11 in 2024-09-17T16:25:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:42 smithi173 ceph-mon[30530]: pgmap v1196: 17 pgs: 1 creating+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s; 25/104682 objects misplaced (0.024%); 27 MiB/s, 437 objects/s recovering 2024-09-17T16:25:42.954 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:42.975 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:42.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:43.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:43.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:43.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:43.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:43.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:43.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:43.307 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:25:43.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:43.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:43.355 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:43.456 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:43.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:43.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:43.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 37353 37337 68.6277 95.125 0.00661619 0.0145652 2024-09-17T16:25:43.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:43.621 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:43.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:43.623 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:25:43.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:43.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-17T16:25:43.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:43.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:43.849 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:43.872 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:43.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:43.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:43.877 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:43 smithi071 ceph-mon[30002]: osdmap e551: 12 total, 12 up, 11 in 2024-09-17T16:25:43.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:43 smithi173 ceph-mon[30530]: osdmap e551: 12 total, 12 up, 11 in 2024-09-17T16:25:43.896 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:43 smithi017 ceph-mon[27571]: osdmap e551: 12 total, 12 up, 11 in 2024-09-17T16:25:43.975 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:43.998 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:43.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:44.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:25:44.100 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:44.104 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:44.121 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:44.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:44.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:44.238 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:25:44.238 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:25:44.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:44.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:44.348 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:44.379 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:44.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:44.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:44.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:44.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:44.513 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:25:44.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 38640 38624 68.9649 80.4375 0.0187062 0.0144925 2024-09-17T16:25:44.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:44.637 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:44.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:44.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:44.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:44.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:44.807 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:44 smithi017 ceph-mon[27571]: pgmap v1198: 17 pgs: 1 creating+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 62 MiB/s wr, 991 op/s; 25/106302 objects misplaced (0.024%); 13 MiB/s, 206 objects/s recovering 2024-09-17T16:25:44.807 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4019034733' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:25:44.807 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:25:44.807 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:44 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:44.807 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:25:44.807 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:44 smithi017 ceph-mon[27571]: osdmap e552: 12 total, 12 up, 11 in 2024-09-17T16:25:44.866 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:44.891 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:44.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:44.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:44 smithi173 ceph-mon[30530]: pgmap v1198: 17 pgs: 1 creating+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 62 MiB/s wr, 991 op/s; 25/106302 objects misplaced (0.024%); 13 MiB/s, 206 objects/s recovering 2024-09-17T16:25:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4019034733' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:25:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:25:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:44 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:25:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:44 smithi173 ceph-mon[30530]: osdmap e552: 12 total, 12 up, 11 in 2024-09-17T16:25:44.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:45.013 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:45.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:45.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:44 smithi071 ceph-mon[30002]: pgmap v1198: 17 pgs: 1 creating+peering, 1 active+recovering+undersized+remapped, 15 active+clean; 2.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 62 MiB/s wr, 991 op/s; 25/106302 objects misplaced (0.024%); 13 MiB/s, 206 objects/s recovering 2024-09-17T16:25:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4019034733' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:25:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:25:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:44 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:25:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:25:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:44 smithi071 ceph-mon[30002]: osdmap e552: 12 total, 12 up, 11 in 2024-09-17T16:25:45.115 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:45.141 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:45.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:45.243 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:45.269 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:45.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:45.371 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:45.402 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:45.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:45.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:45.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:45.539 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:45.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 40021 40005 69.4466 86.3125 0.0056373 0.0143937 2024-09-17T16:25:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:45.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:45.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:45.664 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:45.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:45.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:25:45.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:45.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:45.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:45.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:45.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:45.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:46.016 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:46.029 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:25:46.042 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:46.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:46.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:46.166 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:46.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:46.267 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:46.290 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:46.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:46.392 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:46.416 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:46.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:46.518 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:46.537 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:25:46.537 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:25:46.540 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 15 40948 40933 69.1371 58 0.0069441 0.0144516 2024-09-17T16:25:46.542 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:46.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:46.644 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:46.666 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:46.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:46.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:46.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:46.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:46.794 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:46.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:46.895 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:46.924 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:46.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:47.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:47.052 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:47.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:47.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:47.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:47.163 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:46 smithi071 ceph-mon[30002]: pgmap v1200: 17 pgs: 1 creating+peering, 16 active+clean; 2.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s; 7.8 MiB/s, 124 objects/s recovering 2024-09-17T16:25:47.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:47.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:47.201 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:25:47.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:46 smithi017 ceph-mon[27571]: pgmap v1200: 17 pgs: 1 creating+peering, 16 active+clean; 2.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s; 7.8 MiB/s, 124 objects/s recovering 2024-09-17T16:25:47.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:47.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:47.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:46 smithi173 ceph-mon[30530]: pgmap v1200: 17 pgs: 1 creating+peering, 16 active+clean; 2.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s; 7.8 MiB/s, 124 objects/s recovering 2024-09-17T16:25:47.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:47.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:47.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:47.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:47.506 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:25:47.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:47.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 42194 42178 69.3652 77.8125 0.00600293 0.0144088 2024-09-17T16:25:47.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:47.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:47.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:47.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:47.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:47.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:47.821 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:47.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:47.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:47.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:47.951 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:47.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:47.973 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:47.995 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-17T16:25:48.042 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:25:48.043 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:25:48.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:48.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:48.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:48.178 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:48.201 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:48.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:48.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:48.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:48.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:48.427 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:48.452 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:48.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:48.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 43660 43644 69.9358 91.625 0.00493125 0.0142917 2024-09-17T16:25:48.553 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:48.586 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:48.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:48.686 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:48.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:48.713 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:48.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:48.772 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-17T16:25:48.769+0000 7ff60759e640 0 --2- 172.21.15.17:0/4009519841 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7ff60810b320 0x7ff608111b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T16:25:48.814 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:48.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:25:48.822 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:25:48.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:25:48.841 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:48.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:48.849 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:25:48.849 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:25:48.872 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:25:48.909 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:25:48.909 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:25:48.909 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:25:48.916 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:25:48.916 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:25:48.938 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:25:48.938 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:25:48.942 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:48.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:25:48.960 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:25:48.964 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:48.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:48.984 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:25:48.985 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:25:49.012 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:25:49.012 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:25:49.040 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:25:49.040 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:25:49.066 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:49.079 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:25:49.079 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:25:49.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:48 smithi017 ceph-mon[27571]: pgmap v1201: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 55 MiB/s wr, 883 op/s; 252 KiB/s, 3 objects/s recovering 2024-09-17T16:25:49.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/36391565' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:49.093 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:25:49.101 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:49.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:49.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:49.116 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:25:49.116 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:25:49.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:49.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:49.229 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:49.229 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:49.242 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:48 smithi071 ceph-mon[30002]: pgmap v1201: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 55 MiB/s wr, 883 op/s; 252 KiB/s, 3 objects/s recovering 2024-09-17T16:25:49.243 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/36391565' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:49.330 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:49.352 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:49.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:48 smithi173 ceph-mon[30530]: pgmap v1201: 17 pgs: 17 active+clean; 2.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 55 MiB/s wr, 883 op/s; 252 KiB/s, 3 objects/s recovering 2024-09-17T16:25:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/36391565' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:49.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:49.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:49.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:49.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:49.498 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:25:49.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:25:49.539724+0000 min lat: 0.00108766 max lat: 2.21118 avg lat: 0.01431 2024-09-17T16:25:49.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:25:49.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 44720 44704 69.8435 66.25 0.0231644 0.01431 2024-09-17T16:25:49.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:49.610 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:49.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:49.633 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:25:49.633 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:25:49.633 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.633 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:44804 6.a 6:5a328864:::benchmark_data_smithi173_102017_object44803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-09-17T16:25:49.633 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:25:49.623221+0000", 2024-09-17T16:25:49.633 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0075918289999999996, 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0076123650000000003, 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 44804 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:49.634 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:49.623221+0000", 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:49.623221+0000", 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:49.623222+0000", 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.75e-07 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:49.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:49.623306+0000", 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3943000000000006e-05 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:49.623316+0000", 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0375e-05 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:49.623321+0000", 2024-09-17T16:25:49.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0209999999999999e-06 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:49.623363+0000", 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1826000000000002e-05 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:49.623458+0000", 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4489999999999998e-05 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:49.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.638 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:25:49.638 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:49.623543+0000", 2024-09-17T16:25:49.638 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5006000000000006e-05 2024-09-17T16:25:49.638 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:49.638 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:49.638 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:25:49.638 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:49.624501+0000", 2024-09-17T16:25:49.638 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000958656 2024-09-17T16:25:49.639 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:49.639 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:49.639 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:49.639 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:49.639 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:25:49.639 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 1 2024-09-17T16:25:49.639 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:25:49.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:49.735 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:49.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:49.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:49.871 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:49.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:49.973 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:49.995 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:49.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:50.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:50.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:50.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:50.225 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:50.254 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:50.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:50.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:50.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:50.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:50.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:50.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:50.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:50.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:50.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 45512 45496 69.3472 49.5 0.0051934 0.0144146 2024-09-17T16:25:50.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:50.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:50.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:50.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:50.758 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:50.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:50.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:50 smithi071 ceph-mon[30002]: pgmap v1202: 17 pgs: 17 active+clean; 2.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 54 MiB/s wr, 867 op/s; 200 KiB/s, 3 objects/s recovering 2024-09-17T16:25:50.860 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:50.881 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:50.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:50.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:50 smithi173 ceph-mon[30530]: pgmap v1202: 17 pgs: 17 active+clean; 2.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 54 MiB/s wr, 867 op/s; 200 KiB/s, 3 objects/s recovering 2024-09-17T16:25:50.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:25:50.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:50 smithi017 ceph-mon[27571]: pgmap v1202: 17 pgs: 17 active+clean; 2.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 54 MiB/s wr, 867 op/s; 200 KiB/s, 3 objects/s recovering 2024-09-17T16:25:51.019 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:51.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:51.121 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:51.145 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:51.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:51.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:51.259 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:25:51.276 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:51.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:51.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:51.401 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:51.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:51.503 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:51.537 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:51.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:51.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 46702 46686 69.4668 74.375 0.0207833 0.0143897 2024-09-17T16:25:51.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:51.672 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:51.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:51.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:51.775 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:51.809 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:25:51.809 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:25:51.809 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:25:51.809 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:25:51.809 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:25:51.810 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:51.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:51.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:51.936 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:51.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:52.038 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:52.067 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:52.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:52.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:52.169 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:52.192 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:52.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:52.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:52.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:52.321 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:52.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:52.414 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-17T16:25:52.422 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:52.441 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:52.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:25:52.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:52.484 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:52.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:52.522 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:52 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:52.522 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:52 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:52.522 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:52 smithi017 ceph-mon[27571]: osdmap e553: 12 total, 12 up, 11 in 2024-09-17T16:25:52.522 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:25:52.522 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:25:52.522 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:25:52.522 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:25:52.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 47852 47836 69.5226 71.875 0.00841291 0.0143758 2024-09-17T16:25:52.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:52.611 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:52.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:52.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:52.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:52.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:52.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:52.776 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:25:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:52 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:52 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:52 smithi071 ceph-mon[30002]: osdmap e553: 12 total, 12 up, 11 in 2024-09-17T16:25:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:25:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:25:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:25:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:25:52.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:52.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:52.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:52 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:52 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:52 smithi173 ceph-mon[30530]: osdmap e553: 12 total, 12 up, 11 in 2024-09-17T16:25:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:25:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:25:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:25:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:25:52.966 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:52.990 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:52.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:52.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:53.075 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:53.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:53.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:53.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:53.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:53.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:53.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:53.338 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:25:53.338 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:25:53.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:25:53.338 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:25:53.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.338 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:76068 4.5 4:a01bffc8:::benchmark_data_smithi173_77474_object76067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:18:40.015930+0000", 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 433.31868720799997, 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39555464400000001, 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 76068 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015930+0000", 2024-09-17T16:25:53.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015930+0000", 2024-09-17T16:25:53.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015931+0000", 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.44e-06 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015982+0000", 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0076e-05 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.015997+0000", 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5531000000000001e-05 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016004+0000", 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6930000000000001e-06 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016063+0000", 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9621999999999999e-05 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016194+0000", 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013074899999999999 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.016340+0000", 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000145601 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.018468+0000", 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021282110000000001 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.018797+0000", 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032921399999999998 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.411385+0000", 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39258796099999999 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.411442+0000", 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6997000000000003e-05 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:18:40.411485+0000", 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2549000000000002e-05 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.345 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:102166 4.a 4:583138b7:::benchmark_data_smithi173_77474_object102165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:11.515734+0000", 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 401.81888331300001, 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31708639999999999, 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 102166 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.346 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515734+0000", 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515734+0000", 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515736+0000", 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.903e-06 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515777+0000", 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1304000000000003e-05 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515791+0000", 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3648e-05 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515798+0000", 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4139999999999997e-06 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515850+0000", 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2157000000000002e-05 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.515951+0000", 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100321 2024-09-17T16:25:53.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.516061+0000", 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000110636 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.518923+0000", 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028621470000000002 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519104+0000", 2024-09-17T16:25:53.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018084500000000001 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832702+0000", 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31359804299999999 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832773+0000", 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0612999999999996e-05 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832820+0000", 2024-09-17T16:25:53.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7369e-05 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:102166 4.a e349/306 ondisk, result = 0)", 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:11.519407+0000", 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 401.81520977600002, 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31341929800000001, 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519407+0000", 2024-09-17T16:25:53.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:53.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519407+0000", 2024-09-17T16:25:53.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519408+0000", 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5300000000000003e-07 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.519417+0000", 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7660000000000004e-06 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832605+0000", 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31318752500000002 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832629+0000", 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4039000000000001e-05 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832686+0000", 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7722000000000001e-05 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832696+0000", 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3610000000000007e-06 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:11.832827+0000", 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000131132 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:144445 4.5 4:a72f5a89:::benchmark_data_smithi173_77474_object42270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:59.250222+0000", 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 354.08439489699998, 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32863163499999998, 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:25:53.358 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 144445 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250222+0000", 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250222+0000", 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250224+0000", 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2240000000000001e-06 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250229+0000", 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4720000000000003e-06 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250242+0000", 2024-09-17T16:25:53.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2508e-05 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250251+0000", 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0049999999999993e-06 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250311+0000", 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0441e-05 2024-09-17T16:25:53.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250442+0000", 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013124299999999999 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.250564+0000", 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012152200000000001 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254897+0000", 2024-09-17T16:25:53.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0043328849999999999 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.259207+0000", 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0043099599999999998 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578783+0000", 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31957687099999998 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.363 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578813+0000", 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9745000000000001e-05 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578854+0000", 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0759000000000001e-05 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.364 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.25516.0:144445 4.5 e363/301 ondisk, result = 0)", 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:19:59.254966+0000", 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 354.07965125200002, 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32389465200000001, 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254966+0000", 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254966+0000", 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254967+0000", 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6899999999999996e-07 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.254972+0000", 2024-09-17T16:25:53.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9259999999999999e-06 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578690+0000", 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32371775800000002 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578694+0000", 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3189999999999996e-06 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578774+0000", 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0020999999999998e-05 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578779+0000", 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0669999999999999e-06 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:19:59.578861+0000", 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1692000000000003e-05 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.368 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:184107 4.9 e389/383 4:98a0eb83:::benchmark_data_smithi173_77474_object81932:head v 389'11321, mlcod=389'11318)", 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:53.719658+0000", 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 299.61495934700002, 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.73134149999999998, 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.719658+0000", 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.719658+0000", 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.719659+0000", 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8400000000000004e-07 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.719663+0000", 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.831e-06 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.446826+0000", 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.72716291899999996 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.446836+0000", 2024-09-17T16:25:53.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3319999999999998e-06 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.446925+0000", 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9407000000000006e-05 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.446953+0000", 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8351e-05 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.450963+0000", 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040093389999999998 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.450999+0000", 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6637000000000002e-05 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.373 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:197771 4.1 e401/301 4:8a203159:::benchmark_data_smithi173_77474_object95596:head v 401'12502, mlcod=401'12499)", 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:13.590532+0000", 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 279.74408532000001, 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.325052122, 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590532+0000", 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590532+0000", 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590533+0000", 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5700000000000002e-07 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590540+0000", 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.939e-06 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.911473+0000", 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32093297300000001 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.911477+0000", 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6759999999999998e-06 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.911514+0000", 2024-09-17T16:25:53.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7191999999999999e-05 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.911533+0000", 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8754999999999999e-05 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.915546+0000", 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040127770000000004 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.915584+0000", 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8053000000000003e-05 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(4.8 401/383 [PushReplyOp(4:16ec698f:::benchmark_data_smithi173_77474_object98014:head)])", 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:14.329740+0000", 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 279.00487692000002, 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31203499200000001, 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.378 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.329740+0000", 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.329740+0000", 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.329740+0000", 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.31e-07 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.329744+0000", 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0979999999999998e-06 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.640369+0000", 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.31062495299999998 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.640377+0000", 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7800000000000006e-06 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.641583+0000", 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012057770000000001 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:14.641775+0000", 2024-09-17T16:25:53.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000192253 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:2268 5.2 5:448d2a5f:::benchmark_data_smithi173_90735_object2267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:51.492843+0000", 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 241.841774461, 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26093462899999997, 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.382 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 2268 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492843+0000", 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492843+0000", 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492844+0000", 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2899999999999999e-06 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492888+0000", 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3458999999999998e-05 2024-09-17T16:25:53.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492901+0000", 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3258000000000001e-05 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492906+0000", 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4620000000000004e-06 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.492949+0000", 2024-09-17T16:25:53.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2256000000000003e-05 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.493035+0000", 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6211999999999994e-05 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.493123+0000", 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7849999999999994e-05 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.493462+0000", 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00033951800000000002 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.494234+0000", 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00077194099999999995 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.753697+0000", 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25946257700000003 2024-09-17T16:25:53.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.753735+0000", 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8816000000000002e-05 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:51.753777+0000", 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1990000000000003e-05 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.388 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:54626 5.19 5:9d4a6242:::benchmark_data_smithi173_90735_object54625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:39.758709+0000", 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.57590834300001, 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.414626938, 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:53.389 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 54626 2024-09-17T16:25:53.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.390 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758709+0000", 2024-09-17T16:25:53.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758709+0000", 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758710+0000", 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2389999999999999e-06 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758803+0000", 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3035000000000006e-05 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758813+0000", 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6900000000000004e-06 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758818+0000", 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8080000000000004e-06 2024-09-17T16:25:53.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.758950+0000", 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000132719 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.759057+0000", 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000106562 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 5,7,8", 2024-09-17T16:25:53.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.759170+0000", 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011260999999999999 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.760775+0000", 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001605605 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.761105+0000", 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032938999999999999 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.763632+0000", 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0025276539999999998 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172496+0000", 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40886421000000001 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172702+0000", 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000206011 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.173336+0000", 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00063340499999999997 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26158.0:54626 5.19 e450/444 ondisk, result = 0)", 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:39.760626+0000", 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.57399081899999, 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.412716058, 2024-09-17T16:25:53.396 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.760626+0000", 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.760626+0000", 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.760627+0000", 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9500000000000002e-07 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.760632+0000", 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4689999999999999e-06 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172298+0000", 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41166605099999998 2024-09-17T16:25:53.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172309+0000", 2024-09-17T16:25:53.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1286e-05 2024-09-17T16:25:53.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172438+0000", 2024-09-17T16:25:53.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012923200000000001 2024-09-17T16:25:53.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.172470+0000", 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1585000000000003e-05 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.173342+0000", 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00087274 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.401 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(5.19 450/444 [PushReplyOp(5:99cc975c:::benchmark_data_smithi173_90735_object22010:head)])", 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:39.919313+0000", 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.41530414600001, 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26979225699999998, 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.919313+0000", 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.919313+0000", 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.919314+0000", 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7999999999999996e-07 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:39.919317+0000", 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.517e-06 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.188344+0000", 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26902674500000001 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.188349+0000", 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0100000000000003e-06 2024-09-17T16:25:53.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.188388+0000", 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9285000000000001e-05 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:40.189105+0000", 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00071721999999999997 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPull(5.12 e472/455 cost 1049576)", 2024-09-17T16:25:53.406 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:23:13.988499+0000", 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 159.346118416, 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28772111500000003, 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:13.988499+0000", 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:13.988499+0000", 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:13.988500+0000", 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2309999999999999e-06 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:13.988513+0000", 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2965999999999999e-05 2024-09-17T16:25:53.408 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:14.275869+0000", 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.287355468 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:14.275874+0000", 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9059999999999999e-06 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:14.275915+0000", 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0499e-05 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:23:14.276220+0000", 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030504499999999999 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:148038 5.11 5:8c5c1e75:::benchmark_data_smithi173_90735_object65267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-17T16:25:53.410 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:24:27.901276+0000", 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 85.433341677000001, 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26851852999999998, 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 148038 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.901276+0000", 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.901276+0000", 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.901276+0000", 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9199999999999999e-07 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.901281+0000", 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4569999999999998e-06 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.901292+0000", 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0881e-05 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.901297+0000", 2024-09-17T16:25:53.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2800000000000003e-06 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.901332+0000", 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4446000000000002e-05 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.901435+0000", 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010388899999999999 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,9", 2024-09-17T16:25:53.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.901497+0000", 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1650999999999996e-05 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.902993+0000", 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001495461 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.903013+0000", 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9978000000000001e-05 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:28.169697+0000", 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.266684213 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:28.169742+0000", 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5036e-05 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:28.169794+0000", 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2346000000000003e-05 2024-09-17T16:25:53.416 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26158.0:148038 5.11 e507/450 ondisk, result = 0)", 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:24:27.902708+0000", 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 85.431909257000001, 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26709717399999999, 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.902708+0000", 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.902708+0000", 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.902708+0000", 2024-09-17T16:25:53.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2800000000000002e-07 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:27.902711+0000", 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.774e-06 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:28.169430+0000", 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.266719068 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.419 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:28.169436+0000", 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0100000000000001e-06 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:28.169684+0000", 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024772999999999999 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:28.169691+0000", 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1280000000000002e-06 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:28.169805+0000", 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000114036 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:158679 5.2 5:4405d441:::benchmark_data_smithi173_90735_object75908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:24:38.797105+0000", 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 74.537512133999996, 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.280267139, 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.421 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 158679 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.797105+0000", 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.797105+0000", 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.797106+0000", 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.111e-06 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.797110+0000", 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9639999999999997e-06 2024-09-17T16:25:53.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.797121+0000", 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0923e-05 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.797126+0000", 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2109999999999997e-06 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.797200+0000", 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3819000000000006e-05 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.797277+0000", 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6416999999999997e-05 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,10", 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.797339+0000", 2024-09-17T16:25:53.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2298999999999994e-05 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.799414+0000", 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020746499999999999 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.825705+0000", 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.026291796999999999 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:39.077308+0000", 2024-09-17T16:25:53.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.251602779 2024-09-17T16:25:53.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:39.077344+0000", 2024-09-17T16:25:53.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6018e-05 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:39.077372+0000", 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8150999999999999e-05 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26158.0:158679 5.2 e512/441 ondisk, result = 0)", 2024-09-17T16:25:53.428 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:24:38.799699+0000", 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 74.534918449000003, 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27767710899999998, 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.799699+0000", 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.799699+0000", 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.799699+0000", 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2300000000000001e-07 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:38.799702+0000", 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9830000000000001e-06 2024-09-17T16:25:53.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:39.077251+0000", 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.277548293 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:39.077256+0000", 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5820000000000001e-06 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:39.077295+0000", 2024-09-17T16:25:53.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8824e-05 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:39.077302+0000", 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4730000000000003e-06 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:24:39.077376+0000", 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3430999999999995e-05 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.432 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:17638 6.a 6:5f824b0d:::benchmark_data_smithi173_102017_object17637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:25:22.116190+0000", 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 31.218427057, 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.402092703, 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:25:53.433 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 17638 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116190+0000", 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116190+0000", 2024-09-17T16:25:53.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116191+0000", 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8599999999999998e-07 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116256+0000", 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4834999999999998e-05 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116264+0000", 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1329999999999999e-06 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116268+0000", 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2370000000000003e-06 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116296+0000", 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7438000000000001e-05 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116363+0000", 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7587000000000002e-05 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116436+0000", 2024-09-17T16:25:53.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2534999999999996e-05 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116929+0000", 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049371199999999999 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.117182+0000", 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000252141 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518211+0000", 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40102907199999999 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518251+0000", 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0295e-05 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518283+0000", 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2032000000000001e-05 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.439 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26665.0:17638 6.a e538/527 ondisk, result = 0)", 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:25:22.117495+0000", 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 31.217122108000002, 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.400793913, 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.117495+0000", 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.117495+0000", 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.117496+0000", 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4000000000000002e-07 2024-09-17T16:25:53.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.117498+0000", 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8899999999999999e-06 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518017+0000", 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40051810799999998 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518030+0000", 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3507e-05 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518173+0000", 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000142781 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518190+0000", 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7368999999999999e-05 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518289+0000", 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8818000000000002e-05 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:27647 6.a 6:52a81611:::benchmark_data_smithi173_102017_object27646:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:25:32.709430+0000", 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 20.625187197999999, 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43289498700000001, 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:25:53.444 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 27647 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709430+0000", 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:25:53.445 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709430+0000", 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709432+0000", 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5209999999999999e-06 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709489+0000", 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7837000000000002e-05 2024-09-17T16:25:53.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709506+0000", 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6500000000000001e-05 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709514+0000", 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1899999999999995e-06 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709575+0000", 2024-09-17T16:25:53.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0844000000000002e-05 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709702+0000", 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012716499999999999 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709798+0000", 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5816000000000006e-05 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.710756+0000", 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00095821100000000005 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.968078+0000", 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25732213199999998 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:33.142260+0000", 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.174181488 2024-09-17T16:25:53.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:33.142294+0000", 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3816999999999998e-05 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:33.142325+0000", 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1466e-05 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:25:53.450 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:25:53.451 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:25:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:53.452 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:53.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:53.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 15 48709 48694 69.1612 53.625 0.056244 0.0144529 2024-09-17T16:25:53.554 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:53.583 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:53.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:53.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:25:53.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:53.710 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:53.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:53.812 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:53 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:25:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:53 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:25:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:53 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:25:53.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1813150687' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:53.835 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:53.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:53 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:25:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:53 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:25:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:53 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:25:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1813150687' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:53.937 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:53.960 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:53.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:53.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:53 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:25:53.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:53 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:25:53.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:53 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:25:53.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1813150687' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:25:54.003 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:25:54.060 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:54.083 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:54.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:54.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:54.210 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:54.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:54.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:25:54.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:54.343 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:54.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:54.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:54.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:54.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:54.535 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:54 smithi173 ceph-mon[30530]: pgmap v1205: 17 pgs: 17 active+clean; 2.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s; 170 KiB/s, 2 objects/s recovering 2024-09-17T16:25:54.535 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:54 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:25:54.535 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:54 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:25:54.535 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1813150687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:54.536 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:54 smithi173 ceph-mon[30530]: osdmap e554: 12 total, 12 up, 11 in 2024-09-17T16:25:54.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 49656 49640 68.9381 59.125 0.00583031 0.0144987 2024-09-17T16:25:54.555 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:25:54.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:54.602 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:54.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:54.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:54.704 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:54.728 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:54.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:54 smithi071 ceph-mon[30002]: pgmap v1205: 17 pgs: 17 active+clean; 2.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s; 170 KiB/s, 2 objects/s recovering 2024-09-17T16:25:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:54 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:25:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:54 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:25:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1813150687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:54 smithi071 ceph-mon[30002]: osdmap e554: 12 total, 12 up, 11 in 2024-09-17T16:25:54.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:25:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:54.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:54.858 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:25:54.959 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:54.983 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:54.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:54.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:54 smithi017 ceph-mon[27571]: pgmap v1205: 17 pgs: 17 active+clean; 2.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s; 170 KiB/s, 2 objects/s recovering 2024-09-17T16:25:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:54 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:25:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:54 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:25:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1813150687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:25:54.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:54 smithi017 ceph-mon[27571]: osdmap e554: 12 total, 12 up, 11 in 2024-09-17T16:25:55.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:55.105 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:55.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:55.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:55.109 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:25:55.109 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:25:55.109 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:25:55.109 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:25:55.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:55.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:55.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:55.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:55.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:55.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:55.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:55.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:55.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:55.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 50711 50695 68.8728 65.9375 0.00555106 0.014514 2024-09-17T16:25:55.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:55.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:55.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:55.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:55 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:55.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:55 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:55.678 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:55 smithi173 ceph-mon[30530]: osdmap e555: 12 total, 12 up, 11 in 2024-09-17T16:25:55.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:55 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:55 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:55 smithi017 ceph-mon[27571]: osdmap e555: 12 total, 12 up, 11 in 2024-09-17T16:25:55.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:55.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:55 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:55 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:55 smithi071 ceph-mon[30002]: osdmap e555: 12 total, 12 up, 11 in 2024-09-17T16:25:55.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:55.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:55.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:55.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:25:55.982 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:56.006 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:56.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:56.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:56.130 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:56.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:56.231 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:56.256 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:56.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:56.262 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:25:56.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:56.380 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:56.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:56.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:56.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:56.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:56.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 51815 51799 68.8754 69 0.00284809 0.0145062 2024-09-17T16:25:56.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:25:56.631 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:56.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:56.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:25:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:56 smithi173 ceph-mon[30530]: pgmap v1208: 17 pgs: 17 active+clean; 2.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 61 MiB/s wr, 970 op/s 2024-09-17T16:25:56.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:56.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:56.756 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:56.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:56.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-17T16:25:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:56 smithi071 ceph-mon[30002]: pgmap v1208: 17 pgs: 17 active+clean; 2.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 61 MiB/s wr, 970 op/s 2024-09-17T16:25:56.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:56.864 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:56 smithi017 ceph-mon[27571]: pgmap v1208: 17 pgs: 17 active+clean; 2.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 61 MiB/s wr, 970 op/s 2024-09-17T16:25:56.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:56.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:56.930 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:56.985 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:57.027 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:57.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:57.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:57.150 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:25:57.150 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:25:57.150 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:25:57.151 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:25:57.151 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:25:57.171 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:57.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:57.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:57.294 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:57.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:57.396 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:57.435 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:57.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:57.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:57.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 52846 52830 68.7828 64.4375 0.00649171 0.0145316 2024-09-17T16:25:57.560 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:57.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:57.581 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:25:57.581 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:25:57.609 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:25:57.609 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:25:57.645 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:25:57.646 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:25:57.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:25:57.667 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:25:57.681 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:25:57.681 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:25:57.699 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:57.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:25:57.713 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:25:57.714 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:25:57.746 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:25:57.747 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:25:57.780 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:25:57.780 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:25:57.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:57.802 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:25:57.802 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:25:57.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:25:57.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:57.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:57.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:57.824 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:25:57.824 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:25:57.845 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:25:57.845 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:25:57.874 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:25:57.875 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:25:57.900 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:25:57.900 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:25:57.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:57.947 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:57.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:58.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:58.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:58.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:58.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:58.110 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:25:58.173 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:58.195 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:58.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:58.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:25:58.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:58.321 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:58.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:58.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:25:58.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:58.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:25:58.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 53436 53420 68.1316 36.875 0.00364168 0.0146717 2024-09-17T16:25:58.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:58.590 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:58.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:58.634 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:25:58.634 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:25:58.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:25:58.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:58.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:58.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:25:58.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:25:58.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:25:58.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:58 smithi071 ceph-mon[30002]: osdmap e556: 12 total, 12 up, 11 in 2024-09-17T16:25:58.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3048391750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:58.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:25:58 smithi071 ceph-mon[30002]: pgmap v1210: 17 pgs: 17 active+clean; 3.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 53 MiB/s wr, 853 op/s 2024-09-17T16:25:58.818 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:58.842 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:58.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:25:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:25:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:58 smithi173 ceph-mon[30530]: osdmap e556: 12 total, 12 up, 11 in 2024-09-17T16:25:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3048391750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:25:58 smithi173 ceph-mon[30530]: pgmap v1210: 17 pgs: 17 active+clean; 3.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 53 MiB/s wr, 853 op/s 2024-09-17T16:25:58.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:58.968 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:58.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:25:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:25:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:58 smithi017 ceph-mon[27571]: osdmap e556: 12 total, 12 up, 11 in 2024-09-17T16:25:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3048391750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:25:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:25:58 smithi017 ceph-mon[27571]: pgmap v1210: 17 pgs: 17 active+clean; 3.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 53 MiB/s wr, 853 op/s 2024-09-17T16:25:59.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:25:59.093 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:59.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:25:59.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:25:59.216 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:59.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:25:59.319 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:25:59.350 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:59.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:59.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:25:59.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:25:59.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:25:59.475 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:25:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:59.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:25:59.541 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 54581 54565 68.2001 71.5625 0.00532425 0.0146561 2024-09-17T16:25:59.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:25:59.607 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:59.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:25:59.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:25:59.740 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:25:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:59.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:25:59.759 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:25:59.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:25:59.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:25:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:25:59.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:25:59.970 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:00.002 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:00.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:00.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:00.033 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:00.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:00.135 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:00.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:00.237 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:00.265 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:00.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:00.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:00.309 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:26:00.309 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:26:00.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:00.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:00.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:00.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:00.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:00.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:00.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 55922 55906 68.5061 83.8125 0.0210216 0.0145919 2024-09-17T16:26:00.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:00.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:00.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:00.751 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:00.774 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:00.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:00.789 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:00 smithi017 ceph-mon[27571]: pgmap v1211: 17 pgs: 17 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:26:00.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-17T16:26:00.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:00 smithi071 ceph-mon[30002]: pgmap v1211: 17 pgs: 17 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:26:00.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:00.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:00 smithi173 ceph-mon[30530]: pgmap v1211: 17 pgs: 17 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:26:00.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:00.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:01.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:01.040 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:01.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:01.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:26:01.137 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:01.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:01.163 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:01.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:01.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:01.265 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:01.292 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:01.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:01.345 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:26:01.396 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:01.419 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:01.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:01.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:01.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:01.531 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:26:01.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 56963 56947 68.4398 65.0625 0.0302143 0.0146054 2024-09-17T16:26:01.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:01.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:01.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:01.678 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:01.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:01.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:01.780 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2397603672' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:26:01.780 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:01 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:26:01.799 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:26:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:01.803 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:01.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:01.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2397603672' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:26:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:01 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:26:01.834 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2397603672' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:26:01.834 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:01 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:26:01.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:01.914 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:26:01.914 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:26:01.926 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:01.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:02.028 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:02.056 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:02.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:02.158 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:02.182 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:02.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:02.284 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:02.310 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:02.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:02.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:02.435 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:02.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:02.537 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:02.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 57879 57863 68.2285 57.25 0.00495629 0.0146281 2024-09-17T16:26:02.559 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:02.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:02.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:26:02.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:02.689 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:02.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:02.791 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:02 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:02 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:26:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:02 smithi071 ceph-mon[30002]: osdmap e557: 12 total, 12 up, 11 in 2024-09-17T16:26:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:02 smithi071 ceph-mon[30002]: pgmap v1213: 17 pgs: 17 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-17T16:26:02.824 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:02.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:02.864 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:26:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:02 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:02 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:26:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:02 smithi173 ceph-mon[30530]: osdmap e557: 12 total, 12 up, 11 in 2024-09-17T16:26:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:02 smithi173 ceph-mon[30530]: pgmap v1213: 17 pgs: 17 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-17T16:26:02.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:02.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:02.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:02 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:02 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:26:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:02 smithi017 ceph-mon[27571]: osdmap e557: 12 total, 12 up, 11 in 2024-09-17T16:26:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:02 smithi017 ceph-mon[27571]: pgmap v1213: 17 pgs: 17 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-17T16:26:03.056 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:03.077 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:03.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:03.179 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:03.201 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:03.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:03.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:03.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:03.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:03.431 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:26:03.431 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:26:03.434 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:03.456 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:03.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:03.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 58928 58912 68.179 65.5625 0.0358698 0.0146608 2024-09-17T16:26:03.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:03.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:03.586 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:03.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:03.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:03.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:03.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:03.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:03.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:03.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:03.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:03.969 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:03.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:04.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:04.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:04.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:04.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:26:04.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:04.261 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:04.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:04.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:04.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:04.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:04.487 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:04.492 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:26:04.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:04.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-09-17T16:26:04.521 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:04.522 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:04.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 15 60173 60158 68.3552 77.875 0.0104774 0.0146243 2024-09-17T16:26:04.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:04.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:04.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:04.753 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:04.778 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:04.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:04.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:04.906 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:04.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:05.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:05.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:05.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:05.051 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:26:05.051 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:26:05.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:04 smithi017 ceph-mon[27571]: pgmap v1214: 17 pgs: 17 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:26:05.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:04 smithi017 ceph-mon[27571]: osdmap e558: 12 total, 12 up, 11 in 2024-09-17T16:26:05.121 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:05.142 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:05.164 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:05.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:05.266 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:05.287 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:05.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:05.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:04 smithi071 ceph-mon[30002]: pgmap v1214: 17 pgs: 17 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:26:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:04 smithi071 ceph-mon[30002]: osdmap e558: 12 total, 12 up, 11 in 2024-09-17T16:26:05.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:05.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:04 smithi173 ceph-mon[30530]: pgmap v1214: 17 pgs: 17 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:26:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:04 smithi173 ceph-mon[30530]: osdmap e558: 12 total, 12 up, 11 in 2024-09-17T16:26:05.412 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:05.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:05.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:05.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 61367 61351 68.4659 74.5625 0.00308319 0.014599 2024-09-17T16:26:05.548 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:05.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:05.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:05.671 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:05.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:05.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:26:05.773 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:05.799 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:05.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:05.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:05.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:26:05.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:05.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:05.931 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:05.963 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:26:05.963 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:26:06.000 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:26:06.000 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:26:06.032 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:06.035 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:26:06.035 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:26:06.058 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:26:06.075 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:06.075 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:26:06.076 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:26:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:06.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:06.107 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:26:06.108 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:26:06.135 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:26:06.136 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:26:06.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:26:06.156 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:26:06.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:26:06.177 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:26:06.177 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:06.200 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:26:06.201 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:26:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:06.202 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:06.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:06.231 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:26:06.232 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:26:06.232 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2671145227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:06.270 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:26:06.271 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:26:06.303 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:06.305 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:26:06.305 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:26:06.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2671145227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:06.337 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2671145227' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:06.352 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:06.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:06.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:06.479 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:06.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:06.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 62276 62260 68.2614 56.8125 0.00354912 0.0146445 2024-09-17T16:26:06.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:06.606 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:06.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:06.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:06.608 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:26:06.608 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:26:06.608 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:26:06.608 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:26:06.608 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:26:06.708 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:06.741 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:06.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:06.843 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:06.877 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:06.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:06.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:07.005 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:07.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:07.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:07.108 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:07.139 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:07.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:07.170 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:07.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:06 smithi017 ceph-mon[27571]: pgmap v1216: 17 pgs: 17 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 55 MiB/s wr, 880 op/s 2024-09-17T16:26:07.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:07.243 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:06 smithi173 ceph-mon[30530]: pgmap v1216: 17 pgs: 17 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 55 MiB/s wr, 880 op/s 2024-09-17T16:26:07.257 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:06 smithi071 ceph-mon[30002]: pgmap v1216: 17 pgs: 17 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 55 MiB/s wr, 880 op/s 2024-09-17T16:26:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:07.268 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:07.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:07.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:26:07.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:07.393 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:07.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:07.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:07.517 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:07.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:07.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 63744 63728 68.6662 91.75 0.0237923 0.0145578 2024-09-17T16:26:07.593 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:26:07.619 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:07.649 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:07.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:07.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:07.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:07.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:07.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:07.877 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:07.913 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:07.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:08.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:08.040 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:08.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:08.142 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:08.166 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:08.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:143899 3.1 3:8d15b72a:::benchmark_data_smithi173_66272_object71943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.475047+0000", 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 527.68810740000004, 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37967805900000001, 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.168 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 143899 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475047+0000", 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475047+0000", 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475048+0000", 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.065e-06 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475053+0000", 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8330000000000001e-06 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475067+0000", 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.416e-05 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.475075+0000", 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7049999999999999e-06 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.478661+0000", 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00358662 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.478828+0000", 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016719200000000001 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:26:08.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.478988+0000", 2024-09-17T16:26:08.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015920299999999999 2024-09-17T16:26:08.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:26:08.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.487257+0000", 2024-09-17T16:26:08.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0082690630000000001 2024-09-17T16:26:08.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.498341+0000", 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011084053999999999 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.854410+0000", 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35606905799999999 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.854451+0000", 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.155e-05 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.854725+0000", 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00027355599999999999 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1 e290,272 3:8b678a21:::benchmark_data_smithi173_66272_object17829:head,261'5698)", 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.665092+0000", 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 527.498062151, 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46414988299999999, 2024-09-17T16:26:08.178 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665092+0000", 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665092+0000", 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665093+0000", 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9599999999999999e-07 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.665097+0000", 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2830000000000003e-06 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.128735+0000", 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46363770199999998 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.128747+0000", 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2779000000000001e-05 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.128815+0000", 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7749999999999993e-05 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.129242+0000", 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042677299999999997 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:76068 4.5 e335/301 4:a01bffc8:::benchmark_data_smithi173_77474_object76067:head v 335'4877, mlcod=335'4875)", 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:18:40.016492+0000", 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 448.14666218999997, 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39477241400000002, 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.182 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016492+0000", 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016492+0000", 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016493+0000", 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3659999999999999e-06 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.016567+0000", 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3520000000000001e-05 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409280+0000", 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39271265799999999 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409289+0000", 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8789999999999992e-06 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409338+0000", 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9029000000000001e-05 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.409355+0000", 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7727000000000002e-05 2024-09-17T16:26:08.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.411210+0000", 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018547570000000001 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:18:40.411265+0000", 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4478000000000003e-05 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(4.a 389/383 [PushReplyOp(4:5cbc523f:::benchmark_data_smithi173_77474_object91105:head)])", 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:20:53.654477+0000", 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 314.50867720500003, 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46374295300000001, 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.654477+0000", 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.654477+0000", 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.654477+0000", 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.55e-07 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.654480+0000", 2024-09-17T16:26:08.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.021e-06 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.117956+0000", 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46347616600000002 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.117966+0000", 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4749999999999999e-06 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.118056+0000", 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9608e-05 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.118220+0000", 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016452799999999999 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.190 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(4.5 389/383 [PushReplyOp(4:aaa668bb:::benchmark_data_smithi173_77474_object91191:head)])", 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:20:53.787848+0000", 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 314.375306362, 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.677856495, 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.787848+0000", 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.787848+0000", 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.787848+0000", 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.06e-07 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:53.787850+0000", 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3010000000000001e-06 2024-09-17T16:26:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.465463+0000", 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.677612942 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.465468+0000", 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2089999999999996e-06 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.465525+0000", 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6440999999999997e-05 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:20:54.465704+0000", 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017949599999999999 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25516.0:204338 4.7 4:e82f76a0:::benchmark_data_smithi173_77474_object102163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.614506+0000", 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 284.548648302, 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59349127700000004, 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25516", 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 204338 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614506+0000", 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614506+0000", 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614508+0000", 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.745e-06 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614514+0000", 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6170000000000003e-06 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614528+0000", 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3977e-05 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614536+0000", 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1030000000000002e-06 2024-09-17T16:26:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614584+0000", 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7456000000000001e-05 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614674+0000", 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0142000000000005e-05 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.614753+0000", 2024-09-17T16:26:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8603999999999995e-05 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.618482+0000", 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037292520000000002 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.622334+0000", 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038525199999999999 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207921+0000", 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58558639499999998 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207962+0000", 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1196999999999999e-05 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207997+0000", 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5268999999999997e-05 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25516.0:204340 4.a 4:583138b7:::benchmark_data_smithi173_77474_object102165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.617529+0000", 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 284.54562518900002, 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61758273900000005, 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25516", 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 204340 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617529+0000", 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617529+0000", 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617530+0000", 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617539+0000", 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7830000000000004e-06 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617554+0000", 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4467999999999999e-05 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617562+0000", 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8709999999999997e-06 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617616+0000", 2024-09-17T16:26:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4682000000000003e-05 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617738+0000", 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121927 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.617861+0000", 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123084 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.621672+0000", 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038110869999999999 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.193552+0000", 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57187970399999999 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235037+0000", 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041484961000000001 2024-09-17T16:26:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235078+0000", 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0890000000000003e-05 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235112+0000", 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3945000000000001e-05 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25516.0:204338 4.7 e408/306 ondisk, result = 0)", 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.619785+0000", 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 284.54336972499999, 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58821733300000001, 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.619785+0000", 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.619785+0000", 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.619785+0000", 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4799999999999998e-07 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.619788+0000", 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2909999999999999e-06 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207799+0000", 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58801086199999997 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207811+0000", 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1856e-05 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207912+0000", 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100862 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.207917+0000", 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1919999999999996e-06 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.208002+0000", 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4722000000000006e-05 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25516.0:204340 4.a e408/393 ondisk, result = 0)", 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.622256+0000", 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 284.54089830700002, 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61286083199999997, 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.622256+0000", 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.622256+0000", 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.622257+0000", 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0500000000000003e-07 2024-09-17T16:26:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.622261+0000", 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3850000000000004e-06 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.234958+0000", 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61269643900000004 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.234966+0000", 2024-09-17T16:26:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2940000000000006e-06 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235024+0000", 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8069000000000003e-05 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235032+0000", 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8709999999999997e-06 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.235117+0000", 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5068999999999995e-05 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:41596 5.17 5:edc74594:::benchmark_data_smithi173_90735_object41595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e441)", 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:23.961409+0000", 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 224.201745176, 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0288790649999999, 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 41596 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961409+0000", 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961409+0000", 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961410+0000", 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961444+0000", 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3828000000000002e-05 2024-09-17T16:26:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961454+0000", 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9760000000000007e-06 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961460+0000", 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9720000000000003e-06 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.032985+0000", 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.071524061 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.032994+0000", 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3030000000000005e-06 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.990202+0000", 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95720833599999999 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.990288+0000", 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5996999999999997e-05 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:41597 5.14 5:2fd5d8a7:::benchmark_data_smithi173_90735_object41596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e441)", 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:23.961480+0000", 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 224.20167454700001, 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0299093850000001, 2024-09-17T16:26:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 41597 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961480+0000", 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961480+0000", 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961480+0000", 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.68e-07 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961501+0000", 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0988999999999998e-05 2024-09-17T16:26:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961509+0000", 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2830000000000004e-06 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961515+0000", 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0390000000000001e-06 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.022776+0000", 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.061260904999999997 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.022785+0000", 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2969999999999992e-06 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.991313+0000", 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96852803600000004 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.991389+0000", 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6267999999999996e-05 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:41603 5.18 5:1fe2546e:::benchmark_data_smithi173_90735_object41602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e441)", 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:23.961531+0000", 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 224.201623321, 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.040207801, 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 41603 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961531+0000", 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961531+0000", 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961531+0000", 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9100000000000004e-07 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961552+0000", 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0920999999999998e-05 2024-09-17T16:26:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961560+0000", 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9119999999999998e-06 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961566+0000", 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1580000000000002e-06 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036249+0000", 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.074682889000000002 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036257+0000", 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7170000000000008e-06 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001493+0000", 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96523577699999996 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001739+0000", 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024593599999999998 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:41610 5.18 5:1c8bbb85:::benchmark_data_smithi173_90735_object41609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e441)", 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:23.961621+0000", 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 224.20153366299999, 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.040819079, 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 41610 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961621+0000", 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961621+0000", 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961621+0000", 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8100000000000003e-07 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961654+0000", 2024-09-17T16:26:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2790999999999998e-05 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961662+0000", 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7700000000000001e-06 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:23.961668+0000", 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9159999999999998e-06 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036392+0000", 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.074724225000000005 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036398+0000", 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2720000000000003e-06 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001792+0000", 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96539429499999996 2024-09-17T16:26:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.002440+0000", 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00064732899999999998 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:41615 5.10 5:09bf280c:::benchmark_data_smithi173_90735_object41614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:24.035550+0000", 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 224.127603816, 2024-09-17T16:26:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96589714400000004, 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 41615 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035550+0000", 2024-09-17T16:26:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035550+0000", 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035553+0000", 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2239999999999998e-06 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035582+0000", 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9641999999999998e-05 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035596+0000", 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3346000000000001e-05 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.035606+0000", 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0032e-05 2024-09-17T16:26:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036421+0000", 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081489300000000002 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.036429+0000", 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.799e-06 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.998358+0000", 2024-09-17T16:26:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.961928597 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.998483+0000", 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012492100000000001 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:24.998643+0000", 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015998499999999999 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001077+0000", 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002434322 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001309+0000", 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023223699999999999 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001357+0000", 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7206999999999997e-05 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001416+0000", 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9110000000000002e-05 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:25.001448+0000", 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1829000000000003e-05 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:82755 5.7 5:e29d80c7:::benchmark_data_smithi173_90735_object82754:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.908516+0000", 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.254637833, 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.660800991, 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 82755 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908516+0000", 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908516+0000", 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908518+0000", 2024-09-17T16:26:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6819999999999999e-06 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908541+0000", 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2518999999999999e-05 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908556+0000", 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5136e-05 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908562+0000", 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1909999999999997e-06 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908608+0000", 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5762e-05 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908709+0000", 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101384 2024-09-17T16:26:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.908805+0000", 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6137999999999993e-05 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.910192+0000", 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013868350000000001 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.914165+0000", 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039727699999999996 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569146+0000", 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65498125500000004 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569247+0000", 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010063799999999999 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569317+0000", 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0680999999999996e-05 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26158.0:82755 5.7 e475/441 ondisk, result = 0)", 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.910504+0000", 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.252649937, 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65882479199999999, 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.910504+0000", 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.910504+0000", 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.910505+0000", 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4200000000000005e-07 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.910510+0000", 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4909999999999999e-06 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569010+0000", 2024-09-17T16:26:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65850046799999995 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569022+0000", 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2257e-05 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569117+0000", 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4108000000000001e-05 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569136+0000", 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9956999999999999e-05 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.569329+0000", 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019266899999999999 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:82763 5.13 e475/453 5:cd63b9f8:::benchmark_data_smithi173_90735_object82762:head v 475'4007, mlcod=475'4003)", 2024-09-17T16:26:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.918146+0000", 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.24500827099999, 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62102570099999999, 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.918146+0000", 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.918146+0000", 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.918147+0000", 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4799999999999998e-07 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.918166+0000", 2024-09-17T16:26:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9106999999999998e-05 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.537841+0000", 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61967572500000001 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.537850+0000", 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9560000000000003e-06 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.537915+0000", 2024-09-17T16:26:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4109999999999994e-05 2024-09-17T16:26:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.537939+0000", 2024-09-17T16:26:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4128999999999999e-05 2024-09-17T16:26:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.539124+0000", 2024-09-17T16:26:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011852399999999999 2024-09-17T16:26:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.539172+0000", 2024-09-17T16:26:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7786e-05 2024-09-17T16:26:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:82768 5.6 e475/463 5:63886666:::benchmark_data_smithi173_90735_object82767:head v 475'3819, mlcod=475'3817)", 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.925908+0000", 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.23724581499999, 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83103535299999998, 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.925908+0000", 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.925908+0000", 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.925909+0000", 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4600000000000004e-07 2024-09-17T16:26:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.925968+0000", 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8801000000000002e-05 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.755595+0000", 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82962728200000002 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.755603+0000", 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2389999999999996e-06 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.755677+0000", 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4873e-05 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.755695+0000", 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7167999999999999e-05 2024-09-17T16:26:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.756901+0000", 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012064560000000001 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.756944+0000", 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2787999999999998e-05 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:95335 5.7 5:e4eaf7db:::benchmark_data_smithi173_90735_object12564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:32.800982+0000", 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 155.36217241400001, 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44480024899999998, 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 95335 2024-09-17T16:26:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.800982+0000", 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.800982+0000", 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.800983+0000", 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.181e-06 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.800988+0000", 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9849999999999997e-06 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.800999+0000", 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1228e-05 2024-09-17T16:26:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.801006+0000", 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1679999999999998e-06 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.801055+0000", 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8681999999999999e-05 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.801174+0000", 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118672 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.801331+0000", 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015736600000000001 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.802689+0000", 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001357886 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.803076+0000", 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038712499999999999 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245680+0000", 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44260339799999998 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245736+0000", 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6468000000000002e-05 2024-09-17T16:26:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245782+0000", 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6090000000000001e-05 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26158.0:95335 5.7 e481/441 ondisk, result = 0)", 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:32.803052+0000", 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 155.36010248400001, 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44273757400000002, 2024-09-17T16:26:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.803052+0000", 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.803052+0000", 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.803052+0000", 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7899999999999998e-07 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:32.803056+0000", 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.489e-06 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245613+0000", 2024-09-17T16:26:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.442557058 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245618+0000", 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9930000000000003e-06 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245665+0000", 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6489000000000002e-05 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245674+0000", 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1230000000000006e-06 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:33.245789+0000", 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011574300000000001 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:08.276 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:26:08.277 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:08.302 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:08.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:08.403 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:08.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:08.429 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:08.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:08.530 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:08.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 65041 65025 68.8762 81.0625 0.0436473 0.0145108 2024-09-17T16:26:08.558 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:08.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:08.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:08.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-09-17T16:26:08.660 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:08.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:08.701 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:08.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:08.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:08.841 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:08.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:08.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:08.988 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:08.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:08.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:09.055 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:09.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:08 smithi071 ceph-mon[30002]: pgmap v1217: 17 pgs: 17 active+clean; 3.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-17T16:26:09.092 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:09.127 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:09.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:09.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:08 smithi173 ceph-mon[30530]: pgmap v1217: 17 pgs: 17 active+clean; 3.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-17T16:26:09.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:08 smithi017 ceph-mon[27571]: pgmap v1217: 17 pgs: 17 active+clean; 3.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-17T16:26:09.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:26:09.228 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:09.260 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:09.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:09.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:09.362 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:09.453 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:09.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:26:09.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:26:09.541364+0000 min lat: 0.00108766 max lat: 2.21118 avg lat: 0.0145296 2024-09-17T16:26:09.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:26:09.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 66053 66037 68.7824 63.25 0.00329271 0.0145296 2024-09-17T16:26:09.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:09.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:09.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:09.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:09.715 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:09.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:09.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:09.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:26:09.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:09.839 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:09.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:09.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:09.972 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:09.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:10.014 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:26:10.049 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:26:10.050 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:26:10.050 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:26:10.050 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:26:10.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:10.102 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:10.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:10.204 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:10.226 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:10.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/462308042' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:10 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/462308042' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:10 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:10.328 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:10.357 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:10.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:10.367 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/462308042' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:10.368 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:10 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:10.459 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:10.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:10.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:10.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 67116 67100 68.7438 66.4375 0.00939141 0.0145406 2024-09-17T16:26:10.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:10.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:10.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:10.637 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-09-17T16:26:10.722 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:10.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:26:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:10.764 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:10.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:10.864 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:10.908 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:10.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:11.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:11.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:11.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:11.065 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:26:11.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:11.173 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:11.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:11 smithi017 ceph-mon[27571]: pgmap v1218: 17 pgs: 17 active+clean; 3.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-17T16:26:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:11 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:11 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:26:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:11 smithi017 ceph-mon[27571]: osdmap e559: 12 total, 12 up, 11 in 2024-09-17T16:26:11.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:11.299 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:11.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:11 smithi071 ceph-mon[30002]: pgmap v1218: 17 pgs: 17 active+clean; 3.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-17T16:26:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:11 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:11 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:26:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:11 smithi071 ceph-mon[30002]: osdmap e559: 12 total, 12 up, 11 in 2024-09-17T16:26:11.346 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:11 smithi173 ceph-mon[30530]: pgmap v1218: 17 pgs: 17 active+clean; 3.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-17T16:26:11.346 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:11 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:11.346 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:11 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:26:11.346 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:11 smithi173 ceph-mon[30530]: osdmap e559: 12 total, 12 up, 11 in 2024-09-17T16:26:11.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:11.428 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:11.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:11.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:11.470 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:11.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:11.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 68281 68265 68.8094 72.8125 0.00374771 0.0145281 2024-09-17T16:26:11.568 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:11.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:11.631 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:26:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:26:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:26:11.631 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:26:11.631 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:26:11.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:11.705 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:11.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:11.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:11.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:11.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:11.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:11.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:11.963 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:11.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:11.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:12.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:12.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:12.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:12.189 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:12.217 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:12.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:12.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:12.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:12.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:12.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:12.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:26:12.449 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:12.475 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:12.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:12.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 69383 69367 68.8103 68.875 0.0424021 0.0145243 2024-09-17T16:26:12.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:12.601 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:12.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:12.699 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:26:12.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:12.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:12.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:12.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:12.853 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:12.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:12.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:12.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:12.976 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:12.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:12.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:13.078 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:13.104 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:13.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:13.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:13.120 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:13 smithi173 ceph-mon[30530]: pgmap v1220: 17 pgs: 17 active+clean; 4.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-17T16:26:13.206 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:13.230 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:13.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:13.257 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:26:13.257 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:26:13.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:26:13.257 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:124280 3.0 3.53481990 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:16:48.935442+0000", 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 564.31876434499998, 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41915761200000001, 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124280 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.258 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935442+0000", 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935442+0000", 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935443+0000", 2024-09-17T16:26:13.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4270000000000001e-06 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935448+0000", 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2820000000000004e-06 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935460+0000", 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2091e-05 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935467+0000", 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4489999999999996e-06 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935529+0000", 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1877000000000004e-05 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:48.935541+0000", 2024-09-17T16:26:13.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2145e-05 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.354580+0000", 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41903870599999998 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:16:49.354599+0000", 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9635000000000001e-05 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.262 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25036.0:135118 3.e e281/273 3:7ce4ae2e:::benchmark_data_smithi173_66272_object63162:head v 281'8595, mlcod=281'8592)", 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:07.271192+0000", 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 545.98301399399998, 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34001416000000001, 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.271192+0000", 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.271192+0000", 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.271193+0000", 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6499999999999999e-07 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.271197+0000", 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6029999999999998e-06 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.607697+0000", 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33649947400000002 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.607701+0000", 2024-09-17T16:26:13.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7400000000000004e-06 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.607746+0000", 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4199999999999997e-05 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.607768+0000", 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2829000000000001e-05 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.611170+0000", 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034019469999999998 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.611206+0000", 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5701999999999998e-05 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.267 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(3.0 281/273 [PushOp(3:09b7de72:::benchmark_data_smithi173_66272_object64042:head, version: 245'4009, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:09b7de72:::benchmark_data_smithi173_66272_object64042:head@245'4009, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:07.277690+0000", 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 545.97651547099997, 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30499425000000002, 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.277690+0000", 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.277690+0000", 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.277691+0000", 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3899999999999999e-07 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.277805+0000", 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114574 2024-09-17T16:26:13.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.582375+0000", 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.304569592 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.582392+0000", 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6841999999999999e-05 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.582482+0000", 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0155999999999995e-05 2024-09-17T16:26:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.582496+0000", 2024-09-17T16:26:13.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4231e-05 2024-09-17T16:26:13.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:07.582685+0000", 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018871599999999999 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e284,272 3:86b32a61:::benchmark_data_smithi173_66272_object3793:head,254'4797)", 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:09.831735+0000", 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 543.42247121699995, 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43025157200000003, 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.272 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:09.831735+0000", 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:09.831735+0000", 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:09.831736+0000", 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.581e-06 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:09.831743+0000", 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4030000000000004e-06 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:10.261847+0000", 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43010377999999999 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:10.261853+0000", 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.934e-06 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:10.261917+0000", 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4029999999999995e-05 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:10.261940+0000", 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2642000000000001e-05 2024-09-17T16:26:13.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:10.261986+0000", 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6202000000000003e-05 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25036.0:143877 3.0 3:010a2738:::benchmark_data_smithi173_66272_object71921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.450695+0000", 2024-09-17T16:26:13.276 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 532.80351136100001, 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.47867361000000003, 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25036", 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 143877 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450695+0000", 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450695+0000", 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450696+0000", 2024-09-17T16:26:13.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3540000000000001e-06 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450701+0000", 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7269999999999998e-06 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450712+0000", 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1715999999999999e-05 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450720+0000", 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5349999999999999e-06 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450766+0000", 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6044999999999997e-05 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450855+0000", 2024-09-17T16:26:13.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9117999999999996e-05 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,9", 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.450938+0000", 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3379000000000002e-05 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.454980+0000", 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040412570000000004 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:26:13.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.455817+0000", 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083735500000000002 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.929284+0000", 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.473467058 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.929322+0000", 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7676000000000003e-05 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.929368+0000", 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6390000000000001e-05 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25036.0:143908 3.e e290/273 3:70cae513:::benchmark_data_smithi173_66272_object71952:head v 290'9136, mlcod=290'9133)", 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.489614+0000", 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 532.76459201, 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35349719099999999, 2024-09-17T16:26:13.283 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.489614+0000", 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.489614+0000", 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.489615+0000", 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0419999999999999e-06 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.489620+0000", 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1490000000000002e-06 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.839741+0000", 2024-09-17T16:26:13.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.350120926 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.839756+0000", 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4943000000000001e-05 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.839921+0000", 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000165327 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.839942+0000", 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0633e-05 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.843071+0000", 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003129246 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.843111+0000", 2024-09-17T16:26:13.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9925000000000001e-05 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e290,272 3:81df571b:::benchmark_data_smithi173_66272_object17841:head,261'5699)", 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:17:20.667457+0000", 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 532.58674936800003, 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46443020699999998, 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:26:13.288 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.667457+0000", 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.667457+0000", 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.667457+0000", 2024-09-17T16:26:13.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.82e-07 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:20.667461+0000", 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8600000000000003e-06 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.131740+0000", 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46427869199999999 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.131748+0000", 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3319999999999992e-06 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.131816+0000", 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7775999999999996e-05 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.131826+0000", 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0421000000000001e-05 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:17:21.131887+0000", 2024-09-17T16:26:13.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0544000000000002e-05 2024-09-17T16:26:13.292 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.292 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.292 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.292 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:102161 4.9 e349/306 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head v 349'6286, mlcod=349'6284)", 2024-09-17T16:26:13.292 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:19:11.509193+0000", 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 421.74501344499998, 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90182478799999999, 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:11.509193+0000", 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:11.509193+0000", 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:11.509193+0000", 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4300000000000002e-07 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:11.509286+0000", 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2286000000000004e-05 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.408350+0000", 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89906420200000003 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.408359+0000", 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5310000000000007e-06 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.408445+0000", 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6104000000000001e-05 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.408464+0000", 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8043e-05 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.410981+0000", 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002517278 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:19:12.411017+0000", 2024-09-17T16:26:13.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6501000000000002e-05 2024-09-17T16:26:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:204335 4.9 e408/383 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head v 408'12572, mlcod=408'12569)", 2024-09-17T16:26:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.607078+0000", 2024-09-17T16:26:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 289.647127576, 2024-09-17T16:26:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77592090000000002, 2024-09-17T16:26:13.300 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607078+0000", 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607078+0000", 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607079+0000", 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8199999999999998e-07 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607086+0000", 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9600000000000003e-06 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.378990+0000", 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77190383900000004 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.379000+0000", 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.379065+0000", 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5287e-05 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.379088+0000", 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3334999999999999e-05 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.382963+0000", 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038744439999999999 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.382999+0000", 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6715999999999999e-05 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.304 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:2279 5.f e424/419 5:f5772ab5:::benchmark_data_smithi173_90735_object2278:head v 424'148, mlcod=424'146)", 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:51.502595+0000", 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 261.751610637, 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25253113999999999, 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502595+0000", 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502595+0000", 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502597+0000", 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.311e-06 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502640+0000", 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3847000000000002e-05 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754667+0000", 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25202682199999998 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754673+0000", 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1489999999999999e-06 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754725+0000", 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2009999999999998e-05 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754741+0000", 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5798e-05 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.755084+0000", 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034313300000000002 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.755126+0000", 2024-09-17T16:26:13.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2070000000000002e-05 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:15935 5.f e429/419 5:f3ff7eb6:::benchmark_data_smithi173_90735_object15934:head v 429'990, mlcod=429'987)", 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:02.437369+0000", 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 250.81683713000001, 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79522149499999994, 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437369+0000", 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437369+0000", 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437370+0000", 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9400000000000005e-07 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437426+0000", 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6798000000000001e-05 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231438+0000", 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79401129999999998 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231449+0000", 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1688000000000001e-05 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231596+0000", 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000146546 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231620+0000", 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4315999999999999e-05 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.232554+0000", 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093371600000000002 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.232590+0000", 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6436999999999997e-05 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:54616 5.17 e450/444 5:eee108da:::benchmark_data_smithi173_90735_object54615:head v 450'3019, mlcod=0'0)", 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:39.740381+0000", 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 213.51382521299999, 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58705162700000002, 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.313 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740381+0000", 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740381+0000", 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740382+0000", 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.249e-06 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740443+0000", 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0998000000000001e-05 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325597+0000", 2024-09-17T16:26:13.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58515414300000002 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325606+0000", 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5760000000000006e-06 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325653+0000", 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.774e-05 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325673+0000", 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9253e-05 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.327389+0000", 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001716104 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.327432+0000", 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3563999999999998e-05 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.317 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.318 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:63127 5.f e461/455 5:f7e3e25a:::benchmark_data_smithi173_90735_object63126:head v 461'3883, mlcod=425'480)", 2024-09-17T16:26:13.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:13 smithi071 ceph-mon[30002]: pgmap v1220: 17 pgs: 17 active+clean; 4.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-17T16:26:13.319 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:55.384543+0000", 2024-09-17T16:26:13.319 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 197.86966255199999, 2024-09-17T16:26:13.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23654767900000001, 2024-09-17T16:26:13.319 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.319 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.319 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.319 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384543+0000", 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384543+0000", 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384545+0000", 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.339e-06 2024-09-17T16:26:13.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384593+0000", 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8733000000000002e-05 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619021+0000", 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23442753799999999 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619029+0000", 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7270000000000007e-06 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619090+0000", 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1680999999999994e-05 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619117+0000", 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7013e-05 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.621043+0000", 2024-09-17T16:26:13.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001925679 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.621091+0000", 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7969000000000001e-05 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:77257 5.17 e472/470 5:ee127e34:::benchmark_data_smithi173_90735_object77256:head v 472'3755, mlcod=472'3752)", 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:13.811785+0000", 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 179.442421075, 2024-09-17T16:26:13.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.255431142, 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811785+0000", 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811785+0000", 2024-09-17T16:26:13.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811786+0000", 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8700000000000005e-07 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811954+0000", 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016799800000000001 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065415+0000", 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25346157200000002 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065422+0000", 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3969999999999999e-06 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065477+0000", 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5699000000000003e-05 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065491+0000", 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4069e-05 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.067186+0000", 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001694922 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.067216+0000", 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9697999999999999e-05 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.327 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:82762 5.d e475/455 5:bc524506:::benchmark_data_smithi173_90735_object82761:head v 475'5183, mlcod=439'2263)", 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.917268+0000", 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 173.33693834799999, 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57018889800000006, 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917268+0000", 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917268+0000", 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917269+0000", 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1480000000000001e-06 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917322+0000", 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3705000000000002e-05 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485165+0000", 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.56784279500000001 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485171+0000", 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0639999999999999e-06 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485221+0000", 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9313e-05 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485242+0000", 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1049000000000001e-05 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.487416+0000", 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021741849999999999 2024-09-17T16:26:13.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.487457+0000", 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0639000000000002e-05 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:114282 5.d e489/482 5:bfd83ddc:::benchmark_data_smithi173_90735_object31511:head v 489'7194, mlcod=489'7192)", 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:53.959997+0000", 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 139.29420873699999, 2024-09-17T16:26:13.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33615526200000001, 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.959997+0000", 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.959997+0000", 2024-09-17T16:26:13.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.959999+0000", 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.61e-06 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.960006+0000", 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3170000000000004e-06 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292652+0000", 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33264596400000002 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292658+0000", 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9859999999999999e-06 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292772+0000", 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011348799999999999 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292793+0000", 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1630000000000001e-05 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.296107+0000", 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003314141 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.296152+0000", 2024-09-17T16:26:13.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5126000000000001e-05 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:158720 5.f e512/481 5:f5ceb914:::benchmark_data_smithi173_90735_object75949:head v 512'9748, mlcod=512'9745)", 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 94.396211124000004, 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.233160072, 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.337 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857996+0000", 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7399999999999991e-07 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.858002+0000", 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5030000000000002e-06 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087206+0000", 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22920367999999999 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087213+0000", 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0129999999999996e-06 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087267+0000", 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3978999999999999e-05 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087289+0000", 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1597999999999999e-05 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.091102+0000", 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038130719999999998 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.091155+0000", 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3253e-05 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:160454 5.17 e514/470 5:eb54973c:::benchmark_data_smithi173_90735_object77683:head v 514'6398, mlcod=514'6395)", 2024-09-17T16:26:13.341 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 92.523571415000006, 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18539098100000001, 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2900000000000002e-07 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730643+0000", 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3300000000000001e-06 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913880+0000", 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.183237224 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913887+0000", 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2400000000000001e-06 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913943+0000", 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5859999999999997e-05 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913958+0000", 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4854999999999999e-05 2024-09-17T16:26:13.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.915994+0000", 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00203572 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.916026+0000", 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1822999999999997e-05 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:12912 6.d e535/527 6:ba9121f7:::benchmark_data_smithi173_102017_object12911:head v 535'804, mlcod=535'801)", 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 55.550375926000001, 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27481599400000001, 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703831+0000", 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3340000000000001e-06 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703859+0000", 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7898e-05 2024-09-17T16:26:13.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977854+0000", 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27399516899999998 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977863+0000", 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6309999999999992e-06 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977932+0000", 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.923e-05 2024-09-17T16:26:13.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977956+0000", 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3606000000000001e-05 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.978613+0000", 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065716699999999995 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.978646+0000", 2024-09-17T16:26:13.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2959000000000001e-05 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:13282 6.d e535/527 6:b884e4f4:::benchmark_data_smithi173_102017_object13281:head v 535'823, mlcod=535'820)", 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:18.148010+0000", 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 55.106195501000002, 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.174234948, 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:13.350 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148010+0000", 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148010+0000", 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148011+0000", 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9699999999999994e-07 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148035+0000", 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3946000000000001e-05 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321530+0000", 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17349503999999999 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321535+0000", 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9930000000000003e-06 2024-09-17T16:26:13.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321583+0000", 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7472000000000002e-05 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321598+0000", 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5305e-05 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.322200+0000", 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00060203700000000004 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.322245+0000", 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5157999999999997e-05 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:26:13.354 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:26:13.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:13.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:13 smithi017 ceph-mon[27571]: pgmap v1220: 17 pgs: 17 active+clean; 4.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-17T16:26:13.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:13.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:13.407 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:26:13.484 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:13.510 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:13.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:13.542 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 70429 70413 68.7566 65.375 0.0240807 0.0145398 2024-09-17T16:26:13.611 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:13.632 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:13.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:13.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:13.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:13.762 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:13.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:13.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:13.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:13.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:14.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:14.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:14.123 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:14.160 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:14.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:14.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:26:14.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:26:14.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:26:14.261 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:14.273 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:26:14.273 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:26:14.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:14.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:14.393 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:26:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:26:14.420 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:14.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:14.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:14.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 71681 71665 68.9025 78.25 0.00656211 0.0145081 2024-09-17T16:26:14.551 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:14.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:14.623 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:26:14.653 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:14.689 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:14.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:14.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-09-17T16:26:14.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:14.843 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:14.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:14.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:14.971 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:14.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:15.055 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:26:15.055 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:26:15.072 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:15.099 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:26:15.100 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:26:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:15.125 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:15.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:15.140 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:26:15.141 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:26:15.169 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:26:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:26:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:72385 6.5 6:ab38378c:::benchmark_data_smithi173_102017_object72384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-17T16:26:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:15.161184+0000", 2024-09-17T16:26:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0049454590000000001, 2024-09-17T16:26:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.004966837, 2024-09-17T16:26:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:26:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:26:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:26:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:26:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:26:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:26:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 72385 2024-09-17T16:26:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:26:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:26:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161184+0000", 2024-09-17T16:26:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:26:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:26:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161184+0000", 2024-09-17T16:26:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161185+0000", 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2729999999999999e-06 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161263+0000", 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7811000000000004e-05 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161277+0000", 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4418e-05 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161286+0000", 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2490000000000001e-06 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:26:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161337+0000", 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1251000000000001e-05 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161418+0000", 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1705999999999993e-05 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,9", 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161501+0000", 2024-09-17T16:26:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2114999999999996e-05 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164117+0000", 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0026166679999999999 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.165156+0000", 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010385189999999999 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:26:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:26:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:26:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:72386 6.5 6:aedb8103:::benchmark_data_smithi173_102017_object72385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-17T16:26:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:15.161608+0000", 2024-09-17T16:26:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0045212280000000004, 2024-09-17T16:26:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0045893069999999999, 2024-09-17T16:26:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 72386 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:26:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161608+0000", 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161608+0000", 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161609+0000", 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1699999999999997e-07 2024-09-17T16:26:15.178 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161695+0000", 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6112000000000005e-05 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161708+0000", 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2952999999999999e-05 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.179 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.161716+0000", 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6020000000000002e-06 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.162962+0000", 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012461600000000001 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:26:15.180 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.163045+0000", 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3060999999999996e-05 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,9", 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.163134+0000", 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8976999999999999e-05 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164249+0000", 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011155710000000001 2024-09-17T16:26:15.181 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.165308+0000", 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010585950000000001 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:72391 6.3 6:cd4a3b37:::benchmark_data_smithi173_102017_object72390:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:15.164628+0000", 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0015008059999999999, 2024-09-17T16:26:15.182 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001607724, 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 72391 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.183 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164628+0000", 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164628+0000", 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164630+0000", 2024-09-17T16:26:15.184 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.167e-06 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164672+0000", 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2027999999999998e-05 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164684+0000", 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2748999999999999e-05 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.185 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164693+0000", 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5709999999999998e-06 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164731+0000", 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7703000000000001e-05 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:26:15.186 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164822+0000", 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1498999999999999e-05 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,8", 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:15.164905+0000", 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2878000000000002e-05 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 3 2024-09-17T16:26:15.187 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:26:15.188 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:26:15.188 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:26:15.196 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:15.211 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:15.217 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:26:15.217 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:26:15.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:15 smithi017 ceph-mon[27571]: pgmap v1221: 17 pgs: 17 active+clean; 4.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-17T16:26:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:15 smithi017 ceph-mon[27571]: osdmap e560: 12 total, 12 up, 11 in 2024-09-17T16:26:15.258 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:26:15.258 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:26:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:15.290 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:15.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:15.295 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:26:15.296 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:26:15.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:15 smithi071 ceph-mon[30002]: pgmap v1221: 17 pgs: 17 active+clean; 4.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-17T16:26:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:15 smithi071 ceph-mon[30002]: osdmap e560: 12 total, 12 up, 11 in 2024-09-17T16:26:15.329 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:26:15.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:26:15.365 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:26:15.366 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:26:15.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:15.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:15 smithi173 ceph-mon[30530]: pgmap v1221: 17 pgs: 17 active+clean; 4.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-17T16:26:15.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:15 smithi173 ceph-mon[30530]: osdmap e560: 12 total, 12 up, 11 in 2024-09-17T16:26:15.404 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:26:15.405 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:26:15.429 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:15.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:15.440 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:26:15.440 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:26:15.474 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:26:15.474 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:26:15.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:15.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 72836 72820 68.9522 72.1875 0.00702647 0.0144966 2024-09-17T16:26:15.560 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:15.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:15.662 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:15.696 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:15.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:15.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:15.798 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:15.832 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:15.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:15.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:15.905 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:26:15.934 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:15.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:15.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:16.059 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:16.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3170360297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:16.099 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:16.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:16.146 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3170360297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:16.201 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:16.204 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:26:16.229 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:16.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:16.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:16.367 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:16.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3170360297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:16.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:16.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:16.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:16.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 74063 74047 69.0676 76.6875 0.0099011 0.014472 2024-09-17T16:26:16.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:16.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:16.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:16.619 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:16.720 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:16.730 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:26:16.731 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:26:16.749 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:16.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:16.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:16.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:16.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:16.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:16.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:17.017 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:17.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:17.027 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:26:17.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:17.143 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:17.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:17.244 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:17.280 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:17.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:17 smithi071 ceph-mon[30002]: pgmap v1223: 17 pgs: 17 active+clean; 4.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-17T16:26:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3156196332' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:17 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:17.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:17.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:17 smithi173 ceph-mon[30530]: pgmap v1223: 17 pgs: 17 active+clean; 4.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-17T16:26:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3156196332' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:17 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:17.396 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:17 smithi017 ceph-mon[27571]: pgmap v1223: 17 pgs: 17 active+clean; 4.2 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-17T16:26:17.396 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3156196332' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:17.396 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:17 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:17.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:17.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:17.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:17.440 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:26:17.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:17.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 75402 75386 69.2825 83.6875 0.00866453 0.0144291 2024-09-17T16:26:17.547 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:17.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:17.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:17.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:17.678 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:17.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:17.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:17.754 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:26:17.780 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:17.813 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:17.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:17.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:17.942 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:17.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:18.044 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:18.072 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:18.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:18.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:18.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:18.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:18.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:18 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:18.212 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:18 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:26:18.212 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:18 smithi071 ceph-mon[30002]: osdmap e561: 12 total, 12 up, 11 in 2024-09-17T16:26:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:18 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:18 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:26:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:18 smithi017 ceph-mon[27571]: osdmap e561: 12 total, 12 up, 11 in 2024-09-17T16:26:18.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:18.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:18.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:18.326 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:26:18.326 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:26:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:18 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:18 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:26:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:18 smithi173 ceph-mon[30530]: osdmap e561: 12 total, 12 up, 11 in 2024-09-17T16:26:18.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:18.452 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:18.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:18.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 76519 76503 69.2901 69.8125 0.00983908 0.0144234 2024-09-17T16:26:18.555 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:18.579 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:18.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:18.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:18.645 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-17T16:26:18.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:18.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:18.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:18.703 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:18.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:18.838 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:18.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:18.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:18.957 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:26:18.963 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:18.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:19.023 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:19.065 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:19.104 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:19.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:19.180 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:26:19.205 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:19.227 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:19.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:19 smithi071 ceph-mon[30002]: pgmap v1225: 17 pgs: 17 active+clean; 4.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-17T16:26:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:19 smithi071 ceph-mon[30002]: 6.6 scrub starts 2024-09-17T16:26:19.328 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:19.366 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:19.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:19.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:19 smithi173 ceph-mon[30530]: pgmap v1225: 17 pgs: 17 active+clean; 4.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-17T16:26:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:19 smithi173 ceph-mon[30530]: 6.6 scrub starts 2024-09-17T16:26:19.468 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:19.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:19 smithi017 ceph-mon[27571]: pgmap v1225: 17 pgs: 17 active+clean; 4.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-17T16:26:19.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:19 smithi017 ceph-mon[27571]: 6.6 scrub starts 2024-09-17T16:26:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:19.503 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:19.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:19.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 77421 77405 69.1055 56.375 0.0393811 0.0144644 2024-09-17T16:26:19.603 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:19.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:19.628 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:19.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:19.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:19.758 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:19.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:19.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:19.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:19.889 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:19.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:19.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:26:19.991 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:20.014 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:20.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:20.117 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:20.139 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:20.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:20.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:20.266 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:20.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:20.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:20 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:26:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:20 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:26:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:20 smithi071 ceph-mon[30002]: osdmap e562: 12 total, 12 up, 11 in 2024-09-17T16:26:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:20 smithi071 ceph-mon[30002]: 6.6 scrub ok 2024-09-17T16:26:20.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:20.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:20 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:26:20.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:20 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:26:20.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:20 smithi017 ceph-mon[27571]: osdmap e562: 12 total, 12 up, 11 in 2024-09-17T16:26:20.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:20 smithi017 ceph-mon[27571]: 6.6 scrub ok 2024-09-17T16:26:20.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:20.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:20.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:20.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:20 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:26:20.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:20 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:26:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:20 smithi173 ceph-mon[30530]: osdmap e562: 12 total, 12 up, 11 in 2024-09-17T16:26:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:20 smithi173 ceph-mon[30530]: 6.6 scrub ok 2024-09-17T16:26:20.494 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:20.516 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:20.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:20.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 78422 78406 69.0133 62.5625 0.0110021 0.0144806 2024-09-17T16:26:20.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:20.641 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:20.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:20.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:20.767 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:20.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:20.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:20.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:20.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:20.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:20.997 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:21.021 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:21.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:21.122 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:21.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:21.146 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:21.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:21.248 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:21.271 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:21.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:21.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:21.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:21 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:26:21.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:21 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:26:21.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:21 smithi017 ceph-mon[27571]: pgmap v1227: 17 pgs: 17 active+clean; 4.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-17T16:26:21.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:21 smithi017 ceph-mon[27571]: osdmap e563: 12 total, 12 up, 11 in 2024-09-17T16:26:21.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:21 smithi017 ceph-mon[27571]: 6.c scrub starts 2024-09-17T16:26:21.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:21.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:21 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:26:21.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:21 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:26:21.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:21 smithi173 ceph-mon[30530]: pgmap v1227: 17 pgs: 17 active+clean; 4.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-17T16:26:21.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:21 smithi173 ceph-mon[30530]: osdmap e563: 12 total, 12 up, 11 in 2024-09-17T16:26:21.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:21 smithi173 ceph-mon[30530]: 6.c scrub starts 2024-09-17T16:26:21.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:21.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:21.499 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:21.527 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:21.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:21.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 79258 79242 68.7804 52.25 0.0240002 0.0145324 2024-09-17T16:26:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:21 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:26:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:21 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:26:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:21 smithi071 ceph-mon[30002]: pgmap v1227: 17 pgs: 17 active+clean; 4.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-17T16:26:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:21 smithi071 ceph-mon[30002]: osdmap e563: 12 total, 12 up, 11 in 2024-09-17T16:26:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:21 smithi071 ceph-mon[30002]: 6.c scrub starts 2024-09-17T16:26:21.629 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:21.654 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:21.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:21.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:21.779 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:21.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:21.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:21.881 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:21.904 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:21.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:22.006 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:22.031 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:22.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:22.133 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:22.193 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:22.195 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:22.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:22.209 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:22.214 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:22.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:22.323 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:22.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:22.426 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:22.448 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:22.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:22.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:22.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 80188 80172 68.6344 58.125 0.0148131 0.014564 2024-09-17T16:26:22.550 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:22.579 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:22.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:22.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:22.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:22.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:22.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:22.806 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:22.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:22.831 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:22.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:22.859 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:26:22.859 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:26:22.899 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:26:22.900 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:26:22.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:22.937 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:26:22.938 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:26:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:22.973 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:22.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:22.979 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:26:22.979 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:26:23.011 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:26:23.011 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:26:23.047 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:26:23.048 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:26:23.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:23.088 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:26:23.088 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:26:23.097 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:23.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:23.127 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:26:23.127 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:26:23.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:26:23.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:26:23.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:23 smithi017 ceph-mon[27571]: 6.f scrub starts 2024-09-17T16:26:23.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:23 smithi017 ceph-mon[27571]: pgmap v1229: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-09-17T16:26:23.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:23 smithi017 ceph-mon[27571]: 6.5 scrub starts 2024-09-17T16:26:23.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:23 smithi017 ceph-mon[27571]: 6.c scrub ok 2024-09-17T16:26:23.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3914117387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:23.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:23.200 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:26:23.200 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:26:23.227 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:23.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:23.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:23.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:23.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:23 smithi173 ceph-mon[30530]: 6.f scrub starts 2024-09-17T16:26:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:23 smithi173 ceph-mon[30530]: pgmap v1229: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-09-17T16:26:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:23 smithi173 ceph-mon[30530]: 6.5 scrub starts 2024-09-17T16:26:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:23 smithi173 ceph-mon[30530]: 6.c scrub ok 2024-09-17T16:26:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3914117387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:23.432 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:26:23.433 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:26:23.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:23.481 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:23.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:23.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 81213 81197 68.5726 64.0625 0.00477301 0.0145766 2024-09-17T16:26:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:23 smithi071 ceph-mon[30002]: 6.f scrub starts 2024-09-17T16:26:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:23 smithi071 ceph-mon[30002]: pgmap v1229: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-09-17T16:26:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:23 smithi071 ceph-mon[30002]: 6.5 scrub starts 2024-09-17T16:26:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:23 smithi071 ceph-mon[30002]: 6.c scrub ok 2024-09-17T16:26:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3914117387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:23.582 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:23.603 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:81227 6.0 6:048fa960:::benchmark_data_smithi173_102017_object81226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e563)", 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:23.562988+0000", 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.038484995000000001, 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.038511113, 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:23.604 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 81227 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.562988+0000", 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.562988+0000", 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:23.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.562989+0000", 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7430000000000001e-06 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.563061+0000", 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1229000000000005e-05 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:23.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.563077+0000", 2024-09-17T16:26:23.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6697e-05 2024-09-17T16:26:23.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:23.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.563086+0000", 2024-09-17T16:26:23.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6009999999999995e-06 2024-09-17T16:26:23.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.563155+0000", 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8631000000000001e-05 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.563308+0000", 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015388899999999999 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.563474+0000", 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016568799999999999 2024-09-17T16:26:23.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.566305+0000", 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028310589999999999 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:23.567011+0000", 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070554999999999999 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:23.609 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:26:23.610 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 1 2024-09-17T16:26:23.610 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:26:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:23.611 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:23.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:23.665 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:26:23.665 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:26:23.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:23.737 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:23.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:23.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:23.840 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:23.872 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:23.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:23.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:24.000 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:24.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:24.103 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:24.127 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:24.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:24.229 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:24.268 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:24.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:24.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:24.370 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:24 smithi173 ceph-mon[30530]: 6.f scrub ok 2024-09-17T16:26:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:24 smithi173 ceph-mon[30530]: 6.5 scrub ok 2024-09-17T16:26:24.403 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:24.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:24.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:24 smithi017 ceph-mon[27571]: 6.f scrub ok 2024-09-17T16:26:24.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:24 smithi017 ceph-mon[27571]: 6.5 scrub ok 2024-09-17T16:26:24.505 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:24.537 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:24.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:24.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 82235 82219 68.5099 63.875 0.0169339 0.0145893 2024-09-17T16:26:24.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:24 smithi071 ceph-mon[30002]: 6.f scrub ok 2024-09-17T16:26:24.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:24 smithi071 ceph-mon[30002]: 6.5 scrub ok 2024-09-17T16:26:24.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:24.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:24.666 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:24.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:24.790 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:24.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:24.892 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:24.919 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:24.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:24.971 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:26:25.021 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:25.047 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:25.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:25.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:25.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:25.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:25.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:25 smithi017 ceph-mon[27571]: pgmap v1230: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 54 MiB/s wr, 868 op/s 2024-09-17T16:26:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1214793047' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:25 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:26:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:25 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:26:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1214793047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:26:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:25 smithi017 ceph-mon[27571]: osdmap e564: 12 total, 12 up, 11 in 2024-09-17T16:26:25.275 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:25.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:25.306 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:25.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:25 smithi173 ceph-mon[30530]: pgmap v1230: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 54 MiB/s wr, 868 op/s 2024-09-17T16:26:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1214793047' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:25 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:26:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:25 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:26:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1214793047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:26:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:25 smithi173 ceph-mon[30530]: osdmap e564: 12 total, 12 up, 11 in 2024-09-17T16:26:25.408 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:25.437 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:25.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:25.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:25.539 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:25.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 83274 83258 68.4628 64.9375 0.00666256 0.0146015 2024-09-17T16:26:25.563 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:25.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:25 smithi071 ceph-mon[30002]: pgmap v1230: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 54 MiB/s wr, 868 op/s 2024-09-17T16:26:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1214793047' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:25.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:25 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:26:25.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:25 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:26:25.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1214793047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:26:25.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:25 smithi071 ceph-mon[30002]: osdmap e564: 12 total, 12 up, 11 in 2024-09-17T16:26:25.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:25.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:25.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:25.745 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:26:25.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:25.824 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:25.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:25.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:25.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:25.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:25.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:26.051 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:26.084 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:26.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:26.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:26.210 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:26.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:26.312 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:26.317 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:26.348 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:26.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:26.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:26 smithi173 ceph-mon[30530]: 6.8 scrub starts 2024-09-17T16:26:26.449 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:26.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:26 smithi017 ceph-mon[27571]: 6.8 scrub starts 2024-09-17T16:26:26.483 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:26.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:26.543 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 84173 84157 68.3033 56.1875 0.0358966 0.0146355 2024-09-17T16:26:26.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:26 smithi071 ceph-mon[30002]: 6.8 scrub starts 2024-09-17T16:26:26.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:26.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:26.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:26.714 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:26.746 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:26.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:26.848 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:26.874 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:26.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:26.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:26:26.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:26.997 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:26.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:27.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:27.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:27.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:27.185 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:26:27.186 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:26:27.186 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-17T16:26:27.186 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd in 5 2024-09-17T16:26:27.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:27 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:26:27.225 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:27 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:26:27.225 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:27 smithi017 ceph-mon[27571]: pgmap v1232: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 58 MiB/s wr, 932 op/s 2024-09-17T16:26:27.225 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:27 smithi017 ceph-mon[27571]: osdmap e565: 12 total, 12 up, 11 in 2024-09-17T16:26:27.225 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:27.271 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:27.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:27.372 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:27 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:26:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:27 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:26:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:27 smithi173 ceph-mon[30530]: pgmap v1232: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 58 MiB/s wr, 932 op/s 2024-09-17T16:26:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:27 smithi173 ceph-mon[30530]: osdmap e565: 12 total, 12 up, 11 in 2024-09-17T16:26:27.396 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:27.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:27.498 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:27.537 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:27.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:27.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 85143 85127 68.2048 60.625 0.0131179 0.0146558 2024-09-17T16:26:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:27 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:26:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:27 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:26:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:27 smithi071 ceph-mon[30002]: pgmap v1232: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 58 MiB/s wr, 932 op/s 2024-09-17T16:26:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:27 smithi071 ceph-mon[30002]: osdmap e565: 12 total, 12 up, 11 in 2024-09-17T16:26:27.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:27.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:27.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:27.769 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:27.797 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:27.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:27.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:27.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:27.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:28.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:28.045 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:28.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:28.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:28.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:28.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:28.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:28.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:26:28.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:28.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:28.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:28.300 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:28.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:28 smithi173 ceph-mon[30530]: 6.8 scrub ok 2024-09-17T16:26:28.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:28.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:28 smithi017 ceph-mon[27571]: 6.8 scrub ok 2024-09-17T16:26:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:28.430 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:28.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:28.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:28.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 86040 86024 68.0511 56.0625 0.00759271 0.0146888 2024-09-17T16:26:28.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:28 smithi071 ceph-mon[30002]: 6.8 scrub ok 2024-09-17T16:26:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:28.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:28.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:28.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:28.695 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:28.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:28.797 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:28.822 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:26:28.822 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:26:28.823 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:26:28.823 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:26:28.823 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:26:28.826 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:28.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:28.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:28.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:28.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:29.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:29.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:29.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:29.181 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:29.207 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:29.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:29.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:29.309 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:29.333 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:29.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:29.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:29 smithi173 ceph-mon[30530]: pgmap v1234: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:26:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:29 smithi173 ceph-mon[30530]: 6.4 scrub starts 2024-09-17T16:26:29.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:29.476 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:29.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:29 smithi017 ceph-mon[27571]: pgmap v1234: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:26:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:29 smithi017 ceph-mon[27571]: 6.4 scrub starts 2024-09-17T16:26:29.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:26:29.542907+0000 min lat: 0.00108766 max lat: 2.21118 avg lat: 0.0147387 2024-09-17T16:26:29.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:26:29.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 86838 86822 67.8238 49.875 0.0111964 0.0147387 2024-09-17T16:26:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:29 smithi071 ceph-mon[30002]: pgmap v1234: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:26:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:29 smithi071 ceph-mon[30002]: 6.4 scrub starts 2024-09-17T16:26:29.577 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:29.601 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:29.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:29.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:29.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:29.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:29.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:29.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:29.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:29.958 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:29.982 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:29.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:30.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:30.115 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:30.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:30.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:30.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:30.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:30.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:30.342 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:30.363 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:30.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:30.465 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:30.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:30.492 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:30.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:30.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 87705 87689 67.6554 54.1875 0.00966561 0.0147731 2024-09-17T16:26:30.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:30.618 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:30.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:30.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:30.754 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:30.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:30 smithi071 ceph-mon[30002]: 6.3 scrub starts 2024-09-17T16:26:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:30 smithi071 ceph-mon[30002]: 6.3 scrub ok 2024-09-17T16:26:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:30 smithi071 ceph-mon[30002]: osdmap e566: 12 total, 12 up, 11 in 2024-09-17T16:26:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:30 smithi071 ceph-mon[30002]: pgmap v1236: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 60 MiB/s wr, 964 op/s 2024-09-17T16:26:30.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:30.888 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:30.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:30.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:30 smithi173 ceph-mon[30530]: 6.3 scrub starts 2024-09-17T16:26:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:30 smithi173 ceph-mon[30530]: 6.3 scrub ok 2024-09-17T16:26:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:30 smithi173 ceph-mon[30530]: osdmap e566: 12 total, 12 up, 11 in 2024-09-17T16:26:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:30 smithi173 ceph-mon[30530]: pgmap v1236: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 60 MiB/s wr, 964 op/s 2024-09-17T16:26:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:30 smithi017 ceph-mon[27571]: 6.3 scrub starts 2024-09-17T16:26:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:30 smithi017 ceph-mon[27571]: 6.3 scrub ok 2024-09-17T16:26:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:30 smithi017 ceph-mon[27571]: osdmap e566: 12 total, 12 up, 11 in 2024-09-17T16:26:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:30 smithi017 ceph-mon[27571]: pgmap v1236: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 60 MiB/s wr, 964 op/s 2024-09-17T16:26:30.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:31.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:31.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:31.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:31.142 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:31.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:31.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:31.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:31.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:31.371 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:31.394 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:31.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:31.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:31.496 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:31.525 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:31.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:31.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 88645 88629 67.5468 58.75 0.0110428 0.0147965 2024-09-17T16:26:31.627 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:31.659 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:31.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:31.761 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:31.786 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:31.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:31 smithi071 ceph-mon[30002]: 6.4 scrub ok 2024-09-17T16:26:31.889 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:31.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:31 smithi173 ceph-mon[30530]: 6.4 scrub ok 2024-09-17T16:26:31.920 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:31.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:31.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:31 smithi017 ceph-mon[27571]: 6.4 scrub ok 2024-09-17T16:26:32.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:32.045 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:32.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:32.140 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:26:32.147 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:32.158 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:32.166 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:32.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:32.212 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:32.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:32.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:32.352 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:32.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:32.454 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:32.495 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:32.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:32.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 89683 89667 67.5145 64.875 0.0181266 0.0148062 2024-09-17T16:26:32.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:32 smithi017 ceph-mon[27571]: 6.e scrub starts 2024-09-17T16:26:32.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:32 smithi017 ceph-mon[27571]: pgmap v1237: 17 pgs: 17 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-17T16:26:32.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:32.619 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:32.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:32.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:32.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:32.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:32 smithi071 ceph-mon[30002]: 6.e scrub starts 2024-09-17T16:26:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:32 smithi071 ceph-mon[30002]: pgmap v1237: 17 pgs: 17 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-17T16:26:32.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:32.884 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:32.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:32.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:32 smithi173 ceph-mon[30530]: 6.e scrub starts 2024-09-17T16:26:32.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:32 smithi173 ceph-mon[30530]: pgmap v1237: 17 pgs: 17 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-17T16:26:32.986 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:33.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:33.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:33.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:33.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:33.141 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:33.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:33.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:33.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:33.265 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:33.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:33.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:33.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:33.389 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:33.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:33.491 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:33.520 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:33.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:33.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:33.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 90572 90556 67.3722 55.5625 0.00327098 0.0148362 2024-09-17T16:26:33.622 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:33.646 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:33.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:33.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:33.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:33.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:33 smithi071 ceph-mon[30002]: 6.e scrub ok 2024-09-17T16:26:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2964600493' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-17T16:26:33.870 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:33.891 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:33.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:33 smithi173 ceph-mon[30530]: 6.e scrub ok 2024-09-17T16:26:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2964600493' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-17T16:26:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:33 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:26:33.489+0000 7f086a07d640 -1 mon.a@0(leader).osd e566 definitely_dead 0 2024-09-17T16:26:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:33 smithi017 ceph-mon[27571]: 6.e scrub ok 2024-09-17T16:26:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2964600493' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-17T16:26:33.992 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:34.013 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:34.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:34.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:34.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:34.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:34.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:34.260 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:34.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:34.280 INFO:teuthology.orchestra.run.smithi017.stderr:marked in osd.5. 2024-09-17T16:26:34.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:34.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:34.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:34.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:34.508 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:34.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:34.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 90631 90615 66.6229 3.6875 0.00340142 0.0148817 2024-09-17T16:26:34.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:34.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:34.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:34.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:34.755 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:34.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:34.857 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:34.881 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:34.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2964600493' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-17T16:26:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:34 smithi173 ceph-mon[30530]: osdmap e567: 12 total, 12 up, 12 in 2024-09-17T16:26:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:34 smithi173 ceph-mon[30530]: pgmap v1239: 17 pgs: 17 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-17T16:26:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:34 smithi173 ceph-mon[30530]: osdmap e568: 12 total, 12 up, 12 in 2024-09-17T16:26:34.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2964600493' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-17T16:26:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:34 smithi017 ceph-mon[27571]: osdmap e567: 12 total, 12 up, 12 in 2024-09-17T16:26:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:34 smithi017 ceph-mon[27571]: pgmap v1239: 17 pgs: 17 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-17T16:26:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:34 smithi017 ceph-mon[27571]: osdmap e568: 12 total, 12 up, 12 in 2024-09-17T16:26:35.004 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:35.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:35.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2964600493' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-17T16:26:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:34 smithi071 ceph-mon[30002]: osdmap e567: 12 total, 12 up, 12 in 2024-09-17T16:26:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:34 smithi071 ceph-mon[30002]: pgmap v1239: 17 pgs: 17 active+clean; 5.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-17T16:26:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:34 smithi071 ceph-mon[30002]: osdmap e568: 12 total, 12 up, 12 in 2024-09-17T16:26:35.106 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:35.128 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:35.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:35.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:35.254 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:35.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:35.356 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:35.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:35.380 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:35.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:35.481 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:35.504 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:35.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:35.524 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:26:35.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 90631 90615 65.8482 0 - 0.0148817 2024-09-17T16:26:35.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:35.632 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:35.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:35.734 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:35.755 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:35.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:35.857 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:35.879 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:35.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:35 smithi173 ceph-mon[30530]: 6.b scrub starts 2024-09-17T16:26:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3956081500' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:26:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:35 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3956081500' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:26:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:35 smithi173 ceph-mon[30530]: osdmap e569: 12 total, 12 up, 12 in 2024-09-17T16:26:35.980 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:35 smithi017 ceph-mon[27571]: 6.b scrub starts 2024-09-17T16:26:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3956081500' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:26:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:35 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3956081500' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:26:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:35 smithi017 ceph-mon[27571]: osdmap e569: 12 total, 12 up, 12 in 2024-09-17T16:26:36.012 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:36.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:36.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:35 smithi071 ceph-mon[30002]: 6.b scrub starts 2024-09-17T16:26:36.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3956081500' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:26:36.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:35 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:36.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3956081500' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:26:36.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:35 smithi071 ceph-mon[30002]: osdmap e569: 12 total, 12 up, 12 in 2024-09-17T16:26:36.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:36.140 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:36.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:36.150 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-17T16:26:36.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:36.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:36.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:36.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:36.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:36.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:36.509 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:36.535 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:36.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:36.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 91047 91031 65.3901 13 0.0045025 0.0152867 2024-09-17T16:26:36.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:36.663 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:36.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:36.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:36 smithi017 ceph-mon[27571]: 6.b scrub ok 2024-09-17T16:26:36.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:36 smithi017 ceph-mon[27571]: pgmap v1242: 17 pgs: 1 remapped+peering, 16 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 748 op/s 2024-09-17T16:26:36.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:36.793 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:36.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:36.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:36 smithi173 ceph-mon[30530]: 6.b scrub ok 2024-09-17T16:26:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:36 smithi173 ceph-mon[30530]: pgmap v1242: 17 pgs: 1 remapped+peering, 16 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 748 op/s 2024-09-17T16:26:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:36.934 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:36.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:37.035 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:37.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:36 smithi071 ceph-mon[30002]: 6.b scrub ok 2024-09-17T16:26:37.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:36 smithi071 ceph-mon[30002]: pgmap v1242: 17 pgs: 1 remapped+peering, 16 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 748 op/s 2024-09-17T16:26:37.074 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:37.076 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:37.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:37.115 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:37.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:37.218 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:37.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:37.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:37.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:37.348 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:37.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:37.450 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:37.492 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:37.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:37.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 91852 91836 65.2187 50.3125 0.0141999 0.0153272 2024-09-17T16:26:37.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:37.620 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:37.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:37.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:37.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:37.748 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:37.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:37.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:37.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:37.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:37.977 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:37.987 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:26:37.987 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:26:37.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:26:37.987 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:124293 3.e 3:74c76eb5:::benchmark_data_smithi173_66272_object52337:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e274)", 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 587.61092108900004, 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.55229465499999997, 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:26:37.988 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 124293 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371915+0000", 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371917+0000", 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1430000000000001e-06 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371923+0000", 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1709999999999999e-06 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371936+0000", 2024-09-17T16:26:37.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2947e-05 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.371951+0000", 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4547e-05 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.372152+0000", 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020128999999999999 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.372168+0000", 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6218999999999998e-05 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.916603+0000", 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54443496499999999 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.919612+0000", 2024-09-17T16:26:37.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030085120000000001 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.919736+0000", 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012435899999999999 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.923884+0000", 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041481460000000001 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.923928+0000", 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4159999999999997e-05 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924005+0000", 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6178000000000001e-05 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924024+0000", 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9318e-05 2024-09-17T16:26:37.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924094+0000", 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9605999999999997e-05 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:16:50.924210+0000", 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116094 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:37.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:132741 3.e 3:729d5b9e:::benchmark_data_smithi173_66272_object60785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 574.350912386, 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43202703199999998, 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 132741 2024-09-17T16:26:37.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631924+0000", 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631925+0000", 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3239999999999999e-06 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631930+0000", 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6700000000000002e-06 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631942+0000", 2024-09-17T16:26:37.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1857e-05 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631947+0000", 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8980000000000001e-06 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.631998+0000", 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0365000000000003e-05 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:37.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.632091+0000", 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3592999999999997e-05 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.632178+0000", 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6648000000000001e-05 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635422+0000", 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003244289 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.636256+0000", 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083370299999999998 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.636486+0000", 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022978699999999999 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063733+0000", 2024-09-17T16:26:38.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42724667900000002 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063826+0000", 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3579999999999995e-05 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063951+0000", 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012463900000000001 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25036.0:132741 3.e e278/273 ondisk, result = 0)", 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 574.34690101599995, 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42802421600000001, 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:26:38.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635935+0000", 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635936+0000", 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.023e-06 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:03.635941+0000", 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5000000000000001e-06 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.060074+0000", 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42413389000000001 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.060083+0000", 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6670000000000002e-06 2024-09-17T16:26:38.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063717+0000", 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036338640000000001 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063726+0000", 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.789e-06 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:04.063959+0000", 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023348299999999999 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:135118 3.e 3:7ce4ae2e:::benchmark_data_smithi173_66272_object63162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 570.71242765299996, 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34112115999999998, 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:38.007 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 135118 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270408+0000", 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270411+0000", 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3520000000000001e-06 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270420+0000", 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:26:38.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270443+0000", 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2789000000000002e-05 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270453+0000", 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0322e-05 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270692+0000", 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023867100000000001 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.270833+0000", 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014147500000000001 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.271004+0000", 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017102299999999999 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279153+0000", 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081486939999999997 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279218+0000", 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5320999999999993e-05 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.279235+0000", 2024-09-17T16:26:38.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6402999999999999e-05 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611337+0000", 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33210225300000001 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611397+0000", 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9757999999999999e-05 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:07.611530+0000", 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013292299999999999 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:136353 3.e 3:75c6ba64:::benchmark_data_smithi173_66272_object64397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 568.03404344, 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34635525499999997, 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:38.014 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 136353 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948793+0000", 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948795+0000", 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4269999999999998e-06 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948804+0000", 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948823+0000", 2024-09-17T16:26:38.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8892000000000001e-05 2024-09-17T16:26:38.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948832+0000", 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0629999999999995e-06 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.948903+0000", 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0796000000000005e-05 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.949058+0000", 2024-09-17T16:26:38.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015499099999999999 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.949219+0000", 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016127900000000001 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953488+0000", 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042689440000000002 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953702+0000", 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021406499999999999 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.953721+0000", 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8936999999999998e-05 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294938+0000", 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.341217465 2024-09-17T16:26:38.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295013+0000", 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4389999999999998e-05 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295148+0000", 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000135148 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.25036.0:136353 3.e e284/273 ondisk, result = 0)", 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 568.03037112100003, 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.342690457, 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952465+0000", 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952466+0000", 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0389999999999999e-06 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:09.952472+0000", 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6910000000000002e-06 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294851+0000", 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34237905899999999 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294856+0000", 2024-09-17T16:26:38.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3499999999999996e-06 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294916+0000", 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9580000000000002e-05 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.294928+0000", 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2571999999999999e-05 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:10.295155+0000", 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000227166 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.25036.0:143899 3.1 e290/272 3:8d15b72a:::benchmark_data_smithi173_66272_object71943:head v 290'9096, mlcod=261'5689)", 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 557.50371654000003, 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37496784399999999, 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.026 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479120+0000", 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9699999999999996e-07 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.479128+0000", 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850526+0000", 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37139821699999997 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850534+0000", 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1760000000000004e-06 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850583+0000", 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9013e-05 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.850601+0000", 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8916e-05 2024-09-17T16:26:38.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.854050+0000", 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034480639999999998 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.854087+0000", 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7846000000000003e-05 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25036.0:143908 3.e 3:70cae513:::benchmark_data_smithi173_66272_object71952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 557.49366537699996, 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35428895300000002, 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25036", 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/3308646636", 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 143908 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489171+0000", 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489172+0000", 2024-09-17T16:26:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6700000000000001e-06 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489177+0000", 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.651e-06 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489191+0000", 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3497e-05 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489197+0000", 2024-09-17T16:26:38.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0719999999999996e-06 2024-09-17T16:26:38.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489244+0000", 2024-09-17T16:26:38.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7037999999999999e-05 2024-09-17T16:26:38.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489341+0000", 2024-09-17T16:26:38.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7664000000000005e-05 2024-09-17T16:26:38.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,9,11", 2024-09-17T16:26:38.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.489424+0000", 2024-09-17T16:26:38.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2916000000000004e-05 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493180+0000", 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037554379999999998 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.493210+0000", 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0076000000000001e-05 2024-09-17T16:26:38.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.494016+0000", 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080592900000000004 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843310+0000", 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34929427400000002 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843362+0000", 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2549000000000001e-05 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:17:20.843460+0000", 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7178999999999999e-05 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(4.6 383/382 [PushOp(4:63469528:::benchmark_data_smithi173_77474_object91937:head, version: 345'5625, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:63469528:::benchmark_data_smithi173_77474_object91937:head@345'5625, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:26:38.038 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 355.58640497699997, 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29598169099999999, 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396432+0000", 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9999999999999997e-07 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396467+0000", 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5184000000000001e-05 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692000+0000", 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29553331999999999 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692039+0000", 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8491999999999997e-05 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692173+0000", 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134411 2024-09-17T16:26:38.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692186+0000", 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3118e-05 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692413+0000", 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022656600000000001 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.043 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_backfill(progress 4.6 e 383/383 lb 4:6341a44b:::benchmark_data_smithi173_77474_object88529:head)", 2024-09-17T16:26:38.043 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:26:38.043 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 355.58634515099999, 2024-09-17T16:26:38.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29619305299999998, 2024-09-17T16:26:38.043 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.043 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.043 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:26:38.043 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:26:38.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:26:38.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:26:38.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:26:38.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396495+0000", 2024-09-17T16:26:38.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9450000000000003e-06 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692562+0000", 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29606637800000002 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692584+0000", 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2526000000000001e-05 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692645+0000", 2024-09-17T16:26:38.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1309999999999994e-05 2024-09-17T16:26:38.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692657+0000", 2024-09-17T16:26:38.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1798e-05 2024-09-17T16:26:38.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:38.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:38.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:38.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692684+0000", 2024-09-17T16:26:38.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6821e-05 2024-09-17T16:26:38.049 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:82763 5.13 e475/453 5:cd63b9f8:::benchmark_data_smithi173_90735_object82762:head v 475'4007, mlcod=475'4003)", 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 198.06463689700001, 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56469166699999995, 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918200+0000", 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1500000000000004e-07 2024-09-17T16:26:38.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918245+0000", 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5034000000000003e-05 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481583+0000", 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56333854699999997 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481588+0000", 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5800000000000002e-06 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481652+0000", 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4309999999999999e-05 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481669+0000", 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6880000000000001e-05 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.482864+0000", 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011944200000000001 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.482891+0000", 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7180999999999999e-05 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.054 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPull(5.12 e475/455 cost 1049576)", 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 198.023377329, 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54846142799999997, 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959460+0000", 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5099999999999998e-07 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959470+0000", 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9380000000000004e-06 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507615+0000", 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54814523100000001 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507626+0000", 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1484000000000001e-05 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507695+0000", 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8990000000000002e-05 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507920+0000", 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022483400000000001 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(5.d 475/455 [PushOp(5:b9f39261:::benchmark_data_smithi173_90735_object37371:head, version: 439'2383, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:b9f39261:::benchmark_data_smithi173_90735_object37371:head@439'2383, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 197.50245967800001, 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45551481399999999, 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480378+0000", 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6470000000000001e-06 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480434+0000", 2024-09-17T16:26:38.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6162999999999997e-05 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934248+0000", 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45381411300000002 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934260+0000", 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1382e-05 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934362+0000", 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000102681 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934379+0000", 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6498e-05 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.935891+0000", 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015123300000000001 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:95580 5.b e482/419 5:d4bc1ca6:::benchmark_data_smithi173_90735_object12809:head v 482'5935, mlcod=482'5933)", 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 184.250368864, 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39904708700000002, 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:26:38.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732468+0000", 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732477+0000", 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4880000000000007e-06 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128932+0000", 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.396455369 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128940+0000", 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5089999999999996e-06 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128990+0000", 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0068999999999998e-05 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.129012+0000", 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2218e-05 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.131477+0000", 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002464843 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.131514+0000", 2024-09-17T16:26:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7515000000000003e-05 2024-09-17T16:26:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:95584 5.b e482/419 5:dc96c928:::benchmark_data_smithi173_90735_object12813:head v 482'5936, mlcod=482'5933)", 2024-09-17T16:26:38.069 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 184.09872167, 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39023148299999999, 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884116+0000", 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9319999999999999e-06 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884127+0000", 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0198e-05 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272085+0000", 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38795868300000003 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272089+0000", 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1729999999999997e-06 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272129+0000", 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9937000000000001e-05 2024-09-17T16:26:38.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272145+0000", 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5863e-05 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.274307+0000", 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021616040000000001 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.274346+0000", 2024-09-17T16:26:38.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9093e-05 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17628 6.9 6:9546f3b4:::benchmark_data_smithi173_102017_object17627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:21.997751+0000", 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 75.985084868000001, 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49739271200000001, 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:38.074 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17628 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997751+0000", 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997751+0000", 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997757+0000", 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3489999999999999e-06 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997802+0000", 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5543000000000001e-05 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997822+0000", 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9443999999999999e-05 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997831+0000", 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9740000000000008e-06 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998743+0000", 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091201499999999996 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998896+0000", 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015307299999999999 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.999045+0000", 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014901199999999999 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000028+0000", 2024-09-17T16:26:38.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00098290300000000003 2024-09-17T16:26:38.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.450171+0000", 2024-09-17T16:26:38.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45014342099999999 2024-09-17T16:26:38.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.495073+0000", 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.044902035999999999 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.495110+0000", 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7134000000000001e-05 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.495144+0000", 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3807999999999999e-05 2024-09-17T16:26:38.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17643 6.9 6:992c966e:::benchmark_data_smithi173_102017_object17642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.493337+0000", 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 75.489499053000003, 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42163531900000001, 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:26:38.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17643 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493337+0000", 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493337+0000", 2024-09-17T16:26:38.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493339+0000", 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4229999999999999e-06 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493447+0000", 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107039 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493469+0000", 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2408999999999998e-05 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493478+0000", 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6300000000000004e-06 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493543+0000", 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5235999999999997e-05 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493660+0000", 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011726999999999999 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493766+0000", 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010634899999999999 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.494765+0000", 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099869900000000003 2024-09-17T16:26:38.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.494990+0000", 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000224739 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.914894+0000", 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41990391300000002 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.914938+0000", 2024-09-17T16:26:38.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4121e-05 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.914972+0000", 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4490999999999999e-05 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:27649 6.d 6:bbb9887f:::benchmark_data_smithi173_102017_object27648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:32.713773+0000", 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 65.269062868000006, 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29677943299999998, 2024-09-17T16:26:38.087 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 27649 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713773+0000", 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713773+0000", 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713774+0000", 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1310000000000001e-06 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713850+0000", 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5955e-05 2024-09-17T16:26:38.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713862+0000", 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1539000000000001e-05 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713870+0000", 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.695e-06 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713929+0000", 2024-09-17T16:26:38.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9944000000000001e-05 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.714043+0000", 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113013 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,10", 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.714177+0000", 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134401 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.715264+0000", 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010869860000000001 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.717509+0000", 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022450970000000002 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010391+0000", 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29288151099999998 2024-09-17T16:26:38.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010435+0000", 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4580999999999999e-05 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010553+0000", 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011758 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26665.0:27649 6.d e545/539 ondisk, result = 0)", 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:32.715533+0000", 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 65.267303315000007, 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.295026646, 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.715533+0000", 2024-09-17T16:26:38.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.715533+0000", 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.715533+0000", 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.715539+0000", 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7300000000000002e-06 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010314+0000", 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.294775175 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010326+0000", 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2106000000000001e-05 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010375+0000", 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9042000000000002e-05 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010384+0000", 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7579999999999998e-06 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010559+0000", 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017520599999999999 2024-09-17T16:26:38.097 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(6.d 545/539 [PushReplyOp(6:b170362a:::benchmark_data_smithi173_102017_object16956:head)])", 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:33.019918+0000", 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.962917821999994, 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29282309400000001, 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019918+0000", 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019918+0000", 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019919+0000", 2024-09-17T16:26:38.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7100000000000002e-07 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019923+0000", 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9149999999999998e-06 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.312357+0000", 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29243439700000001 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:38.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.312364+0000", 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5250000000000002e-06 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.312419+0000", 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5792999999999999e-05 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.312741+0000", 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032189300000000002 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:38.102 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:38.102 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:26:38.149 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:38.183 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:38.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:38.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:38.307 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:38.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:38.409 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:38.440 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:38.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:38.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:38.544 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 92543 92527 64.9712 43.1875 0.00576392 0.0153857 2024-09-17T16:26:38.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:38.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:38.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:38.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:38.691 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:38.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:38.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:38.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:38.815 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:38.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:38.917 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:38.941 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:38.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:38.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:39.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:39.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:39.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:39.066 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:39.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:26:39.166 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:39.188 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:39.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:38 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:26:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:38 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:26:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:38 smithi017 ceph-mon[27571]: pgmap v1243: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 12 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 479 op/s; 5710/272274 objects degraded (2.097%); 16811/272274 objects misplaced (6.174%); 373 KiB/s, 5 objects/s recovering 2024-09-17T16:26:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:38 smithi017 ceph-mon[27571]: osdmap e570: 12 total, 12 up, 12 in 2024-09-17T16:26:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:38 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 5710/272274 objects degraded (2.097%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:26:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1022278572' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:39.292 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:38 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:26:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:38 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:26:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:38 smithi071 ceph-mon[30002]: pgmap v1243: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 12 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 479 op/s; 5710/272274 objects degraded (2.097%); 16811/272274 objects misplaced (6.174%); 373 KiB/s, 5 objects/s recovering 2024-09-17T16:26:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:38 smithi071 ceph-mon[30002]: osdmap e570: 12 total, 12 up, 12 in 2024-09-17T16:26:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:38 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 5710/272274 objects degraded (2.097%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:26:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1022278572' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:39.333 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:39.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:39.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:38 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:26:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:38 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:26:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:38 smithi173 ceph-mon[30530]: pgmap v1243: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 12 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 479 op/s; 5710/272274 objects degraded (2.097%); 16811/272274 objects misplaced (6.174%); 373 KiB/s, 5 objects/s recovering 2024-09-17T16:26:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:38 smithi173 ceph-mon[30530]: osdmap e570: 12 total, 12 up, 12 in 2024-09-17T16:26:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:38 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 5710/272274 objects degraded (2.097%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:26:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1022278572' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:39.436 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:39.470 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:39.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:39.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 8 93145 93137 64.6728 38.125 0.00872072 0.0154574 2024-09-17T16:26:39.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:39.593 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:39.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:39.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:39.716 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:39.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:39.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:39.840 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:39.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:39.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:39.966 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:39.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:40.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:40.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:40.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:40.290 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:40.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:39 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:26:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:39 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:26:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:39 smithi071 ceph-mon[30002]: osdmap e571: 12 total, 12 up, 12 in 2024-09-17T16:26:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:39 smithi071 ceph-mon[30002]: osdmap e572: 12 total, 12 up, 12 in 2024-09-17T16:26:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:39 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:26:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:39 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:26:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:39 smithi173 ceph-mon[30530]: osdmap e571: 12 total, 12 up, 12 in 2024-09-17T16:26:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:39 smithi173 ceph-mon[30530]: osdmap e572: 12 total, 12 up, 12 in 2024-09-17T16:26:40.415 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:40.437 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:40.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:39 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:26:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:39 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:26:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:39 smithi017 ceph-mon[27571]: osdmap e571: 12 total, 12 up, 12 in 2024-09-17T16:26:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:39 smithi017 ceph-mon[27571]: osdmap e572: 12 total, 12 up, 12 in 2024-09-17T16:26:40.539 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:40.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.715 2024-09-17T16:26:40.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 93145 2024-09-17T16:26:40.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-09-17T16:26:40.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-09-17T16:26:40.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 64.1742 2024-09-17T16:26:40.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 24.8689 2024-09-17T16:26:40.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 137.625 2024-09-17T16:26:40.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0 2024-09-17T16:26:40.545 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1026 2024-09-17T16:26:40.546 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 397.903 2024-09-17T16:26:40.546 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2202 2024-09-17T16:26:40.546 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 0 2024-09-17T16:26:40.546 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0154753 2024-09-17T16:26:40.546 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.0483656 2024-09-17T16:26:40.546 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 2.47323 2024-09-17T16:26:40.546 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00108766 2024-09-17T16:26:40.561 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T16:26:40.562 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:40.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:40.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:40.665 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:40.690 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:40.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:40.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:40.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:40.819 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:40.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:40.921 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:40.953 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:40.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:41.055 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:41.077 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:41.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:41.151 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:26:41.151 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:26:41.152 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T16:26:41.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:26:41.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:41.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:41.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:41.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:41.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:40 smithi071 ceph-mon[30002]: pgmap v1247: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 12 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 534 op/s; 5710/274473 objects degraded (2.080%); 16811/274473 objects misplaced (6.125%); 508 KiB/s, 7 objects/s recovering 2024-09-17T16:26:41.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:41.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:41.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:41.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:40 smithi173 ceph-mon[30530]: pgmap v1247: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 12 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 534 op/s; 5710/274473 objects degraded (2.080%); 16811/274473 objects misplaced (6.125%); 508 KiB/s, 7 objects/s recovering 2024-09-17T16:26:41.429 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:41.452 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:41.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:41.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:40 smithi017 ceph-mon[27571]: pgmap v1247: 17 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 12 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 534 op/s; 5710/274473 objects degraded (2.080%); 16811/274473 objects misplaced (6.125%); 508 KiB/s, 7 objects/s recovering 2024-09-17T16:26:41.552 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:41.585 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:41.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:41.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:41.652 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:26:41.652 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:26:41.653 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:26:41.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:26:41.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:41.711 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:41.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:41.812 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:41.838 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:41.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:41.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:41.969 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:41.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:42.071 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:42.096 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:42.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:42.198 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:42.230 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:42.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:42.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:42.356 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:42.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:42.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:42.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:42.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:42.481 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:42.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:42.582 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:42.611 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:42.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:42.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:42.745 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:42.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:42.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:42.871 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:42.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:42.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:42.973 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:43.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:43.103 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:43.135 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:43.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:43.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:43.237 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:43.263 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:43.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:43.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:42 smithi071 ceph-mon[30002]: pgmap v1248: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 37 MiB/s wr, 653 op/s; 11386/277758 objects degraded (4.099%); 16458/277758 objects misplaced (5.925%); 4.1 MiB/s, 64 objects/s recovering 2024-09-17T16:26:43.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:43.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:43.390 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:42 smithi173 ceph-mon[30530]: pgmap v1248: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 37 MiB/s wr, 653 op/s; 11386/277758 objects degraded (4.099%); 16458/277758 objects misplaced (5.925%); 4.1 MiB/s, 64 objects/s recovering 2024-09-17T16:26:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:42 smithi017 ceph-mon[27571]: pgmap v1248: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 37 MiB/s wr, 653 op/s; 11386/277758 objects degraded (4.099%); 16458/277758 objects misplaced (5.925%); 4.1 MiB/s, 64 objects/s recovering 2024-09-17T16:26:43.490 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:43.519 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:43.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:43.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:43.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:43.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:43.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:43.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:43.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:43.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:43.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:43.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:43.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:43.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:44.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:44.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:44.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:44.161 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:44.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:44.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:44.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:44.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:44.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:44 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:26:44.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:44 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:26:44.386 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:44.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:44 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:26:44.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:44 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:26:44.420 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:44.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:44 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:26:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:44 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:26:44.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:44.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:44.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:44.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:44.685 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:44.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:44.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:44.788 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:44.813 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:44.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:44.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:44.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:44.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:45.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:45.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:45.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:45.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:45.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:45.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:45.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:45 smithi071 ceph-mon[30002]: pgmap v1249: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 25 MiB/s wr, 501 op/s; 11386/277581 objects degraded (4.102%); 16327/277581 objects misplaced (5.882%); 5.1 MiB/s, 80 objects/s recovering 2024-09-17T16:26:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:45 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11386/277581 objects degraded (4.102%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:26:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:45 smithi071 ceph-mon[30002]: osdmap e573: 12 total, 12 up, 12 in 2024-09-17T16:26:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:45 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:26:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:45 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:26:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:45 smithi071 ceph-mon[30002]: osdmap e574: 12 total, 12 up, 12 in 2024-09-17T16:26:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:45 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:26:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:45 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:26:45.330 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:45.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:45 smithi173 ceph-mon[30530]: pgmap v1249: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 25 MiB/s wr, 501 op/s; 11386/277581 objects degraded (4.102%); 16327/277581 objects misplaced (5.882%); 5.1 MiB/s, 80 objects/s recovering 2024-09-17T16:26:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:45 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11386/277581 objects degraded (4.102%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:26:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:45 smithi173 ceph-mon[30530]: osdmap e573: 12 total, 12 up, 12 in 2024-09-17T16:26:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:45 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:26:45.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:45 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:26:45.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:45 smithi173 ceph-mon[30530]: osdmap e574: 12 total, 12 up, 12 in 2024-09-17T16:26:45.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:45 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:26:45.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:45 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:26:45.432 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:45.458 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:45.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:45.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:45 smithi017 ceph-mon[27571]: pgmap v1249: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 25 MiB/s wr, 501 op/s; 11386/277581 objects degraded (4.102%); 16327/277581 objects misplaced (5.882%); 5.1 MiB/s, 80 objects/s recovering 2024-09-17T16:26:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:45 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11386/277581 objects degraded (4.102%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:26:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:45 smithi017 ceph-mon[27571]: osdmap e573: 12 total, 12 up, 12 in 2024-09-17T16:26:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:45 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:26:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:45 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:26:45.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:45 smithi017 ceph-mon[27571]: osdmap e574: 12 total, 12 up, 12 in 2024-09-17T16:26:45.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:45 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:26:45.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:45 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:26:45.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:45.585 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:45.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:45.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:45.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:45.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:45.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:45.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:45.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:45.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:45.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:45.985 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:45.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:46.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:46.117 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:46.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:46.219 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:46.248 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:46.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:46.350 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:46.382 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:46.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:46.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:46.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:46.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:46.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:46.636 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:46.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:46.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:46.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:46.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:46 smithi071 ceph-mon[30002]: osdmap e575: 12 total, 12 up, 12 in 2024-09-17T16:26:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:46 smithi071 ceph-mon[30002]: pgmap v1253: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 17 MiB/s wr, 504 op/s; 11385/275328 objects degraded (4.135%); 16327/275328 objects misplaced (5.930%); 5.1 MiB/s, 80 objects/s recovering 2024-09-17T16:26:46.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:46.888 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:46.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:46.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:46 smithi173 ceph-mon[30530]: osdmap e575: 12 total, 12 up, 12 in 2024-09-17T16:26:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:46 smithi173 ceph-mon[30530]: pgmap v1253: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 17 MiB/s wr, 504 op/s; 11385/275328 objects degraded (4.135%); 16327/275328 objects misplaced (5.930%); 5.1 MiB/s, 80 objects/s recovering 2024-09-17T16:26:46.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:46 smithi017 ceph-mon[27571]: osdmap e575: 12 total, 12 up, 12 in 2024-09-17T16:26:46.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:46 smithi017 ceph-mon[27571]: pgmap v1253: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 17 MiB/s wr, 504 op/s; 11385/275328 objects degraded (4.135%); 16327/275328 objects misplaced (5.930%); 5.1 MiB/s, 80 objects/s recovering 2024-09-17T16:26:46.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:47.015 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:47.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:47.047 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:26:47.117 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:47.154 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:47.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:47.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:47.288 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:47.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:47.390 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:47.413 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:47.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:47.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:47.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:47.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:47.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:47.664 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:47.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:47.757 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:47.766 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:47.786 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:47.849 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:47.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:47.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:47.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:47.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:47.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:48.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:48.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:48.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:48.201 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:48.237 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:48.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:48.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:48.340 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:48.378 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:48.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:48.480 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:48.523 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:48.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:48.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:48.664 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:48.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:48.728 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:48.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:48.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:48.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:48.833 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:26:48.833 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:26:48.833 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 575, 2024-09-17T16:26:48.833 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:26:48.833 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:26:45.530024+0000", 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:26:33.541808+0000", 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5801984, 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "noscrub", 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:26:48.834 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 6, 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.835 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:26:48.836 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:26:48.837 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "446", 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:26:48.838 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:26:48.839 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:26:48.840 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:26:48.841 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 6, 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_4", 2024-09-17T16:26:48.842 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:25:01.713250+0000", 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:26:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 16, 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 16, 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 16, 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 16, 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 16, 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:26:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "531", 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:26:48.845 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:26:48.846 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:26:48.847 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 3.440000057220459, 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.75, 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1.0900000333786011, 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3.75, 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:26:48.848 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.849 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 568, 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.850 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.851 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:26:48.851 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:26:48.851 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.851 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.851 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.851 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.852 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:26:48.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:26:48.854 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 568, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.856 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.858 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.859 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:26:48.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 527, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.862 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.863 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:26:48.864 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:26:48.865 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:26:48.866 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:26:48.867 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 568, 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.869 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.870 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.872 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:26:48.873 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 549, 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.874 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.875 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.877 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.878 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.879 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.845672607421875, 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 567, 2024-09-17T16:26:48.880 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:26:48.881 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:26:48.882 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.884 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.66790771484375, 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 568, 2024-09-17T16:26:48.886 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:26:48.887 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:26:48.888 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:26:48.889 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:48.891 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.892 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.8061676025390625, 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.73492431640625, 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 527, 2024-09-17T16:26:48.893 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:26:48.894 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.895 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.896 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.898 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 527, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.899 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.900 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:26:48.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:26:48.902 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:26:48.903 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.2821197509765625, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.905 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.906 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:26:48.906 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:26:48.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.908 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:26:48.908 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:26:48.908 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.908 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.908 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.908 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.909 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.910 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.911 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:26:48.912 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 527, 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.913 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.914 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.916 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.917 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:26:48.918 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.588836669921875, 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 291, 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:26:48.919 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.920 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.921 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.922 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:26:48.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:26:48.923 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:26:48.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.923 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:26:48.923 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:26:48.923 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:26:48.923 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.923 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:26:48.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:26:18.763422+0000", 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.925 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:25:51.068888+0000", 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:26:19.885704+0000", 2024-09-17T16:26:48.926 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:26:44.911102+0000", 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.927 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:26:25.416827+0000", 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.928 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:26:43.236707+0000", 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.929 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:26:13.073839+0000", 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:25:54.370583+0000", 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.930 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:26:44.342350+0000", 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:26:48.931 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.932 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.932 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.932 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.932 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.932 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:25:57.264390+0000", 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:26:38.611855+0000", 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.933 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:26:37.896630+0000", 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [ 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "1.0", 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:26:48.934 INFO:teuthology.orchestra.run.smithi017.stdout: 4, 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: 6, 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: 8 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [ 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "6.4", 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: 3 2024-09-17T16:26:48.935 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "6.7", 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: 3, 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: 4 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "6.a", 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: 6, 2024-09-17T16:26:48.936 INFO:teuthology.orchestra.run.smithi017.stdout: 0 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "6.c", 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: 3, 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: 8 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "6.d", 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:26:48.937 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: 10 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:26:48.938 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:26:48.939 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:48.940 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:26:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:48.978 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:48.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:49.078 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:49.107 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:49.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:49.209 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:48 smithi017 ceph-mon[27571]: pgmap v1254: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s wr, 600 op/s; 11380/268350 objects degraded (4.241%); 15921/268350 objects misplaced (5.933%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T16:26:49.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/390473728' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:26:49.242 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:49.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:49.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:48 smithi071 ceph-mon[30002]: pgmap v1254: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s wr, 600 op/s; 11380/268350 objects degraded (4.241%); 15921/268350 objects misplaced (5.933%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T16:26:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/390473728' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:26:49.344 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:49.376 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:49.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:49.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:48 smithi173 ceph-mon[30530]: pgmap v1254: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s wr, 600 op/s; 11380/268350 objects degraded (4.241%); 15921/268350 objects misplaced (5.933%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T16:26:49.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/390473728' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:26:49.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:49.504 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:49.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:49.533 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:26:49.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:49.633 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:49.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:49.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:49.743 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:26:49.768 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:49.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:49.871 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:49.894 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:49.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:49.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:50.020 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:50.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:50.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/11140437' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:50.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:49 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:50.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:49 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11380/268350 objects degraded (4.241%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:26:50.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:49 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:50.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:49 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:26:50.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:49 smithi017 ceph-mon[27571]: osdmap e576: 12 total, 12 up, 12 in 2024-09-17T16:26:50.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:50.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:50.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:50.171 INFO:tasks.thrashosds.thrasher:j is {'epoch': 575, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:26:45.530024+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:26:33.541808+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '446', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-17T16:25:01.713250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '531', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.440000057220459, 'score_stable': 2.75, 'optimal_score': 1.090000033378601, 'raw_score_acting': 3.75, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 568, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 568, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 568, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 0.845672607421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 567, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 0.66790771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 568, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 0.8061676025390625, 'primary_affinity': 0.73492431640625, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 527, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0.2821197509765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 0.588836669921875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:26:18.763422+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:25:51.068888+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:26:19.885704+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:26:44.911102+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:26:25.416827+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:26:43.236707+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:26:13.073839+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:25:54.370583+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:26:44.342350+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:25:57.264390+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:26:38.611855+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:26:37.896630+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [4, 6, 8]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '6.4', 'osds': [1, 3]}, {'pgid': '6.7', 'osds': [3, 4]}, {'pgid': '6.a', 'osds': [6, 0]}, {'pgid': '6.c', 'osds': [3, 8]}, {'pgid': '6.d', 'osds': [0, 1, 10]}], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:26:50.171 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-17T16:26:50.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:50.270 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:50.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/11140437' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:49 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:49 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11380/268350 objects degraded (4.241%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:26:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:49 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:49 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:26:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:49 smithi071 ceph-mon[30002]: osdmap e576: 12 total, 12 up, 12 in 2024-09-17T16:26:50.372 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:50.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/11140437' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:50.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:49 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:26:50.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:49 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11380/268350 objects degraded (4.241%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:26:50.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:49 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:50.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:49 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:26:50.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:49 smithi173 ceph-mon[30530]: osdmap e576: 12 total, 12 up, 12 in 2024-09-17T16:26:50.402 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:50.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:50.505 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:50.543 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:50.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:50.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:50.675 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:50.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:50.777 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:50.802 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:50.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:50.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:50.936 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:50.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:51.038 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:51.063 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:51.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:51.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:51.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:50 smithi071 ceph-mon[30002]: pgmap v1256: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s wr, 713 op/s; 11380/264891 objects degraded (4.296%); 15921/264891 objects misplaced (6.010%); 4.3 MiB/s, 69 objects/s recovering 2024-09-17T16:26:51.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:51.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:51.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:51.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:51.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:51.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:50 smithi173 ceph-mon[30530]: pgmap v1256: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s wr, 713 op/s; 11380/264891 objects degraded (4.296%); 15921/264891 objects misplaced (6.010%); 4.3 MiB/s, 69 objects/s recovering 2024-09-17T16:26:51.416 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:51.439 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:51.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:50 smithi017 ceph-mon[27571]: pgmap v1256: 17 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 12 active+clean; 5.4 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s wr, 713 op/s; 11380/264891 objects degraded (4.296%); 15921/264891 objects misplaced (6.010%); 4.3 MiB/s, 69 objects/s recovering 2024-09-17T16:26:51.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:51.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:51.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:51.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:51.692 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:51.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:51.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:51.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:51.824 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:51.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:51.925 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:51.963 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:51.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:52.065 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:52.106 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:52.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:52.208 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:52.231 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:52.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:52.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:52.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:52.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:52.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:52.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:52.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:52.591 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:52.619 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:52.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:52.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:52.755 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:52.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:52.857 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:52.881 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:52.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:52.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:53.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:53.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:53.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:53.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:53.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:53.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:53.167 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:53.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:53.267 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:53.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:53.303 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:52 smithi017 ceph-mon[27571]: pgmap v1257: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 12 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 11426/258396 objects degraded (4.422%); 15538/258396 objects misplaced (6.013%); 6.2 MiB/s, 101 objects/s recovering 2024-09-17T16:26:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:52 smithi071 ceph-mon[30002]: pgmap v1257: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 12 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 11426/258396 objects degraded (4.422%); 15538/258396 objects misplaced (6.013%); 6.2 MiB/s, 101 objects/s recovering 2024-09-17T16:26:53.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:53.392 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:53.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:53.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:53.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:52 smithi173 ceph-mon[30530]: pgmap v1257: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 12 active+clean; 5.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 11426/258396 objects degraded (4.422%); 15538/258396 objects misplaced (6.013%); 6.2 MiB/s, 101 objects/s recovering 2024-09-17T16:26:53.495 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:53.537 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:53.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:53.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:53.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:53.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:53.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:53.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:53.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:53.817 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:53.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:53.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:53.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:53.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:54.041 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:54.066 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:54.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:54.168 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:54.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:54.195 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:54.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:54.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:26:54.234 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:26:54.234 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:26:54.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.234 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:102723 6.4 6:27f918eb:::benchmark_data_smithi173_102017_object9575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-09-17T16:26:54.234 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:54.200949+0000", 2024-09-17T16:26:54.234 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.030358561999999999, 2024-09-17T16:26:54.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030391306, 2024-09-17T16:26:54.234 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:54.234 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:54.234 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 102723 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.200949+0000", 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:54.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.200949+0000", 2024-09-17T16:26:54.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:54.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:54.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.200951+0000", 2024-09-17T16:26:54.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.84e-06 2024-09-17T16:26:54.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.200959+0000", 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2309999999999999e-06 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.200972+0000", 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3946e-05 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:54.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.200981+0000", 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1799999999999996e-06 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.207047+0000", 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060659290000000003 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.207294+0000", 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024740800000000001 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,5", 2024-09-17T16:26:54.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.207753+0000", 2024-09-17T16:26:54.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045898199999999998 2024-09-17T16:26:54.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:26:54.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.217553+0000", 2024-09-17T16:26:54.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.009799973 2024-09-17T16:26:54.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.217568+0000", 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5140000000000001e-05 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:102736 6.4 6:2a4953ea:::benchmark_data_smithi173_102017_object9588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:54.217886+0000", 2024-09-17T16:26:54.240 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.013422454, 2024-09-17T16:26:54.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013543108999999999, 2024-09-17T16:26:54.241 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:54.241 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:54.241 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:26:54.241 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:54.241 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:26:54.241 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:26:54.241 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 102736 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.217886+0000", 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.217886+0000", 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:54.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.217887+0000", 2024-09-17T16:26:54.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.643e-06 2024-09-17T16:26:54.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:54.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.217894+0000", 2024-09-17T16:26:54.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3090000000000001e-06 2024-09-17T16:26:54.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:54.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.217911+0000", 2024-09-17T16:26:54.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7044000000000001e-05 2024-09-17T16:26:54.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:54.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.217919+0000", 2024-09-17T16:26:54.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8920000000000006e-06 2024-09-17T16:26:54.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.225815+0000", 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0078954170000000001 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.225908+0000", 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3398000000000003e-05 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,5", 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.226033+0000", 2024-09-17T16:26:54.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000125055 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26665.0:102729 6.b e577/545 6:df370af2:::benchmark_data_smithi173_102017_object9581:head v 577'6438, mlcod=577'6436)", 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:54.212751+0000", 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.018557292, 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.018740995999999999, 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:54.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.212751+0000", 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.212751+0000", 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.212752+0000", 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1909999999999999e-06 2024-09-17T16:26:54.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.212760+0000", 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.14e-06 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227427+0000", 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014666943999999999 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:54.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227436+0000", 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9020000000000005e-06 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227496+0000", 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0319999999999998e-05 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227518+0000", 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1793e-05 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.249 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26665.0:102723 6.4 e577/568)", 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:54.227389+0000", 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.0039191759999999999, 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041476450000000002, 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "queued for pg", 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:54.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227389+0000", 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227389+0000", 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227390+0000", 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.046e-06 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227396+0000", 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9139999999999996e-06 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227468+0000", 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2600000000000003e-05 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227474+0000", 2024-09-17T16:26:54.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6509999999999998e-06 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:102741 6.4 6.76bed8e4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:54.227702+0000", 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.0036056560000000001, 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038746340000000001, 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "queued for pg", 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:26:54.253 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:26:54.254 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:26:54.254 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 102741 2024-09-17T16:26:54.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.254 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:26:54.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227702+0000", 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227702+0000", 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227704+0000", 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5009999999999999e-06 2024-09-17T16:26:54.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227709+0000", 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4500000000000003e-06 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227723+0000", 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3591000000000001e-05 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:54.227731+0000", 2024-09-17T16:26:54.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8480000000000001e-06 2024-09-17T16:26:54.257 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.257 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:26:54.257 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.257 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:26:54.257 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:26:54.257 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 5 2024-09-17T16:26:54.257 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:26:54.272 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:26:54.272 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:26:54.272 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:26:54.273 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:26:54.273 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:26:54.273 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:26:54.273 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:26:54.273 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:26:54.289 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:26:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:26:54.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:54 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:26:54.335 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:26:54.336 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:26:54.342 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:54.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:54.370 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:26:54.371 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:26:54.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:54 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:26:54.406 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:26:54.407 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:26:54.441 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:26:54.441 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:26:54.445 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:54.471 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:26:54.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:26:54.478 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:54.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:54.511 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:26:54.512 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:26:54.547 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:26:54.548 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:26:54.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:54.602 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:54.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:54.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:54.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:54.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:54.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:54.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:54.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:54.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:54.953 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:54.985 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:54.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:55.088 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:55.113 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:55.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:55.172 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:26:55.172 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:26:55.173 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-17T16:26:55.173 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-17T16:26:55.180 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:26:55.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:55 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:26:55.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:55 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:26:55.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:55 smithi017 ceph-mon[27571]: pgmap v1258: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s; 11465/256620 objects degraded (4.468%); 15355/256620 objects misplaced (5.984%); 6.5 MiB/s, 106 objects/s recovering 2024-09-17T16:26:55.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:55 smithi017 ceph-mon[27571]: osdmap e577: 12 total, 12 up, 12 in 2024-09-17T16:26:55.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/123732030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:55.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:55 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11465/256620 objects degraded (4.468%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:26:55.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:55 smithi017 ceph-mon[27571]: osdmap e578: 12 total, 12 up, 12 in 2024-09-17T16:26:55.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:26:55.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:26:55.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:55 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:26:55.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:55.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:55.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:55 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:26:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:55 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:26:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:55 smithi071 ceph-mon[30002]: pgmap v1258: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s; 11465/256620 objects degraded (4.468%); 15355/256620 objects misplaced (5.984%); 6.5 MiB/s, 106 objects/s recovering 2024-09-17T16:26:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:55 smithi071 ceph-mon[30002]: osdmap e577: 12 total, 12 up, 12 in 2024-09-17T16:26:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/123732030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:55 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11465/256620 objects degraded (4.468%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:26:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:55 smithi071 ceph-mon[30002]: osdmap e578: 12 total, 12 up, 12 in 2024-09-17T16:26:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:26:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:26:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:55 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:26:55.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:26:55.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:55.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:26:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:55 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:26:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:55 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:26:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:55 smithi173 ceph-mon[30530]: pgmap v1258: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s; 11465/256620 objects degraded (4.468%); 15355/256620 objects misplaced (5.984%); 6.5 MiB/s, 106 objects/s recovering 2024-09-17T16:26:55.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:55 smithi173 ceph-mon[30530]: osdmap e577: 12 total, 12 up, 12 in 2024-09-17T16:26:55.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/123732030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:26:55.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:55 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11465/256620 objects degraded (4.468%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:26:55.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:55 smithi173 ceph-mon[30530]: osdmap e578: 12 total, 12 up, 12 in 2024-09-17T16:26:55.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:26:55.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:26:55.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:55 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:26:55.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:26:55.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:55.528 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:55.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:55.631 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:55.669 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:55.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:55.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:55.805 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:55.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:55.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:55.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:55.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:55.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:56.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:56.060 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:56.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:56.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:56.109 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:56.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:56.225 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:56.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:56.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:56.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:56.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:56.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:56.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:56.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:56.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:56.625 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:56.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:56.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:56.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:56.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:26:56.853 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:56.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:56.886 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:56.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:56.960 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:26:56.960 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:26:56.960 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:26:56.960 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:26:56.960 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:26:56.960 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:26:56.960 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:26:56.960 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:26:56.961 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:26:56.961 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:26:56.961 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:26:56.961 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:26:56.987 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:57.015 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:57.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:57.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:57.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:57.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:57.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:57.270 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:57.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:57 smithi071 ceph-mon[30002]: pgmap v1261: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 11463/254148 objects degraded (4.510%); 15355/254148 objects misplaced (6.042%); 4.6 MiB/s, 75 objects/s recovering 2024-09-17T16:26:57.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:57 smithi017 ceph-mon[27571]: pgmap v1261: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 11463/254148 objects degraded (4.510%); 15355/254148 objects misplaced (6.042%); 4.6 MiB/s, 75 objects/s recovering 2024-09-17T16:26:57.372 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:26:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:57 smithi173 ceph-mon[30530]: pgmap v1261: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 11463/254148 objects degraded (4.510%); 15355/254148 objects misplaced (6.042%); 4.6 MiB/s, 75 objects/s recovering 2024-09-17T16:26:57.397 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:57.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:57.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:26:57.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:57.521 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:57.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:57.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:57.660 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:57.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:57.701 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:26:57.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:57.809 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:57.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:57.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:57.932 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:57.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:57.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:26:58.028 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:26:58.034 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:58.097 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:58.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:58.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:58.228 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:58.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/806146731' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:58 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:58.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:26:58.361 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:58.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:26:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/806146731' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:58 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:58.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/806146731' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:58 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:26:58.491 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:58.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:58.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:58.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:26:58.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:58.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:26:58.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:26:58.746 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:58.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:58.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:26:58.853 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:26:58.882 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:58.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:26:58.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:26:59.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:59.024 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:59.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:26:59.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:26:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:59.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:59.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:26:59.251 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:59.286 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:59.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:59 smithi071 ceph-mon[30002]: pgmap v1262: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 11568/248052 objects degraded (4.664%); 14787/248052 objects misplaced (5.961%); 7.2 MiB/s, 118 objects/s recovering 2024-09-17T16:26:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:59 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:59 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:26:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:26:59 smithi071 ceph-mon[30002]: osdmap e579: 12 total, 12 up, 12 in 2024-09-17T16:26:59.388 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:26:59.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:59 smithi173 ceph-mon[30530]: pgmap v1262: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 11568/248052 objects degraded (4.664%); 14787/248052 objects misplaced (5.961%); 7.2 MiB/s, 118 objects/s recovering 2024-09-17T16:26:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:59 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:59 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:26:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:26:59 smithi173 ceph-mon[30530]: osdmap e579: 12 total, 12 up, 12 in 2024-09-17T16:26:59.419 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:59.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:26:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:59 smithi017 ceph-mon[27571]: pgmap v1262: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 11568/248052 objects degraded (4.664%); 14787/248052 objects misplaced (5.961%); 7.2 MiB/s, 118 objects/s recovering 2024-09-17T16:26:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:59 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:26:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:59 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:26:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:26:59 smithi017 ceph-mon[27571]: osdmap e579: 12 total, 12 up, 12 in 2024-09-17T16:26:59.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:26:59.548 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:26:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:59.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:26:59.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:59.687 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:59.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:59.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:26:59.824 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:26:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:59.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:26:59.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:26:59.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:26:59.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:26:59.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:00.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:00.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:00.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:00.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:00.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:00.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:00.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:00.301 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:00.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:00 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11568/248052 objects degraded (4.664%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:00 smithi071 ceph-mon[30002]: osdmap e580: 12 total, 12 up, 12 in 2024-09-17T16:27:00.324 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:00.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:00 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11568/248052 objects degraded (4.664%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:00 smithi173 ceph-mon[30530]: osdmap e580: 12 total, 12 up, 12 in 2024-09-17T16:27:00.426 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:00.469 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:00.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:00.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:00 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11568/248052 objects degraded (4.664%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:00 smithi017 ceph-mon[27571]: osdmap e580: 12 total, 12 up, 12 in 2024-09-17T16:27:00.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:00.605 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:00.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:00.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:00.707 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:00.737 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:00.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:00.839 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:00.866 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:00.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:00.968 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:01.009 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:01.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:01.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:01.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:01.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:01.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:01.286 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:01.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:01.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:01 smithi071 ceph-mon[30002]: pgmap v1265: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 11568/244884 objects degraded (4.724%); 14787/244884 objects misplaced (6.038%); 4.8 MiB/s, 78 objects/s recovering 2024-09-17T16:27:01.388 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:01.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:01 smithi173 ceph-mon[30530]: pgmap v1265: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 11568/244884 objects degraded (4.724%); 14787/244884 objects misplaced (6.038%); 4.8 MiB/s, 78 objects/s recovering 2024-09-17T16:27:01.413 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:01.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:01 smithi017 ceph-mon[27571]: pgmap v1265: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 5.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 11568/244884 objects degraded (4.724%); 14787/244884 objects misplaced (6.038%); 4.8 MiB/s, 78 objects/s recovering 2024-09-17T16:27:01.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:01.538 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:01.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:01.640 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:01.666 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:01.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:01.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:01.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:01.802 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:01.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:01.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:01.927 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:01.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:01.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:02.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:02.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:02.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:02.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:02.157 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:02.192 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:02.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:02.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:02.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:02.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:02.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:02.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:02.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:02.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:02.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:02.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:02.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:02.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:02.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:02.701 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:02.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:02.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:02.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:02.827 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:02.928 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:02.954 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:02.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:03.056 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:03.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:03.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:03.209 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:03.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:03.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:03 smithi071 ceph-mon[30002]: pgmap v1266: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s; 11650/238344 objects degraded (4.888%); 14388/238344 objects misplaced (6.037%); 6.3 MiB/s, 105 objects/s recovering 2024-09-17T16:27:03.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:03.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:03.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:03 smithi173 ceph-mon[30530]: pgmap v1266: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s; 11650/238344 objects degraded (4.888%); 14388/238344 objects misplaced (6.037%); 6.3 MiB/s, 105 objects/s recovering 2024-09-17T16:27:03.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:03.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:03.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:03.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:03 smithi017 ceph-mon[27571]: pgmap v1266: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s; 11650/238344 objects degraded (4.888%); 14388/238344 objects misplaced (6.037%); 6.3 MiB/s, 105 objects/s recovering 2024-09-17T16:27:03.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:03.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:03.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:03.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:03.711 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:03.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:03.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:03.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:03.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:03.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:03.936 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:03.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:03.967 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:03.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:04.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:04.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:04.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:04.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:04.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:04.228 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:04.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:04.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:04.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:04.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:04.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:04.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:04.478 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:04.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:04.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:04.601 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:04.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:04.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:04.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:04.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:04.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:04.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:04.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:04.951 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:04.974 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:04.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:04.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:05.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:05.113 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:05.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:05.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:05.215 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:05.241 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:05.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:05.290 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:05 smithi017 ceph-mon[27571]: pgmap v1267: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 742 op/s; 11716/236313 objects degraded (4.958%); 14169/236313 objects misplaced (5.996%); 7.0 MiB/s, 116 objects/s recovering 2024-09-17T16:27:05.290 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:05 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11716/236313 objects degraded (4.958%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:05 smithi071 ceph-mon[30002]: pgmap v1267: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 742 op/s; 11716/236313 objects degraded (4.958%); 14169/236313 objects misplaced (5.996%); 7.0 MiB/s, 116 objects/s recovering 2024-09-17T16:27:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:05 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11716/236313 objects degraded (4.958%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:05.343 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:05.380 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:05.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:05.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:05 smithi173 ceph-mon[30530]: pgmap v1267: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 742 op/s; 11716/236313 objects degraded (4.958%); 14169/236313 objects misplaced (5.996%); 7.0 MiB/s, 116 objects/s recovering 2024-09-17T16:27:05.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:05 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11716/236313 objects degraded (4.958%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:05.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:05.509 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:05.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:05.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:05.611 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:05.637 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:05.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:05.739 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:05.764 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:05.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:05.866 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:05.901 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:05.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:06.003 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:06.038 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:06.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:06.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:06.162 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:06.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:06.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:06.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:06.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:06.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:06.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:06.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:06.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:06.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:06.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:06.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:06.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:06.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:06.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:06.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:06.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:06.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:06.902 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:06.937 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:06.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:07.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:07.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:07.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:07.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:07.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:07.192 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:07.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:07.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:07.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:07.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:07 smithi017 ceph-mon[27571]: pgmap v1268: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 11713/233775 objects degraded (5.010%); 14169/233775 objects misplaced (6.061%); 3.5 MiB/s, 59 objects/s recovering 2024-09-17T16:27:07.305 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:27:07.305 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:27:07.305 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:27:07.305 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:27:07.305 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:27:07.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:07 smithi071 ceph-mon[30002]: pgmap v1268: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 11713/233775 objects degraded (5.010%); 14169/233775 objects misplaced (6.061%); 3.5 MiB/s, 59 objects/s recovering 2024-09-17T16:27:07.328 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:07.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:07 smithi173 ceph-mon[30530]: pgmap v1268: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 11713/233775 objects degraded (5.010%); 14169/233775 objects misplaced (6.061%); 3.5 MiB/s, 59 objects/s recovering 2024-09-17T16:27:07.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:07.471 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:07.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:07.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:07.616 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:07.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:07.718 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:07.743 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:07.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:07.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:07.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:07.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:07.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:07.970 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:08.001 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:08.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:08.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:08.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:08.138 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:08.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:08.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:08.265 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:08.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:08.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:08.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:08.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:08.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:08.516 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:08.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:08.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:08.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:08.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:08.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:08.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:08.775 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:08.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:08.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:08.902 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:08.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:09.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:09.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:09.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:09.128 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:09.163 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:09.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:09.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:09.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:09.300 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:09.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:09.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:09 smithi071 ceph-mon[30002]: pgmap v1269: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s; 11864/227691 objects degraded (5.211%); 13550/227691 objects misplaced (5.951%); 5.8 MiB/s, 94 objects/s recovering 2024-09-17T16:27:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:09 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:27:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:09 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:27:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:09 smithi173 ceph-mon[30530]: pgmap v1269: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s; 11864/227691 objects degraded (5.211%); 13550/227691 objects misplaced (5.951%); 5.8 MiB/s, 94 objects/s recovering 2024-09-17T16:27:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:09 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:27:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:09 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:27:09.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:09.430 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:09.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:09.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:09 smithi017 ceph-mon[27571]: pgmap v1269: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 684 op/s; 11864/227691 objects degraded (5.211%); 13550/227691 objects misplaced (5.951%); 5.8 MiB/s, 94 objects/s recovering 2024-09-17T16:27:09.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:09 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:27:09.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:09 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:27:09.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:09.561 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:09.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:09.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:09.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:09.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:09.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:09.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:09.813 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:09.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:09.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:09.957 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:09.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:10.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:10.088 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:10.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:10.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:10.223 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:10.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:10 smithi071 ceph-mon[30002]: osdmap e581: 12 total, 12 up, 12 in 2024-09-17T16:27:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:10 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:27:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:10 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:27:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11864/227691 objects degraded (5.211%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:10.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:10.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:10.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:10 smithi173 ceph-mon[30530]: osdmap e581: 12 total, 12 up, 12 in 2024-09-17T16:27:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:10 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:27:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:10 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:27:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11864/227691 objects degraded (5.211%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:10.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:10.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:10 smithi017 ceph-mon[27571]: osdmap e581: 12 total, 12 up, 12 in 2024-09-17T16:27:10.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:10 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:27:10.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:10 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:27:10.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11864/227691 objects degraded (5.211%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:10.493 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:10.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:27:10.531 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:27:10.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-17T16:27:10.595 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:10.653 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:10.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:10.755 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:10.788 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:10.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:10.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:10.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:10.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:11.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:11.038 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:11.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:11.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:11.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:11.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:11 smithi071 ceph-mon[30002]: pgmap v1271: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 11864/224610 objects degraded (5.282%); 13550/224610 objects misplaced (6.033%); 5.7 MiB/s, 94 objects/s recovering 2024-09-17T16:27:11.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:11 smithi071 ceph-mon[30002]: osdmap e582: 12 total, 12 up, 12 in 2024-09-17T16:27:11.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:11.351 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:11.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:11 smithi173 ceph-mon[30530]: pgmap v1271: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 11864/224610 objects degraded (5.282%); 13550/224610 objects misplaced (6.033%); 5.7 MiB/s, 94 objects/s recovering 2024-09-17T16:27:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:11 smithi173 ceph-mon[30530]: osdmap e582: 12 total, 12 up, 12 in 2024-09-17T16:27:11.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:11.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:11.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:11.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:11 smithi017 ceph-mon[27571]: pgmap v1271: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 675 op/s; 11864/224610 objects degraded (5.282%); 13550/224610 objects misplaced (6.033%); 5.7 MiB/s, 94 objects/s recovering 2024-09-17T16:27:11.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:11 smithi017 ceph-mon[27571]: osdmap e582: 12 total, 12 up, 12 in 2024-09-17T16:27:11.578 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:11.601 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:11.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:11.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:11.726 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:11.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:11.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:11.861 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:11.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:11.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:11.988 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:11.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:12.090 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:12.112 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:12.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:12.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:12.245 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:12.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:12.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:12.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:12.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:12.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:12.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:12.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:12.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:12.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:12.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:12.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:12.743 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:12.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:12.844 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:12.873 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:12.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:12.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:13.008 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:13.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:13.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:13.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:13.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:13.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:13.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:13.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:13.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:13 smithi017 ceph-mon[27571]: pgmap v1273: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s; 11927/218712 objects degraded (5.453%); 13151/218712 objects misplaced (6.013%); 6.2 MiB/s, 100 objects/s recovering 2024-09-17T16:27:13.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:13.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:13.382 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:13 smithi173 ceph-mon[30530]: pgmap v1273: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s; 11927/218712 objects degraded (5.453%); 13151/218712 objects misplaced (6.013%); 6.2 MiB/s, 100 objects/s recovering 2024-09-17T16:27:13.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:13.506 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:13.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:13 smithi071 ceph-mon[30002]: pgmap v1273: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 733 op/s; 11927/218712 objects degraded (5.453%); 13151/218712 objects misplaced (6.013%); 6.2 MiB/s, 100 objects/s recovering 2024-09-17T16:27:13.607 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:13.629 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:13.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:13.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:13.763 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:13.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:13.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:13.897 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:13.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:13.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:14.020 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:14.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:14.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:14.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:14.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:14.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:14.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:14.267 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:14.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:14.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:14.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:14.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:14.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:14.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:14.515 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:14.617 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:14.640 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:14.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:14.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:14.770 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:14.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:14.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:14.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:14.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:14.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:15.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:15.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:15.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:15.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:15.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:15.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:15.244 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:15.268 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:15.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:15.370 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:15 smithi173 ceph-mon[30530]: pgmap v1274: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 11949/217017 objects degraded (5.506%); 12948/217017 objects misplaced (5.966%); 7.6 MiB/s, 123 objects/s recovering 2024-09-17T16:27:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:15 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11949/217017 objects degraded (5.506%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:15.395 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:15.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:15 smithi017 ceph-mon[27571]: pgmap v1274: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 11949/217017 objects degraded (5.506%); 12948/217017 objects misplaced (5.966%); 7.6 MiB/s, 123 objects/s recovering 2024-09-17T16:27:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:15 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11949/217017 objects degraded (5.506%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:15.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:15.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:15.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:15.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:15.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:15 smithi071 ceph-mon[30002]: pgmap v1274: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 698 op/s; 11949/217017 objects degraded (5.506%); 12948/217017 objects misplaced (5.966%); 7.6 MiB/s, 123 objects/s recovering 2024-09-17T16:27:15.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:15 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11949/217017 objects degraded (5.506%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:15.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:15.645 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:15.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:15.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:15.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:15.772 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:15.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:15.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:15.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:15.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:15.998 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:16.022 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:16.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:16.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:16.147 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:16.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:16.249 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:16.275 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:16.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:16.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:16.376 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:16.403 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:16.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:16.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:16.528 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:16.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:16.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:16.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:16.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:16.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:16.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:16.758 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:16.783 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:16.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:16.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:16.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:16.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:16.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:17.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:17.037 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:17.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:17.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:17.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:17.162 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:17.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:17 smithi017 ceph-mon[27571]: pgmap v1275: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 11945/215223 objects degraded (5.550%); 12948/215223 objects misplaced (6.016%); 3.9 MiB/s, 65 objects/s recovering 2024-09-17T16:27:17.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:17.287 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:17.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:17.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:17.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:17 smithi173 ceph-mon[30530]: pgmap v1275: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 11945/215223 objects degraded (5.550%); 12948/215223 objects misplaced (6.016%); 3.9 MiB/s, 65 objects/s recovering 2024-09-17T16:27:17.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:17.420 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:17.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:17.543 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:17.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:17.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:17 smithi071 ceph-mon[30002]: pgmap v1275: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 11945/215223 objects degraded (5.550%); 12948/215223 objects misplaced (6.016%); 3.9 MiB/s, 65 objects/s recovering 2024-09-17T16:27:17.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:17.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:17.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:17.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:17.773 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:17.795 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:17.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:17.897 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:17.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:17.920 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:17.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:18.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:18.046 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:18.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:18.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:18.148 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:18.174 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:18.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:18.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:18.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:18.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:18.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:18.405 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:18.427 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:18.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:18.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:27:18.529 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:18.570 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:18.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:18.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:18.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:18.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:18.799 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:18.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:18.826 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:18.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:18.928 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:18.950 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:18.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:18.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:19.053 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:19.075 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:19.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:19.177 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:19.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:19.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:19.301 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:19.323 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:19.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:19.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:19 smithi173 ceph-mon[30530]: pgmap v1276: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 11957/212277 objects degraded (5.633%); 12239/212277 objects misplaced (5.766%); 8.4 MiB/s, 136 objects/s recovering 2024-09-17T16:27:19.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:19.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:19.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:19.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:19 smithi017 ceph-mon[27571]: pgmap v1276: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 11957/212277 objects degraded (5.633%); 12239/212277 objects misplaced (5.766%); 8.4 MiB/s, 136 objects/s recovering 2024-09-17T16:27:19.551 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:19.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:19 smithi071 ceph-mon[30002]: pgmap v1276: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 11957/212277 objects degraded (5.633%); 12239/212277 objects misplaced (5.766%); 8.4 MiB/s, 136 objects/s recovering 2024-09-17T16:27:19.573 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:19.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:19.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:19.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:19.697 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:19.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:19.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:19.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:19.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:19.927 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:19.951 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:19.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:20.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:20.075 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:20.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:20.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:20.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:20.204 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:20.304 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:20.328 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:20.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:20.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:20.329 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:20 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11957/212277 objects degraded (5.633%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:20 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11957/212277 objects degraded (5.633%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:20.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:20.453 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:20.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:20 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11957/212277 objects degraded (5.633%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:20.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:20.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:20.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:20.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:20.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:20.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:20.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:20.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:20.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:20.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:20.960 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:20.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:21.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:21.086 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:21.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:21.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:21.211 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:21.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:21.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:21.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:21 smithi071 ceph-mon[30002]: pgmap v1277: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 420 op/s; 11957/211998 objects degraded (5.640%); 12239/211998 objects misplaced (5.773%); 7.5 MiB/s, 121 objects/s recovering 2024-09-17T16:27:21.334 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:21.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:21.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:21 smithi173 ceph-mon[30530]: pgmap v1277: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 420 op/s; 11957/211998 objects degraded (5.640%); 12239/211998 objects misplaced (5.773%); 7.5 MiB/s, 121 objects/s recovering 2024-09-17T16:27:21.436 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:21.458 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:21.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:21 smithi017 ceph-mon[27571]: pgmap v1277: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 420 op/s; 11957/211998 objects degraded (5.640%); 12239/211998 objects misplaced (5.773%); 7.5 MiB/s, 121 objects/s recovering 2024-09-17T16:27:21.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:21.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:21.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:21.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:21.710 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:21.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:21.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:21.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:21.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:21.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:21.961 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:21.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:22.063 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:22.090 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:22.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:22.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:22.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:22.213 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:22.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:22.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:22.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:22.339 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:22.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:22.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:22.466 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:22.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:22.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:22.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:22.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:22.697 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:22.723 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:22.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:22.825 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:22.846 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:22.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:22.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:22.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:22.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:22.992 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:27:22.999 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:27:23.033 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:27:23.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:23.099 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:23.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:23.201 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:23.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:23 smithi017 ceph-mon[27571]: pgmap v1278: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 354 op/s; 11957/211956 objects degraded (5.641%); 11657/211956 objects misplaced (5.500%); 9.4 MiB/s, 151 objects/s recovering 2024-09-17T16:27:23.225 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:23.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:23.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:23.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:23.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:23 smithi173 ceph-mon[30530]: pgmap v1278: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 354 op/s; 11957/211956 objects degraded (5.641%); 11657/211956 objects misplaced (5.500%); 9.4 MiB/s, 151 objects/s recovering 2024-09-17T16:27:23.455 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:23.479 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:23.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:23 smithi071 ceph-mon[30002]: pgmap v1278: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 354 op/s; 11957/211956 objects degraded (5.641%); 11657/211956 objects misplaced (5.500%); 9.4 MiB/s, 151 objects/s recovering 2024-09-17T16:27:23.581 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:23.604 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:23.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:23.706 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:23.730 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:23.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:23.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:23.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:23.853 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:23.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:23.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:23.979 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:23.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:24.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:24.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:24.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:24.209 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:24.230 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:24.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:24.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:24.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:24.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:24.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:24.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:24.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:24.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:24.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:24.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:24.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:24.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:24.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:24.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:24.842 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:24.863 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:24.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:24.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:24.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:27:24.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:24.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:24.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:25.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:25.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:25.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:25.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:25.242 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:25.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:25.344 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:25.371 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:25.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:25 smithi173 ceph-mon[30530]: pgmap v1279: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 187 op/s; 11957/211956 objects degraded (5.641%); 11402/211956 objects misplaced (5.379%); 8.9 MiB/s, 143 objects/s recovering 2024-09-17T16:27:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:25 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11957/211956 objects degraded (5.641%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:25.419 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:25 smithi017 ceph-mon[27571]: pgmap v1279: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 187 op/s; 11957/211956 objects degraded (5.641%); 11402/211956 objects misplaced (5.379%); 8.9 MiB/s, 143 objects/s recovering 2024-09-17T16:27:25.419 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:25 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11957/211956 objects degraded (5.641%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:25.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:25.498 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:25.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:25 smithi071 ceph-mon[30002]: pgmap v1279: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 187 op/s; 11957/211956 objects degraded (5.641%); 11402/211956 objects misplaced (5.379%); 8.9 MiB/s, 143 objects/s recovering 2024-09-17T16:27:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:25 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11957/211956 objects degraded (5.641%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:25.601 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:25.627 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:25.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:25.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:25.756 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:25.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:25.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:25.882 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:25.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:25.984 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:26.005 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:26.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:26.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:26.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:26.129 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:26.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:26.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:26.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:26.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:26.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:26.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:26.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:26.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:26.509 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:26.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:26.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:26.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:26.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:26.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:26.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:26.761 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:26.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:26.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:26.884 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:26.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:26.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:27.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:27.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:27.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:27.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:27.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:27.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:27.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:27.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:27.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:27.363 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:27.387 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:27.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:27 smithi173 ceph-mon[30530]: pgmap v1280: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 11957/211956 objects degraded (5.641%); 11402/211956 objects misplaced (5.379%); 8.0 MiB/s, 128 objects/s recovering 2024-09-17T16:27:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:27 smithi017 ceph-mon[27571]: pgmap v1280: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 11957/211956 objects degraded (5.641%); 11402/211956 objects misplaced (5.379%); 8.0 MiB/s, 128 objects/s recovering 2024-09-17T16:27:27.489 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:27.515 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:27.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:27 smithi071 ceph-mon[30002]: pgmap v1280: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 11957/211956 objects degraded (5.641%); 11402/211956 objects misplaced (5.379%); 8.0 MiB/s, 128 objects/s recovering 2024-09-17T16:27:27.617 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:27.638 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:27.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:27.740 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:27.763 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:27.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:27.865 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:27.886 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:27.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:27.989 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:28.014 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:28.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:28.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:28.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:28.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:28.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:28 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:27:28.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:28 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:27:28.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:28 smithi017 ceph-mon[27571]: osdmap e583: 12 total, 12 up, 12 in 2024-09-17T16:27:28.244 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:28.266 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:28.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:28.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:28.393 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:28.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:28.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:28 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:27:28.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:28 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:27:28.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:28 smithi173 ceph-mon[30530]: osdmap e583: 12 total, 12 up, 12 in 2024-09-17T16:27:28.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:28.516 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:28.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:28 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:27:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:28 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:27:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:28 smithi071 ceph-mon[30002]: osdmap e583: 12 total, 12 up, 12 in 2024-09-17T16:27:28.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:28.640 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:28.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:28.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:28.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:28.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:28.870 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:28.892 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:28.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:28.995 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:29.019 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:29.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:29.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:29.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:29.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:29.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:29.300 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:29.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:29.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:29 smithi017 ceph-mon[27571]: pgmap v1282: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 10 op/s; 11957/211956 objects degraded (5.641%); 10367/211956 objects misplaced (4.891%); 12 MiB/s, 187 objects/s recovering 2024-09-17T16:27:29.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:29 smithi173 ceph-mon[30530]: pgmap v1282: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 10 op/s; 11957/211956 objects degraded (5.641%); 10367/211956 objects misplaced (4.891%); 12 MiB/s, 187 objects/s recovering 2024-09-17T16:27:29.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:29.438 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:29.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:29.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:29.540 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:29 smithi071 ceph-mon[30002]: pgmap v1282: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 10 op/s; 11957/211956 objects degraded (5.641%); 10367/211956 objects misplaced (4.891%); 12 MiB/s, 187 objects/s recovering 2024-09-17T16:27:29.577 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:29.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:29.679 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:29.716 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:29.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:29.818 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:29.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:29.849 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:29.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:29.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:29.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:29.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:29.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:30.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:30.109 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:30.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:30.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:30.239 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:30.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:30.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:30.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:30.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:30.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:30.465 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:30.506 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:30.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:30.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:27:30.534 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:27:30.535 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:27:30.607 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:30.632 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:30.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:30.734 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:30.763 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:30.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:30.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:30 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:27:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:30 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:27:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:30 smithi071 ceph-mon[30002]: pgmap v1283: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 10 op/s; 11957/211677 objects degraded (5.649%); 10367/211677 objects misplaced (4.898%); 12 MiB/s, 187 objects/s recovering 2024-09-17T16:27:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:30 smithi071 ceph-mon[30002]: osdmap e584: 12 total, 12 up, 12 in 2024-09-17T16:27:30.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:30 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:27:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:30 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:27:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:30 smithi173 ceph-mon[30530]: pgmap v1283: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 10 op/s; 11957/211677 objects degraded (5.649%); 10367/211677 objects misplaced (4.898%); 12 MiB/s, 187 objects/s recovering 2024-09-17T16:27:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:30 smithi173 ceph-mon[30530]: osdmap e584: 12 total, 12 up, 12 in 2024-09-17T16:27:30.897 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:30.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:30.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:30 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:27:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:30 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:27:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:30 smithi017 ceph-mon[27571]: pgmap v1283: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 10 op/s; 11957/211677 objects degraded (5.649%); 10367/211677 objects misplaced (4.898%); 12 MiB/s, 187 objects/s recovering 2024-09-17T16:27:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:30 smithi017 ceph-mon[27571]: osdmap e584: 12 total, 12 up, 12 in 2024-09-17T16:27:31.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:31.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:31.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:31.125 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:31.147 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:31.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:31.249 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:31.285 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:31.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:31.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:31.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:31.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:31.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:31.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:31.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:31.636 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:31.673 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:31.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:31.776 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:31.807 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:31.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:31 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11957/211677 objects degraded (5.649%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:31 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11957/211677 objects degraded (5.649%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:31.909 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:31.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:31.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:31.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:31 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11957/211677 objects degraded (5.649%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:27:32.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:32.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:32.057 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:32.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:32.188 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:32.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:32.290 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:32.314 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:32.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:32.416 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:32.448 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:32.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:32.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:32.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:32.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:32.676 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:32.710 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:32.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:32.813 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:32 smithi071 ceph-mon[30002]: pgmap v1285: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 11954/208875 objects degraded (5.723%); 9799/208875 objects misplaced (4.691%); 13 MiB/s, 200 objects/s recovering 2024-09-17T16:27:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:32 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:27:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:32 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:27:32.841 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:32.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:32 smithi173 ceph-mon[30530]: pgmap v1285: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 11954/208875 objects degraded (5.723%); 9799/208875 objects misplaced (4.691%); 13 MiB/s, 200 objects/s recovering 2024-09-17T16:27:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:32 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:27:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:32 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:27:32.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:32.967 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:32.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:32.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:32.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:32 smithi017 ceph-mon[27571]: pgmap v1285: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 128 op/s; 11954/208875 objects degraded (5.723%); 9799/208875 objects misplaced (4.691%); 13 MiB/s, 200 objects/s recovering 2024-09-17T16:27:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:32 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:27:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:32 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:27:33.070 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:33.096 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:33.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:33.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:33.221 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:33.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:33.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:33.323 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:33.359 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:33.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:33.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:33.491 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:33.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:33.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:33.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:33.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:33.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:33.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:33.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:33.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:33 smithi071 ceph-mon[30002]: osdmap e585: 12 total, 12 up, 12 in 2024-09-17T16:27:33.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:33.887 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:33.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:33.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:33 smithi173 ceph-mon[30530]: osdmap e585: 12 total, 12 up, 12 in 2024-09-17T16:27:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:33 smithi017 ceph-mon[27571]: osdmap e585: 12 total, 12 up, 12 in 2024-09-17T16:27:33.989 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:34.020 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:34.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:34.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:34.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:34.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:34.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:34.268 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:34.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:34.370 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:34.395 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:34.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:34.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:34.519 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:34.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:34.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:34.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:34.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:34 smithi017 ceph-mon[27571]: pgmap v1287: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 207 op/s; 11954/207675 objects degraded (5.756%); 9512/207675 objects misplaced (4.580%); 7.8 MiB/s, 124 objects/s recovering 2024-09-17T16:27:34.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2318003228' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:27:34.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:34.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:34.746 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:34.780 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:34.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:34 smithi071 ceph-mon[30002]: pgmap v1287: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 207 op/s; 11954/207675 objects degraded (5.756%); 9512/207675 objects misplaced (4.580%); 7.8 MiB/s, 124 objects/s recovering 2024-09-17T16:27:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2318003228' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:27:34.881 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:34 smithi173 ceph-mon[30530]: pgmap v1287: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 207 op/s; 11954/207675 objects degraded (5.756%); 9512/207675 objects misplaced (4.580%); 7.8 MiB/s, 124 objects/s recovering 2024-09-17T16:27:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2318003228' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:27:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:34.904 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:34.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:35.006 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:35.031 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:35.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:35.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:35.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:35.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:35.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:35.259 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:35.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:35.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:35.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:27:35.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:27:35.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:27:35.322 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:27:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:27:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:27:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:27:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:27:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:27:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:27:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:27:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:27:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:27:35.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:27:35.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:35.433 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:35.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:35.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:35.557 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:35.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:35.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:35.680 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:35.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:35.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:35.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:35.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:35.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:35.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:35.937 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:35.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:35.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:36.040 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:36.079 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:36.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:36.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:36.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:36.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:36.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:36.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:36 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11954/205347 objects degraded (5.821%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:36.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:36.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:36.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:36 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11954/205347 objects degraded (5.821%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:36.428 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:36.452 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:36.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:36.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:36 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11954/205347 objects degraded (5.821%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:36.554 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:36.593 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:36.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:36.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:36.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:36.720 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:36.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:36.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:36.851 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:36.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:36.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:36.975 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:36.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:37.077 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:37.100 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:37.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:37.202 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:37.228 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:37.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:37.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:37 smithi071 ceph-mon[30002]: pgmap v1288: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 275 op/s; 11954/205347 objects degraded (5.821%); 9512/205347 objects misplaced (4.632%); 6.7 MiB/s, 107 objects/s recovering 2024-09-17T16:27:37.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:37.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:37.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:37 smithi173 ceph-mon[30530]: pgmap v1288: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 275 op/s; 11954/205347 objects degraded (5.821%); 9512/205347 objects misplaced (4.632%); 6.7 MiB/s, 107 objects/s recovering 2024-09-17T16:27:37.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:37.481 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:37.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:37.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:37 smithi017 ceph-mon[27571]: pgmap v1288: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 275 op/s; 11954/205347 objects degraded (5.821%); 9512/205347 objects misplaced (4.632%); 6.7 MiB/s, 107 objects/s recovering 2024-09-17T16:27:37.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:37.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:37.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:37.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:37.746 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:37.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:37.848 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:37.875 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:37.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:37.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:37.999 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:38.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:38.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:38.125 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:38.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:38.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:38.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:38.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:38.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:38.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:38.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:38.477 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:38.504 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:38.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:38.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:38.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:38.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:38.729 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:38.755 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:38.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:38.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:38.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:38.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:38.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:38.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:38.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:39.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:39.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:39.108 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:39.137 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:39.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:39.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:39.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:39.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:39.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:39 smithi071 ceph-mon[30002]: pgmap v1289: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 545 op/s; 11948/198588 objects degraded (6.016%); 8994/198588 objects misplaced (4.529%); 11 MiB/s, 172 objects/s recovering 2024-09-17T16:27:39.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:39.387 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:39.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:39.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:39 smithi173 ceph-mon[30530]: pgmap v1289: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 545 op/s; 11948/198588 objects degraded (6.016%); 8994/198588 objects misplaced (4.529%); 11 MiB/s, 172 objects/s recovering 2024-09-17T16:27:39.488 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:39 smithi017 ceph-mon[27571]: pgmap v1289: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 545 op/s; 11948/198588 objects degraded (6.016%); 8994/198588 objects misplaced (4.529%); 11 MiB/s, 172 objects/s recovering 2024-09-17T16:27:39.513 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:39.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:39.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:39.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:39.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:39.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:39.740 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:39.765 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:39.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:39.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:39.902 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:39.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:40.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:40.027 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:40.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:40.052 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:27:40.052 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:27:40.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:40.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:40.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:40.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:40.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:40.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:40.281 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:27:40.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:40.402 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:40.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:40.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:40.530 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:40.530 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:40.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:40.659 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:40.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:40.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:40.798 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:40.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:40.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:40.900 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:40.927 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:40.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:41.029 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:41.062 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:41.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:41.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:41.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:41 smithi017 ceph-mon[27571]: pgmap v1290: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 11948/196941 objects degraded (6.067%); 8994/196941 objects misplaced (4.567%); 8.7 MiB/s, 140 objects/s recovering 2024-09-17T16:27:41.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:41 smithi017 ceph-mon[27571]: 6.5 scrub starts 2024-09-17T16:27:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:27:41.142 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:27:41.142 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-17T16:27:41.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:41.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:41.221 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:41.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:41.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:41.361 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:41.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:41.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:41 smithi173 ceph-mon[30530]: pgmap v1290: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 11948/196941 objects degraded (6.067%); 8994/196941 objects misplaced (4.567%); 8.7 MiB/s, 140 objects/s recovering 2024-09-17T16:27:41.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:41 smithi173 ceph-mon[30530]: 6.5 scrub starts 2024-09-17T16:27:41.462 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:41.512 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:41.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:41 smithi071 ceph-mon[30002]: pgmap v1290: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 11948/196941 objects degraded (6.067%); 8994/196941 objects misplaced (4.567%); 8.7 MiB/s, 140 objects/s recovering 2024-09-17T16:27:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:41 smithi071 ceph-mon[30002]: 6.5 scrub starts 2024-09-17T16:27:41.613 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:41.637 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:41.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:41.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:27:41.739 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:41.763 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:41.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:41.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:41.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:41.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:41.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:41.998 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:42.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:42.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:42.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:42.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:42.182 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:42 smithi017 ceph-mon[27571]: 6.5 scrub ok 2024-09-17T16:27:42.264 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:42.291 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:42.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:42.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:42.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:42 smithi173 ceph-mon[30530]: 6.5 scrub ok 2024-09-17T16:27:42.442 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:42.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:42.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:42.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:42 smithi071 ceph-mon[30002]: 6.5 scrub ok 2024-09-17T16:27:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:42.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:42.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:42.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:42.698 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:42.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:42.723 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:27:42.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:42.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:42.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:42.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:42.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:42.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:42.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:43.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:43.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:43.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:43.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:43.079 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:27:43.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:43.209 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:43.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:43 smithi017 ceph-mon[27571]: pgmap v1291: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 11940/190965 objects degraded (6.252%); 8587/190965 objects misplaced (4.497%); 7.6 MiB/s, 122 objects/s recovering 2024-09-17T16:27:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:43 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11940/190965 objects degraded (6.252%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3942939808' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:27:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:43 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:27:43.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:43.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:43.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:43 smithi173 ceph-mon[30530]: pgmap v1291: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 11940/190965 objects degraded (6.252%); 8587/190965 objects misplaced (4.497%); 7.6 MiB/s, 122 objects/s recovering 2024-09-17T16:27:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:43 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11940/190965 objects degraded (6.252%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3942939808' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:27:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:43 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:27:43.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:43.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:43.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:43.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:43 smithi071 ceph-mon[30002]: pgmap v1291: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 11940/190965 objects degraded (6.252%); 8587/190965 objects misplaced (4.497%); 7.6 MiB/s, 122 objects/s recovering 2024-09-17T16:27:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:43 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11940/190965 objects degraded (6.252%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3942939808' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:27:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:43 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:27:43.589 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:43.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:43.691 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:43.724 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:43.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:43.826 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:43.848 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:43.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:43.950 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:43.973 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:43.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:44.075 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:44.105 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:44.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:44.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:44.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:27:44.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:44.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:44 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:27:44.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:27:44.217 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:44 smithi017 ceph-mon[27571]: osdmap e586: 12 total, 12 up, 12 in 2024-09-17T16:27:44.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:44.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:44.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:44.357 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:44.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:44.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:44 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:27:44.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:27:44.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:44 smithi173 ceph-mon[30530]: osdmap e586: 12 total, 12 up, 12 in 2024-09-17T16:27:44.459 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:44.485 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:44.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:44.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:44 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:27:44.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:27:44.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:44 smithi071 ceph-mon[30002]: osdmap e586: 12 total, 12 up, 12 in 2024-09-17T16:27:44.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:44.618 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:44.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:44.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:44.754 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:44.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:44.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:44.856 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:44.881 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:44.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:44.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:45.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:45.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:45.109 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:45.144 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:45.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:45.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:45.284 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:45.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:45.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:45 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:27:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:45 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:27:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:45 smithi173 ceph-mon[30530]: pgmap v1293: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 11940/189249 objects degraded (6.309%); 8362/189249 objects misplaced (4.419%); 7.2 MiB/s, 116 objects/s recovering 2024-09-17T16:27:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:45 smithi173 ceph-mon[30530]: osdmap e587: 12 total, 12 up, 12 in 2024-09-17T16:27:45.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:45.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:45 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:27:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:45 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:27:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:45 smithi017 ceph-mon[27571]: pgmap v1293: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 11940/189249 objects degraded (6.309%); 8362/189249 objects misplaced (4.419%); 7.2 MiB/s, 116 objects/s recovering 2024-09-17T16:27:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:45 smithi017 ceph-mon[27571]: osdmap e587: 12 total, 12 up, 12 in 2024-09-17T16:27:45.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:45.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:45.537 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:45 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:27:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:45 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:27:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:45 smithi071 ceph-mon[30002]: pgmap v1293: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 11940/189249 objects degraded (6.309%); 8362/189249 objects misplaced (4.419%); 7.2 MiB/s, 116 objects/s recovering 2024-09-17T16:27:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:45 smithi071 ceph-mon[30002]: osdmap e587: 12 total, 12 up, 12 in 2024-09-17T16:27:45.638 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:45.663 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:45.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:45.765 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:45.795 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:45.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:45.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:45.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:45.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:45.969 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:27:46.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:46.056 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:46.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:46.158 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:46.184 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:46.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:46.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:46.329 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:46.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:46.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:46.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:46.466 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:46.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:46.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:46.591 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:46.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:46.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:46.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:46.716 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:46.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:46.818 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:46.856 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:46.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:46.958 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:46.992 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:46.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:47.094 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:47.121 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:47.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:47.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:47.260 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:47.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:47.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:47.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:47.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:47 smithi017 ceph-mon[27571]: 6.f scrub starts 2024-09-17T16:27:47.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:47 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:27:47.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:47 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:27:47.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:47 smithi017 ceph-mon[27571]: pgmap v1295: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 11939/186795 objects degraded (6.391%); 8362/186795 objects misplaced (4.477%); 4.9 MiB/s, 80 objects/s recovering 2024-09-17T16:27:47.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:47 smithi017 ceph-mon[27571]: osdmap e588: 12 total, 12 up, 12 in 2024-09-17T16:27:47.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:47.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:47.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:47 smithi173 ceph-mon[30530]: 6.f scrub starts 2024-09-17T16:27:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:47 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:27:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:47 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:27:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:47 smithi173 ceph-mon[30530]: pgmap v1295: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 11939/186795 objects degraded (6.391%); 8362/186795 objects misplaced (4.477%); 4.9 MiB/s, 80 objects/s recovering 2024-09-17T16:27:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:47 smithi173 ceph-mon[30530]: osdmap e588: 12 total, 12 up, 12 in 2024-09-17T16:27:47.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:47.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:47.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:47 smithi071 ceph-mon[30002]: 6.f scrub starts 2024-09-17T16:27:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:47 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:27:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:47 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:27:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:47 smithi071 ceph-mon[30002]: pgmap v1295: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 11939/186795 objects degraded (6.391%); 8362/186795 objects misplaced (4.477%); 4.9 MiB/s, 80 objects/s recovering 2024-09-17T16:27:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:47 smithi071 ceph-mon[30002]: osdmap e588: 12 total, 12 up, 12 in 2024-09-17T16:27:47.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:47.636 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:47.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:47.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:47.773 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:47.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:47.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:47.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:47.899 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:47.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:48.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:48.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:48.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:48.126 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:48.168 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:48.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:48.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:48.270 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:48.277 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:27:48.277 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:27:48.278 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:27:48.278 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:27:48.278 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:27:48.278 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:27:48.278 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:27:48.278 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:27:48.278 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:27:48.278 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:27:48.278 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:27:48.278 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:27:48.299 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:48.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:48.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:48 smithi173 ceph-mon[30530]: 6.f scrub ok 2024-09-17T16:27:48.400 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:48.431 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:48.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:48 smithi017 ceph-mon[27571]: 6.f scrub ok 2024-09-17T16:27:48.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:48.556 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:48.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:48.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:48 smithi071 ceph-mon[30002]: 6.f scrub ok 2024-09-17T16:27:48.571 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:27:48.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:48.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:48.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:48.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:48.828 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:48.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:48.933 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:48.963 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:48.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:49.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:49.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:49.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:49.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:49.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:49.220 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:49.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:27:49.286 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:27:49.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-17T16:27:49.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:49.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:49.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:49 smithi173 ceph-mon[30530]: pgmap v1297: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 645 op/s; 11935/179334 objects degraded (6.655%); 7723/179334 objects misplaced (4.306%); 9.0 MiB/s, 144 objects/s recovering 2024-09-17T16:27:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:49 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11935/179334 objects degraded (6.655%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:49.419 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:49 smithi017 ceph-mon[27571]: pgmap v1297: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 645 op/s; 11935/179334 objects degraded (6.655%); 7723/179334 objects misplaced (4.306%); 9.0 MiB/s, 144 objects/s recovering 2024-09-17T16:27:49.419 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:49 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11935/179334 objects degraded (6.655%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:49.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:49.483 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:49.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:49 smithi071 ceph-mon[30002]: pgmap v1297: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 645 op/s; 11935/179334 objects degraded (6.655%); 7723/179334 objects misplaced (4.306%); 9.0 MiB/s, 144 objects/s recovering 2024-09-17T16:27:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:49 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11935/179334 objects degraded (6.655%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:49.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:49.608 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:49.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:49.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:49.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:49.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:49.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:49.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:49.836 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:49.861 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:49.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:49.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:49.932 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:27:49.932 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:27:49.939 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:27:49.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:49.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:49.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:50.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:50.115 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:50.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:50.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:50.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:50.237 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:50.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:50.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:50.377 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:50.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:50.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:50.507 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:50.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:50.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:50.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:50.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:50.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:50.768 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:50.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:50.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:50 smithi071 ceph-mon[30002]: pgmap v1298: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 11935/177399 objects degraded (6.728%); 7723/177399 objects misplaced (4.353%); 5.8 MiB/s, 93 objects/s recovering 2024-09-17T16:27:50.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:50.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:50.894 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:50.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:50 smithi173 ceph-mon[30530]: pgmap v1298: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 11935/177399 objects degraded (6.728%); 7723/177399 objects misplaced (4.353%); 5.8 MiB/s, 93 objects/s recovering 2024-09-17T16:27:50.909 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:27:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:50 smithi017 ceph-mon[27571]: pgmap v1298: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 11 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 11935/177399 objects degraded (6.728%); 7723/177399 objects misplaced (4.353%); 5.8 MiB/s, 93 objects/s recovering 2024-09-17T16:27:50.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:51.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:51.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:51.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:51.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:51.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:51.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:51.279 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:51.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:51.381 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:51.407 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:51.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:51.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:51.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:51.532 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:51.632 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:51.665 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:51.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:51.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:51.796 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:51.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:51 smithi071 ceph-mon[30002]: 6.3 scrub starts 2024-09-17T16:27:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:51 smithi173 ceph-mon[30530]: 6.3 scrub starts 2024-09-17T16:27:51.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:51.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:51.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:51 smithi017 ceph-mon[27571]: 6.3 scrub starts 2024-09-17T16:27:52.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:52.045 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:52.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:52.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:52.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:52.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:52.269 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:52.293 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:52.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:52.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:52.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:52.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:52.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:52.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:52.548 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:52.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:52.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:52.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:52.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:52.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:52.773 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:52.805 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:52.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:52.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:52 smithi173 ceph-mon[30530]: 6.3 scrub ok 2024-09-17T16:27:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:52 smithi173 ceph-mon[30530]: pgmap v1299: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s; 11934/171270 objects degraded (6.968%); 7281/171270 objects misplaced (4.251%); 8.4 MiB/s, 135 objects/s recovering 2024-09-17T16:27:52.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:52.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:52.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:52.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:52 smithi017 ceph-mon[27571]: 6.3 scrub ok 2024-09-17T16:27:52.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:52 smithi017 ceph-mon[27571]: pgmap v1299: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s; 11934/171270 objects degraded (6.968%); 7281/171270 objects misplaced (4.251%); 8.4 MiB/s, 135 objects/s recovering 2024-09-17T16:27:53.032 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:53.057 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:53.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:53.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:52 smithi071 ceph-mon[30002]: 6.3 scrub ok 2024-09-17T16:27:53.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:52 smithi071 ceph-mon[30002]: pgmap v1299: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s; 11934/171270 objects degraded (6.968%); 7281/171270 objects misplaced (4.251%); 8.4 MiB/s, 135 objects/s recovering 2024-09-17T16:27:53.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:53.192 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:53.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:53.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:53.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:53.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:53.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:53.448 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:53.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:53.550 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:53.587 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:53.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:53.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:53.712 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:53.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:53.713 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:27:53.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:27:53.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:53.843 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:53.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:53.945 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:53.994 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:53.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:54.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:54.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:54.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:54.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:54.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:54.247 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:54.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:54.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:54.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:54.376 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:54.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:54.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:54.505 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:54.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:54.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:54.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:54.621 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:54 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:27:54.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:54 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:27:54.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:54 smithi017 ceph-mon[27571]: osdmap e589: 12 total, 12 up, 12 in 2024-09-17T16:27:54.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:54 smithi017 ceph-mon[27571]: pgmap v1301: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 11934/169611 objects degraded (7.036%); 7075/169611 objects misplaced (4.171%); 10 MiB/s, 161 objects/s recovering 2024-09-17T16:27:54.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:54 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11934/171270 objects degraded (6.968%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:54.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:54.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:54.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:54.709 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:27:54.709 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:27:54.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:54.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:27:54.744 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:27:54.765 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:54.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:54.777 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:27:54.778 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:27:54.810 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:27:54.810 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:27:54.836 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:27:54.836 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:27:54.865 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:27:54.865 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:27:54.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:54.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:54.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:54.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:54 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:27:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:54 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:27:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:54 smithi173 ceph-mon[30530]: osdmap e589: 12 total, 12 up, 12 in 2024-09-17T16:27:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:54 smithi173 ceph-mon[30530]: pgmap v1301: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 11934/169611 objects degraded (7.036%); 7075/169611 objects misplaced (4.171%); 10 MiB/s, 161 objects/s recovering 2024-09-17T16:27:54.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:54 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11934/171270 objects degraded (6.968%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:54.897 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:27:54.897 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:27:54.934 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:27:54.934 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:27:54.962 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:27:54.962 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:27:54.987 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:27:54.987 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:27:54.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:55.016 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:27:55.016 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:27:55.020 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:55.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:55.053 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:27:55.053 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:27:55.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:54 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:27:55.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:54 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:27:55.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:54 smithi071 ceph-mon[30002]: osdmap e589: 12 total, 12 up, 12 in 2024-09-17T16:27:55.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:54 smithi071 ceph-mon[30002]: pgmap v1301: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 11934/169611 objects degraded (7.036%); 7075/169611 objects misplaced (4.171%); 10 MiB/s, 161 objects/s recovering 2024-09-17T16:27:55.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:54 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11934/171270 objects degraded (6.968%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:55.123 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:55.153 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:55.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:55.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:55.256 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:55.285 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:55.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:55.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:55.410 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:55.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:55.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:55.538 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:55.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:55.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:55.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:55.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:55.765 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:55.792 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:55.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:55.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:55.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2683089904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:27:55.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:55 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:27:55.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:55.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:55.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:55.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2683089904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:27:55.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:55 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:27:56.019 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:56.055 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:56.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:56.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2683089904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:27:56.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:55 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:27:56.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:56.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:56.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:56.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:56.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:56.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:56.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:27:56.435 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:56.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:27:56.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:56.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:56.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:56.582 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:27:56.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:56.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:56.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:56.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:56 smithi017 ceph-mon[27571]: pgmap v1302: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 11932/167226 objects degraded (7.135%); 7075/167226 objects misplaced (4.231%); 8.2 MiB/s, 131 objects/s recovering 2024-09-17T16:27:56.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/541358963' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:27:56.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:56.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:56.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:56 smithi173 ceph-mon[30530]: pgmap v1302: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 11932/167226 objects degraded (7.135%); 7075/167226 objects misplaced (4.231%); 8.2 MiB/s, 131 objects/s recovering 2024-09-17T16:27:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/541358963' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:27:56.911 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:56.946 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:56.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:56.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:57.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:27:57.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:56 smithi071 ceph-mon[30002]: pgmap v1302: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 11932/167226 objects degraded (7.135%); 7075/167226 objects misplaced (4.231%); 8.2 MiB/s, 131 objects/s recovering 2024-09-17T16:27:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/541358963' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:27:57.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:57.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:27:57.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:57.208 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:57.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:57.310 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:27:57.338 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:57.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:27:57.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:57.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:57.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:57.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:57.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:57.596 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:57.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:57.697 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:57.732 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:57.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:57.834 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:57.871 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:57.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:57 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:27:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/541358963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:27:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:57 smithi173 ceph-mon[30530]: osdmap e590: 12 total, 12 up, 12 in 2024-09-17T16:27:57.900 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:57 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:27:57.900 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/541358963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:27:57.900 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:57 smithi017 ceph-mon[27571]: osdmap e590: 12 total, 12 up, 12 in 2024-09-17T16:27:57.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:27:57.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:57.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:27:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:57 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:27:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/541358963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:27:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:57 smithi071 ceph-mon[30002]: osdmap e590: 12 total, 12 up, 12 in 2024-09-17T16:27:58.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:58.117 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:58.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:58.219 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:27:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:58.257 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:58.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:27:58.360 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:27:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:58.388 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:58.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:27:58.490 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:58.523 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:58.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:58.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:58.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:58.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:58.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:27:58.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:58.775 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:27:58.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:27:58.876 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:27:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:27:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:58 smithi173 ceph-mon[30530]: osdmap e591: 12 total, 12 up, 12 in 2024-09-17T16:27:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:58 smithi173 ceph-mon[30530]: pgmap v1305: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 11926/159780 objects degraded (7.464%); 6336/159780 objects misplaced (3.965%); 9.8 MiB/s, 158 objects/s recovering 2024-09-17T16:27:58.929 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:58.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:27:58.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:27:58.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:58 smithi017 ceph-mon[27571]: osdmap e591: 12 total, 12 up, 12 in 2024-09-17T16:27:58.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:58 smithi017 ceph-mon[27571]: pgmap v1305: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 11926/159780 objects degraded (7.464%); 6336/159780 objects misplaced (3.965%); 9.8 MiB/s, 158 objects/s recovering 2024-09-17T16:27:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:58.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:59.012 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:27:59.016 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:27:59.016 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:27:59.016 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:27:59.016 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:27:59.016 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:27:59.016 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:27:59.016 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:27:59.017 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:27:59.017 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:27:59.017 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:27:59.017 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:27:59.017 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:27:59.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:59.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:27:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:27:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:58 smithi071 ceph-mon[30002]: osdmap e591: 12 total, 12 up, 12 in 2024-09-17T16:27:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:58 smithi071 ceph-mon[30002]: pgmap v1305: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 11926/159780 objects degraded (7.464%); 6336/159780 objects misplaced (3.965%); 9.8 MiB/s, 158 objects/s recovering 2024-09-17T16:27:59.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:59.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:27:59.200 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:27:59.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:59.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:27:59.337 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:59.371 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:59.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:59.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:27:59.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:59.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:27:59.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:27:59.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:27:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:59.619 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:27:59.665 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:27:59.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:27:59.765 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:59.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:27:59.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:27:59.893 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:27:59 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11926/159780 objects degraded (7.464%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:59.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:27:59 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11926/159780 objects degraded (7.464%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:27:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:27:59.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:27:59.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:00.032 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:00.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:27:59 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11926/159780 objects degraded (7.464%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:00.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:00.094 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:00.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:00.103 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:28:00.103 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:28:00.103 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:28:00.103 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:28:00.103 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:28:00.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:00.220 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:00.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:00.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:00.355 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:00.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:00.457 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:00.480 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:00.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:00.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:00.582 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:00.619 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:00.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:00.654 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:28:00.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:00.746 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:00.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:00.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:00.891 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:00.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:00.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:00 smithi173 ceph-mon[30530]: pgmap v1306: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 613 op/s; 11926/157851 objects degraded (7.555%); 6336/157851 objects misplaced (4.014%); 7.2 MiB/s, 116 objects/s recovering 2024-09-17T16:28:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:00 smithi017 ceph-mon[27571]: pgmap v1306: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 613 op/s; 11926/157851 objects degraded (7.555%); 6336/157851 objects misplaced (4.014%); 7.2 MiB/s, 116 objects/s recovering 2024-09-17T16:28:00.993 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:01.028 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:01.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:00 smithi071 ceph-mon[30002]: pgmap v1306: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 613 op/s; 11926/157851 objects degraded (7.555%); 6336/157851 objects misplaced (4.014%); 7.2 MiB/s, 116 objects/s recovering 2024-09-17T16:28:01.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:01.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:01.151 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:01.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:01.280 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:01.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:01.382 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:01.414 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:01.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:01.516 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:01.546 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:01.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:01.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:01.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:01.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:01.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:01.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:01.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:01.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:01.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:01 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:28:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:01 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:28:01.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:01.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:01.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:01.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:01.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:01 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:28:01.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:01 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:28:02.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:02.046 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:02.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:02.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:01 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:28:02.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:01 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:28:02.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:02.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:02.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:02.271 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:02.314 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:02.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:02.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:02.416 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:02.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:02.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:02.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:02.551 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:02.585 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:02.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:02.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:02.688 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:02.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:02.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:02.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:02.852 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:02.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:02.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:02 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:28:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:02 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:28:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:02 smithi173 ceph-mon[30530]: osdmap e592: 12 total, 12 up, 12 in 2024-09-17T16:28:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:02 smithi173 ceph-mon[30530]: pgmap v1308: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s; 11920/151266 objects degraded (7.880%); 5915/151266 objects misplaced (3.910%); 12 MiB/s, 195 objects/s recovering 2024-09-17T16:28:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:02 smithi173 ceph-mon[30530]: osdmap e593: 12 total, 12 up, 12 in 2024-09-17T16:28:02.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:02.985 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:02.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:02 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:28:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:02 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:28:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:02 smithi017 ceph-mon[27571]: osdmap e592: 12 total, 12 up, 12 in 2024-09-17T16:28:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:02 smithi017 ceph-mon[27571]: pgmap v1308: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s; 11920/151266 objects degraded (7.880%); 5915/151266 objects misplaced (3.910%); 12 MiB/s, 195 objects/s recovering 2024-09-17T16:28:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:02 smithi017 ceph-mon[27571]: osdmap e593: 12 total, 12 up, 12 in 2024-09-17T16:28:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:02 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:28:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:02 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:28:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:02 smithi071 ceph-mon[30002]: osdmap e592: 12 total, 12 up, 12 in 2024-09-17T16:28:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:02 smithi071 ceph-mon[30002]: pgmap v1308: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s; 11920/151266 objects degraded (7.880%); 5915/151266 objects misplaced (3.910%); 12 MiB/s, 195 objects/s recovering 2024-09-17T16:28:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:02 smithi071 ceph-mon[30002]: osdmap e593: 12 total, 12 up, 12 in 2024-09-17T16:28:03.087 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:03.117 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:03.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:03.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:03.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:03.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:03.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:03.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:03.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:03.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:03.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:03.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:03.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:03.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:03.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:03.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:03.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:03.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:03.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:03.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:03.841 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:03.866 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:03.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:03.968 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:04.004 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:04.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:04.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:04.137 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:04.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:04.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:04.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:04.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:04.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:04.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:04.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:04.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:04.514 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:04.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:04.616 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:04.656 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:04.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:04.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:04.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:04.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:04.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:04.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:04.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:04.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:05.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:05.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:05.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:05.138 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:05.164 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:05.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:05.266 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:05.293 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:05.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:05.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:05 smithi071 ceph-mon[30002]: pgmap v1310: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 11920/149466 objects degraded (7.975%); 5679/149466 objects misplaced (3.800%); 6.4 MiB/s, 104 objects/s recovering 2024-09-17T16:28:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:05 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11920/149466 objects degraded (7.975%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:05 smithi071 ceph-mon[30002]: osdmap e594: 12 total, 12 up, 12 in 2024-09-17T16:28:05.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:05 smithi173 ceph-mon[30530]: pgmap v1310: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 11920/149466 objects degraded (7.975%); 5679/149466 objects misplaced (3.800%); 6.4 MiB/s, 104 objects/s recovering 2024-09-17T16:28:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:05 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11920/149466 objects degraded (7.975%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:05 smithi173 ceph-mon[30530]: osdmap e594: 12 total, 12 up, 12 in 2024-09-17T16:28:05.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:05.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:05.419 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:05 smithi017 ceph-mon[27571]: pgmap v1310: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 11920/149466 objects degraded (7.975%); 5679/149466 objects misplaced (3.800%); 6.4 MiB/s, 104 objects/s recovering 2024-09-17T16:28:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:05 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11920/149466 objects degraded (7.975%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:05 smithi017 ceph-mon[27571]: osdmap e594: 12 total, 12 up, 12 in 2024-09-17T16:28:05.519 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:05.543 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:05.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:05.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:05.680 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:05.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:05.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:05.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:05.821 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:05.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:05.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:05.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:05.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:05.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:06.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:06.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:06.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:06.179 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:28:06.199 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:28:06.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-17T16:28:06.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:06.241 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:06.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:06.344 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:06.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:28:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:06.444 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:06.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:06.501 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:06.545 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:06.570 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:06.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:06.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:06.672 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:06.726 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:06.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:06.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:06.898 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:06.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:06.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:07.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:07.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:07.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:07.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:07.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:07.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:07.209 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:07.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:07 smithi017 ceph-mon[27571]: pgmap v1312: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 11920/146757 objects degraded (8.122%); 5679/146757 objects misplaced (3.870%); 6.8 MiB/s, 110 objects/s recovering 2024-09-17T16:28:07.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:07 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:28:07.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:07 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:28:07.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:07 smithi017 ceph-mon[27571]: osdmap e595: 12 total, 12 up, 12 in 2024-09-17T16:28:07.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:07.271 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:07.288 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:07.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:07 smithi071 ceph-mon[30002]: pgmap v1312: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 11920/146757 objects degraded (8.122%); 5679/146757 objects misplaced (3.870%); 6.8 MiB/s, 110 objects/s recovering 2024-09-17T16:28:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:07 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:28:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:07 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:28:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:07 smithi071 ceph-mon[30002]: osdmap e595: 12 total, 12 up, 12 in 2024-09-17T16:28:07.314 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:07.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:07 smithi173 ceph-mon[30530]: pgmap v1312: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 11920/146757 objects degraded (8.122%); 5679/146757 objects misplaced (3.870%); 6.8 MiB/s, 110 objects/s recovering 2024-09-17T16:28:07.417 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:07 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:28:07.417 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:07 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:28:07.417 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:07 smithi173 ceph-mon[30530]: osdmap e595: 12 total, 12 up, 12 in 2024-09-17T16:28:07.417 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:07.441 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:07.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:07.543 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:07.568 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:07.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:07.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:07.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:07.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:07.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:07.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:07.818 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:07.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:07.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:07.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:08.044 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:08.072 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:08.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:08.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:08.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:08.218 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:08.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:08.315 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:28:08.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:08.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:08.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:08.440 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:28:08.440 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:28:08.440 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:28:08.440 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:28:08.441 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:28:08.441 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:28:08.441 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:28:08.441 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:28:08.441 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:28:08.441 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:28:08.441 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:28:08.441 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:28:08.444 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:08.467 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:08.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:08 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:28:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:08 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:28:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3186390666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2061155803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:08.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:08.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:08.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:08 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:28:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:08 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:28:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3186390666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2061155803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:08.693 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:08.736 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:08.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:08 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:28:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:08 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:28:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3186390666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:08.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2061155803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:08.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:08.872 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:08.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:08.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:09.001 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:09.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:09.008 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:28:09.009 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:28:09.033 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:28:09.033 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:28:09.054 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:28:09.054 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:28:09.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:09.116 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:28:09.116 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:28:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:09.136 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:09.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:09.147 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:28:09.147 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:28:09.175 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:28:09.175 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:28:09.205 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:28:09.206 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:28:09.235 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:28:09.236 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:28:09.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:09.260 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:28:09.261 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:28:09.264 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:09.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:09.291 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:28:09.291 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:28:09.322 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:28:09.322 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:28:09.348 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:28:09.349 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:28:09.366 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:09.398 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:09.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:09.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:09.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:09.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:09 smithi071 ceph-mon[30002]: pgmap v1314: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 11918/139032 objects degraded (8.572%); 5000/139032 objects misplaced (3.596%); 9.5 MiB/s, 152 objects/s recovering 2024-09-17T16:28:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:09 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3186390666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:28:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:09 smithi071 ceph-mon[30002]: osdmap e596: 12 total, 12 up, 12 in 2024-09-17T16:28:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:28:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:09 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:28:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:09 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:28:09.625 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:09 smithi173 ceph-mon[30530]: pgmap v1314: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 11918/139032 objects degraded (8.572%); 5000/139032 objects misplaced (3.596%); 9.5 MiB/s, 152 objects/s recovering 2024-09-17T16:28:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:09 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3186390666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:28:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:09 smithi173 ceph-mon[30530]: osdmap e596: 12 total, 12 up, 12 in 2024-09-17T16:28:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:28:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:09 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:28:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:09 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:28:09.656 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:09.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:09 smithi017 ceph-mon[27571]: pgmap v1314: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 679 op/s; 11918/139032 objects degraded (8.572%); 5000/139032 objects misplaced (3.596%); 9.5 MiB/s, 152 objects/s recovering 2024-09-17T16:28:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:09 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3186390666' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:28:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:09 smithi017 ceph-mon[27571]: osdmap e596: 12 total, 12 up, 12 in 2024-09-17T16:28:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:28:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:09 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:28:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:09 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:28:09.758 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:09.780 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:09.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:09.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:09.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:09.909 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:09.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:09.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:10.011 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:10.039 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:10.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:10.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:10.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:10.167 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:10.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:10.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:10.269 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:10.299 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:10.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:10.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:10.433 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:10.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:10.534 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11918/139032 objects degraded (8.572%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:10 smithi071 ceph-mon[30002]: osdmap e597: 12 total, 12 up, 12 in 2024-09-17T16:28:10.565 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:10.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11918/139032 objects degraded (8.572%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:10 smithi173 ceph-mon[30530]: osdmap e597: 12 total, 12 up, 12 in 2024-09-17T16:28:10.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:10.667 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11918/139032 objects degraded (8.572%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:10.668 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:10 smithi017 ceph-mon[27571]: osdmap e597: 12 total, 12 up, 12 in 2024-09-17T16:28:10.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:10.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:10.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:10.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:10.824 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:10.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:10.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:10.961 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:10.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:10.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:11.063 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:11.096 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:11.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:11.197 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:11.225 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:11.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:11.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:11.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:11.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:11.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:11.396 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:11 smithi017 ceph-mon[27571]: pgmap v1317: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 11918/137307 objects degraded (8.680%); 5000/137307 objects misplaced (3.641%); 7.8 MiB/s, 125 objects/s recovering 2024-09-17T16:28:11.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:11.479 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:11.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:11.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:11 smithi071 ceph-mon[30002]: pgmap v1317: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 11918/137307 objects degraded (8.680%); 5000/137307 objects misplaced (3.641%); 7.8 MiB/s, 125 objects/s recovering 2024-09-17T16:28:11.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:11.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:11.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:11 smithi173 ceph-mon[30530]: pgmap v1317: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 711 op/s; 11918/137307 objects degraded (8.680%); 5000/137307 objects misplaced (3.641%); 7.8 MiB/s, 125 objects/s recovering 2024-09-17T16:28:11.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:11.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:11.728 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:11.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:11.859 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:11.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:11.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:11.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:11.987 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:11.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:11.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:12.090 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:12.118 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:12.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:12.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:12.257 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:12.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:12.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:12.383 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:12.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:12.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:12.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:12.515 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:12.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:12.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:12.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:12.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:12.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:12.766 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:12.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:12.868 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:12.903 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:12.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:13.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:13.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:13.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:28:13.032 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:28:13.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-17T16:28:13.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:13.154 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:13.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:13.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:28:13.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:13.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:13.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:13.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:13.384 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:13.384 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:13.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:13.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:13.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:13.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:13.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:13.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:13 smithi071 ceph-mon[30002]: pgmap v1318: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s; 11913/131355 objects degraded (9.069%); 4525/131355 objects misplaced (3.445%); 12 MiB/s, 193 objects/s recovering 2024-09-17T16:28:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:13 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:28:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:13 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:28:13.627 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:13 smithi173 ceph-mon[30530]: pgmap v1318: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s; 11913/131355 objects degraded (9.069%); 4525/131355 objects misplaced (3.445%); 12 MiB/s, 193 objects/s recovering 2024-09-17T16:28:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:13 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:28:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:13 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:28:13.671 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:13.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:13 smithi017 ceph-mon[27571]: pgmap v1318: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 855 op/s; 11913/131355 objects degraded (9.069%); 4525/131355 objects misplaced (3.445%); 12 MiB/s, 193 objects/s recovering 2024-09-17T16:28:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:13 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:28:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:13 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:28:13.773 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:13.812 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:13.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:13.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:13.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:13.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:14.039 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:14.067 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:14.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:14.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:14.099 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:14.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:28:14.156 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:14.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:14.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:14.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:14.213 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:28:14.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:14.339 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:14.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:14.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:14.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:14.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:14.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:14.588 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:14 smithi017 ceph-mon[27571]: osdmap e598: 12 total, 12 up, 12 in 2024-09-17T16:28:14.592 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:14.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:14 smithi173 ceph-mon[30530]: osdmap e598: 12 total, 12 up, 12 in 2024-09-17T16:28:14.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:14.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:14.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:14 smithi071 ceph-mon[30002]: osdmap e598: 12 total, 12 up, 12 in 2024-09-17T16:28:14.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:14.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:14.842 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:14.921 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:28:14.921 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:28:14.921 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:28:14.921 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:28:14.921 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:28:14.921 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:28:14.921 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:28:14.921 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:28:14.921 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:28:14.922 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:28:14.922 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:28:14.922 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:28:14.943 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:14.992 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:14.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:14.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:15.094 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:15.124 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:15.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:15.225 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:15.250 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:15.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:15.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:15.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:15.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:15.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:15.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:15.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:15.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:15.603 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:15.628 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:15.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:15.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:15 smithi173 ceph-mon[30530]: pgmap v1320: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 11913/129549 objects degraded (9.196%); 4233/129549 objects misplaced (3.267%); 8.0 MiB/s, 128 objects/s recovering 2024-09-17T16:28:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:15 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11913/129549 objects degraded (9.196%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:15 smithi173 ceph-mon[30530]: osdmap e599: 12 total, 12 up, 12 in 2024-09-17T16:28:15.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:15 smithi017 ceph-mon[27571]: pgmap v1320: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 11913/129549 objects degraded (9.196%); 4233/129549 objects misplaced (3.267%); 8.0 MiB/s, 128 objects/s recovering 2024-09-17T16:28:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:15 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11913/129549 objects degraded (9.196%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:15 smithi017 ceph-mon[27571]: osdmap e599: 12 total, 12 up, 12 in 2024-09-17T16:28:15.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:15.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:15 smithi071 ceph-mon[30002]: pgmap v1320: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 11913/129549 objects degraded (9.196%); 4233/129549 objects misplaced (3.267%); 8.0 MiB/s, 128 objects/s recovering 2024-09-17T16:28:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:15 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11913/129549 objects degraded (9.196%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:15 smithi071 ceph-mon[30002]: osdmap e599: 12 total, 12 up, 12 in 2024-09-17T16:28:15.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:15.877 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:15.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:15.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:28:15.912 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:28:15.912 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-09-17T16:28:15.948 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:15.950 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:15.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:16.002 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:16.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:16.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:16.147 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:16.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:16.249 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:16.294 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:16.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:16.336 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:28:16.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3973331944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:28:16.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:16.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:16.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:16.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:16.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:16.550 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:16.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3973331944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:28:16.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:16.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:16.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:16.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:16.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:16.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:16.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:16.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3973331944' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:28:16.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:16.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:16.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:16.935 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:16.935 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:17.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:17.059 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:17.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:17.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:17.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:17.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:17.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:17.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:17.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:17.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:17.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:17.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:17.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:17.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:17.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:17.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:17.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:17 smithi017 ceph-mon[27571]: pgmap v1322: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 11913/127035 objects degraded (9.378%); 4233/127035 objects misplaced (3.332%); 7.5 MiB/s, 120 objects/s recovering 2024-09-17T16:28:17.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:17 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:28:17.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:17 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:28:17.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:17 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:17.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3973331944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:28:17.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:17 smithi017 ceph-mon[27571]: osdmap e600: 12 total, 12 up, 12 in 2024-09-17T16:28:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:17 smithi173 ceph-mon[30530]: pgmap v1322: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 11913/127035 objects degraded (9.378%); 4233/127035 objects misplaced (3.332%); 7.5 MiB/s, 120 objects/s recovering 2024-09-17T16:28:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:17 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:28:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:17 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:28:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:17 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3973331944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:28:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:17 smithi173 ceph-mon[30530]: osdmap e600: 12 total, 12 up, 12 in 2024-09-17T16:28:17.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:17.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:17.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:17.717 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:28:17.717 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:28:17.758 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:28:17.759 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:28:17.781 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:17.793 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:28:17.793 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:28:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:17 smithi071 ceph-mon[30002]: pgmap v1322: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 11913/127035 objects degraded (9.378%); 4233/127035 objects misplaced (3.332%); 7.5 MiB/s, 120 objects/s recovering 2024-09-17T16:28:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:17 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:28:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:17 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:28:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:17 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3973331944' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:28:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:17 smithi071 ceph-mon[30002]: osdmap e600: 12 total, 12 up, 12 in 2024-09-17T16:28:17.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:17.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:17.835 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:28:17.835 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:28:17.869 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:28:17.869 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:28:17.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:28:17.898 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:28:17.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:17.935 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:28:17.935 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:28:17.950 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:17.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:17.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:28:17.971 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:28:18.010 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:28:18.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:28:18.045 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:28:18.046 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:28:18.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:18.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:18.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:18.085 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:28:18.086 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:28:18.125 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:28:18.126 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:28:18.176 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:18.223 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:18.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:18.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:18.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:18.353 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:18.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:18.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:18.454 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:18.486 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:18.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:18.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:18.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:18.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/247805196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:18.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:18.736 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:18.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:18.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/247805196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/247805196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:18.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:18.870 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:18.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:18.956 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:18.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:18.985 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:28:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:18.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:18.995 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:19.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:19.142 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:19.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:19.243 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:19.280 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:19.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:19.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:19.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:19.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:19.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:28:19.505 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:19.538 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:19 smithi017 ceph-mon[27571]: pgmap v1324: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 680 op/s; 11911/119106 objects degraded (10.000%); 3440/119106 objects misplaced (2.888%); 11 MiB/s, 181 objects/s recovering 2024-09-17T16:28:19.555 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:19.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:19.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:19.632 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:28:19.632 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:28:19.632 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:28:19.632 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:28:19.633 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:28:19.633 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:28:19.633 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:28:19.633 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:28:19.633 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:28:19.633 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:28:19.633 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:28:19.633 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:28:19.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:19 smithi173 ceph-mon[30530]: pgmap v1324: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 680 op/s; 11911/119106 objects degraded (10.000%); 3440/119106 objects misplaced (2.888%); 11 MiB/s, 181 objects/s recovering 2024-09-17T16:28:19.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:19.692 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:19.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:19.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:19.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:19.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:19 smithi071 ceph-mon[30002]: pgmap v1324: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 680 op/s; 11911/119106 objects degraded (10.000%); 3440/119106 objects misplaced (2.888%); 11 MiB/s, 181 objects/s recovering 2024-09-17T16:28:19.820 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:19.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:19.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:19.953 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:19.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:20.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:20.085 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:20.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:20.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:20.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:20.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:20.276 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:20.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:20.338 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:20.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:20.421 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:20 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11911/119106 objects degraded (10.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:20.421 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:20 smithi017 ceph-mon[27571]: osdmap e601: 12 total, 12 up, 12 in 2024-09-17T16:28:20.440 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:20.471 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:20.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:20.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:20.601 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:20.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:20 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11911/119106 objects degraded (10.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:20 smithi173 ceph-mon[30530]: osdmap e601: 12 total, 12 up, 12 in 2024-09-17T16:28:20.703 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:20.731 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:20.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:20 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11911/119106 objects degraded (10.000%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:20 smithi071 ceph-mon[30002]: osdmap e601: 12 total, 12 up, 12 in 2024-09-17T16:28:20.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:20.854 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:20.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:20.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:20.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:20.897 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:28:20.898 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:28:20.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-09-17T16:28:20.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:20.989 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:20.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:21.091 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:21.125 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:21.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:21.227 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:21.258 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:21.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:21.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:21.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:21.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:21.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:21.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:21.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:21.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:21.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:21.633 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:21 smithi017 ceph-mon[27571]: pgmap v1326: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 11911/117147 objects degraded (10.168%); 3440/117147 objects misplaced (2.936%); 8.3 MiB/s, 132 objects/s recovering 2024-09-17T16:28:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:21.637 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:21.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:21 smithi173 ceph-mon[30530]: pgmap v1326: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 11911/117147 objects degraded (10.168%); 3440/117147 objects misplaced (2.936%); 8.3 MiB/s, 132 objects/s recovering 2024-09-17T16:28:21.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:21.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:21.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:21.773 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:28:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:28:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:28:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:28:21.773 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:28:21.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:21 smithi071 ceph-mon[30002]: pgmap v1326: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 11911/117147 objects degraded (10.168%); 3440/117147 objects misplaced (2.936%); 8.3 MiB/s, 132 objects/s recovering 2024-09-17T16:28:21.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:21.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:21.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:21.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:22.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:22.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:22.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:22.135 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:22.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:22.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:22.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:22.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:22.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:22.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:28:22.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:22.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:22.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:22.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:22.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:22.529 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:22.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:22.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:28:22.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:22.661 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:22.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:22.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:22.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:22.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:22.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:22.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:22.912 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:22.913 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:22.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:23.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:23.037 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:23.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:23.139 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:23.173 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:23.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:23.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:23.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:23.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:23.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:23.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:23.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:23.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:23.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:23.546 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:23.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:23.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:23 smithi173 ceph-mon[30530]: pgmap v1327: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 11902/110706 objects degraded (10.751%); 2747/110706 objects misplaced (2.481%); 13 MiB/s, 202 objects/s recovering 2024-09-17T16:28:23.648 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:23.679 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:23.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:23.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:23 smithi017 ceph-mon[27571]: pgmap v1327: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 11902/110706 objects degraded (10.751%); 2747/110706 objects misplaced (2.481%); 13 MiB/s, 202 objects/s recovering 2024-09-17T16:28:23.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:23.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:23.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:23.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:23 smithi071 ceph-mon[30002]: pgmap v1327: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s; 11902/110706 objects degraded (10.751%); 2747/110706 objects misplaced (2.481%); 13 MiB/s, 202 objects/s recovering 2024-09-17T16:28:23.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:23.930 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:23.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:24.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:24.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:24.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:24.155 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:24.186 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:24.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:24.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:24.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:24.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:24.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:24.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:24.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:24.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:24.565 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:24.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:24.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:24.689 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:24.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:24.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:24.813 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:24.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:24.866 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:24.866 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:24.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:24.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:24.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:25.039 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:25.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:25.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:25.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:25.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:25.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:25.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:25.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:25.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:25.436 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:25.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:25.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:25.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:25.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:25.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:25.564 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:28:25.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:25 smithi017 ceph-mon[27571]: pgmap v1328: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 11902/108915 objects degraded (10.928%); 2433/108915 objects misplaced (2.234%); 14 MiB/s, 226 objects/s recovering 2024-09-17T16:28:25.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:25 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11902/108915 objects degraded (10.928%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:25.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:25 smithi017 ceph-mon[27571]: osdmap e602: 12 total, 12 up, 12 in 2024-09-17T16:28:25.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/560553352' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:25.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:25 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:25 smithi173 ceph-mon[30530]: pgmap v1328: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 11902/108915 objects degraded (10.928%); 2433/108915 objects misplaced (2.234%); 14 MiB/s, 226 objects/s recovering 2024-09-17T16:28:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:25 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11902/108915 objects degraded (10.928%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:25 smithi173 ceph-mon[30530]: osdmap e602: 12 total, 12 up, 12 in 2024-09-17T16:28:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/560553352' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:25.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:25 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:25.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:25.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:25.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:28:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:28:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:28:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:28:25.761 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:28:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:28:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:28:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:28:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:28:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:28:25.762 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:28:25.763 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:28:25.767 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:28:25.767 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:28:25.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:28:25.767 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:102161 4.9 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 553.95754464699996, 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.902645841, 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:28:25.768 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 102161 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:28:25.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508645+0000", 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8700000000000005e-07 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508680+0000", 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4382999999999999e-05 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508688+0000", 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5059999999999995e-06 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508699+0000", 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0781000000000001e-05 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508757+0000", 2024-09-17T16:28:25.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8026999999999999e-05 2024-09-17T16:28:25.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.508920+0000", 2024-09-17T16:28:25.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163308 2024-09-17T16:28:25.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:28:25.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.509034+0000", 2024-09-17T16:28:25.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113961 2024-09-17T16:28:25.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.512221+0000", 2024-09-17T16:28:25.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003186805 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:11.515116+0000", 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002895299 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411212+0000", 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89609538399999999 2024-09-17T16:28:25.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411258+0000", 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6217999999999997e-05 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:19:12.411290+0000", 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2481999999999998e-05 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204335 4.9 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 421.85949790199999, 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77672725300000001, 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:28:25.776 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204335 2024-09-17T16:28:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:28:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:28:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606692+0000", 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2600000000000002e-07 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606696+0000", 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7170000000000002e-06 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606707+0000", 2024-09-17T16:28:25.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1486e-05 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606712+0000", 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1560000000000003e-06 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606769+0000", 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6836000000000003e-05 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606856+0000", 2024-09-17T16:28:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6749999999999994e-05 2024-09-17T16:28:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-17T16:28:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606942+0000", 2024-09-17T16:28:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6001999999999994e-05 2024-09-17T16:28:25.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611141+0000", 2024-09-17T16:28:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041987810000000004 2024-09-17T16:28:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611206+0000", 2024-09-17T16:28:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5581999999999997e-05 2024-09-17T16:28:25.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611481+0000", 2024-09-17T16:28:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027492400000000001 2024-09-17T16:28:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611502+0000", 2024-09-17T16:28:25.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0971999999999998e-05 2024-09-17T16:28:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383271+0000", 2024-09-17T16:28:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77176877600000005 2024-09-17T16:28:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383334+0000", 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2928000000000005e-05 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383419+0000", 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4617000000000007e-05 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17641 6.7 6.aff5e7f7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 183.344834048, 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86676560000000002, 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17641 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.787 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121357+0000", 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4020000000000001e-06 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121396+0000", 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9561999999999998e-05 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121410+0000", 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3645000000000001e-05 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121415+0000", 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6169999999999998e-06 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121465+0000", 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9648000000000003e-05 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121474+0000", 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.568e-06 2024-09-17T16:28:25.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988093+0000", 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86661935700000003 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988121+0000", 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7801e-05 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17642 6.7 6.5a462497 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 183.272048418, 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79401586899999999, 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.792 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17642 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194142+0000", 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2399999999999997e-07 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194163+0000", 2024-09-17T16:28:25.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1841e-05 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194175+0000", 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1487e-05 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194180+0000", 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2639999999999999e-06 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194227+0000", 2024-09-17T16:28:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6675000000000003e-05 2024-09-17T16:28:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:28:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194233+0000", 2024-09-17T16:28:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1229999999999997e-06 2024-09-17T16:28:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988148+0000", 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79391546800000001 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988157+0000", 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1869999999999997e-06 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90607 6.c 6.63d0154c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:33.678865+0000", 2024-09-17T16:28:25.797 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 111.787324032, 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.84340117199999998, 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90607 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678865+0000", 2024-09-17T16:28:25.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678865+0000", 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678866+0000", 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3600000000000003e-07 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678922+0000", 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6008999999999999e-05 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678931+0000", 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1060000000000006e-06 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678935+0000", 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2250000000000002e-06 2024-09-17T16:28:25.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678988+0000", 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2604e-05 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678995+0000", 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2230000000000001e-06 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.522239+0000", 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.84324350999999997 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.522266+0000", 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7759e-05 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.802 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115483 6.b 6:d60f373c:::benchmark_data_smithi173_102017_object22335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.040530+0000", 2024-09-17T16:28:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.425658788999996, 2024-09-17T16:28:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.027404495000001, 2024-09-17T16:28:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.803 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115483 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040530+0000", 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040530+0000", 2024-09-17T16:28:25.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040531+0000", 2024-09-17T16:28:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5000000000000002e-07 2024-09-17T16:28:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040535+0000", 2024-09-17T16:28:25.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6260000000000001e-06 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040545+0000", 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0001e-05 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040551+0000", 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2210000000000002e-06 2024-09-17T16:28:25.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.044294+0000", 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037428040000000002 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.044410+0000", 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011616999999999999 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.044515+0000", 2024-09-17T16:28:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104713 2024-09-17T16:28:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.052307+0000", 2024-09-17T16:28:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0077919399999999998 2024-09-17T16:28:25.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.054367+0000", 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020602149999999998 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067817+0000", 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.013449883 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067881+0000", 2024-09-17T16:28:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4331000000000005e-05 2024-09-17T16:28:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067935+0000", 2024-09-17T16:28:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3940999999999997e-05 2024-09-17T16:28:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.810 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115488 6.b 6:d53f7bd0:::benchmark_data_smithi173_102017_object22340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.048310+0000", 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.417879670000005, 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.019735976, 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.811 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115488 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048310+0000", 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048310+0000", 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048311+0000", 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.393e-06 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048326+0000", 2024-09-17T16:28:25.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5081e-05 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048336+0000", 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7140000000000005e-06 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048342+0000", 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4110000000000001e-06 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048387+0000", 2024-09-17T16:28:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5049000000000002e-05 2024-09-17T16:28:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048484+0000", 2024-09-17T16:28:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6362999999999999e-05 2024-09-17T16:28:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048584+0000", 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100679 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.052398+0000", 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038139070000000001 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.054404+0000", 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002006056 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067973+0000", 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.013568521 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.068008+0000", 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5404000000000002e-05 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.068046+0000", 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7397999999999997e-05 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.817 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115496 6.b 6:d757f72e:::benchmark_data_smithi173_102017_object22348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.060444+0000", 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.405744867999999, 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.011022887999999, 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.818 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115496 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060444+0000", 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060444+0000", 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060446+0000", 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6589999999999999e-06 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060451+0000", 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.429e-06 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060462+0000", 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0485999999999999e-05 2024-09-17T16:28:25.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060469+0000", 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8759999999999999e-06 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060543+0000", 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4586999999999996e-05 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060653+0000", 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000109873 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060753+0000", 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100114 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.064621+0000", 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038672899999999998 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.075055+0000", 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010434218 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071408+0000", 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.99635303 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071443+0000", 2024-09-17T16:28:25.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4671000000000001e-05 2024-09-17T16:28:25.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071467+0000", 2024-09-17T16:28:25.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4655e-05 2024-09-17T16:28:25.824 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.824 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.824 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.825 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115501 6.b 6:d2a1b698:::benchmark_data_smithi173_102017_object22353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.825 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.067982+0000", 2024-09-17T16:28:25.825 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.398206934000001, 2024-09-17T16:28:25.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.003590427000001, 2024-09-17T16:28:25.825 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.825 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.825 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.826 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.826 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.826 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.826 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115501 2024-09-17T16:28:25.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.826 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067982+0000", 2024-09-17T16:28:25.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067982+0000", 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067984+0000", 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067989+0000", 2024-09-17T16:28:25.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5439999999999998e-06 2024-09-17T16:28:25.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068001+0000", 2024-09-17T16:28:25.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1437000000000001e-05 2024-09-17T16:28:25.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068008+0000", 2024-09-17T16:28:25.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2629999999999997e-06 2024-09-17T16:28:25.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068575+0000", 2024-09-17T16:28:25.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00056690400000000002 2024-09-17T16:28:25.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068713+0000", 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013839599999999999 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068972+0000", 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025855700000000003 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.075175+0000", 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0062034830000000001 2024-09-17T16:28:25.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.080658+0000", 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054827449999999998 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071502+0000", 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.990844352 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071534+0000", 2024-09-17T16:28:25.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2150000000000002e-05 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071573+0000", 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8297999999999998e-05 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115511 6.b 6:dffba458:::benchmark_data_smithi173_102017_object22363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.086508+0000", 2024-09-17T16:28:25.832 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.379681120000001, 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.993681085, 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115511 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086508+0000", 2024-09-17T16:28:25.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086508+0000", 2024-09-17T16:28:25.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086510+0000", 2024-09-17T16:28:25.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1509999999999998e-06 2024-09-17T16:28:25.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086519+0000", 2024-09-17T16:28:25.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1570000000000006e-06 2024-09-17T16:28:25.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086535+0000", 2024-09-17T16:28:25.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5914e-05 2024-09-17T16:28:25.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086544+0000", 2024-09-17T16:28:25.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6670000000000002e-06 2024-09-17T16:28:25.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086616+0000", 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1768999999999997e-05 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086756+0000", 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013999699999999999 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086877+0000", 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000121729 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092902+0000", 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060247219999999997 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092914+0000", 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1815e-05 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080089+0000", 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.987174871000001 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080145+0000", 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5692000000000001e-05 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080189+0000", 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4601000000000002e-05 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115517 6.b 6:dccfd4a2:::benchmark_data_smithi173_102017_object22369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.092979+0000", 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.373210103999995, 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.987374223, 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115517 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092979+0000", 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092979+0000", 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092980+0000", 2024-09-17T16:28:25.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.82e-07 2024-09-17T16:28:25.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092982+0000", 2024-09-17T16:28:25.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5509999999999998e-06 2024-09-17T16:28:25.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092990+0000", 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8089999999999993e-06 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092995+0000", 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2460000000000003e-06 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.093046+0000", 2024-09-17T16:28:25.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0569000000000003e-05 2024-09-17T16:28:25.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.093179+0000", 2024-09-17T16:28:25.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013340499999999999 2024-09-17T16:28:25.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.093304+0000", 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012481600000000001 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.097499+0000", 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041949659999999996 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.100969+0000", 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034705370000000001 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080244+0000", 2024-09-17T16:28:25.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.979274189 2024-09-17T16:28:25.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080301+0000", 2024-09-17T16:28:25.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7593000000000002e-05 2024-09-17T16:28:25.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080353+0000", 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.206e-05 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115529 6.b 6:d3cef629:::benchmark_data_smithi173_102017_object22381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.122349+0000", 2024-09-17T16:28:25.847 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.343839989000003, 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.959155643999999, 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115529 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122349+0000", 2024-09-17T16:28:25.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122349+0000", 2024-09-17T16:28:25.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122351+0000", 2024-09-17T16:28:25.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9599999999999999e-06 2024-09-17T16:28:25.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122358+0000", 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1400000000000002e-06 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122371+0000", 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2791e-05 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122379+0000", 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8450000000000003e-06 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122456+0000", 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7334000000000002e-05 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122594+0000", 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013791399999999999 2024-09-17T16:28:25.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122736+0000", 2024-09-17T16:28:25.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014201799999999999 2024-09-17T16:28:25.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.126464+0000", 2024-09-17T16:28:25.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037274119999999998 2024-09-17T16:28:25.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.126918+0000", 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045421399999999998 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.081386+0000", 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.954468022 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.081467+0000", 2024-09-17T16:28:25.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0822999999999995e-05 2024-09-17T16:28:25.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.081505+0000", 2024-09-17T16:28:25.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8170999999999998e-05 2024-09-17T16:28:25.855 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.855 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.856 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.856 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115584 6.b 6:d2471696:::benchmark_data_smithi173_102017_object22436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.856 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.222584+0000", 2024-09-17T16:28:25.856 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.243605564999996, 2024-09-17T16:28:25.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.861712117, 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115584 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222584+0000", 2024-09-17T16:28:25.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222584+0000", 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222585+0000", 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222590+0000", 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222599+0000", 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0569999999999999e-06 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222606+0000", 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0040000000000002e-06 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222665+0000", 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8761999999999997e-05 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222796+0000", 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131798 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222915+0000", 2024-09-17T16:28:25.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011872800000000001 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226472+0000", 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003557218 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226919+0000", 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00044620499999999997 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084197+0000", 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.857278443 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084256+0000", 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8581e-05 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084296+0000", 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0209000000000001e-05 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115586 6.b 6:defa5317:::benchmark_data_smithi173_102017_object22438:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.226877+0000", 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.239312158000004, 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.857535156000001, 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115586 2024-09-17T16:28:25.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.864 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226877+0000", 2024-09-17T16:28:25.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226877+0000", 2024-09-17T16:28:25.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226879+0000", 2024-09-17T16:28:25.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5409999999999999e-06 2024-09-17T16:28:25.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226885+0000", 2024-09-17T16:28:25.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.759e-06 2024-09-17T16:28:25.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226898+0000", 2024-09-17T16:28:25.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3103e-05 2024-09-17T16:28:25.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226907+0000", 2024-09-17T16:28:25.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2060000000000007e-06 2024-09-17T16:28:25.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226955+0000", 2024-09-17T16:28:25.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7991999999999997e-05 2024-09-17T16:28:25.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.227075+0000", 2024-09-17T16:28:25.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012014 2024-09-17T16:28:25.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.870 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.227184+0000", 2024-09-17T16:28:25.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010894400000000001 2024-09-17T16:28:25.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.230180+0000", 2024-09-17T16:28:25.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002995841 2024-09-17T16:28:25.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.230383+0000", 2024-09-17T16:28:25.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000203043 2024-09-17T16:28:25.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084340+0000", 2024-09-17T16:28:25.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.853957060000001 2024-09-17T16:28:25.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084380+0000", 2024-09-17T16:28:25.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0500000000000002e-05 2024-09-17T16:28:25.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084412+0000", 2024-09-17T16:28:25.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2026999999999997e-05 2024-09-17T16:28:25.874 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.874 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.874 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.874 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115600 6.b 6:d47603e1:::benchmark_data_smithi173_102017_object22452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.874 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.270754+0000", 2024-09-17T16:28:25.874 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.195435078000003, 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.813756322, 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115600 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270754+0000", 2024-09-17T16:28:25.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270754+0000", 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270756+0000", 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7129999999999999e-06 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270763+0000", 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0990000000000001e-06 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270778+0000", 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5316e-05 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270787+0000", 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6300000000000004e-06 2024-09-17T16:28:25.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270852+0000", 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5432000000000006e-05 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270992+0000", 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013987099999999999 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.271065+0000", 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3156000000000002e-05 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.274226+0000", 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031605219999999998 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.274496+0000", 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000270492 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084443+0000", 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.809946601 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084482+0000", 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8633000000000001e-05 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084510+0000", 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8856999999999998e-05 2024-09-17T16:28:25.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115609 6.b 6:d5c07821:::benchmark_data_smithi173_102017_object22461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.328401+0000", 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 71.137787880000005, 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.756214807999999, 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.881 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.882 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115609 2024-09-17T16:28:25.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.882 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328401+0000", 2024-09-17T16:28:25.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328401+0000", 2024-09-17T16:28:25.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328403+0000", 2024-09-17T16:28:25.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9010000000000001e-06 2024-09-17T16:28:25.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328409+0000", 2024-09-17T16:28:25.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5509999999999999e-06 2024-09-17T16:28:25.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328423+0000", 2024-09-17T16:28:25.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4318000000000001e-05 2024-09-17T16:28:25.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328430+0000", 2024-09-17T16:28:25.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0210000000000002e-06 2024-09-17T16:28:25.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328486+0000", 2024-09-17T16:28:25.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6168999999999997e-05 2024-09-17T16:28:25.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328615+0000", 2024-09-17T16:28:25.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012836600000000001 2024-09-17T16:28:25.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328743+0000", 2024-09-17T16:28:25.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012822699999999999 2024-09-17T16:28:25.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.332302+0000", 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035596270000000001 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.332400+0000", 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7771999999999998e-05 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084544+0000", 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.752143610999999 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084585+0000", 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1352e-05 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084616+0000", 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0892999999999997e-05 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.890 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115610 6.b 6:d3fb650b:::benchmark_data_smithi173_102017_object22462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.524259+0000", 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 70.941929939000005, 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.567226706, 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.891 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115610 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524259+0000", 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524259+0000", 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524262+0000", 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6259999999999999e-06 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524273+0000", 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1076e-05 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524294+0000", 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1111000000000002e-05 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524303+0000", 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6920000000000008e-06 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524368+0000", 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5178999999999994e-05 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524462+0000", 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4438999999999995e-05 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524539+0000", 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6746e-05 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.527502+0000", 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029623760000000001 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.528273+0000", 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077131499999999996 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.091400+0000", 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.563127508999999 2024-09-17T16:28:25.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.091448+0000", 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8084999999999998e-05 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.091486+0000", 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7552000000000002e-05 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115634 6.b 6:d0fb3e7a:::benchmark_data_smithi173_102017_object22486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.738056+0000", 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 70.728133063000001, 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.356902781000001, 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115634 2024-09-17T16:28:25.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738056+0000", 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738056+0000", 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738058+0000", 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5969999999999999e-06 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738064+0000", 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5740000000000001e-06 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738080+0000", 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5451000000000002e-05 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738088+0000", 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1000000000000004e-06 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738152+0000", 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4474999999999995e-05 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738284+0000", 2024-09-17T16:28:25.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131435 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738414+0000", 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000130068 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.741749+0000", 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033351320000000002 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.741984+0000", 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023470500000000001 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094858+0000", 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.352874224000001 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094915+0000", 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6904999999999998e-05 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094959+0000", 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4115e-05 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115641 6.b 6:d9f11449:::benchmark_data_smithi173_102017_object22493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 70.701844562000005, 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.330730932, 2024-09-17T16:28:25.904 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115641 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:28:25.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764348+0000", 2024-09-17T16:28:25.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0539999999999999e-06 2024-09-17T16:28:25.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764355+0000", 2024-09-17T16:28:25.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2089999999999999e-06 2024-09-17T16:28:25.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764362+0000", 2024-09-17T16:28:25.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.533e-06 2024-09-17T16:28:25.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764422+0000", 2024-09-17T16:28:25.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0105000000000001e-05 2024-09-17T16:28:25.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764557+0000", 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013532 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764684+0000", 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126761 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.768205+0000", 2024-09-17T16:28:25.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035209350000000002 2024-09-17T16:28:25.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.768669+0000", 2024-09-17T16:28:25.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046402699999999999 2024-09-17T16:28:25.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094999+0000", 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.326329940000001 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095044+0000", 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095076+0000", 2024-09-17T16:28:25.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1754999999999997e-05 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115643 6.b 6:d327719e:::benchmark_data_smithi173_102017_object22495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.775625+0000", 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 70.690564074999998, 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.319555412, 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:28:25.912 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115643 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775625+0000", 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775625+0000", 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775627+0000", 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.671e-06 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775634+0000", 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2370000000000003e-06 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775650+0000", 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6436999999999999e-05 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775660+0000", 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4639999999999995e-06 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775720+0000", 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9599000000000003e-05 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775841+0000", 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000121878 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775957+0000", 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001158 2024-09-17T16:28:25.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.779558+0000", 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036009620000000001 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.779623+0000", 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4332999999999995e-05 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:28:25.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095106+0000", 2024-09-17T16:28:25.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.315483789 2024-09-17T16:28:25.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:28:25.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095149+0000", 2024-09-17T16:28:25.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2670999999999999e-05 2024-09-17T16:28:25.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:28:25.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:28:25.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:28:25.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095181+0000", 2024-09-17T16:28:25.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1571e-05 2024-09-17T16:28:25.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.919 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:28:25.919 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:28:25.919 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:28:25.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:25.920 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:25 smithi071 ceph-mon[30002]: pgmap v1328: 17 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 11902/108915 objects degraded (10.928%); 2433/108915 objects misplaced (2.234%); 14 MiB/s, 226 objects/s recovering 2024-09-17T16:28:25.920 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:25 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11902/108915 objects degraded (10.928%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:25.920 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:25 smithi071 ceph-mon[30002]: osdmap e602: 12 total, 12 up, 12 in 2024-09-17T16:28:25.921 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/560553352' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:25.921 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:25 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:25.946 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:25.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:26.047 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:26.090 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:26.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:26.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:26.229 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:26.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:26.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:26.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:26.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:26.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:26.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:26.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:26.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:26.612 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:26.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:26.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:26 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:28:26.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:26 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:28:26.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:26 smithi017 ceph-mon[27571]: osdmap e603: 12 total, 12 up, 12 in 2024-09-17T16:28:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:26 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:28:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:26 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:28:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:26 smithi173 ceph-mon[30530]: osdmap e603: 12 total, 12 up, 12 in 2024-09-17T16:28:26.715 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:26.749 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:26.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:26.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:26.780 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:28:26.780 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:28:26.812 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:28:26.812 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:28:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:26 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:28:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:26 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:28:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:26 smithi071 ceph-mon[30002]: osdmap e603: 12 total, 12 up, 12 in 2024-09-17T16:28:26.844 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:28:26.844 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:28:26.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:26.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:26.877 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:28:26.877 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:28:26.911 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:28:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:28:26.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:28:26.945 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:28:26.978 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:26.980 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:28:26.981 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:28:27.004 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:27.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:27.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:27.021 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:28:27.021 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:28:27.058 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:28:27.058 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:28:27.099 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:28:27.099 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:28:27.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:27.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:27.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:27.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:27.135 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:28:27.135 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:28:27.165 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:28:27.166 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:28:27.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:27.254 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:27.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:27.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:27.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:27.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:27.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:27.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:27.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:27 smithi017 ceph-mon[27571]: pgmap v1331: 17 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 11902/106404 objects degraded (11.186%); 2123/106404 objects misplaced (1.995%); 12 MiB/s, 207 objects/s recovering 2024-09-17T16:28:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1838613077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:27.509 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:27.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:27.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:27.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:27.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:27 smithi173 ceph-mon[30530]: pgmap v1331: 17 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 11902/106404 objects degraded (11.186%); 2123/106404 objects misplaced (1.995%); 12 MiB/s, 207 objects/s recovering 2024-09-17T16:28:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1838613077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:27.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:27.775 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:27.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:27.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:27 smithi071 ceph-mon[30002]: pgmap v1331: 17 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 11902/106404 objects degraded (11.186%); 2123/106404 objects misplaced (1.995%); 12 MiB/s, 207 objects/s recovering 2024-09-17T16:28:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1838613077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:27.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:27.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:27.919 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:27.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:28.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:28.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:28.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:28.144 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:28.178 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:28.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:28.280 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:28.301 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:28.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:28.302 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:28.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:28.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:28.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:28.527 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:28.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:28.569 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:28.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:28.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:28.699 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:28.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:28.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:28.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:28.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:28.925 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:28.957 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:28.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:28.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:29.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:29.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:29.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:29.183 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:29.210 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:29.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:29.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:29.346 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:29.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:29.447 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:29.473 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:29.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:29.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:29.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:29.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:29.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:29 smithi173 ceph-mon[30530]: pgmap v1332: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 9916/100797 objects degraded (9.838%); 3234/100797 objects misplaced (3.208%); 17 MiB/s, 275 objects/s recovering 2024-09-17T16:28:29.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:29.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:29.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:29 smithi017 ceph-mon[27571]: pgmap v1332: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 9916/100797 objects degraded (9.838%); 3234/100797 objects misplaced (3.208%); 17 MiB/s, 275 objects/s recovering 2024-09-17T16:28:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:29 smithi071 ceph-mon[30002]: pgmap v1332: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 9916/100797 objects degraded (9.838%); 3234/100797 objects misplaced (3.208%); 17 MiB/s, 275 objects/s recovering 2024-09-17T16:28:29.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:29.853 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:29.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:29.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:29.988 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:29.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:30.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:30.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:30.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:30.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:30.245 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:30.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:30.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:30.372 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:30.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:30.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:30.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:30.510 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:30.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:30.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:30.639 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:30.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:30.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:30 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 9916/100797 objects degraded (9.838%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:30 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 9916/100797 objects degraded (9.838%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:30.741 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:30.767 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:30.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:30.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:30 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 9916/100797 objects degraded (9.838%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:30.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:30.893 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:30.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:30.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:30.995 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:31.026 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:31.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:31.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:31.128 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:31.159 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:31.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:31.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:31.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:31.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:31.385 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:31.416 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:31.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:31.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:31.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:31.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:31.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:31 smithi173 ceph-mon[30530]: pgmap v1333: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 9916/99228 objects degraded (9.993%); 3234/99228 objects misplaced (3.259%); 11 MiB/s, 187 objects/s recovering 2024-09-17T16:28:31.668 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:31.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:31 smithi017 ceph-mon[27571]: pgmap v1333: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 9916/99228 objects degraded (9.993%); 3234/99228 objects misplaced (3.259%); 11 MiB/s, 187 objects/s recovering 2024-09-17T16:28:31.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:31.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:31.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:31 smithi071 ceph-mon[30002]: pgmap v1333: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 9916/99228 objects degraded (9.993%); 3234/99228 objects misplaced (3.259%); 11 MiB/s, 187 objects/s recovering 2024-09-17T16:28:31.893 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:31.918 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:31.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:32.020 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:32.049 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:32.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:32.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:32.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:32.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:32.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:32.275 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:32.309 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:32.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:32.411 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:32.444 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:32.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:32.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:32.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:32.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:32.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:32.697 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:32.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:32.799 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:32.822 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:32.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:32.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:32.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:32.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:32.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:33.049 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:33.070 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:33.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:33.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:33.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:33.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:33.299 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:33.329 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:33.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:33.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:33.464 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:33.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:33.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:33.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:33.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:33 smithi173 ceph-mon[30530]: pgmap v1334: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 9940/93090 objects degraded (10.678%); 2853/93090 objects misplaced (3.065%); 11 MiB/s, 192 objects/s recovering 2024-09-17T16:28:33.689 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:33.713 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:33.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:33 smithi017 ceph-mon[27571]: pgmap v1334: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 9940/93090 objects degraded (10.678%); 2853/93090 objects misplaced (3.065%); 11 MiB/s, 192 objects/s recovering 2024-09-17T16:28:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:33 smithi071 ceph-mon[30002]: pgmap v1334: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 9940/93090 objects degraded (10.678%); 2853/93090 objects misplaced (3.065%); 11 MiB/s, 192 objects/s recovering 2024-09-17T16:28:33.815 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:33.838 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:33.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:33.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:33.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:33.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:34.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:34.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:34.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:34.186 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:34.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:34.212 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:34.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:34.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:34.337 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:34.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:34.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:34.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:34.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:34.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:34.592 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:34.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:34.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:28:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:34 smithi173 ceph-mon[30530]: osdmap e604: 12 total, 12 up, 12 in 2024-09-17T16:28:34.658 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:34 smithi071 ceph-mon[30002]: osdmap e604: 12 total, 12 up, 12 in 2024-09-17T16:28:34.694 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:34 smithi017 ceph-mon[27571]: osdmap e604: 12 total, 12 up, 12 in 2024-09-17T16:28:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:34.740 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:34.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:34.842 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:34.873 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:34.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:34.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:34.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:35.005 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:35.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:35.024 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:28:35.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:35.133 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:35.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:35.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:35.260 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:35.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:35.362 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:35.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:35.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:28:35.447 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:28:35.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-09-17T16:28:35.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:35 smithi071 ceph-mon[30002]: pgmap v1336: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 9746/91287 objects degraded (10.676%); 2617/91287 objects misplaced (2.867%); 13 MiB/s, 234 objects/s recovering 2024-09-17T16:28:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:35 smithi071 ceph-mon[30002]: osdmap e605: 12 total, 12 up, 12 in 2024-09-17T16:28:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:35 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 9746/91287 objects degraded (10.676%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:35.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:35.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:35.620 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:35.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:35 smithi173 ceph-mon[30530]: pgmap v1336: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 9746/91287 objects degraded (10.676%); 2617/91287 objects misplaced (2.867%); 13 MiB/s, 234 objects/s recovering 2024-09-17T16:28:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:35 smithi173 ceph-mon[30530]: osdmap e605: 12 total, 12 up, 12 in 2024-09-17T16:28:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:35 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 9746/91287 objects degraded (10.676%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:35.650 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:35.650 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:35.657 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:35.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:35 smithi017 ceph-mon[27571]: pgmap v1336: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 9746/91287 objects degraded (10.676%); 2617/91287 objects misplaced (2.867%); 13 MiB/s, 234 objects/s recovering 2024-09-17T16:28:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:35 smithi017 ceph-mon[27571]: osdmap e605: 12 total, 12 up, 12 in 2024-09-17T16:28:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:35 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 9746/91287 objects degraded (10.676%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:35.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:35.789 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:35.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:35.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:35.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:35.926 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:35.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:36.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:36.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:36.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:36.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:36.186 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:36.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:36.287 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:36.323 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:36.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:36.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:36.453 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:36.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:36.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:28:36.556 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:36.591 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:36.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:36.693 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:36.724 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:36.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:36.827 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:36.849 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:36.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:36.951 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:36.955 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:28:36.984 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:36.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:36.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:37.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:37.123 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:37.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:37.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:37.225 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:37.249 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:37.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:37.351 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:37.385 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:37.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:37.487 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:37.513 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:37.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:37.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:37.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:37.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:37 smithi173 ceph-mon[30530]: pgmap v1338: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 8025/88821 objects degraded (9.035%); 4206/88821 objects misplaced (4.735%); 5.7 MiB/s, 114 objects/s recovering 2024-09-17T16:28:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:37 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:28:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:37 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:28:37.647 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:37 smithi071 ceph-mon[30002]: pgmap v1338: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 8025/88821 objects degraded (9.035%); 4206/88821 objects misplaced (4.735%); 5.7 MiB/s, 114 objects/s recovering 2024-09-17T16:28:37.648 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:37 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:28:37.648 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:37 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:28:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:37 smithi017 ceph-mon[27571]: pgmap v1338: 17 pgs: 4 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 8025/88821 objects degraded (9.035%); 4206/88821 objects misplaced (4.735%); 5.7 MiB/s, 114 objects/s recovering 2024-09-17T16:28:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:37 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:28:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:37 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:28:37.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:37.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:37.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:37.865 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:37.898 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:37.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:37.915 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:37.915 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:38.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:38.036 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:38.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:38.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:38.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:38.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:38.261 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:38.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:38.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:38.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:38.439 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:38.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:38.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:38.569 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:38.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:38.671 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:38.674 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:38 smithi017 ceph-mon[27571]: osdmap e606: 12 total, 12 up, 12 in 2024-09-17T16:28:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:38.705 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:38.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:38.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:38 smithi071 ceph-mon[30002]: osdmap e606: 12 total, 12 up, 12 in 2024-09-17T16:28:38.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:38.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:38 smithi173 ceph-mon[30530]: osdmap e606: 12 total, 12 up, 12 in 2024-09-17T16:28:38.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:38.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:38.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:38.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:39.056 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:39.087 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:39.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:39.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:39.189 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:39.220 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:39.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:39.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:39.349 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:39.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:39.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:39.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:39.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:39.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:39.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:39.604 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:39.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:39.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:39.706 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:39.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:39 smithi071 ceph-mon[30002]: pgmap v1340: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 7943/83571 objects degraded (9.504%); 3665/83571 objects misplaced (4.385%); 9.4 MiB/s, 197 objects/s recovering 2024-09-17T16:28:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:39.733 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:39.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:39.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:39 smithi017 ceph-mon[27571]: pgmap v1340: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 7943/83571 objects degraded (9.504%); 3665/83571 objects misplaced (4.385%); 9.4 MiB/s, 197 objects/s recovering 2024-09-17T16:28:39.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:39 smithi173 ceph-mon[30530]: pgmap v1340: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 7943/83571 objects degraded (9.504%); 3665/83571 objects misplaced (4.385%); 9.4 MiB/s, 197 objects/s recovering 2024-09-17T16:28:39.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:39.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:39.858 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:39.959 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:39.985 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:39.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:40.087 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:40.118 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:40.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:40.219 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:40.243 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:40.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:40.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:40.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:40.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:40.470 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:40.497 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:40.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:40.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:40.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:28:40.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:40.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:40.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:40.623 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:40.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:40 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:28:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:40 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:28:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:40 smithi017 ceph-mon[27571]: osdmap e607: 12 total, 12 up, 12 in 2024-09-17T16:28:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:40 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7943/83571 objects degraded (9.504%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:40.769 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:40.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:40 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:28:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:40 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:28:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:40 smithi071 ceph-mon[30002]: osdmap e607: 12 total, 12 up, 12 in 2024-09-17T16:28:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:40 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7943/83571 objects degraded (9.504%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:40.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:40.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:40.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:40.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:40 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:28:40.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:40 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:28:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:40 smithi173 ceph-mon[30530]: osdmap e607: 12 total, 12 up, 12 in 2024-09-17T16:28:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:40 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7943/83571 objects degraded (9.504%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:40.936 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:28:40.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:41.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:41.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:41.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:41.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:41.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:41.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:41.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:41.241 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:41.267 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:41.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:41.369 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:41.401 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:41.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:41.503 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:41.505 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:41.505 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:41.537 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:41.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:41.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:41.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:41.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:41.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:41 smithi017 ceph-mon[27571]: pgmap v1342: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 7943/81873 objects degraded (9.702%); 3665/81873 objects misplaced (4.476%); 7.0 MiB/s, 125 objects/s recovering 2024-09-17T16:28:41.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:41.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:41.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:41 smithi071 ceph-mon[30002]: pgmap v1342: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 7943/81873 objects degraded (9.702%); 3665/81873 objects misplaced (4.476%); 7.0 MiB/s, 125 objects/s recovering 2024-09-17T16:28:41.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:41.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:41 smithi173 ceph-mon[30530]: pgmap v1342: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 7943/81873 objects degraded (9.702%); 3665/81873 objects misplaced (4.476%); 7.0 MiB/s, 125 objects/s recovering 2024-09-17T16:28:41.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:41.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:42.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:42.045 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:42.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:42.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:42.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:42.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:42.172 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:42.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:28:42.272 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:42.312 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:42.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:42.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:42.448 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:42.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:42.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:42.553 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:28:42.576 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:42.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:42.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:42.706 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:42.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:42.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:42.807 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:42.838 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:42.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:42.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:42.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:42.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:42.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:43.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:43.091 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:43.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:43.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:43.133 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:43.133 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:43.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:43.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:43.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:43.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:43.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:43.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:43.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:43.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:43.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:43.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:43.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:43.601 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:43.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:43.678 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:43.702 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:43 smithi017 ceph-mon[27571]: pgmap v1343: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 7975/75900 objects degraded (10.507%); 2914/75900 objects misplaced (3.839%); 11 MiB/s, 194 objects/s recovering 2024-09-17T16:28:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:43.741 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:43.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:43 smithi071 ceph-mon[30002]: pgmap v1343: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 7975/75900 objects degraded (10.507%); 2914/75900 objects misplaced (3.839%); 11 MiB/s, 194 objects/s recovering 2024-09-17T16:28:43.842 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:43.874 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:43.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:43 smithi173 ceph-mon[30530]: pgmap v1343: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 7975/75900 objects degraded (10.507%); 2914/75900 objects misplaced (3.839%); 11 MiB/s, 194 objects/s recovering 2024-09-17T16:28:43.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:43.984 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:28:43.997 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:43.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:44.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:44.127 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:44.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:44.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:44.270 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:44.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:44.348 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:28:44.372 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:44.405 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:44.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:44.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:44.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:44.534 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:44.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:44.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:44.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:28:44.663 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:28:44.664 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:28:44.747 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:28:44.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:44.838 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:44.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:44.887 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:44.887 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:44.940 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:44.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:44.976 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:44.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:45.077 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:45.128 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:45.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:45.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:45.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:45.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:45.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:45.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:45.390 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:45.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:45.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:45.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:45.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:45.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:45.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:45.519 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:28:45.616 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:45.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:28:45.655 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:45.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:45 smithi017 ceph-mon[27571]: pgmap v1344: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 7975/73956 objects degraded (10.783%); 2914/73956 objects misplaced (3.940%); 9.8 MiB/s, 167 objects/s recovering 2024-09-17T16:28:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:45 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7975/73956 objects degraded (10.783%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:45.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:45.790 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:45.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:45 smithi071 ceph-mon[30002]: pgmap v1344: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 7975/73956 objects degraded (10.783%); 2914/73956 objects misplaced (3.940%); 9.8 MiB/s, 167 objects/s recovering 2024-09-17T16:28:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:45 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7975/73956 objects degraded (10.783%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:45.892 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:45 smithi173 ceph-mon[30530]: pgmap v1344: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 7975/73956 objects degraded (10.783%); 2914/73956 objects misplaced (3.940%); 9.8 MiB/s, 167 objects/s recovering 2024-09-17T16:28:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:45 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7975/73956 objects degraded (10.783%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:45.917 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:45.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:45.968 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:28:45.986 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:28:45.986 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:28:45.986 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-09-17T16:28:46.019 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:46.045 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:46.057 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:46.086 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:46.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:46.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:46.218 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:46.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:46.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:46.346 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:46.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:46.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:46.448 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:46.482 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:46.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:46.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:46.539 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:46.539 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:46.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:46.609 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:46.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:46.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:46.739 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:46.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:46.842 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:46.864 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:46.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:46.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:46.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:46.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:46.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:47.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:47.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:47.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:47.215 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:47.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:28:47.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:47.240 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:47.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:47.342 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:47.377 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:47.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:47.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:28:47.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:28:47.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:28:47.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:28:47.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:28:47.469 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:28:47.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:28:47.471 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:28:47.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:28:47.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:28:47.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:28:47.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:28:47.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:28:47.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:28:47.478 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:47.506 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:47.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:47.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:47 smithi071 ceph-mon[30002]: pgmap v1345: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 7975/70674 objects degraded (11.284%); 2750/70674 objects misplaced (3.891%); 6.4 MiB/s, 102 objects/s recovering 2024-09-17T16:28:47.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2537694135' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:47.591 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:28:47.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:47.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:47.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:47.731 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:47 smithi017 ceph-mon[27571]: pgmap v1345: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 7975/70674 objects degraded (11.284%); 2750/70674 objects misplaced (3.891%); 6.4 MiB/s, 102 objects/s recovering 2024-09-17T16:28:47.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2537694135' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:47.759 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:47.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:47.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:47.889 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:47.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:47.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:47 smithi173 ceph-mon[30530]: pgmap v1345: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 7975/70674 objects degraded (11.284%); 2750/70674 objects misplaced (3.891%); 6.4 MiB/s, 102 objects/s recovering 2024-09-17T16:28:47.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2537694135' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:47.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:48.030 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:48.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:48.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:48.155 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:48.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:48.257 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:48.283 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:48.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:48.386 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:48.413 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:48.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:48.433 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:28:48.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:48.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:48.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:48.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:48.640 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:48.663 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:48.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1875858238' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:28:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:48 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:28:48.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:48.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:48.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:48.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1875858238' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:28:48.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:48 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:28:48.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:48.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1875858238' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:28:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:48 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:28:48.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:48.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:48.924 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:28:48.924 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:28:48.924 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:28:48.924 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:28:49.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:49.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:49.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:49.134 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:49.158 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:49.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:49.185 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:49.260 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:49.283 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:49.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:49.385 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:49.407 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:49.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:49.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:49.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:49.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:49 smithi071 ceph-mon[30002]: pgmap v1346: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 7835/63675 objects degraded (12.305%); 2372/63675 objects misplaced (3.725%); 7.6 MiB/s, 140 objects/s recovering 2024-09-17T16:28:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:49 smithi071 ceph-mon[30002]: 6.1 scrub starts 2024-09-17T16:28:49.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:49 smithi071 ceph-mon[30002]: 6.1 scrub ok 2024-09-17T16:28:49.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:49 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:49.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:49 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:28:49.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:49 smithi071 ceph-mon[30002]: osdmap e608: 12 total, 12 up, 12 in 2024-09-17T16:28:49.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:49.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:49.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:49.667 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:28:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:49 smithi017 ceph-mon[27571]: pgmap v1346: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 7835/63675 objects degraded (12.305%); 2372/63675 objects misplaced (3.725%); 7.6 MiB/s, 140 objects/s recovering 2024-09-17T16:28:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:49 smithi017 ceph-mon[27571]: 6.1 scrub starts 2024-09-17T16:28:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:49 smithi017 ceph-mon[27571]: 6.1 scrub ok 2024-09-17T16:28:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:49 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:49 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:28:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:49 smithi017 ceph-mon[27571]: osdmap e608: 12 total, 12 up, 12 in 2024-09-17T16:28:49.764 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:49.781 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:28:49.781 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:28:49.781 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:28:49.781 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:28:49.781 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:28:49.781 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:28:49.782 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:28:49.782 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:28:49.782 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:28:49.782 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:28:49.782 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:28:49.782 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:28:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:49.801 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:49.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:49 smithi173 ceph-mon[30530]: pgmap v1346: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 663 op/s; 7835/63675 objects degraded (12.305%); 2372/63675 objects misplaced (3.725%); 7.6 MiB/s, 140 objects/s recovering 2024-09-17T16:28:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:49 smithi173 ceph-mon[30530]: 6.1 scrub starts 2024-09-17T16:28:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:49 smithi173 ceph-mon[30530]: 6.1 scrub ok 2024-09-17T16:28:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:49 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:49 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:28:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:49 smithi173 ceph-mon[30530]: osdmap e608: 12 total, 12 up, 12 in 2024-09-17T16:28:49.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:49.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:49.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:50.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:50.049 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:28:50.072 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:50.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:50.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:50.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:50.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:50.306 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:50.335 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:50.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:50.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:50.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:50.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:50.540 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:28:50.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:50.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:50.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:50.656 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:28:50.656 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:28:50.656 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:28:50.656 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:28:50.656 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:28:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:28:50.680 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:28:50.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-17T16:28:50.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:50.714 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:50 smithi017 ceph-mon[27571]: osdmap e609: 12 total, 12 up, 12 in 2024-09-17T16:28:50.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:50 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7835/63675 objects degraded (12.305%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:50.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:50.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:50 smithi071 ceph-mon[30002]: osdmap e609: 12 total, 12 up, 12 in 2024-09-17T16:28:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:50 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7835/63675 objects degraded (12.305%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:50.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:50.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:50.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:50 smithi173 ceph-mon[30530]: osdmap e609: 12 total, 12 up, 12 in 2024-09-17T16:28:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:50 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7835/63675 objects degraded (12.305%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:50.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:50.983 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:51.007 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:51.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:51.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:51.137 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:51.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:51.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:28:51.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:51.269 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:51.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:51.331 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:28:51.371 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:51.412 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:51.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:51.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:51.541 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:51.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:51.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:51 smithi071 ceph-mon[30002]: pgmap v1349: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 7835/62229 objects degraded (12.591%); 2372/62229 objects misplaced (3.812%); 3.9 MiB/s, 85 objects/s recovering 2024-09-17T16:28:51.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:51.666 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:51.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:51.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:51.673 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:28:51.677 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:28:51.677 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:28:51.719 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:28:51.719 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:28:51.719 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:51 smithi017 ceph-mon[27571]: pgmap v1349: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 7835/62229 objects degraded (12.591%); 2372/62229 objects misplaced (3.812%); 3.9 MiB/s, 85 objects/s recovering 2024-09-17T16:28:51.722 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:51.754 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:28:51.755 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:28:51.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:51.796 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:28:51.797 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:28:51.806 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:51.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:51.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:51.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:28:51.837 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:28:51.873 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:28:51.873 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:28:51.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:51 smithi173 ceph-mon[30530]: pgmap v1349: 17 pgs: 1 active+recovery_wait, 3 active+recovering+undersized+degraded+remapped, 13 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 7835/62229 objects degraded (12.591%); 2372/62229 objects misplaced (3.812%); 3.9 MiB/s, 85 objects/s recovering 2024-09-17T16:28:51.904 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:28:51.904 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:28:51.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:51.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:51.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:51.937 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:28:51.938 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:28:51.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:28:51.973 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:28:52.005 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:28:52.005 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:28:52.033 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:28:52.033 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:28:52.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:52.062 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:28:52.062 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:28:52.069 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:52.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:52.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:52.171 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:52.206 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:52.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:52.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:52.331 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:52.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:52.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:52.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:52.465 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:52.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:52.469 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:28:52.469 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:28:52.469 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:28:52.469 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:28:52.469 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:28:52.469 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:28:52.469 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:28:52.470 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:28:52.470 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:28:52.470 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:28:52.470 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:28:52.470 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:28:52.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:52.589 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:52.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:52.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:52.721 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:52.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1788565690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1788565690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:52.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:52.850 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:52.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1788565690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:28:52.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:52.976 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:52.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:53.078 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:53.105 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:53.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:53.114 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:53.114 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:53.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:53.238 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:53.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:53.340 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:53.381 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:53.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:53.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:53.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:53.517 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:53.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:53.618 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:53.649 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:53.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:53.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:53 smithi017 ceph-mon[27571]: pgmap v1350: 17 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 7681/57309 objects degraded (13.403%); 1915/57309 objects misplaced (3.342%); 7.4 MiB/s, 161 objects/s recovering 2024-09-17T16:28:53.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:28:53.781 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:53.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:28:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:53 smithi071 ceph-mon[30002]: pgmap v1350: 17 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 7681/57309 objects degraded (13.403%); 1915/57309 objects misplaced (3.342%); 7.4 MiB/s, 161 objects/s recovering 2024-09-17T16:28:53.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:53.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:53 smithi173 ceph-mon[30530]: pgmap v1350: 17 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 14 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 7681/57309 objects degraded (13.403%); 1915/57309 objects misplaced (3.342%); 7.4 MiB/s, 161 objects/s recovering 2024-09-17T16:28:53.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:53.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:54.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:54.033 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:54.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:54.135 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:54.163 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:54.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:54.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:54.290 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:54.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:54.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:54.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:54.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:54.516 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:54.543 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:54.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:54.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:54.673 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:54.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:54.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:54.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:54.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:54.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:54.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:54.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:54.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:55.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:55.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:55.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:55.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:55.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:55.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:55.277 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:55.302 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:55.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:55.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:55.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:55.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:55.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:55.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:55.553 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:55.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:55.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:55.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:55.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:55.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:55 smithi017 ceph-mon[27571]: pgmap v1351: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 7681/55776 objects degraded (13.771%); 1915/55776 objects misplaced (3.433%); 6.1 MiB/s, 141 objects/s recovering 2024-09-17T16:28:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:55 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7681/55776 objects degraded (13.771%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:55.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:55.810 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:55.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:55.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:55.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:55 smithi071 ceph-mon[30002]: pgmap v1351: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 7681/55776 objects degraded (13.771%); 1915/55776 objects misplaced (3.433%); 6.1 MiB/s, 141 objects/s recovering 2024-09-17T16:28:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:55 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7681/55776 objects degraded (13.771%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:55 smithi173 ceph-mon[30530]: pgmap v1351: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 7681/55776 objects degraded (13.771%); 1915/55776 objects misplaced (3.433%); 6.1 MiB/s, 141 objects/s recovering 2024-09-17T16:28:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:55 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7681/55776 objects degraded (13.771%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:28:55.912 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:55.965 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:55.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:56.066 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:56.089 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:56.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:56.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:56.227 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:56.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:56.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:28:56.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:56.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:56.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:28:56.457 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:56.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:56.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:56.618 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:56.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:56.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:56.754 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:56.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:56.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:56.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:56.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:56.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:56.982 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:57.014 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:57.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:57.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:57.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:57.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:57.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:57.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:57.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:57.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:57.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:28:57.367 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:28:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:57.407 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:57.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:28:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:57 smithi017 ceph-mon[27571]: pgmap v1352: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 466 op/s; 7681/52476 objects degraded (14.637%); 1727/52476 objects misplaced (3.291%); 5.0 MiB/s, 99 objects/s recovering 2024-09-17T16:28:57.507 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:57.544 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:57.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:57 smithi071 ceph-mon[30002]: pgmap v1352: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 466 op/s; 7681/52476 objects degraded (14.637%); 1727/52476 objects misplaced (3.291%); 5.0 MiB/s, 99 objects/s recovering 2024-09-17T16:28:57.595 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:28:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:57 smithi173 ceph-mon[30530]: pgmap v1352: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 466 op/s; 7681/52476 objects degraded (14.637%); 1727/52476 objects misplaced (3.291%); 5.0 MiB/s, 99 objects/s recovering 2024-09-17T16:28:57.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:57.670 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:57.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:57.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:57.802 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:57.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:57.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:28:57.892 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:28:57.892 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:28:57.893 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-17T16:28:57.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:57.926 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:57.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:57.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:58.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:58.053 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:58.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:58.146 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:58.146 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:58.154 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:58.202 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:58.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:58.294 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:28:58.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:28:58.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:58.329 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:28:58.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:58.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:58.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:58.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:58.465 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:28:58.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:28:58.595 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:58.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:58.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:28:58.697 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:28:58.713 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1845853143' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:58.726 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:58.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:28:58.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1845853143' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:58.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1845853143' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:28:58.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:58.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:58.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:58.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:28:58.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:58.977 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:28:59.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:59.014 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:59.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:28:59.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:59.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:28:59.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:28:59.262 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:28:59.272 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:59.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:28:59.374 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:28:59.411 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:59.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:28:59.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:59.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:59.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:28:59.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:59 smithi173 ceph-mon[30530]: pgmap v1353: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 973 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 7705/46689 objects degraded (16.503%); 1552/46689 objects misplaced (3.324%); 5.2 MiB/s, 99 objects/s recovering 2024-09-17T16:28:59.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:59 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:59.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1845853143' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:28:59.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:28:59 smithi173 ceph-mon[30530]: osdmap e610: 12 total, 12 up, 12 in 2024-09-17T16:28:59.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:28:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:59.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:28:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:59 smithi017 ceph-mon[27571]: pgmap v1353: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 973 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 7705/46689 objects degraded (16.503%); 1552/46689 objects misplaced (3.324%); 5.2 MiB/s, 99 objects/s recovering 2024-09-17T16:28:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:59 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1845853143' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:28:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:28:59 smithi017 ceph-mon[27571]: osdmap e610: 12 total, 12 up, 12 in 2024-09-17T16:28:59.768 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:28:59.793 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:28:59.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:59.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:28:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:59 smithi071 ceph-mon[30002]: pgmap v1353: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 973 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 7705/46689 objects degraded (16.503%); 1552/46689 objects misplaced (3.324%); 5.2 MiB/s, 99 objects/s recovering 2024-09-17T16:28:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:59 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:28:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1845853143' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:28:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:28:59 smithi071 ceph-mon[30002]: osdmap e610: 12 total, 12 up, 12 in 2024-09-17T16:28:59.817 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:28:59.817 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:28:59.862 INFO:teuthology.orchestra.run.smithi017.stderr:no valid command found; 10 closest matches: 2024-09-17T16:28:59.863 INFO:teuthology.orchestra.run.smithi017.stderr:0 2024-09-17T16:28:59.863 INFO:teuthology.orchestra.run.smithi017.stderr:1 2024-09-17T16:28:59.863 INFO:teuthology.orchestra.run.smithi017.stderr:2 2024-09-17T16:28:59.863 INFO:teuthology.orchestra.run.smithi017.stderr:abort 2024-09-17T16:28:59.863 INFO:teuthology.orchestra.run.smithi017.stderr:assert 2024-09-17T16:28:59.863 INFO:teuthology.orchestra.run.smithi017.stderr:bench [] [] [] [] 2024-09-17T16:28:59.863 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:28:59.863 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs files list 2024-09-17T16:28:59.863 INFO:teuthology.orchestra.run.smithi017.stderr:bluefs stats 2024-09-17T16:28:59.864 INFO:teuthology.orchestra.run.smithi017.stderr:bluestore allocator dump block 2024-09-17T16:28:59.864 INFO:teuthology.orchestra.run.smithi017.stderr:Error EINVAL: invalid command 2024-09-17T16:28:59.895 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:28:59.939 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:28:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:28:59.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:00.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:00.067 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:00.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:00.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:00.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:00.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:00.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:00.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:00.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:00.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:00.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:00.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:00.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:00.554 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:00 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7705/46689 objects degraded (16.503%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:00.554 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:00 smithi173 ceph-mon[30530]: osdmap e611: 12 total, 12 up, 12 in 2024-09-17T16:29:00.569 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:00.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:00.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:29:00.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:00 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7705/46689 objects degraded (16.503%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:00.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:00 smithi017 ceph-mon[27571]: osdmap e611: 12 total, 12 up, 12 in 2024-09-17T16:29:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:29:00.646 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-17T16:29:00.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:00.702 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:00.708 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:00.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:00.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:00.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:00.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:00 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7705/46689 objects degraded (16.503%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:00.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:00 smithi071 ceph-mon[30002]: osdmap e611: 12 total, 12 up, 12 in 2024-09-17T16:29:00.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:00.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:00.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:00.886 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:29:00.936 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:00.966 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:00.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:00.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:01.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:01.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:01.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:01.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:01.213 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:01.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:01.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:01.344 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:01.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:01.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:01.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:01.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:01.507 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:29:01.507 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:29:01.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:01 smithi173 ceph-mon[30530]: pgmap v1356: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 904 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 7705/43368 objects degraded (17.767%); 1552/43368 objects misplaced (3.579%); 2.6 MiB/s, 42 objects/s recovering 2024-09-17T16:29:01.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:01.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:29:01.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:01.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:01.702 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:01.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:01.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:01 smithi017 ceph-mon[27571]: pgmap v1356: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 904 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 7705/43368 objects degraded (17.767%); 1552/43368 objects misplaced (3.579%); 2.6 MiB/s, 42 objects/s recovering 2024-09-17T16:29:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:01 smithi071 ceph-mon[30002]: pgmap v1356: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 904 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 7705/43368 objects degraded (17.767%); 1552/43368 objects misplaced (3.579%); 2.6 MiB/s, 42 objects/s recovering 2024-09-17T16:29:01.833 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:01.858 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:01.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:01.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:01.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:01.987 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:01.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:02.088 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:02.118 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:02.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:02.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:29:02.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:02.251 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:02.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:02.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:02.354 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:02.378 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:02.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:02.480 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:02.507 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:29:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:02.525 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:02.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:02.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:02.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:02.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:02 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:02 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:02.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:02.777 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:02 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:02.777 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:02 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:02.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:02.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:02.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:02 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:02 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:02.878 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:02.903 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:02.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:02.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:03.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:03.036 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:03.037 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:29:03.037 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:29:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:03.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:03.138 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:03.162 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:03.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:03.264 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:03.294 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:03.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:03.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:03.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:03.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:03.466 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:29:03.466 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:29:03.466 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:29:03.466 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:29:03.498 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:29:03.499 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:29:03.519 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:03.538 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:29:03.538 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:29:03.577 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:29:03.577 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:29:03.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:03.578 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:03.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:03.618 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:29:03.619 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:29:03.638 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:03.652 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:29:03.653 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:29:03.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:03.687 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:29:03.687 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:29:03.688 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:03 smithi017 ceph-mon[27571]: pgmap v1357: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 768 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 7768/36834 objects degraded (21.089%); 1100/36834 objects misplaced (2.986%); 5.7 MiB/s, 90 objects/s recovering 2024-09-17T16:29:03.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:03 smithi017 ceph-mon[27571]: osdmap e612: 12 total, 12 up, 12 in 2024-09-17T16:29:03.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4264263639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:03.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:03.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:03.723 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:29:03.724 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:29:03.763 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:29:03.764 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:29:03.793 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:29:03.793 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:29:03.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:03 smithi071 ceph-mon[30002]: pgmap v1357: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 768 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 7768/36834 objects degraded (21.089%); 1100/36834 objects misplaced (2.986%); 5.7 MiB/s, 90 objects/s recovering 2024-09-17T16:29:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:03 smithi071 ceph-mon[30002]: osdmap e612: 12 total, 12 up, 12 in 2024-09-17T16:29:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4264263639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:03.828 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:29:03.829 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:29:03.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:03.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:03.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:03 smithi173 ceph-mon[30530]: pgmap v1357: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 768 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 7768/36834 objects degraded (21.089%); 1100/36834 objects misplaced (2.986%); 5.7 MiB/s, 90 objects/s recovering 2024-09-17T16:29:03.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:03 smithi173 ceph-mon[30530]: osdmap e612: 12 total, 12 up, 12 in 2024-09-17T16:29:03.849 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4264263639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:03.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:29:03.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:03.958 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:03.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:04.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:04.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:04.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:04.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:04.211 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:29:04.227 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:04.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:04.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:04.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:04.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:04.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:04.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:04.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:04.511 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:04.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:04.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:04.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:04.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:04.741 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:04.757 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:29:04.758 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:29:04.758 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:29:04.758 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:29:04.758 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:29:04.775 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:04.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:04.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:04.916 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:04.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:05.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:05.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:05.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:05.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:05.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:05.275 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:05.310 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:05.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:05.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:05.443 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:05.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:05.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:29:05.545 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:05.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:05.576 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:05.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:05.592 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:29:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:05 smithi173 ceph-mon[30530]: pgmap v1359: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 729 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 651 op/s; 7768/34962 objects degraded (22.218%); 1100/34962 objects misplaced (3.146%); 4.1 MiB/s, 64 objects/s recovering 2024-09-17T16:29:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:05 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7768/34962 objects degraded (22.218%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:05 smithi173 ceph-mon[30530]: osdmap e613: 12 total, 12 up, 12 in 2024-09-17T16:29:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4075364630' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:29:05.646 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:29:05.647 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:29:05.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:29:05.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:05.709 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:05.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:05.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:05 smithi017 ceph-mon[27571]: pgmap v1359: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 729 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 651 op/s; 7768/34962 objects degraded (22.218%); 1100/34962 objects misplaced (3.146%); 4.1 MiB/s, 64 objects/s recovering 2024-09-17T16:29:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:05 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7768/34962 objects degraded (22.218%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:05 smithi017 ceph-mon[27571]: osdmap e613: 12 total, 12 up, 12 in 2024-09-17T16:29:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4075364630' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:29:05.767 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:29:05.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:05 smithi071 ceph-mon[30002]: pgmap v1359: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 729 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 651 op/s; 7768/34962 objects degraded (22.218%); 1100/34962 objects misplaced (3.146%); 4.1 MiB/s, 64 objects/s recovering 2024-09-17T16:29:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:05 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7768/34962 objects degraded (22.218%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:05 smithi071 ceph-mon[30002]: osdmap e613: 12 total, 12 up, 12 in 2024-09-17T16:29:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4075364630' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:29:05.832 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:05.833 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:05.934 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:05.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:05.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:06.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:06.088 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:06.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:06.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:06.217 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:06.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:06.303 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:29:06.303 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:29:06.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:29:06.303 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:29:06.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.303 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:75081 5.e 5:7e559e6f:::benchmark_data_smithi173_90735_object75080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e470)", 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:11.118325+0000", 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 355.18171095100001, 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38537186400000001, 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75081 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118325+0000", 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118325+0000", 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118327+0000", 2024-09-17T16:29:06.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0849999999999999e-06 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118355+0000", 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7823000000000001e-05 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118376+0000", 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0421e-05 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118386+0000", 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0331e-05 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118442+0000", 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6623999999999999e-05 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118547+0000", 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104461 2024-09-17T16:29:06.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:29:06.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118639+0000", 2024-09-17T16:29:06.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2054999999999999e-05 2024-09-17T16:29:06.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:29:06.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.120640+0000", 2024-09-17T16:29:06.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002000693 2024-09-17T16:29:06.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.120805+0000", 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016576899999999999 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.503590+0000", 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38278434099999997 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.503651+0000", 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1233000000000001e-05 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.503697+0000", 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6028e-05 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:82739 5.e 5:70c6ff22:::benchmark_data_smithi173_90735_object82738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-17T16:29:06.310 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.881300+0000", 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 346.41873628500002, 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65333291699999996, 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 82739 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881300+0000", 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881300+0000", 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881301+0000", 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.584e-06 2024-09-17T16:29:06.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881325+0000", 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3825999999999999e-05 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881340+0000", 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4647999999999999e-05 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881348+0000", 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7819999999999994e-06 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881409+0000", 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1302000000000004e-05 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881527+0000", 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118077 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881614+0000", 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7399999999999997e-05 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.883793+0000", 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021789309999999998 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.885245+0000", 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014513760000000001 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534560+0000", 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64931495400000006 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534603+0000", 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3602e-05 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534633+0000", 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9434999999999999e-05 2024-09-17T16:29:06.316 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26158.0:82739 5.e e475/419 ondisk, result = 0)", 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.884046+0000", 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 346.41599029899999, 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65059267499999995, 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.884046+0000", 2024-09-17T16:29:06.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.884046+0000", 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.884046+0000", 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7299999999999996e-07 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.884050+0000", 2024-09-17T16:29:06.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.427e-06 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534498+0000", 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65044812500000004 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534503+0000", 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4609999999999999e-06 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534550+0000", 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6136e-05 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534555+0000", 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.711e-06 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534639+0000", 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3242e-05 2024-09-17T16:29:06.321 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:13328 6.5 e535/527 6:a9f49cc4:::benchmark_data_smithi173_102017_object13327:head v 535'822, mlcod=535'819)", 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:18.187814+0000", 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 228.11222258800001, 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20311574499999999, 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.187814+0000", 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.187814+0000", 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.187814+0000", 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1299999999999999e-07 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.187884+0000", 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9604999999999995e-05 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390246+0000", 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202361655 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390250+0000", 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2629999999999997e-06 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390289+0000", 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8942000000000001e-05 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390304+0000", 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4768e-05 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390899+0000", 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059588800000000004 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390929+0000", 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9910999999999999e-05 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.326 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:13492 6.4 e535/527 6:27b8e258:::benchmark_data_smithi173_102017_object13491:head v 535'822, mlcod=535'819)", 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:18.460867+0000", 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 227.83916877499999, 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23653454300000001, 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460867+0000", 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460867+0000", 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460868+0000", 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1699999999999998e-07 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460931+0000", 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3336000000000005e-05 2024-09-17T16:29:06.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696639+0000", 2024-09-17T16:29:06.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23570720000000001 2024-09-17T16:29:06.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696643+0000", 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7280000000000003e-06 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696694+0000", 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1141999999999999e-05 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696718+0000", 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3328000000000001e-05 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.697364+0000", 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00064611300000000003 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.697402+0000", 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8078999999999999e-05 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.334 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:13511 6.5 e535/527 6:a930b115:::benchmark_data_smithi173_102017_object13510:head v 535'831, mlcod=535'828)", 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:18.469756+0000", 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 227.83028007199999, 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.187752747, 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.469756+0000", 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.469756+0000", 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.469757+0000", 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.51e-07 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.469828+0000", 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1217000000000006e-05 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.656807+0000", 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18697961900000001 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.656813+0000", 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1329999999999998e-06 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.656874+0000", 2024-09-17T16:29:06.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1865000000000005e-05 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.656887+0000", 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2525e-05 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.657481+0000", 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059367700000000003 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.657509+0000", 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8160000000000001e-05 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:16253 6.1 e537/527 6:8947cbc4:::benchmark_data_smithi173_102017_object16252:head v 537'1058, mlcod=537'1053)", 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:21.027292+0000", 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 225.27274412599999, 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19452012199999999, 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.339 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.027292+0000", 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.027292+0000", 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.027293+0000", 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2200000000000004e-07 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.027310+0000", 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7105e-05 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.220867+0000", 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19355778100000001 2024-09-17T16:29:06.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.220872+0000", 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6580000000000001e-06 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.220941+0000", 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9183000000000005e-05 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.220959+0000", 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7975999999999999e-05 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.221770+0000", 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081104400000000002 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.221812+0000", 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1853000000000001e-05 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.343 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:17620 6.5 e538/527 6:a62c44a9:::benchmark_data_smithi173_102017_object17619:head v 538'1058, mlcod=537'1054)", 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:21.992300+0000", 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 224.30773567099999, 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50495759100000004, 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992300+0000", 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992300+0000", 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992301+0000", 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8500000000000001e-07 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992450+0000", 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149061 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496350+0000", 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50389984499999996 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496361+0000", 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0807e-05 2024-09-17T16:29:06.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496468+0000", 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106811 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496508+0000", 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0079e-05 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.497225+0000", 2024-09-17T16:29:06.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00071726500000000002 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.497258+0000", 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3037999999999998e-05 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:17628 6.9 e538/527 6:9546f3b4:::benchmark_data_smithi173_102017_object17627:head v 538'1099, mlcod=537'1096)", 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:21.999183+0000", 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 224.300852854, 2024-09-17T16:29:06.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45068112500000002, 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999183+0000", 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999183+0000", 2024-09-17T16:29:06.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999184+0000", 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3e-07 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999231+0000", 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7206000000000002e-05 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448811+0000", 2024-09-17T16:29:06.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44957967100000001 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448826+0000", 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5480000000000001e-05 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448977+0000", 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015076600000000001 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448996+0000", 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9069999999999999e-05 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.449834+0000", 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083746999999999999 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.449864+0000", 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0732000000000003e-05 2024-09-17T16:29:06.352 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:17644 6.5 e538/527 6:a59309a0:::benchmark_data_smithi173_102017_object17643:head v 538'1059, mlcod=537'1056)", 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:22.496194+0000", 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 223.80384237999999, 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.174573757, 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496194+0000", 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496194+0000", 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496195+0000", 2024-09-17T16:29:06.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5230000000000001e-06 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496318+0000", 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012246800000000001 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.669757+0000", 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.173438754 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.669768+0000", 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1863e-05 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.669928+0000", 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000159603 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.669963+0000", 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5336000000000002e-05 2024-09-17T16:29:06.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.670733+0000", 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000770161 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.670768+0000", 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4048999999999999e-05 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:27647 6.a e545/527 6:52a81611:::benchmark_data_smithi173_102017_object27646:head v 545'1782, mlcod=545'1779)", 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:32.709980+0000", 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 213.59005644499999, 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43213955799999998, 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.709980+0000", 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.709980+0000", 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.709981+0000", 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.23e-07 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.710035+0000", 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4273e-05 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141017+0000", 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43098198300000001 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141021+0000", 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.515e-06 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141072+0000", 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0322999999999999e-05 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141086+0000", 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4409e-05 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.142088+0000", 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010016109999999999 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.142119+0000", 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1621000000000001e-05 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:51892 6.5 e555/527 6:a9ca83c4:::benchmark_data_smithi173_102017_object51891:head v 555'3158, mlcod=555'3155)", 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:56.646438+0000", 2024-09-17T16:29:06.362 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 189.65359865100001, 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.198588405, 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.646438+0000", 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.646438+0000", 2024-09-17T16:29:06.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.646438+0000", 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.646493+0000", 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4975999999999997e-05 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.843107+0000", 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19661374200000001 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.843114+0000", 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8630000000000002e-06 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.843187+0000", 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3695999999999994e-05 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:06.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.843204+0000", 2024-09-17T16:29:06.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6208000000000001e-05 2024-09-17T16:29:06.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.844992+0000", 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017883969999999999 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.845026+0000", 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3800999999999997e-05 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:51898 6.f e555/527 6:f7f1db19:::benchmark_data_smithi173_102017_object51897:head v 555'3218, mlcod=555'3215)", 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:56.654459+0000", 2024-09-17T16:29:06.368 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 189.64557669199999, 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17637305, 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.654459+0000", 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.654459+0000", 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.654461+0000", 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1289999999999999e-06 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.654567+0000", 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106883 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.829210+0000", 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17464289099999999 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.829216+0000", 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4190000000000001e-06 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.829267+0000", 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1507e-05 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.829281+0000", 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.361e-05 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.830800+0000", 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015194780000000001 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:56.830833+0000", 2024-09-17T16:29:06.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2132999999999999e-05 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:85438 6.f e565/527 6:f29abae7:::benchmark_data_smithi173_102017_object85437:head v 565'5353, mlcod=565'5351)", 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:26:27.791488+0000", 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 158.508548265, 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.243614578, 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.373 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791488+0000", 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791488+0000", 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791489+0000", 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1900000000000001e-07 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791572+0000", 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3040000000000005e-05 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032638+0000", 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24106640200000001 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032645+0000", 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.55e-06 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032734+0000", 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9209000000000006e-05 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032755+0000", 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0618000000000001e-05 2024-09-17T16:29:06.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.035062+0000", 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023077140000000002 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.035102+0000", 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0126000000000002e-05 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:87629 6.9 e566/527 6:975355a4:::benchmark_data_smithi173_102017_object87628:head v 566'5443, mlcod=566'5441)", 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:26:30.268719+0000", 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 156.031317586, 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22957050300000001, 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268719+0000", 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268719+0000", 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268721+0000", 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.12e-06 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268753+0000", 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2620999999999998e-05 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494429+0000", 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22567617500000001 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494433+0000", 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3000000000000002e-06 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494476+0000", 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2908999999999999e-05 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494495+0000", 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9335999999999999e-05 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.498267+0000", 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037720879999999998 2024-09-17T16:29:06.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.498289+0000", 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1954e-05 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:107227 6.5 e580/527 6:a39a6f47:::benchmark_data_smithi173_102017_object14079:head v 580'6674, mlcod=580'6670)", 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:00.798388+0000", 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 125.50164850199999, 2024-09-17T16:29:06.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.185674703, 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:00.798388+0000", 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:00.798388+0000", 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:00.798388+0000", 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4200000000000005e-07 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:00.798394+0000", 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7599999999999999e-06 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:00.981093+0000", 2024-09-17T16:29:06.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18269913800000001 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:00.981099+0000", 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8109999999999999e-06 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:00.981153+0000", 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3439000000000001e-05 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:00.981175+0000", 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2099e-05 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:00.984021+0000", 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028467200000000001 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:00.984062+0000", 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0893999999999998e-05 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.386 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:114687 6.9 e582/527 6:96e26e86:::benchmark_data_smithi173_102017_object21539:head v 582'7080, mlcod=582'7078)", 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:12.569600+0000", 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 113.730436152, 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24650798600000001, 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569600+0000", 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569600+0000", 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569601+0000", 2024-09-17T16:29:06.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.569999999999999e-07 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569607+0000", 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2380000000000002e-06 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812466+0000", 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24285875200000001 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812471+0000", 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9960000000000001e-06 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812530+0000", 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9333999999999998e-05 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812551+0000", 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1158999999999999e-05 2024-09-17T16:29:06.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.816069+0000", 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035176840000000001 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.816108+0000", 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8865999999999997e-05 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:115587 6.5 e582/527 6:ae192cb4:::benchmark_data_smithi173_102017_object22439:head v 582'7137, mlcod=582'7134)", 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:14.230950+0000", 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 112.06908625299999, 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29268783799999998, 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230950+0000", 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230950+0000", 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230951+0000", 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5700000000000001e-07 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230956+0000", 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6210000000000001e-06 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520676+0000", 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.289719173 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520684+0000", 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.833e-06 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520770+0000", 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5866999999999996e-05 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520790+0000", 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9266000000000002e-05 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.523606+0000", 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028161639999999999 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.523638+0000", 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2057000000000002e-05 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:116184 6.f e583/527 6:f2e9c488:::benchmark_data_smithi173_102017_object23036:head v 583'7277, mlcod=583'7273)", 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:29.905958+0000", 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 96.394077894999995, 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23584665899999999, 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.396 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905958+0000", 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905958+0000", 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905960+0000", 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.282e-06 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905967+0000", 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.61e-06 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136208+0000", 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23024043699999999 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136213+0000", 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.955e-06 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136316+0000", 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103074 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136341+0000", 2024-09-17T16:29:06.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5809e-05 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.141768+0000", 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054267589999999997 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.141805+0000", 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6733000000000002e-05 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.400 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:116211 6.f e583/527 6:f64da5b7:::benchmark_data_smithi173_102017_object23063:head v 583'7278, mlcod=583'7274)", 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:29.939601+0000", 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 96.360435159999994, 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20233385800000001, 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.939601+0000", 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.939601+0000", 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.939602+0000", 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.16e-07 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.939608+0000", 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4840000000000001e-06 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136235+0000", 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19662713400000001 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136240+0000", 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1799999999999998e-06 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.139513+0000", 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032737299999999999 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.139537+0000", 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3550999999999999e-05 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.141912+0000", 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023747249999999998 2024-09-17T16:29:06.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:06.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:06.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:06.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.141935+0000", 2024-09-17T16:29:06.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3238e-05 2024-09-17T16:29:06.405 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.405 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.405 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.405 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:06.405 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:06.405 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:29:06.423 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:06.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:06.430 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:06.531 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:06.555 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:06.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:06.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:06.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:06 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:06.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4075364630' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:29:06.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:06 smithi017 ceph-mon[27571]: osdmap e614: 12 total, 12 up, 12 in 2024-09-17T16:29:06.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:06.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:06.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:06.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:06.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:06.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:06 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4075364630' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:29:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:06 smithi071 ceph-mon[30002]: osdmap e614: 12 total, 12 up, 12 in 2024-09-17T16:29:06.879 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:06 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:06.879 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4075364630' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:29:06.879 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:06 smithi173 ceph-mon[30530]: osdmap e614: 12 total, 12 up, 12 in 2024-09-17T16:29:06.909 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:06.941 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:06.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:07.037 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:29:07.043 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:07.068 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:07.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:07.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:07.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:07.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:07.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:07.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:29:07.364 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:07.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:07.464 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:07.466 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:07.494 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:07.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:07.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:07.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:07.622 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:07.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:07.626 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:29:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:07 smithi173 ceph-mon[30530]: pgmap v1362: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 661 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 648 op/s; 7768/31698 objects degraded (24.506%); 856/31698 objects misplaced (2.700%); 6.6 MiB/s, 105 objects/s recovering 2024-09-17T16:29:07.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:07.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:07 smithi017 ceph-mon[27571]: pgmap v1362: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 661 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 648 op/s; 7768/31698 objects degraded (24.506%); 856/31698 objects misplaced (2.700%); 6.6 MiB/s, 105 objects/s recovering 2024-09-17T16:29:07.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:07.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:07 smithi071 ceph-mon[30002]: pgmap v1362: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 661 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 648 op/s; 7768/31698 objects degraded (24.506%); 856/31698 objects misplaced (2.700%); 6.6 MiB/s, 105 objects/s recovering 2024-09-17T16:29:07.850 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:07.888 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:07.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:07.963 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:29:07.964 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:29:07.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:08.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:08.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:08.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:29:08.081 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:29:08.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:08.115 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:29:08.115 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:29:08.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:08.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:08.150 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:29:08.151 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:29:08.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:29:08.186 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:29:08.220 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:29:08.220 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:29:08.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:08.241 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:29:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:29:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:179593 6.1 e614/527 6:8a3d67e6:::benchmark_data_smithi173_102017_object86445:head v 614'11479, mlcod=614'11473)", 2024-09-17T16:29:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:08.227215+0000", 2024-09-17T16:29:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.011551364999999999, 2024-09-17T16:29:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011571984, 2024-09-17T16:29:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:08.227215+0000", 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:08.227215+0000", 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:08.227216+0000", 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.026e-06 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:08.227223+0000", 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3139999999999998e-06 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:08.234406+0000", 2024-09-17T16:29:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0071831869999999997 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:08.234415+0000", 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0909999999999999e-06 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:08.234574+0000", 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015911899999999999 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:08.234594+0000", 2024-09-17T16:29:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9839000000000001e-05 2024-09-17T16:29:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:29:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-09-17T16:29:08.245 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:29:08.256 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:29:08.256 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:29:08.266 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:08.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:08.294 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:29:08.294 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:29:08.330 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:29:08.331 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:29:08.365 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:29:08.365 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:29:08.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:08.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:08.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:08.399 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:29:08.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:29:08.407 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:29:08.435 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:29:08.435 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:29:08.474 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:29:08.475 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:29:08.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:08.556 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:08.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:08.580 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/980912898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:08.580 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:08 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:29:08.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:08.689 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:08.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:08.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:08.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:08.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/980912898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:08.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:08 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:29:08.817 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/980912898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:08.817 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:08 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:29:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:08.822 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:08.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:08.883 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1362,"stamp":"2024-09-17T16:29:05.979966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":692847160,"num_objects":10566,"num_object_clones":0,"num_object_copies":31698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7768,"num_objects_misplaced":856,"num_objects_unfound":0,"num_objects_dirty":10566,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":175851,"num_write_kb":5963431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25063,"num_bytes_recovered":1589388352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160720,"ondisk_log_size":160720,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8273184,"kb_used_data":3129484,"kb_used_omap":58,"kb_used_meta":5143365,"kb_avail":1116668640,"statfs":{"total":1151940427776,"available":1143468687360,"internally_reserved":0,"allocated":3204591616,"data_stored":3199822199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":5266805923},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-254935040,"num_objects":-3890,"num_object_clones":0,"num_object_copies":-11670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":63,"num_objects_misplaced":-696,"num_objects_unfound":0,"num_objects_dirty":-3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3890,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":41484288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001357"},"pg_stats":[{"pgid":"6.f","version":"611'10988","reported_seq":11273,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-09-17T16:29:01.802616+0000","last_change":"2024-09-17T16:27:46.912831+0000","last_active":"2024-09-17T16:29:01.802616+0000","last_peered":"2024-09-17T16:29:01.802616+0000","last_clean":"2024-09-17T16:29:01.802616+0000","last_became_active":"2024-09-17T16:25:03.056038+0000","last_became_peered":"2024-09-17T16:25:03.056038+0000","last_unstale":"2024-09-17T16:29:01.802616+0000","last_undegraded":"2024-09-17T16:29:01.802616+0000","last_fullsized":"2024-09-17T16:29:01.802616+0000","mapping_epoch":527,"log_start":"536'900","ondisk_log_start":"536'900","created":527,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"588'7921","last_scrub_stamp":"2024-09-17T16:27:46.912623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:27:46.912623+0000","objects_scrubbed":3791,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:51.645905+0000","scrub_duration":1060,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46530560,"num_objects":710,"num_object_clones":0,"num_object_copies":2130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10988,"num_write_kb":374336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"613'11101","reported_seq":13518,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-09-17T16:29:04.911237+0000","last_change":"2024-09-17T16:26:32.744077+0000","last_active":"2024-09-17T16:29:04.911237+0000","last_peered":"2024-09-17T16:29:04.911237+0000","last_clean":"2024-09-17T16:29:04.911237+0000","last_became_active":"2024-09-17T16:25:41.576009+0000","last_became_peered":"2024-09-17T16:25:41.576009+0000","last_unstale":"2024-09-17T16:29:04.911237+0000","last_undegraded":"2024-09-17T16:29:04.911237+0000","last_fullsized":"2024-09-17T16:29:04.911237+0000","mapping_epoch":549,"log_start":"540'1100","ondisk_log_start":"540'1100","created":527,"last_epoch_clean":550,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5655","last_scrub_stamp":"2024-09-17T16:26:32.744015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:26:32.744015+0000","objects_scrubbed":5605,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36241408,"num_objects":553,"num_object_clones":0,"num_object_copies":1659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11101,"num_write_kb":372928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":70778880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"614'10911","reported_seq":24304,"reported_epoch":614,"state":"active+clean","last_fresh":"2024-09-17T16:29:05.695038+0000","last_change":"2024-09-17T16:28:25.662397+0000","last_active":"2024-09-17T16:29:05.695038+0000","last_peered":"2024-09-17T16:29:05.695038+0000","last_clean":"2024-09-17T16:29:05.695038+0000","last_became_active":"2024-09-17T16:28:25.584586+0000","last_became_peered":"2024-09-17T16:28:25.584586+0000","last_unstale":"2024-09-17T16:29:05.695038+0000","last_undegraded":"2024-09-17T16:29:05.695038+0000","last_fullsized":"2024-09-17T16:29:05.695038+0000","mapping_epoch":602,"log_start":"536'900","ondisk_log_start":"536'900","created":527,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:25:02.000918+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719448,"num_objects":485,"num_object_clones":0,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10911,"num_write_kb":364609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6628,"num_bytes_recovered":430178304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"611'11030","reported_seq":22708,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-09-17T16:29:01.668913+0000","last_change":"2024-09-17T16:28:49.487612+0000","last_active":"2024-09-17T16:29:01.668913+0000","last_peered":"2024-09-17T16:29:01.668913+0000","last_clean":"2024-09-17T16:29:01.668913+0000","last_became_active":"2024-09-17T16:28:49.444632+0000","last_became_peered":"2024-09-17T16:28:49.444632+0000","last_unstale":"2024-09-17T16:29:01.668913+0000","last_undegraded":"2024-09-17T16:29:01.668913+0000","last_fullsized":"2024-09-17T16:29:01.668913+0000","mapping_epoch":608,"log_start":"537'1000","ondisk_log_start":"537'1000","created":527,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'5014","last_scrub_stamp":"2024-09-17T16:26:22.357111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:26:22.357111+0000","objects_scrubbed":4965,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1975,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50987008,"num_objects":778,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11030,"num_write_kb":377856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5714,"num_bytes_recovered":351010816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,8],"acting":[3,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"611'11015","reported_seq":13438,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-09-17T16:29:01.587613+0000","last_change":"2024-09-17T16:26:34.960806+0000","last_active":"2024-09-17T16:29:01.587613+0000","last_peered":"2024-09-17T16:29:01.587613+0000","last_clean":"2024-09-17T16:29:01.587613+0000","last_became_active":"2024-09-17T16:25:33.529867+0000","last_became_peered":"2024-09-17T16:25:33.529867+0000","last_unstale":"2024-09-17T16:29:01.587613+0000","last_undegraded":"2024-09-17T16:29:01.587613+0000","last_fullsized":"2024-09-17T16:29:01.587613+0000","mapping_epoch":545,"log_start":"537'1000","ondisk_log_start":"537'1000","created":527,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5710","last_scrub_stamp":"2024-09-17T16:26:34.960740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:26:34.960740+0000","objects_scrubbed":5710,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46727168,"num_objects":713,"num_object_clones":0,"num_object_copies":2139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11015,"num_write_kb":375296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1084,"num_bytes_recovered":71041024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"614'11121","reported_seq":14156,"reported_epoch":614,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-17T16:29:05.616180+0000","last_change":"2024-09-17T16:28:32.802232+0000","last_active":"2024-09-17T16:29:05.616180+0000","last_peered":"2024-09-17T16:29:05.616180+0000","last_clean":"2024-09-17T16:26:33.602387+0000","last_became_active":"2024-09-17T16:26:35.984418+0000","last_became_peered":"2024-09-17T16:26:35.984418+0000","last_unstale":"2024-09-17T16:29:05.616180+0000","last_undegraded":"2024-09-17T16:26:35.626426+0000","last_fullsized":"2024-09-17T16:26:35.527300+0000","mapping_epoch":568,"log_start":"537'1100","ondisk_log_start":"537'1100","created":527,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:25:02.000918+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36110336,"num_objects":551,"num_object_clones":0,"num_object_copies":1653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3936,"num_objects_misplaced":318,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11121,"num_write_kb":373504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1423,"num_bytes_recovered":92012544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[6,0],"avail_no_missing":["6","0","9"],"object_location_counts":[{"shards":"0,6","objects":-3703},{"shards":"0,6,9","objects":318}],"blocked_by":[],"up_primary":5,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"611'10808","reported_seq":10987,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-09-17T16:29:01.350579+0000","last_change":"2024-09-17T16:25:03.056876+0000","last_active":"2024-09-17T16:29:01.350579+0000","last_peered":"2024-09-17T16:29:01.350579+0000","last_clean":"2024-09-17T16:29:01.350579+0000","last_became_active":"2024-09-17T16:25:03.056558+0000","last_became_peered":"2024-09-17T16:25:03.056558+0000","last_unstale":"2024-09-17T16:29:01.350579+0000","last_undegraded":"2024-09-17T16:29:01.350579+0000","last_fullsized":"2024-09-17T16:29:01.350579+0000","mapping_epoch":527,"log_start":"535'800","ondisk_log_start":"535'800","created":527,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:25:02.000918+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44302336,"num_objects":676,"num_object_clones":0,"num_object_copies":2028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10808,"num_write_kb":367488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"611'10777","reported_seq":11017,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-09-17T16:29:01.695897+0000","last_change":"2024-09-17T16:26:27.360880+0000","last_active":"2024-09-17T16:29:01.695897+0000","last_peered":"2024-09-17T16:29:01.695897+0000","last_clean":"2024-09-17T16:29:01.695897+0000","last_became_active":"2024-09-17T16:25:03.055684+0000","last_became_peered":"2024-09-17T16:25:03.055684+0000","last_unstale":"2024-09-17T16:29:01.695897+0000","last_undegraded":"2024-09-17T16:29:01.695897+0000","last_fullsized":"2024-09-17T16:29:01.695897+0000","mapping_epoch":527,"log_start":"535'700","ondisk_log_start":"535'700","created":527,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"565'5232","last_scrub_stamp":"2024-09-17T16:26:27.360655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:26:27.360655+0000","objects_scrubbed":5170,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46858240,"num_objects":715,"num_object_clones":0,"num_object_copies":2145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":367744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,11],"acting":[3,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"611'10795","reported_seq":11030,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-09-17T16:29:01.699669+0000","last_change":"2024-09-17T16:26:19.419963+0000","last_active":"2024-09-17T16:29:01.699669+0000","last_peered":"2024-09-17T16:29:01.699669+0000","last_clean":"2024-09-17T16:29:01.699669+0000","last_became_active":"2024-09-17T16:25:03.036007+0000","last_became_peered":"2024-09-17T16:25:03.036007+0000","last_unstale":"2024-09-17T16:29:01.699669+0000","last_undegraded":"2024-09-17T16:29:01.699669+0000","last_fullsized":"2024-09-17T16:29:01.699669+0000","mapping_epoch":527,"log_start":"535'700","ondisk_log_start":"535'700","created":527,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'4781","last_scrub_stamp":"2024-09-17T16:26:19.419830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:26:19.419830+0000","objects_scrubbed":4750,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46727168,"num_objects":713,"num_object_clones":0,"num_object_copies":2139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10795,"num_write_kb":368256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"612'10927","reported_seq":11215,"reported_epoch":612,"state":"active+clean","last_fresh":"2024-09-17T16:29:03.039655+0000","last_change":"2024-09-17T16:27:41.232123+0000","last_active":"2024-09-17T16:29:03.039655+0000","last_peered":"2024-09-17T16:29:03.039655+0000","last_clean":"2024-09-17T16:29:03.039655+0000","last_became_active":"2024-09-17T16:25:03.055626+0000","last_became_peered":"2024-09-17T16:25:03.055626+0000","last_unstale":"2024-09-17T16:29:03.039655+0000","last_undegraded":"2024-09-17T16:29:03.039655+0000","last_fullsized":"2024-09-17T16:29:03.039655+0000","mapping_epoch":527,"log_start":"536'900","ondisk_log_start":"536'900","created":527,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'7616","last_scrub_stamp":"2024-09-17T16:27:41.232066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:27:41.232066+0000","objects_scrubbed":4003,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:09.266447+0000","scrub_duration":844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44630016,"num_objects":681,"num_object_clones":0,"num_object_copies":2043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10927,"num_write_kb":371456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"611'10773","reported_seq":24385,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-09-17T16:29:02.226174+0000","last_change":"2024-09-17T16:28:47.996888+0000","last_active":"2024-09-17T16:29:02.226174+0000","last_peered":"2024-09-17T16:29:02.226174+0000","last_clean":"2024-09-17T16:29:02.226174+0000","last_became_active":"2024-09-17T16:28:34.447224+0000","last_became_peered":"2024-09-17T16:28:34.447224+0000","last_unstale":"2024-09-17T16:29:02.226174+0000","last_undegraded":"2024-09-17T16:29:02.226174+0000","last_fullsized":"2024-09-17T16:29:02.226174+0000","mapping_epoch":604,"log_start":"535'700","ondisk_log_start":"535'700","created":527,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5381","last_scrub_stamp":"2024-09-17T16:26:30.343206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:26:30.343206+0000","objects_scrubbed":5333,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:49.689771+0000","scrub_duration":1631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45023232,"num_objects":687,"num_object_clones":0,"num_object_copies":2061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10773,"num_write_kb":366720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6673,"num_bytes_recovered":418709504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"612'11053","reported_seq":11340,"reported_epoch":612,"state":"active+clean","last_fresh":"2024-09-17T16:29:03.067380+0000","last_change":"2024-09-17T16:27:50.944281+0000","last_active":"2024-09-17T16:29:03.067380+0000","last_peered":"2024-09-17T16:29:03.067380+0000","last_clean":"2024-09-17T16:29:03.067380+0000","last_became_active":"2024-09-17T16:25:03.024824+0000","last_became_peered":"2024-09-17T16:25:03.024824+0000","last_unstale":"2024-09-17T16:29:03.067380+0000","last_undegraded":"2024-09-17T16:29:03.067380+0000","last_fullsized":"2024-09-17T16:29:03.067380+0000","mapping_epoch":527,"log_start":"536'1000","ondisk_log_start":"536'1000","created":527,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"588'8171","last_scrub_stamp":"2024-09-17T16:27:50.944159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:27:50.944159+0000","objects_scrubbed":3525,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:57.491888+0000","scrub_duration":652,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41353216,"num_objects":631,"num_object_clones":0,"num_object_copies":1893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11053,"num_write_kb":373888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,8],"acting":[7,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"613'10936","reported_seq":11119,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-09-17T16:29:04.888379+0000","last_change":"2024-09-17T16:25:03.024334+0000","last_active":"2024-09-17T16:29:04.888379+0000","last_peered":"2024-09-17T16:29:04.888379+0000","last_clean":"2024-09-17T16:29:04.888379+0000","last_became_active":"2024-09-17T16:25:03.023597+0000","last_became_peered":"2024-09-17T16:25:03.023597+0000","last_unstale":"2024-09-17T16:29:04.888379+0000","last_undegraded":"2024-09-17T16:29:04.888379+0000","last_fullsized":"2024-09-17T16:29:04.888379+0000","mapping_epoch":527,"log_start":"536'900","ondisk_log_start":"536'900","created":527,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:25:02.000918+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38535168,"num_objects":588,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10936,"num_write_kb":368768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"612'11315","reported_seq":11518,"reported_epoch":612,"state":"active+clean","last_fresh":"2024-09-17T16:29:04.374964+0000","last_change":"2024-09-17T16:28:48.386791+0000","last_active":"2024-09-17T16:29:04.374964+0000","last_peered":"2024-09-17T16:29:04.374964+0000","last_clean":"2024-09-17T16:29:04.374964+0000","last_became_active":"2024-09-17T16:25:03.056580+0000","last_became_peered":"2024-09-17T16:25:03.056580+0000","last_unstale":"2024-09-17T16:29:04.374964+0000","last_undegraded":"2024-09-17T16:29:04.374964+0000","last_fullsized":"2024-09-17T16:29:04.374964+0000","mapping_epoch":527,"log_start":"541'1300","ondisk_log_start":"541'1300","created":527,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"607'10640","last_scrub_stamp":"2024-09-17T16:28:48.386723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:28:48.386723+0000","objects_scrubbed":1286,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:15.826602+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39780352,"num_objects":607,"num_object_clones":0,"num_object_copies":1821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11315,"num_write_kb":381504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.0","version":"611'11156","reported_seq":11333,"reported_epoch":611,"state":"active+clean","last_fresh":"2024-09-17T16:29:01.356656+0000","last_change":"2024-09-17T16:25:03.023256+0000","last_active":"2024-09-17T16:29:01.356656+0000","last_peered":"2024-09-17T16:29:01.356656+0000","last_clean":"2024-09-17T16:29:01.356656+0000","last_became_active":"2024-09-17T16:25:03.022876+0000","last_became_peered":"2024-09-17T16:25:03.022876+0000","last_unstale":"2024-09-17T16:29:01.356656+0000","last_undegraded":"2024-09-17T16:29:01.356656+0000","last_fullsized":"2024-09-17T16:29:01.356656+0000","mapping_epoch":527,"log_start":"540'1100","ondisk_log_start":"540'1100","created":527,"last_epoch_clean":528,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:25:02.000918+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50069504,"num_objects":764,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11156,"num_write_kb":381440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"611'11022","reported_seq":16132,"reported_epoch":611,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-17T16:29:01.691247+0000","last_change":"2024-09-17T16:28:24.387705+0000","last_active":"2024-09-17T16:29:01.691247+0000","last_peered":"2024-09-17T16:29:01.691247+0000","last_clean":"2024-09-17T16:26:33.598087+0000","last_became_active":"2024-09-17T16:26:35.972346+0000","last_became_peered":"2024-09-17T16:26:35.972346+0000","last_unstale":"2024-09-17T16:29:01.691247+0000","last_undegraded":"2024-09-17T16:26:35.672260+0000","last_fullsized":"2024-09-17T16:26:35.523379+0000","mapping_epoch":568,"log_start":"537'1000","ondisk_log_start":"537'1000","created":527,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:25:02.000918+0000","last_clean_scrub_stamp":"2024-09-17T16:25:02.000918+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46661632,"num_objects":712,"num_object_clones":0,"num_object_copies":2136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3832,"num_objects_misplaced":538,"num_objects_unfound":0,"num_objects_dirty":712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":375488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2457,"num_bytes_recovered":154599424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[3,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":538},{"shards":"3,4","objects":-3658}],"blocked_by":[],"up_primary":5,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":181,"reported_epoch":613,"state":"active+clean","last_fresh":"2024-09-17T16:29:04.574600+0000","last_change":"2024-09-17T16:25:09.585157+0000","last_active":"2024-09-17T16:29:04.574600+0000","last_peered":"2024-09-17T16:29:04.574600+0000","last_clean":"2024-09-17T16:29:04.574600+0000","last_became_active":"2024-09-17T16:25:09.545303+0000","last_became_peered":"2024-09-17T16:25:09.545303+0000","last_unstale":"2024-09-17T16:29:04.574600+0000","last_undegraded":"2024-09-17T16:29:04.574600+0000","last_fullsized":"2024-09-17T16:29:04.574600+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:22:06.807231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:22:06.807231+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":692256792,"num_objects":10564,"num_object_clones":0,"num_object_copies":31692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7768,"num_objects_misplaced":856,"num_objects_unfound":0,"num_objects_dirty":10564,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175728,"num_write_kb":5961281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25059,"num_bytes_recovered":1588330496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3135389696,"data_stored":3135373408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160628,"ondisk_log_size":160628,"up":48,"acting":46,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":421224,"kb_used_data":38804,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93323928,"statfs":{"total":95995035648,"available":95563702272,"internally_reserved":0,"allocated":39735296,"data_stored":39339721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":391576004},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547440,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":490500,"kb_used_data":102448,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93254652,"statfs":{"total":95995035648,"available":95492763648,"internally_reserved":0,"allocated":104906752,"data_stored":104498211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":397277632},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1060952,"kb_used_data":552200,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92684200,"statfs":{"total":95995035648,"available":94908620800,"internally_reserved":0,"allocated":565452800,"data_stored":565061652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":520944064},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005306,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534316,"kb_used_data":156060,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93210836,"statfs":{"total":95995035648,"available":95447896064,"internally_reserved":0,"allocated":159805440,"data_stored":159406333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963782,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":977064,"kb_used_data":513492,"kb_used_omap":41,"kb_used_meta":463510,"kb_avail":92768088,"statfs":{"total":95995035648,"available":94994522112,"internally_reserved":0,"allocated":525815808,"data_stored":525420253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":474634547},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495876,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":535544,"kb_used_data":107304,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93209608,"statfs":{"total":95995035648,"available":95446638592,"internally_reserved":0,"allocated":109879296,"data_stored":109475732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":438499782},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724807,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":640824,"kb_used_data":132068,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93104328,"statfs":{"total":95995035648,"available":95338831872,"internally_reserved":0,"allocated":135237632,"data_stored":134833548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986447,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":573484,"kb_used_data":154972,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93171668,"statfs":{"total":95995035648,"available":95407788032,"internally_reserved":0,"allocated":158691328,"data_stored":158292221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149969,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":882524,"kb_used_data":386248,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92862628,"statfs":{"total":95995035648,"available":95091331072,"internally_reserved":0,"allocated":395517952,"data_stored":395126804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":508164551},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411607,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583960,"kb_used_data":233480,"kb_used_omap":1,"kb_used_meta":350462,"kb_avail":93161192,"statfs":{"total":95995035648,"available":95397060608,"internally_reserved":0,"allocated":239083520,"data_stored":238692372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":358873541},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575133,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866972,"kb_used_data":547020,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92878180,"statfs":{"total":95995035648,"available":95107256320,"internally_reserved":0,"allocated":560148480,"data_stored":559753260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738657,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":705820,"kb_used_data":205388,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93039332,"statfs":{"total":95995035648,"available":95272275968,"internally_reserved":0,"allocated":210317312,"data_stored":209922092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":206704640,"data_stored":206700568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":551620608,"data_stored":551616536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":235929600,"data_stored":235929600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":382271488,"data_stored":382271488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":148242432,"data_stored":148242432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":129830912,"data_stored":129826840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":105250816,"data_stored":105250816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":518586368,"data_stored":518586368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":155516928,"data_stored":155516928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":562888704,"data_stored":562888704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":100864000,"data_stored":100859928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":37683200,"data_stored":37683200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:29:08.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:08.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:08.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:08.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:29:09.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:09.076 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:09.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:09.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:09.204 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:09.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:09.306 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:09.320 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:29:09.338 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:09.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:09.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:09.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:09.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:09.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:09.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:09.608 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:09.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:09.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:09.744 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:09 smithi173 ceph-mon[30530]: from='client.27005 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:29:09.744 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:09 smithi173 ceph-mon[30530]: pgmap v1363: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 528 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 7829/25338 objects degraded (30.898%); 633/25338 objects misplaced (2.498%); 4.2 MiB/s, 67 objects/s recovering 2024-09-17T16:29:09.748 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:09.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:09.796 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:09 smithi071 ceph-mon[30002]: from='client.27005 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:29:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:09 smithi071 ceph-mon[30002]: pgmap v1363: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 528 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 7829/25338 objects degraded (30.898%); 633/25338 objects misplaced (2.498%); 4.2 MiB/s, 67 objects/s recovering 2024-09-17T16:29:09.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:09.861 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:29:09.861 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:29:09.881 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:09.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:09.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:09 smithi017 ceph-mon[27571]: from='client.27005 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:29:09.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:09 smithi017 ceph-mon[27571]: pgmap v1363: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 528 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 638 op/s; 7829/25338 objects degraded (30.898%); 633/25338 objects misplaced (2.498%); 4.2 MiB/s, 67 objects/s recovering 2024-09-17T16:29:09.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:10.011 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:10.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:10.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:10.147 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:10.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:10.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:10.249 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:10.279 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:10.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:10.381 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:10.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:10.413 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:10.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:10.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:10.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:10.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:10.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:10.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:29:10.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:10.668 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:10.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7829/25338 objects degraded (30.898%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:10 smithi017 ceph-mon[27571]: osdmap e615: 12 total, 12 up, 12 in 2024-09-17T16:29:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:10 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:10 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:10.769 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:10.801 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:10.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:10.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:10.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7829/25338 objects degraded (30.898%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:10.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:10 smithi071 ceph-mon[30002]: osdmap e615: 12 total, 12 up, 12 in 2024-09-17T16:29:10.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:10 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:10.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:10 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:10.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7829/25338 objects degraded (30.898%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:10.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:10 smithi173 ceph-mon[30530]: osdmap e615: 12 total, 12 up, 12 in 2024-09-17T16:29:10.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:10 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:10.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:10 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:10.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:10.926 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:29:10.930 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:10.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:11.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:11.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:11.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:11.159 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:11.188 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:11.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:11.290 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:11.321 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:11.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:11.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:11.451 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:11.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:11.530 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:29:11.530 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:29:11.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:11.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:11.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:11.595 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:29:11.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:11.720 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:11.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:11.740 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:29:11.777 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:11 smithi017 ceph-mon[27571]: pgmap v1365: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 457 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 7829/21936 objects degraded (35.690%); 633/21936 objects misplaced (2.886%); 4.2 MiB/s, 67 objects/s recovering 2024-09-17T16:29:11.778 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:11 smithi017 ceph-mon[27571]: 6.5 scrub starts 2024-09-17T16:29:11.778 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:11 smithi017 ceph-mon[27571]: osdmap e616: 12 total, 12 up, 12 in 2024-09-17T16:29:11.778 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:11 smithi017 ceph-mon[27571]: 6.5 scrub ok 2024-09-17T16:29:11.778 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/225024159' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:11.778 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:11 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:11 smithi071 ceph-mon[30002]: pgmap v1365: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 457 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 7829/21936 objects degraded (35.690%); 633/21936 objects misplaced (2.886%); 4.2 MiB/s, 67 objects/s recovering 2024-09-17T16:29:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:11 smithi071 ceph-mon[30002]: 6.5 scrub starts 2024-09-17T16:29:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:11 smithi071 ceph-mon[30002]: osdmap e616: 12 total, 12 up, 12 in 2024-09-17T16:29:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:11 smithi071 ceph-mon[30002]: 6.5 scrub ok 2024-09-17T16:29:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/225024159' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:11 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:11.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:11.864 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:11.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:11 smithi173 ceph-mon[30530]: pgmap v1365: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 457 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 723 op/s; 7829/21936 objects degraded (35.690%); 633/21936 objects misplaced (2.886%); 4.2 MiB/s, 67 objects/s recovering 2024-09-17T16:29:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:11 smithi173 ceph-mon[30530]: 6.5 scrub starts 2024-09-17T16:29:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:11 smithi173 ceph-mon[30530]: osdmap e616: 12 total, 12 up, 12 in 2024-09-17T16:29:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:11 smithi173 ceph-mon[30530]: 6.5 scrub ok 2024-09-17T16:29:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/225024159' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:11 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:11.966 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:11.997 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:11.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:12.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:12.135 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:12.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:12.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:12.270 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:12.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:12.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:29:12.372 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:12.411 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:12.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:12.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:12.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:12.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:12.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:12.570 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:12.603 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:29:12.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:12 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:29:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:12 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:29:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:12 smithi173 ceph-mon[30530]: osdmap e617: 12 total, 12 up, 12 in 2024-09-17T16:29:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:12 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:29:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:12 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:29:12.663 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:12.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:12.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:12 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:29:12.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:12 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:29:12.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:12 smithi017 ceph-mon[27571]: osdmap e617: 12 total, 12 up, 12 in 2024-09-17T16:29:12.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:12 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:29:12.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:12 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:29:12.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:12.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:12.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:12 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:29:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:12 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:29:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:12 smithi071 ceph-mon[30002]: osdmap e617: 12 total, 12 up, 12 in 2024-09-17T16:29:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:12 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:29:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:12 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:29:12.888 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:12.929 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:12.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:13.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:13.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:13.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:13.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:13.156 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:13.183 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:13.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:13.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:13.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:13.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:13.346 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:29:13.346 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:29:13.412 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:13.437 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:29:13.437 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:29:13.440 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:13.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:13.473 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:29:13.473 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:29:13.509 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:29:13.510 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:29:13.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:13.544 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:29:13.544 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:29:13.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:13.568 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:13.581 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:29:13.581 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:29:13.617 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:29:13.618 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:29:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:13 smithi173 ceph-mon[30530]: pgmap v1368: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 318 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s; 7895/15234 objects degraded (51.825%); 217/15234 objects misplaced (1.424%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T16:29:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:13 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:29:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:13 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:29:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:13 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:29:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/972415199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:13.646 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:29:13.646 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:29:13.669 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:13.681 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:29:13.681 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:29:13.697 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:13.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:13.716 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:29:13.717 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:29:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:13 smithi017 ceph-mon[27571]: pgmap v1368: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 318 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s; 7895/15234 objects degraded (51.825%); 217/15234 objects misplaced (1.424%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T16:29:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:13 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:29:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:13 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:29:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:13 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:29:13.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/972415199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:13.755 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:29:13.755 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:29:13.793 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:29:13.794 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:29:13.798 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:13 smithi071 ceph-mon[30002]: pgmap v1368: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 318 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s; 7895/15234 objects degraded (51.825%); 217/15234 objects misplaced (1.424%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T16:29:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:13 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:29:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:13 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:29:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:13 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:29:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/972415199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:13.823 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:13.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:13.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:13.828 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:29:13.828 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:29:13.885 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:29:13.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:29:13.885 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T16:29:13.885 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:29:13.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:13.951 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:13.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:14.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:14.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:14.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:14.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:14.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:14.203 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:14.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:14.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:14.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:14.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:14.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:29:14.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:14.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:29:14.470 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:14.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:14.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:14.596 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:14.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:14.656 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:29:14.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:14.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:14.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:14.739 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:14.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:14.854 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:14.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:14.871 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:14.871 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:14.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:14.987 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:14.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:14.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:15.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:15.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:15.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:15.218 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:29:15.218 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:29:15.218 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:29:15.218 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:29:15.218 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:29:15.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:15.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:15.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:15.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:15.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:15.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:15.471 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:15.493 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:15.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:15.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:15.595 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:15.628 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:15.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:15.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:15.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:15.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:15.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:15.812 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:15 smithi017 ceph-mon[27571]: pgmap v1369: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 275 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 7895/13191 objects degraded (59.851%); 217/13191 objects misplaced (1.645%); 3.6 MiB/s, 58 objects/s recovering 2024-09-17T16:29:15.812 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:15 smithi017 ceph-mon[27571]: osdmap e618: 12 total, 12 up, 12 in 2024-09-17T16:29:15.813 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:15 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7895/13191 objects degraded (59.851%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:15.813 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:15 smithi017 ceph-mon[27571]: osdmap e619: 12 total, 12 up, 12 in 2024-09-17T16:29:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:15 smithi071 ceph-mon[30002]: pgmap v1369: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 275 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 7895/13191 objects degraded (59.851%); 217/13191 objects misplaced (1.645%); 3.6 MiB/s, 58 objects/s recovering 2024-09-17T16:29:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:15 smithi071 ceph-mon[30002]: osdmap e618: 12 total, 12 up, 12 in 2024-09-17T16:29:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:15 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7895/13191 objects degraded (59.851%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:15 smithi071 ceph-mon[30002]: osdmap e619: 12 total, 12 up, 12 in 2024-09-17T16:29:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:15 smithi173 ceph-mon[30530]: pgmap v1369: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 275 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 7895/13191 objects degraded (59.851%); 217/13191 objects misplaced (1.645%); 3.6 MiB/s, 58 objects/s recovering 2024-09-17T16:29:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:15 smithi173 ceph-mon[30530]: osdmap e618: 12 total, 12 up, 12 in 2024-09-17T16:29:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:15 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7895/13191 objects degraded (59.851%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:15.818 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:15 smithi173 ceph-mon[30530]: osdmap e619: 12 total, 12 up, 12 in 2024-09-17T16:29:15.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:15.900 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:15.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:16.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:16.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:16.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:16.113 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:16.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:16.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:16.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:16.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:16.255 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:16.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:29:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:16.295 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:16.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:16.396 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:16.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:16.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:16.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:16.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:16.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:16.573 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:29:16.611 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:29:16.612 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:29:16.612 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 620, 2024-09-17T16:29:16.612 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:29:16.612 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:29:16.612 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:29:15.590159+0000", 2024-09-17T16:29:16.612 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:29:16.612 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:26:33.541808+0000", 2024-09-17T16:29:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:29:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5799936, 2024-09-17T16:29:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:29:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:29:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:29:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:29:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:29:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:29:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:29:16.613 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 6, 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:29:16.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:29:16.615 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:29:16.616 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:29:16.616 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:29:16.616 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:29:16.616 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:29:16.616 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:29:16.616 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:29:16.616 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:29:16.616 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:29:16.616 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:29:16.616 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:29:16.617 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:29:16.617 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:29:16.617 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:29:16.617 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:29:16.617 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:29:16.617 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:29:16.617 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:29:16.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.617 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "446", 2024-09-17T16:29:16.617 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:29:16.618 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:29:16.618 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:29:16.618 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:29:16.618 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:29:16.618 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:29:16.618 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:29:16.618 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:29:16.618 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:29:16.618 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:29:16.619 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:29:16.619 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:29:16.619 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:29:16.619 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:29:16.619 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:29:16.619 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:29:16.619 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:29:16.619 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:29:16.619 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:29:16.620 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:29:16.621 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:29:16.621 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:29:16.621 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:29:16.621 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:29:16.621 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:29:16.621 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:29:16.621 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:29:16.621 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:29:16.621 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:29:16.621 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:29:16.622 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:29:16.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.622 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:29:16.622 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:29:16.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.622 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:29:16.622 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:29:16.622 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.623 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 6, 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_4", 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:25:01.713250+0000", 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:29:16.624 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 16, 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 16, 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 16, 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 16, 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 16, 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:29:16.625 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "531", 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:29:16.626 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:29:16.627 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:29:16.628 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:29:16.628 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:29:16.628 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:29:16.628 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:29:16.628 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:29:16.628 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:29:16.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.628 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:29:16.628 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:29:16.628 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:29:16.629 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:29:16.629 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:29:16.629 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:29:16.629 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:29:16.629 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:29:16.629 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:29:16.629 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:29:16.629 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:29:16.629 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 3.440000057220459, 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.75, 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1.0900000333786011, 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3.75, 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:29:16.630 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 601, 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.632 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:29:16.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:29:16.636 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:29:16.637 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.637 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.637 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.637 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.637 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:29:16.637 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:29:16.637 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.637 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 604, 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.638 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:29:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:29:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:29:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.641 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.642 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.643 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.644 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:29:16.644 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:29:16.644 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:29:16.644 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:29:16.644 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.644 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.644 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.644 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:29:16.645 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 527, 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.646 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.647 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:29:16.647 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:29:16.647 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.647 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.647 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.647 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.648 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.649 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.649 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.649 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.649 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:29:16.649 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:29:16.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.650 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.650 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:29:16.650 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:29:16.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.650 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.650 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.650 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.650 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.651 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:29:16.651 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:29:16.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.651 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.651 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:29:16.651 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:29:16.652 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.652 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.652 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:29:16.652 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:29:16.652 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:29:16.652 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:29:16.652 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.652 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.652 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.653 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 608, 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:29:16.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:29:16.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:29:16.656 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:29:16.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:29:16.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.658 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.658 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:29:16.658 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:29:16.658 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.658 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:29:16.659 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.660 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.660 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:29:16.660 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:29:16.660 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:29:16.660 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:29:16.660 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.660 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.660 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:29:16.661 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 549, 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:29:16.662 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.664 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.664 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:29:16.664 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:29:16.664 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.664 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.664 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.664 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.665 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.665 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:29:16.665 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:29:16.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.665 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.665 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:29:16.665 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:29:16.665 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.666 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.667 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.845672607421875, 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.668 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 602, 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.670 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:29:16.670 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:29:16.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.670 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.670 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.670 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.671 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:29:16.672 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.673 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.66790771484375, 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 568, 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.675 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.676 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.677 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.677 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.677 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.677 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:29:16.677 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:29:16.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.678 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.678 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:29:16.678 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:29:16.678 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.678 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.678 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.678 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.679 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.679 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:29:16.679 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:29:16.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.679 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.679 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:29:16.679 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:29:16.679 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.680 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.680 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.680 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.680 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.680 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:29:16.680 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:29:16.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.681 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.681 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:29:16.681 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:29:16.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.681 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.681 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:29:16.681 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:29:16.681 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.682 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.8061676025390625, 2024-09-17T16:29:16.683 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.73492431640625, 2024-09-17T16:29:16.683 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:29:16.683 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:29:16.683 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:29:16.683 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 527, 2024-09-17T16:29:16.683 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:29:16.683 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.683 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.683 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.684 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.684 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:29:16.684 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:29:16.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.684 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.684 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:29:16.684 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:29:16.684 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.685 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.686 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:29:16.686 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:29:16.686 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.686 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.686 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.686 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.686 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.687 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:29:16.687 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:29:16.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.687 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.687 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:29:16.687 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:29:16.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.688 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.688 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.688 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.688 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.689 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:29:16.689 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:29:16.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.689 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.689 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:29:16.689 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:29:16.689 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.689 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.691 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:29:16.691 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:29:16.691 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:29:16.691 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:29:16.691 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.691 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.691 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.692 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.692 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:29:16.692 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:29:16.692 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.692 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.692 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:29:16.692 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:29:16.693 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.693 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.693 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:29:16.693 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 527, 2024-09-17T16:29:16.693 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.693 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.693 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.693 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.694 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.694 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:29:16.694 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:29:16.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.694 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.694 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:29:16.694 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:29:16.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.694 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:29:16.695 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.696 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.698 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.698 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:29:16.698 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:29:16.698 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.699 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:29:16.700 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:29:16.700 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.700 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.700 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:29:16.700 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.2821197509765625, 2024-09-17T16:29:16.700 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.700 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.700 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:29:16.700 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 419, 2024-09-17T16:29:16.700 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:29:16.701 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:29:16.702 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:29:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:29:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:29:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.703 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.704 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.704 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.704 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.704 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:29:16.706 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.707 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:29:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:29:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:29:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 527, 2024-09-17T16:29:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.709 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.710 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.711 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:29:16.713 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.588836669921875, 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 291, 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.714 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:29:16.715 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.716 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.717 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:29:16.719 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:01.534897+0000", 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.720 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:27:57.094446+0000", 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:28:39.044701+0000", 2024-09-17T16:29:16.721 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:14.245748+0000", 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:29:16.722 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:28:36.403304+0000", 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.723 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:14.995509+0000", 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:10.274208+0000", 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:28:16.257648+0000", 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.725 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.726 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.726 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.726 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:27:53.408119+0000", 2024-09-17T16:29:16.726 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.726 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:29:16.726 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.726 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:28:12.484927+0000", 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.728 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:28:01.617561+0000", 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:28:00.894489+0000", 2024-09-17T16:29:16.729 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [ 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "1.0", 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: 4, 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: 6, 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: 8 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:29:16.730 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [ 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "6.7", 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: 3, 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: 4 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "6.a", 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: 6, 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: 0 2024-09-17T16:29:16.731 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:29:16.732 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:29:16.733 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:29:16.734 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:29:16.734 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:29:16.734 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:29:16.734 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:29:16.734 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:29:16.734 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:29:16.734 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:29:16.734 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:29:16.734 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:29:16.734 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:16.735 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:29:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:16 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:16 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:16 smithi017 ceph-mon[27571]: osdmap e620: 12 total, 12 up, 12 in 2024-09-17T16:29:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:16 smithi017 ceph-mon[27571]: pgmap v1373: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 205 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 719 op/s; 7798/9819 objects degraded (79.417%); 87/9819 objects misplaced (0.886%); 4.1 MiB/s, 88 objects/s recovering 2024-09-17T16:29:16.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:16.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:16.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:16.877 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:16 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:16.878 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:16 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:16.878 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:16 smithi173 ceph-mon[30530]: osdmap e620: 12 total, 12 up, 12 in 2024-09-17T16:29:16.878 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:16 smithi173 ceph-mon[30530]: pgmap v1373: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 205 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 719 op/s; 7798/9819 objects degraded (79.417%); 87/9819 objects misplaced (0.886%); 4.1 MiB/s, 88 objects/s recovering 2024-09-17T16:29:16.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:16.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:16.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:17.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:17.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:16 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:16 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:16 smithi071 ceph-mon[30002]: osdmap e620: 12 total, 12 up, 12 in 2024-09-17T16:29:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:16 smithi071 ceph-mon[30002]: pgmap v1373: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 205 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 719 op/s; 7798/9819 objects degraded (79.417%); 87/9819 objects misplaced (0.886%); 4.1 MiB/s, 88 objects/s recovering 2024-09-17T16:29:17.109 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:17.130 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:29:17.130 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:29:17.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:29:17.130 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:29:17.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.130 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:204335 4.9 e408/383 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head v 408'12572, mlcod=408'12569)", 2024-09-17T16:29:17.130 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.607078+0000", 2024-09-17T16:29:17.130 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 473.51999710600001, 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77592090000000002, 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607078+0000", 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607078+0000", 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607079+0000", 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8199999999999998e-07 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607086+0000", 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9600000000000003e-06 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.378990+0000", 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77190383900000004 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.379000+0000", 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.379065+0000", 2024-09-17T16:29:17.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5287e-05 2024-09-17T16:29:17.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.379088+0000", 2024-09-17T16:29:17.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3334999999999999e-05 2024-09-17T16:29:17.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.382963+0000", 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038744439999999999 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.382999+0000", 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6715999999999999e-05 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:2279 5.f e424/419 5:f5772ab5:::benchmark_data_smithi173_90735_object2278:head v 424'148, mlcod=424'146)", 2024-09-17T16:29:17.135 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:51.502595+0000", 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 445.624480167, 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25253113999999999, 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502595+0000", 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502595+0000", 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502597+0000", 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.311e-06 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502640+0000", 2024-09-17T16:29:17.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3847000000000002e-05 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754667+0000", 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25202682199999998 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754673+0000", 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1489999999999999e-06 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754725+0000", 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2009999999999998e-05 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754741+0000", 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5798e-05 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.755084+0000", 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034313300000000002 2024-09-17T16:29:17.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.755126+0000", 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2070000000000002e-05 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:15935 5.f e429/419 5:f3ff7eb6:::benchmark_data_smithi173_90735_object15934:head v 429'990, mlcod=429'987)", 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:02.437369+0000", 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 434.68970666000001, 2024-09-17T16:29:17.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79522149499999994, 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437369+0000", 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437369+0000", 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437370+0000", 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9400000000000005e-07 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437426+0000", 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6798000000000001e-05 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231438+0000", 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79401129999999998 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231449+0000", 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1688000000000001e-05 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231596+0000", 2024-09-17T16:29:17.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000146546 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231620+0000", 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4315999999999999e-05 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.232554+0000", 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093371600000000002 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.232590+0000", 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6436999999999997e-05 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:54616 5.17 e450/444 5:eee108da:::benchmark_data_smithi173_90735_object54615:head v 450'3019, mlcod=0'0)", 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:39.740381+0000", 2024-09-17T16:29:17.145 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 397.38669474300002, 2024-09-17T16:29:17.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58705162700000002, 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740381+0000", 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740381+0000", 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740382+0000", 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.249e-06 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740443+0000", 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0998000000000001e-05 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325597+0000", 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58515414300000002 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325606+0000", 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5760000000000006e-06 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325653+0000", 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.774e-05 2024-09-17T16:29:17.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325673+0000", 2024-09-17T16:29:17.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9253e-05 2024-09-17T16:29:17.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.327389+0000", 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001716104 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.327432+0000", 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3563999999999998e-05 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.151 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:63127 5.f e461/455 5:f7e3e25a:::benchmark_data_smithi173_90735_object63126:head v 461'3883, mlcod=425'480)", 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:55.384543+0000", 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 381.74253208200003, 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23654767900000001, 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384543+0000", 2024-09-17T16:29:17.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384543+0000", 2024-09-17T16:29:17.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384545+0000", 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.339e-06 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384593+0000", 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8733000000000002e-05 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619021+0000", 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23442753799999999 2024-09-17T16:29:17.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619029+0000", 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7270000000000007e-06 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619090+0000", 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1680999999999994e-05 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619117+0000", 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7013e-05 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.621043+0000", 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001925679 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.621091+0000", 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7969000000000001e-05 2024-09-17T16:29:17.156 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:77257 5.17 e472/470 5:ee127e34:::benchmark_data_smithi173_90735_object77256:head v 472'3755, mlcod=472'3752)", 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:13.811785+0000", 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 363.31529060499997, 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.255431142, 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811785+0000", 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811785+0000", 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811786+0000", 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8700000000000005e-07 2024-09-17T16:29:17.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811954+0000", 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016799800000000001 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065415+0000", 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25346157200000002 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065422+0000", 2024-09-17T16:29:17.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3969999999999999e-06 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065477+0000", 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5699000000000003e-05 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065491+0000", 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4069e-05 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.067186+0000", 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001694922 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.067216+0000", 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9697999999999999e-05 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.161 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:82762 5.d e475/455 5:bc524506:::benchmark_data_smithi173_90735_object82761:head v 475'5183, mlcod=439'2263)", 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.917268+0000", 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 357.20980787799999, 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57018889800000006, 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917268+0000", 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917268+0000", 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917269+0000", 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1480000000000001e-06 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917322+0000", 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3705000000000002e-05 2024-09-17T16:29:17.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485165+0000", 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.56784279500000001 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485171+0000", 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0639999999999999e-06 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485221+0000", 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9313e-05 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485242+0000", 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1049000000000001e-05 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.487416+0000", 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021741849999999999 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.487457+0000", 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0639000000000002e-05 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:114282 5.d e489/482 5:bfd83ddc:::benchmark_data_smithi173_90735_object31511:head v 489'7194, mlcod=489'7192)", 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:53.959997+0000", 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 323.16707826700002, 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33615526200000001, 2024-09-17T16:29:17.166 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.959997+0000", 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.959997+0000", 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.959999+0000", 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.61e-06 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.960006+0000", 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3170000000000004e-06 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292652+0000", 2024-09-17T16:29:17.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33264596400000002 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292658+0000", 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9859999999999999e-06 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292772+0000", 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011348799999999999 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292793+0000", 2024-09-17T16:29:17.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1630000000000001e-05 2024-09-17T16:29:17.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.296107+0000", 2024-09-17T16:29:17.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003314141 2024-09-17T16:29:17.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.296152+0000", 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5126000000000001e-05 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:158720 5.f e512/481 5:f5ceb914:::benchmark_data_smithi173_90735_object75949:head v 512'9748, mlcod=512'9745)", 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 278.26908065399999, 2024-09-17T16:29:17.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.233160072, 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857996+0000", 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7399999999999991e-07 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.858002+0000", 2024-09-17T16:29:17.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5030000000000002e-06 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087206+0000", 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22920367999999999 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087213+0000", 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0129999999999996e-06 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087267+0000", 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3978999999999999e-05 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087289+0000", 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1597999999999999e-05 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.091102+0000", 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038130719999999998 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.091155+0000", 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3253e-05 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:160454 5.17 e514/470 5:eb54973c:::benchmark_data_smithi173_90735_object77683:head v 514'6398, mlcod=514'6395)", 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 276.39644094499999, 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18539098100000001, 2024-09-17T16:29:17.177 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2900000000000002e-07 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730643+0000", 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3300000000000001e-06 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913880+0000", 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.183237224 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913887+0000", 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2400000000000001e-06 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913943+0000", 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5859999999999997e-05 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913958+0000", 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4854999999999999e-05 2024-09-17T16:29:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.915994+0000", 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00203572 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.916026+0000", 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1822999999999997e-05 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:12912 6.d e535/527 6:ba9121f7:::benchmark_data_smithi173_102017_object12911:head v 535'804, mlcod=535'801)", 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 239.42324545599999, 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27481599400000001, 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703831+0000", 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3340000000000001e-06 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703859+0000", 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7898e-05 2024-09-17T16:29:17.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977854+0000", 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27399516899999998 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977863+0000", 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6309999999999992e-06 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977932+0000", 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.923e-05 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977956+0000", 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3606000000000001e-05 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.978613+0000", 2024-09-17T16:29:17.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065716699999999995 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.978646+0000", 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2959000000000001e-05 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:13282 6.d e535/527 6:b884e4f4:::benchmark_data_smithi173_102017_object13281:head v 535'823, mlcod=535'820)", 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:18.148010+0000", 2024-09-17T16:29:17.186 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 238.979065031, 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.174234948, 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148010+0000", 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148010+0000", 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148011+0000", 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9699999999999994e-07 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148035+0000", 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3946000000000001e-05 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321530+0000", 2024-09-17T16:29:17.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17349503999999999 2024-09-17T16:29:17.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321535+0000", 2024-09-17T16:29:17.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9930000000000003e-06 2024-09-17T16:29:17.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321583+0000", 2024-09-17T16:29:17.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7472000000000002e-05 2024-09-17T16:29:17.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321598+0000", 2024-09-17T16:29:17.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5305e-05 2024-09-17T16:29:17.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.322200+0000", 2024-09-17T16:29:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00060203700000000004 2024-09-17T16:29:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.322245+0000", 2024-09-17T16:29:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5157999999999997e-05 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:112033 6.8 e580/527 6:13e325ea:::benchmark_data_smithi173_102017_object18885:head v 580'6954, mlcod=580'6951)", 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:08.095238+0000", 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 129.03183746299999, 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.111567449, 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.192 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.095238+0000", 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.095238+0000", 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.095239+0000", 2024-09-17T16:29:17.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5499999999999997e-07 2024-09-17T16:29:17.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.095245+0000", 2024-09-17T16:29:17.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5060000000000001e-06 2024-09-17T16:29:17.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:17.195 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:17.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:17.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.204212+0000", 2024-09-17T16:29:17.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.108966758 2024-09-17T16:29:17.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.204221+0000", 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6230000000000003e-06 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.204281+0000", 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0285999999999998e-05 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.204298+0000", 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6456e-05 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.206772+0000", 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024745560000000002 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.206805+0000", 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3408999999999998e-05 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:112146 6.8 e580/527 6:15337d2c:::benchmark_data_smithi173_102017_object18998:head v 580'6960, mlcod=580'6957)", 2024-09-17T16:29:17.198 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:08.246278+0000", 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 128.880797575, 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.095665551000000001, 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.246278+0000", 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.246278+0000", 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.246280+0000", 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.604e-06 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.246289+0000", 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0299999999999999e-06 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.339212+0000", 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.092923729999999996 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.339219+0000", 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.798e-06 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.339265+0000", 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6349e-05 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.339282+0000", 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6283e-05 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.341902+0000", 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026199970000000002 2024-09-17T16:29:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.341943+0000", 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.176e-05 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:112335 6.8 e580/527 6:182a3819:::benchmark_data_smithi173_102017_object19187:head v 580'6971, mlcod=580'6968)", 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:08.529169+0000", 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 128.59790661100001, 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.141299961, 2024-09-17T16:29:17.203 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.529169+0000", 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.529169+0000", 2024-09-17T16:29:17.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.529170+0000", 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.249e-06 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.529179+0000", 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3990000000000004e-06 2024-09-17T16:29:17.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.666665+0000", 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13748685099999999 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.666671+0000", 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7930000000000003e-06 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.666723+0000", 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1683e-05 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.666738+0000", 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4997999999999999e-05 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.670438+0000", 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036998560000000001 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:08.670469+0000", 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1131999999999999e-05 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:113603 6.8 e582/527 6:1d7478f4:::benchmark_data_smithi173_102017_object20455:head v 582'7045, mlcod=582'7042)", 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 126.51734924500001, 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.183501209, 2024-09-17T16:29:17.208 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609728+0000", 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.176e-06 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609742+0000", 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3703000000000001e-05 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790389+0000", 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18064696399999999 2024-09-17T16:29:17.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790402+0000", 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3239e-05 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790530+0000", 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012778799999999999 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790567+0000", 2024-09-17T16:29:17.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6999000000000003e-05 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.793200+0000", 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026325070000000001 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.793227+0000", 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7832999999999999e-05 2024-09-17T16:29:17.212 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:133513 6.8 e590/527 6:16578546:::benchmark_data_smithi173_102017_object40365:head v 590'8302, mlcod=590'8299)", 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:57.014805+0000", 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 80.112270076000001, 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.115763746, 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.213 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:57.014805+0000", 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:57.014805+0000", 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:57.014806+0000", 2024-09-17T16:29:17.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.004e-06 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:57.014813+0000", 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6409999999999996e-06 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:57.127515+0000", 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.112701632 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:57.127523+0000", 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1019999999999997e-06 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:57.127580+0000", 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6755999999999997e-05 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:57.127597+0000", 2024-09-17T16:29:17.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7450999999999999e-05 2024-09-17T16:29:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:57.130539+0000", 2024-09-17T16:29:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029423679999999999 2024-09-17T16:29:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:57.130569+0000", 2024-09-17T16:29:17.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9791999999999999e-05 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:149167 6.8 e601/527 6:10c778a4:::benchmark_data_smithi173_102017_object56019:head v 601'9251, mlcod=601'9247)", 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:28:20.018530+0000", 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 57.108545491999998, 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.095189022999999998, 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:28:20.018530+0000", 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:28:20.018530+0000", 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:28:20.018531+0000", 2024-09-17T16:29:17.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3170000000000001e-06 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:28:20.018539+0000", 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0590000000000003e-06 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:28:20.110242+0000", 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.091702354 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:28:20.110251+0000", 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0299999999999999e-06 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:28:20.110306+0000", 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4873999999999997e-05 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:28:20.110322+0000", 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6721000000000001e-05 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:28:20.113691+0000", 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033684259999999999 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:28:20.113719+0000", 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8242000000000002e-05 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:185287 6.8 e620/527 6:127bdc49:::benchmark_data_smithi173_102017_object92139:head v 620'11427, mlcod=620'11423)", 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:16.099702+0000", 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.0273731690000001, 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.098379468999999997, 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.099702+0000", 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.099702+0000", 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.099703+0000", 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5000000000000001e-07 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.099708+0000", 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2900000000000002e-06 2024-09-17T16:29:17.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.193777+0000", 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.094068323999999995 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.193784+0000", 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3030000000000002e-06 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.193839+0000", 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5050999999999999e-05 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.193861+0000", 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2297999999999999e-05 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.198047+0000", 2024-09-17T16:29:17.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041859280000000002 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.198082+0000", 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4425000000000003e-05 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:185512 6.8 e620/527 6:10641d04:::benchmark_data_smithi173_102017_object92364:head v 620'11444, mlcod=620'11440)", 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:16.427597+0000", 2024-09-17T16:29:17.227 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.69947820999999999, 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.093896919999999995, 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.427597+0000", 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:29:17.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.427597+0000", 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.427599+0000", 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2440000000000001e-06 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.427606+0000", 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6000000000000001e-06 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.516739+0000", 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.089132873000000001 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.516747+0000", 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9410000000000007e-06 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.516808+0000", 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1037000000000005e-05 2024-09-17T16:29:17.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.516832+0000", 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4034999999999999e-05 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.521444+0000", 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046118640000000002 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:29:17.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:29:17.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:16.521494+0000", 2024-09-17T16:29:17.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0325999999999999e-05 2024-09-17T16:29:17.232 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.232 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.232 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.232 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:29:17.232 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:29:17.232 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:29:17.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:17.319 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:17.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:17.421 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:17.448 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:17.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:17.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:17.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:17.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:17.618 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:29:17.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:17.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:17.695 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:17.796 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:17.818 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:17.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:17.862 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:17 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:17.862 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:17 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:17.862 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3287144592' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:29:17.862 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:17 smithi173 ceph-mon[30530]: osdmap e621: 12 total, 12 up, 12 in 2024-09-17T16:29:17.862 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3422700686' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:29:17.862 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:17 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:29:17.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:17.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:17.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:17 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:17 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3287144592' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:29:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:17 smithi017 ceph-mon[27571]: osdmap e621: 12 total, 12 up, 12 in 2024-09-17T16:29:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3422700686' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:29:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:17 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:29:18.024 INFO:tasks.thrashosds.thrasher:j is {'epoch': 620, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:29:15.590159+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:26:33.541808+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '446', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-17T16:25:01.713250+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '531', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.440000057220459, 'score_stable': 2.75, 'optimal_score': 1.090000033378601, 'raw_score_acting': 3.75, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 601, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 604, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 608, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 549, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 0.845672607421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 602, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 0.66790771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 568, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 0.8061676025390625, 'primary_affinity': 0.73492431640625, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 527, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0.2821197509765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 527, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 0.588836669921875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:01.534897+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:27:57.094446+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:28:39.044701+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:14.245748+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:28:36.403304+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:14.995509+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:10.274208+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:28:16.257648+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:27:53.408119+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:28:12.484927+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:28:01.617561+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:28:00.894489+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [4, 6, 8]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '6.7', 'osds': [3, 4]}, {'pgid': '6.a', 'osds': [6, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:29:18.024 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-17T16:29:18.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:17 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:17 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3287144592' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:29:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:17 smithi071 ceph-mon[30002]: osdmap e621: 12 total, 12 up, 12 in 2024-09-17T16:29:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3422700686' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:29:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:17 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:29:18.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:18.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:18.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:18.160 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 93145 objects 2024-09-17T16:29:18.160 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :157.599 2024-09-17T16:29:18.166 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:18.188 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:18.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:18.290 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:18.314 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:18.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:18.316 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:29:18.339 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:18.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:18.439 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:18.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:18.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:18.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:18.563 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:18.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:18.639 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:29:18.665 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:18.692 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:18.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:18.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:18.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:18 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:18.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:18 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:29:18.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:18 smithi071 ceph-mon[30002]: osdmap e622: 12 total, 12 up, 12 in 2024-09-17T16:29:18.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:18 smithi071 ceph-mon[30002]: pgmap v1376: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 80 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 7689/3831 objects degraded (200.705%); 3.5 MiB/s, 121 objects/s recovering 2024-09-17T16:29:18.794 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:18.821 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:18.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:18 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:18 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:29:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:18 smithi173 ceph-mon[30530]: osdmap e622: 12 total, 12 up, 12 in 2024-09-17T16:29:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:18 smithi173 ceph-mon[30530]: pgmap v1376: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 80 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 7689/3831 objects degraded (200.705%); 3.5 MiB/s, 121 objects/s recovering 2024-09-17T16:29:18.905 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:18 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:18.905 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:18 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:29:18.905 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:18 smithi017 ceph-mon[27571]: osdmap e622: 12 total, 12 up, 12 in 2024-09-17T16:29:18.905 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:18 smithi017 ceph-mon[27571]: pgmap v1376: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 80 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 835 op/s; 7689/3831 objects degraded (200.705%); 3.5 MiB/s, 121 objects/s recovering 2024-09-17T16:29:18.922 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:18.948 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:18.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:19.050 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:19.075 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:19.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:19.177 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:19.178 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:29:19.178 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:29:19.200 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:19.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:19.301 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:19.380 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:19.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:19.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:19.504 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:19.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:19.549 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:29:19.550 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:29:19.550 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:29:19.550 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:29:19.550 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:29:19.550 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:29:19.550 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:29:19.550 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:29:19.551 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:29:19.551 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:29:19.551 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:29:19.551 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:29:19.551 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:29:19.551 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:29:19.551 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:29:19.551 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:29:19.551 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:29:19.552 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:29:19.552 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:29:19.552 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:29:19.552 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:29:19.552 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:29:19.552 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:29:19.552 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:29:19.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:19.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:19.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:19.633 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-17T16:29:19.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-17T16:29:19.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:19.755 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:19.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:19.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3700902568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:19.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:19 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 7689/3831 objects degraded (200.705%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:19.837 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:19 smithi017 ceph-mon[27571]: osdmap e623: 12 total, 12 up, 12 in 2024-09-17T16:29:19.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:19.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:19.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3700902568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:19 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 7689/3831 objects degraded (200.705%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:19 smithi173 ceph-mon[30530]: osdmap e623: 12 total, 12 up, 12 in 2024-09-17T16:29:19.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:19.988 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3700902568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:19.988 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:19 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 7689/3831 objects degraded (200.705%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:19.988 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:19 smithi071 ceph-mon[30002]: osdmap e623: 12 total, 12 up, 12 in 2024-09-17T16:29:20.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:20.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:20.020 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:29:20.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:20.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:20.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:20.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:20.261 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:20.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:20.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:20.314 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:29:20.363 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:20.386 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:20.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:20.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:20.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:20.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:20.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:20.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:20.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:20.734 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:20 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:20.734 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:20 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:20.734 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:20 smithi071 ceph-mon[30002]: pgmap v1378: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 31 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 7689/1470 objects degraded (523.061%); 2.7 MiB/s, 96 objects/s recovering 2024-09-17T16:29:20.734 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:20 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:20.734 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:20 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:20.734 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:20 smithi071 ceph-mon[30002]: osdmap e624: 12 total, 12 up, 12 in 2024-09-17T16:29:20.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:20 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:20 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:20 smithi017 ceph-mon[27571]: pgmap v1378: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 31 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 7689/1470 objects degraded (523.061%); 2.7 MiB/s, 96 objects/s recovering 2024-09-17T16:29:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:20 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:20 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:20 smithi017 ceph-mon[27571]: osdmap e624: 12 total, 12 up, 12 in 2024-09-17T16:29:20.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:20.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:20.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:20.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:20.867 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:29:20.867 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:29:20.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:20 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:20 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:20 smithi173 ceph-mon[30530]: pgmap v1378: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 31 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 7689/1470 objects degraded (523.061%); 2.7 MiB/s, 96 objects/s recovering 2024-09-17T16:29:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:20 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:20 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:20 smithi173 ceph-mon[30530]: osdmap e624: 12 total, 12 up, 12 in 2024-09-17T16:29:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:20.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:20.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:20.997 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:21.020 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:21.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:21.123 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:21.146 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:21.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:21.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:21.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:21.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:21.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:21.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:21.376 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:21.399 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:21.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:21.408 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:29:21.501 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:21.523 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:21.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:21.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:21.645 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:21.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:21.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:21 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:21 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:21.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:21.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:21.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:21.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:21.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:21 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:21 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:21.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:21.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:21.923 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:21 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:21.923 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:21 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:22.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:22.023 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:22.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:22.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:22.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:22.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:22.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:29:22.250 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:22.299 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:22.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:22.400 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:22.422 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:22.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:22.483 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:29:22.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:29:22.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:22.576 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:22.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:22.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:22.678 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:22.705 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:22.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:22.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:22.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:22.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:22.895 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:22 smithi071 ceph-mon[30002]: pgmap v1380: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.2 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 6717/36 objects degraded (18658.333%); 762 KiB/s, 217 objects/s recovering 2024-09-17T16:29:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:22 smithi173 ceph-mon[30530]: pgmap v1380: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.2 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 6717/36 objects degraded (18658.333%); 762 KiB/s, 217 objects/s recovering 2024-09-17T16:29:22.935 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:22.959 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:22.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:22 smithi017 ceph-mon[27571]: pgmap v1380: 17 pgs: 2 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.2 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 6717/36 objects degraded (18658.333%); 762 KiB/s, 217 objects/s recovering 2024-09-17T16:29:23.010 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:29:23.010 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:29:23.025 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:29:23.025 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:29:23.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:29:23.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:23.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:23.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:23.189 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:23.211 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:23.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:23.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:23.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:23.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:23.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:23.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:23.440 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:23.463 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:23.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:23.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:23.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:23.586 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:23.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:23.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:23.653 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:29:23.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:23.709 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:23.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:23.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:23.811 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:23.831 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:23.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:23.921 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4096538506' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:29:23.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:23.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:23.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:23.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:24.064 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:24.088 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:24.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:24.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:24.218 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:24.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:24.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4096538506' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:29:24.319 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:24.341 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:24.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:24.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4096538506' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:29:24.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:24.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:24.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:24.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:24.589 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:24.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:24.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:24.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:24.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:24.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:24.836 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_4' removed 2024-09-17T16:29:24.838 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:24.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:24.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:24.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:24.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:24.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:25.066 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:25.088 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:25.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:25.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:25.212 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:25.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4096538506' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:29:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:24 smithi017 ceph-mon[27571]: osdmap e625: 12 total, 12 up, 12 in 2024-09-17T16:29:25.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:24 smithi017 ceph-mon[27571]: pgmap v1382: 1 pgs: 1 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:25.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:24 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:25.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:24 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:25.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:24 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 6717/36 objects degraded (18658.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4096538506' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:29:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:24 smithi071 ceph-mon[30002]: osdmap e625: 12 total, 12 up, 12 in 2024-09-17T16:29:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:24 smithi071 ceph-mon[30002]: pgmap v1382: 1 pgs: 1 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:24 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:24 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:25.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:24 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 6717/36 objects degraded (18658.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:25.315 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:25.336 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:25.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4096538506' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:29:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:24 smithi173 ceph-mon[30530]: osdmap e625: 12 total, 12 up, 12 in 2024-09-17T16:29:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:24 smithi173 ceph-mon[30530]: pgmap v1382: 1 pgs: 1 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:24 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:24 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:24 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 6717/36 objects degraded (18658.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:29:25.438 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:25.466 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:25.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:25.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:25.589 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:25.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:25.691 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:25.718 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:25.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:25.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:25.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:25.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:25.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:25.969 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:25.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:26.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:26.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:26.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:26.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:26.223 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:26.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:26.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:25 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6717/36 objects degraded (18658.333%), 2 pgs degraded, 2 pgs undersized) 2024-09-17T16:29:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:25 smithi017 ceph-mon[27571]: osdmap e626: 12 total, 12 up, 12 in 2024-09-17T16:29:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:25 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6717/36 objects degraded (18658.333%), 2 pgs degraded, 2 pgs undersized) 2024-09-17T16:29:26.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:25 smithi071 ceph-mon[30002]: osdmap e626: 12 total, 12 up, 12 in 2024-09-17T16:29:26.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:26.348 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:26.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:26.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:25 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6717/36 objects degraded (18658.333%), 2 pgs degraded, 2 pgs undersized) 2024-09-17T16:29:26.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:25 smithi173 ceph-mon[30530]: osdmap e626: 12 total, 12 up, 12 in 2024-09-17T16:29:26.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:26.472 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:26.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:26.555 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:26.565 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:26.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:26.580 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:26.596 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:26.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:26.698 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:26.726 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:26.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:26.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:26.852 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:26.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:26.954 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:26.977 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:26.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:27.079 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:27.103 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:27.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:27.205 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:27.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:27.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:27.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:26 smithi017 ceph-mon[27571]: pgmap v1384: 1 pgs: 1 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:27.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:26 smithi071 ceph-mon[30002]: pgmap v1384: 1 pgs: 1 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:27.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:27.361 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:27.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:27.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:26 smithi173 ceph-mon[30530]: pgmap v1384: 1 pgs: 1 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:27.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:27.491 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:27.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:27.593 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:27.617 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:27.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:27.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:27.744 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:27.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:27.789 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:29:27.789 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:29:27.811 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:29:27.811 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:29:27.831 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:29:27.831 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:29:27.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:27.852 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:29:27.853 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:29:27.872 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:27.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:27.874 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:29:27.874 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:29:27.896 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:29:27.896 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:29:27.918 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:29:27.918 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:29:27.942 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:29:27.942 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:29:27.965 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:29:27.965 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:29:27.973 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:27.987 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:29:27.987 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:29:27.995 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:27.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:27.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:28.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:29:28.009 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:29:28.030 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:29:28.031 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:29:28.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:28.123 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:28.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:28.225 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:28.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1448467811' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:28.250 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:28.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:28.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1448467811' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:28.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:28.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:28.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:28.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:28.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1448467811' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:28.481 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:28.504 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:28.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:28.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:28.633 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:28.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:28.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:28.761 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:28.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:28.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:28.888 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:28.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:28.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:29.014 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:29.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:29.069 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:29:29.070 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:29:29.070 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:29:29.070 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:29:29.070 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:28 smithi017 ceph-mon[27571]: pgmap v1385: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:29.117 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:29.140 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:29.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:29.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:29.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:29.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:29.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:28 smithi071 ceph-mon[30002]: pgmap v1385: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:29.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:29.395 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:29.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:29.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:28 smithi173 ceph-mon[30530]: pgmap v1385: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:29.497 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:29.520 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:29.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:29.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:29.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:29.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:29.754 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:29.777 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:29.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:29.842 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:29:29.879 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:29.905 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:29.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:30.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:30.032 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:30.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:30.134 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:30.159 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:30.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:30.260 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:30.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:30.284 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:30.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:30.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:30.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:30.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:30.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:30.513 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:30.536 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:30.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:30.638 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:30.659 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:30.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:30.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:30.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:30 smithi017 ceph-mon[27571]: pgmap v1386: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:30.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:30.783 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:30.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:30.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:29:30.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:30.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:30 smithi173 ceph-mon[30530]: pgmap v1386: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:30.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:30.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:31.012 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:31.018 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T16:29:31.019 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T16:29:31.019 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-17T16:29:31.019 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-09-17T16:29:31.038 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:31.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:31.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:30 smithi071 ceph-mon[30002]: pgmap v1386: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:31.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:31.148 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:29:31.148 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:29:31.163 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:31.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:31.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:31.291 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:31.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:31.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:31.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:31.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:31.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:31.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:31.544 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:31.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:31.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:31.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:31.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:31 smithi017 ceph-mon[27571]: from='client.17523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:29:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/250409414' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:31.774 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:31.797 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:31.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:31.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:31.831 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1386,"stamp":"2024-09-17T16:29:29.985344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":5,"kb":1124941824,"kb_used":5522852,"kb_used_data":323216,"kb_used_omap":58,"kb_used_meta":5197509,"kb_avail":1119418972,"statfs":{"total":1151940427776,"available":1146285027328,"internally_reserved":0,"allocated":330973184,"data_stored":326146244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":5322249379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-655360,"num_objects":-10,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-6717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-186282,"num_write_kb":-5961281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-26966,"num_bytes_recovered":-1634205696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001799"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":208,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-09-17T16:29:24.926934+0000","last_change":"2024-09-17T16:25:09.585157+0000","last_active":"2024-09-17T16:29:24.926934+0000","last_peered":"2024-09-17T16:29:24.926934+0000","last_clean":"2024-09-17T16:29:24.926934+0000","last_became_active":"2024-09-17T16:25:09.545303+0000","last_became_peered":"2024-09-17T16:25:09.545303+0000","last_unstale":"2024-09-17T16:29:24.926934+0000","last_undegraded":"2024-09-17T16:29:24.926934+0000","last_fullsized":"2024-09-17T16:29:24.926934+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:22:06.807231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:22:06.807231+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:24.121982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285808,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":383640,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93361512,"statfs":{"total":95995035648,"available":95602188288,"internally_reserved":0,"allocated":1060864,"data_stored":664570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":391576004},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547444,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":393368,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93351784,"statfs":{"total":95995035648,"available":95592226816,"internally_reserved":0,"allocated":1060864,"data_stored":664590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":401537472},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":541912,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93203240,"statfs":{"total":95995035648,"available":95440117760,"internally_reserved":0,"allocated":1060864,"data_stored":664590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":553646528},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005311,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":371680,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93373472,"statfs":{"total":95995035648,"available":95614435328,"internally_reserved":0,"allocated":1593344,"data_stored":1193518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963787,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":769948,"kb_used_data":293900,"kb_used_omap":41,"kb_used_meta":475990,"kb_avail":92975204,"statfs":{"total":95995035648,"available":95206608896,"internally_reserved":0,"allocated":300953600,"data_stored":300557326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":487414067},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438304,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93306848,"statfs":{"total":95995035648,"available":95546212352,"internally_reserved":0,"allocated":1593344,"data_stored":1193518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":447019462},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724811,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":501640,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93243512,"statfs":{"total":95995035648,"available":95481356288,"internally_reserved":0,"allocated":1232896,"data_stored":775646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":512424390},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986452,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":411936,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93333216,"statfs":{"total":95995035648,"available":95573213184,"internally_reserved":0,"allocated":1593344,"data_stored":1193518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":501656,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93243496,"statfs":{"total":95995035648,"available":95481339904,"internally_reserved":0,"allocated":1060864,"data_stored":664590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":512424391},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411612,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":347544,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93397608,"statfs":{"total":95995035648,"available":95639150592,"internally_reserved":0,"allocated":1060864,"data_stored":664590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575138,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":335060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410092,"statfs":{"total":95995035648,"available":95651934208,"internally_reserved":0,"allocated":1060864,"data_stored":664590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738662,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":526164,"kb_used_data":17228,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93218988,"statfs":{"total":95995035648,"available":95456243712,"internally_reserved":0,"allocated":17641472,"data_stored":17245198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:29:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:31 smithi173 ceph-mon[30530]: from='client.17523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:29:31.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/250409414' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:31.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:31.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:31.927 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:31.969 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:29:32.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:32.055 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:32.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:32.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:31 smithi071 ceph-mon[30002]: from='client.17523 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:29:32.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/250409414' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:32.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:32.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:32.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:32.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:32.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:32.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:32.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:32.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:32.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:32.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:32.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:32.537 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:32.563 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:32.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:32.664 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:32.669 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:32.671 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:32.689 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:32.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:32.710 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:32.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:32.813 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:32.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:32.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:32 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/250409414' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:29:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:32 smithi173 ceph-mon[30530]: osdmap e627: 12 total, 12 up, 12 in 2024-09-17T16:29:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:32 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:32 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:32 smithi173 ceph-mon[30530]: pgmap v1388: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:32.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:32.937 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:32 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:32.937 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/250409414' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:29:32.937 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:32 smithi017 ceph-mon[27571]: osdmap e627: 12 total, 12 up, 12 in 2024-09-17T16:29:32.937 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:32 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:32.937 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:32 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:32.937 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:32 smithi017 ceph-mon[27571]: pgmap v1388: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:32.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:32.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:33.043 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:32 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/250409414' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:29:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:32 smithi071 ceph-mon[30002]: osdmap e627: 12 total, 12 up, 12 in 2024-09-17T16:29:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:32 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:32 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:32 smithi071 ceph-mon[30002]: pgmap v1388: 1 pgs: 1 active+clean; 577 KiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:33.067 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:33.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:33.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:33.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:33.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:33.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:33.320 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:33.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:33.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:33.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:33.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:33.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:33.575 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:33.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:33.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:33.702 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:33.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:33.804 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:33.826 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:33 smithi017 ceph-mon[27571]: osdmap e628: 12 total, 12 up, 12 in 2024-09-17T16:29:33.829 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:33.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:33.853 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:29:33.853 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:29:33.875 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:29:33.875 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:29:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:33 smithi173 ceph-mon[30530]: osdmap e628: 12 total, 12 up, 12 in 2024-09-17T16:29:33.895 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:29:33.896 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:29:33.916 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:29:33.916 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:29:33.931 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:33.938 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:29:33.938 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:29:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:33.954 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:33.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:33.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:29:33.961 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:29:33.983 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:29:33.983 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:29:34.005 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:29:34.005 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:29:34.027 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:29:34.027 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:29:34.049 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:29:34.050 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:29:34.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:34.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:33 smithi071 ceph-mon[30002]: osdmap e628: 12 total, 12 up, 12 in 2024-09-17T16:29:34.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:29:34.071 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:29:34.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:34.082 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:29:34.083 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:29:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:34.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:34.093 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:29:34.093 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:29:34.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:34.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:34.209 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:34.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:34.334 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:34.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:34.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:34.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:34.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:34.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:34.588 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:34.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:34.609 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_5' created 2024-09-17T16:29:34.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:29:34.690 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:34.726 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:34.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:34.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:34.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:34.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:34.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2594716' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-17T16:29:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3947782517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:34.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:34 smithi173 ceph-mon[30530]: pgmap v1390: 1 pgs: 1 active+clean; 577 KiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:34.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:34.981 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:34.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2594716' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-17T16:29:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3947782517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:34 smithi017 ceph-mon[27571]: pgmap v1390: 1 pgs: 1 active+clean; 577 KiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2594716' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-17T16:29:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3947782517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:34 smithi071 ceph-mon[30002]: pgmap v1390: 1 pgs: 1 active+clean; 577 KiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:35.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:35.110 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:35.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:35.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:35.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:35.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:35.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:35.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:35.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:35.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:35.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:35.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:35.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:35.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:35.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:35.623 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:35.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2594716' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-17T16:29:35.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:35 smithi017 ceph-mon[27571]: osdmap e629: 12 total, 12 up, 12 in 2024-09-17T16:29:35.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:35.737 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-17T16:29:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:35.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:35.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:35.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:35.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:35.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2594716' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-17T16:29:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:35 smithi173 ceph-mon[30530]: osdmap e629: 12 total, 12 up, 12 in 2024-09-17T16:29:35.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:36.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:36.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2594716' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-17T16:29:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:35 smithi071 ceph-mon[30002]: osdmap e629: 12 total, 12 up, 12 in 2024-09-17T16:29:36.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:36.128 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:36.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:36.180 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:29:36.192 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:36.225 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:36.229 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:36.252 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:36.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:36.354 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:36.382 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:36.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:36.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:36.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:36.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:36.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:36.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:36.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:36.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:36.761 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:36.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:36.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:36.820 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:36 smithi017 ceph-mon[27571]: osdmap e630: 12 total, 12 up, 12 in 2024-09-17T16:29:36.820 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:36 smithi017 ceph-mon[27571]: pgmap v1393: 17 pgs: 1 creating+peering, 3 active+clean, 13 unknown; 577 KiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:36.832 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:29:36.833 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:29:36.833 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-09-17T16:29:36.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd out 4 2024-09-17T16:29:36.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:36.890 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:36.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:36.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:36 smithi173 ceph-mon[30530]: osdmap e630: 12 total, 12 up, 12 in 2024-09-17T16:29:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:36 smithi173 ceph-mon[30530]: pgmap v1393: 17 pgs: 1 creating+peering, 3 active+clean, 13 unknown; 577 KiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:36.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:37.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:37.018 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:37.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:36 smithi071 ceph-mon[30002]: osdmap e630: 12 total, 12 up, 12 in 2024-09-17T16:29:37.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:36 smithi071 ceph-mon[30002]: pgmap v1393: 17 pgs: 1 creating+peering, 3 active+clean, 13 unknown; 577 KiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:37.071 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:29:37.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:37.146 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:37.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:37.248 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:37.274 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:37.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:37.318 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:37.319 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:37.355 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:37.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:37.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:37.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:37.503 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:37.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:37.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:37.628 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:37.634 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:29:37.634 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-17T16:29:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:37.654 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:37.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:37.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:37 smithi017 ceph-mon[27571]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:29:37.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:37 smithi017 ceph-mon[27571]: osdmap e631: 12 total, 12 up, 12 in 2024-09-17T16:29:37.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4290544546' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:29:37.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3487646519' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:29:37.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:37.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:37.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:37.883 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:37 smithi173 ceph-mon[30530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:29:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:37 smithi173 ceph-mon[30530]: osdmap e631: 12 total, 12 up, 12 in 2024-09-17T16:29:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4290544546' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:29:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3487646519' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:29:37.909 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:37.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:38.011 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:38.035 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:38.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:37 smithi071 ceph-mon[30002]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:29:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:37 smithi071 ceph-mon[30002]: osdmap e631: 12 total, 12 up, 12 in 2024-09-17T16:29:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4290544546' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:29:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3487646519' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:29:38.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:38.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:38.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:38.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:38.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:38.286 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:38.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:38.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:38.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:38.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:38.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:38 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:29:38.150+0000 7f086a07d640 -1 mon.a@0(leader).osd e632 definitely_dead 0 2024-09-17T16:29:38.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:38.541 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:38.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:38.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:38.669 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:38.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:38 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4290544546' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:29:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3487646519' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:29:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:38 smithi017 ceph-mon[27571]: osdmap e632: 12 total, 12 up, 12 in 2024-09-17T16:29:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:38 smithi017 ceph-mon[27571]: pgmap v1396: 17 pgs: 1 creating+peering, 15 active+clean, 1 unknown; 577 KiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1120107133' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-17T16:29:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:38 smithi017 ceph-mon[27571]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-17T16:29:38.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:38.796 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:38.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:38 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4290544546' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:29:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3487646519' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:29:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:38 smithi173 ceph-mon[30530]: osdmap e632: 12 total, 12 up, 12 in 2024-09-17T16:29:38.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:38 smithi173 ceph-mon[30530]: pgmap v1396: 17 pgs: 1 creating+peering, 15 active+clean, 1 unknown; 577 KiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:38.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1120107133' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-17T16:29:38.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:38 smithi173 ceph-mon[30530]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-17T16:29:38.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:38.924 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:38.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:39.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:38 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4290544546' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:29:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3487646519' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:29:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:38 smithi071 ceph-mon[30002]: osdmap e632: 12 total, 12 up, 12 in 2024-09-17T16:29:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:38 smithi071 ceph-mon[30002]: pgmap v1396: 17 pgs: 1 creating+peering, 15 active+clean, 1 unknown; 577 KiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:29:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1120107133' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-17T16:29:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:38 smithi071 ceph-mon[30002]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-17T16:29:39.074 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:39.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:39.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:39.094 INFO:teuthology.orchestra.run.smithi017.stderr:marked out osd.4. 2024-09-17T16:29:39.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:39.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:39.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:39.289 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:29:39.290 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:29:39.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:39.310 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:29:39.311 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:29:39.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:39.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:39.331 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:29:39.332 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:29:39.352 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:29:39.352 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:29:39.374 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:29:39.374 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:29:39.395 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:29:39.396 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:29:39.417 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:29:39.417 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:29:39.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:39.439 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:29:39.439 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:29:39.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:39.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:39.462 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:29:39.462 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:29:39.484 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:29:39.484 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:29:39.505 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:29:39.506 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:29:39.527 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:29:39.527 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:29:39.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:39.585 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:39.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:39.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:39.711 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:39.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:39.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:39.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:39.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:39.877 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-17T16:29:39.881 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T16:29:39.881 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T16:29:39.883 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:39 smithi017 ceph-mon[27571]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:29:39.883 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1120107133' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-17T16:29:39.883 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:39 smithi017 ceph-mon[27571]: osdmap e633: 12 total, 12 up, 11 in 2024-09-17T16:29:39.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/580211669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:39.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:39 smithi173 ceph-mon[30530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:29:39.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1120107133' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-17T16:29:39.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:39 smithi173 ceph-mon[30530]: osdmap e633: 12 total, 12 up, 11 in 2024-09-17T16:29:39.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/580211669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:39.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:39.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-09-17T16:29:39.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T16:29:39.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_116396 2024-09-17T16:29:39.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:29:39.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-09-17T16:29:39.964 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:39 smithi071 ceph-mon[30002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:29:39.964 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1120107133' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-17T16:29:39.964 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:39 smithi071 ceph-mon[30002]: osdmap e633: 12 total, 12 up, 11 in 2024-09-17T16:29:39.964 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/580211669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:39.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:39.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:40.071 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:40.099 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:40.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:40.201 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:40.223 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:40.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:40.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:40.347 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:40.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:40.449 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:40.483 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:40.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:40.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:40.608 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:40.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:40.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:40.748 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:40.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:40.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:40.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:40 smithi071 ceph-mon[30002]: osdmap e634: 12 total, 12 up, 11 in 2024-09-17T16:29:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:40 smithi071 ceph-mon[30002]: pgmap v1399: 17 pgs: 1 creating+peering, 16 active+clean; 577 KiB data, 4.8 GiB used, 979 GiB / 983 GiB avail 2024-09-17T16:29:40.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:40.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:40.879 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:40.896 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:40 smithi017 ceph-mon[27571]: osdmap e634: 12 total, 12 up, 11 in 2024-09-17T16:29:40.896 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:40 smithi017 ceph-mon[27571]: pgmap v1399: 17 pgs: 1 creating+peering, 16 active+clean; 577 KiB data, 4.8 GiB used, 979 GiB / 983 GiB avail 2024-09-17T16:29:40.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 15 1966 1951 121.931 121.938 0.0104979 0.00811743 2024-09-17T16:29:40.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:41.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:41.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:41.006 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:41.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:41.130 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:41.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:41.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:40 smithi173 ceph-mon[30530]: osdmap e634: 12 total, 12 up, 11 in 2024-09-17T16:29:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:40 smithi173 ceph-mon[30530]: pgmap v1399: 17 pgs: 1 creating+peering, 16 active+clean; 577 KiB data, 4.8 GiB used, 979 GiB / 983 GiB avail 2024-09-17T16:29:41.231 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:41.253 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:41.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:41.338 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:29:41.338 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:29:41.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:29:41.338 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:29:41.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.338 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(4.6 383/382 [PushOp(4:63469528:::benchmark_data_smithi173_77474_object91937:head, version: 345'5625, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:63469528:::benchmark_data_smithi173_77474_object91937:head@345'5625, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 538.43092406300002, 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29598169099999999, 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396431+0000", 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396432+0000", 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9999999999999997e-07 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396467+0000", 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5184000000000001e-05 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692000+0000", 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29553331999999999 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692039+0000", 2024-09-17T16:29:41.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8491999999999997e-05 2024-09-17T16:29:41.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692173+0000", 2024-09-17T16:29:41.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134411 2024-09-17T16:29:41.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692186+0000", 2024-09-17T16:29:41.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3118e-05 2024-09-17T16:29:41.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692413+0000", 2024-09-17T16:29:41.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022656600000000001 2024-09-17T16:29:41.343 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.343 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.343 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.344 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_backfill(progress 4.6 e 383/383 lb 4:6341a44b:::benchmark_data_smithi173_77474_object88529:head)", 2024-09-17T16:29:41.344 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:29:41.344 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 538.43086423700004, 2024-09-17T16:29:41.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29619305299999998, 2024-09-17T16:29:41.344 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.344 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.344 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:29:41.344 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:29:41.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396491+0000", 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.396495+0000", 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9450000000000003e-06 2024-09-17T16:29:41.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692562+0000", 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29606637800000002 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692584+0000", 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2526000000000001e-05 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692645+0000", 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1309999999999994e-05 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692657+0000", 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1798e-05 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:20:42.692684+0000", 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6821e-05 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.348 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.349 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.349 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:82763 5.13 e475/453 5:cd63b9f8:::benchmark_data_smithi173_90735_object82762:head v 475'4007, mlcod=475'4003)", 2024-09-17T16:29:41.349 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:29:41.349 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 380.90915598300001, 2024-09-17T16:29:41.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56469166699999995, 2024-09-17T16:29:41.349 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.349 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918200+0000", 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1500000000000004e-07 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918245+0000", 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5034000000000003e-05 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481583+0000", 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56333854699999997 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481588+0000", 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5800000000000002e-06 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481652+0000", 2024-09-17T16:29:41.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4309999999999999e-05 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481669+0000", 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6880000000000001e-05 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.482864+0000", 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011944200000000001 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.482891+0000", 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7180999999999999e-05 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPull(5.12 e475/455 cost 1049576)", 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 380.86789641500002, 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54846142799999997, 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.354 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959460+0000", 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5099999999999998e-07 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959470+0000", 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9380000000000004e-06 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507615+0000", 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54814523100000001 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507626+0000", 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1484000000000001e-05 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507695+0000", 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8990000000000002e-05 2024-09-17T16:29:41.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507920+0000", 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022483400000000001 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(5.d 475/455 [PushOp(5:b9f39261:::benchmark_data_smithi173_90735_object37371:head, version: 439'2383, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:b9f39261:::benchmark_data_smithi173_90735_object37371:head@439'2383, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:29:41.358 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 380.34697876400003, 2024-09-17T16:29:41.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45551481399999999, 2024-09-17T16:29:41.359 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.359 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.359 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:29:41.359 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:29:41.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480378+0000", 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6470000000000001e-06 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480434+0000", 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6162999999999997e-05 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934248+0000", 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45381411300000002 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934260+0000", 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1382e-05 2024-09-17T16:29:41.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934362+0000", 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000102681 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934379+0000", 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6498e-05 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.935891+0000", 2024-09-17T16:29:41.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015123300000000001 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:95580 5.b e482/419 5:d4bc1ca6:::benchmark_data_smithi173_90735_object12809:head v 482'5935, mlcod=482'5933)", 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 367.09488794999999, 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39904708700000002, 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732468+0000", 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732477+0000", 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4880000000000007e-06 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128932+0000", 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.396455369 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128940+0000", 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5089999999999996e-06 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128990+0000", 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0068999999999998e-05 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.129012+0000", 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2218e-05 2024-09-17T16:29:41.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.131477+0000", 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002464843 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.131514+0000", 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7515000000000003e-05 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:95584 5.b e482/419 5:dc96c928:::benchmark_data_smithi173_90735_object12813:head v 482'5936, mlcod=482'5933)", 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 366.94324075600002, 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39023148299999999, 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:29:41.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884116+0000", 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9319999999999999e-06 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884127+0000", 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0198e-05 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272085+0000", 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38795868300000003 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272089+0000", 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1729999999999997e-06 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272129+0000", 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9937000000000001e-05 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272145+0000", 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5863e-05 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.274307+0000", 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021616040000000001 2024-09-17T16:29:41.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.274346+0000", 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9093e-05 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17628 6.9 6:9546f3b4:::benchmark_data_smithi173_102017_object17627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:21.997751+0000", 2024-09-17T16:29:41.372 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 258.82960395399999, 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49739271200000001, 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17628 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997751+0000", 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997751+0000", 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997757+0000", 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3489999999999999e-06 2024-09-17T16:29:41.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997802+0000", 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5543000000000001e-05 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997822+0000", 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9443999999999999e-05 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997831+0000", 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9740000000000008e-06 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998743+0000", 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091201499999999996 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998896+0000", 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015307299999999999 2024-09-17T16:29:41.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.999045+0000", 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014901199999999999 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000028+0000", 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00098290300000000003 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.450171+0000", 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45014342099999999 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.495073+0000", 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.044902035999999999 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.495110+0000", 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7134000000000001e-05 2024-09-17T16:29:41.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.495144+0000", 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3807999999999999e-05 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17643 6.9 6:992c966e:::benchmark_data_smithi173_102017_object17642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.493337+0000", 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 258.33401813900002, 2024-09-17T16:29:41.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42163531900000001, 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17643 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493337+0000", 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493337+0000", 2024-09-17T16:29:41.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493339+0000", 2024-09-17T16:29:41.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4229999999999999e-06 2024-09-17T16:29:41.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493447+0000", 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107039 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493469+0000", 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2408999999999998e-05 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493478+0000", 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6300000000000004e-06 2024-09-17T16:29:41.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493543+0000", 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5235999999999997e-05 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493660+0000", 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011726999999999999 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493766+0000", 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010634899999999999 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.494765+0000", 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099869900000000003 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.494990+0000", 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000224739 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.914894+0000", 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41990391300000002 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.914938+0000", 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4121e-05 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.914972+0000", 2024-09-17T16:29:41.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4490999999999999e-05 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:27649 6.d 6:bbb9887f:::benchmark_data_smithi173_102017_object27648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:32.713773+0000", 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 248.11358195400001, 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29677943299999998, 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.387 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 27649 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713773+0000", 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713773+0000", 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713774+0000", 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1310000000000001e-06 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713850+0000", 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5955e-05 2024-09-17T16:29:41.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713862+0000", 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1539000000000001e-05 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713870+0000", 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.695e-06 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.390 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.713929+0000", 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9944000000000001e-05 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.714043+0000", 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113013 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,10", 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.714177+0000", 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134401 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.715264+0000", 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010869860000000001 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.717509+0000", 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022450970000000002 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010391+0000", 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29288151099999998 2024-09-17T16:29:41.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010435+0000", 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4580999999999999e-05 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010553+0000", 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011758 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26665.0:27649 6.d e545/539 ondisk, result = 0)", 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:32.715533+0000", 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 248.11182240100001, 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.295026646, 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.715533+0000", 2024-09-17T16:29:41.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.715533+0000", 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.715533+0000", 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.715539+0000", 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7300000000000002e-06 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010314+0000", 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.294775175 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010326+0000", 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2106000000000001e-05 2024-09-17T16:29:41.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010375+0000", 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9042000000000002e-05 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010384+0000", 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7579999999999998e-06 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.010559+0000", 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017520599999999999 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(6.d 545/539 [PushReplyOp(6:b170362a:::benchmark_data_smithi173_102017_object16956:head)])", 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:33.019918+0000", 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 247.807436908, 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29282309400000001, 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:29:41.398 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019918+0000", 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019918+0000", 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019919+0000", 2024-09-17T16:29:41.399 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7100000000000002e-07 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019923+0000", 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9149999999999998e-06 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.312357+0000", 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29243439700000001 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.312364+0000", 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5250000000000002e-06 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.312419+0000", 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5792999999999999e-05 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.312741+0000", 2024-09-17T16:29:41.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032189300000000002 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:105206 6.d 6:bffb0a05:::benchmark_data_smithi173_102017_object12058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:57.892731+0000", 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 162.93462386100001, 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27620592999999999, 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.402 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 105206 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:57.892731+0000", 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:57.892731+0000", 2024-09-17T16:29:41.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:57.892732+0000", 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.13e-06 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:57.892737+0000", 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6059999999999996e-06 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:57.892746+0000", 2024-09-17T16:29:41.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5290000000000008e-06 2024-09-17T16:29:41.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:57.892751+0000", 2024-09-17T16:29:41.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1379999999999998e-06 2024-09-17T16:29:41.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:57.995434+0000", 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.10268289899999999 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for rw locks", 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:57.995526+0000", 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2455999999999997e-05 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:58.003028+0000", 2024-09-17T16:29:41.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075021410000000004 2024-09-17T16:29:41.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.407 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:58.003127+0000", 2024-09-17T16:29:41.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9160000000000006e-05 2024-09-17T16:29:41.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,5,10", 2024-09-17T16:29:41.407 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:58.003264+0000", 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000136427 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:58.019137+0000", 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015873221999999999 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:58.022468+0000", 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033313370000000002 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:58.057874+0000", 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.035405144 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:58.166072+0000", 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.108198735 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:58.166261+0000", 2024-09-17T16:29:41.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018913 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:58.168937+0000", 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026758760000000002 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:114687 6.9 6:96e26e86:::benchmark_data_smithi173_102017_object21539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:12.569195+0000", 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 148.25815999100001, 2024-09-17T16:29:41.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24756941900000001, 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 114687 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.569195+0000", 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.569195+0000", 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.569196+0000", 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.173e-06 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.569211+0000", 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4968e-05 2024-09-17T16:29:41.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.569253+0000", 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2138999999999998e-05 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.569261+0000", 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9570000000000002e-06 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.413 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.569308+0000", 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6331000000000001e-05 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.569425+0000", 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116937 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.569523+0000", 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8250000000000003e-05 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.572903+0000", 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033796859999999998 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.572994+0000", 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1539000000000006e-05 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.816661+0000", 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.243666508 2024-09-17T16:29:41.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.816724+0000", 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2885999999999994e-05 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.816765+0000", 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1044999999999998e-05 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:114688 6.0 6:070d0d40:::benchmark_data_smithi173_102017_object21540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:12.570735+0000", 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 148.25661973199999, 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.244633352, 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 114688 2024-09-17T16:29:41.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.570735+0000", 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.570735+0000", 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.570737+0000", 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7570000000000001e-06 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.570742+0000", 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.113e-06 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.570755+0000", 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3106e-05 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.570762+0000", 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6800000000000004e-06 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.570816+0000", 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3990999999999999e-05 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.570897+0000", 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0813e-05 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.570961+0000", 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3886999999999993e-05 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.574336+0000", 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033750899999999999 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.574749+0000", 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00041279400000000002 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.815243+0000", 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.240494454 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.815298+0000", 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5293000000000001e-05 2024-09-17T16:29:41.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.815369+0000", 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0374000000000001e-05 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:114694 6.0 6:0cb9f150:::benchmark_data_smithi173_102017_object21546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:12.590117+0000", 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 148.237238249, 2024-09-17T16:29:41.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23263436700000001, 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 114694 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.590117+0000", 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.590117+0000", 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.590119+0000", 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7689999999999999e-06 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.590124+0000", 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5450000000000003e-06 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.590138+0000", 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4028e-05 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.590145+0000", 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.844e-06 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.590203+0000", 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8034e-05 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.590290+0000", 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7324999999999995e-05 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.590375+0000", 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4241999999999997e-05 2024-09-17T16:29:41.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.593555+0000", 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031805359999999999 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.593754+0000", 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019875599999999999 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.822650+0000", 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22889646299999999 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.822706+0000", 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5943999999999998e-05 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:12.822751+0000", 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4880999999999999e-05 2024-09-17T16:29:41.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(6.d 582/568 [PushReplyOp(6:bd4f72bd:::benchmark_data_smithi173_102017_object30472:head)])", 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:21.019823+0000", 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 139.80753174500001, 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48108583100000002, 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.019823+0000", 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.019823+0000", 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.019825+0000", 2024-09-17T16:29:41.431 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.085e-06 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.019830+0000", 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6720000000000001e-06 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.497888+0000", 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47805754900000003 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.497896+0000", 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6370000000000005e-06 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.497969+0000", 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2470000000000002e-05 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.500909+0000", 2024-09-17T16:29:41.433 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029404180000000002 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:137287 6.9 6:94a197ff:::benchmark_data_smithi173_102017_object44139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e592)", 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:28:02.514003+0000", 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 98.313352199999997, 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21399362799999999, 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.434 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 137287 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.514003+0000", 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.514003+0000", 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.514004+0000", 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2999999999999999e-07 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.514008+0000", 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0899999999999998e-06 2024-09-17T16:29:41.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.514018+0000", 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0538e-05 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.514027+0000", 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7609999999999996e-06 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.518183+0000", 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041560989999999999 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.518462+0000", 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027834600000000001 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.518937+0000", 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047556299999999997 2024-09-17T16:29:41.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.524268+0000", 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053309400000000002 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.530496+0000", 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0062280670000000003 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.727868+0000", 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.19737153199999999 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.727922+0000", 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4265999999999998e-05 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:28:02.727997+0000", 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4596000000000002e-05 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.440 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:186218 6.9 6:9ec80bfb:::benchmark_data_smithi173_102017_object93070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e621)", 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:29:17.436586+0000", 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.390768775000002, 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.313966564, 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:29:41.441 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186218 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.436586+0000", 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.436586+0000", 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.436589+0000", 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3240000000000001e-06 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.436596+0000", 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6369999999999999e-06 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.436614+0000", 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8087000000000001e-05 2024-09-17T16:29:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.436625+0000", 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0322e-05 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.436720+0000", 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4944000000000004e-05 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.436861+0000", 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000141349 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.437209+0000", 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00034789400000000001 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.442972+0000", 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057632689999999997 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.443033+0000", 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0690000000000003e-05 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.750433+0000", 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30739994700000001 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.750499+0000", 2024-09-17T16:29:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5942e-05 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.750553+0000", 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4159000000000001e-05 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.26665.0:186218 6.9 e621/527 ondisk, result = 0)", 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:29:17.442290+0000", 2024-09-17T16:29:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.385065629, 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30827019, 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.442290+0000", 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.442290+0000", 2024-09-17T16:29:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.442291+0000", 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.72e-06 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.442301+0000", 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9639999999999998e-06 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:29:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.750304+0000", 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30800324499999998 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.750312+0000", 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8960000000000003e-06 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.750414+0000", 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000101631 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.750425+0000", 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0978999999999999e-05 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.750560+0000", 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013475499999999999 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:29:41.451 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:29:41.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:41.474 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:41.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:41.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:41.607 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:41.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:41.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:41.741 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:41.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:41 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:29:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:41 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:29:41.843 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:41.865 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:41.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:41 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:29:41.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:41 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:29:41.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 3133 3117 97.4002 72.875 0.00130719 0.0101602 2024-09-17T16:29:41.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:41.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:41 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:29:41.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:41 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:29:41.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:41.990 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:41.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:42.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:42.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:42.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:42.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:42.242 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:42.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:42.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:42.365 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:42.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:42.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:42.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:42.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:42.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:42.619 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:42.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:42.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:42.750 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:42.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:42.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:29:42.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:42.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:29:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:42.900 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:42.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:42.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: pgmap v1400: 17 pgs: 17 active+clean; 121 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 22 MiB/s wr, 359 op/s 2024-09-17T16:29:42.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2923698731,config=1059521453,health=1997216375,kv=934586102}) 2024-09-17T16:29:42.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2076377730,logm=879219170}) 2024-09-17T16:29:42.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2369990081}) 2024-09-17T16:29:42.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3472724531}) 2024-09-17T16:29:42.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986593371}) 2024-09-17T16:29:42.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3892863403}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608203479}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1028765151}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830156865}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017219221}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3163991451}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976657088}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994577394}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228186772}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513011902}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1019089576}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1562040657}) 2024-09-17T16:29:42.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1939701297}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202739429}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429485943}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933017628}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490763907}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413101616}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3892563289}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179993420}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826373082}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3775024339}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1896053547}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3392735833}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864710863}) 2024-09-17T16:29:42.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=718497686}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581223697}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3141518112}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992159658}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4223213282}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3578162085}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3032310862}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=314108633}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1078773366}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1428740331}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=752716589}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1830851690}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462908408}) 2024-09-17T16:29:42.930 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3332743360}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3373886626}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998995822}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933290962}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3947496753}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743822187}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3757397767}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3757839890}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=46,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=16} crc {logm=652706751,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=1831427809,mon_config_key=1161575990}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4107285624}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1670109086}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=69,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=10} crc {mon_config_key=2179969976,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=2349339223,osdmap=2877664439}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1636934316}) 2024-09-17T16:29:42.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=13} crc {osdmap=1621240093}) 2024-09-17T16:29:42.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 4342 4326 90.1189 75.5625 0.007171 0.0110605 2024-09-17T16:29:43.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:43.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:43.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:43.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: pgmap v1400: 17 pgs: 17 active+clean; 121 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 22 MiB/s wr, 359 op/s 2024-09-17T16:29:43.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2923698731,config=1059521453,health=1997216375,kv=934586102}) 2024-09-17T16:29:43.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2076377730,logm=879219170}) 2024-09-17T16:29:43.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2369990081}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3472724531}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986593371}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3892863403}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608203479}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1028765151}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830156865}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017219221}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3163991451}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976657088}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994577394}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228186772}) 2024-09-17T16:29:43.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513011902}) 2024-09-17T16:29:43.067 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1019089576}) 2024-09-17T16:29:43.067 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1562040657}) 2024-09-17T16:29:43.067 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1939701297}) 2024-09-17T16:29:43.067 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202739429}) 2024-09-17T16:29:43.067 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429485943}) 2024-09-17T16:29:43.067 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933017628}) 2024-09-17T16:29:43.067 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490763907}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413101616}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3892563289}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179993420}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826373082}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3775024339}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1896053547}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3392735833}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864710863}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=718497686}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581223697}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3141518112}) 2024-09-17T16:29:43.068 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992159658}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4223213282}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3578162085}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3032310862}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=314108633}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1078773366}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1428740331}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=752716589}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1830851690}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462908408}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3332743360}) 2024-09-17T16:29:43.069 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3373886626}) 2024-09-17T16:29:43.070 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998995822}) 2024-09-17T16:29:43.070 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933290962}) 2024-09-17T16:29:43.070 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3947496753}) 2024-09-17T16:29:43.070 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743822187}) 2024-09-17T16:29:43.071 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3757397767}) 2024-09-17T16:29:43.071 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3757839890}) 2024-09-17T16:29:43.071 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=46,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=16} crc {logm=652706751,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=1831427809,mon_config_key=1161575990}) 2024-09-17T16:29:43.071 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4107285624}) 2024-09-17T16:29:43.071 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1670109086}) 2024-09-17T16:29:43.071 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=69,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=10} crc {mon_config_key=2179969976,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=2349339223,osdmap=2877664439}) 2024-09-17T16:29:43.071 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1636934316}) 2024-09-17T16:29:43.072 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=13} crc {osdmap=1621240093}) 2024-09-17T16:29:43.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: pgmap v1400: 17 pgs: 17 active+clean; 121 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 22 MiB/s wr, 359 op/s 2024-09-17T16:29:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=2923698731,config=1059521453,health=1997216375,kv=934586102}) 2024-09-17T16:29:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=2076377730,logm=879219170}) 2024-09-17T16:29:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2369990081}) 2024-09-17T16:29:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3472724531}) 2024-09-17T16:29:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3986593371}) 2024-09-17T16:29:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3892863403}) 2024-09-17T16:29:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608203479}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1028765151}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830156865}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017219221}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3163991451}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976657088}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994577394}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228186772}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513011902}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1019089576}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1562040657}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1939701297}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202739429}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429485943}) 2024-09-17T16:29:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933017628}) 2024-09-17T16:29:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490763907}) 2024-09-17T16:29:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413101616}) 2024-09-17T16:29:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3892563289}) 2024-09-17T16:29:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179993420}) 2024-09-17T16:29:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826373082}) 2024-09-17T16:29:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3775024339}) 2024-09-17T16:29:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1896053547}) 2024-09-17T16:29:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3392735833}) 2024-09-17T16:29:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1864710863}) 2024-09-17T16:29:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=718497686}) 2024-09-17T16:29:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1581223697}) 2024-09-17T16:29:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3141518112}) 2024-09-17T16:29:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992159658}) 2024-09-17T16:29:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4223213282}) 2024-09-17T16:29:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3578162085}) 2024-09-17T16:29:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3032310862}) 2024-09-17T16:29:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=314108633}) 2024-09-17T16:29:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1078773366}) 2024-09-17T16:29:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1428740331}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=752716589}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1830851690}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462908408}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3332743360}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3373886626}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998995822}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933290962}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3947496753}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743822187}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3757397767}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3757839890}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=46,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=14,mon_config_key=16} crc {logm=652706751,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=1831427809,mon_config_key=1161575990}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4107285624}) 2024-09-17T16:29:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1670109086}) 2024-09-17T16:29:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=69,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=10} crc {mon_config_key=2179969976,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=2349339223,osdmap=2877664439}) 2024-09-17T16:29:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1636934316}) 2024-09-17T16:29:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=13} crc {osdmap=1621240093}) 2024-09-17T16:29:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:43.151 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:43.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:43.243 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:43.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:43.274 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:43.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:43.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:43.397 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:43.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:43.499 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:43.527 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:43.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:43.629 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:43.659 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:43.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:43.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:43.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:43.787 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:43.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:43.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:43.911 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:43.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:43.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:43.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 5751 5735 89.6017 88.0625 0.0134372 0.0110742 2024-09-17T16:29:44.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:44.037 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:44.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:44.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:44.161 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:44.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:44.263 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:44.287 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:44.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:44.287 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:29:44.388 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:44.421 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:44.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:44.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:44.546 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:44.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:44.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:44.661 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:29:44.684 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:44.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:44.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:44.789 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:29:44.789 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:29:44.789 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:29:44.789 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:29:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:44.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:44.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:44.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:44.914 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:29:44.914 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:29:44.914 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-17T16:29:44.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd in 4 2024-09-17T16:29:44.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:44.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:44.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:44 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:44.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:44 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:44.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:44 smithi017 ceph-mon[27571]: osdmap e635: 12 total, 12 up, 11 in 2024-09-17T16:29:44.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:44 smithi017 ceph-mon[27571]: pgmap v1402: 17 pgs: 17 active+clean; 148 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 25 MiB/s wr, 392 op/s; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:29:44.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3322680022' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:44.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:44.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 15 6850 6835 85.4301 68.75 0.0145386 0.0116746 2024-09-17T16:29:45.040 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:45.064 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:45.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:44 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:45.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:44 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:45.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:44 smithi071 ceph-mon[30002]: osdmap e635: 12 total, 12 up, 11 in 2024-09-17T16:29:45.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:44 smithi071 ceph-mon[30002]: pgmap v1402: 17 pgs: 17 active+clean; 148 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 25 MiB/s wr, 392 op/s; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:29:45.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3322680022' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:45.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:45.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:44 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:44 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:44 smithi173 ceph-mon[30530]: osdmap e635: 12 total, 12 up, 11 in 2024-09-17T16:29:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:44 smithi173 ceph-mon[30530]: pgmap v1402: 17 pgs: 17 active+clean; 148 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 25 MiB/s wr, 392 op/s; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:29:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3322680022' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:29:45.168 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:45.198 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:45.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:45.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:45.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:45.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:45.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:45.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:45.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:45.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:45.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:45.547 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:45.579 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:45.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:45.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:45.713 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:45.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:45.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:45 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:29:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:45 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:29:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:45 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:29:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:45 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:29:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:45 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:29:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:45 smithi017 ceph-mon[27571]: osdmap e636: 12 total, 12 up, 11 in 2024-09-17T16:29:45.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:45.837 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:45.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:45.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:29:45.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:45.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 8013 7997 83.295 72.625 0.00686559 0.0119735 2024-09-17T16:29:45.960 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:45.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:46.061 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:46.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:45 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:29:46.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:45 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:29:46.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:45 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:29:46.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:45 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:29:46.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:45 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:29:46.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:45 smithi071 ceph-mon[30002]: osdmap e636: 12 total, 12 up, 11 in 2024-09-17T16:29:46.084 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:46.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:45 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:29:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:45 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:29:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:45 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:29:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:45 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:29:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:45 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:29:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:45 smithi173 ceph-mon[30530]: osdmap e636: 12 total, 12 up, 11 in 2024-09-17T16:29:46.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:46.210 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:46.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:46.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:46.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:46.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:46.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:46.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:46.467 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:46.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:46.569 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:46.602 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:46.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:46.626 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:46.626 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:46.685 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:46.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:46.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:46.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:46.815 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:46 smithi017 ceph-mon[27571]: pgmap v1404: 17 pgs: 17 active+clean; 245 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 39 MiB/s wr, 617 op/s; 81 KiB/s, 0 objects/s recovering 2024-09-17T16:29:46.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:46.859 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:46.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:46.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 9572 9556 85.3142 97.4375 0.00346112 0.0117056 2024-09-17T16:29:46.961 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:46.986 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:46.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:46.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:47.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:46 smithi071 ceph-mon[30002]: pgmap v1404: 17 pgs: 17 active+clean; 245 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 39 MiB/s wr, 617 op/s; 81 KiB/s, 0 objects/s recovering 2024-09-17T16:29:47.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:47.116 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:47.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:47.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:46 smithi173 ceph-mon[30530]: pgmap v1404: 17 pgs: 17 active+clean; 245 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 39 MiB/s wr, 617 op/s; 81 KiB/s, 0 objects/s recovering 2024-09-17T16:29:47.219 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:47.240 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:47.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:47.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:47.342 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:47.364 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:47.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:47.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:47.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:47.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:47.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:47.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:47.728 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:47.750 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:47.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:47.851 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:47.861 INFO:teuthology.orchestra.run.smithi017.stderr:marked in osd.4. 2024-09-17T16:29:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:47.875 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:47.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:47.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 10920 10904 85.1801 84.25 0.00200771 0.0116227 2024-09-17T16:29:47.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:47.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:47.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:48.013 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:47 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:29:47.803+0000 7f086a07d640 -1 mon.a@0(leader).osd e636 definitely_dead 0 2024-09-17T16:29:48.013 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4038462935' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-17T16:29:48.100 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:48.125 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:48.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:48.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4038462935' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-17T16:29:48.160 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:29:48.160 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:29:48.161 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:29:48.161 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:29:48.175 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:29:48.175 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:29:48.196 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:29:48.196 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:29:48.218 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:29:48.218 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:29:48.227 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:48.239 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:29:48.239 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:29:48.249 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:48.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:48.261 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:29:48.264 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:29:48.285 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:29:48.285 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:29:48.306 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:29:48.306 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:29:48.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4038462935' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-17T16:29:48.328 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:29:48.328 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:29:48.349 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:29:48.350 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:29:48.351 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:48.370 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:29:48.371 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:29:48.373 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:48.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:48.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:48.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:48.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:48.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:29:48.598 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:48.627 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:48.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:48.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:48.754 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:48.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:48.831 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:29:48.831 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:29:48.831 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:29:48.831 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:29:48.831 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:29:48.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:48.882 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:48.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:48.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 10954 10938 75.9517 2.125 0.169412 0.0120289 2024-09-17T16:29:48.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:49.006 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:49.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:49.108 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:49.130 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:49.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:49.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4038462935' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-17T16:29:49.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:48 smithi017 ceph-mon[27571]: osdmap e637: 12 total, 12 up, 12 in 2024-09-17T16:29:49.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:48 smithi017 ceph-mon[27571]: pgmap v1406: 17 pgs: 17 active+clean; 544 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:29:49.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1593765653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:49.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:48 smithi017 ceph-mon[27571]: osdmap e638: 12 total, 12 up, 12 in 2024-09-17T16:29:49.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4038462935' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-17T16:29:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:48 smithi173 ceph-mon[30530]: osdmap e637: 12 total, 12 up, 12 in 2024-09-17T16:29:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:48 smithi173 ceph-mon[30530]: pgmap v1406: 17 pgs: 17 active+clean; 544 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:29:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1593765653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:48 smithi173 ceph-mon[30530]: osdmap e638: 12 total, 12 up, 12 in 2024-09-17T16:29:49.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:49.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:49.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4038462935' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-17T16:29:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:48 smithi071 ceph-mon[30002]: osdmap e637: 12 total, 12 up, 12 in 2024-09-17T16:29:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:48 smithi071 ceph-mon[30002]: pgmap v1406: 17 pgs: 17 active+clean; 544 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s; 86 KiB/s, 0 objects/s recovering 2024-09-17T16:29:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1593765653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:48 smithi071 ceph-mon[30002]: osdmap e638: 12 total, 12 up, 12 in 2024-09-17T16:29:49.359 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:49.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:49.382 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:49.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:49.484 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:49.506 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:49.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:49.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:29:49.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:49.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:49.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:49.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:49.757 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:49.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:49.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:49.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:49.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:49.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:49.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 11064 11048 69.0439 6.875 0.00490169 0.0144731 2024-09-17T16:29:49.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:50.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:50.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:50.115 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:50.138 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:50.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:50.241 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:50.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:50.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:50.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:50.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:50.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:50.498 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:50.531 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:50.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:50.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:50.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:50.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:50.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:50.789 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:50.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:50.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:50.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:50.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:50.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 12362 12346 70.1416 81.125 0.00333242 0.0142438 2024-09-17T16:29:51.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:51.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:51.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:51.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:51.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:51.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:50 smithi173 ceph-mon[30530]: osdmap e639: 12 total, 12 up, 12 in 2024-09-17T16:29:51.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:50 smithi173 ceph-mon[30530]: pgmap v1409: 17 pgs: 1 active+recovery_wait+degraded, 16 active+clean; 561 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 4/26907 objects degraded (0.015%) 2024-09-17T16:29:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:51.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:51.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:51.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:50 smithi017 ceph-mon[27571]: osdmap e639: 12 total, 12 up, 12 in 2024-09-17T16:29:51.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:50 smithi017 ceph-mon[27571]: pgmap v1409: 17 pgs: 1 active+recovery_wait+degraded, 16 active+clean; 561 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 4/26907 objects degraded (0.015%) 2024-09-17T16:29:51.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:51.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:51.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:50 smithi071 ceph-mon[30002]: osdmap e639: 12 total, 12 up, 12 in 2024-09-17T16:29:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:50 smithi071 ceph-mon[30002]: pgmap v1409: 17 pgs: 1 active+recovery_wait+degraded, 16 active+clean; 561 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 4/26907 objects degraded (0.015%) 2024-09-17T16:29:51.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:51.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:51.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:51.519 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:51.546 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:51.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:51.648 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:51.672 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:51.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:51.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:51.774 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:51.798 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:51.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:51.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:51.935 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:51.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:51.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 13716 13700 71.348 84.625 0.011871 0.0140055 2024-09-17T16:29:52.037 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:52.061 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:52.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:52.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:52.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:51 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 4/26907 objects degraded (0.015%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:29:52.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:52.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:52.189 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:52.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:51 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 4/26907 objects degraded (0.015%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:29:52.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:52.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:51 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 4/26907 objects degraded (0.015%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:29:52.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:52.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:52.417 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:52.440 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:52.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:52.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:52.564 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:52.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:52.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:52.688 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:52.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:52.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:52.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:52.825 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:52.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:52.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:52.941 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 15095 15079 72.489 86.1875 0.030868 0.0137603 2024-09-17T16:29:52.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:52.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:52.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:53.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:53.076 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:53.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:53.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:52 smithi173 ceph-mon[30530]: pgmap v1410: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 13 active+clean; 761 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s; 4/36522 objects degraded (0.011%); 1223/36522 objects misplaced (3.349%); 8.1 MiB/s, 131 objects/s recovering 2024-09-17T16:29:53.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:53.204 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:53.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:53.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:52 smithi017 ceph-mon[27571]: pgmap v1410: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 13 active+clean; 761 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s; 4/36522 objects degraded (0.011%); 1223/36522 objects misplaced (3.349%); 8.1 MiB/s, 131 objects/s recovering 2024-09-17T16:29:53.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:53.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:52 smithi071 ceph-mon[30002]: pgmap v1410: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 13 active+clean; 761 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s; 4/36522 objects degraded (0.011%); 1223/36522 objects misplaced (3.349%); 8.1 MiB/s, 131 objects/s recovering 2024-09-17T16:29:53.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:53.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:53.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:53.469 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:53.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:53.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:29:53.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:53.594 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:53.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:53.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:53.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:53.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:53.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:53.849 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:53.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:53.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 16439 16423 73.3105 84 0.00624161 0.0136013 2024-09-17T16:29:53.950 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:53.972 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:53.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:54.073 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:54.095 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:54.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:54.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:54.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:54.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:54.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:54.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:54.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:54.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:54.445 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:54.466 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:54.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:54.470 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-17T16:29:54.560 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:54.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:54.584 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:29:54.589 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:54.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:54.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:54.715 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:54.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:54.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:54.840 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:54.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:54.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 17123 17107 71.2731 42.75 0.0361185 0.0140046 2024-09-17T16:29:54.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:54.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:54.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:55.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:55.082 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:54 smithi017 ceph-mon[27571]: osdmap e640: 12 total, 12 up, 12 in 2024-09-17T16:29:55.083 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:54 smithi017 ceph-mon[27571]: pgmap v1412: 17 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 12 active+clean; 817 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 728 op/s; 4/39210 objects degraded (0.010%); 1873/39210 objects misplaced (4.777%); 8.1 MiB/s, 131 objects/s recovering 2024-09-17T16:29:55.083 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:54 smithi017 ceph-mon[27571]: osdmap e641: 12 total, 12 up, 12 in 2024-09-17T16:29:55.089 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:55.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:55.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:54 smithi173 ceph-mon[30530]: osdmap e640: 12 total, 12 up, 12 in 2024-09-17T16:29:55.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:54 smithi173 ceph-mon[30530]: pgmap v1412: 17 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 12 active+clean; 817 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 728 op/s; 4/39210 objects degraded (0.010%); 1873/39210 objects misplaced (4.777%); 8.1 MiB/s, 131 objects/s recovering 2024-09-17T16:29:55.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:54 smithi173 ceph-mon[30530]: osdmap e641: 12 total, 12 up, 12 in 2024-09-17T16:29:55.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:55.226 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:55.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:54 smithi071 ceph-mon[30002]: osdmap e640: 12 total, 12 up, 12 in 2024-09-17T16:29:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:54 smithi071 ceph-mon[30002]: pgmap v1412: 17 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 12 active+clean; 817 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 728 op/s; 4/39210 objects degraded (0.010%); 1873/39210 objects misplaced (4.777%); 8.1 MiB/s, 131 objects/s recovering 2024-09-17T16:29:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:54 smithi071 ceph-mon[30002]: osdmap e641: 12 total, 12 up, 12 in 2024-09-17T16:29:55.318 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:29:55.318 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:29:55.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:55.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:55.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:55.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:55.476 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:55.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:55.578 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:55.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:55.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:55.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:55.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:55.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:55.829 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:55.850 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:55.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:55.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 17774 17758 69.3611 40.6875 0.001975 0.0141751 2024-09-17T16:29:55.951 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:55.972 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:55.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:56.075 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:56.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:56.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:56.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:56.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:56.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:56.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:29:56.343 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:56.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:29:56.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:56.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:56.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:56.569 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:56.594 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:29:56.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:56.595 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:56.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:56.696 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:56.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:56.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:56.764 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:29:56.818 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:29:56.840 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:56.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:29:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:56 smithi173 ceph-mon[30530]: osdmap e642: 12 total, 12 up, 12 in 2024-09-17T16:29:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:56 smithi173 ceph-mon[30530]: pgmap v1415: 17 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 884 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 860 op/s; 1873/42402 objects misplaced (4.417%); 8.4 MiB/s, 132 objects/s recovering 2024-09-17T16:29:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/72759542' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:29:56.941 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:56.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 18171 18155 66.7405 24.8125 0.0102664 0.0149753 2024-09-17T16:29:56.980 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:56.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:56 smithi017 ceph-mon[27571]: osdmap e642: 12 total, 12 up, 12 in 2024-09-17T16:29:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:56 smithi017 ceph-mon[27571]: pgmap v1415: 17 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 884 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 860 op/s; 1873/42402 objects misplaced (4.417%); 8.4 MiB/s, 132 objects/s recovering 2024-09-17T16:29:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/72759542' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:29:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:56 smithi071 ceph-mon[30002]: osdmap e642: 12 total, 12 up, 12 in 2024-09-17T16:29:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:56 smithi071 ceph-mon[30002]: pgmap v1415: 17 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 13 active+clean; 884 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 860 op/s; 1873/42402 objects misplaced (4.417%); 8.4 MiB/s, 132 objects/s recovering 2024-09-17T16:29:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/72759542' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:29:57.082 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:29:57.119 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:57.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:29:57.221 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:57.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:57.357 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:29:57.390 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:57.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:29:57.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:57.515 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:57.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:57.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:57.616 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:57.637 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:57.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:57.740 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:57.762 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:57.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:57.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:57.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:57.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:57.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:57 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/39210 objects degraded (0.010%), 1 pg degraded) 2024-09-17T16:29:57.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:57 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:29:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:57 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/72759542' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:29:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:57 smithi173 ceph-mon[30530]: osdmap e643: 12 total, 12 up, 12 in 2024-09-17T16:29:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3217637666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:57.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 19720 19704 68.4107 96.8125 0.0125015 0.0146078 2024-09-17T16:29:57.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:57 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/39210 objects degraded (0.010%), 1 pg degraded) 2024-09-17T16:29:57.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:57 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:29:57.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:57 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:57.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/72759542' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:29:57.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:57 smithi017 ceph-mon[27571]: osdmap e643: 12 total, 12 up, 12 in 2024-09-17T16:29:57.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3217637666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:57.989 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:29:58.024 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:58.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:58.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:29:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:57 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/39210 objects degraded (0.010%), 1 pg degraded) 2024-09-17T16:29:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:57 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:29:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:57 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:29:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/72759542' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:29:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:57 smithi071 ceph-mon[30002]: osdmap e643: 12 total, 12 up, 12 in 2024-09-17T16:29:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3217637666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:29:58.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:58.131 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:29:58.131 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:29:58.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:58.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:58.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:29:58.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:29:58.200 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:29:58.200 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:29:58.235 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:29:58.235 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:29:58.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:58.265 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:29:58.265 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:29:58.276 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:58.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:58.292 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:29:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:29:58.314 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:29:58.314 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:29:58.341 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:29:58.341 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:29:58.374 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:29:58.374 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:29:58.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:58.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:58.404 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:29:58.404 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:29:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:58.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:58.435 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:29:58.436 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:29:58.463 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:29:58.463 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:29:58.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:29:58.528 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:58.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:58.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:29:58.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:58.653 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:58.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:58.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:58.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:58.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:58.883 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:29:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:29:58 smithi173 ceph-mon[30530]: pgmap v1417: 17 pgs: 1 active+recovering+remapped, 1 peering, 15 active+clean; 1.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s; 255/53919 objects misplaced (0.473%); 25 MiB/s, 394 objects/s recovering 2024-09-17T16:29:58.905 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:58.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:58.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:29:58.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 21104 21088 69.3623 86.5 0.00578104 0.0143937 2024-09-17T16:29:58.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:29:58 smithi017 ceph-mon[27571]: pgmap v1417: 17 pgs: 1 active+recovering+remapped, 1 peering, 15 active+clean; 1.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s; 255/53919 objects misplaced (0.473%); 25 MiB/s, 394 objects/s recovering 2024-09-17T16:29:59.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:59.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:59.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:59.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:29:58 smithi071 ceph-mon[30002]: pgmap v1417: 17 pgs: 1 active+recovering+remapped, 1 peering, 15 active+clean; 1.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s; 255/53919 objects misplaced (0.473%); 25 MiB/s, 394 objects/s recovering 2024-09-17T16:29:59.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:59.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:59.157 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:59.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:29:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:59.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:59.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:29:59.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:59.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:59.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:59.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:59.470 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:29:59.470 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:29:59.471 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-17T16:29:59.471 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:29:59.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:29:59.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:59.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:29:59.631 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:29:59.655 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:29:59.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:59.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:29:59.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:29:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:59.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:29:59.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:29:59.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:29:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:29:59.906 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:29:59.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:29:59.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:29:59.941372+0000 min lat: 0.00119669 max lat: 2.03842 avg lat: 0.0146453 2024-09-17T16:29:59.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:29:59.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 21694 21678 67.7378 36.875 0.00165149 0.0146453 2024-09-17T16:30:00.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:00.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:00.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:00.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:00.153 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:00.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:00.255 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:00.277 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:00.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:00.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:00.402 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:00.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:00.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:00.525 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:00.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:00.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:00.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:00.650 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:00.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:00.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:00.774 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:00.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:00.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:00 smithi173 ceph-mon[30530]: osdmap e644: 12 total, 12 up, 12 in 2024-09-17T16:30:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:00 smithi173 ceph-mon[30530]: pgmap v1419: 17 pgs: 1 creating+peering, 1 peering, 15 active+clean; 1.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 918 op/s; 22 MiB/s, 347 objects/s recovering 2024-09-17T16:30:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:00 smithi173 ceph-mon[30530]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-17T16:30:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:00 smithi173 ceph-mon[30530]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-17T16:30:00.909 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:00.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:00.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 22371 22355 66.5268 42.3125 0.00504178 0.0150241 2024-09-17T16:30:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:00 smithi017 ceph-mon[27571]: osdmap e644: 12 total, 12 up, 12 in 2024-09-17T16:30:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:00 smithi017 ceph-mon[27571]: pgmap v1419: 17 pgs: 1 creating+peering, 1 peering, 15 active+clean; 1.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 918 op/s; 22 MiB/s, 347 objects/s recovering 2024-09-17T16:30:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:00 smithi017 ceph-mon[27571]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-17T16:30:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:00 smithi017 ceph-mon[27571]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-17T16:30:01.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:01.046 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:01.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:00 smithi071 ceph-mon[30002]: osdmap e644: 12 total, 12 up, 12 in 2024-09-17T16:30:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:00 smithi071 ceph-mon[30002]: pgmap v1419: 17 pgs: 1 creating+peering, 1 peering, 15 active+clean; 1.1 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 918 op/s; 22 MiB/s, 347 objects/s recovering 2024-09-17T16:30:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:00 smithi071 ceph-mon[30002]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-17T16:30:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:00 smithi071 ceph-mon[30002]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-17T16:30:01.148 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:01.169 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:01.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:01.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:01.271 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:01.296 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:01.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:01.331 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:01.396 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:01.418 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:01.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:01.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:01.547 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:01.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:01.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:01.680 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:01.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:01 smithi017 ceph-mon[27571]: osdmap e645: 12 total, 12 up, 12 in 2024-09-17T16:30:01.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:01.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:01.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:01 smithi173 ceph-mon[30530]: osdmap e645: 12 total, 12 up, 12 in 2024-09-17T16:30:01.913 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:01.935 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:01.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:01.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 23684 23668 67.2327 82.0625 0.0153724 0.0148571 2024-09-17T16:30:02.037 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:02.060 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:02.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:02.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:01 smithi071 ceph-mon[30002]: osdmap e645: 12 total, 12 up, 12 in 2024-09-17T16:30:02.162 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:02.197 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:02.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:02.299 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:02.320 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:02.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:02.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:02.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:02.445 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:02.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:02.546 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:02.558 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:02.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:02.581 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:02.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:02.682 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:02.708 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:02.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:02.810 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:02.815 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:30:02.816 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:30:02.816 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:02 smithi017 ceph-mon[27571]: pgmap v1421: 17 pgs: 1 creating+peering, 16 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s; 20 MiB/s, 312 objects/s recovering 2024-09-17T16:30:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:02.832 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:02.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:02 smithi173 ceph-mon[30530]: pgmap v1421: 17 pgs: 1 creating+peering, 16 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s; 20 MiB/s, 312 objects/s recovering 2024-09-17T16:30:02.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:02.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 24741 24725 67.1816 66.0625 0.00160749 0.0148473 2024-09-17T16:30:02.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:02.956 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:03.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:30:03.056 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:03.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:02 smithi071 ceph-mon[30002]: pgmap v1421: 17 pgs: 1 creating+peering, 16 active+clean; 1.4 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s; 20 MiB/s, 312 objects/s recovering 2024-09-17T16:30:03.080 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:03.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:03.182 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:03.204 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:03.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:03.306 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:03.329 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:03.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:03.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:03.457 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:03.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:03.559 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:03.586 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:03.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:03.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:03.710 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:03.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:03.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:03.836 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:03.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:03.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:03 smithi173 ceph-mon[30530]: from='client.27118 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:03.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:03.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:30:03.942 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 25931 25915 67.4809 74.375 0.0637083 0.014768 2024-09-17T16:30:03.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:03.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:03.965 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:03.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:03 smithi017 ceph-mon[27571]: from='client.27118 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:03 smithi071 ceph-mon[30002]: from='client.27118 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:04.066 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:04.089 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:04.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:04.191 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:04.213 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:04.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:04.315 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:04.337 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:04.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:04.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:04.466 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:04.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:04.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:04.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:04.598 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:04.677 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:30:04.677 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:30:04.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.677 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:26087 7.a 7:56ce0713:::benchmark_data_smithi173_116396_object26086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-17T16:30:04.677 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.130471+0000", 2024-09-17T16:30:04.677 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.074052343000000007, 2024-09-17T16:30:04.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.074084641000000007, 2024-09-17T16:30:04.677 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:04.677 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26087 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.130471+0000", 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:04.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.130471+0000", 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.130472+0000", 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.217e-06 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:04.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.130530+0000", 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810999999999999e-05 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.130541+0000", 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1547e-05 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.130548+0000", 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0180000000000004e-06 2024-09-17T16:30:04.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.130622+0000", 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.130749+0000", 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000127169 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:30:04.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.130874+0000", 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012496 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.132743+0000", 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018687059999999999 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.132910+0000", 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016746 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:04.682 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:26093 7.a 7:50c2a194:::benchmark_data_smithi173_116396_object26092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.134213+0000", 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.070310536000000007, 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.070394193999999993, 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:04.683 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26093 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.134213+0000", 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.134213+0000", 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.134214+0000", 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.657e-06 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.134275+0000", 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0475e-05 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.134287+0000", 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2745000000000001e-05 2024-09-17T16:30:04.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.134295+0000", 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.644e-06 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.134365+0000", 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9523000000000005e-05 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.134546+0000", 2024-09-17T16:30:04.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018125500000000001 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.134705+0000", 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015927399999999999 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.135896+0000", 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011904039999999999 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27088.0:26095 7.3 e645/640 7:c8cf6237:::benchmark_data_smithi173_116396_object26094:head v 645'1671, mlcod=645'1668)", 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.135379+0000", 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.069144355000000005, 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.069266091000000002, 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.135379+0000", 2024-09-17T16:30:04.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.135379+0000", 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.135379+0000", 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3e-07 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:04.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.135419+0000", 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9403999999999997e-05 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.135450+0000", 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1041999999999999e-05 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.135457+0000", 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.99e-06 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.135532+0000", 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4861e-05 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.135552+0000", 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0027999999999999e-05 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.136622+0000", 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010704849999999999 2024-09-17T16:30:04.691 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:26093 7.a e645/629)", 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.136255+0000", 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.068268181999999997, 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.068419760999999996, 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "queued for pg", 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:04.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.136255+0000", 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.136255+0000", 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.136256+0000", 2024-09-17T16:30:04.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0500000000000003e-07 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.136259+0000", 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9219999999999999e-06 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.136273+0000", 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3444e-05 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.136279+0000", 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7359999999999998e-06 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:26018 7.8 7:151f2f24:::benchmark_data_smithi173_116396_object26017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.168393247, 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.16857, 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:04.695 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26018 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036131+0000", 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.029e-06 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036175+0000", 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4579000000000001e-05 2024-09-17T16:30:04.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036188+0000", 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.218e-05 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036203+0000", 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5494000000000001e-05 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:04.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036337+0000", 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013415599999999999 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036422+0000", 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4427999999999999e-05 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036500+0000", 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8009000000000005e-05 2024-09-17T16:30:04.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037194+0000", 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00069466700000000005 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037673+0000", 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047842099999999998 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:04.700 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:30:04.701 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 5 2024-09-17T16:30:04.701 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:30:04.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:04.723 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:04.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:04.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:04.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:04.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:04.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:04.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:04 smithi173 ceph-mon[30530]: pgmap v1422: 17 pgs: 1 creating+peering, 16 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 976 op/s; 11 MiB/s, 172 objects/s recovering 2024-09-17T16:30:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:04 smithi173 ceph-mon[30530]: osdmap e646: 12 total, 12 up, 12 in 2024-09-17T16:30:04.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 27172 27156 67.884 77.5625 0.00324036 0.0147239 2024-09-17T16:30:04.954 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:04.981 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:04.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:04.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:04 smithi017 ceph-mon[27571]: pgmap v1422: 17 pgs: 1 creating+peering, 16 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 976 op/s; 11 MiB/s, 172 objects/s recovering 2024-09-17T16:30:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:04 smithi017 ceph-mon[27571]: osdmap e646: 12 total, 12 up, 12 in 2024-09-17T16:30:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:04 smithi071 ceph-mon[30002]: pgmap v1422: 17 pgs: 1 creating+peering, 16 active+clean; 1.4 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 976 op/s; 11 MiB/s, 172 objects/s recovering 2024-09-17T16:30:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:04 smithi071 ceph-mon[30002]: osdmap e646: 12 total, 12 up, 12 in 2024-09-17T16:30:05.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:05.106 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:05.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:05.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:05.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:05.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:05.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:05.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:05.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:05.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:05.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:05.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:05.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:05.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:05.607 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:05.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:05.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:05.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:05.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:05.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:05.743 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:30:05.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:05.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:05.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:05.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:05.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 15 28431 28416 68.3016 78.75 0.00238152 0.0146299 2024-09-17T16:30:05.958 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:05.979 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:05.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:06.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:06.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:06.206 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:06.228 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:06.243 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:06.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:06.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:06.276 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1421,"stamp":"2024-09-17T16:30:01.992590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1458307616,"num_objects":22245,"num_object_clones":0,"num_object_copies":66735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22245,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":22366,"num_write_kb":1425702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2676,"num_bytes_recovered":176170048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22335,"ondisk_log_size":22335,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9899404,"kb_used_data":4541908,"kb_used_omap":58,"kb_used_meta":5357189,"kb_avail":1115042420,"statfs":{"total":1151940427776,"available":1141803438080,"internally_reserved":0,"allocated":4650913792,"data_stored":4646887654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":5485761699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531562496,"num_objects":8111,"num_object_clones":0,"num_object_copies":24333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1873,"num_objects_unfound":0,"num_objects_dirty":8111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":519104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1876,"num_bytes_recovered":122945536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001933"},"pg_stats":[{"pgid":"7.f","version":"645'1433","reported_seq":1476,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:01.278868+0000","last_change":"2024-09-17T16:29:35.616372+0000","last_active":"2024-09-17T16:30:01.278868+0000","last_peered":"2024-09-17T16:30:01.278868+0000","last_clean":"2024-09-17T16:30:01.278868+0000","last_became_active":"2024-09-17T16:29:35.616046+0000","last_became_peered":"2024-09-17T16:29:35.616046+0000","last_unstale":"2024-09-17T16:30:01.278868+0000","last_undegraded":"2024-09-17T16:30:01.278868+0000","last_fullsized":"2024-09-17T16:30:01.278868+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1433,"log_dups_size":0,"ondisk_log_size":1433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:35.871596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93913088,"num_objects":1433,"num_object_clones":0,"num_object_copies":4299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1433,"num_write_kb":91712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"645'1548","reported_seq":1591,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:01.921031+0000","last_change":"2024-09-17T16:29:35.659373+0000","last_active":"2024-09-17T16:30:01.921031+0000","last_peered":"2024-09-17T16:30:01.921031+0000","last_clean":"2024-09-17T16:30:01.921031+0000","last_became_active":"2024-09-17T16:29:35.659021+0000","last_became_peered":"2024-09-17T16:29:35.659021+0000","last_unstale":"2024-09-17T16:30:01.921031+0000","last_undegraded":"2024-09-17T16:30:01.921031+0000","last_fullsized":"2024-09-17T16:30:01.921031+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:43.706367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101449728,"num_objects":1548,"num_object_clones":0,"num_object_copies":4644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1548,"num_write_kb":99072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"643'1160","reported_seq":1199,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-17T16:29:57.282515+0000","last_change":"2024-09-17T16:29:35.618928+0000","last_active":"2024-09-17T16:29:57.282515+0000","last_peered":"2024-09-17T16:29:57.282515+0000","last_clean":"2024-09-17T16:29:57.282515+0000","last_became_active":"2024-09-17T16:29:35.618644+0000","last_became_peered":"2024-09-17T16:29:35.618644+0000","last_unstale":"2024-09-17T16:29:57.282515+0000","last_undegraded":"2024-09-17T16:29:57.282515+0000","last_fullsized":"2024-09-17T16:29:57.282515+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1160,"log_dups_size":0,"ondisk_log_size":1160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:50.763262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76021760,"num_objects":1160,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1160,"num_write_kb":74240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"645'1374","reported_seq":1417,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:00.629773+0000","last_change":"2024-09-17T16:29:35.617815+0000","last_active":"2024-09-17T16:30:00.629773+0000","last_peered":"2024-09-17T16:30:00.629773+0000","last_clean":"2024-09-17T16:30:00.629773+0000","last_became_active":"2024-09-17T16:29:35.617430+0000","last_became_peered":"2024-09-17T16:29:35.617430+0000","last_unstale":"2024-09-17T16:30:00.629773+0000","last_undegraded":"2024-09-17T16:30:00.629773+0000","last_fullsized":"2024-09-17T16:30:00.629773+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1374,"log_dups_size":0,"ondisk_log_size":1374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:48.198410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90046464,"num_objects":1374,"num_object_clones":0,"num_object_copies":4122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1374,"num_write_kb":87936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"643'1168","reported_seq":1207,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-17T16:29:57.283854+0000","last_change":"2024-09-17T16:29:35.686174+0000","last_active":"2024-09-17T16:29:57.283854+0000","last_peered":"2024-09-17T16:29:57.283854+0000","last_clean":"2024-09-17T16:29:57.283854+0000","last_became_active":"2024-09-17T16:29:35.685764+0000","last_became_peered":"2024-09-17T16:29:35.685764+0000","last_unstale":"2024-09-17T16:29:57.283854+0000","last_undegraded":"2024-09-17T16:29:57.283854+0000","last_fullsized":"2024-09-17T16:29:57.283854+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1168,"log_dups_size":0,"ondisk_log_size":1168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:57.545117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76546048,"num_objects":1168,"num_object_clones":0,"num_object_copies":3504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1168,"num_write_kb":74752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"645'1472","reported_seq":1514,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:01.919560+0000","last_change":"2024-09-17T16:29:35.686166+0000","last_active":"2024-09-17T16:30:01.919560+0000","last_peered":"2024-09-17T16:30:01.919560+0000","last_clean":"2024-09-17T16:30:01.919560+0000","last_became_active":"2024-09-17T16:29:35.685807+0000","last_became_peered":"2024-09-17T16:29:35.685807+0000","last_unstale":"2024-09-17T16:30:01.919560+0000","last_undegraded":"2024-09-17T16:30:01.919560+0000","last_fullsized":"2024-09-17T16:30:01.919560+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1472,"log_dups_size":0,"ondisk_log_size":1472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:39.776288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":1472,"num_object_clones":0,"num_object_copies":4416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1472,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,11],"acting":[2,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"645'1445","reported_seq":1488,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:01.250468+0000","last_change":"2024-09-17T16:29:35.659376+0000","last_active":"2024-09-17T16:30:01.250468+0000","last_peered":"2024-09-17T16:30:01.250468+0000","last_clean":"2024-09-17T16:30:01.250468+0000","last_became_active":"2024-09-17T16:29:35.659034+0000","last_became_peered":"2024-09-17T16:29:35.659034+0000","last_unstale":"2024-09-17T16:30:01.250468+0000","last_undegraded":"2024-09-17T16:30:01.250468+0000","last_fullsized":"2024-09-17T16:30:01.250468+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:52.950535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94699520,"num_objects":1445,"num_object_clones":0,"num_object_copies":4335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1445,"num_write_kb":92480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"645'1445","reported_seq":2852,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:01.905502+0000","last_change":"2024-09-17T16:29:56.602948+0000","last_active":"2024-09-17T16:30:01.905502+0000","last_peered":"2024-09-17T16:30:01.905502+0000","last_clean":"2024-09-17T16:30:01.905502+0000","last_became_active":"2024-09-17T16:29:56.600296+0000","last_became_peered":"2024-09-17T16:29:56.600296+0000","last_unstale":"2024-09-17T16:30:01.905502+0000","last_undegraded":"2024-09-17T16:30:01.905502+0000","last_fullsized":"2024-09-17T16:30:01.905502+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:03.960970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94699520,"num_objects":1445,"num_object_clones":0,"num_object_copies":4335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1445,"num_write_kb":92480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":44105728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"643'1313","reported_seq":1352,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-17T16:29:59.487539+0000","last_change":"2024-09-17T16:29:35.616770+0000","last_active":"2024-09-17T16:29:59.487539+0000","last_peered":"2024-09-17T16:29:59.487539+0000","last_clean":"2024-09-17T16:29:59.487539+0000","last_became_active":"2024-09-17T16:29:35.616545+0000","last_became_peered":"2024-09-17T16:29:35.616545+0000","last_unstale":"2024-09-17T16:29:59.487539+0000","last_undegraded":"2024-09-17T16:29:59.487539+0000","last_fullsized":"2024-09-17T16:29:59.487539+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1313,"log_dups_size":0,"ondisk_log_size":1313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:49.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86048768,"num_objects":1313,"num_object_clones":0,"num_object_copies":3939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1313,"num_write_kb":84032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"645'1431","reported_seq":1474,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:01.244384+0000","last_change":"2024-09-17T16:29:35.686052+0000","last_active":"2024-09-17T16:30:01.244384+0000","last_peered":"2024-09-17T16:30:01.244384+0000","last_clean":"2024-09-17T16:30:01.244384+0000","last_became_active":"2024-09-17T16:29:35.685539+0000","last_became_peered":"2024-09-17T16:29:35.685539+0000","last_unstale":"2024-09-17T16:30:01.244384+0000","last_undegraded":"2024-09-17T16:30:01.244384+0000","last_fullsized":"2024-09-17T16:30:01.244384+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1431,"log_dups_size":0,"ondisk_log_size":1431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:40.718377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93782016,"num_objects":1431,"num_object_clones":0,"num_object_copies":4293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1431,"num_write_kb":91584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"645'1445","reported_seq":2871,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:01.254484+0000","last_change":"2024-09-17T16:29:56.616980+0000","last_active":"2024-09-17T16:30:01.254484+0000","last_peered":"2024-09-17T16:30:01.254484+0000","last_clean":"2024-09-17T16:30:01.254484+0000","last_became_active":"2024-09-17T16:29:56.600312+0000","last_became_peered":"2024-09-17T16:29:56.600312+0000","last_unstale":"2024-09-17T16:30:01.254484+0000","last_undegraded":"2024-09-17T16:30:01.254484+0000","last_fullsized":"2024-09-17T16:30:01.254484+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:42.145938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94699520,"num_objects":1445,"num_object_clones":0,"num_object_copies":4335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1445,"num_write_kb":92480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":44695552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"645'1480","reported_seq":2878,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:01.725048+0000","last_change":"2024-09-17T16:29:54.745801+0000","last_active":"2024-09-17T16:30:01.725048+0000","last_peered":"2024-09-17T16:30:01.725048+0000","last_clean":"2024-09-17T16:30:01.725048+0000","last_became_active":"2024-09-17T16:29:54.595605+0000","last_became_peered":"2024-09-17T16:29:54.595605+0000","last_unstale":"2024-09-17T16:30:01.725048+0000","last_undegraded":"2024-09-17T16:30:01.725048+0000","last_fullsized":"2024-09-17T16:30:01.725048+0000","mapping_epoch":640,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:37.374814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96993280,"num_objects":1480,"num_object_clones":0,"num_object_copies":4440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1480,"num_write_kb":94720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"645'1425","reported_seq":1468,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:01.700429+0000","last_change":"2024-09-17T16:29:35.659242+0000","last_active":"2024-09-17T16:30:01.700429+0000","last_peered":"2024-09-17T16:30:01.700429+0000","last_clean":"2024-09-17T16:30:01.700429+0000","last_became_active":"2024-09-17T16:29:35.658918+0000","last_became_peered":"2024-09-17T16:29:35.658918+0000","last_unstale":"2024-09-17T16:30:01.700429+0000","last_undegraded":"2024-09-17T16:30:01.700429+0000","last_fullsized":"2024-09-17T16:30:01.700429+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1425,"log_dups_size":0,"ondisk_log_size":1425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:56.097200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93388800,"num_objects":1425,"num_object_clones":0,"num_object_copies":4275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1425,"num_write_kb":91200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"643'1294","reported_seq":2632,"reported_epoch":644,"state":"creating+peering","last_fresh":"2024-09-17T16:29:59.589500+0000","last_change":"2024-09-17T16:29:59.947910+0000","last_active":"2024-09-17T16:29:59.947910+0000","last_peered":"2024-09-17T16:29:59.589500+0000","last_clean":"2024-09-17T16:29:47.846982+0000","last_became_active":"2024-09-17T16:29:49.890448+0000","last_became_peered":"2024-09-17T16:29:49.890448+0000","last_unstale":"2024-09-17T16:29:59.589500+0000","last_undegraded":"2024-09-17T16:29:59.589500+0000","last_fullsized":"2024-09-17T16:29:59.589500+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1294,"log_dups_size":0,"ondisk_log_size":1294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84803584,"num_objects":1294,"num_object_clones":0,"num_object_copies":3882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1294,"num_write_kb":82816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":42598400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"645'1381","reported_seq":1424,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:00.670426+0000","last_change":"2024-09-17T16:29:35.617588+0000","last_active":"2024-09-17T16:30:00.670426+0000","last_peered":"2024-09-17T16:30:00.670426+0000","last_clean":"2024-09-17T16:30:00.670426+0000","last_became_active":"2024-09-17T16:29:35.617197+0000","last_became_peered":"2024-09-17T16:29:35.617197+0000","last_unstale":"2024-09-17T16:30:00.670426+0000","last_undegraded":"2024-09-17T16:30:00.670426+0000","last_fullsized":"2024-09-17T16:30:00.670426+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1381,"log_dups_size":0,"ondisk_log_size":1381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:46.697820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90505216,"num_objects":1381,"num_object_clones":0,"num_object_copies":4143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1381,"num_write_kb":88384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"645'1429","reported_seq":1472,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-09-17T16:30:01.407059+0000","last_change":"2024-09-17T16:29:35.686727+0000","last_active":"2024-09-17T16:30:01.407059+0000","last_peered":"2024-09-17T16:30:01.407059+0000","last_clean":"2024-09-17T16:30:01.407059+0000","last_became_active":"2024-09-17T16:29:35.686233+0000","last_became_peered":"2024-09-17T16:29:35.686233+0000","last_unstale":"2024-09-17T16:30:01.407059+0000","last_undegraded":"2024-09-17T16:30:01.407059+0000","last_fullsized":"2024-09-17T16:30:01.407059+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":1429,"log_dups_size":0,"ondisk_log_size":1429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:36.781435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93650944,"num_objects":1429,"num_object_clones":0,"num_object_copies":4287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1429,"num_write_kb":91456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":32,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-17T16:29:59.589249+0000","last_change":"2024-09-17T16:29:49.984493+0000","last_active":"2024-09-17T16:29:59.589249+0000","last_peered":"2024-09-17T16:29:59.589249+0000","last_clean":"2024-09-17T16:29:59.589249+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:29:59.589249+0000","last_undegraded":"2024-09-17T16:29:59.589249+0000","last_fullsized":"2024-09-17T16:29:59.589249+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:29:40.527128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:29:40.527128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:10.179932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1457717248,"num_objects":22243,"num_object_clones":0,"num_object_copies":66729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22243,"num_write_kb":1423552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2672,"num_bytes_recovered":175112192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4507041792,"data_stored":4507041792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22243,"ondisk_log_size":22243,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285814,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681912,"kb_used_data":293924,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93063240,"statfs":{"total":95995035648,"available":95296757760,"internally_reserved":0,"allocated":300978176,"data_stored":300643915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":397277636},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547451,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846920,"kb_used_data":440880,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92898232,"statfs":{"total":95995035648,"available":95127789568,"internally_reserved":0,"allocated":451461120,"data_stored":451122372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":415758784},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743680,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1012404,"kb_used_data":439780,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92732748,"statfs":{"total":95995035648,"available":94958333952,"internally_reserved":0,"allocated":450334720,"data_stored":450000479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":586348992},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005317,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":625456,"kb_used_data":255520,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93119696,"statfs":{"total":95995035648,"available":95354568704,"internally_reserved":0,"allocated":261652480,"data_stored":261319173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963793,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":853868,"kb_used_data":386136,"kb_used_omap":41,"kb_used_meta":467670,"kb_avail":92891284,"statfs":{"total":95995035648,"available":95120674816,"internally_reserved":0,"allocated":395403264,"data_stored":395073509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":478894387},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495887,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":512960,"kb_used_data":88876,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93232192,"statfs":{"total":95995035648,"available":95469764608,"internally_reserved":0,"allocated":91009024,"data_stored":90671231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":434239942},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":597624,"kb_used_data":88868,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93147528,"statfs":{"total":95995035648,"available":95383068672,"internally_reserved":0,"allocated":91000832,"data_stored":90666591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986458,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781824,"kb_used_data":345260,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92963328,"statfs":{"total":95995035648,"available":95194447872,"internally_reserved":0,"allocated":353546240,"data_stored":353208447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149981,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1333128,"kb_used_data":760496,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92412024,"statfs":{"total":95995035648,"available":94629912576,"internally_reserved":0,"allocated":778747904,"data_stored":778409156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":586348999},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411619,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891268,"kb_used_data":544944,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92853884,"statfs":{"total":95995035648,"available":95082377216,"internally_reserved":0,"allocated":558022656,"data_stored":557683908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575144,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696112,"kb_used_data":358104,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93049040,"statfs":{"total":95995035648,"available":95282216960,"internally_reserved":0,"allocated":366698496,"data_stored":366368741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1065928,"kb_used_data":539120,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92679224,"statfs":{"total":95995035648,"available":94903525376,"internally_reserved":0,"allocated":552058880,"data_stored":551720132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":563150848,"data_stored":563150848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":381681664,"data_stored":381681664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":584712192,"data_stored":584712192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":787480576,"data_stored":787480576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352321536,"data_stored":352321536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":90505216,"data_stored":90505216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":90046464,"data_stored":90046464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":160235520,"data_stored":160235520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":263323648,"data_stored":263323648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":472645632,"data_stored":472645632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":460718080,"data_stored":460718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":300220416,"data_stored":300220416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:30:06.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:06.367 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:06.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:06.469 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:06.497 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:06.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:06.599 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:06.625 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:06.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:06.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:06.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:06.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:06.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:06.872 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:06.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:06.931 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:06.931 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:06.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 29602 29586 68.4801 73.125 0.0156191 0.0145948 2024-09-17T16:30:06.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:07.007 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:07.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:07.109 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:07.132 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:07 smithi017 ceph-mon[27571]: pgmap v1424: 17 pgs: 17 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-09-17T16:30:07.138 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:07.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:07.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:07.240 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:07.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:07.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:07.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:07 smithi071 ceph-mon[30002]: pgmap v1424: 17 pgs: 17 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-09-17T16:30:07.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:07.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:07.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:07.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:07 smithi173 ceph-mon[30530]: pgmap v1424: 17 pgs: 17 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-09-17T16:30:07.487 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:07.494 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:30:07.494 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:30:07.494 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:30:07.494 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:30:07.494 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:30:07.511 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:07.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:07.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:07.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:07.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:07.745 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:07.779 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:07.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:07.881 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:07.903 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:07.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:07.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 30702 30686 68.4894 68.75 0.00204707 0.0145818 2024-09-17T16:30:08.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:08.041 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:08.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:08.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:08.062 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:30:08.142 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:08.165 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:08.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/842250641' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:08 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:08.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:08.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:08.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:08.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/842250641' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:08 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:08.393 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:08.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/842250641' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:08.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:08 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:08.415 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:08.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:08.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:08.540 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:08.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:08.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:08.704 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:08.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:08.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:08.832 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:08.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:08.934 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:08.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 32186 32170 69.3257 92.75 0.0170761 0.0144147 2024-09-17T16:30:08.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:08.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:09.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:09.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:09.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:09.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:09.217 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:09.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:09 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:09 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:09 smithi071 ceph-mon[30002]: pgmap v1425: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-17T16:30:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:09 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:09 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:30:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:09 smithi071 ceph-mon[30002]: osdmap e647: 12 total, 12 up, 12 in 2024-09-17T16:30:09.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:09.340 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:09.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:09.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:09 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:09 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:09 smithi173 ceph-mon[30530]: pgmap v1425: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-17T16:30:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:09 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:09 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:30:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:09 smithi173 ceph-mon[30530]: osdmap e647: 12 total, 12 up, 12 in 2024-09-17T16:30:09.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:09 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:09.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:09 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:09.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:09 smithi017 ceph-mon[27571]: pgmap v1425: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-09-17T16:30:09.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:09 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:09.415 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:09 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:30:09.415 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:09 smithi017 ceph-mon[27571]: osdmap e647: 12 total, 12 up, 12 in 2024-09-17T16:30:09.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:09.469 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:09.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:09.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:09.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:09.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:09.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:09.721 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:09.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:09.735 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:30:09.735 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:30:09.757 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:30:09.757 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:30:09.779 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:30:09.779 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:30:09.800 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:30:09.800 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:30:09.822 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:30:09.822 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:30:09.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:09.843 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:30:09.843 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:30:09.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:09.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:09.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:09.864 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:30:09.865 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:30:09.886 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:30:09.886 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:30:09.910 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:30:09.910 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:30:09.933 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:30:09.933 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:30:09.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 33221 33205 69.1709 64.6875 0.00209154 0.0144034 2024-09-17T16:30:09.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:09.955 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:30:09.955 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:30:09.967 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:09.968 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:09.981 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:30:09.981 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:30:10.069 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:10.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:10.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:10.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:10.217 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:10.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:10.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:10.319 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:10.341 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:10.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:10.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:30:10.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:10.466 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:10.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:10.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:10.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:10.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:10.695 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:10.725 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:10.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:10.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:10.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:10.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:10 smithi173 ceph-mon[30530]: osdmap e648: 12 total, 12 up, 12 in 2024-09-17T16:30:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/750450088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:10 smithi173 ceph-mon[30530]: pgmap v1428: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:30:10.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 34401 34385 69.3184 73.75 0.00835168 0.0144198 2024-09-17T16:30:10.954 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:10.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:10 smithi017 ceph-mon[27571]: osdmap e648: 12 total, 12 up, 12 in 2024-09-17T16:30:10.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/750450088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:10.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:10 smithi017 ceph-mon[27571]: pgmap v1428: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:30:10.987 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:10.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:11.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:10 smithi071 ceph-mon[30002]: osdmap e648: 12 total, 12 up, 12 in 2024-09-17T16:30:11.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/750450088' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:11.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:10 smithi071 ceph-mon[30002]: pgmap v1428: 17 pgs: 17 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:30:11.089 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:11.100 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:11.111 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:11.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:11.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:11.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:11.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:11.281 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:30:11.281 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:30:11.281 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-09-17T16:30:11.281 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T16:30:11.281 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T16:30:11.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-09-17T16:30:11.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:11.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:11.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:11.460 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:30:11.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:11.497 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:11.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:11.585 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:30:11.599 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:11.620 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:11.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:11.675 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:11.675 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:11.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:11.743 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:11.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:11.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:11.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:11.867 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:11.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:11.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 35495 35479 69.2888 68.375 0.0132901 0.0144229 2024-09-17T16:30:11.969 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:12.003 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:12.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:12.095 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:30:12.095 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T16:30:12.095 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:30:12.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:12.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:12.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:12.141 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:12.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:12.264 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:12.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:12.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:12.393 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:12.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:12.494 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:30:12.494 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:12.522 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:12.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:12.616 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:12.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:12.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:12.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:12.754 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:12.778 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:12.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:12.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:12.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:12.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:12.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 36460 36444 69.0166 60.3125 0.00242732 0.0144211 2024-09-17T16:30:13.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:13.008 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:30:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:13.028 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:13.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:13.130 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:13.151 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:13.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:13.218 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:13.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:13.310 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:13.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:13.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:13 smithi071 ceph-mon[30002]: pgmap v1429: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-17T16:30:13.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:13 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:30:13.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:13 smithi017 ceph-mon[27571]: pgmap v1429: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-17T16:30:13.356 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:13 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:30:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:13 smithi173 ceph-mon[30530]: pgmap v1429: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-17T16:30:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:13 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:30:13.412 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:13.434 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:13.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:13.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:13.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:13.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:13.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:13.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:13.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:13.799 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:13.821 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:13.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:13.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:13.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 37877 37861 69.5913 88.5625 0.0507838 0.0143469 2024-09-17T16:30:13.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:13.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:14.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:14.058 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:30:14.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:14.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:14.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:14.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:14.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:14.302 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:14.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/88524957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:30:14.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:14 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:30:14.329 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:14.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:14.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/88524957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:30:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:14 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:30:14.430 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:14.453 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:14.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/88524957' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:30:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:14 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:30:14.555 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:14.576 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:14.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:14.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:14.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:14.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:14.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:14.830 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:14.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:14.933 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:14.943 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 39263 39247 70.0778 86.625 0.0102646 0.0142619 2024-09-17T16:30:14.966 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:14.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:14.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:15.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:15.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:15.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:15.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:15.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:15.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:15 smithi071 ceph-mon[30002]: pgmap v1430: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-17T16:30:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:15 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:15 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:30:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:15 smithi071 ceph-mon[30002]: osdmap e649: 12 total, 12 up, 12 in 2024-09-17T16:30:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:15 smithi071 ceph-mon[30002]: osdmap e650: 12 total, 12 up, 12 in 2024-09-17T16:30:15.324 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:15.357 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:15.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:15 smithi173 ceph-mon[30530]: pgmap v1430: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-17T16:30:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:15 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:15 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:30:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:15 smithi173 ceph-mon[30530]: osdmap e649: 12 total, 12 up, 12 in 2024-09-17T16:30:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:15 smithi173 ceph-mon[30530]: osdmap e650: 12 total, 12 up, 12 in 2024-09-17T16:30:15.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:15.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:15.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:15 smithi017 ceph-mon[27571]: pgmap v1430: 17 pgs: 17 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-17T16:30:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:15 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:15 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:30:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:15 smithi017 ceph-mon[27571]: osdmap e649: 12 total, 12 up, 12 in 2024-09-17T16:30:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:15 smithi017 ceph-mon[27571]: osdmap e650: 12 total, 12 up, 12 in 2024-09-17T16:30:15.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:15.609 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:15.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:15.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:15.734 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:15.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:15.835 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:15.857 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:15.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:15.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 40540 40524 70.348 79.8125 0.00598179 0.014208 2024-09-17T16:30:15.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:15.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:15.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:16.093 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:16.114 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:16.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:16.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:16.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:16.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:16.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:16.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:16.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:16.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:16.490 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:16.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:16.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:16.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:16.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:16.716 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:16.737 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:16.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:16.839 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:16.860 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:16.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:16.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 41326 41310 69.7742 49.125 0.00296006 0.0142145 2024-09-17T16:30:16.962 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:16.985 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:16.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:17.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:17.109 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:17.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:17.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:17.253 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:17.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:17.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:17 smithi017 ceph-mon[27571]: pgmap v1433: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-17T16:30:17.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:30:17.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:30:17.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:17 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:30:17.355 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:17.378 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:17.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:17.380 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:17.380 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:17 smithi173 ceph-mon[30530]: pgmap v1433: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-17T16:30:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:17 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:30:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:17 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:30:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:17 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:30:17.480 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:17.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:17.517 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:17.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:17 smithi071 ceph-mon[30002]: pgmap v1433: 17 pgs: 17 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-17T16:30:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:17 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:30:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:17 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:30:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:17 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:30:17.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:17.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:17.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:17.652 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:17.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:17.766 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:17.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:17.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:17.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:17.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:17.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 42633 42617 70.0876 81.6875 0.00201196 0.0142584 2024-09-17T16:30:17.951 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:30:17.951 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:30:17.951 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T16:30:17.952 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:30:17.993 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:18.024 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:18.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:18.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:18.126 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:18.161 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:18.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:18.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:30:18.263 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:18.287 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:18.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:18.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:18.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:18.411 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:18.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:18.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:18.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:18.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:18.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:18.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:18.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:18.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:18.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:18.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:18.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:18.813 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:18.830 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:30:18.830 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:30:18.876 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:30:18.877 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:30:18.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:18.911 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:30:18.912 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:30:18.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:18.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:18.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 43959 43943 70.4153 82.875 0.00463337 0.014196 2024-09-17T16:30:18.948 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:30:18.949 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:30:18.980 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:30:18.981 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:30:19.012 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:30:19.013 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:30:19.021 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:19.042 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:19.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:19.045 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:30:19.046 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:30:19.053 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:30:19.082 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:30:19.082 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:30:19.118 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:30:19.118 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:30:19.144 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:19.157 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:30:19.157 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:30:19.165 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:19.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:19.196 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:30:19.196 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:30:19.215 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:19.267 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:19.309 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:19.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:19.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:19 smithi017 ceph-mon[27571]: pgmap v1434: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-17T16:30:19.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2812489045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:19.363 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:30:19.363 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:30:19.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:19 smithi173 ceph-mon[30530]: pgmap v1434: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-17T16:30:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2812489045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:19.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:19.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:19.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:19.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:19.534 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:19.559 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:19.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:19.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:19 smithi071 ceph-mon[30002]: pgmap v1434: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-17T16:30:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2812489045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:19.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:19.683 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:19.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:19.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:19.806 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:19.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:19.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:19.931 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:19.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:19.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:19.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:30:19.943124+0000 min lat: 0.00119669 max lat: 2.03842 avg lat: 0.0142339 2024-09-17T16:30:19.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:30:19.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 44964 44948 70.2251 62.8125 0.0164086 0.0142339 2024-09-17T16:30:20.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:20.063 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:20.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:20.165 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:20.186 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:20.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:20.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:20.288 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:20.311 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:20.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:20.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4029329106' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:30:20.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:20.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:20.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4029329106' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:30:20.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:20.545 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:30:20.545 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:30:20.545 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 650, 2024-09-17T16:30:20.545 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:30:20.545 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:30:20.545 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:30:14.590979+0000", 2024-09-17T16:30:20.545 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:30:20.545 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:29:47.857157+0000", 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5804032, 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "nodeep-scrub", 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:30:20.546 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 7, 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:30:20.547 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:30:20.548 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:30:20.549 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "446", 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:30:20.550 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:30:20.551 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:30:20.552 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:30:20.553 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 7, 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_5", 2024-09-17T16:30:20.554 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:29:33.619938+0000", 2024-09-17T16:30:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:30:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:30:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:30:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:30:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:30:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:30:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:30:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:30:20.555 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 16, 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 16, 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 16, 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 16, 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 16, 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:30:20.556 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "632", 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:30:20.557 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:30:20.558 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:30:20.559 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 2.75, 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.75, 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1.0900000333786011, 2024-09-17T16:30:20.560 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.561 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 642, 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:30:20.562 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.563 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.564 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:30:20.566 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.567 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 643, 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:30:20.568 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.569 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.570 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:30:20.572 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.573 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 642, 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:30:20.574 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:30:20.575 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.576 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.577 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:30:20.578 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.579 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 640, 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.580 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.581 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.582 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.583 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 644, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.585 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.587 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:30:20.588 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.589 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.589 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.589 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.589 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:30:20.592 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.593 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.845672607421875, 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 629, 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:30:20.594 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.595 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.596 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:30:20.598 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.66790771484375, 2024-09-17T16:30:20.599 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:30:20.600 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.600 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.600 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:30:20.600 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 629, 2024-09-17T16:30:20.600 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.600 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.600 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.600 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.601 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.602 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.603 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.604 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.605 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.8061676025390625, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.73492431640625, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 633, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.606 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.607 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:30:20.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:30:20.610 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 629, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.612 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.615 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:30:20.616 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:30:20.617 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.2821197509765625, 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 629, 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.619 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.620 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.621 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.622 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.623 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 633, 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.624 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.625 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.628 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:30:20.629 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:30:20.629 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:30:20.629 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:30:20.629 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.629 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.629 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.629 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.629 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.588836669921875, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 291, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:30:20.630 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.632 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.633 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:30:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:30:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:30:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:30:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:30:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.634 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:30:20.635 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:30:20.635 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:30:20.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.635 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.635 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:30:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:30:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:30:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:30:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:30:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:30:20.636 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:30:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:30:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:30:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.637 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:30:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:30:07.802347+0000", 2024-09-17T16:30:20.638 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:30:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.639 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:24.007677+0000", 2024-09-17T16:30:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.640 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:30:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:43.220359+0000", 2024-09-17T16:30:20.641 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:30:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.642 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:14.245748+0000", 2024-09-17T16:30:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:30:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.644 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:44.363440+0000", 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:14.995509+0000", 2024-09-17T16:30:20.645 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:10.274208+0000", 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.646 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:31.706687+0000", 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.647 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:16.204396+0000", 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.648 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:20.398457+0000", 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:20.405370+0000", 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:29:19.601397+0000", 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [ 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "1.0", 2024-09-17T16:30:20.650 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: 4, 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: 6, 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: 8 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:30:20.651 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:30:20.652 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:30:20.653 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:30:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:30:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:30:20.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:20.654 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:30:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:20.655 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:20.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:20.656 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4029329106' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:30:20.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:20.823 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:20.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:20.924 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:20.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 46220 46204 70.4267 78.5 0.0824453 0.0141869 2024-09-17T16:30:20.946 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:20.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:20.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:21.049 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:21.077 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:21.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:21.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:21.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:21.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:21.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:21.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:21.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:21.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:21.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:21 smithi173 ceph-mon[30530]: pgmap v1435: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 924 op/s 2024-09-17T16:30:21.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:21.459 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:21.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:21.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:21 smithi017 ceph-mon[27571]: pgmap v1435: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 924 op/s 2024-09-17T16:30:21.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:21.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:21 smithi071 ceph-mon[30002]: pgmap v1435: 17 pgs: 17 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 924 op/s 2024-09-17T16:30:21.599 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:21.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:21.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:21.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:21.725 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:21.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:21.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:21.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:21.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:21.858 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:21.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 15 47226 47211 70.2483 62.9375 0.00455221 0.0142281 2024-09-17T16:30:21.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:21.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:21.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:22.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:22.109 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:22.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:22.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:22.253 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:22.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:22.355 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:22.376 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:22.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:22.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:22.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:22.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:22.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:22.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:22.624 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:22.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:22.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:22.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:22.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:22.849 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:22.881 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:22.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:22.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 48421 48405 70.35 74.625 0.00455512 0.0142089 2024-09-17T16:30:22.982 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:23.012 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:23.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:23.065 INFO:tasks.thrashosds.thrasher:j is {'epoch': 650, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:30:14.590979+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:29:47.857157+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '446', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-09-17T16:29:33.619938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '632', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.75, 'score_stable': 2.75, 'optimal_score': 1.090000033378601, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 642, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 643, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 642, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 640, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 644, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 0.845672607421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 629, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 0.66790771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 629, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 0.8061676025390625, 'primary_affinity': 0.73492431640625, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 633, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 629, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0.2821197509765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 629, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 633, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 0.588836669921875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:30:07.802347+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:24.007677+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:43.220359+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:14.245748+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:44.363440+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:14.995509+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:10.274208+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:31.706687+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:16.204396+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:20.398457+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:20.405370+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:29:19.601397+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [4, 6, 8]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:30:23.066 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:30:23.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:23.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:23.138 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:23.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:23.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:23.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:23.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:23.386 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:23.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:23.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:23 smithi173 ceph-mon[30530]: pgmap v1436: 17 pgs: 17 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-17T16:30:23.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:23 smithi017 ceph-mon[27571]: pgmap v1436: 17 pgs: 17 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-17T16:30:23.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:23.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:23.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:23.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:23 smithi071 ceph-mon[30002]: pgmap v1436: 17 pgs: 17 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-17T16:30:23.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:23.653 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:23.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:23.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:23.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:23.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:23.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:23.892 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:30:23.893 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:30:23.893 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:30:23.893 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:30:23.893 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:30:23.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:23.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:23.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 49595 49579 70.4186 73.375 0.00559172 0.0141938 2024-09-17T16:30:24.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:24.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:24.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:24.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:24.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:24.160 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:24.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:24.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:24.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:24.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:24.288 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:24.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:24.412 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:24.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:24.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:24.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:24.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:24.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:24.664 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:24.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:24.766 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:24.801 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:24.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:24.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:24.926 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:24.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:24.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 15 51075 51060 70.9105 92.5625 0.00351546 0.014097 2024-09-17T16:30:25.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:25.063 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:25.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:25.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:25.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:25.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:25.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:25 smithi017 ceph-mon[27571]: pgmap v1437: 17 pgs: 17 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-17T16:30:25.252 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:25.295 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:25.302 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:25.304 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:30:25.316 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:25.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:25.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:25 smithi173 ceph-mon[30530]: pgmap v1437: 17 pgs: 17 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-17T16:30:25.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:25.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:25.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:25.546 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:25.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:25 smithi071 ceph-mon[30002]: pgmap v1437: 17 pgs: 17 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-17T16:30:25.593 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:25.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:25.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:25.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:25.718 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:25.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:25.820 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:25.856 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:25.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:25.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 52477 52461 71.2723 87.5625 0.00327872 0.0140245 2024-09-17T16:30:25.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:25.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:25.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:26.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:26.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:26.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:26.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:26.252 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:26.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:26.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:26.375 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:26.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:26.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:26.411 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:26.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:26.514 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:26.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:26.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:26.516 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:30:26.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:26.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:26.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:26.648 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:30:26.648 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:30:26.740 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:26.763 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:26.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:26.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:26.901 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:26.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:26.944 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 53582 53566 71.2252 69.0625 0.00631402 0.0140335 2024-09-17T16:30:27.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:27.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:27.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:27.093 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:30:27.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:27.165 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:27.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:27.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:27.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:27.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:27 smithi173 ceph-mon[30530]: pgmap v1438: 17 pgs: 17 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-17T16:30:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1893158167' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:27 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:27.396 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:27.418 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:27.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:27.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:27 smithi017 ceph-mon[27571]: pgmap v1438: 17 pgs: 17 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-17T16:30:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1893158167' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:27 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:27.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:27.548 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:27.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:27 smithi071 ceph-mon[30002]: pgmap v1438: 17 pgs: 17 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-17T16:30:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1893158167' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:27 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:27.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:27.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:27.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:27.773 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:27.797 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:27.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:27.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:27.924 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:27.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:27.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 54572 54556 71.0303 61.875 0.00375781 0.0140683 2024-09-17T16:30:28.027 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:28.049 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:28.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:28.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:28.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:28.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:28.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:28.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:28.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:28.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:28.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:28 smithi173 ceph-mon[30530]: from='client.27155 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:28.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:28 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:30:28.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:28 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:30:28.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:28 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:30:28.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:28 smithi173 ceph-mon[30530]: osdmap e651: 12 total, 12 up, 12 in 2024-09-17T16:30:28.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:28.429 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:28.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:28 smithi017 ceph-mon[27571]: from='client.27155 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:28 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:30:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:28 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:30:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:28 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:30:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:28 smithi017 ceph-mon[27571]: osdmap e651: 12 total, 12 up, 12 in 2024-09-17T16:30:28.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:28 smithi071 ceph-mon[30002]: from='client.27155 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:28 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:30:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:28 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:30:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:28 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:30:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:28 smithi071 ceph-mon[30002]: osdmap e651: 12 total, 12 up, 12 in 2024-09-17T16:30:28.566 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:28.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:28.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:28.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:28.689 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:28.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:28.791 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:28.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:28.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:28.922 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:28.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 55464 55448 70.7184 55.75 0.00291185 0.0141276 2024-09-17T16:30:28.948 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:28.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:29.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:29.072 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:29.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:29.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:29.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:29.210 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:29.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:29.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:29.336 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:29.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:29.405 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:29 smithi017 ceph-mon[27571]: pgmap v1440: 17 pgs: 17 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-17T16:30:29.438 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:29.469 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:29.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:29 smithi071 ceph-mon[30002]: pgmap v1440: 17 pgs: 17 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-17T16:30:29.571 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:29.606 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:29.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:29.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:29 smithi173 ceph-mon[30530]: pgmap v1440: 17 pgs: 17 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-17T16:30:29.696 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:30:29.696 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:30:29.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:30:29.696 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:30:29.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.696 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.25516.0:204335 4.9 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-17T16:30:29.696 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:30:29.696 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 546.08453974999998, 2024-09-17T16:30:29.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77672725300000001, 2024-09-17T16:30:29.696 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.25516", 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 204335 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606691+0000", 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606692+0000", 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2600000000000002e-07 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606696+0000", 2024-09-17T16:30:29.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7170000000000002e-06 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606707+0000", 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1486e-05 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606712+0000", 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1560000000000003e-06 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606769+0000", 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6836000000000003e-05 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606856+0000", 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6749999999999994e-05 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-17T16:30:29.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.606942+0000", 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6001999999999994e-05 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611141+0000", 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041987810000000004 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611206+0000", 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5581999999999997e-05 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611481+0000", 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00027492400000000001 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:23.611502+0000", 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0971999999999998e-05 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383271+0000", 2024-09-17T16:30:29.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77176877600000005 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383334+0000", 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2928000000000005e-05 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:21:24.383419+0000", 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4617000000000007e-05 2024-09-17T16:30:29.703 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17641 6.7 6.aff5e7f7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 307.56987589599999, 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86676560000000002, 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.704 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17641 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121355+0000", 2024-09-17T16:30:29.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121357+0000", 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4020000000000001e-06 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121396+0000", 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9561999999999998e-05 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121410+0000", 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3645000000000001e-05 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121415+0000", 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6169999999999998e-06 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121465+0000", 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9648000000000003e-05 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:30:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.121474+0000", 2024-09-17T16:30:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.568e-06 2024-09-17T16:30:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988093+0000", 2024-09-17T16:30:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.86661935700000003 2024-09-17T16:30:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988121+0000", 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7801e-05 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17642 6.7 6.5a462497 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 307.49709026599999, 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79401586899999999, 2024-09-17T16:30:29.709 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17642 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194141+0000", 2024-09-17T16:30:29.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194142+0000", 2024-09-17T16:30:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2399999999999997e-07 2024-09-17T16:30:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194163+0000", 2024-09-17T16:30:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1841e-05 2024-09-17T16:30:29.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194175+0000", 2024-09-17T16:30:29.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1487e-05 2024-09-17T16:30:29.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194180+0000", 2024-09-17T16:30:29.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2639999999999999e-06 2024-09-17T16:30:29.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194227+0000", 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6675000000000003e-05 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.194233+0000", 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1229999999999997e-06 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988148+0000", 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79391546800000001 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.988157+0000", 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1869999999999997e-06 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90607 6.c 6.63d0154c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:33.678865+0000", 2024-09-17T16:30:29.715 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 236.01236588, 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.84340117199999998, 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90607 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.716 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678865+0000", 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678865+0000", 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678866+0000", 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3600000000000003e-07 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678922+0000", 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6008999999999999e-05 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678931+0000", 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1060000000000006e-06 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678935+0000", 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2250000000000002e-06 2024-09-17T16:30:29.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678988+0000", 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2604e-05 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678995+0000", 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2230000000000001e-06 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.522239+0000", 2024-09-17T16:30:29.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.84324350999999997 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.522266+0000", 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7759e-05 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115483 6.b 6:d60f373c:::benchmark_data_smithi173_102017_object22335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.040530+0000", 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.650700637, 2024-09-17T16:30:29.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.027404495000001, 2024-09-17T16:30:29.721 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.721 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.721 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.721 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.721 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.721 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.721 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115483 2024-09-17T16:30:29.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.721 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040530+0000", 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040530+0000", 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040531+0000", 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5000000000000002e-07 2024-09-17T16:30:29.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040535+0000", 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6260000000000001e-06 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040545+0000", 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0001e-05 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040551+0000", 2024-09-17T16:30:29.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2210000000000002e-06 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.044294+0000", 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037428040000000002 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.044410+0000", 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011616999999999999 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.044515+0000", 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104713 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.052307+0000", 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0077919399999999998 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.054367+0000", 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020602149999999998 2024-09-17T16:30:29.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067817+0000", 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.013449883 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067881+0000", 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4331000000000005e-05 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067935+0000", 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3940999999999997e-05 2024-09-17T16:30:29.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115488 6.b 6:d53f7bd0:::benchmark_data_smithi173_102017_object22340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.048310+0000", 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.64292151800001, 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.019735976, 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.727 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115488 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048310+0000", 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048310+0000", 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048311+0000", 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.393e-06 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048326+0000", 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5081e-05 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048336+0000", 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7140000000000005e-06 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048342+0000", 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4110000000000001e-06 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048387+0000", 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5049000000000002e-05 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048484+0000", 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6362999999999999e-05 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048584+0000", 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100679 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.052398+0000", 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038139070000000001 2024-09-17T16:30:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.054404+0000", 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002006056 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067973+0000", 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.013568521 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.068008+0000", 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5404000000000002e-05 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.068046+0000", 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7397999999999997e-05 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115496 6.b 6:d757f72e:::benchmark_data_smithi173_102017_object22348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.060444+0000", 2024-09-17T16:30:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.63078671599999, 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.011022887999999, 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115496 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060444+0000", 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060444+0000", 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060446+0000", 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6589999999999999e-06 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060451+0000", 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.429e-06 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060462+0000", 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0485999999999999e-05 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060469+0000", 2024-09-17T16:30:29.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8759999999999999e-06 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060543+0000", 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4586999999999996e-05 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060653+0000", 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000109873 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060753+0000", 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100114 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.064621+0000", 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038672899999999998 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.075055+0000", 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010434218 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071408+0000", 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.99635303 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071443+0000", 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4671000000000001e-05 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071467+0000", 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4655e-05 2024-09-17T16:30:29.739 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115501 6.b 6:d2a1b698:::benchmark_data_smithi173_102017_object22353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.067982+0000", 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.62324878199999, 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.003590427000001, 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.740 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.741 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.741 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115501 2024-09-17T16:30:29.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.741 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067982+0000", 2024-09-17T16:30:29.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.742 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067982+0000", 2024-09-17T16:30:29.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.742 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067984+0000", 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067989+0000", 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5439999999999998e-06 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068001+0000", 2024-09-17T16:30:29.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1437000000000001e-05 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068008+0000", 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2629999999999997e-06 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068575+0000", 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00056690400000000002 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068713+0000", 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013839599999999999 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068972+0000", 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025855700000000003 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.075175+0000", 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0062034830000000001 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.080658+0000", 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054827449999999998 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071502+0000", 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.990844352 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071534+0000", 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2150000000000002e-05 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071573+0000", 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8297999999999998e-05 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.747 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115511 6.b 6:dffba458:::benchmark_data_smithi173_102017_object22363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.086508+0000", 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.604722968, 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.993681085, 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115511 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086508+0000", 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086508+0000", 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086510+0000", 2024-09-17T16:30:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1509999999999998e-06 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086519+0000", 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1570000000000006e-06 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086535+0000", 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5914e-05 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086544+0000", 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6670000000000002e-06 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086616+0000", 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1768999999999997e-05 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086756+0000", 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013999699999999999 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086877+0000", 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000121729 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092902+0000", 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060247219999999997 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092914+0000", 2024-09-17T16:30:29.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1815e-05 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080089+0000", 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.987174871000001 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080145+0000", 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5692000000000001e-05 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080189+0000", 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4601000000000002e-05 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115517 6.b 6:dccfd4a2:::benchmark_data_smithi173_102017_object22369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.092979+0000", 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.598251952, 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.987374223, 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.754 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115517 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092979+0000", 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092979+0000", 2024-09-17T16:30:29.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092980+0000", 2024-09-17T16:30:29.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.82e-07 2024-09-17T16:30:29.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092982+0000", 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5509999999999998e-06 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092990+0000", 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8089999999999993e-06 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092995+0000", 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2460000000000003e-06 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.093046+0000", 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0569000000000003e-05 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.093179+0000", 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013340499999999999 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.093304+0000", 2024-09-17T16:30:29.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012481600000000001 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.097499+0000", 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041949659999999996 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.100969+0000", 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034705370000000001 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080244+0000", 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.979274189 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080301+0000", 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7593000000000002e-05 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080353+0000", 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.206e-05 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.760 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115529 6.b 6:d3cef629:::benchmark_data_smithi173_102017_object22381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.122349+0000", 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.56888183699999, 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.959155643999999, 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.761 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115529 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122349+0000", 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122349+0000", 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122351+0000", 2024-09-17T16:30:29.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9599999999999999e-06 2024-09-17T16:30:29.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122358+0000", 2024-09-17T16:30:29.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1400000000000002e-06 2024-09-17T16:30:29.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122371+0000", 2024-09-17T16:30:29.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2791e-05 2024-09-17T16:30:29.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122379+0000", 2024-09-17T16:30:29.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8450000000000003e-06 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122456+0000", 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7334000000000002e-05 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122594+0000", 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013791399999999999 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122736+0000", 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014201799999999999 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.126464+0000", 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037274119999999998 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.126918+0000", 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045421399999999998 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.081386+0000", 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.954468022 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.081467+0000", 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0822999999999995e-05 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.081505+0000", 2024-09-17T16:30:29.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8170999999999998e-05 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115584 6.b 6:d2471696:::benchmark_data_smithi173_102017_object22436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.222584+0000", 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.46864741300001, 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.861712117, 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.768 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115584 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222584+0000", 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222584+0000", 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222585+0000", 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222590+0000", 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:30:29.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222599+0000", 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0569999999999999e-06 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222606+0000", 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0040000000000002e-06 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222665+0000", 2024-09-17T16:30:29.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8761999999999997e-05 2024-09-17T16:30:29.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222796+0000", 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131798 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222915+0000", 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011872800000000001 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226472+0000", 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003557218 2024-09-17T16:30:29.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226919+0000", 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00044620499999999997 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084197+0000", 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.857278443 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084256+0000", 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8581e-05 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084296+0000", 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0209000000000001e-05 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115586 6.b 6:defa5317:::benchmark_data_smithi173_102017_object22438:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.776 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.226877+0000", 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.46435400600001, 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.857535156000001, 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115586 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226877+0000", 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226877+0000", 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226879+0000", 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5409999999999999e-06 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226885+0000", 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.759e-06 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226898+0000", 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3103e-05 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226907+0000", 2024-09-17T16:30:29.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2060000000000007e-06 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226955+0000", 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7991999999999997e-05 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.227075+0000", 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012014 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.227184+0000", 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010894400000000001 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.230180+0000", 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002995841 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.230383+0000", 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000203043 2024-09-17T16:30:29.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084340+0000", 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.853957060000001 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084380+0000", 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0500000000000002e-05 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084412+0000", 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2026999999999997e-05 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115600 6.b 6:d47603e1:::benchmark_data_smithi173_102017_object22452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.270754+0000", 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.42047692599999, 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.813756322, 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.783 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115600 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270754+0000", 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270754+0000", 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270756+0000", 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7129999999999999e-06 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270763+0000", 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0990000000000001e-06 2024-09-17T16:30:29.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270778+0000", 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5316e-05 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270787+0000", 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6300000000000004e-06 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270852+0000", 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5432000000000006e-05 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270992+0000", 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013987099999999999 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.271065+0000", 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3156000000000002e-05 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.274226+0000", 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031605219999999998 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.274496+0000", 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000270492 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084443+0000", 2024-09-17T16:30:29.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.809946601 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084482+0000", 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8633000000000001e-05 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084510+0000", 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8856999999999998e-05 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.789 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115609 6.b 6:d5c07821:::benchmark_data_smithi173_102017_object22461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.328401+0000", 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.36282972800001, 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.756214807999999, 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.790 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115609 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328401+0000", 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328401+0000", 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328403+0000", 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9010000000000001e-06 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328409+0000", 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5509999999999999e-06 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328423+0000", 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4318000000000001e-05 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328430+0000", 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0210000000000002e-06 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328486+0000", 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6168999999999997e-05 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328615+0000", 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012836600000000001 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328743+0000", 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012822699999999999 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.332302+0000", 2024-09-17T16:30:29.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035596270000000001 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.332400+0000", 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7771999999999998e-05 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084544+0000", 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.752143610999999 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084585+0000", 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1352e-05 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084616+0000", 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0892999999999997e-05 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115610 6.b 6:d3fb650b:::benchmark_data_smithi173_102017_object22462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.796 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.524259+0000", 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 195.16697178699999, 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.567226706, 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115610 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524259+0000", 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524259+0000", 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524262+0000", 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6259999999999999e-06 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524273+0000", 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1076e-05 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524294+0000", 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1111000000000002e-05 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524303+0000", 2024-09-17T16:30:29.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6920000000000008e-06 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524368+0000", 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5178999999999994e-05 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524462+0000", 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4438999999999995e-05 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524539+0000", 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6746e-05 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.527502+0000", 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029623760000000001 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.528273+0000", 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077131499999999996 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.091400+0000", 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.563127508999999 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.091448+0000", 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8084999999999998e-05 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.091486+0000", 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7552000000000002e-05 2024-09-17T16:30:29.802 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115634 6.b 6:d0fb3e7a:::benchmark_data_smithi173_102017_object22486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.738056+0000", 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 194.95317491099999, 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.356902781000001, 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.803 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115634 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738056+0000", 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738056+0000", 2024-09-17T16:30:29.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738058+0000", 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5969999999999999e-06 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738064+0000", 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5740000000000001e-06 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738080+0000", 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5451000000000002e-05 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738088+0000", 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1000000000000004e-06 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738152+0000", 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4474999999999995e-05 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738284+0000", 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131435 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738414+0000", 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000130068 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.741749+0000", 2024-09-17T16:30:29.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033351320000000002 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.741984+0000", 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023470500000000001 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094858+0000", 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.352874224000001 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094915+0000", 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6904999999999998e-05 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094959+0000", 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4115e-05 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115641 6.b 6:d9f11449:::benchmark_data_smithi173_102017_object22493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.809 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 194.92688641000001, 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.330730932, 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115641 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764348+0000", 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0539999999999999e-06 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764355+0000", 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2089999999999999e-06 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764362+0000", 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.533e-06 2024-09-17T16:30:29.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764422+0000", 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0105000000000001e-05 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764557+0000", 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013532 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764684+0000", 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126761 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.768205+0000", 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035209350000000002 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.768669+0000", 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046402699999999999 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094999+0000", 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.326329940000001 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095044+0000", 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095076+0000", 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1754999999999997e-05 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115643 6.b 6:d327719e:::benchmark_data_smithi173_102017_object22495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.775625+0000", 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 194.91560592299999, 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.319555412, 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:30:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:30:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:29.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:29.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115643 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775625+0000", 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775625+0000", 2024-09-17T16:30:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775627+0000", 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.671e-06 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775634+0000", 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2370000000000003e-06 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775650+0000", 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6436999999999999e-05 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775660+0000", 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4639999999999995e-06 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775720+0000", 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9599000000000003e-05 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775841+0000", 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000121878 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775957+0000", 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001158 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.779558+0000", 2024-09-17T16:30:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036009620000000001 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.779623+0000", 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4332999999999995e-05 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095106+0000", 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.315483789 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095149+0000", 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2670999999999999e-05 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095181+0000", 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1571e-05 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27088.0:46453 7.7 e650/629 7:e2b17e74:::benchmark_data_smithi173_116396_object46452:head v 650'2853, mlcod=650'2849)", 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:21.139817+0000", 2024-09-17T16:30:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.551413664, 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30567191599999999, 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:21.139817+0000", 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:30:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:21.139817+0000", 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:21.139818+0000", 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000004e-07 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:21.139867+0000", 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8469e-05 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:21.443134+0000", 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30326708600000002 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:21.443141+0000", 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.182e-06 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:21.443211+0000", 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0617999999999994e-05 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:21.443231+0000", 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9749e-05 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:21.445361+0000", 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021294109999999999 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:21.445489+0000", 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012873100000000001 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.827 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.828 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.828 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:30:29.828 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:30:29.828 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:30:29.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:29.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 56770 56754 70.9364 81.625 0.0110977 0.0140913 2024-09-17T16:30:29.956 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:29.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:29.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:30.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:30.087 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:30.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:30.189 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:30.216 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:30.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:30.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:30.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:30.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:30.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:30.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:30.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:30.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:30 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:30 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:30 smithi017 ceph-mon[27571]: osdmap e652: 12 total, 12 up, 12 in 2024-09-17T16:30:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:30 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:30 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:30 smithi071 ceph-mon[30002]: osdmap e652: 12 total, 12 up, 12 in 2024-09-17T16:30:30.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:30.587 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:30.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:30 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:30 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:30 smithi173 ceph-mon[30530]: osdmap e652: 12 total, 12 up, 12 in 2024-09-17T16:30:30.688 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:30.723 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:30.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:30.828 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:30.850 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:30.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:30.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:30.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 57839 57823 70.8554 66.8125 0.012118 0.0141071 2024-09-17T16:30:30.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:30.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:30.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:30.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:31.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:31.110 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:31.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:31.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:31.213 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:31.221 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1438,"stamp":"2024-09-17T16:30:25.997805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3126788640,"num_objects":47704,"num_object_clones":0,"num_object_copies":143112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47704,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":47825,"num_write_kb":3055078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2676,"num_bytes_recovered":176170048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47794,"ondisk_log_size":47794,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":14857064,"kb_used_data":9409384,"kb_used_omap":58,"kb_used_meta":5447429,"kb_avail":1110084760,"statfs":{"total":1151940427776,"available":1136726794240,"internally_reserved":0,"allocated":9635209216,"data_stored":9630863264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":5578167459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,4,5,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":816185344,"num_objects":12454,"num_object_clones":0,"num_object_copies":37362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":797056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.404670"},"pg_stats":[{"pgid":"7.f","version":"650'2904","reported_seq":2956,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:21.419479+0000","last_change":"2024-09-17T16:29:35.616372+0000","last_active":"2024-09-17T16:30:21.419479+0000","last_peered":"2024-09-17T16:30:21.419479+0000","last_clean":"2024-09-17T16:30:21.419479+0000","last_became_active":"2024-09-17T16:29:35.616046+0000","last_became_peered":"2024-09-17T16:29:35.616046+0000","last_unstale":"2024-09-17T16:30:21.419479+0000","last_undegraded":"2024-09-17T16:30:21.419479+0000","last_fullsized":"2024-09-17T16:30:21.419479+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:35.871596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190316544,"num_objects":2904,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2904,"num_write_kb":185856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"650'3039","reported_seq":3092,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:21.846468+0000","last_change":"2024-09-17T16:29:35.659373+0000","last_active":"2024-09-17T16:30:21.846468+0000","last_peered":"2024-09-17T16:30:21.846468+0000","last_clean":"2024-09-17T16:30:21.846468+0000","last_became_active":"2024-09-17T16:29:35.659021+0000","last_became_peered":"2024-09-17T16:29:35.659021+0000","last_unstale":"2024-09-17T16:30:21.846468+0000","last_undegraded":"2024-09-17T16:30:21.846468+0000","last_fullsized":"2024-09-17T16:30:21.846468+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":3039,"log_dups_size":0,"ondisk_log_size":3039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:43.706367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199163904,"num_objects":3039,"num_object_clones":0,"num_object_copies":9117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3039,"num_write_kb":194496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"650'2914","reported_seq":2967,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:22.271556+0000","last_change":"2024-09-17T16:29:35.618928+0000","last_active":"2024-09-17T16:30:22.271556+0000","last_peered":"2024-09-17T16:30:22.271556+0000","last_clean":"2024-09-17T16:30:22.271556+0000","last_became_active":"2024-09-17T16:29:35.618644+0000","last_became_peered":"2024-09-17T16:29:35.618644+0000","last_unstale":"2024-09-17T16:30:22.271556+0000","last_undegraded":"2024-09-17T16:30:22.271556+0000","last_fullsized":"2024-09-17T16:30:22.271556+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2914,"log_dups_size":0,"ondisk_log_size":2914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:50.763262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":8742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2914,"num_write_kb":186496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"650'3318","reported_seq":3371,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:25.632363+0000","last_change":"2024-09-17T16:29:35.617815+0000","last_active":"2024-09-17T16:30:25.632363+0000","last_peered":"2024-09-17T16:30:25.632363+0000","last_clean":"2024-09-17T16:30:25.632363+0000","last_became_active":"2024-09-17T16:29:35.617430+0000","last_became_peered":"2024-09-17T16:29:35.617430+0000","last_unstale":"2024-09-17T16:30:25.632363+0000","last_undegraded":"2024-09-17T16:30:25.632363+0000","last_fullsized":"2024-09-17T16:30:25.632363+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":3318,"log_dups_size":0,"ondisk_log_size":3318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:48.198410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217448448,"num_objects":3318,"num_object_clones":0,"num_object_copies":9954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3318,"num_write_kb":212352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"650'3009","reported_seq":3062,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:22.290595+0000","last_change":"2024-09-17T16:29:35.686174+0000","last_active":"2024-09-17T16:30:22.290595+0000","last_peered":"2024-09-17T16:30:22.290595+0000","last_clean":"2024-09-17T16:30:22.290595+0000","last_became_active":"2024-09-17T16:29:35.685764+0000","last_became_peered":"2024-09-17T16:29:35.685764+0000","last_unstale":"2024-09-17T16:30:22.290595+0000","last_undegraded":"2024-09-17T16:30:22.290595+0000","last_fullsized":"2024-09-17T16:30:22.290595+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":3009,"log_dups_size":0,"ondisk_log_size":3009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:57.545117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197197824,"num_objects":3009,"num_object_clones":0,"num_object_copies":9027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3009,"num_write_kb":192576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"650'2913","reported_seq":2966,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:21.854183+0000","last_change":"2024-09-17T16:29:35.686166+0000","last_active":"2024-09-17T16:30:21.854183+0000","last_peered":"2024-09-17T16:30:21.854183+0000","last_clean":"2024-09-17T16:30:21.854183+0000","last_became_active":"2024-09-17T16:29:35.685807+0000","last_became_peered":"2024-09-17T16:29:35.685807+0000","last_unstale":"2024-09-17T16:30:21.854183+0000","last_undegraded":"2024-09-17T16:30:21.854183+0000","last_fullsized":"2024-09-17T16:30:21.854183+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2913,"log_dups_size":0,"ondisk_log_size":2913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:39.776288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190906368,"num_objects":2913,"num_object_clones":0,"num_object_copies":8739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2913,"num_write_kb":186432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,11],"acting":[2,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"650'2896","reported_seq":2949,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:21.143821+0000","last_change":"2024-09-17T16:29:35.659376+0000","last_active":"2024-09-17T16:30:21.143821+0000","last_peered":"2024-09-17T16:30:21.143821+0000","last_clean":"2024-09-17T16:30:21.143821+0000","last_became_active":"2024-09-17T16:29:35.659034+0000","last_became_peered":"2024-09-17T16:29:35.659034+0000","last_unstale":"2024-09-17T16:30:21.143821+0000","last_undegraded":"2024-09-17T16:30:21.143821+0000","last_fullsized":"2024-09-17T16:30:21.143821+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:52.950535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189792256,"num_objects":2896,"num_object_clones":0,"num_object_copies":8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":185344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"650'2891","reported_seq":4309,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:21.850770+0000","last_change":"2024-09-17T16:29:56.602948+0000","last_active":"2024-09-17T16:30:21.850770+0000","last_peered":"2024-09-17T16:30:21.850770+0000","last_clean":"2024-09-17T16:30:21.850770+0000","last_became_active":"2024-09-17T16:29:56.600296+0000","last_became_peered":"2024-09-17T16:29:56.600296+0000","last_unstale":"2024-09-17T16:30:21.850770+0000","last_undegraded":"2024-09-17T16:30:21.850770+0000","last_fullsized":"2024-09-17T16:30:21.850770+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2891,"log_dups_size":0,"ondisk_log_size":2891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:03.960970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189464576,"num_objects":2891,"num_object_clones":0,"num_object_copies":8673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2891,"num_write_kb":185024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":44105728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"650'3116","reported_seq":3169,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:24.486320+0000","last_change":"2024-09-17T16:29:35.616770+0000","last_active":"2024-09-17T16:30:24.486320+0000","last_peered":"2024-09-17T16:30:24.486320+0000","last_clean":"2024-09-17T16:30:24.486320+0000","last_became_active":"2024-09-17T16:29:35.616545+0000","last_became_peered":"2024-09-17T16:29:35.616545+0000","last_unstale":"2024-09-17T16:30:24.486320+0000","last_undegraded":"2024-09-17T16:30:24.486320+0000","last_fullsized":"2024-09-17T16:30:24.486320+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":3116,"log_dups_size":0,"ondisk_log_size":3116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:49.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204210176,"num_objects":3116,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3116,"num_write_kb":199424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"650'2901","reported_seq":2954,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:21.109526+0000","last_change":"2024-09-17T16:29:35.686052+0000","last_active":"2024-09-17T16:30:21.109526+0000","last_peered":"2024-09-17T16:30:21.109526+0000","last_clean":"2024-09-17T16:30:21.109526+0000","last_became_active":"2024-09-17T16:29:35.685539+0000","last_became_peered":"2024-09-17T16:29:35.685539+0000","last_unstale":"2024-09-17T16:30:21.109526+0000","last_undegraded":"2024-09-17T16:30:21.109526+0000","last_fullsized":"2024-09-17T16:30:21.109526+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2901,"log_dups_size":0,"ondisk_log_size":2901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:40.718377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190119936,"num_objects":2901,"num_object_clones":0,"num_object_copies":8703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2901,"num_write_kb":185664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"650'2985","reported_seq":4421,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:21.130476+0000","last_change":"2024-09-17T16:29:56.616980+0000","last_active":"2024-09-17T16:30:21.130476+0000","last_peered":"2024-09-17T16:30:21.130476+0000","last_clean":"2024-09-17T16:30:21.130476+0000","last_became_active":"2024-09-17T16:29:56.600312+0000","last_became_peered":"2024-09-17T16:29:56.600312+0000","last_unstale":"2024-09-17T16:30:21.130476+0000","last_undegraded":"2024-09-17T16:30:21.130476+0000","last_fullsized":"2024-09-17T16:30:21.130476+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2985,"log_dups_size":0,"ondisk_log_size":2985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:42.145938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195624960,"num_objects":2985,"num_object_clones":0,"num_object_copies":8955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2985,"num_write_kb":191040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":44695552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"650'2982","reported_seq":4390,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:21.737026+0000","last_change":"2024-09-17T16:29:54.745801+0000","last_active":"2024-09-17T16:30:21.737026+0000","last_peered":"2024-09-17T16:30:21.737026+0000","last_clean":"2024-09-17T16:30:21.737026+0000","last_became_active":"2024-09-17T16:29:54.595605+0000","last_became_peered":"2024-09-17T16:29:54.595605+0000","last_unstale":"2024-09-17T16:30:21.737026+0000","last_undegraded":"2024-09-17T16:30:21.737026+0000","last_fullsized":"2024-09-17T16:30:21.737026+0000","mapping_epoch":640,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2982,"log_dups_size":0,"ondisk_log_size":2982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:37.374814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195428352,"num_objects":2982,"num_object_clones":0,"num_object_copies":8946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2982,"num_write_kb":190848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"650'2867","reported_seq":2920,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:21.737074+0000","last_change":"2024-09-17T16:29:35.659242+0000","last_active":"2024-09-17T16:30:21.737074+0000","last_peered":"2024-09-17T16:30:21.737074+0000","last_clean":"2024-09-17T16:30:21.737074+0000","last_became_active":"2024-09-17T16:29:35.658918+0000","last_became_peered":"2024-09-17T16:29:35.658918+0000","last_unstale":"2024-09-17T16:30:21.737074+0000","last_undegraded":"2024-09-17T16:30:21.737074+0000","last_fullsized":"2024-09-17T16:30:21.737074+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2867,"log_dups_size":0,"ondisk_log_size":2867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:56.097200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187891712,"num_objects":2867,"num_object_clones":0,"num_object_copies":8601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":183488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"650'3202","reported_seq":4560,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:24.927230+0000","last_change":"2024-09-17T16:30:00.594164+0000","last_active":"2024-09-17T16:30:24.927230+0000","last_peered":"2024-09-17T16:30:24.927230+0000","last_clean":"2024-09-17T16:30:24.927230+0000","last_became_active":"2024-09-17T16:30:00.593037+0000","last_became_peered":"2024-09-17T16:30:00.593037+0000","last_unstale":"2024-09-17T16:30:24.927230+0000","last_undegraded":"2024-09-17T16:30:24.927230+0000","last_fullsized":"2024-09-17T16:30:24.927230+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":3202,"log_dups_size":0,"ondisk_log_size":3202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:00.869201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209846272,"num_objects":3202,"num_object_clones":0,"num_object_copies":9606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3202,"num_write_kb":204928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":42598400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"650'2834","reported_seq":2886,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:20.715154+0000","last_change":"2024-09-17T16:29:35.617588+0000","last_active":"2024-09-17T16:30:20.715154+0000","last_peered":"2024-09-17T16:30:20.715154+0000","last_clean":"2024-09-17T16:30:20.715154+0000","last_became_active":"2024-09-17T16:29:35.617197+0000","last_became_peered":"2024-09-17T16:29:35.617197+0000","last_unstale":"2024-09-17T16:30:20.715154+0000","last_undegraded":"2024-09-17T16:30:20.715154+0000","last_fullsized":"2024-09-17T16:30:20.715154+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2834,"log_dups_size":0,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:46.697820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185729024,"num_objects":2834,"num_object_clones":0,"num_object_copies":8502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":181376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"650'2931","reported_seq":2983,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:21.145903+0000","last_change":"2024-09-17T16:29:35.686727+0000","last_active":"2024-09-17T16:30:21.145903+0000","last_peered":"2024-09-17T16:30:21.145903+0000","last_clean":"2024-09-17T16:30:21.145903+0000","last_became_active":"2024-09-17T16:29:35.686233+0000","last_became_peered":"2024-09-17T16:29:35.686233+0000","last_unstale":"2024-09-17T16:30:21.145903+0000","last_undegraded":"2024-09-17T16:30:21.145903+0000","last_fullsized":"2024-09-17T16:30:21.145903+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":2931,"log_dups_size":0,"ondisk_log_size":2931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:36.781435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192086016,"num_objects":2931,"num_object_clones":0,"num_object_copies":8793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2931,"num_write_kb":187584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":44,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-17T16:30:14.607389+0000","last_change":"2024-09-17T16:29:49.984493+0000","last_active":"2024-09-17T16:30:14.607389+0000","last_peered":"2024-09-17T16:30:14.607389+0000","last_clean":"2024-09-17T16:30:14.607389+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:30:14.607389+0000","last_undegraded":"2024-09-17T16:30:14.607389+0000","last_fullsized":"2024-09-17T16:30:14.607389+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:29:40.527128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:29:40.527128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:10.179932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3126198272,"num_objects":47702,"num_object_clones":0,"num_object_copies":143106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47702,"num_write_kb":3052928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2672,"num_bytes_recovered":175112192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9531031552,"data_stored":9531031552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47702,"ondisk_log_size":47702,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1017020,"kb_used_data":606828,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92728132,"statfs":{"total":95995035648,"available":94953607168,"internally_reserved":0,"allocated":621391872,"data_stored":621030579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":420018628},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547456,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1463548,"kb_used_data":1035308,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92281604,"statfs":{"total":95995035648,"available":94496362496,"internally_reserved":0,"allocated":1060155392,"data_stored":1059794119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":438499776},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743684,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1502140,"kb_used_data":929516,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92243012,"statfs":{"total":95995035648,"available":94456844288,"internally_reserved":0,"allocated":951824384,"data_stored":951463111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":586348992},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005322,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":970820,"kb_used_data":596724,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92774332,"statfs":{"total":95995035648,"available":95000915968,"internally_reserved":0,"allocated":611045376,"data_stored":610680551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963798,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037436,"kb_used_data":569708,"kb_used_omap":41,"kb_used_meta":467670,"kb_avail":92707716,"statfs":{"total":95995035648,"available":94932701184,"internally_reserved":0,"allocated":583380992,"data_stored":583019719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":478894387},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495892,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":644228,"kb_used_data":211828,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93100924,"statfs":{"total":95995035648,"available":95335346176,"internally_reserved":0,"allocated":216911872,"data_stored":216547047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":442759622},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":677244,"kb_used_data":180972,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93067908,"statfs":{"total":95995035648,"available":95301537792,"internally_reserved":0,"allocated":185315328,"data_stored":184954055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986463,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1242116,"kb_used_data":809716,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92503036,"statfs":{"total":95995035648,"available":94723108864,"internally_reserved":0,"allocated":829149184,"data_stored":828784359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149985,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2065596,"kb_used_data":1484652,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91679556,"statfs":{"total":95995035648,"available":93879865344,"internally_reserved":0,"allocated":1520283648,"data_stored":1519922375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":594868679},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411623,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1493116,"kb_used_data":1123180,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92252036,"statfs":{"total":95995035648,"available":94466084864,"internally_reserved":0,"allocated":1150136320,"data_stored":1149775047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1080828,"kb_used_data":746988,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92664324,"statfs":{"total":95995035648,"available":94888267776,"internally_reserved":0,"allocated":764915712,"data_stored":764554439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738673,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1662972,"kb_used_data":1113964,"kb_used_omap":1,"kb_used_meta":548990,"kb_avail":92082180,"statfs":{"total":95995035648,"available":94292152320,"internally_reserved":0,"allocated":1140699136,"data_stored":1140337863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":562166213},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1145176064,"data_stored":1145176064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":782630912,"data_stored":782630912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1161494528,"data_stored":1161494528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1534525440,"data_stored":1534525440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":842989568,"data_stored":842989568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":185729024,"data_stored":185729024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":217448448,"data_stored":217448448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":397475840,"data_stored":397475840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":616890368,"data_stored":616890368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":950861824,"data_stored":950861824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1074724864,"data_stored":1074724864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":621084672,"data_stored":621084672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:30:31.222 INFO:tasks.thrashosds.thrasher:Setting 7.9 to [3, 1, 2, 4, 5, 10] 2024-09-17T16:30:31.222 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '7.9', '3', '1', '2', '4', '5', '10'] 2024-09-17T16:30:31.222 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 7.9 3 1 2 4 5 10 2024-09-17T16:30:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:31.249 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:31.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:31.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:31.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:31.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:31.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:31.454 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:30:31.481 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:31 smithi017 ceph-mon[27571]: pgmap v1442: 17 pgs: 17 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-09-17T16:30:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:31 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:31 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:31.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:31 smithi017 ceph-mon[27571]: osdmap e653: 12 total, 12 up, 12 in 2024-09-17T16:30:31.519 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:31.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:31.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:31 smithi071 ceph-mon[30002]: pgmap v1442: 17 pgs: 17 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-09-17T16:30:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:31 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:31 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:31 smithi071 ceph-mon[30002]: osdmap e653: 12 total, 12 up, 12 in 2024-09-17T16:30:31.621 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:31.644 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:31 smithi173 ceph-mon[30530]: pgmap v1442: 17 pgs: 17 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-09-17T16:30:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:31 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:31 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:31 smithi173 ceph-mon[30530]: osdmap e653: 12 total, 12 up, 12 in 2024-09-17T16:30:31.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:31.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:31.747 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:31.787 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:31.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:31.889 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:31.911 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:31.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:31.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 58910 58894 70.78 66.9375 0.107981 0.0140841 2024-09-17T16:30:32.013 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:32.035 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:32.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:32.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:32.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:32.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:32.211 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:32.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:30:32.256 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:32.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:32.326 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:32.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:32.428 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:32.452 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:32 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:32.452 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:32 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:32.453 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:32 smithi071 ceph-mon[30002]: osdmap e654: 12 total, 12 up, 12 in 2024-09-17T16:30:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:32.454 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:32.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:32 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:32 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:32 smithi017 ceph-mon[27571]: osdmap e654: 12 total, 12 up, 12 in 2024-09-17T16:30:32.545 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:30:32.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:32.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:32.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:32.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:32 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:32 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:32 smithi173 ceph-mon[30530]: osdmap e654: 12 total, 12 up, 12 in 2024-09-17T16:30:32.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:32.721 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:32.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:32.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:32.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:32.875 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:32.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 60195 60179 70.9597 80.3125 0.00583086 0.0140827 2024-09-17T16:30:32.948 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:32.973 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:32.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:33.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:33.077 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:30:33.077 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:30:33.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.077 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:60324 7.1 7:8e4ca64e:::benchmark_data_smithi173_116396_object60323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-09-17T16:30:33.077 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:30:33.053339+0000", 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.014064297, 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.014084625999999999, 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 60324 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:33.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.053339+0000", 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.053339+0000", 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:33.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.053339+0000", 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4299999999999999e-07 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.053382+0000", 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2636999999999999e-05 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.053387+0000", 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0760000000000002e-06 2024-09-17T16:30:33.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.053390+0000", 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3519999999999998e-06 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.056707+0000", 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00331672 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:33.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.056820+0000", 2024-09-17T16:30:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011258 2024-09-17T16:30:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:30:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.056918+0000", 2024-09-17T16:30:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8444999999999997e-05 2024-09-17T16:30:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:30:33.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.058065+0000", 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001146492 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.066119+0000", 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0080543079999999996 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.083 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:60335 7.1 7:80ee570f:::benchmark_data_smithi173_116396_object60334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:30:33.061666+0000", 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0057369489999999999, 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0058045589999999999, 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 60335 2024-09-17T16:30:33.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.085 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:33.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:33.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.061666+0000", 2024-09-17T16:30:33.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:33.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:33.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.061666+0000", 2024-09-17T16:30:33.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.061667+0000", 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1319999999999999e-06 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.061734+0000", 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6161000000000006e-05 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.061753+0000", 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9448000000000001e-05 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.061761+0000", 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6480000000000003e-06 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.061803+0000", 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2357000000000001e-05 2024-09-17T16:30:33.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:33.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.061880+0000", 2024-09-17T16:30:33.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6879999999999996e-05 2024-09-17T16:30:33.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:30:33.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.061968+0000", 2024-09-17T16:30:33.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7769999999999995e-05 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.064143+0000", 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002174858 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.066188+0000", 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020453049999999999 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:33.089 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:60338 7.1 7:88d699b7:::benchmark_data_smithi173_116396_object60337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:30:33.062108+0000", 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0052956440000000004, 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.005401416, 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:33.090 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 60338 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.062108+0000", 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.062108+0000", 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.062108+0000", 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7899999999999997e-07 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.062152+0000", 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3579999999999999e-05 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.062160+0000", 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4689999999999997e-06 2024-09-17T16:30:33.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.062167+0000", 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.64e-06 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.063233+0000", 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010655809999999999 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:33.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.063310+0000", 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7173000000000001e-05 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.063396+0000", 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5715000000000002e-05 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.064303+0000", 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00090774900000000001 2024-09-17T16:30:33.094 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:60244 7.8 e655/642 7:125c6726:::benchmark_data_smithi173_116396_object60243:head v 655'3717, mlcod=655'3714)", 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:30:32.987141+0000", 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.080261958999999994, 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.080402355999999994, 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:33.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:32.987141+0000", 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:32.987141+0000", 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:32.987142+0000", 2024-09-17T16:30:33.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5000000000000001e-07 2024-09-17T16:30:33.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:33.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:32.987268+0000", 2024-09-17T16:30:33.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012603399999999999 2024-09-17T16:30:33.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:33.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.066909+0000", 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.079640394000000003 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.066915+0000", 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.708e-06 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.066955+0000", 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9561999999999998e-05 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:33.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:33.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:33.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:33.066969+0000", 2024-09-17T16:30:33.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4181e-05 2024-09-17T16:30:33.099 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:33.099 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:33.099 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:33.099 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:33.099 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:30:33.099 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 4 2024-09-17T16:30:33.099 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:30:33.128 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:33.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:33.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:33.253 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:33.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:33.355 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:33.375 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:33 smithi071 ceph-mon[30002]: pgmap v1445: 17 pgs: 17 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:30:33.375 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:33 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:33.376 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:33 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:33.376 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:33 smithi071 ceph-mon[30002]: osdmap e655: 12 total, 12 up, 12 in 2024-09-17T16:30:33.391 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:33.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:33.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:33 smithi017 ceph-mon[27571]: pgmap v1445: 17 pgs: 17 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:30:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:33 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:33 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:33 smithi017 ceph-mon[27571]: osdmap e655: 12 total, 12 up, 12 in 2024-09-17T16:30:33.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:33.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:33.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:33.621 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:33.643 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:33.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:33 smithi173 ceph-mon[30530]: pgmap v1445: 17 pgs: 17 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:30:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:33 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:33 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:33 smithi173 ceph-mon[30530]: osdmap e655: 12 total, 12 up, 12 in 2024-09-17T16:30:33.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:33.768 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:33.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:33.870 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:33.892 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:33.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:33.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 61464 61448 71.1143 79.3125 0.00540291 0.0140557 2024-09-17T16:30:33.984 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:30:33.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:34.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:34.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:34.123 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:34.145 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:34.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:34.190 INFO:teuthology.orchestra.run.smithi017.stderr:set 7.9 pg_upmap_items mapping to [3->1,2->4,5->10] 2024-09-17T16:30:34.248 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:34.270 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:30:34.282 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:34.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3708863267' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [3, 1, 2, 4, 5, 10]} : dispatch 2024-09-17T16:30:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:34 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [3, 1, 2, 4, 5, 10]} : dispatch 2024-09-17T16:30:34.384 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:34.407 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:34.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:34.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:34.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3708863267' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [3, 1, 2, 4, 5, 10]} : dispatch 2024-09-17T16:30:34.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:34 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [3, 1, 2, 4, 5, 10]} : dispatch 2024-09-17T16:30:34.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:34.529 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:34.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:34.631 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3708863267' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [3, 1, 2, 4, 5, 10]} : dispatch 2024-09-17T16:30:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:34 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [3, 1, 2, 4, 5, 10]} : dispatch 2024-09-17T16:30:34.654 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:34.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:34.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:34.756 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:34.791 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:34.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:34.813 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:30:34.813 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:30:34.813 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:30:34.813 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:30:34.813 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:30:34.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:34.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:34.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:34.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 62335 62319 70.811 54.4375 0.0225541 0.0141137 2024-09-17T16:30:35.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:35.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:35.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:35.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:35.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:35.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:35.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:35.258 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:30:35.258 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:30:35.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:35.289 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:30:35.289 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:30:35.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:35.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:35.319 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:30:35.319 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:30:35.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:30:35.347 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:30:35.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:35 smithi017 ceph-mon[27571]: pgmap v1447: 17 pgs: 17 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:30:35.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:35 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [3, 1, 2, 4, 5, 10]}]': finished 2024-09-17T16:30:35.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:35 smithi017 ceph-mon[27571]: osdmap e656: 12 total, 12 up, 12 in 2024-09-17T16:30:35.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:35 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:35.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:35 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:35.379 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:30:35.379 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:30:35.399 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:35.413 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:30:35.413 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:30:35.432 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:35.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:35.443 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:30:35.443 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:30:35.444 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:35 smithi071 ceph-mon[30002]: pgmap v1447: 17 pgs: 17 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:30:35.444 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:35 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [3, 1, 2, 4, 5, 10]}]': finished 2024-09-17T16:30:35.444 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:35 smithi071 ceph-mon[30002]: osdmap e656: 12 total, 12 up, 12 in 2024-09-17T16:30:35.444 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:35 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:35.444 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:35 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:35.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:30:35.479 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:30:35.479 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:30:35.512 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:30:35.513 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:30:35.533 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:35.554 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:30:35.554 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:30:35.571 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:35.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:35.589 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:30:35.590 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:30:35.628 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:30:35.628 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:30:35.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:35 smithi173 ceph-mon[30530]: pgmap v1447: 17 pgs: 17 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-17T16:30:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:35 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [3, 1, 2, 4, 5, 10]}]': finished 2024-09-17T16:30:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:35 smithi173 ceph-mon[30530]: osdmap e656: 12 total, 12 up, 12 in 2024-09-17T16:30:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:35 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:35 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:35.672 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:35.696 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:35.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:35.749 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:30:35.799 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:35.821 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:35.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:35.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:35.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:35.945 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:35.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:35.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 63377 63361 70.7093 65.125 0.026732 0.0141185 2024-09-17T16:30:36.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:36.077 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:36.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:36.179 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:36.202 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:30:36.203 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:36.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:36.299 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:36.300 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:36.305 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:36.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:36.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:36.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:36.455 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:36.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:36.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:36 smithi017 ceph-mon[27571]: osdmap e657: 12 total, 12 up, 12 in 2024-09-17T16:30:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1843301036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/205902689' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:30:36.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:36 smithi071 ceph-mon[30002]: osdmap e657: 12 total, 12 up, 12 in 2024-09-17T16:30:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1843301036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:36.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/205902689' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:30:36.588 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:36.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:36 smithi173 ceph-mon[30530]: osdmap e657: 12 total, 12 up, 12 in 2024-09-17T16:30:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1843301036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/205902689' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:30:36.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:36.713 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:36.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:36.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:36.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:36.838 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:36.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:36.940 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:36.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 64403 64387 70.5937 64.125 0.00324835 0.0141582 2024-09-17T16:30:36.968 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:36.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:37.009 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:30:37.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:37.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:37.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:37.194 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:37.220 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:37.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:37.283 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:30:37.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:37.345 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:37.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:37.447 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:37.457 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:37 smithi071 ceph-mon[30002]: pgmap v1450: 17 pgs: 17 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s 2024-09-17T16:30:37.457 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:37 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:37.457 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/205902689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:30:37.457 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:37 smithi071 ceph-mon[30002]: osdmap e658: 12 total, 12 up, 12 in 2024-09-17T16:30:37.457 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:37 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:37.457 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:37 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:37.471 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:37.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:37 smithi017 ceph-mon[27571]: pgmap v1450: 17 pgs: 17 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s 2024-09-17T16:30:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:37 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/205902689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:30:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:37 smithi017 ceph-mon[27571]: osdmap e658: 12 total, 12 up, 12 in 2024-09-17T16:30:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:37 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:37 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:37.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:37.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:37.598 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:37 smithi173 ceph-mon[30530]: pgmap v1450: 17 pgs: 17 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s 2024-09-17T16:30:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:37 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/205902689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:30:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:37 smithi173 ceph-mon[30530]: osdmap e658: 12 total, 12 up, 12 in 2024-09-17T16:30:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:37 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:37 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:37.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:37.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:37.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:37.828 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:37.851 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:37.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:37.941 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:37.941 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:37.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 65581 65565 70.6458 73.625 0.0160899 0.0141498 2024-09-17T16:30:37.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:37.978 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:37.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:38.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:38.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:38.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:38.209 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:38.243 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:38.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:38.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:38.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:38.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:38.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:38.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:38 smithi017 ceph-mon[27571]: osdmap e659: 12 total, 12 up, 12 in 2024-09-17T16:30:38.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:38.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:38.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:38 smithi071 ceph-mon[30002]: osdmap e659: 12 total, 12 up, 12 in 2024-09-17T16:30:38.604 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:38.627 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:38.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:38.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:38 smithi173 ceph-mon[30530]: osdmap e659: 12 total, 12 up, 12 in 2024-09-17T16:30:38.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:38.762 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:38.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:38.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:38.888 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:38.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:38.918 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:30:38.945 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 66810 66794 70.7502 76.8125 0.00772125 0.0141299 2024-09-17T16:30:38.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:39.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:39.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:39.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:39.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:39.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:39.226 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:39 smithi071 ceph-mon[30002]: pgmap v1453: 17 pgs: 17 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:30:39.227 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:39 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:39.241 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:30:39.249 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:39.289 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:39.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:39.391 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:39.425 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:39.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:39 smithi017 ceph-mon[27571]: pgmap v1453: 17 pgs: 17 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:30:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:39 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:39 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:39.526 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:39.529 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:39 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:39.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:39 smithi173 ceph-mon[30530]: pgmap v1453: 17 pgs: 17 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:30:39.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:39 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:39.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:39 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:39.556 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:39.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:39.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:39.682 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:39.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:39.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:39.805 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:39.805 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:39.810 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:39.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:39.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:39.912 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:39.943 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:39.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:39.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:30:39.944822+0000 min lat: 0.00119669 max lat: 2.03842 avg lat: 0.0140892 2024-09-17T16:30:39.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:30:39.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 68114 68098 70.9293 81.5 0.00325222 0.0140892 2024-09-17T16:30:40.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:40.072 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:40.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:40.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:40.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:40.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:40.302 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:40.329 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:40.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:40.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:40.456 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:40.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:40 smithi017 ceph-mon[27571]: osdmap e660: 12 total, 12 up, 12 in 2024-09-17T16:30:40.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:30:40.558 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:40.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:40 smithi071 ceph-mon[30002]: osdmap e660: 12 total, 12 up, 12 in 2024-09-17T16:30:40.632 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:40.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:40.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:30:40.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:40 smithi173 ceph-mon[30530]: osdmap e660: 12 total, 12 up, 12 in 2024-09-17T16:30:40.734 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:40.774 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:40.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:40.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:40.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:40.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:40.930 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:40.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 69079 69063 70.7551 60.3125 0.00432832 0.0141269 2024-09-17T16:30:41.001 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:41.024 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:41.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:41.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:30:41.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:41.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:41.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:41.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:41.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:41.292 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:41.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:41.369 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:30:41.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:41.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:41.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:41.437 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:41 smithi017 ceph-mon[27571]: pgmap v1455: 17 pgs: 17 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:30:41.493 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:41 smithi071 ceph-mon[30002]: pgmap v1455: 17 pgs: 17 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:30:41.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:41.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:41.543 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:41.576 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:30:41.576 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:30:41.609 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:30:41.609 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:30:41.634 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:30:41.635 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:30:41.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:41.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:41 smithi173 ceph-mon[30530]: pgmap v1455: 17 pgs: 17 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-17T16:30:41.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:41.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:41.668 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:30:41.668 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:30:41.690 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:41.697 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:30:41.697 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:30:41.724 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:30:41.724 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:30:41.749 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:30:41.750 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:30:41.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:41.778 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:30:41.779 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:30:41.795 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:41.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:41.809 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:30:41.809 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:30:41.836 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:30:41.836 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:30:41.863 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:30:41.863 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:30:41.888 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:30:41.889 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:30:41.897 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:41.900 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:30:41.900 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:30:41.900 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:30:41.900 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:30:41.900 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.900 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:176912 4.2 4:44b32f3d:::benchmark_data_smithi173_77474_object74737:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e383)", 2024-09-17T16:30:41.900 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:42.499251+0000", 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 599.39820626400001, 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34693342300000002, 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 176912 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499251+0000", 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499251+0000", 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499253+0000", 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.874e-06 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499260+0000", 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.187e-06 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499285+0000", 2024-09-17T16:30:41.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5329000000000001e-05 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499301+0000", 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5590000000000002e-05 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499413+0000", 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011231 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.904 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499434+0000", 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0835e-05 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.846068+0000", 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34663389999999999 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.846184+0000", 2024-09-17T16:30:41.905 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011639800000000001 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:176918 4.2 4:4746b7f5:::benchmark_data_smithi173_77474_object74743:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e383)", 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:42.499325+0000", 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 599.39813184399998, 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37077265799999998, 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.906 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 176918 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499325+0000", 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499325+0000", 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499326+0000", 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6400000000000002e-07 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499329+0000", 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0570000000000001e-06 2024-09-17T16:30:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499346+0000", 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7367000000000002e-05 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499353+0000", 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6599999999999998e-06 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.909 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499521+0000", 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016777900000000001 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.499530+0000", 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7030000000000001e-06 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.846482+0000", 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34695168900000001 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.911 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.911 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.846605+0000", 2024-09-17T16:30:41.911 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000122402 2024-09-17T16:30:41.911 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.911 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.911 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9,11", 2024-09-17T16:30:41.911 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.846744+0000", 2024-09-17T16:30:41.911 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013938000000000001 2024-09-17T16:30:41.911 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.911 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.858906+0000", 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.012162506999999999 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.867247+0000", 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0083402130000000008 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:41.912 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.867355+0000", 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000108434 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.869979+0000", 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002623785 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.870021+0000", 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1816000000000001e-05 2024-09-17T16:30:41.913 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:42.870098+0000", 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7302e-05 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(4.2 389/383 [PushReplyOp(4:4e55586c:::benchmark_data_smithi173_77474_object100589:head)])", 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:53.998631+0000", 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 587.89882573299997, 2024-09-17T16:30:41.914 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44632730900000001, 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.998631+0000", 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.998631+0000", 2024-09-17T16:30:41.915 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.916 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.916 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.916 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.916 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.998632+0000", 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.17e-07 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:53.998634+0000", 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9879999999999999e-06 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.442705+0000", 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44407097200000001 2024-09-17T16:30:41.917 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.442709+0000", 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5210000000000004e-06 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.444667+0000", 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019580940000000001 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.918 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:54.444959+0000", 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00029161699999999998 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(4.a 391/383 [PushOp(4:5f33903b:::benchmark_data_smithi173_77474_object97064:head, version: 347'6156, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:5f33903b:::benchmark_data_smithi173_77474_object97064:head@347'6156, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:55.758638+0000", 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 586.13881922099995, 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70397542800000001, 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.919 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:55.758638+0000", 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:55.758638+0000", 2024-09-17T16:30:41.920 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:55.758638+0000", 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.99e-07 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:55.758703+0000", 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4610999999999995e-05 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:56.462316+0000", 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70361271599999997 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:56.462343+0000", 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7909999999999999e-05 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:56.462460+0000", 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000116815 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.922 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:56.462470+0000", 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5799999999999998e-06 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:56.462613+0000", 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000143597 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:186293 4.f e393/383 4:ffe1d3e0:::benchmark_data_smithi173_77474_object84118:head v 393'11582, mlcod=393'11579)", 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:20:58.185863+0000", 2024-09-17T16:30:41.923 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 583.71159396400003, 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.42666476800000003, 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.185863+0000", 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.185863+0000", 2024-09-17T16:30:41.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.185865+0000", 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.277e-06 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.185873+0000", 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.371e-06 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.610471+0000", 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.42459858 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.610476+0000", 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7149999999999997e-06 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.610557+0000", 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1149000000000005e-05 2024-09-17T16:30:41.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.610573+0000", 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5951e-05 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.612500+0000", 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019271710000000001 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:20:58.612528+0000", 2024-09-17T16:30:41.927 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7554000000000001e-05 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.25516.0:197771 4.1 4:8a203159:::benchmark_data_smithi173_77474_object95596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:13.590237+0000", 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 568.30722055599995, 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32554565600000002, 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.25516", 2024-09-17T16:30:41.928 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1957552998", 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 197771 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590237+0000", 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590237+0000", 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590238+0000", 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.271e-06 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590243+0000", 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1989999999999997e-06 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590254+0000", 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0638e-05 2024-09-17T16:30:41.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590259+0000", 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0069999999999997e-06 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590300+0000", 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1007999999999998e-05 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590385+0000", 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5731999999999998e-05 2024-09-17T16:30:41.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 6,11", 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.590473+0000", 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7978000000000004e-05 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.594248+0000", 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0037743640000000001 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.594779+0000", 2024-09-17T16:30:41.932 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00053105800000000005 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.915695+0000", 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32091583899999998 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.915732+0000", 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6918000000000001e-05 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.933 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:13.915782+0000", 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0643999999999998e-05 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.25516.0:204340 4.a e408/393 4:583138b7:::benchmark_data_smithi173_77474_object102165:head v 408'12934, mlcod=408'12929)", 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:21:23.617934+0000", 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 558.27952352600005, 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57527535799999996, 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.934 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:23.617934+0000", 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:23.617934+0000", 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:23.617934+0000", 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8299999999999996e-07 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:23.617941+0000", 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3990000000000001e-06 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.190571+0000", 2024-09-17T16:30:41.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57263039199999999 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.190579+0000", 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9230000000000002e-06 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.190659+0000", 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9571000000000001e-05 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.190686+0000", 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7177000000000001e-05 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.193171+0000", 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024847900000000002 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:21:24.193209+0000", 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8322999999999999e-05 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.938 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26158.0:15935 5.f 5:f3ff7eb6:::benchmark_data_smithi173_90735_object15934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:22:02.436987+0000", 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 519.46047019399998, 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79589018300000003, 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26158", 2024-09-17T16:30:41.939 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 15935 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.436987+0000", 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.436987+0000", 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.436988+0000", 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2500000000000004e-07 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.437037+0000", 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9310000000000001e-05 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.437048+0000", 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0861e-05 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.437053+0000", 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3360000000000002e-06 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.437092+0000", 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9104999999999999e-05 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.437161+0000", 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8837999999999994e-05 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,11", 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.437237+0000", 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5368000000000001e-05 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.437799+0000", 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00056245199999999996 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:41.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:02.438118+0000", 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000318652 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:03.232765+0000", 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79464739100000004 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:03.232831+0000", 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6156999999999997e-05 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:22:03.232877+0000", 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5887999999999998e-05 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:90586 6.7 6.f12f0c47 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e567)", 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:33.608289+0000", 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 248.289167883, 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.91384783400000003, 2024-09-17T16:30:41.945 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 90586 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.608289+0000", 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.608289+0000", 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.608290+0000", 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7700000000000003e-07 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.608322+0000", 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2045999999999998e-05 2024-09-17T16:30:41.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.608334+0000", 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1736000000000001e-05 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.608341+0000", 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0859999999999996e-06 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.608383+0000", 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2222000000000003e-05 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.608388+0000", 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0390000000000004e-06 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:34.522124+0000", 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.913735827 2024-09-17T16:30:41.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:34.522137+0000", 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3001e-05 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:90596 6.7 6.618cec7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:33.612416+0000", 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 248.28504151800001, 2024-09-17T16:30:41.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.90987580000000001, 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 90596 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612416+0000", 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612416+0000", 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612417+0000", 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2249999999999999e-06 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612464+0000", 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7494999999999998e-05 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612475+0000", 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0208e-05 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612480+0000", 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1320000000000002e-06 2024-09-17T16:30:41.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612518+0000", 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8152999999999999e-05 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612524+0000", 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3210000000000001e-06 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:34.522281+0000", 2024-09-17T16:30:41.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.90975704499999999 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:34.522291+0000", 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0220999999999999e-05 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:90597 6.7 6.6fd0c667 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:33.612545+0000", 2024-09-17T16:30:41.955 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 248.28491184399999, 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.90978444400000003, 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 90597 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612545+0000", 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612545+0000", 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612546+0000", 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6000000000000004e-07 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612607+0000", 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1050000000000007e-05 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612613+0000", 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5189999999999998e-06 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612617+0000", 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.614e-06 2024-09-17T16:30:41.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.959 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 69878 69862 70.4193 49.9375 0.0535105 0.0141918 2024-09-17T16:30:41.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612661+0000", 2024-09-17T16:30:41.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4005999999999998e-05 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:33.612669+0000", 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5480000000000004e-06 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:34.522320+0000", 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.90965125099999999 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:34.522330+0000", 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8959999999999998e-06 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(6.4 582/568 [PushOp(6:281d5555:::benchmark_data_smithi173_102017_object25803:head, version: 544'1611, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:281d5555:::benchmark_data_smithi173_102017_object25803:head@544'1611, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:27:17.007662+0000", 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 204.88979480200001, 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.84428071299999996, 2024-09-17T16:30:41.961 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.007662+0000", 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.007662+0000", 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.007663+0000", 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.06e-07 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.007691+0000", 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7974999999999999e-05 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.963 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.850066+0000", 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.84237536899999998 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.850093+0000", 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6602999999999999e-05 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.850247+0000", 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000153821 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.850254+0000", 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.571e-06 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.851943+0000", 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016886679999999999 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(6.d 582/568 [PushOp(6:bcd2660c:::benchmark_data_smithi173_102017_object27055:head, version: 544'1655, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:bcd2660c:::benchmark_data_smithi173_102017_object27055:head@544'1655, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:27:17.245935+0000", 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 204.651522054, 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55889859500000005, 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.245935+0000", 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.245935+0000", 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.245936+0000", 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7000000000000004e-07 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.245965+0000", 2024-09-17T16:30:41.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9263000000000001e-05 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.802794+0000", 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55682863199999999 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.802805+0000", 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0985000000000001e-05 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.802878+0000", 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3701999999999993e-05 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.802887+0000", 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9930000000000001e-06 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:17.804834+0000", 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019463499999999999 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.969 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(6.d 582/568 [PushOp(6:b87983ab:::benchmark_data_smithi173_102017_object27770:head, version: 547'1705, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:b87983ab:::benchmark_data_smithi173_102017_object27770:head@547'1705, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:27:18.356685+0000", 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 203.54077211000001, 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.97865760400000001, 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:18.356685+0000", 2024-09-17T16:30:41.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:18.356685+0000", 2024-09-17T16:30:41.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:41.972 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:41.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:41.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:18.356687+0000", 2024-09-17T16:30:41.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.654e-06 2024-09-17T16:30:41.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:18.356743+0000", 2024-09-17T16:30:41.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6499999999999998e-05 2024-09-17T16:30:41.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.332805+0000", 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.97606177199999999 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.332832+0000", 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.741e-05 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.333069+0000", 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00023670599999999999 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.333112+0000", 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3294999999999998e-05 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.335343+0000", 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022302670000000002 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.975 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(6.4 582/568 [PushOp(6:2813109f:::benchmark_data_smithi173_102017_object26489:head, version: 544'1657, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:2813109f:::benchmark_data_smithi173_102017_object26489:head@544'1657, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:27:18.433815+0000", 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 203.463642212, 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79163000800000005, 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:18.433815+0000", 2024-09-17T16:30:41.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:18.433815+0000", 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:18.433816+0000", 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4000000000000001e-07 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:18.433841+0000", 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4929000000000001e-05 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.222561+0000", 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.788720058 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.222607+0000", 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6128999999999998e-05 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.222815+0000", 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000208063 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.222826+0000", 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1199999999999999e-05 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.225445+0000", 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002618989 2024-09-17T16:30:41.979 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(6.c 582/568 [PushOp(6:31643fcf:::benchmark_data_smithi173_102017_object26295:head, version: 544'1665, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:31643fcf:::benchmark_data_smithi173_102017_object26295:head@544'1665, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:27:19.031451+0000", 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 202.86600627600001, 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79046347699999997, 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:41.980 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.031451+0000", 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.031451+0000", 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.031452+0000", 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2699999999999999e-07 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.031477+0000", 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5137e-05 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.819833+0000", 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.78835625499999995 2024-09-17T16:30:41.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.819844+0000", 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0737000000000001e-05 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.819901+0000", 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7550999999999998e-05 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.819910+0000", 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6500000000000002e-06 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:19.821914+0000", 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020044199999999998 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.984 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(6.4 582/568 [PushOp(6:27f9d298:::benchmark_data_smithi173_102017_object30133:head, version: 547'1876, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:27f9d298:::benchmark_data_smithi173_102017_object30133:head@547'1876, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:27:22.176892+0000", 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 199.72056507600001, 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52406880899999997, 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:22.176892+0000", 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:22.176892+0000", 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:22.176893+0000", 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7999999999999995e-07 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:22.176921+0000", 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8496e-05 2024-09-17T16:30:41.986 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:22.699205+0000", 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52228376399999998 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:22.699221+0000", 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5605000000000001e-05 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:22.699287+0000", 2024-09-17T16:30:41.987 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6777000000000001e-05 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:22.699295+0000", 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4839999999999998e-06 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:22.700961+0000", 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016660030000000001 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(6.d 582/568 [PushOp(6:be78c722:::benchmark_data_smithi173_102017_object33645:head, version: 548'2095, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:be78c722:::benchmark_data_smithi173_102017_object33645:head@548'2095, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:27:23.806894+0000", 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 198.09056361899999, 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.484966534, 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:23.806894+0000", 2024-09-17T16:30:41.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:23.806894+0000", 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:23.806894+0000", 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1799999999999995e-07 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.990 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:23.806917+0000", 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2643e-05 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:24.289850+0000", 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.48293332900000002 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:24.289859+0000", 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8939999999999999e-06 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:24.289918+0000", 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9079999999999997e-05 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:24.289927+0000", 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9970000000000004e-06 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:24.291860+0000", 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019329729999999999 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.992 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(6.4 582/568 [PushOp(6:2ac6c1b5:::benchmark_data_smithi173_102017_object34172:head, version: 548'2088, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:2ac6c1b5:::benchmark_data_smithi173_102017_object34172:head@548'2088, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:27:25.653430+0000", 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 196.24402769, 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40288464099999999, 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.993 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:25.653430+0000", 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:25.653430+0000", 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:25.653430+0000", 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7e-07 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.994 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:25.653491+0000", 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1087999999999995e-05 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:26.054472+0000", 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40098117700000002 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:26.054481+0000", 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0920000000000004e-06 2024-09-17T16:30:41.995 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:26.054564+0000", 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2780000000000004e-05 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:26.054577+0000", 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2941e-05 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:26.056314+0000", 2024-09-17T16:30:41.996 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017370930000000001 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(6.d 582/568 [PushOp(6:b5542677:::benchmark_data_smithi173_102017_object36940:head, version: 551'2289, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:b5542677:::benchmark_data_smithi173_102017_object36940:head@551'2289, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:27:26.403135+0000", 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 195.49432232300001, 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.663566192, 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:41.997 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:26.403135+0000", 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:26.403135+0000", 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:26.403136+0000", 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5300000000000004e-07 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:26.403162+0000", 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6384e-05 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:27.064091+0000", 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.66092949199999995 2024-09-17T16:30:41.999 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:27.064119+0000", 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7177000000000001e-05 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:27.064316+0000", 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019788 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:42.000 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:27.064339+0000", 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.298e-05 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:27:27.066701+0000", 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023616259999999999 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:42.001 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:30:42.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:42.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:42.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:42.140 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:30:42.140 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:30:42.141 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_5 2024-09-17T16:30:42.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:30:42.198 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:42.236 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:42.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:42.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:42.371 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:42.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:42.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:42.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1676326890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:42.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:42 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:30:42.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:42 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:30:42.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:42 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:30:42.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:42 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:30:42.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:42.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1676326890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:30:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:30:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:30:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:42 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:30:42.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:42.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:42.626 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:42.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1676326890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:42 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:30:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:42 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:30:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:42 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:30:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:42 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:30:42.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:42.759 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:42.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:42.860 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:42.884 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:42.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:42.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:42.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:30:42.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 70950 70934 70.3649 67 0.00675333 0.0142053 2024-09-17T16:30:42.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:43.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:43.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:43.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:43.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:43.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:43.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:43.244 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:30:43.245 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:43.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:43.272 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:43.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:43.414 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:43.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:43.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:43.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:43.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:43.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:43.644 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:43.668 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:43.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:43.756 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:30:43.756 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:30:43.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.756 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:71615 7.c 7:3e19d7d5:::benchmark_data_smithi173_116396_object71614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e660)", 2024-09-17T16:30:43.756 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:30:43.699500+0000", 2024-09-17T16:30:43.757 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.039586024999999997, 2024-09-17T16:30:43.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.039612885, 2024-09-17T16:30:43.757 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:43.757 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:43.757 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:30:43.758 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:43.758 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:30:43.758 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:43.758 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 71615 2024-09-17T16:30:43.758 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.758 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:43.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.759 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:43.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.699500+0000", 2024-09-17T16:30:43.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:43.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:43.760 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.699500+0000", 2024-09-17T16:30:43.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:43.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.699501+0000", 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.218e-06 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.699549+0000", 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8133999999999997e-05 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.699560+0000", 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0808999999999999e-05 2024-09-17T16:30:43.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.699565+0000", 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3839999999999996e-06 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.699604+0000", 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8611e-05 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:43.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.699688+0000", 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4739000000000003e-05 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.699775+0000", 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6219999999999998e-05 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.701532+0000", 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017569319999999999 2024-09-17T16:30:43.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:43.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.702782+0000", 2024-09-17T16:30:43.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012505789999999999 2024-09-17T16:30:43.764 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:43.764 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:43.764 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:43.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:71621 7.c 7:333e2574:::benchmark_data_smithi173_116396_object71620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e660)", 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:30:43.705401+0000", 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.033684655000000001, 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.033762397999999999, 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 71621 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.765 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.705401+0000", 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.705401+0000", 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.705403+0000", 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7400000000000001e-06 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.705447+0000", 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4456999999999998e-05 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.705463+0000", 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5848000000000001e-05 2024-09-17T16:30:43.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.705470+0000", 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3490000000000003e-06 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.705516+0000", 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5800000000000002e-05 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:30:43.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.705643+0000", 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012693700000000001 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.705738+0000", 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4773999999999998e-05 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.707676+0000", 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019379499999999999 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:30:43.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:30:43.770 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:43.770 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:30:43.709054+0000", 2024-09-17T16:30:43.770 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013781380000000001 2024-09-17T16:30:43.770 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:43.770 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:30:43.770 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:43.770 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:30:43.770 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:30:43.770 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 2 2024-09-17T16:30:43.770 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:30:43.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:43.797 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:43.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:43.899 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:43.920 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:43.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:43.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 71787 71771 70.0828 52.3125 0.114207 0.0142617 2024-09-17T16:30:44.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:44.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:44.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:44.117 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:30:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:30:44.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:44.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:44.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:44.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:44.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:44.303 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:44.374 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:30:44.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:30:44.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:30:44.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:30:44.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:30:44.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:30:44.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2954852339' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:44.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:44.403 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:44.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:44.426 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:44.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:44.527 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:44.558 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:44.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:44.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:44.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:30:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:30:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:30:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:30:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:30:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:30:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2954852339' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:44.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:44.681 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:44.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:44.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:44.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:44.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:30:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:30:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:30:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:30:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:30:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2954852339' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:30:44.815 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:30:44.844 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:30:44.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:44.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:44.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:44.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 72859 72843 70.0353 67 0.0350022 0.014272 2024-09-17T16:30:45.039 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:45.073 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:45.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:45.174 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:45.205 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:45.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:45.307 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:45.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:45.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:45.365 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1457,"stamp":"2024-09-17T16:30:44.001031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4549116448,"num_objects":69407,"num_object_clones":0,"num_object_copies":208221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69407,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":69528,"num_write_kb":4444070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2676,"num_bytes_recovered":176170048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69497,"ondisk_log_size":69497,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":18994264,"kb_used_data":13403580,"kb_used_omap":58,"kb_used_meta":5590405,"kb_avail":1105947560,"statfs":{"total":1151940427776,"available":1132490301440,"internally_reserved":0,"allocated":13725265920,"data_stored":13720387037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":5724574883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":630063104,"num_objects":9614,"num_object_clones":0,"num_object_copies":28842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9614,"num_write_kb":615296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.798448"},"pg_stats":[{"pgid":"7.f","version":"660'4311","reported_seq":4383,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:41.422868+0000","last_change":"2024-09-17T16:29:35.616372+0000","last_active":"2024-09-17T16:30:41.422868+0000","last_peered":"2024-09-17T16:30:41.422868+0000","last_clean":"2024-09-17T16:30:41.422868+0000","last_became_active":"2024-09-17T16:29:35.616046+0000","last_became_peered":"2024-09-17T16:29:35.616046+0000","last_unstale":"2024-09-17T16:30:41.422868+0000","last_undegraded":"2024-09-17T16:30:41.422868+0000","last_fullsized":"2024-09-17T16:30:41.422868+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:36.995583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282525696,"num_objects":4311,"num_object_clones":0,"num_object_copies":12933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4311,"num_write_kb":275904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"660'4471","reported_seq":4544,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:41.844184+0000","last_change":"2024-09-17T16:29:35.659373+0000","last_active":"2024-09-17T16:30:41.844184+0000","last_peered":"2024-09-17T16:30:41.844184+0000","last_clean":"2024-09-17T16:30:41.844184+0000","last_became_active":"2024-09-17T16:29:35.659021+0000","last_became_peered":"2024-09-17T16:29:35.659021+0000","last_unstale":"2024-09-17T16:30:41.844184+0000","last_undegraded":"2024-09-17T16:30:41.844184+0000","last_fullsized":"2024-09-17T16:30:41.844184+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:43.706367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293011456,"num_objects":4471,"num_object_clones":0,"num_object_copies":13413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4471,"num_write_kb":286144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"660'4315","reported_seq":4388,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:42.254837+0000","last_change":"2024-09-17T16:29:35.618928+0000","last_active":"2024-09-17T16:30:42.254837+0000","last_peered":"2024-09-17T16:30:42.254837+0000","last_clean":"2024-09-17T16:30:42.254837+0000","last_became_active":"2024-09-17T16:29:35.618644+0000","last_became_peered":"2024-09-17T16:29:35.618644+0000","last_unstale":"2024-09-17T16:30:42.254837+0000","last_undegraded":"2024-09-17T16:30:42.254837+0000","last_fullsized":"2024-09-17T16:30:42.254837+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:50.763262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282787840,"num_objects":4315,"num_object_clones":0,"num_object_copies":12945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4315,"num_write_kb":276160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"660'4374","reported_seq":4446,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:40.607087+0000","last_change":"2024-09-17T16:29:35.617815+0000","last_active":"2024-09-17T16:30:40.607087+0000","last_peered":"2024-09-17T16:30:40.607087+0000","last_clean":"2024-09-17T16:30:40.607087+0000","last_became_active":"2024-09-17T16:29:35.617430+0000","last_became_peered":"2024-09-17T16:29:35.617430+0000","last_unstale":"2024-09-17T16:30:40.607087+0000","last_undegraded":"2024-09-17T16:30:40.607087+0000","last_fullsized":"2024-09-17T16:30:40.607087+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:48.198410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286654464,"num_objects":4374,"num_object_clones":0,"num_object_copies":13122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":279936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"660'4415","reported_seq":4488,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:42.290704+0000","last_change":"2024-09-17T16:29:35.686174+0000","last_active":"2024-09-17T16:30:42.290704+0000","last_peered":"2024-09-17T16:30:42.290704+0000","last_clean":"2024-09-17T16:30:42.290704+0000","last_became_active":"2024-09-17T16:29:35.685764+0000","last_became_peered":"2024-09-17T16:29:35.685764+0000","last_unstale":"2024-09-17T16:30:42.290704+0000","last_undegraded":"2024-09-17T16:30:42.290704+0000","last_fullsized":"2024-09-17T16:30:42.290704+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:57.545117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289341440,"num_objects":4415,"num_object_clones":0,"num_object_copies":13245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4415,"num_write_kb":282560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"660'4299","reported_seq":4370,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:41.933698+0000","last_change":"2024-09-17T16:29:35.686166+0000","last_active":"2024-09-17T16:30:41.933698+0000","last_peered":"2024-09-17T16:30:41.933698+0000","last_clean":"2024-09-17T16:30:41.933698+0000","last_became_active":"2024-09-17T16:29:35.685807+0000","last_became_peered":"2024-09-17T16:29:35.685807+0000","last_unstale":"2024-09-17T16:30:41.933698+0000","last_undegraded":"2024-09-17T16:30:41.933698+0000","last_fullsized":"2024-09-17T16:30:41.933698+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281739264,"num_objects":4299,"num_object_clones":0,"num_object_copies":12897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4299,"num_write_kb":275136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,11],"acting":[2,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"660'4348","reported_seq":4420,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:41.426402+0000","last_change":"2024-09-17T16:29:35.659376+0000","last_active":"2024-09-17T16:30:41.426402+0000","last_peered":"2024-09-17T16:30:41.426402+0000","last_clean":"2024-09-17T16:30:41.426402+0000","last_became_active":"2024-09-17T16:29:35.659034+0000","last_became_peered":"2024-09-17T16:29:35.659034+0000","last_unstale":"2024-09-17T16:30:41.426402+0000","last_undegraded":"2024-09-17T16:30:41.426402+0000","last_fullsized":"2024-09-17T16:30:41.426402+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:52.950535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284950528,"num_objects":4348,"num_object_clones":0,"num_object_copies":13044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4348,"num_write_kb":278272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"660'4293","reported_seq":5729,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:41.866884+0000","last_change":"2024-09-17T16:29:56.602948+0000","last_active":"2024-09-17T16:30:41.866884+0000","last_peered":"2024-09-17T16:30:41.866884+0000","last_clean":"2024-09-17T16:30:41.866884+0000","last_became_active":"2024-09-17T16:29:56.600296+0000","last_became_peered":"2024-09-17T16:29:56.600296+0000","last_unstale":"2024-09-17T16:30:41.866884+0000","last_undegraded":"2024-09-17T16:30:41.866884+0000","last_fullsized":"2024-09-17T16:30:41.866884+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4293,"log_dups_size":0,"ondisk_log_size":4293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:03.960970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281346048,"num_objects":4293,"num_object_clones":0,"num_object_copies":12879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4293,"num_write_kb":274752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":44105728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"660'4193","reported_seq":4265,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:39.493232+0000","last_change":"2024-09-17T16:29:35.616770+0000","last_active":"2024-09-17T16:30:39.493232+0000","last_peered":"2024-09-17T16:30:39.493232+0000","last_clean":"2024-09-17T16:30:39.493232+0000","last_became_active":"2024-09-17T16:29:35.616545+0000","last_became_peered":"2024-09-17T16:29:35.616545+0000","last_unstale":"2024-09-17T16:30:39.493232+0000","last_undegraded":"2024-09-17T16:30:39.493232+0000","last_fullsized":"2024-09-17T16:30:39.493232+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4193,"log_dups_size":0,"ondisk_log_size":4193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:49.218795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274792448,"num_objects":4193,"num_object_clones":0,"num_object_copies":12579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":268352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"660'4339","reported_seq":4412,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:41.253747+0000","last_change":"2024-09-17T16:29:35.686052+0000","last_active":"2024-09-17T16:30:41.253747+0000","last_peered":"2024-09-17T16:30:41.253747+0000","last_clean":"2024-09-17T16:30:41.253747+0000","last_became_active":"2024-09-17T16:29:35.685539+0000","last_became_peered":"2024-09-17T16:29:35.685539+0000","last_unstale":"2024-09-17T16:30:41.253747+0000","last_undegraded":"2024-09-17T16:30:41.253747+0000","last_fullsized":"2024-09-17T16:30:41.253747+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284360704,"num_objects":4339,"num_object_clones":0,"num_object_copies":13017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4339,"num_write_kb":277696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"660'4403","reported_seq":5859,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:41.331207+0000","last_change":"2024-09-17T16:29:56.616980+0000","last_active":"2024-09-17T16:30:41.331207+0000","last_peered":"2024-09-17T16:30:41.331207+0000","last_clean":"2024-09-17T16:30:41.331207+0000","last_became_active":"2024-09-17T16:29:56.600312+0000","last_became_peered":"2024-09-17T16:29:56.600312+0000","last_unstale":"2024-09-17T16:30:41.331207+0000","last_undegraded":"2024-09-17T16:30:41.331207+0000","last_fullsized":"2024-09-17T16:30:41.331207+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:42.145938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288555008,"num_objects":4403,"num_object_clones":0,"num_object_copies":13209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4403,"num_write_kb":281792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":44695552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"660'4402","reported_seq":5830,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:41.626100+0000","last_change":"2024-09-17T16:29:54.745801+0000","last_active":"2024-09-17T16:30:41.626100+0000","last_peered":"2024-09-17T16:30:41.626100+0000","last_clean":"2024-09-17T16:30:41.626100+0000","last_became_active":"2024-09-17T16:29:54.595605+0000","last_became_peered":"2024-09-17T16:29:54.595605+0000","last_unstale":"2024-09-17T16:30:41.626100+0000","last_undegraded":"2024-09-17T16:30:41.626100+0000","last_fullsized":"2024-09-17T16:30:41.626100+0000","mapping_epoch":640,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:37.698855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288489472,"num_objects":4402,"num_object_clones":0,"num_object_copies":13206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4402,"num_write_kb":281728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"660'4302","reported_seq":4375,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:41.712906+0000","last_change":"2024-09-17T16:29:35.659242+0000","last_active":"2024-09-17T16:30:41.712906+0000","last_peered":"2024-09-17T16:30:41.712906+0000","last_clean":"2024-09-17T16:30:41.712906+0000","last_became_active":"2024-09-17T16:29:35.658918+0000","last_became_peered":"2024-09-17T16:29:35.658918+0000","last_unstale":"2024-09-17T16:30:41.712906+0000","last_undegraded":"2024-09-17T16:30:41.712906+0000","last_fullsized":"2024-09-17T16:30:41.712906+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4302,"log_dups_size":0,"ondisk_log_size":4302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:56.097200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281935872,"num_objects":4302,"num_object_clones":0,"num_object_copies":12906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":275328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"660'4260","reported_seq":5638,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:39.966094+0000","last_change":"2024-09-17T16:30:00.594164+0000","last_active":"2024-09-17T16:30:39.966094+0000","last_peered":"2024-09-17T16:30:39.966094+0000","last_clean":"2024-09-17T16:30:39.966094+0000","last_became_active":"2024-09-17T16:30:00.593037+0000","last_became_peered":"2024-09-17T16:30:00.593037+0000","last_unstale":"2024-09-17T16:30:39.966094+0000","last_undegraded":"2024-09-17T16:30:39.966094+0000","last_fullsized":"2024-09-17T16:30:39.966094+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:00.869201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279183360,"num_objects":4260,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4260,"num_write_kb":272640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":42598400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"660'4316","reported_seq":4389,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:40.672601+0000","last_change":"2024-09-17T16:29:35.617588+0000","last_active":"2024-09-17T16:30:40.672601+0000","last_peered":"2024-09-17T16:30:40.672601+0000","last_clean":"2024-09-17T16:30:40.672601+0000","last_became_active":"2024-09-17T16:29:35.617197+0000","last_became_peered":"2024-09-17T16:29:35.617197+0000","last_unstale":"2024-09-17T16:30:40.672601+0000","last_undegraded":"2024-09-17T16:30:40.672601+0000","last_fullsized":"2024-09-17T16:30:40.672601+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:46.697820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282853376,"num_objects":4316,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4316,"num_write_kb":276224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"660'4364","reported_seq":4436,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:41.437802+0000","last_change":"2024-09-17T16:29:35.686727+0000","last_active":"2024-09-17T16:30:41.437802+0000","last_peered":"2024-09-17T16:30:41.437802+0000","last_clean":"2024-09-17T16:30:41.437802+0000","last_became_active":"2024-09-17T16:29:35.686233+0000","last_became_peered":"2024-09-17T16:29:35.686233+0000","last_unstale":"2024-09-17T16:30:41.437802+0000","last_undegraded":"2024-09-17T16:30:41.437802+0000","last_fullsized":"2024-09-17T16:30:41.437802+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4364,"log_dups_size":0,"ondisk_log_size":4364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285999104,"num_objects":4364,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4364,"num_write_kb":279296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":64,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-17T16:30:39.220283+0000","last_change":"2024-09-17T16:29:49.984493+0000","last_active":"2024-09-17T16:30:39.220283+0000","last_peered":"2024-09-17T16:30:39.220283+0000","last_clean":"2024-09-17T16:30:39.220283+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:30:39.220283+0000","last_undegraded":"2024-09-17T16:30:39.220283+0000","last_fullsized":"2024-09-17T16:30:39.220283+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:29:40.527128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:29:40.527128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:10.179932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4548526080,"num_objects":69405,"num_object_clones":0,"num_object_copies":208215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69405,"num_write_kb":4441920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2672,"num_bytes_recovered":175112192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13624934400,"data_stored":13624934400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69405,"ondisk_log_size":69405,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1217400,"kb_used_data":811364,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92527752,"statfs":{"total":95995035648,"available":94748418048,"internally_reserved":0,"allocated":830836736,"data_stored":830430660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":415758788},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547459,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1811064,"kb_used_data":1378660,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91934088,"statfs":{"total":95995035648,"available":94140506112,"internally_reserved":0,"allocated":1411747840,"data_stored":1411341784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743688,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1976248,"kb_used_data":1389732,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91768904,"statfs":{"total":95995035648,"available":93971357696,"internally_reserved":0,"allocated":1423085568,"data_stored":1422679512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":600570304},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005325,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172532,"kb_used_data":802592,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92572620,"statfs":{"total":95995035648,"available":94794362880,"internally_reserved":0,"allocated":821854208,"data_stored":821449065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963802,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1185716,"kb_used_data":709668,"kb_used_omap":41,"kb_used_meta":475990,"kb_avail":92559436,"statfs":{"total":95995035648,"available":94780862464,"internally_reserved":0,"allocated":726700032,"data_stored":726293976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":487414067},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495895,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":716544,"kb_used_data":279980,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93028608,"statfs":{"total":95995035648,"available":95261294592,"internally_reserved":0,"allocated":286699520,"data_stored":286289912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":447019462},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":780596,"kb_used_data":276004,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92964556,"statfs":{"total":95995035648,"available":95195705344,"internally_reserved":0,"allocated":282628096,"data_stored":282222040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":516684230},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986466,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509632,"kb_used_data":1077228,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92235520,"statfs":{"total":95995035648,"available":94449172480,"internally_reserved":0,"allocated":1103081472,"data_stored":1102671864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149989,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2844596,"kb_used_data":2231716,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":90900556,"statfs":{"total":95995035648,"available":93082169344,"internally_reserved":0,"allocated":2285277184,"data_stored":2284871128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":627571143},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411627,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2044664,"kb_used_data":1674724,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91700488,"statfs":{"total":95995035648,"available":93901299712,"internally_reserved":0,"allocated":1714917376,"data_stored":1714511320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575153,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1473204,"kb_used_data":1112996,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92271948,"statfs":{"total":95995035648,"available":94486474752,"internally_reserved":0,"allocated":1139707904,"data_stored":1139301848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738677,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2262068,"kb_used_data":1658916,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":91483084,"statfs":{"total":95995035648,"available":93678678016,"internally_reserved":0,"allocated":1698729984,"data_stored":1698323928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1707016192,"data_stored":1707016192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1147928576,"data_stored":1147928576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1718222848,"data_stored":1718222848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2284257280,"data_stored":2284257280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1118502912,"data_stored":1118502912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":282853376,"data_stored":282853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":286654464,"data_stored":286654464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":578748416,"data_stored":578748416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":826671104,"data_stored":826671104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1422983168,"data_stored":1422983168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1421148160,"data_stored":1421148160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":829947904,"data_stored":829947904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:30:45.366 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_5 pgp_num 16 2024-09-17T16:30:45.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:45.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:45.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:45.558 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:45.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:45 smithi071 ceph-mon[30002]: pgmap v1457: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-17T16:30:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:45 smithi071 ceph-mon[30002]: from='client.17670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:45 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:45 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:30:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:45 smithi071 ceph-mon[30002]: osdmap e661: 12 total, 12 up, 12 in 2024-09-17T16:30:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:45 smithi071 ceph-mon[30002]: osdmap e662: 12 total, 12 up, 12 in 2024-09-17T16:30:45.585 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:45.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:45 smithi173 ceph-mon[30530]: pgmap v1457: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-17T16:30:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:45 smithi173 ceph-mon[30530]: from='client.17670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:45 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:45 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:30:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:45 smithi173 ceph-mon[30530]: osdmap e661: 12 total, 12 up, 12 in 2024-09-17T16:30:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:45 smithi173 ceph-mon[30530]: osdmap e662: 12 total, 12 up, 12 in 2024-09-17T16:30:45.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:45.709 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:45.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:45 smithi017 ceph-mon[27571]: pgmap v1457: 17 pgs: 17 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-17T16:30:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:45 smithi017 ceph-mon[27571]: from='client.17670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:45 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:45.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:45 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:30:45.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:45 smithi017 ceph-mon[27571]: osdmap e661: 12 total, 12 up, 12 in 2024-09-17T16:30:45.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:45 smithi017 ceph-mon[27571]: osdmap e662: 12 total, 12 up, 12 in 2024-09-17T16:30:45.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:45.868 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:45.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:45.934 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:45.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 73732 73716 69.8008 54.5625 0.00736862 0.014319 2024-09-17T16:30:45.949 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:45.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:45.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:45.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:46.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:46.109 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:30:46.109 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:30:46.110 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:30:46.110 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:30:46.110 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:30:46.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:46.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:46.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:46.246 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:46.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:46.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:46.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:46.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:46.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:46.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:46.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:46.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:46.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:46.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:46.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:46.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:46.766 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:46.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:46.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:46.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:46.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:46.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 74748 74732 69.7067 63.5 0.00224227 0.0143253 2024-09-17T16:30:46.985 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:30:46.985 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:30:46.991 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:47.006 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:30:47.006 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:30:47.012 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:47.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:47.031 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:30:47.032 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:30:47.069 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:30:47.069 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:30:47.107 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:30:47.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:47.145 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:30:47.145 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:30:47.153 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:47.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:47.165 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:30:47.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:30:47.177 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:30:47.209 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:30:47.209 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:30:47.245 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:30:47.246 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:30:47.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:47.278 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:30:47.278 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:30:47.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:47.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:47.306 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:30:47.306 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:30:47.358 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 7 pgp_num to 16 2024-09-17T16:30:47.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:47.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:47.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:47.435 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:30:47.461 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:30:47.461 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:30:47.506 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:47.542 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:47.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:47.557 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:47 smithi071 ceph-mon[30002]: pgmap v1460: 17 pgs: 17 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:30:47.557 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4117168572' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:30:47.557 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:47 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:30:47.557 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3532552898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:47.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:47 smithi017 ceph-mon[27571]: pgmap v1460: 17 pgs: 17 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:30:47.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4117168572' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:30:47.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:47 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:30:47.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3532552898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:47.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:47 smithi173 ceph-mon[30530]: pgmap v1460: 17 pgs: 17 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:30:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4117168572' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:30:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:47 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"} : dispatch 2024-09-17T16:30:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3532552898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:47.676 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:47.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:47.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:47.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:47.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:47.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:30:47.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:47.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 75727 75711 69.5813 61.1875 0.0994567 0.014363 2024-09-17T16:30:47.951 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:47.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:47.973 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:47.973 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:48.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:48.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:48.075 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:48.176 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:48.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:48.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:48.300 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:48.332 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:48.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:48.434 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:48.457 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:48.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:48.558 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:48.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:48 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T16:30:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:48 smithi071 ceph-mon[30002]: osdmap e663: 12 total, 12 up, 12 in 2024-09-17T16:30:48.581 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:48.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:48 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T16:30:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:48 smithi173 ceph-mon[30530]: osdmap e663: 12 total, 12 up, 12 in 2024-09-17T16:30:48.683 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:48.713 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:48.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:48.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:48 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-09-17T16:30:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:48 smithi017 ceph-mon[27571]: osdmap e663: 12 total, 12 up, 12 in 2024-09-17T16:30:48.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:30:48.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:48.836 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:48.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:48.906 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:48.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:48.946 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 76489 76473 69.2631 47.625 0.0113033 0.0144323 2024-09-17T16:30:48.980 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:48.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:48.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:49.039 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:30:49.081 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:49.105 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:49.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:49.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:49.229 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:49.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:49.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:49.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:49.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:49.440 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:49 smithi173 ceph-mon[30530]: pgmap v1462: 17 pgs: 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 919 op/s 2024-09-17T16:30:49.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:49.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:49.493 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:49.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:49.575 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:49.576 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:49.594 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:49.619 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:49.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:49.629 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:49 smithi071 ceph-mon[30002]: pgmap v1462: 17 pgs: 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 919 op/s 2024-09-17T16:30:49.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:49 smithi017 ceph-mon[27571]: pgmap v1462: 17 pgs: 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 919 op/s 2024-09-17T16:30:49.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:49.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:49.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:49.886 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:49.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:49.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 77254 77238 68.9565 47.8125 0.00591749 0.0144889 2024-09-17T16:30:49.987 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:50.021 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:50.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:50.123 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:50.151 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:50.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:50.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:50.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:50.276 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:50.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:50.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:30:50.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:50.400 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:50.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:50.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:50.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:50.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:50.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:50.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:30:50.635 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:50 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:50 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:50 smithi173 ceph-mon[30530]: osdmap e664: 12 total, 12 up, 12 in 2024-09-17T16:30:50.652 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:30:50.656 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:50.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:50 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:50 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:50 smithi017 ceph-mon[27571]: osdmap e664: 12 total, 12 up, 12 in 2024-09-17T16:30:50.758 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:50.779 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:50.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:50 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:50 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:50 smithi071 ceph-mon[30002]: osdmap e664: 12 total, 12 up, 12 in 2024-09-17T16:30:50.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:50.903 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:50.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:50.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 78079 78063 68.7115 51.5625 0.0201662 0.0145362 2024-09-17T16:30:51.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:51.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:51.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:51.133 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:51.167 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:51.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:51.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:51.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:51.291 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:51.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:51.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:51.295 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:51.295 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:51.393 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:51.417 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:51.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:51.519 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:51.562 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:51.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:51.629 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:51 smithi173 ceph-mon[30530]: pgmap v1464: 17 pgs: 17 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 949 op/s 2024-09-17T16:30:51.629 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4010739' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:30:51.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:51.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:51.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:51 smithi017 ceph-mon[27571]: pgmap v1464: 17 pgs: 17 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 949 op/s 2024-09-17T16:30:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4010739' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:30:51.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:51.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:51 smithi071 ceph-mon[30002]: pgmap v1464: 17 pgs: 17 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 949 op/s 2024-09-17T16:30:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4010739' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:30:51.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:51.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:51.921 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:51.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 79074 79058 68.6208 62.1875 0.014432 0.0145654 2024-09-17T16:30:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:51.951 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:51.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:51.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:30:52.000 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:30:52.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:52.072 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:52.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:52.175 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:52.197 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:52.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:52.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:52.276 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:30:52.298 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:52.321 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:52.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:52.424 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:52.446 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:52.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:52.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:52.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:52.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:52.578 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:52 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4010739' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:30:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:52 smithi173 ceph-mon[30530]: osdmap e665: 12 total, 12 up, 12 in 2024-09-17T16:30:52.679 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:52.701 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:52.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:52.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:52 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4010739' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:30:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:52 smithi017 ceph-mon[27571]: osdmap e665: 12 total, 12 up, 12 in 2024-09-17T16:30:52.804 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:52.806 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:52.807 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:52 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:30:52.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4010739' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:30:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:52 smithi071 ceph-mon[30002]: osdmap e665: 12 total, 12 up, 12 in 2024-09-17T16:30:52.825 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:52.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:52.927 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:52.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 79864 79848 68.3571 49.375 0.00814972 0.0146236 2024-09-17T16:30:52.964 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:52.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:53.066 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:53.093 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:53.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:53.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:53.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:53.218 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:53.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:53.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:53.354 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:53.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:53.456 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:53.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:53.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:53.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:53.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:30:53.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:53.622 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:53.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:53.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:53 smithi173 ceph-mon[30530]: pgmap v1466: 17 pgs: 17 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-17T16:30:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:53 smithi173 ceph-mon[30530]: 7.4 scrub starts 2024-09-17T16:30:53.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:53.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:53 smithi017 ceph-mon[27571]: pgmap v1466: 17 pgs: 17 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-17T16:30:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:53 smithi017 ceph-mon[27571]: 7.4 scrub starts 2024-09-17T16:30:53.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:53.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:53.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:53 smithi071 ceph-mon[30002]: pgmap v1466: 17 pgs: 17 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-17T16:30:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:53 smithi071 ceph-mon[30002]: 7.4 scrub starts 2024-09-17T16:30:53.820 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:30:53.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:53.866 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:30:53.866 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:30:53.866 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-17T16:30:53.866 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:30:53.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:53.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:53.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 80837 80821 68.255 60.8125 0.00786547 0.0146463 2024-09-17T16:30:53.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:54.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:54.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:54.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:54.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:54.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:54.243 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:54.266 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:54.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:54.368 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:54.384 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:54.384 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:54.390 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:54.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:54.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:54.515 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:54.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:54.617 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:54.638 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:54.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:54 smithi173 ceph-mon[30530]: 7.4 scrub ok 2024-09-17T16:30:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:54 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:54 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:54 smithi017 ceph-mon[27571]: 7.4 scrub ok 2024-09-17T16:30:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:54 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:54 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:54.740 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:54.762 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:54.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:54 smithi071 ceph-mon[30002]: 7.4 scrub ok 2024-09-17T16:30:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:54 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:54 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:54.864 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:54.893 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:54.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:54.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 81803 81787 68.1499 60.375 0.0131289 0.014667 2024-09-17T16:30:54.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:55.018 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:55.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:55.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:30:55.119 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:55.148 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:55.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:55.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:55.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:55.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:55.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:55.373 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:30:55.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:55.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:55.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:55.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:55.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:55.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:55.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:55.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:55 smithi173 ceph-mon[30530]: pgmap v1467: 17 pgs: 17 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 642 op/s 2024-09-17T16:30:55.666 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:55 smithi173 ceph-mon[30530]: osdmap e666: 12 total, 12 up, 12 in 2024-09-17T16:30:55.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:55.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:55.688 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:55.721 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:55 smithi017 ceph-mon[27571]: pgmap v1467: 17 pgs: 17 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 642 op/s 2024-09-17T16:30:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:55 smithi017 ceph-mon[27571]: osdmap e666: 12 total, 12 up, 12 in 2024-09-17T16:30:55.777 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:55.800 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:55.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:55.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:55 smithi071 ceph-mon[30002]: pgmap v1467: 17 pgs: 17 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 642 op/s 2024-09-17T16:30:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:55 smithi071 ceph-mon[30002]: osdmap e666: 12 total, 12 up, 12 in 2024-09-17T16:30:55.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:55.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:55.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:55.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 82678 82662 67.9727 54.6875 0.00326366 0.0146995 2024-09-17T16:30:55.953 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:30:55.954 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:30:56.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:56.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:56.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:56.159 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:56.184 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:56.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:56.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:56.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:56.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:56.415 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:56.457 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:56.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:56.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:56.567 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:30:56.567 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:30:56.580 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:56.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:56.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:56.604 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:30:56.604 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:30:56.642 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:30:56.642 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:30:56.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:56 smithi017 ceph-mon[27571]: 7.7 scrub starts 2024-09-17T16:30:56.679 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:30:56.680 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:30:56.682 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:56.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:56.721 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:56.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:56.723 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:30:56.723 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:30:56.762 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:30:56.762 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:30:56.804 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:30:56.804 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:30:56.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:56 smithi071 ceph-mon[30002]: 7.7 scrub starts 2024-09-17T16:30:56.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:56.841 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:30:56.842 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:30:56.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:56.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:56.879 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:30:56.879 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:30:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:56 smithi173 ceph-mon[30530]: 7.7 scrub starts 2024-09-17T16:30:56.915 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:30:56.916 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:30:56.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 83636 83620 67.8675 59.875 0.0403586 0.0147283 2024-09-17T16:30:56.948 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:56.954 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:30:56.954 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:30:56.970 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:56.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:56.994 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:30:56.994 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:30:57.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:57.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:30:57.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:57.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:57.107 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:30:57.108 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:30:57.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:57.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:57.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:57.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:57.348 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:57.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:57.394 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:30:57.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:57.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:57.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:57.576 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:30:57.597 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:30:57.614 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:57.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:57 smithi017 ceph-mon[27571]: pgmap v1469: 17 pgs: 17 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 813 op/s 2024-09-17T16:30:57.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:57 smithi017 ceph-mon[27571]: 7.7 scrub ok 2024-09-17T16:30:57.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2522829512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:57.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:30:57.665 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:57 smithi173 ceph-mon[30530]: pgmap v1469: 17 pgs: 17 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 813 op/s 2024-09-17T16:30:57.665 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:57 smithi173 ceph-mon[30530]: 7.7 scrub ok 2024-09-17T16:30:57.665 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2522829512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:57.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:57.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:57.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:57.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:57 smithi071 ceph-mon[30002]: pgmap v1469: 17 pgs: 17 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 813 op/s 2024-09-17T16:30:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:57 smithi071 ceph-mon[30002]: 7.7 scrub ok 2024-09-17T16:30:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2522829512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:30:57.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:57.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:57.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:57.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 84349 84333 67.5687 44.5625 0.00824994 0.0147943 2024-09-17T16:30:57.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:30:57.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:57.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:30:58.007 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:30:58.007 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:75081 5.e 5:7e559e6f:::benchmark_data_smithi173_90735_object75080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e470)", 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:11.118325+0000", 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 466.886465509, 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38537186400000001, 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:30:58.008 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75081 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118325+0000", 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118325+0000", 2024-09-17T16:30:58.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118327+0000", 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0849999999999999e-06 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118355+0000", 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7823000000000001e-05 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118376+0000", 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0421e-05 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118386+0000", 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0331e-05 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118442+0000", 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6623999999999999e-05 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118547+0000", 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104461 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.118639+0000", 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2054999999999999e-05 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:30:58.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.120640+0000", 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002000693 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.120805+0000", 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016576899999999999 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.503590+0000", 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38278434099999997 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.503651+0000", 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1233000000000001e-05 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:11.503697+0000", 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6028e-05 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.014 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26158.0:82739 5.e 5:70c6ff22:::benchmark_data_smithi173_90735_object82738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.881300+0000", 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 458.12349084300001, 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65333291699999996, 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26158", 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1732642448", 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 82739 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881300+0000", 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881300+0000", 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881301+0000", 2024-09-17T16:30:58.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.584e-06 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881325+0000", 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3825999999999999e-05 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881340+0000", 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4647999999999999e-05 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881348+0000", 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7819999999999994e-06 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881409+0000", 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1302000000000004e-05 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881527+0000", 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118077 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.881614+0000", 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7399999999999997e-05 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.883793+0000", 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021789309999999998 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.885245+0000", 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014513760000000001 2024-09-17T16:30:58.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534560+0000", 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64931495400000006 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534603+0000", 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3602e-05 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534633+0000", 2024-09-17T16:30:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9434999999999999e-05 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26158.0:82739 5.e e475/419 ondisk, result = 0)", 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.884046+0000", 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 458.12074485699998, 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65059267499999995, 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.884046+0000", 2024-09-17T16:30:58.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.884046+0000", 2024-09-17T16:30:58.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.884046+0000", 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7299999999999996e-07 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.884050+0000", 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.427e-06 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534498+0000", 2024-09-17T16:30:58.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65044812500000004 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534503+0000", 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4609999999999999e-06 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534550+0000", 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6136e-05 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534555+0000", 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.711e-06 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.534639+0000", 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3242e-05 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:13328 6.5 e535/527 6:a9f49cc4:::benchmark_data_smithi173_102017_object13327:head v 535'822, mlcod=535'819)", 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:18.187814+0000", 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 339.816977146, 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20311574499999999, 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.187814+0000", 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.187814+0000", 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.187814+0000", 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1299999999999999e-07 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.187884+0000", 2024-09-17T16:30:58.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9604999999999995e-05 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390246+0000", 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202361655 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390250+0000", 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2629999999999997e-06 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390289+0000", 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8942000000000001e-05 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390304+0000", 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4768e-05 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390899+0000", 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059588800000000004 2024-09-17T16:30:58.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.390929+0000", 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9910999999999999e-05 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:13492 6.4 e535/527 6:27b8e258:::benchmark_data_smithi173_102017_object13491:head v 535'822, mlcod=535'819)", 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:18.460867+0000", 2024-09-17T16:30:58.031 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 339.54392333300001, 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23653454300000001, 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460867+0000", 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460867+0000", 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460868+0000", 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1699999999999998e-07 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460931+0000", 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3336000000000005e-05 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696639+0000", 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23570720000000001 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696643+0000", 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7280000000000003e-06 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696694+0000", 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1141999999999999e-05 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696718+0000", 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3328000000000001e-05 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.697364+0000", 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00064611300000000003 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.697402+0000", 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8078999999999999e-05 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:17620 6.5 e538/527 6:a62c44a9:::benchmark_data_smithi173_102017_object17619:head v 538'1058, mlcod=537'1054)", 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:21.992300+0000", 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 336.01249022899998, 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50495759100000004, 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992300+0000", 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992300+0000", 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992301+0000", 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8500000000000001e-07 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992450+0000", 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149061 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496350+0000", 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50389984499999996 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496361+0000", 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0807e-05 2024-09-17T16:30:58.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496468+0000", 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106811 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496508+0000", 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0079e-05 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.497225+0000", 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00071726500000000002 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.497258+0000", 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3037999999999998e-05 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:17628 6.9 e538/527 6:9546f3b4:::benchmark_data_smithi173_102017_object17627:head v 538'1099, mlcod=537'1096)", 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:21.999183+0000", 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 336.00560741200002, 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45068112500000002, 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999183+0000", 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999183+0000", 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999184+0000", 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3e-07 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999231+0000", 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7206000000000002e-05 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448811+0000", 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44957967100000001 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448826+0000", 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5480000000000001e-05 2024-09-17T16:30:58.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448977+0000", 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015076600000000001 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448996+0000", 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9069999999999999e-05 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.449834+0000", 2024-09-17T16:30:58.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083746999999999999 2024-09-17T16:30:58.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.449864+0000", 2024-09-17T16:30:58.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0732000000000003e-05 2024-09-17T16:30:58.045 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.045 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.045 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:27647 6.a e545/527 6:52a81611:::benchmark_data_smithi173_102017_object27646:head v 545'1782, mlcod=545'1779)", 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:32.709980+0000", 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 325.29481100300001, 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43213955799999998, 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.709980+0000", 2024-09-17T16:30:58.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.709980+0000", 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.709981+0000", 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.23e-07 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.710035+0000", 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4273e-05 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141017+0000", 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43098198300000001 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141021+0000", 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.515e-06 2024-09-17T16:30:58.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141072+0000", 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0322999999999999e-05 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141086+0000", 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4409e-05 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.142088+0000", 2024-09-17T16:30:58.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010016109999999999 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.142119+0000", 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1621000000000001e-05 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:85438 6.f e565/527 6:f29abae7:::benchmark_data_smithi173_102017_object85437:head v 565'5353, mlcod=565'5351)", 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:26:27.791488+0000", 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 270.21330282299999, 2024-09-17T16:30:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.243614578, 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791488+0000", 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791488+0000", 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791489+0000", 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1900000000000001e-07 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791572+0000", 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3040000000000005e-05 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032638+0000", 2024-09-17T16:30:58.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24106640200000001 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032645+0000", 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.55e-06 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032734+0000", 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9209000000000006e-05 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032755+0000", 2024-09-17T16:30:58.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0618000000000001e-05 2024-09-17T16:30:58.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.035062+0000", 2024-09-17T16:30:58.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023077140000000002 2024-09-17T16:30:58.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.035102+0000", 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0126000000000002e-05 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:87629 6.9 e566/527 6:975355a4:::benchmark_data_smithi173_102017_object87628:head v 566'5443, mlcod=566'5441)", 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:26:30.268719+0000", 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 267.73607214399999, 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22957050300000001, 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.055 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268719+0000", 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268719+0000", 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268721+0000", 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.12e-06 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268753+0000", 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2620999999999998e-05 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494429+0000", 2024-09-17T16:30:58.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22567617500000001 2024-09-17T16:30:58.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494433+0000", 2024-09-17T16:30:58.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3000000000000002e-06 2024-09-17T16:30:58.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494476+0000", 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2908999999999999e-05 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494495+0000", 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9335999999999999e-05 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.498267+0000", 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037720879999999998 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.498289+0000", 2024-09-17T16:30:58.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1954e-05 2024-09-17T16:30:58.060 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.060 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.060 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.060 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:114687 6.9 e582/527 6:96e26e86:::benchmark_data_smithi173_102017_object21539:head v 582'7080, mlcod=582'7078)", 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:12.569600+0000", 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 225.43519071, 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24650798600000001, 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569600+0000", 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569600+0000", 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569601+0000", 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.569999999999999e-07 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569607+0000", 2024-09-17T16:30:58.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2380000000000002e-06 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812466+0000", 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24285875200000001 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812471+0000", 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9960000000000001e-06 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812530+0000", 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9333999999999998e-05 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812551+0000", 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1158999999999999e-05 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.816069+0000", 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035176840000000001 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.816108+0000", 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8865999999999997e-05 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:115587 6.5 e582/527 6:ae192cb4:::benchmark_data_smithi173_102017_object22439:head v 582'7137, mlcod=582'7134)", 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:14.230950+0000", 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 223.77384081100001, 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29268783799999998, 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.065 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230950+0000", 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230950+0000", 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230951+0000", 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5700000000000001e-07 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230956+0000", 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6210000000000001e-06 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520676+0000", 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.289719173 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520684+0000", 2024-09-17T16:30:58.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.833e-06 2024-09-17T16:30:58.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520770+0000", 2024-09-17T16:30:58.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5866999999999996e-05 2024-09-17T16:30:58.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520790+0000", 2024-09-17T16:30:58.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9266000000000002e-05 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.523606+0000", 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028161639999999999 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.523638+0000", 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2057000000000002e-05 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.070 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:116184 6.f e583/527 6:f2e9c488:::benchmark_data_smithi173_102017_object23036:head v 583'7277, mlcod=583'7273)", 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:29.905958+0000", 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 208.098832453, 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23584665899999999, 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905958+0000", 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905958+0000", 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905960+0000", 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.282e-06 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905967+0000", 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.61e-06 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136208+0000", 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23024043699999999 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136213+0000", 2024-09-17T16:30:58.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.955e-06 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136316+0000", 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103074 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136341+0000", 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5809e-05 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.141768+0000", 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054267589999999997 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.141805+0000", 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6733000000000002e-05 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:186285 6.f e621/527 6:ff66d09f:::benchmark_data_smithi173_102017_object93137:head v 621'11698, mlcod=621'11694)", 2024-09-17T16:30:58.075 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:17.540727+0000", 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 100.464063903, 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53518362600000002, 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540727+0000", 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540727+0000", 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540728+0000", 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.361e-06 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540742+0000", 2024-09-17T16:30:58.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4087000000000001e-05 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.071882+0000", 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53114022100000002 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.071897+0000", 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4039000000000001e-05 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.072016+0000", 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011962499999999999 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.072036+0000", 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0035999999999999e-05 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.075868+0000", 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038314 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.075910+0000", 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2857e-05 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:10923 7.a e637/629 7:59531872:::benchmark_data_smithi173_116396_object10922:head v 637'660, mlcod=636'657)", 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:48.036837+0000", 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 69.967953890000004, 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22698770900000001, 2024-09-17T16:30:58.080 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.036837+0000", 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.036837+0000", 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.036839+0000", 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.722e-06 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.036878+0000", 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9056e-05 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.263099+0000", 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22622150799999999 2024-09-17T16:30:58.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.263103+0000", 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3789999999999999e-06 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.263152+0000", 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8316000000000003e-05 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.263174+0000", 2024-09-17T16:30:58.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1715000000000001e-05 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.263793+0000", 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00061959600000000004 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.263825+0000", 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1417000000000001e-05 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.084 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:21072 7.5 7:a6915c4f:::benchmark_data_smithi173_116396_object21071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:58.907216+0000", 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 59.097574424000001, 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32350631499999999, 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:58.085 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21072 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907216+0000", 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907216+0000", 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907218+0000", 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.638e-06 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907236+0000", 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7849000000000001e-05 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907248+0000", 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1933000000000001e-05 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907254+0000", 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0689999999999998e-06 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907308+0000", 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3940000000000002e-05 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907441+0000", 2024-09-17T16:30:58.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133685 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907543+0000", 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101369 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.908430+0000", 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088757899999999999 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.908686+0000", 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025581999999999997 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230610+0000", 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32192386200000001 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230673+0000", 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2668000000000004e-05 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230723+0000", 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9903e-05 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:21129 7.5 7:af8cedc1:::benchmark_data_smithi173_116396_object21128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:59.022800+0000", 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.981990629000002, 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20906143999999999, 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.092 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.092 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:30:58.092 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:30:58.092 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:58.092 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21129 2024-09-17T16:30:58.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.092 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.022800+0000", 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.022800+0000", 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.022801+0000", 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3790000000000001e-06 2024-09-17T16:30:58.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.022825+0000", 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3472000000000002e-05 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.022840+0000", 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5118e-05 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.022849+0000", 2024-09-17T16:30:58.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4679999999999992e-06 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.022927+0000", 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8212000000000003e-05 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.023043+0000", 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115956 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:30:58.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.023127+0000", 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3981999999999997e-05 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.024220+0000", 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001093484 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.025005+0000", 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00078454399999999997 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.231796+0000", 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20679097900000001 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.231832+0000", 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6055e-05 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.231862+0000", 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9791e-05 2024-09-17T16:30:58.097 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27088.0:21072 7.5 e643/629 ondisk, result = 0)", 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:59.024245+0000", 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.980545497000001, 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20648555800000001, 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.024245+0000", 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.024245+0000", 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.024246+0000", 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8800000000000002e-07 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.024249+0000", 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3019999999999999e-06 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230525+0000", 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20627600300000001 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230531+0000", 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1800000000000001e-06 2024-09-17T16:30:58.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230595+0000", 2024-09-17T16:30:58.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4064000000000002e-05 2024-09-17T16:30:58.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230603+0000", 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8129999999999996e-06 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230731+0000", 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000127608 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27088.0:21129 7.5 e643/629 ondisk, result = 0)", 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:59.024934+0000", 2024-09-17T16:30:58.102 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.979856841, 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20693272200000001, 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.024934+0000", 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.024934+0000", 2024-09-17T16:30:58.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.024934+0000", 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8699999999999999e-07 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.024935+0000", 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.263e-06 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.231745+0000", 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20680923600000001 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.231749+0000", 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6689999999999997e-06 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.231787+0000", 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8009999999999997e-05 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.231792+0000", 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.118e-06 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.231867+0000", 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4239000000000005e-05 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.106 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:21141 7.5 7:a860228b:::benchmark_data_smithi173_116396_object21140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:59.030274+0000", 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.974517136000003, 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20378579299999999, 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21141 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:30:58.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.030274+0000", 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.030274+0000", 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.030275+0000", 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6470000000000001e-06 2024-09-17T16:30:58.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.030332+0000", 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6988999999999999e-05 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.030347+0000", 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4290999999999999e-05 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.030353+0000", 2024-09-17T16:30:58.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4999999999999996e-06 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.030407+0000", 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4351000000000002e-05 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.030562+0000", 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015507800000000001 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:30:58.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.030722+0000", 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015945999999999999 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.031737+0000", 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001014643 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.032060+0000", 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032366399999999998 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.233956+0000", 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20189596600000001 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.234012+0000", 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6174000000000001e-05 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.234059+0000", 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7030000000000002e-05 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.112 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.113 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.113 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:30:58.113 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:30:58.113 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:30:58.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:30:58.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:58.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:30:58.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:58.274 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:58.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:58.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:58.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:58.398 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:58.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:58 smithi017 ceph-mon[27571]: from='client.17700 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:58 smithi017 ceph-mon[27571]: 7.8 scrub starts 2024-09-17T16:30:58.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:30:58.522 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:58.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:58.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:30:58.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:30:58.645 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:58.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:30:58.734 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1469,"stamp":"2024-09-17T16:30:56.006793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5220008480,"num_objects":79644,"num_object_clones":0,"num_object_copies":238932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79644,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":79765,"num_write_kb":5099238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2676,"num_bytes_recovered":176170048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79734,"ondisk_log_size":79734,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":21057908,"kb_used_data":15417936,"kb_used_omap":58,"kb_used_meta":5638981,"kb_avail":1103883916,"statfs":{"total":1151940427776,"available":1130377129984,"internally_reserved":0,"allocated":15787966464,"data_stored":15783537549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":5774316707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":353173504,"num_objects":5389,"num_object_clones":0,"num_object_copies":16167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5389,"num_write_kb":344896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.627507"},"pg_stats":[{"pgid":"7.f","version":"664'4917","reported_seq":4998,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-09-17T16:30:51.451363+0000","last_change":"2024-09-17T16:29:35.616372+0000","last_active":"2024-09-17T16:30:51.451363+0000","last_peered":"2024-09-17T16:30:51.451363+0000","last_clean":"2024-09-17T16:30:51.451363+0000","last_became_active":"2024-09-17T16:29:35.616046+0000","last_became_peered":"2024-09-17T16:29:35.616046+0000","last_unstale":"2024-09-17T16:30:51.451363+0000","last_undegraded":"2024-09-17T16:30:51.451363+0000","last_fullsized":"2024-09-17T16:30:51.451363+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:51.095153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322240512,"num_objects":4917,"num_object_clones":0,"num_object_copies":14751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4917,"num_write_kb":314688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"665'5011","reported_seq":5095,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-09-17T16:30:51.893166+0000","last_change":"2024-09-17T16:29:35.659373+0000","last_active":"2024-09-17T16:30:51.893166+0000","last_peered":"2024-09-17T16:30:51.893166+0000","last_clean":"2024-09-17T16:30:51.893166+0000","last_became_active":"2024-09-17T16:29:35.659021+0000","last_became_peered":"2024-09-17T16:29:35.659021+0000","last_unstale":"2024-09-17T16:30:51.893166+0000","last_undegraded":"2024-09-17T16:30:51.893166+0000","last_fullsized":"2024-09-17T16:30:51.893166+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328400896,"num_objects":5011,"num_object_clones":0,"num_object_copies":15033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":320704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"665'4855","reported_seq":4939,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-09-17T16:30:52.273627+0000","last_change":"2024-09-17T16:29:35.618928+0000","last_active":"2024-09-17T16:30:52.273627+0000","last_peered":"2024-09-17T16:30:52.273627+0000","last_clean":"2024-09-17T16:30:52.273627+0000","last_became_active":"2024-09-17T16:29:35.618644+0000","last_became_peered":"2024-09-17T16:29:35.618644+0000","last_unstale":"2024-09-17T16:30:52.273627+0000","last_undegraded":"2024-09-17T16:30:52.273627+0000","last_fullsized":"2024-09-17T16:30:52.273627+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:51.555722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318177280,"num_objects":4855,"num_object_clones":0,"num_object_copies":14565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":310720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"666'5234","reported_seq":5320,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-09-17T16:30:55.635271+0000","last_change":"2024-09-17T16:29:35.617815+0000","last_active":"2024-09-17T16:30:55.635271+0000","last_peered":"2024-09-17T16:30:55.635271+0000","last_clean":"2024-09-17T16:30:55.635271+0000","last_became_active":"2024-09-17T16:29:35.617430+0000","last_became_peered":"2024-09-17T16:29:35.617430+0000","last_unstale":"2024-09-17T16:30:55.635271+0000","last_undegraded":"2024-09-17T16:30:55.635271+0000","last_fullsized":"2024-09-17T16:30:55.635271+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5234,"log_dups_size":0,"ondisk_log_size":5234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:48.321444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343015424,"num_objects":5234,"num_object_clones":0,"num_object_copies":15702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":334976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"665'5029","reported_seq":5113,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-09-17T16:30:52.281043+0000","last_change":"2024-09-17T16:29:35.686174+0000","last_active":"2024-09-17T16:30:52.281043+0000","last_peered":"2024-09-17T16:30:52.281043+0000","last_clean":"2024-09-17T16:30:52.281043+0000","last_became_active":"2024-09-17T16:29:35.685764+0000","last_became_peered":"2024-09-17T16:29:35.685764+0000","last_unstale":"2024-09-17T16:30:52.281043+0000","last_undegraded":"2024-09-17T16:30:52.281043+0000","last_fullsized":"2024-09-17T16:30:52.281043+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5029,"log_dups_size":0,"ondisk_log_size":5029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:04.181228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329580544,"num_objects":5029,"num_object_clones":0,"num_object_copies":15087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":321856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"665'4886","reported_seq":4970,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-09-17T16:30:51.904387+0000","last_change":"2024-09-17T16:29:35.686166+0000","last_active":"2024-09-17T16:30:51.904387+0000","last_peered":"2024-09-17T16:30:51.904387+0000","last_clean":"2024-09-17T16:30:51.904387+0000","last_became_active":"2024-09-17T16:29:35.685807+0000","last_became_peered":"2024-09-17T16:29:35.685807+0000","last_unstale":"2024-09-17T16:30:51.904387+0000","last_undegraded":"2024-09-17T16:30:51.904387+0000","last_fullsized":"2024-09-17T16:30:51.904387+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:51.591230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320208896,"num_objects":4886,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4886,"num_write_kb":312704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,11],"acting":[2,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"664'4939","reported_seq":5021,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-09-17T16:30:51.454609+0000","last_change":"2024-09-17T16:29:35.659376+0000","last_active":"2024-09-17T16:30:51.454609+0000","last_peered":"2024-09-17T16:30:51.454609+0000","last_clean":"2024-09-17T16:30:51.454609+0000","last_became_active":"2024-09-17T16:29:35.659034+0000","last_became_peered":"2024-09-17T16:29:35.659034+0000","last_unstale":"2024-09-17T16:30:51.454609+0000","last_undegraded":"2024-09-17T16:30:51.454609+0000","last_fullsized":"2024-09-17T16:30:51.454609+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4939,"log_dups_size":0,"ondisk_log_size":4939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:50.084410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323682304,"num_objects":4939,"num_object_clones":0,"num_object_copies":14817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4939,"num_write_kb":316096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"665'4874","reported_seq":6322,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-09-17T16:30:51.896175+0000","last_change":"2024-09-17T16:29:56.602948+0000","last_active":"2024-09-17T16:30:51.896175+0000","last_peered":"2024-09-17T16:30:51.896175+0000","last_clean":"2024-09-17T16:30:51.896175+0000","last_became_active":"2024-09-17T16:29:56.600296+0000","last_became_peered":"2024-09-17T16:29:56.600296+0000","last_unstale":"2024-09-17T16:30:51.896175+0000","last_undegraded":"2024-09-17T16:30:51.896175+0000","last_fullsized":"2024-09-17T16:30:51.896175+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4874,"log_dups_size":0,"ondisk_log_size":4874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319422464,"num_objects":4874,"num_object_clones":0,"num_object_copies":14622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4874,"num_write_kb":311936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":44105728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"666'4976","reported_seq":5062,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-09-17T16:30:54.477986+0000","last_change":"2024-09-17T16:29:35.616770+0000","last_active":"2024-09-17T16:30:54.477986+0000","last_peered":"2024-09-17T16:30:54.477986+0000","last_clean":"2024-09-17T16:30:54.477986+0000","last_became_active":"2024-09-17T16:29:35.616545+0000","last_became_peered":"2024-09-17T16:29:35.616545+0000","last_unstale":"2024-09-17T16:30:54.477986+0000","last_undegraded":"2024-09-17T16:30:54.477986+0000","last_fullsized":"2024-09-17T16:30:54.477986+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326107136,"num_objects":4976,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":318464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"664'4897","reported_seq":4979,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-09-17T16:30:51.343660+0000","last_change":"2024-09-17T16:29:35.686052+0000","last_active":"2024-09-17T16:30:51.343660+0000","last_peered":"2024-09-17T16:30:51.343660+0000","last_clean":"2024-09-17T16:30:51.343660+0000","last_became_active":"2024-09-17T16:29:35.685539+0000","last_became_peered":"2024-09-17T16:29:35.685539+0000","last_unstale":"2024-09-17T16:30:51.343660+0000","last_undegraded":"2024-09-17T16:30:51.343660+0000","last_fullsized":"2024-09-17T16:30:51.343660+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4897,"log_dups_size":0,"ondisk_log_size":4897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:51.280804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320929792,"num_objects":4897,"num_object_clones":0,"num_object_copies":14691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4897,"num_write_kb":313408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"664'4985","reported_seq":6450,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-09-17T16:30:51.443331+0000","last_change":"2024-09-17T16:29:56.616980+0000","last_active":"2024-09-17T16:30:51.443331+0000","last_peered":"2024-09-17T16:30:51.443331+0000","last_clean":"2024-09-17T16:30:51.443331+0000","last_became_active":"2024-09-17T16:29:56.600312+0000","last_became_peered":"2024-09-17T16:29:56.600312+0000","last_unstale":"2024-09-17T16:30:51.443331+0000","last_undegraded":"2024-09-17T16:30:51.443331+0000","last_fullsized":"2024-09-17T16:30:51.443331+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4985,"log_dups_size":0,"ondisk_log_size":4985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326696960,"num_objects":4985,"num_object_clones":0,"num_object_copies":14955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4985,"num_write_kb":319040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":44695552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"665'4954","reported_seq":6393,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-09-17T16:30:51.741044+0000","last_change":"2024-09-17T16:29:54.745801+0000","last_active":"2024-09-17T16:30:51.741044+0000","last_peered":"2024-09-17T16:30:51.741044+0000","last_clean":"2024-09-17T16:30:51.741044+0000","last_became_active":"2024-09-17T16:29:54.595605+0000","last_became_peered":"2024-09-17T16:29:54.595605+0000","last_unstale":"2024-09-17T16:30:51.741044+0000","last_undegraded":"2024-09-17T16:30:51.741044+0000","last_fullsized":"2024-09-17T16:30:51.741044+0000","mapping_epoch":640,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4954,"log_dups_size":0,"ondisk_log_size":4954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:48.844412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324665344,"num_objects":4954,"num_object_clones":0,"num_object_copies":14862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4954,"num_write_kb":317056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"665'4896","reported_seq":4980,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-09-17T16:30:51.757081+0000","last_change":"2024-09-17T16:29:35.659242+0000","last_active":"2024-09-17T16:30:51.757081+0000","last_peered":"2024-09-17T16:30:51.757081+0000","last_clean":"2024-09-17T16:30:51.757081+0000","last_became_active":"2024-09-17T16:29:35.658918+0000","last_became_peered":"2024-09-17T16:29:35.658918+0000","last_unstale":"2024-09-17T16:30:51.757081+0000","last_undegraded":"2024-09-17T16:30:51.757081+0000","last_fullsized":"2024-09-17T16:30:51.757081+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:47.848857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320864256,"num_objects":4896,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4896,"num_write_kb":313344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"666'5106","reported_seq":6496,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-09-17T16:30:54.953711+0000","last_change":"2024-09-17T16:30:00.594164+0000","last_active":"2024-09-17T16:30:54.953711+0000","last_peered":"2024-09-17T16:30:54.953711+0000","last_clean":"2024-09-17T16:30:54.953711+0000","last_became_active":"2024-09-17T16:30:00.593037+0000","last_became_peered":"2024-09-17T16:30:00.593037+0000","last_unstale":"2024-09-17T16:30:54.953711+0000","last_undegraded":"2024-09-17T16:30:54.953711+0000","last_fullsized":"2024-09-17T16:30:54.953711+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5106,"log_dups_size":0,"ondisk_log_size":5106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:48.292945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334626816,"num_objects":5106,"num_object_clones":0,"num_object_copies":15318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5106,"num_write_kb":326784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":42598400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"666'5119","reported_seq":5204,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-09-17T16:30:55.665660+0000","last_change":"2024-09-17T16:29:35.617588+0000","last_active":"2024-09-17T16:30:55.665660+0000","last_peered":"2024-09-17T16:30:55.665660+0000","last_clean":"2024-09-17T16:30:55.665660+0000","last_became_active":"2024-09-17T16:29:35.617197+0000","last_became_peered":"2024-09-17T16:29:35.617197+0000","last_unstale":"2024-09-17T16:30:55.665660+0000","last_undegraded":"2024-09-17T16:30:55.665660+0000","last_fullsized":"2024-09-17T16:30:55.665660+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5119,"log_dups_size":0,"ondisk_log_size":5119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:51.436418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335478784,"num_objects":5119,"num_object_clones":0,"num_object_copies":15357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5119,"num_write_kb":327616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"664'4964","reported_seq":5046,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-09-17T16:30:51.448807+0000","last_change":"2024-09-17T16:29:35.686727+0000","last_active":"2024-09-17T16:30:51.448807+0000","last_peered":"2024-09-17T16:30:51.448807+0000","last_clean":"2024-09-17T16:30:51.448807+0000","last_became_active":"2024-09-17T16:29:35.686233+0000","last_became_peered":"2024-09-17T16:29:35.686233+0000","last_unstale":"2024-09-17T16:30:51.448807+0000","last_undegraded":"2024-09-17T16:30:51.448807+0000","last_fullsized":"2024-09-17T16:30:51.448807+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:01.056974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325320704,"num_objects":4964,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4964,"num_write_kb":317696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":76,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-09-17T16:30:54.395070+0000","last_change":"2024-09-17T16:29:49.984493+0000","last_active":"2024-09-17T16:30:54.395070+0000","last_peered":"2024-09-17T16:30:54.395070+0000","last_clean":"2024-09-17T16:30:54.395070+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:30:54.395070+0000","last_undegraded":"2024-09-17T16:30:54.395070+0000","last_fullsized":"2024-09-17T16:30:54.395070+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:29:40.527128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:29:40.527128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:10.179932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5219418112,"num_objects":79642,"num_object_clones":0,"num_object_copies":238926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79642,"num_write_kb":5097088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2672,"num_bytes_recovered":175112192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15734407168,"data_stored":15734407168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79642,"ondisk_log_size":79642,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1399372,"kb_used_data":980676,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92345780,"statfs":{"total":95995035648,"available":94562078720,"internally_reserved":0,"allocated":1004212224,"data_stored":1003909740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":428538308},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547462,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2088716,"kb_used_data":1651972,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91656436,"statfs":{"total":95995035648,"available":93856190464,"internally_reserved":0,"allocated":1691619328,"data_stored":1691316864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743690,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2162276,"kb_used_data":1567444,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91582876,"statfs":{"total":95995035648,"available":93780865024,"internally_reserved":0,"allocated":1605062656,"data_stored":1604638930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":609089984},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005328,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1370692,"kb_used_data":970228,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92374460,"statfs":{"total":95995035648,"available":94591447040,"internally_reserved":0,"allocated":993513472,"data_stored":993146669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1232756,"kb_used_data":764844,"kb_used_omap":41,"kb_used_meta":467670,"kb_avail":92512396,"statfs":{"total":95995035648,"available":94732693504,"internally_reserved":0,"allocated":783200256,"data_stored":782837005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":478894387},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495898,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":759764,"kb_used_data":335500,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92985388,"statfs":{"total":95995035648,"available":95217037312,"internally_reserved":0,"allocated":343552000,"data_stored":343245984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":434239942},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831380,"kb_used_data":326788,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92913772,"statfs":{"total":95995035648,"available":95143702528,"internally_reserved":0,"allocated":334630912,"data_stored":334328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":516684230},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986469,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1747780,"kb_used_data":1301492,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":91997372,"statfs":{"total":95995035648,"available":94205308928,"internally_reserved":0,"allocated":1332727808,"data_stored":1332361005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149991,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3101924,"kb_used_data":2489044,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":90643228,"statfs":{"total":95995035648,"available":92818665472,"internally_reserved":0,"allocated":2548781056,"data_stored":2548357330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":627571143},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411629,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2275940,"kb_used_data":1893524,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91469212,"statfs":{"total":95995035648,"available":93664473088,"internally_reserved":0,"allocated":1938968576,"data_stored":1938544850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575155,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1608552,"kb_used_data":1256660,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92136600,"statfs":{"total":95995035648,"available":94347878400,"internally_reserved":0,"allocated":1286819840,"data_stored":1286396114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738679,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2478756,"kb_used_data":1879764,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91266396,"statfs":{"total":95995035648,"available":93456789504,"internally_reserved":0,"allocated":1924878336,"data_stored":1924454610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1934557184,"data_stored":1934557184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1295450112,"data_stored":1295450112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1945436160,"data_stored":1945436160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2577924096,"data_stored":2577924096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1341849600,"data_stored":1341849600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":335478784,"data_stored":335478784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":343146496,"data_stored":343146496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":654049280,"data_stored":654049280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":995098624,"data_stored":995098624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1606615040,"data_stored":1606615040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1700724736,"data_stored":1700724736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1004077056,"data_stored":1004077056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:30:58.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-17T16:30:58.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:58.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:58.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:58.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:58 smithi173 ceph-mon[30530]: from='client.17700 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:58.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:58 smithi173 ceph-mon[30530]: 7.8 scrub starts 2024-09-17T16:30:58.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:58 smithi071 ceph-mon[30002]: from='client.17700 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:30:58.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:58 smithi071 ceph-mon[30002]: 7.8 scrub starts 2024-09-17T16:30:58.829 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:58.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:30:58.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:58.897 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:30:58.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 85329 85313 67.4886 61.25 0.0145923 0.0148109 2024-09-17T16:30:58.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:30:59.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:59.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:30:59.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:59.160 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:59.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:59.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:59.262 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:59.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:30:59.296 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:59.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:59.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:59.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:30:59.421 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:59.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:30:59.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:30:59.545 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:30:59.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:59.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:30:59.558 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:30:59.602 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:30:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:59 smithi173 ceph-mon[30530]: pgmap v1470: 17 pgs: 17 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-17T16:30:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:59 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:30:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:59 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:30:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:30:59 smithi173 ceph-mon[30530]: osdmap e667: 12 total, 12 up, 12 in 2024-09-17T16:30:59.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:30:59.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:30:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:59.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:30:59.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:59 smithi017 ceph-mon[27571]: pgmap v1470: 17 pgs: 17 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-17T16:30:59.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:59 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:30:59.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:59 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:30:59.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:30:59 smithi017 ceph-mon[27571]: osdmap e667: 12 total, 12 up, 12 in 2024-09-17T16:30:59.777 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:30:59.813 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:59.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:30:59.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:59 smithi071 ceph-mon[30002]: pgmap v1470: 17 pgs: 17 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-17T16:30:59.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:59 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:30:59.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:59 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:30:59.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:30:59 smithi071 ceph-mon[30002]: osdmap e667: 12 total, 12 up, 12 in 2024-09-17T16:30:59.915 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:30:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:30:59.945 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:30:59.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:30:59.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:30:59.946544+0000 min lat: 0.00119669 max lat: 2.03842 avg lat: 0.0148409 2024-09-17T16:30:59.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:30:59.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 86220 86204 67.341 55.6875 0.0670079 0.0148409 2024-09-17T16:31:00.047 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:00.070 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:31:00.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:00.071 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:00.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:00.145 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:31:00.145 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:31:00.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:00.145 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:86408 7.b e667/629 7:dd8ebcb3:::benchmark_data_smithi173_116396_object86407:head v 667'5457, mlcod=667'5455)", 2024-09-17T16:31:00.145 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:00.140799+0000", 2024-09-17T16:31:00.145 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0020021980000000002, 2024-09-17T16:31:00.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020220400000000001, 2024-09-17T16:31:00.145 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:00.140799+0000", 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:00.140799+0000", 2024-09-17T16:31:00.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:00.140800+0000", 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.006e-06 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:00.140867+0000", 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7343999999999998e-05 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:00.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:00.140887+0000", 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0086e-05 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:00.140903+0000", 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5832e-05 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:00.140961+0000", 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8260999999999997e-05 2024-09-17T16:31:00.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:00.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:00.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:00.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:00.140982+0000", 2024-09-17T16:31:00.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0489999999999999e-05 2024-09-17T16:31:00.149 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:00.149 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:00.149 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:00.149 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:00.149 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:31:00.149 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-09-17T16:31:00.150 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:31:00.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:00.211 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:00.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:00.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:00.337 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:00.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:00.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:00.421 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2609946087' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:00.436 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:31:00.438 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:00.498 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:00.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:00.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:00.622 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:00.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:00.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2609946087' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:00 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:00.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:00 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:00.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:00.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:00.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2609946087' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:00.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:00 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:00.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:00.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:00.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:00.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:31:00.947 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 87199 87183 67.265 61.1875 0.0347041 0.0148608 2024-09-17T16:31:00.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:00.999 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:01.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:01.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:01.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:01.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:01.167 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:01.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:01.260 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:01.261 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:31:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:01.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:01.362 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:01.384 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:01.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:01.432 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 1 pgp_num to 1 2024-09-17T16:31:01.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:01.515 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:01.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:01.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:01.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:01.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:01.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:01.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:01 smithi173 ceph-mon[30530]: pgmap v1472: 17 pgs: 17 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 702 op/s 2024-09-17T16:31:01.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:01 smithi173 ceph-mon[30530]: 7.8 scrub ok 2024-09-17T16:31:01.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:01 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:31:01.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:01 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:31:01.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:01 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:31:01.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:01 smithi173 ceph-mon[30530]: osdmap e668: 12 total, 12 up, 12 in 2024-09-17T16:31:01.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/689471411' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-17T16:31:01.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:01 smithi173 ceph-mon[30530]: 7.e scrub starts 2024-09-17T16:31:01.721 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:01 smithi017 ceph-mon[27571]: pgmap v1472: 17 pgs: 17 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 702 op/s 2024-09-17T16:31:01.721 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:01 smithi017 ceph-mon[27571]: 7.8 scrub ok 2024-09-17T16:31:01.721 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:01 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:31:01.721 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:01 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:31:01.721 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:01 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:31:01.721 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:01 smithi017 ceph-mon[27571]: osdmap e668: 12 total, 12 up, 12 in 2024-09-17T16:31:01.722 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/689471411' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-17T16:31:01.722 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:01 smithi017 ceph-mon[27571]: 7.e scrub starts 2024-09-17T16:31:01.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:01.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:01.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:01 smithi071 ceph-mon[30002]: pgmap v1472: 17 pgs: 17 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 702 op/s 2024-09-17T16:31:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:01 smithi071 ceph-mon[30002]: 7.8 scrub ok 2024-09-17T16:31:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:01 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:31:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:01 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:31:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:01 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:31:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:01 smithi071 ceph-mon[30002]: osdmap e668: 12 total, 12 up, 12 in 2024-09-17T16:31:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/689471411' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-17T16:31:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:01 smithi071 ceph-mon[30002]: 7.e scrub starts 2024-09-17T16:31:01.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:01.903 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:01.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:01.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 88032 88016 67.0796 52.0625 0.0370466 0.0149024 2024-09-17T16:31:01.971 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:31:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:31:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:31:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:31:01.971 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:31:02.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:02.046 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:02.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:02.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:02.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:02.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:02.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:02.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:02.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:02.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:02.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:02.407 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:02.444 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:02.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:02.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:02.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:02.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:02.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:02.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:02.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/689471411' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-17T16:31:02.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:02 smithi017 ceph-mon[27571]: osdmap e669: 12 total, 12 up, 12 in 2024-09-17T16:31:02.672 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:02.694 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:02.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:02.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:02.796 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:02.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/689471411' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-17T16:31:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:02 smithi071 ceph-mon[30002]: osdmap e669: 12 total, 12 up, 12 in 2024-09-17T16:31:02.861 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:02.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/689471411' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-17T16:31:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:02 smithi173 ceph-mon[30530]: osdmap e669: 12 total, 12 up, 12 in 2024-09-17T16:31:02.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 89019 89003 67.0145 61.6875 0.0218822 0.0149159 2024-09-17T16:31:02.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:02.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:02.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:03.008 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:31:03.008 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:31:03.008 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:31:03.008 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:31:03.009 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:31:03.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:31:03.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:31:03.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:31:03.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:31:03.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:31:03.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:31:03.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:31:03.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:31:03.010 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:31:03.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:03.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:03.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:03.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:03.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:03.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:03.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:03.342 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:03.377 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:03.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:03.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:03.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:03.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:03.607 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:03.636 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:03.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:03.722 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:03 smithi173 ceph-mon[30530]: pgmap v1475: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-17T16:31:03.723 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:03 smithi173 ceph-mon[30530]: 7.e scrub ok 2024-09-17T16:31:03.723 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:03 smithi173 ceph-mon[30530]: 7.6 scrub starts 2024-09-17T16:31:03.723 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3648911028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:03.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:31:03.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:03 smithi017 ceph-mon[27571]: pgmap v1475: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-17T16:31:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:03 smithi017 ceph-mon[27571]: 7.e scrub ok 2024-09-17T16:31:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:03 smithi017 ceph-mon[27571]: 7.6 scrub starts 2024-09-17T16:31:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3648911028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:03.774 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:31:03.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:03.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:03 smithi071 ceph-mon[30002]: pgmap v1475: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-17T16:31:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:03 smithi071 ceph-mon[30002]: 7.e scrub ok 2024-09-17T16:31:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:03 smithi071 ceph-mon[30002]: 7.6 scrub starts 2024-09-17T16:31:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3648911028' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:03.893 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:03.916 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:03.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:03.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:03.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 89925 89909 66.8908 56.625 0.0153708 0.0149444 2024-09-17T16:31:04.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:04.045 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:31:04.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:04.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:04.049 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:04.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:04.171 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:04.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:04.199 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:04.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:04.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:04.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:04.300 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:04.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:04.442 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:04.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:04.544 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:04.599 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:04.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:04.604 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:31:04.604 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:31:04.700 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:04.723 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:04.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:04.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:04.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:04.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:04.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 91023 91007 66.9111 68.625 0.00495484 0.0149403 2024-09-17T16:31:04.963 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:04.997 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:04.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:05.099 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:05.124 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:05.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:05.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:05.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:05.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:05.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:05.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:05.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:05.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:05.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:31:05.458 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:31:05.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:05 smithi173 ceph-mon[30530]: 7.6 scrub ok 2024-09-17T16:31:05.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:05 smithi173 ceph-mon[30530]: pgmap v1476: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 741 op/s 2024-09-17T16:31:05.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:05 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:31:05.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:05 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:31:05.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:05 smithi173 ceph-mon[30530]: osdmap e670: 12 total, 12 up, 12 in 2024-09-17T16:31:05.461 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1812168613' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:31:05.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:05.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:05.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:05.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:05.630 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:05.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:05.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:05 smithi017 ceph-mon[27571]: 7.6 scrub ok 2024-09-17T16:31:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:05 smithi017 ceph-mon[27571]: pgmap v1476: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 741 op/s 2024-09-17T16:31:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:05 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:31:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:05 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:31:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:05 smithi017 ceph-mon[27571]: osdmap e670: 12 total, 12 up, 12 in 2024-09-17T16:31:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1812168613' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:31:05.763 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:31:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:05.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:05.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:05 smithi071 ceph-mon[30002]: 7.6 scrub ok 2024-09-17T16:31:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:05 smithi071 ceph-mon[30002]: pgmap v1476: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 741 op/s 2024-09-17T16:31:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:05 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:31:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:05 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:31:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:05 smithi071 ceph-mon[30002]: osdmap e670: 12 total, 12 up, 12 in 2024-09-17T16:31:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1812168613' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:31:05.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:31:05.866 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:05.887 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:05.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:05.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 15 91908 91893 66.7769 55.375 0.0138552 0.0149696 2024-09-17T16:31:05.989 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:06.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:06.010 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:06.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:06.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:06.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:06.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:06.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:06.284 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:06.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:06.317 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:31:06.317 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:31:06.326 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:06.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:06.410 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:06.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:06.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:06.535 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:06.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:06.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:06.658 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:06.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:06 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:31:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:06 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:31:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:06 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:31:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1812168613' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:31:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:06 smithi017 ceph-mon[27571]: osdmap e671: 12 total, 12 up, 12 in 2024-09-17T16:31:06.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:06.784 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:06.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:06 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:31:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:06 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:31:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:06 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:31:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1812168613' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:31:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:06 smithi071 ceph-mon[30002]: osdmap e671: 12 total, 12 up, 12 in 2024-09-17T16:31:06.886 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:06.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:06 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:31:06.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:06 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:31:06.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:06 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:31:06.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1812168613' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:31:06.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:06 smithi173 ceph-mon[30530]: osdmap e671: 12 total, 12 up, 12 in 2024-09-17T16:31:06.908 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:06.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:06.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 92787 92771 66.6401 54.875 0.0162681 0.0150004 2024-09-17T16:31:07.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:07.040 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:07.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:07.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:07.142 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:07.165 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:07.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:07.267 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:07.302 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:07.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:07.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:07.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:31:07.394 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:31:07.394 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:31:07.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:07.425 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:31:07.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:31:07.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:07.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:07.457 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:31:07.457 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:31:07.495 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:31:07.495 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:31:07.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:07 smithi017 ceph-mon[27571]: pgmap v1479: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 907 op/s 2024-09-17T16:31:07.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1204726035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:07.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:31:07.529 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:31:07.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:07.565 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:07.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:07.566 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:31:07.567 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:31:07.599 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:31:07.600 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:31:07.632 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:31:07.633 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:31:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:07 smithi173 ceph-mon[30530]: pgmap v1479: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 907 op/s 2024-09-17T16:31:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1204726035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:07.665 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:31:07.665 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:31:07.666 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:31:07.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:07.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:31:07.694 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:31:07.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:07.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:07.719 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:31:07.719 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:31:07.752 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:31:07.752 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:31:07.797 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:07 smithi071 ceph-mon[30002]: pgmap v1479: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 907 op/s 2024-09-17T16:31:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1204726035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:07.819 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:07.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:07.846 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:31:07.846 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:31:07.846 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_5 2024-09-17T16:31:07.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:31:07.920 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:07.941 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:07.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:07.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 93721 93705 66.5461 58.375 0.023434 0.0150225 2024-09-17T16:31:08.043 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:08.080 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:08.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:08.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:08.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:08.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:08.261 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:31:08.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:08.332 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:08.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:08.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:08.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:08.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:08.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:08.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:08.586 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:31:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:31:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:31:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:31:08.586 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:31:08.588 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:08.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:08.677 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:08.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:08.710 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:08.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:08.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:08.817 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:08.858 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:08.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:08.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 94730 94714 66.5069 63.0625 0.00906612 0.0150312 2024-09-17T16:31:08.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:09.005 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:09.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:09.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:09.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:09.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:09.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:09.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:09.279 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:09.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:09.381 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:09.406 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:09.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:09.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:09.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:09.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:09.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:31:09.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:09.665 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:09.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:09 smithi017 ceph-mon[27571]: pgmap v1480: 17 pgs: 17 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-17T16:31:09.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:09.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:09.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:09.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:09 smithi071 ceph-mon[30002]: pgmap v1480: 17 pgs: 17 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-17T16:31:09.893 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:09.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:09 smithi173 ceph-mon[30530]: pgmap v1480: 17 pgs: 17 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-17T16:31:09.897 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:31:09.917 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:09.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:09.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:09.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 11 95597 95586 66.3734 54.5 0.00699959 0.0150602 2024-09-17T16:31:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:31:10.008 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:31:10.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:10.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:10.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:10.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:10.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:10.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:10.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:10.289 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:10.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:10.391 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:10.414 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:10.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:10.415 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:31:10.415 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25516.0:204335 4.9 e408/383 4:99bf4a76:::benchmark_data_smithi173_77474_object102160:head v 408'12572, mlcod=408'12569)", 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:23.607078+0000", 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.80246792200001, 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77592090000000002, 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607078+0000", 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607078+0000", 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607079+0000", 2024-09-17T16:31:10.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8199999999999998e-07 2024-09-17T16:31:10.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:23.607086+0000", 2024-09-17T16:31:10.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9600000000000003e-06 2024-09-17T16:31:10.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.378990+0000", 2024-09-17T16:31:10.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77190383900000004 2024-09-17T16:31:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.379000+0000", 2024-09-17T16:31:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:31:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.379065+0000", 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5287e-05 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.379088+0000", 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3334999999999999e-05 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.382963+0000", 2024-09-17T16:31:10.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038744439999999999 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:24.382999+0000", 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6715999999999999e-05 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:2279 5.f e424/419 5:f5772ab5:::benchmark_data_smithi173_90735_object2278:head v 424'148, mlcod=424'146)", 2024-09-17T16:31:10.422 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:21:51.502595+0000", 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 558.906950983, 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25253113999999999, 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502595+0000", 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502595+0000", 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502597+0000", 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.311e-06 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.502640+0000", 2024-09-17T16:31:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3847000000000002e-05 2024-09-17T16:31:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754667+0000", 2024-09-17T16:31:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25202682199999998 2024-09-17T16:31:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754673+0000", 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1489999999999999e-06 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754725+0000", 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2009999999999998e-05 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.754741+0000", 2024-09-17T16:31:10.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5798e-05 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.755084+0000", 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034313300000000002 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:21:51.755126+0000", 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2070000000000002e-05 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.427 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:15935 5.f e429/419 5:f3ff7eb6:::benchmark_data_smithi173_90735_object15934:head v 429'990, mlcod=429'987)", 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:02.437369+0000", 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 547.97217747599996, 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79522149499999994, 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.428 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437369+0000", 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437369+0000", 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437370+0000", 2024-09-17T16:31:10.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9400000000000005e-07 2024-09-17T16:31:10.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:02.437426+0000", 2024-09-17T16:31:10.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6798000000000001e-05 2024-09-17T16:31:10.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231438+0000", 2024-09-17T16:31:10.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79401129999999998 2024-09-17T16:31:10.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231449+0000", 2024-09-17T16:31:10.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1688000000000001e-05 2024-09-17T16:31:10.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231596+0000", 2024-09-17T16:31:10.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000146546 2024-09-17T16:31:10.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.231620+0000", 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4315999999999999e-05 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.232554+0000", 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093371600000000002 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:03.232590+0000", 2024-09-17T16:31:10.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6436999999999997e-05 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:54616 5.17 e450/444 5:eee108da:::benchmark_data_smithi173_90735_object54615:head v 450'3019, mlcod=0'0)", 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:39.740381+0000", 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 510.66916555900002, 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58705162700000002, 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.434 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740381+0000", 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740381+0000", 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740382+0000", 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.249e-06 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:39.740443+0000", 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0998000000000001e-05 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325597+0000", 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58515414300000002 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325606+0000", 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5760000000000006e-06 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325653+0000", 2024-09-17T16:31:10.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.774e-05 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.325673+0000", 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9253e-05 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.327389+0000", 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001716104 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:40.327432+0000", 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3563999999999998e-05 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:63127 5.f e461/455 5:f7e3e25a:::benchmark_data_smithi173_90735_object63126:head v 461'3883, mlcod=425'480)", 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:22:55.384543+0000", 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 495.02500289800003, 2024-09-17T16:31:10.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23654767900000001, 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384543+0000", 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384543+0000", 2024-09-17T16:31:10.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384545+0000", 2024-09-17T16:31:10.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.339e-06 2024-09-17T16:31:10.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.384593+0000", 2024-09-17T16:31:10.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8733000000000002e-05 2024-09-17T16:31:10.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619021+0000", 2024-09-17T16:31:10.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23442753799999999 2024-09-17T16:31:10.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619029+0000", 2024-09-17T16:31:10.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7270000000000007e-06 2024-09-17T16:31:10.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619090+0000", 2024-09-17T16:31:10.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1680999999999994e-05 2024-09-17T16:31:10.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.619117+0000", 2024-09-17T16:31:10.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7013e-05 2024-09-17T16:31:10.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.621043+0000", 2024-09-17T16:31:10.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001925679 2024-09-17T16:31:10.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:22:55.621091+0000", 2024-09-17T16:31:10.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7969000000000001e-05 2024-09-17T16:31:10.446 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.446 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.446 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.446 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:77257 5.17 e472/470 5:ee127e34:::benchmark_data_smithi173_90735_object77256:head v 472'3755, mlcod=472'3752)", 2024-09-17T16:31:10.446 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:13.811785+0000", 2024-09-17T16:31:10.447 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 476.59776142099997, 2024-09-17T16:31:10.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.255431142, 2024-09-17T16:31:10.447 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.447 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.447 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.447 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811785+0000", 2024-09-17T16:31:10.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811785+0000", 2024-09-17T16:31:10.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811786+0000", 2024-09-17T16:31:10.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8700000000000005e-07 2024-09-17T16:31:10.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:13.811954+0000", 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016799800000000001 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065415+0000", 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25346157200000002 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065422+0000", 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3969999999999999e-06 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065477+0000", 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5699000000000003e-05 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.065491+0000", 2024-09-17T16:31:10.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4069e-05 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.067186+0000", 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001694922 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:14.067216+0000", 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9697999999999999e-05 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.452 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:82762 5.d e475/455 5:bc524506:::benchmark_data_smithi173_90735_object82761:head v 475'5183, mlcod=439'2263)", 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:19.917268+0000", 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 470.49227869399999, 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57018889800000006, 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917268+0000", 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917268+0000", 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917269+0000", 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1480000000000001e-06 2024-09-17T16:31:10.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:19.917322+0000", 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3705000000000002e-05 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485165+0000", 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.56784279500000001 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485171+0000", 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0639999999999999e-06 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485221+0000", 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9313e-05 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.485242+0000", 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1049000000000001e-05 2024-09-17T16:31:10.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.487416+0000", 2024-09-17T16:31:10.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021741849999999999 2024-09-17T16:31:10.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:20.487457+0000", 2024-09-17T16:31:10.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0639000000000002e-05 2024-09-17T16:31:10.457 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:114282 5.d e489/482 5:bfd83ddc:::benchmark_data_smithi173_90735_object31511:head v 489'7194, mlcod=489'7192)", 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:23:53.959997+0000", 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 436.44954908300002, 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33615526200000001, 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.959997+0000", 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.959997+0000", 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.959999+0000", 2024-09-17T16:31:10.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.61e-06 2024-09-17T16:31:10.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:53.960006+0000", 2024-09-17T16:31:10.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3170000000000004e-06 2024-09-17T16:31:10.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292652+0000", 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33264596400000002 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292658+0000", 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9859999999999999e-06 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292772+0000", 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011348799999999999 2024-09-17T16:31:10.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.292793+0000", 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1630000000000001e-05 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.296107+0000", 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003314141 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:23:54.296152+0000", 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5126000000000001e-05 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:158720 5.f e512/481 5:f5ceb914:::benchmark_data_smithi173_90735_object75949:head v 512'9748, mlcod=512'9745)", 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 391.55155146999999, 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.233160072, 2024-09-17T16:31:10.463 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857996+0000", 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7399999999999991e-07 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.858002+0000", 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5030000000000002e-06 2024-09-17T16:31:10.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087206+0000", 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22920367999999999 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087213+0000", 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0129999999999996e-06 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087267+0000", 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3978999999999999e-05 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087289+0000", 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1597999999999999e-05 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.091102+0000", 2024-09-17T16:31:10.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038130719999999998 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.091155+0000", 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3253e-05 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:160454 5.17 e514/470 5:eb54973c:::benchmark_data_smithi173_90735_object77683:head v 514'6398, mlcod=514'6395)", 2024-09-17T16:31:10.468 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 389.67891176099999, 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18539098100000001, 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:31:10.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:31:10.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2900000000000002e-07 2024-09-17T16:31:10.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730643+0000", 2024-09-17T16:31:10.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3300000000000001e-06 2024-09-17T16:31:10.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913880+0000", 2024-09-17T16:31:10.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.183237224 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913887+0000", 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2400000000000001e-06 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913943+0000", 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5859999999999997e-05 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913958+0000", 2024-09-17T16:31:10.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4854999999999999e-05 2024-09-17T16:31:10.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.915994+0000", 2024-09-17T16:31:10.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00203572 2024-09-17T16:31:10.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.916026+0000", 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1822999999999997e-05 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:12912 6.d e535/527 6:ba9121f7:::benchmark_data_smithi173_102017_object12911:head v 535'804, mlcod=535'801)", 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 352.70571627200002, 2024-09-17T16:31:10.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27481599400000001, 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703831+0000", 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3340000000000001e-06 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703859+0000", 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7898e-05 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977854+0000", 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27399516899999998 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977863+0000", 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6309999999999992e-06 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977932+0000", 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.923e-05 2024-09-17T16:31:10.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977956+0000", 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3606000000000001e-05 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.978613+0000", 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065716699999999995 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.978646+0000", 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2959000000000001e-05 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:13282 6.d e535/527 6:b884e4f4:::benchmark_data_smithi173_102017_object13281:head v 535'823, mlcod=535'820)", 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:18.148010+0000", 2024-09-17T16:31:10.480 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 352.261535847, 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.174234948, 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148010+0000", 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148010+0000", 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148011+0000", 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9699999999999994e-07 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.148035+0000", 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3946000000000001e-05 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321530+0000", 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17349503999999999 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321535+0000", 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9930000000000003e-06 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321583+0000", 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7472000000000002e-05 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.321598+0000", 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5305e-05 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.322200+0000", 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00060203700000000004 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.322245+0000", 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5157999999999997e-05 2024-09-17T16:31:10.484 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:113603 6.8 e582/527 6:1d7478f4:::benchmark_data_smithi173_102017_object20455:head v 582'7045, mlcod=582'7042)", 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 239.79982006099999, 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.183501209, 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609728+0000", 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.176e-06 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609742+0000", 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3703000000000001e-05 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790389+0000", 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18064696399999999 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790402+0000", 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3239e-05 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790530+0000", 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012778799999999999 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790567+0000", 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6999000000000003e-05 2024-09-17T16:31:10.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.793200+0000", 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026325070000000001 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.793227+0000", 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7832999999999999e-05 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.490 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:21165 7.a e643/629 7:5cfcee13:::benchmark_data_smithi173_116396_object21164:head v 643'1303, mlcod=643'1299)", 2024-09-17T16:31:10.490 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:59.104257+0000", 2024-09-17T16:31:10.490 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 71.305289000000002, 2024-09-17T16:31:10.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14482610500000001, 2024-09-17T16:31:10.490 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.490 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.490 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.490 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.104257+0000", 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.104257+0000", 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.104258+0000", 2024-09-17T16:31:10.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7199999999999998e-07 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.104312+0000", 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3897000000000003e-05 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.247419+0000", 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14310688599999999 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.247431+0000", 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2128e-05 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.247574+0000", 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014306 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.247596+0000", 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1560999999999999e-05 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.249039+0000", 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014438470000000001 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.249083+0000", 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4054000000000002e-05 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.494 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:21246 7.a e643/629 7:54f6627e:::benchmark_data_smithi173_116396_object21245:head v 643'1308, mlcod=643'1304)", 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:59.262208+0000", 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 71.147337859000004, 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15227691099999999, 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.262208+0000", 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.262208+0000", 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.262209+0000", 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9800000000000003e-07 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.262284+0000", 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4741999999999997e-05 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.413509+0000", 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15122511999999999 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.413517+0000", 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2360000000000004e-06 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.413574+0000", 2024-09-17T16:31:10.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6984000000000002e-05 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.413591+0000", 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6449000000000001e-05 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.414452+0000", 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086084499999999995 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.414485+0000", 2024-09-17T16:31:10.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3736999999999999e-05 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:24959 7.b e645/629 7:d9e245de:::benchmark_data_smithi173_116396_object24958:head v 645'1612, mlcod=645'1608)", 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:03.105540+0000", 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 67.304006096999998, 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.169251445, 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.499 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:03.105540+0000", 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:03.105540+0000", 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:03.105541+0000", 2024-09-17T16:31:10.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1100000000000005e-07 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:03.105607+0000", 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5832000000000003e-05 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:03.273711+0000", 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16810393000000001 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:03.273720+0000", 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3090000000000001e-06 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:03.273776+0000", 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5690999999999999e-05 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:03.273797+0000", 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1369000000000002e-05 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:03.274756+0000", 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00095853799999999999 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:03.274792+0000", 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5964999999999999e-05 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:29912 7.a e646/629 7:59a868d4:::benchmark_data_smithi173_116396_object29911:head v 646'1855, mlcod=646'1851)", 2024-09-17T16:31:10.503 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:07.193928+0000", 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.215617922, 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17504670999999999, 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.193928+0000", 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.193928+0000", 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.193929+0000", 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0800000000000003e-07 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.193986+0000", 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6860000000000001e-05 2024-09-17T16:31:10.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.367935+0000", 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17394890399999999 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.367940+0000", 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6619999999999996e-06 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.368000+0000", 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0129e-05 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.368020+0000", 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0040000000000001e-05 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.368937+0000", 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091735799999999995 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.368975+0000", 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7849000000000003e-05 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:29928 7.a e646/629 7:54337644:::benchmark_data_smithi173_116396_object29927:head v 646'1856, mlcod=646'1852)", 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.119345766000002, 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.232420081, 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.508 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290207+0000", 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6579999999999996e-06 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290242+0000", 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4696000000000001e-05 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521561+0000", 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23131880499999999 2024-09-17T16:31:10.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521567+0000", 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.392e-06 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521628+0000", 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0473999999999997e-05 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521650+0000", 2024-09-17T16:31:10.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2909e-05 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.522588+0000", 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093747100000000005 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.522621+0000", 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2675999999999997e-05 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.512 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.513 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.513 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:29955 7.a e646/629 7:5743185d:::benchmark_data_smithi173_116396_object29954:head v 646'1857, mlcod=646'1853)", 2024-09-17T16:31:10.513 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:07.367863+0000", 2024-09-17T16:31:10.513 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.041683399, 2024-09-17T16:31:10.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.166931687, 2024-09-17T16:31:10.514 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.514 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.514 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.514 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.367863+0000", 2024-09-17T16:31:10.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.367863+0000", 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.367864+0000", 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6500000000000008e-07 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.367920+0000", 2024-09-17T16:31:10.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5962999999999999e-05 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.533373+0000", 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.165452671 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.533380+0000", 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9850000000000006e-06 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.533450+0000", 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9807000000000004e-05 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.533479+0000", 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8319000000000001e-05 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.534721+0000", 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001241968 2024-09-17T16:31:10.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.534795+0000", 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4009000000000002e-05 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:46467 7.b e650/629 7:d098567c:::benchmark_data_smithi173_116396_object46466:head v 650'2942, mlcod=650'2939)", 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 49.245899944999998, 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.213043449, 2024-09-17T16:31:10.519 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163647+0000", 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163708+0000", 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1152e-05 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.374934+0000", 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21122532999999999 2024-09-17T16:31:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.374948+0000", 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4756000000000001e-05 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.375068+0000", 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119771 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.375108+0000", 2024-09-17T16:31:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9619000000000001e-05 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.376653+0000", 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001545416 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.376690+0000", 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6631000000000002e-05 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:31:10.524 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:31:10.524 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:31:10.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:10.547 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:10.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:10.617 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:31:10.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:31:10.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:10.697 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:10.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:10.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:10.799 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:10.822 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:10.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3746454810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:10 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:10 smithi173 ceph-mon[30530]: from='client.27247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:31:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:10 smithi173 ceph-mon[30530]: pgmap v1481: 17 pgs: 17 active+clean; 5.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 710 op/s 2024-09-17T16:31:10.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:10.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:10.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:10.948 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.8386 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 95597 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 65.7739 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 18.1341 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 121.938 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 2.125 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1052 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 290.146 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 1951 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 34 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0150993 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.0357509 2024-09-17T16:31:10.949 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 2.03842 2024-09-17T16:31:10.950 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00119669 2024-09-17T16:31:10.953 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T16:31:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3746454810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:10 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:10 smithi017 ceph-mon[27571]: from='client.27247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:31:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:10 smithi017 ceph-mon[27571]: pgmap v1481: 17 pgs: 17 active+clean; 5.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 710 op/s 2024-09-17T16:31:11.047 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:11.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3746454810' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:11.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:10 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:11.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:10 smithi071 ceph-mon[30002]: from='client.27247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:31:11.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:10 smithi071 ceph-mon[30002]: pgmap v1481: 17 pgs: 17 active+clean; 5.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 710 op/s 2024-09-17T16:31:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:11.077 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:11.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:11.178 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:11.201 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:11.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:11.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:11.324 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:11.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:11.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:11.426 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:11.448 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:11.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:11.491 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:31:11.550 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:11.587 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:11.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:11.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:11.689 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:11.721 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:11.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:11.793 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:31:11.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:11 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:31:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:11 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:31:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:11 smithi071 ceph-mon[30002]: osdmap e672: 12 total, 12 up, 12 in 2024-09-17T16:31:11.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:11.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:11.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:11.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:11 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:31:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:11 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:31:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:11 smithi173 ceph-mon[30530]: osdmap e672: 12 total, 12 up, 12 in 2024-09-17T16:31:11.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:11.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:11.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:11.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:11 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:31:11.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:11 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:31:11.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:11 smithi017 ceph-mon[27571]: osdmap e672: 12 total, 12 up, 12 in 2024-09-17T16:31:12.077 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:12.101 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:12.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:12.202 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:12.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:12.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:12.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:12.335 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:12.357 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:12.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:12.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:12.460 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:12.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:12.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:97238 7.6 e672/629 7:680bb552:::benchmark_data_smithi173_116396_object1638:head v 672'6109, mlcod=672'6107)", 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:31:12.539964+0000", 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0013116919999999999, 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013308009999999999, 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:31:12.544 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:12.539964+0000", 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:12.539964+0000", 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:31:12.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:12.539964+0000", 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6199999999999996e-07 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:12.539971+0000", 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7390000000000002e-06 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:31:12.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:12.539998+0000", 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6299999999999999e-05 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:12.540005+0000", 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3810000000000001e-06 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:12.540062+0000", 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7117999999999998e-05 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:31:12.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:31:12.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:31:12.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:12.540078+0000", 2024-09-17T16:31:12.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5784e-05 2024-09-17T16:31:12.548 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:31:12.548 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:31:12.548 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:31:12.548 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:31:12.548 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:31:12.548 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 1 2024-09-17T16:31:12.548 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:31:12.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:12.626 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:12.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:12.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:12.716 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1480,"stamp":"2024-09-17T16:31:08.009383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6047728160,"num_objects":92274,"num_object_clones":0,"num_object_copies":276822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92274,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":92395,"num_write_kb":5907558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2676,"num_bytes_recovered":176170048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92364,"ondisk_log_size":92364,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":23388968,"kb_used_data":17699664,"kb_used_omap":58,"kb_used_meta":5688965,"kb_avail":1101552856,"statfs":{"total":1151940427776,"available":1127990124544,"internally_reserved":0,"allocated":18124455936,"data_stored":18121073404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":5825500323},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":512557056,"num_objects":7821,"num_object_clones":0,"num_object_copies":23463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7821,"num_write_kb":500544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.575581"},"pg_stats":[{"pgid":"7.f","version":"671'5831","reported_seq":5927,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:06.444424+0000","last_change":"2024-09-17T16:29:35.616372+0000","last_active":"2024-09-17T16:31:06.444424+0000","last_peered":"2024-09-17T16:31:06.444424+0000","last_clean":"2024-09-17T16:31:06.444424+0000","last_became_active":"2024-09-17T16:29:35.616046+0000","last_became_peered":"2024-09-17T16:29:35.616046+0000","last_unstale":"2024-09-17T16:31:06.444424+0000","last_undegraded":"2024-09-17T16:31:06.444424+0000","last_fullsized":"2024-09-17T16:31:06.444424+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5831,"log_dups_size":0,"ondisk_log_size":5831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:51.095153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382140416,"num_objects":5831,"num_object_clones":0,"num_object_copies":17493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5831,"num_write_kb":373184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"671'5876","reported_seq":6035,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:06.896011+0000","last_change":"2024-09-17T16:31:02.091075+0000","last_active":"2024-09-17T16:31:06.896011+0000","last_peered":"2024-09-17T16:31:06.896011+0000","last_clean":"2024-09-17T16:31:06.896011+0000","last_became_active":"2024-09-17T16:29:35.659021+0000","last_became_peered":"2024-09-17T16:29:35.659021+0000","last_unstale":"2024-09-17T16:31:06.896011+0000","last_undegraded":"2024-09-17T16:31:06.896011+0000","last_fullsized":"2024-09-17T16:31:06.896011+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"669'5588","last_scrub_stamp":"2024-09-17T16:31:02.090927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:02.090927+0000","objects_scrubbed":5548,"log_size":5876,"log_dups_size":0,"ondisk_log_size":5876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:32:12.750432+0000","scrub_duration":1454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385089536,"num_objects":5876,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":376064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"671'5715","reported_seq":5811,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:07.289976+0000","last_change":"2024-09-17T16:29:35.618928+0000","last_active":"2024-09-17T16:31:07.289976+0000","last_peered":"2024-09-17T16:31:07.289976+0000","last_clean":"2024-09-17T16:31:07.289976+0000","last_became_active":"2024-09-17T16:29:35.618644+0000","last_became_peered":"2024-09-17T16:29:35.618644+0000","last_unstale":"2024-09-17T16:31:07.289976+0000","last_undegraded":"2024-09-17T16:31:07.289976+0000","last_fullsized":"2024-09-17T16:31:07.289976+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:51.555722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374538240,"num_objects":5715,"num_object_clones":0,"num_object_copies":17145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5715,"num_write_kb":365760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"671'5770","reported_seq":5865,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:05.618527+0000","last_change":"2024-09-17T16:29:35.617815+0000","last_active":"2024-09-17T16:31:05.618527+0000","last_peered":"2024-09-17T16:31:05.618527+0000","last_clean":"2024-09-17T16:31:05.618527+0000","last_became_active":"2024-09-17T16:29:35.617430+0000","last_became_peered":"2024-09-17T16:29:35.617430+0000","last_unstale":"2024-09-17T16:31:05.618527+0000","last_undegraded":"2024-09-17T16:31:05.618527+0000","last_fullsized":"2024-09-17T16:31:05.618527+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:48.321444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378142720,"num_objects":5770,"num_object_clones":0,"num_object_copies":17310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":369280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"671'5872","reported_seq":5967,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:07.288335+0000","last_change":"2024-09-17T16:29:35.686174+0000","last_active":"2024-09-17T16:31:07.288335+0000","last_peered":"2024-09-17T16:31:07.288335+0000","last_clean":"2024-09-17T16:31:07.288335+0000","last_became_active":"2024-09-17T16:29:35.685764+0000","last_became_peered":"2024-09-17T16:29:35.685764+0000","last_unstale":"2024-09-17T16:31:07.288335+0000","last_undegraded":"2024-09-17T16:31:07.288335+0000","last_fullsized":"2024-09-17T16:31:07.288335+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5872,"log_dups_size":0,"ondisk_log_size":5872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384827392,"num_objects":5872,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":375808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"671'5781","reported_seq":5877,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:06.913687+0000","last_change":"2024-09-17T16:29:35.686166+0000","last_active":"2024-09-17T16:31:06.913687+0000","last_peered":"2024-09-17T16:31:06.913687+0000","last_clean":"2024-09-17T16:31:06.913687+0000","last_became_active":"2024-09-17T16:29:35.685807+0000","last_became_peered":"2024-09-17T16:29:35.685807+0000","last_unstale":"2024-09-17T16:31:06.913687+0000","last_undegraded":"2024-09-17T16:31:06.913687+0000","last_fullsized":"2024-09-17T16:31:06.913687+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:51.591230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378863616,"num_objects":5781,"num_object_clones":0,"num_object_copies":17343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5781,"num_write_kb":369984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,11],"acting":[2,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"671'5780","reported_seq":5876,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:06.457795+0000","last_change":"2024-09-17T16:29:35.659376+0000","last_active":"2024-09-17T16:31:06.457795+0000","last_peered":"2024-09-17T16:31:06.457795+0000","last_clean":"2024-09-17T16:31:06.457795+0000","last_became_active":"2024-09-17T16:29:35.659034+0000","last_became_peered":"2024-09-17T16:29:35.659034+0000","last_unstale":"2024-09-17T16:31:06.457795+0000","last_undegraded":"2024-09-17T16:31:06.457795+0000","last_fullsized":"2024-09-17T16:31:06.457795+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:50.084410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378798080,"num_objects":5780,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5780,"num_write_kb":369920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"671'5746","reported_seq":7269,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:06.929237+0000","last_change":"2024-09-17T16:31:00.025431+0000","last_active":"2024-09-17T16:31:06.929237+0000","last_peered":"2024-09-17T16:31:06.929237+0000","last_clean":"2024-09-17T16:31:06.929237+0000","last_became_active":"2024-09-17T16:29:56.600296+0000","last_became_peered":"2024-09-17T16:29:56.600296+0000","last_unstale":"2024-09-17T16:31:06.929237+0000","last_undegraded":"2024-09-17T16:31:06.929237+0000","last_fullsized":"2024-09-17T16:31:06.929237+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"667'5344","last_scrub_stamp":"2024-09-17T16:31:00.025251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:00.025251+0000","objects_scrubbed":5255,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:32:29.335483+0000","scrub_duration":2516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376569856,"num_objects":5746,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5746,"num_write_kb":367744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":673,"num_bytes_recovered":44105728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"670'5536","reported_seq":5688,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-09-17T16:31:04.459968+0000","last_change":"2024-09-17T16:30:56.450257+0000","last_active":"2024-09-17T16:31:04.459968+0000","last_peered":"2024-09-17T16:31:04.459968+0000","last_clean":"2024-09-17T16:31:04.459968+0000","last_became_active":"2024-09-17T16:29:35.616545+0000","last_became_peered":"2024-09-17T16:29:35.616545+0000","last_unstale":"2024-09-17T16:31:04.459968+0000","last_undegraded":"2024-09-17T16:31:04.459968+0000","last_fullsized":"2024-09-17T16:31:04.459968+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"666'5094","last_scrub_stamp":"2024-09-17T16:30:56.450199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:30:56.450199+0000","objects_scrubbed":5055,"log_size":5536,"log_dups_size":0,"ondisk_log_size":5536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:32:20.872355+0000","scrub_duration":1222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362807296,"num_objects":5536,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5536,"num_write_kb":354304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.5","version":"671'5768","reported_seq":5864,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:06.346481+0000","last_change":"2024-09-17T16:29:35.686052+0000","last_active":"2024-09-17T16:31:06.346481+0000","last_peered":"2024-09-17T16:31:06.346481+0000","last_clean":"2024-09-17T16:31:06.346481+0000","last_became_active":"2024-09-17T16:29:35.685539+0000","last_became_peered":"2024-09-17T16:29:35.685539+0000","last_unstale":"2024-09-17T16:31:06.346481+0000","last_undegraded":"2024-09-17T16:31:06.346481+0000","last_fullsized":"2024-09-17T16:31:06.346481+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5768,"log_dups_size":0,"ondisk_log_size":5768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:51.280804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378011648,"num_objects":5768,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5768,"num_write_kb":369152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"671'5846","reported_seq":7383,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:06.442596+0000","last_change":"2024-09-17T16:30:53.275385+0000","last_active":"2024-09-17T16:31:06.442596+0000","last_peered":"2024-09-17T16:31:06.442596+0000","last_clean":"2024-09-17T16:31:06.442596+0000","last_became_active":"2024-09-17T16:29:56.600312+0000","last_became_peered":"2024-09-17T16:29:56.600312+0000","last_unstale":"2024-09-17T16:31:06.442596+0000","last_undegraded":"2024-09-17T16:31:06.442596+0000","last_fullsized":"2024-09-17T16:31:06.442596+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"665'5081","last_scrub_stamp":"2024-09-17T16:30:53.275174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:30:53.275174+0000","objects_scrubbed":5055,"log_size":5846,"log_dups_size":0,"ondisk_log_size":5846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:55.038306+0000","scrub_duration":1140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383123456,"num_objects":5846,"num_object_clones":0,"num_object_copies":17538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5846,"num_write_kb":374144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":44695552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"671'5786","reported_seq":7237,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:06.684173+0000","last_change":"2024-09-17T16:29:54.745801+0000","last_active":"2024-09-17T16:31:06.684173+0000","last_peered":"2024-09-17T16:31:06.684173+0000","last_clean":"2024-09-17T16:31:06.684173+0000","last_became_active":"2024-09-17T16:29:54.595605+0000","last_became_peered":"2024-09-17T16:29:54.595605+0000","last_unstale":"2024-09-17T16:31:06.684173+0000","last_undegraded":"2024-09-17T16:31:06.684173+0000","last_fullsized":"2024-09-17T16:31:06.684173+0000","mapping_epoch":640,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5786,"log_dups_size":0,"ondisk_log_size":5786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:48.844412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379191296,"num_objects":5786,"num_object_clones":0,"num_object_copies":17358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5786,"num_write_kb":370304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":667,"num_bytes_recovered":43712512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"671'5774","reported_seq":5870,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:06.699250+0000","last_change":"2024-09-17T16:29:35.659242+0000","last_active":"2024-09-17T16:31:06.699250+0000","last_peered":"2024-09-17T16:31:06.699250+0000","last_clean":"2024-09-17T16:31:06.699250+0000","last_became_active":"2024-09-17T16:29:35.658918+0000","last_became_peered":"2024-09-17T16:29:35.658918+0000","last_unstale":"2024-09-17T16:31:06.699250+0000","last_undegraded":"2024-09-17T16:31:06.699250+0000","last_fullsized":"2024-09-17T16:31:06.699250+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5774,"log_dups_size":0,"ondisk_log_size":5774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:47.848857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378404864,"num_objects":5774,"num_object_clones":0,"num_object_copies":17322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5774,"num_write_kb":369536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"670'5679","reported_seq":7078,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-09-17T16:31:04.955791+0000","last_change":"2024-09-17T16:30:00.594164+0000","last_active":"2024-09-17T16:31:04.955791+0000","last_peered":"2024-09-17T16:31:04.955791+0000","last_clean":"2024-09-17T16:31:04.955791+0000","last_became_active":"2024-09-17T16:30:00.593037+0000","last_became_peered":"2024-09-17T16:30:00.593037+0000","last_unstale":"2024-09-17T16:31:04.955791+0000","last_undegraded":"2024-09-17T16:31:04.955791+0000","last_fullsized":"2024-09-17T16:31:04.955791+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":645,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:48.292945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372178944,"num_objects":5679,"num_object_clones":0,"num_object_copies":17037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5679,"num_write_kb":363456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":42598400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"671'5714","reported_seq":5810,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:05.692936+0000","last_change":"2024-09-17T16:29:35.617588+0000","last_active":"2024-09-17T16:31:05.692936+0000","last_peered":"2024-09-17T16:31:05.692936+0000","last_clean":"2024-09-17T16:31:05.692936+0000","last_became_active":"2024-09-17T16:29:35.617197+0000","last_became_peered":"2024-09-17T16:29:35.617197+0000","last_unstale":"2024-09-17T16:31:05.692936+0000","last_undegraded":"2024-09-17T16:31:05.692936+0000","last_fullsized":"2024-09-17T16:31:05.692936+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":5714,"log_dups_size":0,"ondisk_log_size":5714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:31:51.436418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374472704,"num_objects":5714,"num_object_clones":0,"num_object_copies":17142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5714,"num_write_kb":365696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"671'5798","reported_seq":5958,"reported_epoch":671,"state":"active+clean","last_fresh":"2024-09-17T16:31:06.432817+0000","last_change":"2024-09-17T16:31:04.005952+0000","last_active":"2024-09-17T16:31:06.432817+0000","last_peered":"2024-09-17T16:31:06.432817+0000","last_clean":"2024-09-17T16:31:06.432817+0000","last_became_active":"2024-09-17T16:29:35.686233+0000","last_became_peered":"2024-09-17T16:29:35.686233+0000","last_unstale":"2024-09-17T16:31:06.432817+0000","last_undegraded":"2024-09-17T16:31:06.432817+0000","last_fullsized":"2024-09-17T16:31:06.432817+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"669'5652","last_scrub_stamp":"2024-09-17T16:31:04.005746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:04.005746+0000","objects_scrubbed":5605,"log_size":5798,"log_dups_size":0,"ondisk_log_size":5798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:32:33.039360+0000","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379977728,"num_objects":5798,"num_object_clones":0,"num_object_copies":17394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5798,"num_write_kb":371072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":85,"reported_epoch":670,"state":"active+clean","last_fresh":"2024-09-17T16:31:04.450996+0000","last_change":"2024-09-17T16:29:49.984493+0000","last_active":"2024-09-17T16:31:04.450996+0000","last_peered":"2024-09-17T16:31:04.450996+0000","last_clean":"2024-09-17T16:31:04.450996+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:31:04.450996+0000","last_undegraded":"2024-09-17T16:31:04.450996+0000","last_fullsized":"2024-09-17T16:31:04.450996+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:29:40.527128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:29:40.527128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6047137792,"num_objects":92272,"num_object_clones":0,"num_object_copies":276816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92272,"num_write_kb":5905408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2672,"num_bytes_recovered":175112192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18085314560,"data_stored":18085314560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92272,"ondisk_log_size":92272,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1499676,"kb_used_data":1089480,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92245476,"statfs":{"total":95995035648,"available":94459367424,"internally_reserved":0,"allocated":1115627520,"data_stored":1115347277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":420018628},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547464,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2279640,"kb_used_data":1833352,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":91465512,"statfs":{"total":95995035648,"available":93660684288,"internally_reserved":0,"allocated":1877352448,"data_stored":1877072225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":456980928},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743693,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2468580,"kb_used_data":1851540,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91276572,"statfs":{"total":95995035648,"available":93467209728,"internally_reserved":0,"allocated":1895976960,"data_stored":1895692222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":631830976},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005330,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1472916,"kb_used_data":1080772,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92272236,"statfs":{"total":95995035648,"available":94486769664,"internally_reserved":0,"allocated":1106710528,"data_stored":1106431218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1289228,"kb_used_data":817340,"kb_used_omap":41,"kb_used_meta":471830,"kb_avail":92455924,"statfs":{"total":95995035648,"available":94674866176,"internally_reserved":0,"allocated":836956160,"data_stored":836680402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":483154227},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495900,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":815588,"kb_used_data":369296,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92929564,"statfs":{"total":95995035648,"available":95159873536,"internally_reserved":0,"allocated":378159104,"data_stored":377875329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":456980934},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":873052,"kb_used_data":364296,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92872100,"statfs":{"total":95995035648,"available":95101030400,"internally_reserved":0,"allocated":373039104,"data_stored":372758881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986471,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1901716,"kb_used_data":1447108,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91843436,"statfs":{"total":95995035648,"available":94047678464,"internally_reserved":0,"allocated":1481838592,"data_stored":1481559282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149994,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3533796,"kb_used_data":2929236,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":90211356,"statfs":{"total":95995035648,"available":92376428544,"internally_reserved":0,"allocated":2999537664,"data_stored":2999252926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":619051463},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411632,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2593444,"kb_used_data":2223508,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91151708,"statfs":{"total":95995035648,"available":93339348992,"internally_reserved":0,"allocated":2276872192,"data_stored":2276587454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575158,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1839724,"kb_used_data":1475284,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91905428,"statfs":{"total":95995035648,"available":94111158272,"internally_reserved":0,"allocated":1510690816,"data_stored":1510406078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738682,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2821608,"kb_used_data":2218452,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":90923544,"statfs":{"total":95995035648,"available":93105709056,"internally_reserved":0,"allocated":2271694848,"data_stored":2271410110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2273312768,"data_stored":2273312768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1521352704,"data_stored":1521352704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2282356736,"data_stored":2282356736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3022520320,"data_stored":3022520320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1490747392,"data_stored":1490747392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":374472704,"data_stored":374472704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":378142720,"data_stored":378142720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":728367104,"data_stored":728367104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1108672512,"data_stored":1108672512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1897463808,"data_stored":1897463808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1889861632,"data_stored":1889861632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1118044160,"data_stored":1118044160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:31:12.717 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-17T16:31:12.717 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_5 pg_num 26 2024-09-17T16:31:12.729 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:12.754 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:12.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:12.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:12 smithi071 ceph-mon[30002]: pgmap v1483: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.08k op/s 2024-09-17T16:31:12.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:12.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:12 smithi173 ceph-mon[30530]: pgmap v1483: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.08k op/s 2024-09-17T16:31:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:12.899 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:12.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:12.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:12 smithi017 ceph-mon[27571]: pgmap v1483: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.08k op/s 2024-09-17T16:31:12.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:13.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:13.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:13.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:13.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:13.149 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:13.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:13.250 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:13.273 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:13.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:13.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:31:13.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:13.409 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:13.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:13.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:13.536 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:13.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:13.598 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:31:13.600 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:13.607 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:13.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:13.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:13.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:13.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:13.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:13.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:13.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:13.915 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:13.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:14.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:14.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:14.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:14.129 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:31:14.129 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:31:14.129 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:31:14.129 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:31:14.130 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:31:14.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:14.164 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:14.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:14.253 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:31:14.253 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:31:14.265 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:14.291 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:31:14.291 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:31:14.302 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:14.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:14.330 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:31:14.331 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:31:14.367 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:31:14.367 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:31:14.401 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:31:14.402 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:31:14.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:14.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:14.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:14.436 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:31:14.437 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:31:14.470 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:31:14.470 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:31:14.503 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:31:14.504 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:31:14.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:14.537 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:31:14.537 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:31:14.564 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:14.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:14.574 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:31:14.574 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:31:14.606 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:31:14.606 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:31:14.637 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:31:14.637 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:31:14.666 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:14.699 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:14.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:14.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:14.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:14.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:14.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:14.827 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:31:14.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:14.955 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:14.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:15.057 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:15.079 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:31:15.080 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:15.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:15.182 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:15.204 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:15.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:15.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:15 smithi017 ceph-mon[27571]: pgmap v1484: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 119 B/s rd, 55 MiB/s wr, 946 op/s 2024-09-17T16:31:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/376767537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:15.306 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:15.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:15 smithi071 ceph-mon[30002]: pgmap v1484: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 119 B/s rd, 55 MiB/s wr, 946 op/s 2024-09-17T16:31:15.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/376767537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:15.328 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:15.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:15 smithi173 ceph-mon[30530]: pgmap v1484: 17 pgs: 17 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 119 B/s rd, 55 MiB/s wr, 946 op/s 2024-09-17T16:31:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/376767537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:15.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:15.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:15.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:15.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:31:15.553 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:15.555 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:31:15.555 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:31:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:15.640 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:15.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:15.741 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:15.763 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:15.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:15.866 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:15.915 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:15.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:16.017 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:16.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:16.040 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:16.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:16.074 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 7 pg_num to 26 2024-09-17T16:31:16.142 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:16.165 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:16.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:16.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:16.289 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:16.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1253584058' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:31:16.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:16 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:31:16.382 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:31:16.391 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:16.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1253584058' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:31:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:16 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:31:16.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1253584058' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:31:16.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:16 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:31:16.433 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:16.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:16.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:16.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:16.557 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:16.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:16.659 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:16.690 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:16.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:16.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:16.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:16.822 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:16.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:16.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:16.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:16.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:17.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:17.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:17.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:17.173 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:17.210 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:17.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:17.312 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:17.350 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:17.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:17.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:17 smithi173 ceph-mon[30530]: pgmap v1485: 17 pgs: 17 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 43 MiB/s wr, 857 op/s 2024-09-17T16:31:17.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:17 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:31:17.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:17 smithi173 ceph-mon[30530]: osdmap e673: 12 total, 12 up, 12 in 2024-09-17T16:31:17.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:17.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:17.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:17 smithi017 ceph-mon[27571]: pgmap v1485: 17 pgs: 17 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 43 MiB/s wr, 857 op/s 2024-09-17T16:31:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:17 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:31:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:17 smithi017 ceph-mon[27571]: osdmap e673: 12 total, 12 up, 12 in 2024-09-17T16:31:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:17 smithi071 ceph-mon[30002]: pgmap v1485: 17 pgs: 17 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 43 MiB/s wr, 857 op/s 2024-09-17T16:31:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:17 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:31:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:17 smithi071 ceph-mon[30002]: osdmap e673: 12 total, 12 up, 12 in 2024-09-17T16:31:17.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:17.618 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:17.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:17.719 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:17.753 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:17.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:17.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:17.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:17.884 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:17.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:17.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:18.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:18.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:18.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:18.181 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:18.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:18.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:18.306 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:18.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:18.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:31:18.408 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:18.436 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:18.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:31:18.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:18.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:18.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:18.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:18.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:31:18.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:18.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:18.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:18.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:18.808 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:18.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:18.903 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:18.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:18.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:18.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:19.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:19.057 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:19.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:19.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:19 smithi017 ceph-mon[27571]: pgmap v1487: 17 pgs: 17 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 23 MiB/s wr, 973 op/s 2024-09-17T16:31:19.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:19 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:31:19.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:19 smithi017 ceph-mon[27571]: osdmap e674: 12 total, 12 up, 12 in 2024-09-17T16:31:19.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:19.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:19.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:19.288 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:19.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:19.325 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:19.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:19 smithi173 ceph-mon[30530]: pgmap v1487: 17 pgs: 17 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 23 MiB/s wr, 973 op/s 2024-09-17T16:31:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:19 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:31:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:19 smithi173 ceph-mon[30530]: osdmap e674: 12 total, 12 up, 12 in 2024-09-17T16:31:19.426 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:19.450 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:19.453 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:19.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:19.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:19 smithi071 ceph-mon[30002]: pgmap v1487: 17 pgs: 17 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 23 MiB/s wr, 973 op/s 2024-09-17T16:31:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:19 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:31:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:19 smithi071 ceph-mon[30002]: osdmap e674: 12 total, 12 up, 12 in 2024-09-17T16:31:19.583 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:19.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:19.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:19.664 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:31:19.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:19.706 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:19.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:19.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:19.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:19.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:19.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:19.959 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:19.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:19.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:20.061 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:20.100 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:20.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:20.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:20.102 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:31:20.151 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:31:20.151 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:31:20.202 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:20.226 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:20.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:20.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:20 smithi017 ceph-mon[27571]: osdmap e675: 12 total, 12 up, 12 in 2024-09-17T16:31:20.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:20 smithi017 ceph-mon[27571]: 7.7 scrub starts 2024-09-17T16:31:20.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:20 smithi017 ceph-mon[27571]: 7.0 scrub starts 2024-09-17T16:31:20.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:20 smithi017 ceph-mon[27571]: 7.5 scrub starts 2024-09-17T16:31:20.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:20 smithi017 ceph-mon[27571]: 7.6 scrub starts 2024-09-17T16:31:20.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:20 smithi017 ceph-mon[27571]: 7.8 scrub starts 2024-09-17T16:31:20.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:20 smithi017 ceph-mon[27571]: from='client.17751 172.21.15.17:0/3108338531' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:31:20.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:20 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:31:20.327 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:20.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:20 smithi071 ceph-mon[30002]: osdmap e675: 12 total, 12 up, 12 in 2024-09-17T16:31:20.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:20 smithi071 ceph-mon[30002]: 7.7 scrub starts 2024-09-17T16:31:20.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:20 smithi071 ceph-mon[30002]: 7.0 scrub starts 2024-09-17T16:31:20.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:20 smithi071 ceph-mon[30002]: 7.5 scrub starts 2024-09-17T16:31:20.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:20 smithi071 ceph-mon[30002]: 7.6 scrub starts 2024-09-17T16:31:20.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:20 smithi071 ceph-mon[30002]: 7.8 scrub starts 2024-09-17T16:31:20.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:20 smithi071 ceph-mon[30002]: from='client.17751 172.21.15.17:0/3108338531' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:31:20.347 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:20 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:31:20.351 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:20.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:20 smithi173 ceph-mon[30530]: osdmap e675: 12 total, 12 up, 12 in 2024-09-17T16:31:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:20 smithi173 ceph-mon[30530]: 7.7 scrub starts 2024-09-17T16:31:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:20 smithi173 ceph-mon[30530]: 7.0 scrub starts 2024-09-17T16:31:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:20 smithi173 ceph-mon[30530]: 7.5 scrub starts 2024-09-17T16:31:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:20 smithi173 ceph-mon[30530]: 7.6 scrub starts 2024-09-17T16:31:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:20 smithi173 ceph-mon[30530]: 7.8 scrub starts 2024-09-17T16:31:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:20 smithi173 ceph-mon[30530]: from='client.17751 172.21.15.17:0/3108338531' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:31:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:20 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:31:20.453 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:20.479 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:20.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:20.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:20.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:20.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:20.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:20.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:20.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:20.734 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:20.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:20.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:20.859 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:20.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:20.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:20.982 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:20.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:21.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:21.108 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:21.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:21.155 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:31:21.210 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:21.232 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:21.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: 7.1 scrub starts 2024-09-17T16:31:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: pgmap v1490: 27 pgs: 8 unknown, 19 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s wr, 755 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T16:31:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: 7.2 scrub starts 2024-09-17T16:31:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:31:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: from='client.17751 172.21.15.17:0/3108338531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:31:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:31:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: osdmap e676: 12 total, 12 up, 12 in 2024-09-17T16:31:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: 7.7 scrub starts 2024-09-17T16:31:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: 7.0 scrub starts 2024-09-17T16:31:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: 7.5 scrub starts 2024-09-17T16:31:21.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: 7.6 scrub starts 2024-09-17T16:31:21.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: 7.8 scrub starts 2024-09-17T16:31:21.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:21 smithi071 ceph-mon[30002]: 7.8 scrub ok 2024-09-17T16:31:21.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:21.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:21.356 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:21.384 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: 7.1 scrub starts 2024-09-17T16:31:21.384 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: pgmap v1490: 27 pgs: 8 unknown, 19 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s wr, 755 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T16:31:21.384 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: 7.2 scrub starts 2024-09-17T16:31:21.384 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:31:21.384 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: from='client.17751 172.21.15.17:0/3108338531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:31:21.384 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:31:21.384 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: osdmap e676: 12 total, 12 up, 12 in 2024-09-17T16:31:21.384 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: 7.7 scrub starts 2024-09-17T16:31:21.384 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: 7.0 scrub starts 2024-09-17T16:31:21.385 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: 7.5 scrub starts 2024-09-17T16:31:21.385 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: 7.6 scrub starts 2024-09-17T16:31:21.385 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: 7.8 scrub starts 2024-09-17T16:31:21.385 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:21 smithi173 ceph-mon[30530]: 7.8 scrub ok 2024-09-17T16:31:21.457 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:21.479 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:21.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: 7.1 scrub starts 2024-09-17T16:31:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: pgmap v1490: 27 pgs: 8 unknown, 19 active+clean; 5.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s wr, 755 op/s; 0 B/s, 0 objects/s recovering 2024-09-17T16:31:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: 7.2 scrub starts 2024-09-17T16:31:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:31:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: from='client.17751 172.21.15.17:0/3108338531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:31:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:31:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: osdmap e676: 12 total, 12 up, 12 in 2024-09-17T16:31:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: 7.7 scrub starts 2024-09-17T16:31:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: 7.0 scrub starts 2024-09-17T16:31:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: 7.5 scrub starts 2024-09-17T16:31:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: 7.6 scrub starts 2024-09-17T16:31:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: 7.8 scrub starts 2024-09-17T16:31:21.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:21 smithi017 ceph-mon[27571]: 7.8 scrub ok 2024-09-17T16:31:21.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:21.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:21.603 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:21.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:21.723 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:21.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:21.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:21.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:21.828 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:21.849 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:21.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:21.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:21.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:21.974 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:22.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:22.143 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:22.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:22.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:22.298 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:22.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:22.348 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: 7.c scrub starts 2024-09-17T16:31:22.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: 7.1 scrub starts 2024-09-17T16:31:22.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: 7.b scrub starts 2024-09-17T16:31:22.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: 7.0 scrub ok 2024-09-17T16:31:22.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: 7.1 scrub ok 2024-09-17T16:31:22.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: 7.2 scrub starts 2024-09-17T16:31:22.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: osdmap e677: 12 total, 12 up, 12 in 2024-09-17T16:31:22.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: 7.c scrub ok 2024-09-17T16:31:22.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: 7.7 scrub ok 2024-09-17T16:31:22.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: 7.4 scrub starts 2024-09-17T16:31:22.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:22 smithi017 ceph-mon[27571]: 7.6 scrub ok 2024-09-17T16:31:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: 7.c scrub starts 2024-09-17T16:31:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: 7.1 scrub starts 2024-09-17T16:31:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: 7.b scrub starts 2024-09-17T16:31:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: 7.0 scrub ok 2024-09-17T16:31:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: 7.1 scrub ok 2024-09-17T16:31:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: 7.2 scrub starts 2024-09-17T16:31:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: osdmap e677: 12 total, 12 up, 12 in 2024-09-17T16:31:22.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: 7.c scrub ok 2024-09-17T16:31:22.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: 7.7 scrub ok 2024-09-17T16:31:22.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: 7.4 scrub starts 2024-09-17T16:31:22.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:22 smithi173 ceph-mon[30530]: 7.6 scrub ok 2024-09-17T16:31:22.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:22.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:22.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:22.433 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:22.519 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:31:22.519 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:31:22.527 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: 7.c scrub starts 2024-09-17T16:31:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: 7.1 scrub starts 2024-09-17T16:31:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: 7.b scrub starts 2024-09-17T16:31:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: 7.0 scrub ok 2024-09-17T16:31:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: 7.1 scrub ok 2024-09-17T16:31:22.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: 7.2 scrub starts 2024-09-17T16:31:22.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: osdmap e677: 12 total, 12 up, 12 in 2024-09-17T16:31:22.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: 7.c scrub ok 2024-09-17T16:31:22.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: 7.7 scrub ok 2024-09-17T16:31:22.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: 7.4 scrub starts 2024-09-17T16:31:22.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:22 smithi071 ceph-mon[30002]: 7.6 scrub ok 2024-09-17T16:31:22.573 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:31:22.573 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:31:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:22.597 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:22.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:22.624 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:31:22.624 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:31:22.669 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:31:22.669 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:31:22.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:22.712 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:31:22.712 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:31:22.720 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:22.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:22.756 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:31:22.756 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:31:22.798 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:31:22.798 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:31:22.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:22.843 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:31:22.843 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:31:22.867 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:22.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:22.895 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:31:22.895 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:31:22.942 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:31:22.942 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:31:22.969 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:22.983 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:31:22.983 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:31:23.023 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:23.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:23.032 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:31:23.032 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:31:23.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:23.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:23.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:23.172 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:31:23.172 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:31:23.172 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:31:23.172 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:31:23.172 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:31:23.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:23 smithi017 ceph-mon[27571]: 7.2 scrub ok 2024-09-17T16:31:23.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:23 smithi017 ceph-mon[27571]: 7.5 scrub ok 2024-09-17T16:31:23.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:23 smithi017 ceph-mon[27571]: pgmap v1493: 27 pgs: 4 active+clean+scrubbing, 9 remapped+peering, 14 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:31:23.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:23 smithi017 ceph-mon[27571]: 7.3 scrub starts 2024-09-17T16:31:23.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:23 smithi017 ceph-mon[27571]: osdmap e678: 12 total, 12 up, 12 in 2024-09-17T16:31:23.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3022149452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:23.249 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:23.287 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:23.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:23.390 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:23 smithi173 ceph-mon[30530]: 7.2 scrub ok 2024-09-17T16:31:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:23 smithi173 ceph-mon[30530]: 7.5 scrub ok 2024-09-17T16:31:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:23 smithi173 ceph-mon[30530]: pgmap v1493: 27 pgs: 4 active+clean+scrubbing, 9 remapped+peering, 14 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:31:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:23 smithi173 ceph-mon[30530]: 7.3 scrub starts 2024-09-17T16:31:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:23 smithi173 ceph-mon[30530]: osdmap e678: 12 total, 12 up, 12 in 2024-09-17T16:31:23.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3022149452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:23.411 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:23.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:23.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:23.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:23.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:23 smithi071 ceph-mon[30002]: 7.2 scrub ok 2024-09-17T16:31:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:23 smithi071 ceph-mon[30002]: 7.5 scrub ok 2024-09-17T16:31:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:23 smithi071 ceph-mon[30002]: pgmap v1493: 27 pgs: 4 active+clean+scrubbing, 9 remapped+peering, 14 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:31:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:23 smithi071 ceph-mon[30002]: 7.3 scrub starts 2024-09-17T16:31:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:23 smithi071 ceph-mon[30002]: osdmap e678: 12 total, 12 up, 12 in 2024-09-17T16:31:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3022149452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:31:23.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:23.659 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:23.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:23.739 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:31:23.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:23.783 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:23.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:23.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:23.898 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:31:23.899 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:31:23.899 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T16:31:23.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:31:23.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:23.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:24.011 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:24.032 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:24.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:24.134 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:24.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:24 smithi017 ceph-mon[27571]: 7.9 scrub starts 2024-09-17T16:31:24.157 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:24.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:24.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:24.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:24.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:24.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:24.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:24.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:24 smithi173 ceph-mon[30530]: 7.9 scrub starts 2024-09-17T16:31:24.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:24.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:24.508 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:24.553 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:24.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:24.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:24 smithi071 ceph-mon[30002]: 7.9 scrub starts 2024-09-17T16:31:24.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:24.677 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:24.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:24.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:24.800 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:24.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:24.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:24.902 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:24.944 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:24.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:25.046 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:25.067 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:25.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:25.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:25.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:25.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:25.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:25.295 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:25.329 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:25.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:25 smithi173 ceph-mon[30530]: 7.b scrub ok 2024-09-17T16:31:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:25 smithi173 ceph-mon[30530]: 7.3 scrub ok 2024-09-17T16:31:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:25 smithi173 ceph-mon[30530]: pgmap v1495: 27 pgs: 1 activating+undersized+degraded+remapped, 5 active+clean+scrubbing, 9 remapped+peering, 12 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5830/265923 objects degraded (2.192%); 2915/265923 objects misplaced (1.096%) 2024-09-17T16:31:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:25 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 5830/265923 objects degraded (2.192%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:31:25.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:25 smithi173 ceph-mon[30530]: 7.9 scrub ok 2024-09-17T16:31:25.431 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:25.454 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:25.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:25 smithi017 ceph-mon[27571]: 7.b scrub ok 2024-09-17T16:31:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:25 smithi017 ceph-mon[27571]: 7.3 scrub ok 2024-09-17T16:31:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:25 smithi017 ceph-mon[27571]: pgmap v1495: 27 pgs: 1 activating+undersized+degraded+remapped, 5 active+clean+scrubbing, 9 remapped+peering, 12 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5830/265923 objects degraded (2.192%); 2915/265923 objects misplaced (1.096%) 2024-09-17T16:31:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:25 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 5830/265923 objects degraded (2.192%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:31:25.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:25 smithi017 ceph-mon[27571]: 7.9 scrub ok 2024-09-17T16:31:25.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:25 smithi071 ceph-mon[30002]: 7.b scrub ok 2024-09-17T16:31:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:25 smithi071 ceph-mon[30002]: 7.3 scrub ok 2024-09-17T16:31:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:25 smithi071 ceph-mon[30002]: pgmap v1495: 27 pgs: 1 activating+undersized+degraded+remapped, 5 active+clean+scrubbing, 9 remapped+peering, 12 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5830/265923 objects degraded (2.192%); 2915/265923 objects misplaced (1.096%) 2024-09-17T16:31:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:25 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 5830/265923 objects degraded (2.192%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:31:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:25 smithi071 ceph-mon[30002]: 7.9 scrub ok 2024-09-17T16:31:25.577 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:25.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:25.679 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:25.714 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:25.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:25.815 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:25.858 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:25.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:25.961 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:26.005 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:26.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:26.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:26.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:26.129 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:26.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:26.231 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:26.284 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:26.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:26.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:26.390 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:26.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:26 smithi017 ceph-mon[27571]: 7.4 scrub ok 2024-09-17T16:31:26.412 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:26.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:26.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:26.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:26.536 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:26.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:26.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:26 smithi071 ceph-mon[30002]: 7.4 scrub ok 2024-09-17T16:31:26.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:26.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:26 smithi173 ceph-mon[30530]: 7.4 scrub ok 2024-09-17T16:31:26.659 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:26.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:26.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:26.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:26.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:26.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:26.930 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:26.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:27.032 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:27.071 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:27.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:27.173 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:27.195 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:27.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:27.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:27.344 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:27.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:27.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:27.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:27.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:27.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:27 smithi017 ceph-mon[27571]: pgmap v1496: 27 pgs: 1 activating+undersized+degraded+remapped, 3 active+clean+scrubbing, 9 remapped+peering, 14 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5830/265431 objects degraded (2.196%); 2915/265431 objects misplaced (1.098%) 2024-09-17T16:31:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:27 smithi071 ceph-mon[30002]: pgmap v1496: 27 pgs: 1 activating+undersized+degraded+remapped, 3 active+clean+scrubbing, 9 remapped+peering, 14 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5830/265431 objects degraded (2.196%); 2915/265431 objects misplaced (1.098%) 2024-09-17T16:31:27.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:27.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:27.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:27.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:27 smithi173 ceph-mon[30530]: pgmap v1496: 27 pgs: 1 activating+undersized+degraded+remapped, 3 active+clean+scrubbing, 9 remapped+peering, 14 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 5830/265431 objects degraded (2.196%); 2915/265431 objects misplaced (1.098%) 2024-09-17T16:31:27.696 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:27.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:27.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:27.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:27.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:27.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:27.946 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:27.988 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:27.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:28.090 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:28.134 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:28.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:28.236 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:28.259 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:28.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:28.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:28.403 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:28.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:28.505 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:28.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:28.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:28.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:28.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:28.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:28.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:28.752 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:28.795 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:28.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:28.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:28.921 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:28.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:29.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:29.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:29.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:29.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:29.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:29.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:29.171 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:29.272 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:29.295 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:29.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:29.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:29.419 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:29.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:29.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:29 smithi017 ceph-mon[27571]: pgmap v1497: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 17 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 23611/260850 objects degraded (9.052%); 46143/260850 objects misplaced (17.689%) 2024-09-17T16:31:29.521 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:29.544 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:29.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:29 smithi071 ceph-mon[30002]: pgmap v1497: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 17 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 23611/260850 objects degraded (9.052%); 46143/260850 objects misplaced (17.689%) 2024-09-17T16:31:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:29 smithi173 ceph-mon[30530]: pgmap v1497: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 17 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 23611/260850 objects degraded (9.052%); 46143/260850 objects misplaced (17.689%) 2024-09-17T16:31:29.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:29.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:29.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:29.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:29.794 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:29.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:29.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:29.895 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:29.933 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:29.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:30.035 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:30.090 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:30.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:30.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:30.213 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:30.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:30.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:30.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:30.370 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:30.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:30.472 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:30 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 23611/260850 objects degraded (9.052%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:30.494 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:30.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:30 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 23611/260850 objects degraded (9.052%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:30.597 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:30.618 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:30.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:30.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:30 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 23611/260850 objects degraded (9.052%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:30.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:30.741 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:30.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:30.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:30.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:30.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:30.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:30.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:30.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:30.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:30.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:31.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:31.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:31.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:31.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:31.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:31.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:31.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:31.369 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:31.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:31.471 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:31 smithi017 ceph-mon[27571]: pgmap v1498: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 17 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 23611/260433 objects degraded (9.066%); 46143/260433 objects misplaced (17.718%) 2024-09-17T16:31:31.492 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:31.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:31 smithi071 ceph-mon[30002]: pgmap v1498: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 17 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 23611/260433 objects degraded (9.066%); 46143/260433 objects misplaced (17.718%) 2024-09-17T16:31:31.594 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:31.638 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:31.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:31 smithi173 ceph-mon[30530]: pgmap v1498: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 17 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 23611/260433 objects degraded (9.066%); 46143/260433 objects misplaced (17.718%) 2024-09-17T16:31:31.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:31.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:31.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:31.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:31.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:31.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:31.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:31.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:32.012 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:32.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:32.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:32.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:32.135 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:32.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:32.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:32.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:32.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:32.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:32.408 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:32.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:32 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:31:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:32 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:31:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:32 smithi017 ceph-mon[27571]: osdmap e679: 12 total, 12 up, 12 in 2024-09-17T16:31:32.510 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:32.532 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:32.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:32.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:32 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:31:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:32 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:31:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:32 smithi071 ceph-mon[30002]: osdmap e679: 12 total, 12 up, 12 in 2024-09-17T16:31:32.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:32 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:31:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:32 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:31:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:32 smithi173 ceph-mon[30530]: osdmap e679: 12 total, 12 up, 12 in 2024-09-17T16:31:32.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:32.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:32.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:32.816 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:32.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:32.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:32.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:32.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:33.041 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:33.063 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:33.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:33.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:33.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:33.187 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:33.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:33.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:33.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:33.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:33.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:33.414 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:33.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:33.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:33 smithi017 ceph-mon[27571]: pgmap v1500: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22974/254712 objects degraded (9.020%); 44582/254712 objects misplaced (17.503%); 5.2 MiB/s, 84 objects/s recovering 2024-09-17T16:31:33.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:33.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:33.562 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:33.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:33 smithi071 ceph-mon[30002]: pgmap v1500: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22974/254712 objects degraded (9.020%); 44582/254712 objects misplaced (17.503%); 5.2 MiB/s, 84 objects/s recovering 2024-09-17T16:31:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:33 smithi173 ceph-mon[30530]: pgmap v1500: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22974/254712 objects degraded (9.020%); 44582/254712 objects misplaced (17.503%); 5.2 MiB/s, 84 objects/s recovering 2024-09-17T16:31:33.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:33.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:33.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:33.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:33.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:33.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:33.916 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:33.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:33.939 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:33.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:34.040 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:34.081 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:34.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:34.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:34.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:34.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:34.307 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:34.329 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:34.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:34.432 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:34.454 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:34.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:34.556 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:34.604 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:34.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:34.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:34.729 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:34.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:34.831 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:34.852 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:34.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:34.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:34.979 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:34.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:35.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:35.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:35.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:35.205 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:35.226 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:35.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:35.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:35.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:35.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:35.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:35.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:35.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:35.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:35 smithi017 ceph-mon[27571]: pgmap v1501: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 22974/253572 objects degraded (9.060%); 44220/253572 objects misplaced (17.439%); 5.9 MiB/s, 95 objects/s recovering 2024-09-17T16:31:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:35 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 22974/253572 objects degraded (9.060%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:35 smithi017 ceph-mon[27571]: osdmap e680: 12 total, 12 up, 12 in 2024-09-17T16:31:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:35 smithi071 ceph-mon[30002]: pgmap v1501: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 22974/253572 objects degraded (9.060%); 44220/253572 objects misplaced (17.439%); 5.9 MiB/s, 95 objects/s recovering 2024-09-17T16:31:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:35 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 22974/253572 objects degraded (9.060%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:35 smithi071 ceph-mon[30002]: osdmap e680: 12 total, 12 up, 12 in 2024-09-17T16:31:35.579 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:35.630 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:35.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:35.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:35 smithi173 ceph-mon[30530]: pgmap v1501: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 22974/253572 objects degraded (9.060%); 44220/253572 objects misplaced (17.439%); 5.9 MiB/s, 95 objects/s recovering 2024-09-17T16:31:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:35 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 22974/253572 objects degraded (9.060%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:35 smithi173 ceph-mon[30530]: osdmap e680: 12 total, 12 up, 12 in 2024-09-17T16:31:35.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:35.780 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:35.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:35.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:35.862 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:31:35.874 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:35.884 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:35.957 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:35.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:35.984 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:36.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:36.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:36.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:36.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:36.260 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:36.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:36.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:36.403 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:36.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:36.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:36.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:36.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:36.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:36.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:36.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:36.657 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:36.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:31:36.758 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:36.804 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:36.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:36.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:36.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:36.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:37.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:37.074 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:37.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:37.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:37.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:37.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:37.304 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:37.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:37.327 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:37.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:37.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:37.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:37.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:37 smithi017 ceph-mon[27571]: pgmap v1503: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 353 op/s; 22974/252369 objects degraded (9.103%); 44220/252369 objects misplaced (17.522%); 8.7 MiB/s, 139 objects/s recovering 2024-09-17T16:31:37.554 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:37 smithi071 ceph-mon[30002]: pgmap v1503: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 353 op/s; 22974/252369 objects degraded (9.103%); 44220/252369 objects misplaced (17.522%); 8.7 MiB/s, 139 objects/s recovering 2024-09-17T16:31:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:37.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:37 smithi173 ceph-mon[30530]: pgmap v1503: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 353 op/s; 22974/252369 objects degraded (9.103%); 44220/252369 objects misplaced (17.522%); 8.7 MiB/s, 139 objects/s recovering 2024-09-17T16:31:37.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:37.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:37.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:37.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:37.865 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:37.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:37.967 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:38.016 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:38.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:38.118 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:38.141 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:38.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:38.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:38.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:38.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:38.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:38.367 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:38.389 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:38.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:38.491 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:38.512 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:38.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:38.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:38.655 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:38.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:38.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:38.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:38.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:38.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:38.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:38.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:39.010 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:39.032 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:39.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:39.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:39.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:39.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:39.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:39.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:39.285 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:39.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:39.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:39.409 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:39.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:39.455 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:39 smithi173 ceph-mon[30530]: pgmap v1504: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 22321/245949 objects degraded (9.075%); 42224/245949 objects misplaced (17.168%); 18 MiB/s, 281 objects/s recovering 2024-09-17T16:31:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:39 smithi017 ceph-mon[27571]: pgmap v1504: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 22321/245949 objects degraded (9.075%); 42224/245949 objects misplaced (17.168%); 18 MiB/s, 281 objects/s recovering 2024-09-17T16:31:39.510 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:39.533 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:39.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:39 smithi071 ceph-mon[30002]: pgmap v1504: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 22321/245949 objects degraded (9.075%); 42224/245949 objects misplaced (17.168%); 18 MiB/s, 281 objects/s recovering 2024-09-17T16:31:39.635 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:39.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:39.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:39.778 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:39.799 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:39.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:39.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:39.938 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:39.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:40.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:40.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:40.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:40.169 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:40.204 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:40.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:40.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:40.306 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:40.330 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:40.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:40.432 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:40.461 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:40.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:40 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 22321/245949 objects degraded (9.075%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:40 smithi017 ceph-mon[27571]: osdmap e681: 12 total, 12 up, 12 in 2024-09-17T16:31:40.563 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:40 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 22321/245949 objects degraded (9.075%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:40 smithi071 ceph-mon[30002]: osdmap e681: 12 total, 12 up, 12 in 2024-09-17T16:31:40.586 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:40.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:40 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 22321/245949 objects degraded (9.075%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:40 smithi173 ceph-mon[30530]: osdmap e681: 12 total, 12 up, 12 in 2024-09-17T16:31:40.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:40.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:40.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:40.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:40.871 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:40.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:40.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:40.994 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:40.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:41.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:41.139 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:41.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:41.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:41.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:41.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:41.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:41.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:41.393 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:41.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:41 smithi017 ceph-mon[27571]: pgmap v1506: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 386 op/s; 22321/245427 objects degraded (9.095%); 42224/245427 objects misplaced (17.204%); 9.8 MiB/s, 157 objects/s recovering 2024-09-17T16:31:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/985978825' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:41 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:41.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:41.519 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:41.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:41 smithi071 ceph-mon[30002]: pgmap v1506: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 386 op/s; 22321/245427 objects degraded (9.095%); 42224/245427 objects misplaced (17.204%); 9.8 MiB/s, 157 objects/s recovering 2024-09-17T16:31:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/985978825' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:41 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:41.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:41.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:41.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:41 smithi173 ceph-mon[30530]: pgmap v1506: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 386 op/s; 22321/245427 objects degraded (9.095%); 42224/245427 objects misplaced (17.204%); 9.8 MiB/s, 157 objects/s recovering 2024-09-17T16:31:41.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/985978825' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:41.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:41 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:31:41.746 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:41.768 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:41.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:41.870 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:41.919 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:41.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:42.021 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:42.062 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:42.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:42.091 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:31:42.163 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:42.185 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:42.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:42.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:42.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:42.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:42.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:42.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:42.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:42.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:42.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:42 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:31:42.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:42 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:31:42.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:42 smithi017 ceph-mon[27571]: osdmap e682: 12 total, 12 up, 12 in 2024-09-17T16:31:42.537 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:42 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:31:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:42 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:31:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:42 smithi071 ceph-mon[30002]: osdmap e682: 12 total, 12 up, 12 in 2024-09-17T16:31:42.575 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:42.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:42 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:31:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:42 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:31:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:42 smithi173 ceph-mon[30530]: osdmap e682: 12 total, 12 up, 12 in 2024-09-17T16:31:42.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:42.698 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:42.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:42.800 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:42.826 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:42.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:42.928 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:42.962 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:42.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:43.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:43.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:43.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:43.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:43.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:43.215 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:43.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:43.317 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:43.358 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:43.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:43.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:43.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:43.482 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:43.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:43.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:43 smithi017 ceph-mon[27571]: pgmap v1508: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 21908/241920 objects degraded (9.056%); 41212/241920 objects misplaced (17.035%); 16 MiB/s, 248 objects/s recovering 2024-09-17T16:31:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:43 smithi071 ceph-mon[30002]: pgmap v1508: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 21908/241920 objects degraded (9.056%); 41212/241920 objects misplaced (17.035%); 16 MiB/s, 248 objects/s recovering 2024-09-17T16:31:43.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:43.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:43.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:43 smithi173 ceph-mon[30530]: pgmap v1508: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 21908/241920 objects degraded (9.056%); 41212/241920 objects misplaced (17.035%); 16 MiB/s, 248 objects/s recovering 2024-09-17T16:31:43.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:43.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:43.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:43.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:43.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:43.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:43.972 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:43.994 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:43.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:44.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:44.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:44.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:44.222 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:44.264 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:44.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:44.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:44.366 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:44.407 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:44.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:44.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:44 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:31:44.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:44.555 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:44.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:44.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:44 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:31:44.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:44 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:31:44.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:44.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:44.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:44.781 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:44.803 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:44.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:44.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:44.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:44.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:44.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:45.032 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:45.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:45.070 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:45.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:45.171 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:45.193 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:45.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:45.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:45.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:45.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:45.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:45.449 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:45.450 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:45 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:31:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:45 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:31:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:45 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:31:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:45 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:31:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:45 smithi017 ceph-mon[27571]: pgmap v1509: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 21760/239106 objects degraded (9.101%); 40263/239106 objects misplaced (16.839%); 18 MiB/s, 286 objects/s recovering 2024-09-17T16:31:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:45 smithi017 ceph-mon[27571]: osdmap e683: 12 total, 12 up, 12 in 2024-09-17T16:31:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:45 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 21760/239106 objects degraded (9.101%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:45.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:45 smithi017 ceph-mon[27571]: osdmap e684: 12 total, 12 up, 12 in 2024-09-17T16:31:45.550 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:45 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:31:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:45 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:31:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:45 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:31:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:45 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:31:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:45 smithi071 ceph-mon[30002]: pgmap v1509: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 21760/239106 objects degraded (9.101%); 40263/239106 objects misplaced (16.839%); 18 MiB/s, 286 objects/s recovering 2024-09-17T16:31:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:45 smithi071 ceph-mon[30002]: osdmap e683: 12 total, 12 up, 12 in 2024-09-17T16:31:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:45 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 21760/239106 objects degraded (9.101%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:45 smithi071 ceph-mon[30002]: osdmap e684: 12 total, 12 up, 12 in 2024-09-17T16:31:45.571 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:45.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:45 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:31:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:45 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:31:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:45 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:31:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:45 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:31:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:45 smithi173 ceph-mon[30530]: pgmap v1509: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 21760/239106 objects degraded (9.101%); 40263/239106 objects misplaced (16.839%); 18 MiB/s, 286 objects/s recovering 2024-09-17T16:31:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:45 smithi173 ceph-mon[30530]: osdmap e683: 12 total, 12 up, 12 in 2024-09-17T16:31:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:45 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 21760/239106 objects degraded (9.101%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:45.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:45 smithi173 ceph-mon[30530]: osdmap e684: 12 total, 12 up, 12 in 2024-09-17T16:31:45.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:45.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:45.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:45.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:45.798 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:45.836 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:45.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:45.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:45.974 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:45.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:46.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:46.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:46.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:46.204 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:46.227 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:46.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:46.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:46.374 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:46.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:46.475 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:31:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:31:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:31:46.497 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:46.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:31:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:31:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:31:46.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:46.620 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:46.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:31:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:31:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:31:46.722 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:46.744 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:46.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:46.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:46.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:46.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:46.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:46.973 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:47.014 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:47.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:47.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:47.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:47.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:47.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:47.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:47.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:47.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:47.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:47.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:47.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:47.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:47 smithi017 ceph-mon[27571]: pgmap v1512: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 406 op/s; 21760/238104 objects degraded (9.139%); 40263/238104 objects misplaced (16.910%); 12 MiB/s, 192 objects/s recovering 2024-09-17T16:31:47.509 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:47.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:47 smithi071 ceph-mon[30002]: pgmap v1512: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 406 op/s; 21760/238104 objects degraded (9.139%); 40263/238104 objects misplaced (16.910%); 12 MiB/s, 192 objects/s recovering 2024-09-17T16:31:47.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:47.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:47.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:47 smithi173 ceph-mon[30530]: pgmap v1512: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 406 op/s; 21760/238104 objects degraded (9.139%); 40263/238104 objects misplaced (16.910%); 12 MiB/s, 192 objects/s recovering 2024-09-17T16:31:47.737 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:47.761 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:47.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:47.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:47.885 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:47.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:47.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:48.009 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:48.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:48.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:48.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:48.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:48.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:48.279 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:48.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:48.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:48.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:48.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:48.507 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:48.536 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:48.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:48.638 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:48.661 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:48.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:48.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:48.762 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:48.785 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:48.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:48.887 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:48.909 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:48.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:49.011 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:49.054 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:49.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:49.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:49.196 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:49.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:49.298 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:49.335 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:49.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:49.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:49.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:49.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:49 smithi017 ceph-mon[27571]: pgmap v1513: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 21122/232116 objects degraded (9.100%); 38201/232116 objects misplaced (16.458%); 20 MiB/s, 319 objects/s recovering 2024-09-17T16:31:49.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:49 smithi071 ceph-mon[30002]: pgmap v1513: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 21122/232116 objects degraded (9.100%); 38201/232116 objects misplaced (16.458%); 20 MiB/s, 319 objects/s recovering 2024-09-17T16:31:49.584 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:49.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:49.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:49 smithi173 ceph-mon[30530]: pgmap v1513: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 21122/232116 objects degraded (9.100%); 38201/232116 objects misplaced (16.458%); 20 MiB/s, 319 objects/s recovering 2024-09-17T16:31:49.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:49.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:49.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:49.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:49.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:49.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:49.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:49.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:49.984 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:49.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:50.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:50.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:50.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:50.213 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:50.259 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:50.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:50.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:50.393 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:50.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:50 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:31:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:50 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:31:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:50 smithi017 ceph-mon[27571]: osdmap e685: 12 total, 12 up, 12 in 2024-09-17T16:31:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:50 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 21122/232116 objects degraded (9.100%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:50.495 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:50.543 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:50.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:50 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:31:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:50 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:31:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:50 smithi071 ceph-mon[30002]: osdmap e685: 12 total, 12 up, 12 in 2024-09-17T16:31:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:50 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 21122/232116 objects degraded (9.100%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:50.644 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:50 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:31:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:50 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:31:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:50 smithi173 ceph-mon[30530]: osdmap e685: 12 total, 12 up, 12 in 2024-09-17T16:31:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:50 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 21122/232116 objects degraded (9.100%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:50.667 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:50.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:50.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:50.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:50.791 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:50.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:50.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:50.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:51.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:51.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:51.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:51.144 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:51.186 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:51.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:51.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:51.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:51.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:51.416 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:51.438 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:51.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:51 smithi017 ceph-mon[27571]: pgmap v1515: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 409 op/s; 21122/231732 objects degraded (9.115%); 38201/231732 objects misplaced (16.485%); 13 MiB/s, 201 objects/s recovering 2024-09-17T16:31:51.539 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:51.561 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:51.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:51 smithi071 ceph-mon[30002]: pgmap v1515: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 409 op/s; 21122/231732 objects degraded (9.115%); 38201/231732 objects misplaced (16.485%); 13 MiB/s, 201 objects/s recovering 2024-09-17T16:31:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:51 smithi173 ceph-mon[30530]: pgmap v1515: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 409 op/s; 21122/231732 objects degraded (9.115%); 38201/231732 objects misplaced (16.485%); 13 MiB/s, 201 objects/s recovering 2024-09-17T16:31:51.663 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:51.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:51.686 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:51.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:51.787 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:51.808 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:51.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:51.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:51.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:51.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:51.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:51.934 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:52.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:52.058 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:52.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:52.160 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:52.182 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:52.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:52.284 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:52.307 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:52.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:52.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:52.452 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:52.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:52.554 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:52.577 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:52.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:52.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:52.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:52.702 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:52.803 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:52.824 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:52.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:52.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:52.965 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:52.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:53.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:53.110 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:53.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:53.212 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:53.234 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:53.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:53.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:53.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:53.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:53.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:53.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:53.669 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:53 smithi173 ceph-mon[30530]: pgmap v1516: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s; 20658/227670 objects degraded (9.074%); 37146/227670 objects misplaced (16.316%); 15 MiB/s, 245 objects/s recovering 2024-09-17T16:31:53.669 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:53 smithi071 ceph-mon[30002]: pgmap v1516: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s; 20658/227670 objects degraded (9.074%); 37146/227670 objects misplaced (16.316%); 15 MiB/s, 245 objects/s recovering 2024-09-17T16:31:53.711 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:53.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:53 smithi017 ceph-mon[27571]: pgmap v1516: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s; 20658/227670 objects degraded (9.074%); 37146/227670 objects misplaced (16.316%); 15 MiB/s, 245 objects/s recovering 2024-09-17T16:31:53.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:53.835 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:53.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:53.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:53.959 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:53.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:54.061 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:31:54.084 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:54.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:31:54.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:54.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:54.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:54.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:54.314 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:54.336 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:54.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:54.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:54.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:54.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:54.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:54.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:54.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:54.593 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:54.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:54.694 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:54.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:54.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:54.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:54.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:54.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:54.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:54.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:54.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:54.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:55.072 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:55.114 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:55.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:55.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:55.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:55.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:55.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:55.327 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:31:55.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:55.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:55.364 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:55.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:55.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:55.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:55 smithi071 ceph-mon[30002]: pgmap v1517: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 511 op/s; 20494/224481 objects degraded (9.130%); 36136/224481 objects misplaced (16.098%); 16 MiB/s, 255 objects/s recovering 2024-09-17T16:31:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:55 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 20494/224481 objects degraded (9.130%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:55.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:55.610 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:55.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:55 smithi173 ceph-mon[30530]: pgmap v1517: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 511 op/s; 20494/224481 objects degraded (9.130%); 36136/224481 objects misplaced (16.098%); 16 MiB/s, 255 objects/s recovering 2024-09-17T16:31:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:55 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 20494/224481 objects degraded (9.130%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:55.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:55.736 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:55.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:55.737 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:55 smithi017 ceph-mon[27571]: pgmap v1517: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 511 op/s; 20494/224481 objects degraded (9.130%); 36136/224481 objects misplaced (16.098%); 16 MiB/s, 255 objects/s recovering 2024-09-17T16:31:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:55 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 20494/224481 objects degraded (9.130%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:31:55.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:55.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:55.860 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:55.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:55.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:55.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:56.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:56.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:56.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:56.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:56.240 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:56.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:56.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:56.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:56.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:56.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:56.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:56.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:56.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:56.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:56.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:56.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:56.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:56.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:56.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:56.842 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:56.899 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:56.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:57.001 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:31:57.047 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:57.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:31:57.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:57.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:57.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:57.273 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:31:57.299 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:57.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:31:57.386 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:31:57.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:57.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:57.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:57.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:57 smithi017 ceph-mon[27571]: pgmap v1518: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 20494/223425 objects degraded (9.173%); 36136/223425 objects misplaced (16.174%); 15 MiB/s, 239 objects/s recovering 2024-09-17T16:31:57.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:57.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:57.550 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:57 smithi071 ceph-mon[30002]: pgmap v1518: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 20494/223425 objects degraded (9.173%); 36136/223425 objects misplaced (16.174%); 15 MiB/s, 239 objects/s recovering 2024-09-17T16:31:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:57 smithi173 ceph-mon[30530]: pgmap v1518: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 20494/223425 objects degraded (9.173%); 36136/223425 objects misplaced (16.174%); 15 MiB/s, 239 objects/s recovering 2024-09-17T16:31:57.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:57.711 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:57.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:57.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:57.834 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:57.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:57.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:57.958 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:57.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:58.060 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:58.096 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:58.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:58.100 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:31:58.100 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:31:58.100 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 685, 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:31:49.218516+0000", 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:29:47.857157+0000", 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5799936, 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:31:58.101 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 7, 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:31:58.102 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:31:58.103 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:31:58.104 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "669", 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:31:58.105 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:31:58.106 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.107 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:31:58.108 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:31:58.109 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 7, 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_5", 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:29:33.619938+0000", 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:31:58.110 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 26, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 26, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 26, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 26, 2024-09-17T16:31:58.111 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 26, 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "676", 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:31:58.112 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "674", 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:31:58.113 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:31:58.114 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:31:58.115 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 3.3900001049041748, 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.9600000381469727, 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1.0900000333786011, 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3.690000057220459, 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3.2300000190734863, 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 677, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.117 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.118 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.119 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:31:58.120 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.121 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 677, 2024-09-17T16:31:58.123 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:31:58.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:31:58.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:31:58.126 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.127 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 677, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.129 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:31:58.131 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:31:58.132 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:31:58.133 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 677, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.135 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.138 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:31:58.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.140 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 674, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.141 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.142 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.144 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:31:58.145 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.845672607421875, 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 674, 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.147 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.148 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:31:58.149 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:31:58.150 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.150 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.150 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.150 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.150 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.151 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:31:58.151 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.153 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.66790771484375, 2024-09-17T16:31:58.154 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 676, 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:31:58.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.159 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.8061676025390625, 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.73492431640625, 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:31:58.160 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 676, 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:31:58.161 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.162 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.163 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.165 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:31:58.166 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 676, 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:31:58.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:31:58.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.169 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.171 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.2821197509765625, 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:31:58.172 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 674, 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:31:58.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.174 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.175 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:31:58.177 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 676, 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.179 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.180 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:31:58.183 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.588836669921875, 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 291, 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.184 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.185 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.186 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.187 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:31:58.188 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.189 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:31:30.786946+0000", 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:30:49.020432+0000", 2024-09-17T16:31:58.190 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:30:58.338683+0000", 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:31:43.874166+0000", 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.192 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:31:04.347833+0000", 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.193 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:30:32.162223+0000", 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:30:36.224300+0000", 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:30:54.097393+0000", 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.195 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:30:34.202289+0000", 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.196 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:30:38.759890+0000", 2024-09-17T16:31:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:31:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.197 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:31:48.877195+0000", 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:31:43.725830+0000", 2024-09-17T16:31:58.198 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [ 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "1.0", 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: 4, 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: 6, 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: 8 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-17T16:31:58.199 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [ 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "7.10", 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: 3, 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: 9 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "7.11", 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.200 INFO:teuthology.orchestra.run.smithi017.stdout: 3, 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: 6 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "7.12", 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: 2, 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: 0 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "7.13", 2024-09-17T16:31:58.201 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: 2, 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: 0 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "7.14", 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: 5 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "7.15", 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.202 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "7.16", 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "7.17", 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: 1, 2024-09-17T16:31:58.203 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "7.18", 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: 2, 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: 7 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "7.19", 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: 0, 2024-09-17T16:31:58.204 INFO:teuthology.orchestra.run.smithi017.stdout: 2 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:31:58.205 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:31:58.206 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:31:58.207 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:31:58.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:58.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:58.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:58.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:58.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:58.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:58.458 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:58.479 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:58.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/95162018' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:31:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/95162018' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:31:58.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:58.602 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:58.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/95162018' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:31:58.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:31:58.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:58.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:31:58.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:31:58.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:58.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:31:58.951 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:58.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:58.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:31:59.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:59.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:59.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:59.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:31:59.243 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:59.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:59.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:31:59.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:31:59.367 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:59.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:31:59.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:31:59.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:31:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:59.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:31:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:59 smithi071 ceph-mon[30002]: pgmap v1519: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 19906/217197 objects degraded (9.165%); 34149/217197 objects misplaced (15.723%); 15 MiB/s, 237 objects/s recovering 2024-09-17T16:31:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:59 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:31:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:59 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:31:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:31:59 smithi071 ceph-mon[30002]: osdmap e686: 12 total, 12 up, 12 in 2024-09-17T16:31:59.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:31:59.614 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:59.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:59.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:31:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:59 smithi173 ceph-mon[30530]: pgmap v1519: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 19906/217197 objects degraded (9.165%); 34149/217197 objects misplaced (15.723%); 15 MiB/s, 237 objects/s recovering 2024-09-17T16:31:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:59 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:31:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:59 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:31:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:31:59 smithi173 ceph-mon[30530]: osdmap e686: 12 total, 12 up, 12 in 2024-09-17T16:31:59.716 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:31:59.738 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:31:59.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:59.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:31:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:59 smithi017 ceph-mon[27571]: pgmap v1519: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 19906/217197 objects degraded (9.165%); 34149/217197 objects misplaced (15.723%); 15 MiB/s, 237 objects/s recovering 2024-09-17T16:31:59.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:59 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:31:59.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:59 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:31:59.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:31:59 smithi017 ceph-mon[27571]: osdmap e686: 12 total, 12 up, 12 in 2024-09-17T16:31:59.839 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:31:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:59.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:59.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:31:59.963 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:31:59.985 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:31:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:31:59.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:00.087 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:00.135 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:00.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:00.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:00.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:00.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:00.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:00.393 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:00.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:00.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:00.495 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:00.543 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:00.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:00 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 19906/217197 objects degraded (9.165%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:00 smithi071 ceph-mon[30002]: osdmap e687: 12 total, 12 up, 12 in 2024-09-17T16:32:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:00 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 19906/217197 objects degraded (9.165%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:00 smithi173 ceph-mon[30530]: osdmap e687: 12 total, 12 up, 12 in 2024-09-17T16:32:00.645 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:00.693 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:00.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:00.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:00 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 19906/217197 objects degraded (9.165%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:00 smithi017 ceph-mon[27571]: osdmap e687: 12 total, 12 up, 12 in 2024-09-17T16:32:00.796 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:00.818 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:00.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:00.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:00.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:00.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:00.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:01.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:01.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:01.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:01.169 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:01.192 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:01.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:01.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:01.315 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:01.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:01.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:01.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:01.459 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:01.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:01.561 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:01 smithi071 ceph-mon[30002]: pgmap v1522: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 454 op/s; 19906/216753 objects degraded (9.184%); 34149/216753 objects misplaced (15.755%); 13 MiB/s, 208 objects/s recovering 2024-09-17T16:32:01.593 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:01.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:01 smithi173 ceph-mon[30530]: pgmap v1522: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 454 op/s; 19906/216753 objects degraded (9.184%); 34149/216753 objects misplaced (15.755%); 13 MiB/s, 208 objects/s recovering 2024-09-17T16:32:01.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:01.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:01.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:01 smithi017 ceph-mon[27571]: pgmap v1522: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 454 op/s; 19906/216753 objects degraded (9.184%); 34149/216753 objects misplaced (15.755%); 13 MiB/s, 208 objects/s recovering 2024-09-17T16:32:01.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:01.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:01.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:01.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:01.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:01.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:02.075 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:02.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:02.103 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:02.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:02.243 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:02.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:02.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:02.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:02.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:02.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:02.469 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:02.501 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:02.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:02.603 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:02.624 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:02.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:02.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:02.748 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:02.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:02.849 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:02.873 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:02.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:02 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:32:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:02 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:32:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:02 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:32:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:02 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:32:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:02 smithi173 ceph-mon[30530]: osdmap e688: 12 total, 12 up, 12 in 2024-09-17T16:32:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:02 smithi173 ceph-mon[30530]: pgmap v1524: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 566 op/s; 19502/213222 objects degraded (9.146%); 33046/213222 objects misplaced (15.498%); 20 MiB/s, 317 objects/s recovering 2024-09-17T16:32:02.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:02 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:32:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:02 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:32:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:02 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:32:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:02 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:32:02.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:02 smithi017 ceph-mon[27571]: osdmap e688: 12 total, 12 up, 12 in 2024-09-17T16:32:02.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:02 smithi017 ceph-mon[27571]: pgmap v1524: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 566 op/s; 19502/213222 objects degraded (9.146%); 33046/213222 objects misplaced (15.498%); 20 MiB/s, 317 objects/s recovering 2024-09-17T16:32:02.996 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:02.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:02 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:32:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:02 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:32:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:02 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:32:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:02 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:32:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:02 smithi071 ceph-mon[30002]: osdmap e688: 12 total, 12 up, 12 in 2024-09-17T16:32:03.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:02 smithi071 ceph-mon[30002]: pgmap v1524: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 566 op/s; 19502/213222 objects degraded (9.146%); 33046/213222 objects misplaced (15.498%); 20 MiB/s, 317 objects/s recovering 2024-09-17T16:32:03.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:03.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:03.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:03.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:03.243 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:03.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:03.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:03.367 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:03.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:03.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:03.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:03.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:03.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:03.616 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:03.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:03.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:03.719 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:03.784 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:03.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:03.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:03.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:03 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:32:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:03 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:32:03.910 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:03.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:03.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:03 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:32:03.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:03 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:32:04.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:04.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:04.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:04.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:03 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:32:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:03 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:32:04.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:04.160 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:04.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:04.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:04.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:04.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:04.388 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:04.426 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:04.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:04.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:04.565 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:04.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:04.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:04.667 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:04.689 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:04.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:04.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:04.812 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:04.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:04.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:04.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:04.955 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:04.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:04.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:04 smithi017 ceph-mon[27571]: osdmap e689: 12 total, 12 up, 12 in 2024-09-17T16:32:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:04 smithi017 ceph-mon[27571]: pgmap v1526: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s; 19356/210564 objects degraded (9.192%); 32080/210564 objects misplaced (15.235%); 13 MiB/s, 204 objects/s recovering 2024-09-17T16:32:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:04 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 19502/213222 objects degraded (9.146%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:05.057 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:05.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:04 smithi071 ceph-mon[30002]: osdmap e689: 12 total, 12 up, 12 in 2024-09-17T16:32:05.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:04 smithi071 ceph-mon[30002]: pgmap v1526: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s; 19356/210564 objects degraded (9.192%); 32080/210564 objects misplaced (15.235%); 13 MiB/s, 204 objects/s recovering 2024-09-17T16:32:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:04 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 19502/213222 objects degraded (9.146%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:05.078 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:05.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:05.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:04 smithi173 ceph-mon[30530]: osdmap e689: 12 total, 12 up, 12 in 2024-09-17T16:32:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:04 smithi173 ceph-mon[30530]: pgmap v1526: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s; 19356/210564 objects degraded (9.192%); 32080/210564 objects misplaced (15.235%); 13 MiB/s, 204 objects/s recovering 2024-09-17T16:32:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:04 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 19502/213222 objects degraded (9.146%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:05.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:05.220 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:05.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:05.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:05.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:05.347 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:05.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:05.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:05.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:05.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:05.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:05.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:05.595 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:05.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:05.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:05.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:05.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:05.825 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:05.846 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:05.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:05.948 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:05.995 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:05.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:05.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:06.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:06.134 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:06.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:06.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:06.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:06.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:06.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:06.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:06.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:06.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:06.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:06.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:06.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:06.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:06.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:06.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:06.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:06.764 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:06.865 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:06.888 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:06.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:06.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:07.028 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:07.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:07.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:07.165 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:07.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:07.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:07.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:07.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:07.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:07.394 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:07.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:07 smithi173 ceph-mon[30530]: pgmap v1527: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 19356/209385 objects degraded (9.244%); 32080/209385 objects misplaced (15.321%); 12 MiB/s, 184 objects/s recovering 2024-09-17T16:32:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:07 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:32:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:07 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:32:07.431 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:07.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:07 smithi017 ceph-mon[27571]: pgmap v1527: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 19356/209385 objects degraded (9.244%); 32080/209385 objects misplaced (15.321%); 12 MiB/s, 184 objects/s recovering 2024-09-17T16:32:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:07 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:32:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:07 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:32:07.533 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:07.557 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:07.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:07 smithi071 ceph-mon[30002]: pgmap v1527: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 19356/209385 objects degraded (9.244%); 32080/209385 objects misplaced (15.321%); 12 MiB/s, 184 objects/s recovering 2024-09-17T16:32:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:07 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:32:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:07 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:32:07.659 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:07.681 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:07.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:07.783 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:07.826 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:07.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:07.928 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:07.950 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:07.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:08.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:08.074 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:08.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:08.176 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:08.197 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:08.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:08.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:08.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:08.320 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:08.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:08 smithi173 ceph-mon[30530]: osdmap e690: 12 total, 12 up, 12 in 2024-09-17T16:32:08.422 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:08.467 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:08.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:08.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:08 smithi017 ceph-mon[27571]: osdmap e690: 12 total, 12 up, 12 in 2024-09-17T16:32:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:08 smithi071 ceph-mon[30002]: osdmap e690: 12 total, 12 up, 12 in 2024-09-17T16:32:08.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:08.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:08.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:08.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:08.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:08.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:08.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:08.853 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:08.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:08.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:08.977 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:08.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:09.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:09.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:09.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:09.207 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:09.252 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:09.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:09.355 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:09.378 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:09.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:09 smithi173 ceph-mon[30530]: pgmap v1529: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 18682/202884 objects degraded (9.208%); 29932/202884 objects misplaced (14.753%); 24 MiB/s, 383 objects/s recovering 2024-09-17T16:32:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:09 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:32:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:09 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:32:09.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:09 smithi071 ceph-mon[30002]: pgmap v1529: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 18682/202884 objects degraded (9.208%); 29932/202884 objects misplaced (14.753%); 24 MiB/s, 383 objects/s recovering 2024-09-17T16:32:09.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:09 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:32:09.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:09 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:32:09.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:09 smithi017 ceph-mon[27571]: pgmap v1529: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 18682/202884 objects degraded (9.208%); 29932/202884 objects misplaced (14.753%); 24 MiB/s, 383 objects/s recovering 2024-09-17T16:32:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:09 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:32:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:09 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:32:09.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:09.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:09.606 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:09.657 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:09.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:09.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:09.802 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:09.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:09.904 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:09.925 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:09.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:10.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:10.066 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:10.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:10.168 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:10.211 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:10.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:10.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:10.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:10.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:10.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:10 smithi173 ceph-mon[30530]: osdmap e691: 12 total, 12 up, 12 in 2024-09-17T16:32:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 18682/202884 objects degraded (9.208%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:10.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:10.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:10.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:10 smithi017 ceph-mon[27571]: osdmap e691: 12 total, 12 up, 12 in 2024-09-17T16:32:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 18682/202884 objects degraded (9.208%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:10 smithi071 ceph-mon[30002]: osdmap e691: 12 total, 12 up, 12 in 2024-09-17T16:32:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 18682/202884 objects degraded (9.208%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:10.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:10.590 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:10.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:10.692 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:10.716 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:10.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:10.818 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:10.871 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:10.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:10.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:10.994 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:10.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:11.096 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:11.119 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:11.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:11.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:11.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:11.243 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:11.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:11.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:11.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:11.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:11.373 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:11 smithi173 ceph-mon[30530]: pgmap v1531: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 18682/202425 objects degraded (9.229%); 29932/202425 objects misplaced (14.787%); 12 MiB/s, 199 objects/s recovering 2024-09-17T16:32:11.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:11.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:11 smithi017 ceph-mon[27571]: pgmap v1531: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 18682/202425 objects degraded (9.229%); 29932/202425 objects misplaced (14.787%); 12 MiB/s, 199 objects/s recovering 2024-09-17T16:32:11.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:11.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:11 smithi071 ceph-mon[30002]: pgmap v1531: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 18682/202425 objects degraded (9.229%); 29932/202425 objects misplaced (14.787%); 12 MiB/s, 199 objects/s recovering 2024-09-17T16:32:11.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:11.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:11.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:11.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:11.731 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:11.750 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:82763 5.13 e475/453 5:cd63b9f8:::benchmark_data_smithi173_90735_object82762:head v 475'4007, mlcod=475'4003)", 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 530.73142525000003, 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56469166699999995, 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.751 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918199+0000", 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918200+0000", 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1500000000000004e-07 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.918245+0000", 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5034000000000003e-05 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481583+0000", 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56333854699999997 2024-09-17T16:32:11.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481588+0000", 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5800000000000002e-06 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481652+0000", 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4309999999999999e-05 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.481669+0000", 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6880000000000001e-05 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.482864+0000", 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011944200000000001 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.482891+0000", 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7180999999999999e-05 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.755 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPull(5.12 e475/455 cost 1049576)", 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 530.69016568200004, 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54846142799999997, 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959459+0000", 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959460+0000", 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5099999999999998e-07 2024-09-17T16:32:11.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:19.959470+0000", 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9380000000000004e-06 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507615+0000", 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.54814523100000001 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507626+0000", 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1484000000000001e-05 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507695+0000", 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8990000000000002e-05 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.507920+0000", 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022483400000000001 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.759 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(5.d 475/455 [PushOp(5:b9f39261:::benchmark_data_smithi173_90735_object37371:head, version: 439'2383, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:b9f39261:::benchmark_data_smithi173_90735_object37371:head@439'2383, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 530.16924803100005, 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45551481399999999, 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480376+0000", 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480378+0000", 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6470000000000001e-06 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.480434+0000", 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6162999999999997e-05 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934248+0000", 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45381411300000002 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934260+0000", 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1382e-05 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934362+0000", 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000102681 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.934379+0000", 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6498e-05 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:20.935891+0000", 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015123300000000001 2024-09-17T16:32:11.763 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:95580 5.b e482/419 5:d4bc1ca6:::benchmark_data_smithi173_90735_object12809:head v 482'5935, mlcod=482'5933)", 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 516.91715721699995, 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39904708700000002, 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732467+0000", 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732468+0000", 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-17T16:32:11.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.732477+0000", 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4880000000000007e-06 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128932+0000", 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.396455369 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128940+0000", 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5089999999999996e-06 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.128990+0000", 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0068999999999998e-05 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.129012+0000", 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2218e-05 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.131477+0000", 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002464843 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.131514+0000", 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7515000000000003e-05 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.768 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26158.0:95584 5.b e482/419 5:dc96c928:::benchmark_data_smithi173_90735_object12813:head v 482'5936, mlcod=482'5933)", 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 516.76551002300005, 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39023148299999999, 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884114+0000", 2024-09-17T16:32:11.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884116+0000", 2024-09-17T16:32:11.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9319999999999999e-06 2024-09-17T16:32:11.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:33.884127+0000", 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0198e-05 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272085+0000", 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38795868300000003 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272089+0000", 2024-09-17T16:32:11.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1729999999999997e-06 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272129+0000", 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9937000000000001e-05 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.272145+0000", 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5863e-05 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.274307+0000", 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021616040000000001 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:23:34.274346+0000", 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9093e-05 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17628 6.9 6:9546f3b4:::benchmark_data_smithi173_102017_object17627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:32:11.773 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:21.997751+0000", 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 408.65187322100002, 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49739271200000001, 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17628 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997751+0000", 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997751+0000", 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997757+0000", 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3489999999999999e-06 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997802+0000", 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5543000000000001e-05 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997822+0000", 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9443999999999999e-05 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.997831+0000", 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9740000000000008e-06 2024-09-17T16:32:11.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998743+0000", 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091201499999999996 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.998896+0000", 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015307299999999999 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.999045+0000", 2024-09-17T16:32:11.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014901199999999999 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000028+0000", 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00098290300000000003 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.450171+0000", 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45014342099999999 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.495073+0000", 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.044902035999999999 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.495110+0000", 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7134000000000001e-05 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.495144+0000", 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3807999999999999e-05 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17643 6.9 6:992c966e:::benchmark_data_smithi173_102017_object17642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.493337+0000", 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 408.15628740599999, 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42163531900000001, 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:32:11.780 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17643 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493337+0000", 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493337+0000", 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493339+0000", 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4229999999999999e-06 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493447+0000", 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107039 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493469+0000", 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2408999999999998e-05 2024-09-17T16:32:11.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493478+0000", 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6300000000000004e-06 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493543+0000", 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5235999999999997e-05 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493660+0000", 2024-09-17T16:32:11.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011726999999999999 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.493766+0000", 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010634899999999999 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.494765+0000", 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099869900000000003 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.494990+0000", 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000224739 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.914894+0000", 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41990391300000002 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.914938+0000", 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4121e-05 2024-09-17T16:32:11.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.914972+0000", 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4490999999999999e-05 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(6.d 582/568 [PushReplyOp(6:bd4f72bd:::benchmark_data_smithi173_102017_object30472:head)])", 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:21.019823+0000", 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 289.62980101199997, 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48108583100000002, 2024-09-17T16:32:11.786 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.019823+0000", 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.019823+0000", 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.019825+0000", 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.085e-06 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.019830+0000", 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6720000000000001e-06 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.497888+0000", 2024-09-17T16:32:11.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47805754900000003 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.497896+0000", 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6370000000000005e-06 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.497969+0000", 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2470000000000002e-05 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:21.500909+0000", 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029404180000000002 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:95592 7.6 7:68d68028:::benchmark_data_smithi173_116396_object95591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e671)", 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:09.933181+0000", 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.716443742999999, 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61727390299999996, 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.790 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 95592 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933181+0000", 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933181+0000", 2024-09-17T16:32:11.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933182+0000", 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4279999999999999e-06 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933214+0000", 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2320999999999998e-05 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933231+0000", 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6123000000000001e-05 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933238+0000", 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.956e-06 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933298+0000", 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0278999999999997e-05 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933451+0000", 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015354499999999999 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933603+0000", 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000151495 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.936309+0000", 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027063360000000002 2024-09-17T16:32:11.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.214031+0000", 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27772170200000001 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.550370+0000", 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33633900999999999 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.550416+0000", 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5800000000000002e-05 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.550455+0000", 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8908000000000001e-05 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:95592 7.6 e671/629 ondisk, result = 0)", 2024-09-17T16:32:11.796 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:09.936213+0000", 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.713411115, 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61424870399999998, 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.936213+0000", 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.936213+0000", 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.936214+0000", 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5099999999999999e-07 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.936218+0000", 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0559999999999998e-06 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.550136+0000", 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.61391777300000006 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.550153+0000", 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7212999999999999e-05 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.550335+0000", 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000182175 2024-09-17T16:32:11.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.550364+0000", 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8819e-05 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.550462+0000", 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7917000000000005e-05 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:102657 7.19 7:9baf5427:::benchmark_data_smithi173_116396_object7057:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e678)", 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:22.966767+0000", 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.682857916000003, 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.40274460200000001, 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 102657 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.801 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.966767+0000", 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.966767+0000", 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.966767+0000", 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0100000000000004e-07 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.966775+0000", 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3980000000000002e-06 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.966789+0000", 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4397e-05 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.966796+0000", 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3540000000000002e-06 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.966836+0000", 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9178000000000003e-05 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.966939+0000", 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103411 2024-09-17T16:32:11.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,6,7", 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.967061+0000", 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000121539 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.968746+0000", 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016857829999999999 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.975067+0000", 2024-09-17T16:32:11.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063205370000000002 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:23.014644+0000", 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.039577257999999997 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:23.367547+0000", 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35290236800000002 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:23.367607+0000", 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0843e-05 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:23.369511+0000", 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019037349999999999 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:102657 7.19 e678/677 ondisk, result = 0)", 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:22.970026+0000", 2024-09-17T16:32:11.807 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.679598005000003, 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39950242200000002, 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.970026+0000", 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.970026+0000", 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.970027+0000", 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:22.970032+0000", 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6789999999999996e-06 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:23.365997+0000", 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.395965183 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:23.366007+0000", 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3149999999999998e-06 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:23.367530+0000", 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015231260000000001 2024-09-17T16:32:11.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:23.367540+0000", 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0087e-05 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:23.369529+0000", 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001989153 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:111366 7.19 7:9aebc2a1:::benchmark_data_smithi173_116396_object15766:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e681)", 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:41.239413+0000", 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 29.410211305000001, 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45416669599999998, 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 111366 2024-09-17T16:32:11.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.239413+0000", 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.239413+0000", 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.239415+0000", 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7460000000000001e-06 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.239420+0000", 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4600000000000002e-06 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.239430+0000", 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6490000000000003e-06 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.239436+0000", 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.19e-06 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.241642+0000", 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022060729999999998 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.241799+0000", 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015709200000000001 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,6,7", 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.241945+0000", 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014582500000000001 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.247233+0000", 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052874990000000002 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.247256+0000", 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3657000000000001e-05 2024-09-17T16:32:11.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.250118+0000", 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028616010000000001 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.691462+0000", 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44134400099999999 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.691591+0000", 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129424 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.693580+0000", 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001988479 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:111366 7.19 e681/677 ondisk, result = 0)", 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:41.245421+0000", 2024-09-17T16:32:11.818 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 29.40420387, 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.45098158100000002, 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.245421+0000", 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.245421+0000", 2024-09-17T16:32:11.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.245421+0000", 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9299999999999998e-07 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.245425+0000", 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9990000000000002e-06 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.686834+0000", 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44140849900000001 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.686838+0000", 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5630000000000002e-06 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.691441+0000", 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046026909999999999 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.691454+0000", 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.296e-05 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.696402+0000", 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049482759999999997 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:111413 7.4 7:23072244:::benchmark_data_smithi173_116396_object15813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e681)", 2024-09-17T16:32:11.822 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:41.279161+0000", 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 29.370463375, 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36654876600000003, 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 111413 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.279161+0000", 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.279161+0000", 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.279163+0000", 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9920000000000002e-06 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.279167+0000", 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8909999999999997e-06 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.301381+0000", 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022214093000000001 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.301393+0000", 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1457999999999999e-05 2024-09-17T16:32:11.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.304164+0000", 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027712259999999999 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.304288+0000", 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012415700000000001 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,4", 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.304456+0000", 2024-09-17T16:32:11.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000168138 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.307272+0000", 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002815807 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.376706+0000", 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.069434015000000002 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.645606+0000", 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26889999999999997 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.645665+0000", 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9586999999999997e-05 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.645710+0000", 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4401999999999999e-05 2024-09-17T16:32:11.828 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:111413 7.4 e681/674 ondisk, result = 0)", 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:41.306585+0000", 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 29.343039488999999, 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33913036400000002, 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.306585+0000", 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.306585+0000", 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.306585+0000", 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9399999999999995e-07 2024-09-17T16:32:11.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.306589+0000", 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2490000000000001e-06 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.645516+0000", 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33892746499999998 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.645524+0000", 2024-09-17T16:32:11.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3370000000000002e-06 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.645590+0000", 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6815000000000003e-05 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.645600+0000", 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2609999999999999e-06 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:41.645715+0000", 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011574300000000001 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:111972 7.4 7:264eda1e:::benchmark_data_smithi173_116396_object16372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e682)", 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:42.580443+0000", 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.06918168, 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.40728628900000002, 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.833 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 111972 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.580443+0000", 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.580443+0000", 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.580444+0000", 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1799999999999999e-06 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.580448+0000", 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.602e-06 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.580459+0000", 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1158999999999999e-05 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.580466+0000", 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0489999999999998e-06 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.585037+0000", 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045713409999999996 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.585222+0000", 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018534700000000001 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,4", 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.585368+0000", 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014515900000000001 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.593605+0000", 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082372309999999994 2024-09-17T16:32:11.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.662439+0000", 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.068834160000000005 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.987574+0000", 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32513546100000001 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.987624+0000", 2024-09-17T16:32:11.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9391999999999997e-05 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:42.987729+0000", 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000105208 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:116737 7.6 7:672e1b08:::benchmark_data_smithi173_116396_object21137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e685)", 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:52.339562+0000", 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.310062737999999, 2024-09-17T16:32:11.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34314308300000002, 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 116737 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.339562+0000", 2024-09-17T16:32:11.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.339562+0000", 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.339563+0000", 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6600000000000005e-07 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.339568+0000", 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6030000000000004e-06 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.339581+0000", 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2475e-05 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.339588+0000", 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6590000000000007e-06 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.339653+0000", 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4535999999999993e-05 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.339782+0000", 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129154 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.339904+0000", 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012171300000000001 2024-09-17T16:32:11.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.341912+0000", 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020081629999999999 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.347062+0000", 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051502329999999997 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.682615+0000", 2024-09-17T16:32:11.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33555300199999999 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.682661+0000", 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5652999999999998e-05 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.682705+0000", 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4026000000000002e-05 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:116737 7.6 e685/674 ondisk, result = 0)", 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:52.345977+0000", 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.303647903000002, 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33673836600000001, 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.345977+0000", 2024-09-17T16:32:11.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.345977+0000", 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.345978+0000", 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9099999999999991e-07 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.345982+0000", 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.681795+0000", 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33581253900000002 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.681799+0000", 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.403e-06 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.682602+0000", 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080305299999999997 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.682610+0000", 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3300000000000001e-06 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:52.682715+0000", 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010528599999999999 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.849 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(7.14 686/677 [PushReplyOp(7:2b4d29bd:::benchmark_data_smithi173_116396_object24459:head)])", 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.365387+0000", 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 11.284237364999999, 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33177679799999998, 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.365387+0000", 2024-09-17T16:32:11.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.365387+0000", 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.365388+0000", 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1900000000000001e-07 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:32:11.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.365392+0000", 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3170000000000003e-06 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.691428+0000", 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32603542200000002 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.691433+0000", 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7779999999999998e-06 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.694736+0000", 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033031929999999998 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.697164+0000", 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002428169 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:32:11.853 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:32:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:11.855 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:11.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:11.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:11.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:11.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:12.082 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:12.105 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:12.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:12.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:12.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:12.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:12.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:12.333 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:12.378 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:12.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:12.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:12.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:12.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:12.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:12.630 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:12.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:12.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:12.762 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:12.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:12.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:12.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:12.886 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:12.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:12.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:13.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:13.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:13.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:13.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:13.151 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:13.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:13.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:13.297 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:13.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:13 smithi173 ceph-mon[30530]: pgmap v1532: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 18192/198231 objects degraded (9.177%); 28506/198231 objects misplaced (14.380%); 18 MiB/s, 283 objects/s recovering 2024-09-17T16:32:13.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:13.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:13.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:13 smithi017 ceph-mon[27571]: pgmap v1532: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 18192/198231 objects degraded (9.177%); 28506/198231 objects misplaced (14.380%); 18 MiB/s, 283 objects/s recovering 2024-09-17T16:32:13.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:13.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:13.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:13 smithi071 ceph-mon[30002]: pgmap v1532: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 18192/198231 objects degraded (9.177%); 28506/198231 objects misplaced (14.380%); 18 MiB/s, 283 objects/s recovering 2024-09-17T16:32:13.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:13.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:13.676 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:13.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:13.777 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:13.799 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:13.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:13.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:13.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:13.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:14.032 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:14.078 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:14.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:14.180 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:14.202 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:14.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:14.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:14.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:14.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:32:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:32:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:14 smithi173 ceph-mon[30530]: osdmap e692: 12 total, 12 up, 12 in 2024-09-17T16:32:14.441 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:14.463 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:14.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:32:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:32:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:14 smithi017 ceph-mon[27571]: osdmap e692: 12 total, 12 up, 12 in 2024-09-17T16:32:14.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:32:14.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:32:14.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:14 smithi071 ceph-mon[30002]: osdmap e692: 12 total, 12 up, 12 in 2024-09-17T16:32:14.565 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:14.588 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:14.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:14.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:14.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:14.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:14.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:14.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:14.836 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:14.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:14.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:14.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:14.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:15.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:15.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:15.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:15.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:15.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:15.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:15.319 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:15.359 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:15.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:15.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:15 smithi173 ceph-mon[30530]: pgmap v1534: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 18066/195201 objects degraded (9.255%); 27467/195201 objects misplaced (14.071%); 14 MiB/s, 231 objects/s recovering 2024-09-17T16:32:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:15 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 18066/195201 objects degraded (9.255%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:15 smithi173 ceph-mon[30530]: osdmap e693: 12 total, 12 up, 12 in 2024-09-17T16:32:15.461 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:15.483 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:15.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:15 smithi017 ceph-mon[27571]: pgmap v1534: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 18066/195201 objects degraded (9.255%); 27467/195201 objects misplaced (14.071%); 14 MiB/s, 231 objects/s recovering 2024-09-17T16:32:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:15 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 18066/195201 objects degraded (9.255%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:15 smithi017 ceph-mon[27571]: osdmap e693: 12 total, 12 up, 12 in 2024-09-17T16:32:15.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:15 smithi071 ceph-mon[30002]: pgmap v1534: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 424 op/s; 18066/195201 objects degraded (9.255%); 27467/195201 objects misplaced (14.071%); 14 MiB/s, 231 objects/s recovering 2024-09-17T16:32:15.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:15 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 18066/195201 objects degraded (9.255%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:15.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:15 smithi071 ceph-mon[30002]: osdmap e693: 12 total, 12 up, 12 in 2024-09-17T16:32:15.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:15.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:15.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:15.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:15.738 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:15.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:15.839 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:15.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:15.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:15.963 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:15.986 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:15.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:16.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:16.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:16.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:16.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:16.236 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:16.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:16.338 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:16.360 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:16.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:16.462 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:16.484 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:16.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:16.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:16.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:16.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:16.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:16.714 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:16.751 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:16.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:16.853 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:16.875 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:16.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:16.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:16.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:16.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:17.000 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:17.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:17.122 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:17.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:17.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:17.224 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:17.247 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:17.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:17.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:17.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:17.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:17.475 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:17 smithi017 ceph-mon[27571]: pgmap v1536: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 430 op/s; 18066/193950 objects degraded (9.315%); 27467/193950 objects misplaced (14.162%); 13 MiB/s, 215 objects/s recovering 2024-09-17T16:32:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:17 smithi017 ceph-mon[27571]: osdmap e694: 12 total, 12 up, 12 in 2024-09-17T16:32:17.498 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:17.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:17 smithi071 ceph-mon[30002]: pgmap v1536: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 430 op/s; 18066/193950 objects degraded (9.315%); 27467/193950 objects misplaced (14.162%); 13 MiB/s, 215 objects/s recovering 2024-09-17T16:32:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:17 smithi071 ceph-mon[30002]: osdmap e694: 12 total, 12 up, 12 in 2024-09-17T16:32:17.600 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:17.624 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:17.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:17 smithi173 ceph-mon[30530]: pgmap v1536: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 430 op/s; 18066/193950 objects degraded (9.315%); 27467/193950 objects misplaced (14.162%); 13 MiB/s, 215 objects/s recovering 2024-09-17T16:32:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:17 smithi173 ceph-mon[30530]: osdmap e694: 12 total, 12 up, 12 in 2024-09-17T16:32:17.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:17.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:17.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:17.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:17.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:17.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:17.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:17.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:18.002 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:18.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:18.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:18.147 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:18.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:18.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:18.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:18.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:18.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:18.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:18.415 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:18.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:18 smithi017 ceph-mon[27571]: osdmap e695: 12 total, 12 up, 12 in 2024-09-17T16:32:18.517 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:18.545 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:18.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:18.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:18 smithi071 ceph-mon[30002]: osdmap e695: 12 total, 12 up, 12 in 2024-09-17T16:32:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:18 smithi173 ceph-mon[30530]: osdmap e695: 12 total, 12 up, 12 in 2024-09-17T16:32:18.646 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:18.669 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:18.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:18.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:18.794 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:18.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:18.896 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:18.920 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:18.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:19.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:19.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:19.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:19.145 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:19.167 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:19.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:19.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:19.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:19.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:19.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:19.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:19.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:19.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:19 smithi017 ceph-mon[27571]: 7.10 scrub starts 2024-09-17T16:32:19.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:19 smithi017 ceph-mon[27571]: pgmap v1539: 27 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 17422/187515 objects degraded (9.291%); 25109/187515 objects misplaced (13.390%); 16 MiB/s, 296 objects/s recovering 2024-09-17T16:32:19.526 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:19 smithi071 ceph-mon[30002]: 7.10 scrub starts 2024-09-17T16:32:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:19 smithi071 ceph-mon[30002]: pgmap v1539: 27 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 17422/187515 objects degraded (9.291%); 25109/187515 objects misplaced (13.390%); 16 MiB/s, 296 objects/s recovering 2024-09-17T16:32:19.573 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:19.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:19 smithi173 ceph-mon[30530]: 7.10 scrub starts 2024-09-17T16:32:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:19 smithi173 ceph-mon[30530]: pgmap v1539: 27 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 17422/187515 objects degraded (9.291%); 25109/187515 objects misplaced (13.390%); 16 MiB/s, 296 objects/s recovering 2024-09-17T16:32:19.676 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:19.698 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:19.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:19.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:19.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:19.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:19.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:19.967 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:19.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:20.069 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:20.106 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:20.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:20.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:20.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:20.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:20.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:20.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:20.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:20.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:20 smithi017 ceph-mon[27571]: 7.10 scrub ok 2024-09-17T16:32:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:20 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 17422/187515 objects degraded (9.291%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:20.498 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:20.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:20.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:20 smithi071 ceph-mon[30002]: 7.10 scrub ok 2024-09-17T16:32:20.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:20 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 17422/187515 objects degraded (9.291%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:20.599 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:20.635 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:20.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:20.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:20.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:20 smithi173 ceph-mon[30530]: 7.10 scrub ok 2024-09-17T16:32:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:20 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 17422/187515 objects degraded (9.291%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:32:20.736 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:20.765 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:20.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:20.866 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:20.888 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:20.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:20.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:21.042 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:21.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:21.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:21.144 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:21.166 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:21.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:21.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:21.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:21.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:21.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:21.391 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:21.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:21 smithi173 ceph-mon[30530]: pgmap v1540: 27 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 17422/186996 objects degraded (9.317%); 25109/186996 objects misplaced (13.428%); 13 MiB/s, 241 objects/s recovering 2024-09-17T16:32:21.431 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:21.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:21.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:21 smithi017 ceph-mon[27571]: pgmap v1540: 27 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 17422/186996 objects degraded (9.317%); 25109/186996 objects misplaced (13.428%); 13 MiB/s, 241 objects/s recovering 2024-09-17T16:32:21.533 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:21.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:21 smithi071 ceph-mon[30002]: pgmap v1540: 27 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s; 17422/186996 objects degraded (9.317%); 25109/186996 objects misplaced (13.428%); 13 MiB/s, 241 objects/s recovering 2024-09-17T16:32:21.572 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:21.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:21.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:21.699 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:21.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:21.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:21.824 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:21.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:21.891 INFO:tasks.thrashosds.thrasher:j is {'epoch': 685, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:31:49.218516+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:29:47.857157+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '669', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-09-17T16:29:33.619938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '676', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '674', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.390000104904175, 'score_stable': 2.9600000381469727, 'optimal_score': 1.090000033378601, 'raw_score_acting': 3.690000057220459, 'raw_score_stable': 3.2300000190734863, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 677, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 677, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 677, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 677, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 674, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 0.845672607421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 674, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 0.66790771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 676, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 0.8061676025390625, 'primary_affinity': 0.73492431640625, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 676, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 676, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0.2821197509765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 674, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 676, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 0.588836669921875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:31:30.786946+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:30:49.020432+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:30:58.338683+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:31:43.874166+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:31:04.347833+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:30:32.162223+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:30:36.224300+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:30:54.097393+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:30:34.202289+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:30:38.759890+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:31:48.877195+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:31:43.725830+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [4, 6, 8]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.10', 'osds': [3, 9]}, {'pgid': '7.11', 'osds': [3, 6]}, {'pgid': '7.12', 'osds': [2, 0]}, {'pgid': '7.13', 'osds': [2, 0]}, {'pgid': '7.14', 'osds': [0, 5]}, {'pgid': '7.15', 'osds': [0, 1]}, {'pgid': '7.16', 'osds': [0, 2]}, {'pgid': '7.17', 'osds': [1, 2]}, {'pgid': '7.18', 'osds': [2, 7]}, {'pgid': '7.19', 'osds': [0, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:32:21.892 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:32:21.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:21.964 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:21.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:21.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:22.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:22.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:22.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:22.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:22.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:22.216 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:22.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:22.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:22.339 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:22.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:22.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:22.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:22.465 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:22.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:22.590 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:22.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:22.692 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:22.713 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:22.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:22.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:22.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:22.837 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:22.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:22.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:22.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:22.960 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:22.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:23.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:23.084 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:23.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:23.186 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:23.226 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:23.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:23.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:23.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:23.353 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:23.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:23 smithi017 ceph-mon[27571]: pgmap v1541: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 32 op/s; 16908/182517 objects degraded (9.264%); 23989/182517 objects misplaced (13.143%) 2024-09-17T16:32:23.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:23.462 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:32:23.465 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:32:23.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:23.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:23.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:23.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:32:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:23 smithi071 ceph-mon[30002]: pgmap v1541: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 32 op/s; 16908/182517 objects degraded (9.264%); 23989/182517 objects misplaced (13.143%) 2024-09-17T16:32:23.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:23.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:23.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:23.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:23 smithi173 ceph-mon[30530]: pgmap v1541: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 32 op/s; 16908/182517 objects degraded (9.264%); 23989/182517 objects misplaced (13.143%) 2024-09-17T16:32:23.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:23.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:23.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:23.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:23.885 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:23.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:23.987 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:24.050 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:24.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:24.152 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:24.174 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:24.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:24.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:24.301 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:24.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:24.302 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:24.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:24.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:24.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:24.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:24.607 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:24.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:24.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:24.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:24.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:24.835 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:24.858 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:24.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:24.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:24.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:24.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:24.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:25.088 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:25.134 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:25.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:25.236 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:25.258 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:25.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:25.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:25.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:25.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:25.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:25.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:25.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:25 smithi017 ceph-mon[27571]: pgmap v1542: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 16716/179145 objects degraded (9.331%); 22888/179145 objects misplaced (12.776%) 2024-09-17T16:32:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:25 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 16716/179145 objects degraded (9.331%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:25.507 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:25.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:25 smithi071 ceph-mon[30002]: pgmap v1542: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 16716/179145 objects degraded (9.331%); 22888/179145 objects misplaced (12.776%) 2024-09-17T16:32:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:25 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 16716/179145 objects degraded (9.331%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:25.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:25 smithi173 ceph-mon[30530]: pgmap v1542: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 118 op/s; 16716/179145 objects degraded (9.331%); 22888/179145 objects misplaced (12.776%) 2024-09-17T16:32:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:25 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 16716/179145 objects degraded (9.331%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:25.653 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:25.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:25.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:25.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:25.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:25.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:25.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:25.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:25.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:26.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:26.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:26.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:26.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:26.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:26.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:26.262 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:26.300 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:26.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:26.402 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:26.424 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:26.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:26.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:26 smithi017 ceph-mon[27571]: 7.1 scrub starts 2024-09-17T16:32:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:26 smithi017 ceph-mon[27571]: 7.1 scrub ok 2024-09-17T16:32:26.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:26.547 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:26.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:26.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:26 smithi071 ceph-mon[30002]: 7.1 scrub starts 2024-09-17T16:32:26.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:26 smithi071 ceph-mon[30002]: 7.1 scrub ok 2024-09-17T16:32:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:26 smithi173 ceph-mon[30530]: 7.1 scrub starts 2024-09-17T16:32:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:26 smithi173 ceph-mon[30530]: 7.1 scrub ok 2024-09-17T16:32:26.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:26.671 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:26.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:26.773 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:26.810 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:26.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:26.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:26.933 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:26.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:27.035 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:27.057 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:27.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:27.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:27.159 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:27.200 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:27.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:27.302 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:27.330 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:27.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:27.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:27.472 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:27.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:27.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:27 smithi017 ceph-mon[27571]: pgmap v1543: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 16716/177807 objects degraded (9.401%); 22888/177807 objects misplaced (12.872%) 2024-09-17T16:32:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:27 smithi071 ceph-mon[30002]: pgmap v1543: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 16716/177807 objects degraded (9.401%); 22888/177807 objects misplaced (12.872%) 2024-09-17T16:32:27.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:27.595 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:27.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:27 smithi173 ceph-mon[30530]: pgmap v1543: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 141 op/s; 16716/177807 objects degraded (9.401%); 22888/177807 objects misplaced (12.872%) 2024-09-17T16:32:27.696 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:27.720 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:27.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:27.823 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:27.844 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:27.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:27.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:27.967 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:27.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:27.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:28.069 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:28.093 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:28.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:28.194 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:28.223 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:28.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:28.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:28.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:28.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:28.455 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:28.492 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:28.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:28.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:28.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:28.616 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:28.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:28.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:28.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:28.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:28.893 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:28.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:28.995 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:29.040 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:29.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:29.142 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:29.176 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:29.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:29.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:29.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:29.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:29.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:29.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:29.425 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:29.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:29 smithi017 ceph-mon[27571]: pgmap v1544: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 184 op/s; 15990/170751 objects degraded (9.365%); 20539/170751 objects misplaced (12.029%) 2024-09-17T16:32:29.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:29.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:29.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:29 smithi071 ceph-mon[30002]: pgmap v1544: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 184 op/s; 15990/170751 objects degraded (9.365%); 20539/170751 objects misplaced (12.029%) 2024-09-17T16:32:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:29 smithi173 ceph-mon[30530]: pgmap v1544: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 184 op/s; 15990/170751 objects degraded (9.365%); 20539/170751 objects misplaced (12.029%) 2024-09-17T16:32:29.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:29.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:29.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:29.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:29.803 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:29.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:29.905 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:29.926 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:29.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:30.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:30.052 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:30.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:30.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:30.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:30.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:30.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:30.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:30.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:30.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:30.429 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:30.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:30 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:32:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:30 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:32:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:30 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:32:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:30 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:32:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:30 smithi017 ceph-mon[27571]: osdmap e696: 12 total, 12 up, 12 in 2024-09-17T16:32:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:30 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 15990/170751 objects degraded (9.365%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:30.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:30.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:30.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:30 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:32:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:30 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:32:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:30 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:32:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:30 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:32:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:30 smithi071 ceph-mon[30002]: osdmap e696: 12 total, 12 up, 12 in 2024-09-17T16:32:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:30 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 15990/170751 objects degraded (9.365%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:30 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:32:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:30 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:32:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:30 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:32:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:30 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:32:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:30 smithi173 ceph-mon[30530]: osdmap e696: 12 total, 12 up, 12 in 2024-09-17T16:32:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:30 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 15990/170751 objects degraded (9.365%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:30.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:30.677 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:30.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:30.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:30.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:30.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:30.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:30.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:30.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:30.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:30.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:31.031 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:31.058 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:31.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:31.159 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:31.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:31.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:31.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:31.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:31.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:31.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:31.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:31.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:31.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:31.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:31 smithi017 ceph-mon[27571]: pgmap v1546: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 157 op/s; 15990/170313 objects degraded (9.389%); 20539/170313 objects misplaced (12.060%) 2024-09-17T16:32:31.553 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:31 smithi071 ceph-mon[30002]: pgmap v1546: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 157 op/s; 15990/170313 objects degraded (9.389%); 20539/170313 objects misplaced (12.060%) 2024-09-17T16:32:31.575 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:31.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:31 smithi173 ceph-mon[30530]: pgmap v1546: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 157 op/s; 15990/170313 objects degraded (9.389%); 20539/170313 objects misplaced (12.060%) 2024-09-17T16:32:31.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:31.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:31.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:31.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:31.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:31.849 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:31.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:31.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:31.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:31.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:31.973 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:32.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:32.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:32.103 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:32.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:32.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:32.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:32.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:32.330 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:32.352 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:32.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:32.453 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:32.474 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:32.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:32.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:32.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2223426399' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:32:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2223426399' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:32:32.576 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:32.612 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:32.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2223426399' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:32:32.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:32.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:32.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:32.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:32.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:32.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:32.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:32.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:32.987 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:32:32.987 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:32:32.987 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:32:32.987 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:32:32.988 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:32:32.988 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:32:32.988 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:32:32.988 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:32:32.988 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:32:32.988 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:32:32.988 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:32:32.988 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:32:32.988 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:32:32.988 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:32:32.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:32:32.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:32:32.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:32:32.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:32:32.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:32:32.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:32:32.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:32:32.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:32:32.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:32:32.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:32:32.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:32.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:32.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:33.098 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:33.120 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:33.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:33.222 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:33 smithi017 ceph-mon[27571]: pgmap v1547: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 15456/167076 objects degraded (9.251%); 19894/167076 objects misplaced (11.907%); 14 MiB/s, 222 objects/s recovering 2024-09-17T16:32:33.245 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:33.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:33.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:33.369 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:33.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:33.471 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:33.509 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:33.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:33 smithi071 ceph-mon[30002]: pgmap v1547: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 15456/167076 objects degraded (9.251%); 19894/167076 objects misplaced (11.907%); 14 MiB/s, 222 objects/s recovering 2024-09-17T16:32:33.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:33.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:33.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:33 smithi173 ceph-mon[30530]: pgmap v1547: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 514 op/s; 15456/167076 objects degraded (9.251%); 19894/167076 objects misplaced (11.907%); 14 MiB/s, 222 objects/s recovering 2024-09-17T16:32:33.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:33.757 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:33.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:33.859 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:33.881 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:33.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:33.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:34.004 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:34.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:34.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:34.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:34.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:34.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:34.232 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:34.255 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:34.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:34.357 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:34.399 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:34.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:34.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:34.525 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:34.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:34.627 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:34.652 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:34.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:34.755 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:34.777 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:34.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:34.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:34.912 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:34.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:34.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:35.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:35.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:35.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:35.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:35.162 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:35.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:35.263 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:35.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:35.287 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:35.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:35.389 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:35.426 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:35.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:35.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:35.550 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:35.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:35 smithi071 ceph-mon[30002]: pgmap v1548: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 15284/162969 objects degraded (9.378%); 18060/162969 objects misplaced (11.082%); 15 MiB/s, 246 objects/s recovering 2024-09-17T16:32:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:35 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 15284/162969 objects degraded (9.378%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:35 smithi173 ceph-mon[30530]: pgmap v1548: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 15284/162969 objects degraded (9.378%); 18060/162969 objects misplaced (11.082%); 15 MiB/s, 246 objects/s recovering 2024-09-17T16:32:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:35 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 15284/162969 objects degraded (9.378%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:35.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:35.703 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:35.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:35 smithi017 ceph-mon[27571]: pgmap v1548: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 15284/162969 objects degraded (9.378%); 18060/162969 objects misplaced (11.082%); 15 MiB/s, 246 objects/s recovering 2024-09-17T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:35 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 15284/162969 objects degraded (9.378%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:35.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:35.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:35.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:35.930 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:35.955 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:35.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:36.057 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:36.080 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:36.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:36.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:36.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:36.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:36.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:36.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:36.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:36.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:36.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:36.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:36.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:36.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:36 smithi071 ceph-mon[30002]: 7.0 scrub starts 2024-09-17T16:32:36.587 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:36.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:36 smithi173 ceph-mon[30530]: 7.0 scrub starts 2024-09-17T16:32:36.689 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:36.733 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:36.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:36.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:36 smithi017 ceph-mon[27571]: 7.0 scrub starts 2024-09-17T16:32:36.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:36.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:36.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:36.957 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:37.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:37.001 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:37.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:37.102 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:37.125 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:37.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:37.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:37.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:37.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:37.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:37.374 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:37.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:37.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:37.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:37.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:37 smithi071 ceph-mon[30002]: pgmap v1549: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 15284/161601 objects degraded (9.458%); 18060/161601 objects misplaced (11.176%); 15 MiB/s, 246 objects/s recovering 2024-09-17T16:32:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:37 smithi071 ceph-mon[30002]: 7.0 scrub ok 2024-09-17T16:32:37.600 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:37.623 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:37.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:37 smithi173 ceph-mon[30530]: pgmap v1549: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 15284/161601 objects degraded (9.458%); 18060/161601 objects misplaced (11.176%); 15 MiB/s, 246 objects/s recovering 2024-09-17T16:32:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:37 smithi173 ceph-mon[30530]: 7.0 scrub ok 2024-09-17T16:32:37.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:37 smithi017 ceph-mon[27571]: pgmap v1549: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 15284/161601 objects degraded (9.458%); 18060/161601 objects misplaced (11.176%); 15 MiB/s, 246 objects/s recovering 2024-09-17T16:32:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:37 smithi017 ceph-mon[27571]: 7.0 scrub ok 2024-09-17T16:32:37.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:37.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:37.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:37.873 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:37.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:37.940 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:32:37.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:38.045 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:38.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:38.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:38.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:38.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:38.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:38.262 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:32:38.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:38.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:38.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:38.397 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:38.418 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:38.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:38.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:38.559 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:38.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:38.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2347817515' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:32:38.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2347817515' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:32:38.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:38.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:38.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:38.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2347817515' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:32:38.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:38.812 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:38.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:38.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:38.956 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:38.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:38.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:39.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:39.080 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:39.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:39.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:39.211 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:39.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:39.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:39.357 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:39.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:39.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:39.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:39.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:39.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:39 smithi173 ceph-mon[30530]: pgmap v1550: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 14635/155028 objects degraded (9.440%); 15478/155028 objects misplaced (9.984%); 16 MiB/s, 253 objects/s recovering 2024-09-17T16:32:39.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:39 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:32:39.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2347817515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:32:39.546 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:39 smithi173 ceph-mon[30530]: osdmap e697: 12 total, 12 up, 12 in 2024-09-17T16:32:39.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:39 smithi071 ceph-mon[30002]: pgmap v1550: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 14635/155028 objects degraded (9.440%); 15478/155028 objects misplaced (9.984%); 16 MiB/s, 253 objects/s recovering 2024-09-17T16:32:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:39 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:32:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2347817515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:32:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:39 smithi071 ceph-mon[30002]: osdmap e697: 12 total, 12 up, 12 in 2024-09-17T16:32:39.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:39.609 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:39.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:39.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:39 smithi017 ceph-mon[27571]: pgmap v1550: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 14635/155028 objects degraded (9.440%); 15478/155028 objects misplaced (9.984%); 16 MiB/s, 253 objects/s recovering 2024-09-17T16:32:39.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:39 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:32:39.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2347817515' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:32:39.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:39 smithi017 ceph-mon[27571]: osdmap e697: 12 total, 12 up, 12 in 2024-09-17T16:32:39.712 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:39.751 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:39.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:39.853 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:39.877 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:39.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:39.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:39.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:40.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:40.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:40.149 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:40.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:40.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:40.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:40.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:40.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:40.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:40.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:40.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:40.517 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:40.558 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:40.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:40.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:40 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 14635/155028 objects degraded (9.440%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:40 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 14635/155028 objects degraded (9.440%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:40.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:40.683 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:40.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:40 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 14635/155028 objects degraded (9.440%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:40.784 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:40.808 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:40.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:40.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:40.910 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:40.952 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:40.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:41.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:41.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:41.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:41.179 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:41.203 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:41.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:41.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:41.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:41.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:41.431 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:41.455 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:41.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:41.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:41.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:41 smithi071 ceph-mon[30002]: pgmap v1552: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 14635/154506 objects degraded (9.472%); 15478/154506 objects misplaced (10.018%); 16 MiB/s, 253 objects/s recovering 2024-09-17T16:32:41.579 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:41.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:41 smithi173 ceph-mon[30530]: pgmap v1552: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 14635/154506 objects degraded (9.472%); 15478/154506 objects misplaced (10.018%); 16 MiB/s, 253 objects/s recovering 2024-09-17T16:32:41.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:41.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:41.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:41.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:41 smithi017 ceph-mon[27571]: pgmap v1552: 27 pgs: 1 active+clean+scrubbing, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 17 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 14635/154506 objects degraded (9.472%); 15478/154506 objects misplaced (10.018%); 16 MiB/s, 253 objects/s recovering 2024-09-17T16:32:41.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:41.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:41.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:41.940 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:41.973 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:41.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:42.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:42.097 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:42.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:42.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:42.220 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:42.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:42.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:42.370 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:42.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:42.472 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:42.495 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:42.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:42.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:42.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:42.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:42.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:42.763 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:42.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:42.865 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:42.888 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:42.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:42.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:42.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:43.023 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:43.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:43.125 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:43.148 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:43.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:43.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:43.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:43.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:43.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:43.398 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:43.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:43.500 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:43.540 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:43.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:43.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:43 smithi071 ceph-mon[30002]: pgmap v1553: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 14159/150957 objects degraded (9.379%); 14876/150957 objects misplaced (9.854%); 16 MiB/s, 249 objects/s recovering 2024-09-17T16:32:43.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:43 smithi173 ceph-mon[30530]: pgmap v1553: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 14159/150957 objects degraded (9.379%); 14876/150957 objects misplaced (9.854%); 16 MiB/s, 249 objects/s recovering 2024-09-17T16:32:43.670 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:43.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:43 smithi017 ceph-mon[27571]: pgmap v1553: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 14159/150957 objects degraded (9.379%); 14876/150957 objects misplaced (9.854%); 16 MiB/s, 249 objects/s recovering 2024-09-17T16:32:43.772 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:43.795 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:43.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:43.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:43.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:43.920 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:43.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:44.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:44.057 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:44.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:44.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:44.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:44.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:44.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:44.307 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:44.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:44.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:44.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:44.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:44.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:44.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:44.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:44.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:44.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:44.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:44.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:44.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:44.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:44.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:44.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:44.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:45.042 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:45.065 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:45.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:45.168 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:45.191 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:45.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:45.293 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:45.315 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:45.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:45.417 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:45.440 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:45.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:45.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:45.563 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:45 smithi071 ceph-mon[30002]: pgmap v1554: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s; 13971/146313 objects degraded (9.549%); 12821/146313 objects misplaced (8.763%); 16 MiB/s, 260 objects/s recovering 2024-09-17T16:32:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:45 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 13971/146313 objects degraded (9.549%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:45.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:45 smithi173 ceph-mon[30530]: pgmap v1554: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s; 13971/146313 objects degraded (9.549%); 12821/146313 objects misplaced (8.763%); 16 MiB/s, 260 objects/s recovering 2024-09-17T16:32:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:45 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 13971/146313 objects degraded (9.549%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:45.665 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:45.687 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:45.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:45 smithi017 ceph-mon[27571]: pgmap v1554: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 554 op/s; 13971/146313 objects degraded (9.549%); 12821/146313 objects misplaced (8.763%); 16 MiB/s, 260 objects/s recovering 2024-09-17T16:32:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:45 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 13971/146313 objects degraded (9.549%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:45.789 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:45.837 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:45.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:45.939 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:45.982 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:45.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:46.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:46.116 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:46.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:46.218 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:46.240 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:46.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:46.342 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:46.379 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:46.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:46.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:46.481 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:46.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:32:46.523 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:46.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:46.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:32:46.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:46.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:32:46.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:46.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:46.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:46.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:46.770 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:46.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:46.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:46.893 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:46.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:46.995 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:47.018 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:47.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:47.120 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:47.166 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:47.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:47.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:47.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:47.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:47.397 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:47.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:47.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:47.539 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:47 smithi071 ceph-mon[30002]: pgmap v1555: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 13971/144915 objects degraded (9.641%); 12821/144915 objects misplaced (8.847%); 16 MiB/s, 260 objects/s recovering 2024-09-17T16:32:47.575 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:47.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:47 smithi173 ceph-mon[30530]: pgmap v1555: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 13971/144915 objects degraded (9.641%); 12821/144915 objects misplaced (8.847%); 16 MiB/s, 260 objects/s recovering 2024-09-17T16:32:47.677 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:47.708 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:47.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:47.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:47.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:47 smithi017 ceph-mon[27571]: pgmap v1555: 27 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 13971/144915 objects degraded (9.641%); 12821/144915 objects misplaced (8.847%); 16 MiB/s, 260 objects/s recovering 2024-09-17T16:32:47.809 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:47.833 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:47.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:47.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:47.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:47.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:47.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:48.058 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:48.080 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:48.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:48.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:48.206 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:48.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:48.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:48.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:48.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:48.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:48.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:48.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:48.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:48.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:48 smithi071 ceph-mon[30002]: osdmap e698: 12 total, 12 up, 12 in 2024-09-17T16:32:48.582 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:48.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:48.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:48 smithi017 ceph-mon[27571]: osdmap e698: 12 total, 12 up, 12 in 2024-09-17T16:32:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:48 smithi173 ceph-mon[30530]: osdmap e698: 12 total, 12 up, 12 in 2024-09-17T16:32:48.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:48.707 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:48.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:48.809 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:48.831 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:48.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:48.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:48.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:48.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:49.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:49.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:49.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:49.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:49.209 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:49.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:49.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:49.334 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:49.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:49.436 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:49.477 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:49.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:49 smithi071 ceph-mon[30002]: pgmap v1557: 27 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 9991/137304 objects degraded (7.277%); 13042/137304 objects misplaced (9.499%); 16 MiB/s, 280 objects/s recovering 2024-09-17T16:32:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:49 smithi071 ceph-mon[30002]: osdmap e699: 12 total, 12 up, 12 in 2024-09-17T16:32:49.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:49.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:49.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:49 smithi173 ceph-mon[30530]: pgmap v1557: 27 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 9991/137304 objects degraded (7.277%); 13042/137304 objects misplaced (9.499%); 16 MiB/s, 280 objects/s recovering 2024-09-17T16:32:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:49 smithi173 ceph-mon[30530]: osdmap e699: 12 total, 12 up, 12 in 2024-09-17T16:32:49.708 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:49.729 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:49.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:49 smithi017 ceph-mon[27571]: pgmap v1557: 27 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 9991/137304 objects degraded (7.277%); 13042/137304 objects misplaced (9.499%); 16 MiB/s, 280 objects/s recovering 2024-09-17T16:32:49.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:49 smithi017 ceph-mon[27571]: osdmap e699: 12 total, 12 up, 12 in 2024-09-17T16:32:49.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:49.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:49.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:49.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:49.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:49.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:49.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:50.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:50.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:50.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:50.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:50.247 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:50.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:50.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:50.373 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:50.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:50.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:50.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:50.514 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:50.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:50 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:32:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:50 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:32:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:50 smithi071 ceph-mon[30002]: 7.18 scrub starts 2024-09-17T16:32:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:50 smithi071 ceph-mon[30002]: osdmap e700: 12 total, 12 up, 12 in 2024-09-17T16:32:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:50 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 9991/137304 objects degraded (7.277%), 3 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:50.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:50.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:50.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:50 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:32:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:50 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:32:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:50 smithi173 ceph-mon[30530]: 7.18 scrub starts 2024-09-17T16:32:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:50 smithi173 ceph-mon[30530]: osdmap e700: 12 total, 12 up, 12 in 2024-09-17T16:32:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:50 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 9991/137304 objects degraded (7.277%), 3 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:50 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:32:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:50 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:32:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:50 smithi017 ceph-mon[27571]: 7.18 scrub starts 2024-09-17T16:32:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:50 smithi017 ceph-mon[27571]: osdmap e700: 12 total, 12 up, 12 in 2024-09-17T16:32:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:50 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 9991/137304 objects degraded (7.277%), 3 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:50.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:50.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:50.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:50.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:50.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:50.894 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:50.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:50.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:51.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:51.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:51.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:51.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:51.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:51.255 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:51.299 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:51.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:51.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:51.423 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:51.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:51.526 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:51 smithi071 ceph-mon[30002]: pgmap v1560: 27 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 532 op/s; 9991/136728 objects degraded (7.307%); 13042/136728 objects misplaced (9.539%); 12 MiB/s, 236 objects/s recovering 2024-09-17T16:32:51.566 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:51.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:51.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:51 smithi173 ceph-mon[30530]: pgmap v1560: 27 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 532 op/s; 9991/136728 objects degraded (7.307%); 13042/136728 objects misplaced (9.539%); 12 MiB/s, 236 objects/s recovering 2024-09-17T16:32:51.668 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:51.706 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:51.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:51.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:51.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:51 smithi017 ceph-mon[27571]: pgmap v1560: 27 pgs: 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 18 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 532 op/s; 9991/136728 objects degraded (7.307%); 13042/136728 objects misplaced (9.539%); 12 MiB/s, 236 objects/s recovering 2024-09-17T16:32:51.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:51.830 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:51.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:51.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:51.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:51.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:52.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:52.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:52.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:52.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:52.188 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:52.210 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:52.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:52.312 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:52.335 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:52.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:52.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:52.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:52.464 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:52.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:52.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:52.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:52.696 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:52.720 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:52.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:52.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:52.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:52.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:52.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:52.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:52.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:52.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:53.080 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:53.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:53.121 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:53.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:53.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:53.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:53.248 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:53.348 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:53.370 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:53.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:53.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:53.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:53.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:53 smithi071 ceph-mon[30002]: pgmap v1561: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 9443/133113 objects degraded (7.094%); 12300/133113 objects misplaced (9.240%); 18 MiB/s, 322 objects/s recovering 2024-09-17T16:32:53.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:53.619 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:53.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:53 smithi173 ceph-mon[30530]: pgmap v1561: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 9443/133113 objects degraded (7.094%); 12300/133113 objects misplaced (9.240%); 18 MiB/s, 322 objects/s recovering 2024-09-17T16:32:53.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:53 smithi017 ceph-mon[27571]: pgmap v1561: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 9443/133113 objects degraded (7.094%); 12300/133113 objects misplaced (9.240%); 18 MiB/s, 322 objects/s recovering 2024-09-17T16:32:53.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:53.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:53.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:32:53.850 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:53.891 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:53.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:53.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:54.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:54.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:54.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:54.167 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:54.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:54.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:54.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:54.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:54.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:54.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:54.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:54.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:54.547 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:32:54.547 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:32:54.549 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:54.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:54.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:54.677 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:54.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:54.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:54.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:54.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:54.907 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:54.929 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:54.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:55.032 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:55.054 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:55.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:55.155 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:55.195 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:55.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:55.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:55.297 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:55.321 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:55.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:55.424 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:55.458 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:55.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:55.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:55.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:55.583 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:55.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:55 smithi173 ceph-mon[30530]: from='client.27283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:32:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:55 smithi173 ceph-mon[30530]: pgmap v1562: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 9443/129099 objects degraded (7.315%); 10218/129099 objects misplaced (7.915%); 13 MiB/s, 208 objects/s recovering 2024-09-17T16:32:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:55 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 9443/129099 objects degraded (7.315%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:55.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:55.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:55.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:55 smithi017 ceph-mon[27571]: from='client.27283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:32:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:55 smithi017 ceph-mon[27571]: pgmap v1562: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 9443/129099 objects degraded (7.315%); 10218/129099 objects misplaced (7.915%); 13 MiB/s, 208 objects/s recovering 2024-09-17T16:32:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:55 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 9443/129099 objects degraded (7.315%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:55.813 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:55 smithi071 ceph-mon[30002]: from='client.27283 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:32:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:55 smithi071 ceph-mon[30002]: pgmap v1562: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 9443/129099 objects degraded (7.315%); 10218/129099 objects misplaced (7.915%); 13 MiB/s, 208 objects/s recovering 2024-09-17T16:32:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:55 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 9443/129099 objects degraded (7.315%), 3 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-17T16:32:55.851 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:55.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:55.953 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:55.991 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:55.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:55.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:56.093 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:32:56.115 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:56.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:32:56.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:32:56.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:56.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:32:56.343 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:56.366 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:56.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:56.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:56.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:56.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:56.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:56.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:56.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:56.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:56.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:56.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:56.853 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:56.875 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:56.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:56.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:32:57.000 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:57.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:32:57.103 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:57.127 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:57.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:57.229 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:57.255 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:57.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:57.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:57.270 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:32:57.358 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:57.381 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:57.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:57.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:57.483 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:57.517 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:57.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:57.619 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:57.642 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:57.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:57.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:57.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:57 smithi173 ceph-mon[30530]: pgmap v1563: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 404 op/s; 9443/127584 objects degraded (7.401%); 10218/127584 objects misplaced (8.009%); 9.8 MiB/s, 157 objects/s recovering 2024-09-17T16:32:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:57 smithi173 ceph-mon[30530]: osdmap e701: 12 total, 12 up, 12 in 2024-09-17T16:32:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:57 smithi017 ceph-mon[27571]: pgmap v1563: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 404 op/s; 9443/127584 objects degraded (7.401%); 10218/127584 objects misplaced (8.009%); 9.8 MiB/s, 157 objects/s recovering 2024-09-17T16:32:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:57 smithi017 ceph-mon[27571]: osdmap e701: 12 total, 12 up, 12 in 2024-09-17T16:32:57.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:57.768 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:57.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:57.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:57 smithi071 ceph-mon[30002]: pgmap v1563: 27 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 404 op/s; 9443/127584 objects degraded (7.401%); 10218/127584 objects misplaced (8.009%); 9.8 MiB/s, 157 objects/s recovering 2024-09-17T16:32:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:57 smithi071 ceph-mon[30002]: osdmap e701: 12 total, 12 up, 12 in 2024-09-17T16:32:57.870 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:32:57.892 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:57.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:32:57.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:58.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:58.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:58.118 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:58.140 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:58.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:58.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:58.277 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:58.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:58.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:32:58.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:58.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:32:58.506 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:32:58.538 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:58.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:32:58.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:32:58.680 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:58.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:32:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:58 smithi017 ceph-mon[27571]: osdmap e702: 12 total, 12 up, 12 in 2024-09-17T16:32:58.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:58.812 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:58.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:58.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:58 smithi071 ceph-mon[30002]: osdmap e702: 12 total, 12 up, 12 in 2024-09-17T16:32:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:58 smithi173 ceph-mon[30530]: osdmap e702: 12 total, 12 up, 12 in 2024-09-17T16:32:58.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:58.936 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:58.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:58.937 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:59.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:59.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:59.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:59.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:32:59.183 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:32:59.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:59.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:32:59.285 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:32:59.315 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:59.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:32:59.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:59.442 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:59.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:59.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:59.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:59.566 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:59.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:32:59.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:59.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:32:59.794 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:32:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:59 smithi071 ceph-mon[30002]: pgmap v1566: 27 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 5984/120156 objects degraded (4.980%); 10641/120156 objects misplaced (8.856%); 16 MiB/s, 301 objects/s recovering 2024-09-17T16:32:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:59 smithi071 ceph-mon[30002]: 7.16 scrub starts 2024-09-17T16:32:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:32:59 smithi071 ceph-mon[30002]: osdmap e703: 12 total, 12 up, 12 in 2024-09-17T16:32:59.837 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:32:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:59.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:32:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:59 smithi173 ceph-mon[30530]: pgmap v1566: 27 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 5984/120156 objects degraded (4.980%); 10641/120156 objects misplaced (8.856%); 16 MiB/s, 301 objects/s recovering 2024-09-17T16:32:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:59 smithi173 ceph-mon[30530]: 7.16 scrub starts 2024-09-17T16:32:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:32:59 smithi173 ceph-mon[30530]: osdmap e703: 12 total, 12 up, 12 in 2024-09-17T16:32:59.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:32:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:32:59.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:32:59.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:32:59.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:59 smithi017 ceph-mon[27571]: pgmap v1566: 27 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 19 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 5984/120156 objects degraded (4.980%); 10641/120156 objects misplaced (8.856%); 16 MiB/s, 301 objects/s recovering 2024-09-17T16:32:59.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:59 smithi017 ceph-mon[27571]: 7.16 scrub starts 2024-09-17T16:32:59.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:32:59 smithi017 ceph-mon[27571]: osdmap e703: 12 total, 12 up, 12 in 2024-09-17T16:33:00.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:00.104 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:00.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:00.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:00.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:00.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:00.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:00.362 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:00.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:00.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:00.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:00.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:00.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:00.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:00.612 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:00.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:00.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:00.714 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:00.736 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:00.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:00.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:00 smithi071 ceph-mon[30002]: 7.12 scrub starts 2024-09-17T16:33:00.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:00 smithi071 ceph-mon[30002]: osdmap e704: 12 total, 12 up, 12 in 2024-09-17T16:33:00.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:00 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 5984/120156 objects degraded (4.980%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:00.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:00 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:00.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:00 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:00.838 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:00.861 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:00.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:00.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:00.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:00 smithi173 ceph-mon[30530]: 7.12 scrub starts 2024-09-17T16:33:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:00 smithi173 ceph-mon[30530]: osdmap e704: 12 total, 12 up, 12 in 2024-09-17T16:33:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:00 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 5984/120156 objects degraded (4.980%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:00 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:00.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:00 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:00.964 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:00.985 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:00.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:00.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:00 smithi017 ceph-mon[27571]: 7.12 scrub starts 2024-09-17T16:33:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:00 smithi017 ceph-mon[27571]: osdmap e704: 12 total, 12 up, 12 in 2024-09-17T16:33:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:00 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 5984/120156 objects degraded (4.980%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:00 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:00 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:01.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:01.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:01.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:01.113 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:01.215 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:01.236 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:01.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:01.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:01.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:01.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:01.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:01.518 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:01.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:01.619 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:01.668 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:01.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:01.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:01 smithi017 ceph-mon[27571]: pgmap v1569: 27 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 20 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 5984/119334 objects degraded (5.014%); 10641/119334 objects misplaced (8.917%); 13 MiB/s, 338 objects/s recovering 2024-09-17T16:33:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:01 smithi017 ceph-mon[27571]: osdmap e705: 12 total, 12 up, 12 in 2024-09-17T16:33:01.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:01.792 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:01.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:01.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:01 smithi071 ceph-mon[30002]: pgmap v1569: 27 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 20 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 5984/119334 objects degraded (5.014%); 10641/119334 objects misplaced (8.917%); 13 MiB/s, 338 objects/s recovering 2024-09-17T16:33:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:01 smithi071 ceph-mon[30002]: osdmap e705: 12 total, 12 up, 12 in 2024-09-17T16:33:01.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:01 smithi173 ceph-mon[30530]: pgmap v1569: 27 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 20 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 5984/119334 objects degraded (5.014%); 10641/119334 objects misplaced (8.917%); 13 MiB/s, 338 objects/s recovering 2024-09-17T16:33:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:01 smithi173 ceph-mon[30530]: osdmap e705: 12 total, 12 up, 12 in 2024-09-17T16:33:01.916 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:01.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:02.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:02.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:02.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:02.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:02.169 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:02.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:02.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:02.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:02.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:02.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:02.445 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:02.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:02.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:02.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:02.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:02.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:02.693 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:02.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:02.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:02 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:02 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:02 smithi071 ceph-mon[30002]: osdmap e706: 12 total, 12 up, 12 in 2024-09-17T16:33:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:33:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:33:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:33:02.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:02.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:02 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:02 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:02 smithi173 ceph-mon[30530]: osdmap e706: 12 total, 12 up, 12 in 2024-09-17T16:33:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:33:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:33:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:33:02.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:02.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:02.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:02 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:02 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:02 smithi017 ceph-mon[27571]: osdmap e706: 12 total, 12 up, 12 in 2024-09-17T16:33:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:33:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:33:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:33:03.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:03.083 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:03.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:03.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:03.208 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:03.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:03.310 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:03.332 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:03.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:03.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:03.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:03.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:03.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:03.561 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:03.594 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:03.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:03.696 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:03.733 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:03.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:03 smithi071 ceph-mon[30002]: pgmap v1572: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 342 op/s; 2852/116049 objects degraded (2.458%); 11965/116049 objects misplaced (10.310%); 6.2 MiB/s, 169 objects/s recovering 2024-09-17T16:33:03.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:03.856 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:03.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:03.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:03 smithi173 ceph-mon[30530]: pgmap v1572: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 342 op/s; 2852/116049 objects degraded (2.458%); 11965/116049 objects misplaced (10.310%); 6.2 MiB/s, 169 objects/s recovering 2024-09-17T16:33:03.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:03.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:03.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:03.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:03.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:03 smithi017 ceph-mon[27571]: pgmap v1572: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 342 op/s; 2852/116049 objects degraded (2.458%); 11965/116049 objects misplaced (10.310%); 6.2 MiB/s, 169 objects/s recovering 2024-09-17T16:33:04.087 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:04.127 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:04.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:04.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:04.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:04.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:04.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:04.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:04.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:04.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:04.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:04.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:04.611 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:04.633 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:04.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:04.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:04.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:04.772 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:04.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:04.873 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:04.907 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:04.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:05.009 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:05.031 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:05.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:05.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:05.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:05.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:05.263 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:05.287 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:05.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:05.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:05.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:05.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:05.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:05.513 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:05.536 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:05.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:05.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:05.677 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:05.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:05.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:05.802 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:05.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:05 smithi071 ceph-mon[30002]: pgmap v1573: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 438 op/s; 2852/112833 objects degraded (2.528%); 10760/112833 objects misplaced (9.536%); 8.9 MiB/s, 194 objects/s recovering 2024-09-17T16:33:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:05 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2852/112833 objects degraded (2.528%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:05.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:05 smithi173 ceph-mon[30530]: pgmap v1573: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 438 op/s; 2852/112833 objects degraded (2.528%); 10760/112833 objects misplaced (9.536%); 8.9 MiB/s, 194 objects/s recovering 2024-09-17T16:33:05.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:05 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2852/112833 objects degraded (2.528%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:05.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:05.925 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:05.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:05 smithi017 ceph-mon[27571]: pgmap v1573: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 438 op/s; 2852/112833 objects degraded (2.528%); 10760/112833 objects misplaced (9.536%); 8.9 MiB/s, 194 objects/s recovering 2024-09-17T16:33:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:05 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2852/112833 objects degraded (2.528%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:06.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:06.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:06.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:06.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:06.178 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:06.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:06.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:06.305 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:06.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:06.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:06.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:06.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:06.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:06.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:06.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:06.560 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:06.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:06.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:06.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:06.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:06.818 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:06.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:06.920 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:06.942 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:06.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:07.044 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:07.085 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:07.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:07.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:07.211 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:07.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:07.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:07.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:07.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:07.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:07.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:07.465 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:07.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:07.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:07.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:07.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:07.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:07.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:07.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:07 smithi071 ceph-mon[30002]: pgmap v1574: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 475 op/s; 2852/110799 objects degraded (2.574%); 10760/110799 objects misplaced (9.711%); 7.6 MiB/s, 164 objects/s recovering 2024-09-17T16:33:07.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:07.837 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:07.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:07.857 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:07 smithi017 ceph-mon[27571]: pgmap v1574: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 475 op/s; 2852/110799 objects degraded (2.574%); 10760/110799 objects misplaced (9.711%); 7.6 MiB/s, 164 objects/s recovering 2024-09-17T16:33:07.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:07 smithi173 ceph-mon[30530]: pgmap v1574: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 475 op/s; 2852/110799 objects degraded (2.574%); 10760/110799 objects misplaced (9.711%); 7.6 MiB/s, 164 objects/s recovering 2024-09-17T16:33:07.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:07.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:07.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:08.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:08.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:08.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:08.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:08.222 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:08.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:08.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:08.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:08.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:08.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:08.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:08.471 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:08.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:08.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:08.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:08.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:08.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:08.601 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:08.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:08.722 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:08.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:08 smithi071 ceph-mon[30002]: osdmap e707: 12 total, 12 up, 12 in 2024-09-17T16:33:08.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:08.858 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:08.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:08.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:08 smithi173 ceph-mon[30530]: osdmap e707: 12 total, 12 up, 12 in 2024-09-17T16:33:08.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:08.982 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:08.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:08.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:08 smithi017 ceph-mon[27571]: osdmap e707: 12 total, 12 up, 12 in 2024-09-17T16:33:09.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:09.105 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:09.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:09.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:09.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:09.229 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:09.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:09.331 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:09.354 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:09.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:09.371 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1561,"stamp":"2024-09-17T16:32:52.040828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2908160568,"num_objects":44371,"num_object_clones":0,"num_object_copies":133113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9443,"num_objects_misplaced":12300,"num_objects_unfound":0,"num_objects_dirty":44371,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":111031,"num_write_kb":4023431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19743,"num_bytes_recovered":1279272000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210295,"ondisk_log_size":210295,"up":81,"acting":73,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":17100088,"kb_used_data":10874888,"kb_used_omap":58,"kb_used_meta":6224837,"kb_avail":1107841736,"statfs":{"total":1151940427776,"available":1134429937664,"internally_reserved":0,"allocated":11135885312,"data_stored":11132672957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":6374233251},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-257818624,"num_objects":-3934,"num_object_clones":0,"num_object_copies":-11802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4528,"num_objects_misplaced":-521,"num_objects_unfound":0,"num_objects_dirty":-3934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3934,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1933,"num_bytes_recovered":111673344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001777"},"pg_stats":[{"pgid":"7.17","version":"698'7623","reported_seq":16329,"reported_epoch":698,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:32:47.513536+0000","last_change":"2024-09-17T16:31:22.779049+0000","last_active":"2024-09-17T16:32:47.513536+0000","last_peered":"2024-09-17T16:32:47.513536+0000","last_clean":"2024-09-17T16:31:20.068705+0000","last_became_active":"2024-09-17T16:31:22.668348+0000","last_became_peered":"2024-09-17T16:31:22.668348+0000","last_unstale":"2024-09-17T16:32:47.513536+0000","last_undegraded":"2024-09-17T16:32:47.513536+0000","last_fullsized":"2024-09-17T16:31:22.136755+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":674,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"666'5094","last_scrub_stamp":"2024-09-17T16:30:56.450199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:30:56.450199+0000","objects_scrubbed":5055,"log_size":7623,"log_dups_size":0,"ondisk_log_size":7623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91193344,"num_objects":1392,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1736,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4499,"num_write_kb":188480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2121,"num_bytes_recovered":138936320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,10],"acting":[1,2],"avail_no_missing":["1","3","8"],"object_location_counts":[{"shards":"1,2","objects":813},{"shards":"1,3,7,8","objects":1},{"shards":"1,3,8","objects":578}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"700'7857","reported_seq":16426,"reported_epoch":700,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:32:51.952325+0000","last_change":"2024-09-17T16:32:51.956472+0000","last_active":"2024-09-17T16:32:51.952325+0000","last_peered":"2024-09-17T16:32:51.952325+0000","last_clean":"2024-09-17T16:31:20.005846+0000","last_became_active":"2024-09-17T16:31:22.650227+0000","last_became_peered":"2024-09-17T16:31:22.650227+0000","last_unstale":"2024-09-17T16:32:51.952325+0000","last_undegraded":"2024-09-17T16:32:51.952182+0000","last_fullsized":"2024-09-17T16:31:22.133445+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":674,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"669'5652","last_scrub_stamp":"2024-09-17T16:31:04.005746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:04.005746+0000","objects_scrubbed":5605,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1750,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4665,"num_write_kb":192160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":181665792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[0,2],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,2","objects":1340}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"700'7761","reported_seq":13507,"reported_epoch":700,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-17T16:32:51.958401+0000","last_change":"2024-09-17T16:31:22.815091+0000","last_active":"2024-09-17T16:32:51.958401+0000","last_peered":"2024-09-17T16:32:51.958401+0000","last_clean":"2024-09-17T16:31:20.078411+0000","last_became_active":"2024-09-17T16:31:22.731609+0000","last_became_peered":"2024-09-17T16:31:22.731609+0000","last_unstale":"2024-09-17T16:32:51.958401+0000","last_undegraded":"2024-09-17T16:31:22.721895+0000","last_fullsized":"2024-09-17T16:31:22.132865+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":674,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":7761,"log_dups_size":0,"ondisk_log_size":7761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90669056,"num_objects":1384,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3191,"num_objects_misplaced":1596,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":191232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1395,"num_bytes_recovered":91291648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[0,1],"avail_no_missing":["0","9","11"],"object_location_counts":[{"shards":"0,1","objects":-4},{"shards":"0,8,9,11","objects":1},{"shards":"0,9,11","objects":1387}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"700'7918","reported_seq":15380,"reported_epoch":700,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-17T16:32:51.950204+0000","last_change":"2024-09-17T16:31:22.926055+0000","last_active":"2024-09-17T16:32:51.950204+0000","last_peered":"2024-09-17T16:32:51.950204+0000","last_clean":"2024-09-17T16:31:20.087841+0000","last_became_active":"2024-09-17T16:31:22.915039+0000","last_became_peered":"2024-09-17T16:31:22.915039+0000","last_unstale":"2024-09-17T16:32:51.950204+0000","last_undegraded":"2024-09-17T16:31:22.744239+0000","last_fullsized":"2024-09-17T16:31:22.143013+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":674,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"665'5081","last_scrub_stamp":"2024-09-17T16:30:53.275174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:30:53.275174+0000","objects_scrubbed":5055,"log_size":7918,"log_dups_size":0,"ondisk_log_size":7918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87883776,"num_objects":1341,"num_object_clones":0,"num_object_copies":4023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3060,"num_objects_misplaced":1530,"num_objects_unfound":0,"num_objects_dirty":1341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":193600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1805,"num_bytes_recovered":118292480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[0,5],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":1317},{"shards":"0,5","objects":24}],"blocked_by":[],"up_primary":10,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"697'7735","reported_seq":14782,"reported_epoch":697,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:32:47.247101+0000","last_change":"2024-09-17T16:32:46.957512+0000","last_active":"2024-09-17T16:32:47.247101+0000","last_peered":"2024-09-17T16:32:47.247101+0000","last_clean":"2024-09-17T16:31:20.018091+0000","last_became_active":"2024-09-17T16:31:22.701252+0000","last_became_peered":"2024-09-17T16:31:22.701252+0000","last_unstale":"2024-09-17T16:32:47.247101+0000","last_undegraded":"2024-09-17T16:32:47.247101+0000","last_fullsized":"2024-09-17T16:31:22.136467+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":674,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":7735,"log_dups_size":0,"ondisk_log_size":7735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91291648,"num_objects":1393,"num_object_clones":0,"num_object_copies":4179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4206,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4573,"num_write_kb":190912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1717,"num_bytes_recovered":112492544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,11],"acting":[2,0],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"0,2","objects":-9},{"shards":"2,3,4","objects":1402}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"697'7698","reported_seq":15614,"reported_epoch":697,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:32:47.242647+0000","last_change":"2024-09-17T16:31:22.650559+0000","last_active":"2024-09-17T16:32:47.242647+0000","last_peered":"2024-09-17T16:32:47.242647+0000","last_clean":"2024-09-17T16:31:19.943556+0000","last_became_active":"2024-09-17T16:31:22.607812+0000","last_became_peered":"2024-09-17T16:31:22.607812+0000","last_unstale":"2024-09-17T16:32:47.242647+0000","last_undegraded":"2024-09-17T16:32:47.242647+0000","last_fullsized":"2024-09-17T16:31:22.128023+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":674,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94076928,"num_objects":1436,"num_object_clones":0,"num_object_copies":4308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":388,"num_objects_unfound":0,"num_objects_dirty":1436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":190784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2586,"num_bytes_recovered":169476096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,8],"acting":[2,0],"avail_no_missing":["2","3","10"],"object_location_counts":[{"shards":"0,2","objects":1242},{"shards":"2,3,10","objects":194}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.11","version":"697'7678","reported_seq":18487,"reported_epoch":697,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-17T16:32:47.057170+0000","last_change":"2024-09-17T16:31:22.763919+0000","last_active":"2024-09-17T16:32:47.057170+0000","last_peered":"2024-09-17T16:32:47.057170+0000","last_clean":"2024-09-17T16:31:20.084668+0000","last_became_active":"2024-09-17T16:31:22.677157+0000","last_became_peered":"2024-09-17T16:31:22.677157+0000","last_unstale":"2024-09-17T16:32:47.057170+0000","last_undegraded":"2024-09-17T16:32:47.057170+0000","last_fullsized":"2024-09-17T16:31:22.126072+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":674,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":7678,"log_dups_size":0,"ondisk_log_size":7678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91521024,"num_objects":1397,"num_object_clones":0,"num_object_copies":4191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1248,"num_objects_unfound":0,"num_objects_dirty":1397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4534,"num_write_kb":189760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2661,"num_bytes_recovered":174325760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[3,6],"avail_no_missing":["3","4","10"],"object_location_counts":[{"shards":"3,4,10","objects":416},{"shards":"3,6","objects":981}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"697'7651","reported_seq":13795,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-17T16:32:47.011213+0000","last_change":"2024-09-17T16:32:18.716569+0000","last_active":"2024-09-17T16:32:47.011213+0000","last_peered":"2024-09-17T16:32:47.011213+0000","last_clean":"2024-09-17T16:32:47.011213+0000","last_became_active":"2024-09-17T16:32:17.191360+0000","last_became_peered":"2024-09-17T16:32:17.191360+0000","last_unstale":"2024-09-17T16:32:47.011213+0000","last_undegraded":"2024-09-17T16:32:47.011213+0000","last_fullsized":"2024-09-17T16:32:47.011213+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":674,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":5,"last_scrub":"695'7179","last_scrub_stamp":"2024-09-17T16:32:18.716347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:32:18.716347+0000","objects_scrubbed":1952,"log_size":7651,"log_dups_size":0,"ondisk_log_size":7651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:38.578480+0000","scrub_duration":744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":1472,"num_object_clones":0,"num_object_copies":4416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":480,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,9],"acting":[3,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"700'9320","reported_seq":9478,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:51.952980+0000","last_change":"2024-09-17T16:29:35.616372+0000","last_active":"2024-09-17T16:32:51.952980+0000","last_peered":"2024-09-17T16:32:51.952980+0000","last_clean":"2024-09-17T16:32:51.952980+0000","last_became_active":"2024-09-17T16:29:35.616046+0000","last_became_peered":"2024-09-17T16:29:35.616046+0000","last_unstale":"2024-09-17T16:32:51.952980+0000","last_undegraded":"2024-09-17T16:32:51.952980+0000","last_fullsized":"2024-09-17T16:32:51.952980+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178651136,"num_objects":2726,"num_object_clones":0,"num_object_copies":8178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9320,"num_write_kb":385472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"697'9274","reported_seq":9489,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-17T16:32:47.169680+0000","last_change":"2024-09-17T16:31:02.091075+0000","last_active":"2024-09-17T16:32:47.169680+0000","last_peered":"2024-09-17T16:32:47.169680+0000","last_clean":"2024-09-17T16:32:47.169680+0000","last_became_active":"2024-09-17T16:29:35.659021+0000","last_became_peered":"2024-09-17T16:29:35.659021+0000","last_unstale":"2024-09-17T16:32:47.169680+0000","last_undegraded":"2024-09-17T16:32:47.169680+0000","last_fullsized":"2024-09-17T16:32:47.169680+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"669'5588","last_scrub_stamp":"2024-09-17T16:31:02.090927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:02.090927+0000","objects_scrubbed":5548,"log_size":9274,"log_dups_size":0,"ondisk_log_size":9274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185991168,"num_objects":2838,"num_object_clones":0,"num_object_copies":8514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":387584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"698'8937","reported_seq":9092,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-17T16:32:47.495588+0000","last_change":"2024-09-17T16:29:35.618928+0000","last_active":"2024-09-17T16:32:47.495588+0000","last_peered":"2024-09-17T16:32:47.495588+0000","last_clean":"2024-09-17T16:32:47.495588+0000","last_became_active":"2024-09-17T16:29:35.618644+0000","last_became_peered":"2024-09-17T16:29:35.618644+0000","last_unstale":"2024-09-17T16:32:47.495588+0000","last_undegraded":"2024-09-17T16:32:47.495588+0000","last_fullsized":"2024-09-17T16:32:47.495588+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":8937,"log_dups_size":0,"ondisk_log_size":8937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183369752,"num_objects":2799,"num_object_clones":0,"num_object_copies":8397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8937,"num_write_kb":375489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"700'9348","reported_seq":9568,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:50.629457+0000","last_change":"2024-09-17T16:31:21.144959+0000","last_active":"2024-09-17T16:32:50.629457+0000","last_peered":"2024-09-17T16:32:50.629457+0000","last_clean":"2024-09-17T16:32:50.629457+0000","last_became_active":"2024-09-17T16:29:35.617430+0000","last_became_peered":"2024-09-17T16:29:35.617430+0000","last_unstale":"2024-09-17T16:32:50.629457+0000","last_undegraded":"2024-09-17T16:32:50.629457+0000","last_fullsized":"2024-09-17T16:32:50.629457+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"676'6513","last_scrub_stamp":"2024-09-17T16:31:21.144875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:21.144875+0000","objects_scrubbed":5565,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178913280,"num_objects":2730,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9348,"num_write_kb":386496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"698'9205","reported_seq":9422,"reported_epoch":698,"state":"active+clean","last_fresh":"2024-09-17T16:32:47.473841+0000","last_change":"2024-09-17T16:31:23.585084+0000","last_active":"2024-09-17T16:32:47.473841+0000","last_peered":"2024-09-17T16:32:47.473841+0000","last_clean":"2024-09-17T16:32:47.473841+0000","last_became_active":"2024-09-17T16:29:35.685764+0000","last_became_peered":"2024-09-17T16:29:35.685764+0000","last_unstale":"2024-09-17T16:32:47.473841+0000","last_undegraded":"2024-09-17T16:32:47.473841+0000","last_fullsized":"2024-09-17T16:32:47.473841+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'6514","last_scrub_stamp":"2024-09-17T16:31:23.584787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:23.584787+0000","objects_scrubbed":5590,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189071360,"num_objects":2885,"num_object_clones":0,"num_object_copies":8655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9205,"num_write_kb":386880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"697'9069","reported_seq":9221,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-17T16:32:47.085770+0000","last_change":"2024-09-17T16:29:35.686166+0000","last_active":"2024-09-17T16:32:47.085770+0000","last_peered":"2024-09-17T16:32:47.085770+0000","last_clean":"2024-09-17T16:32:47.085770+0000","last_became_active":"2024-09-17T16:29:35.685807+0000","last_became_peered":"2024-09-17T16:29:35.685807+0000","last_unstale":"2024-09-17T16:32:47.085770+0000","last_undegraded":"2024-09-17T16:32:47.085770+0000","last_fullsized":"2024-09-17T16:32:47.085770+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":9069,"log_dups_size":0,"ondisk_log_size":9069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188678144,"num_objects":2879,"num_object_clones":0,"num_object_copies":8637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9069,"num_write_kb":382336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,11],"acting":[2,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"700'7786","reported_seq":7986,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:51.885740+0000","last_change":"2024-09-17T16:31:24.245183+0000","last_active":"2024-09-17T16:32:51.885740+0000","last_peered":"2024-09-17T16:32:51.885740+0000","last_clean":"2024-09-17T16:32:51.885740+0000","last_became_active":"2024-09-17T16:31:19.084475+0000","last_became_peered":"2024-09-17T16:31:19.084475+0000","last_unstale":"2024-09-17T16:32:51.885740+0000","last_undegraded":"2024-09-17T16:32:51.885740+0000","last_fullsized":"2024-09-17T16:32:51.885740+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'6418","last_scrub_stamp":"2024-09-17T16:31:24.244972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:24.244972+0000","objects_scrubbed":2708,"log_size":7786,"log_dups_size":0,"ondisk_log_size":7786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87031808,"num_objects":1328,"num_object_clones":0,"num_object_copies":3984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1380,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"697'7649","reported_seq":9273,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-17T16:32:46.770614+0000","last_change":"2024-09-17T16:31:20.780523+0000","last_active":"2024-09-17T16:32:46.770614+0000","last_peered":"2024-09-17T16:32:46.770614+0000","last_clean":"2024-09-17T16:32:46.770614+0000","last_became_active":"2024-09-17T16:31:19.090784+0000","last_became_peered":"2024-09-17T16:31:19.090784+0000","last_unstale":"2024-09-17T16:32:46.770614+0000","last_undegraded":"2024-09-17T16:32:46.770614+0000","last_fullsized":"2024-09-17T16:32:46.770614+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"675'6297","last_scrub_stamp":"2024-09-17T16:31:20.780460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:20.780460+0000","objects_scrubbed":2803,"log_size":7649,"log_dups_size":0,"ondisk_log_size":7649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95092736,"num_objects":1451,"num_object_clones":0,"num_object_copies":4353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1352,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"700'7661","reported_seq":7926,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:49.462539+0000","last_change":"2024-09-17T16:31:21.187196+0000","last_active":"2024-09-17T16:32:49.462539+0000","last_peered":"2024-09-17T16:32:49.462539+0000","last_clean":"2024-09-17T16:32:49.462539+0000","last_became_active":"2024-09-17T16:31:19.108699+0000","last_became_peered":"2024-09-17T16:31:19.108699+0000","last_unstale":"2024-09-17T16:32:49.462539+0000","last_undegraded":"2024-09-17T16:32:49.462539+0000","last_fullsized":"2024-09-17T16:32:49.462539+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"676'6279","last_scrub_stamp":"2024-09-17T16:31:21.187110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:21.187110+0000","objects_scrubbed":2768,"log_size":7661,"log_dups_size":0,"ondisk_log_size":7661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:32:50.939942+0000","scrub_duration":963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90767360,"num_objects":1385,"num_object_clones":0,"num_object_copies":4155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1383,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.18","version":"700'7699","reported_seq":18017,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:51.925074+0000","last_change":"2024-09-17T16:32:49.200065+0000","last_active":"2024-09-17T16:32:51.925074+0000","last_peered":"2024-09-17T16:32:51.925074+0000","last_clean":"2024-09-17T16:32:51.925074+0000","last_became_active":"2024-09-17T16:32:48.292361+0000","last_became_peered":"2024-09-17T16:32:48.292361+0000","last_unstale":"2024-09-17T16:32:51.925074+0000","last_undegraded":"2024-09-17T16:32:51.925074+0000","last_fullsized":"2024-09-17T16:32:51.925074+0000","mapping_epoch":698,"log_start":"0'0","ondisk_log_start":"0'0","created":674,"last_epoch_clean":699,"parent":"0.0","parent_split_bits":5,"last_scrub":"667'5344","last_scrub_stamp":"2024-09-17T16:31:00.025251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:00.025251+0000","objects_scrubbed":0,"log_size":7699,"log_dups_size":0,"ondisk_log_size":7699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:33:19.168212+0000","scrub_duration":2516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":1344,"num_object_clones":0,"num_object_copies":4032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4564,"num_write_kb":189056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3241,"num_bytes_recovered":197754880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,7],"acting":[0,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"700'7818","reported_seq":8021,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:51.357041+0000","last_change":"2024-09-17T16:31:21.895865+0000","last_active":"2024-09-17T16:32:51.357041+0000","last_peered":"2024-09-17T16:32:51.357041+0000","last_clean":"2024-09-17T16:32:51.357041+0000","last_became_active":"2024-09-17T16:31:19.087717+0000","last_became_peered":"2024-09-17T16:31:19.087717+0000","last_unstale":"2024-09-17T16:32:51.357041+0000","last_undegraded":"2024-09-17T16:32:51.357041+0000","last_fullsized":"2024-09-17T16:32:51.357041+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"676'6374","last_scrub_stamp":"2024-09-17T16:31:21.895775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:21.895775+0000","objects_scrubbed":2765,"log_size":7818,"log_dups_size":0,"ondisk_log_size":7818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86573056,"num_objects":1321,"num_object_clones":0,"num_object_copies":3963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.4","version":"700'7942","reported_seq":9584,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:51.891734+0000","last_change":"2024-09-17T16:31:24.691212+0000","last_active":"2024-09-17T16:32:51.891734+0000","last_peered":"2024-09-17T16:32:51.891734+0000","last_clean":"2024-09-17T16:32:51.891734+0000","last_became_active":"2024-09-17T16:31:19.083576+0000","last_became_peered":"2024-09-17T16:31:19.083576+0000","last_unstale":"2024-09-17T16:32:51.891734+0000","last_undegraded":"2024-09-17T16:32:51.891734+0000","last_fullsized":"2024-09-17T16:32:51.891734+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'6464","last_scrub_stamp":"2024-09-17T16:31:24.690938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:24.690938+0000","objects_scrubbed":2834,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87883776,"num_objects":1341,"num_object_clones":0,"num_object_copies":4023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1493,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"697'7734","reported_seq":9283,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-17T16:32:46.976189+0000","last_change":"2024-09-17T16:31:23.608549+0000","last_active":"2024-09-17T16:32:46.976189+0000","last_peered":"2024-09-17T16:32:46.976189+0000","last_clean":"2024-09-17T16:32:46.976189+0000","last_became_active":"2024-09-17T16:31:19.116986+0000","last_became_peered":"2024-09-17T16:31:19.116986+0000","last_unstale":"2024-09-17T16:32:46.976189+0000","last_undegraded":"2024-09-17T16:32:46.976189+0000","last_fullsized":"2024-09-17T16:32:46.976189+0000","mapping_epoch":640,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"678'6363","last_scrub_stamp":"2024-09-17T16:31:23.606236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:23.606236+0000","objects_scrubbed":2755,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1582,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90374144,"num_objects":1379,"num_object_clones":0,"num_object_copies":4137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"697'7658","reported_seq":7854,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-17T16:32:47.050263+0000","last_change":"2024-09-17T16:31:21.759157+0000","last_active":"2024-09-17T16:32:47.050263+0000","last_peered":"2024-09-17T16:32:47.050263+0000","last_clean":"2024-09-17T16:32:47.050263+0000","last_became_active":"2024-09-17T16:31:19.104084+0000","last_became_peered":"2024-09-17T16:31:19.104084+0000","last_unstale":"2024-09-17T16:32:47.050263+0000","last_undegraded":"2024-09-17T16:32:47.050263+0000","last_fullsized":"2024-09-17T16:32:47.050263+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"676'6371","last_scrub_stamp":"2024-09-17T16:31:21.759051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:21.759051+0000","objects_scrubbed":2748,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95748096,"num_objects":1461,"num_object_clones":0,"num_object_copies":4383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1287,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1","version":"700'7725","reported_seq":9260,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:49.906776+0000","last_change":"2024-09-17T16:32:25.247569+0000","last_active":"2024-09-17T16:32:49.906776+0000","last_peered":"2024-09-17T16:32:49.906776+0000","last_clean":"2024-09-17T16:32:49.906776+0000","last_became_active":"2024-09-17T16:31:19.087342+0000","last_became_peered":"2024-09-17T16:31:19.087342+0000","last_unstale":"2024-09-17T16:32:49.906776+0000","last_undegraded":"2024-09-17T16:32:49.906776+0000","last_fullsized":"2024-09-17T16:32:49.906776+0000","mapping_epoch":644,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"695'7294","last_scrub_stamp":"2024-09-17T16:32:25.247433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:32:25.247433+0000","objects_scrubbed":1794,"log_size":7725,"log_dups_size":0,"ondisk_log_size":7725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:44.570985+0000","scrub_duration":873,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88932352,"num_objects":1357,"num_object_clones":0,"num_object_copies":4071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1412,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"700'7745","reported_seq":7975,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:50.472473+0000","last_change":"2024-09-17T16:32:36.276373+0000","last_active":"2024-09-17T16:32:50.472473+0000","last_peered":"2024-09-17T16:32:50.472473+0000","last_clean":"2024-09-17T16:32:50.472473+0000","last_became_active":"2024-09-17T16:31:19.101106+0000","last_became_peered":"2024-09-17T16:31:19.101106+0000","last_unstale":"2024-09-17T16:32:50.472473+0000","last_undegraded":"2024-09-17T16:32:50.472473+0000","last_fullsized":"2024-09-17T16:32:50.472473+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"696'7490","last_scrub_stamp":"2024-09-17T16:32:36.276147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:32:36.276147+0000","objects_scrubbed":1591,"log_size":7745,"log_dups_size":0,"ondisk_log_size":7745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:48.287680+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87097344,"num_objects":1329,"num_object_clones":0,"num_object_copies":3987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1387,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"700'7840","reported_seq":13742,"reported_epoch":700,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-17T16:32:51.912819+0000","last_change":"2024-09-17T16:31:22.752156+0000","last_active":"2024-09-17T16:32:51.912819+0000","last_peered":"2024-09-17T16:32:51.912819+0000","last_clean":"2024-09-17T16:31:20.065331+0000","last_became_active":"2024-09-17T16:31:22.726526+0000","last_became_peered":"2024-09-17T16:31:22.726526+0000","last_unstale":"2024-09-17T16:32:51.912819+0000","last_undegraded":"2024-09-17T16:31:22.715125+0000","last_fullsized":"2024-09-17T16:31:22.133106+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":674,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:29:34.602207+0000","objects_scrubbed":0,"log_size":7840,"log_dups_size":0,"ondisk_log_size":7840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86114304,"num_objects":1314,"num_object_clones":0,"num_object_copies":3942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3192,"num_objects_misplaced":1596,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":191040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1434,"num_bytes_recovered":93978624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[0,2],"avail_no_missing":["0","8","10"],"object_location_counts":[{"shards":"0,2","objects":-40},{"shards":"0,8,10","objects":1354}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"700'7872","reported_seq":8140,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:51.640019+0000","last_change":"2024-09-17T16:31:21.452875+0000","last_active":"2024-09-17T16:32:51.640019+0000","last_peered":"2024-09-17T16:32:51.640019+0000","last_clean":"2024-09-17T16:32:51.640019+0000","last_became_active":"2024-09-17T16:31:19.083937+0000","last_became_peered":"2024-09-17T16:31:19.083937+0000","last_unstale":"2024-09-17T16:32:51.640019+0000","last_undegraded":"2024-09-17T16:32:51.640019+0000","last_fullsized":"2024-09-17T16:32:51.640019+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":629,"last_epoch_clean":675,"parent":"0.0","parent_split_bits":0,"last_scrub":"675'6431","last_scrub_stamp":"2024-09-17T16:31:21.452762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:29:34.602207+0000","last_clean_scrub_stamp":"2024-09-17T16:31:21.452762+0000","objects_scrubbed":2788,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88276992,"num_objects":1347,"num_object_clones":0,"num_object_copies":4041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1441,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":162,"reported_epoch":700,"state":"active+clean","last_fresh":"2024-09-17T16:32:49.304126+0000","last_change":"2024-09-17T16:32:28.373307+0000","last_active":"2024-09-17T16:32:49.304126+0000","last_peered":"2024-09-17T16:32:49.304126+0000","last_clean":"2024-09-17T16:32:49.304126+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:32:49.304126+0000","last_undegraded":"2024-09-17T16:32:49.304126+0000","last_fullsized":"2024-09-17T16:32:49.304126+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:32:28.373229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:32:28.373229+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:33:55.346441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":2907570200,"num_objects":44369,"num_object_clones":0,"num_object_copies":133107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9443,"num_objects_misplaced":12300,"num_objects_unfound":0,"num_objects_dirty":44369,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110908,"num_write_kb":4021281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19739,"num_bytes_recovered":1278214144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11076644864,"data_stored":11076632648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210203,"ondisk_log_size":210203,"up":78,"acting":70,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1086020,"kb_used_data":631408,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92659132,"statfs":{"total":95995035648,"available":94882951168,"internally_reserved":0,"allocated":646561792,"data_stored":646297105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":465500612},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547485,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1779020,"kb_used_data":1296636,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91966132,"statfs":{"total":95995035648,"available":94173319168,"internally_reserved":0,"allocated":1327755264,"data_stored":1327486364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":493943232},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1681936,"kb_used_data":1032956,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92063216,"statfs":{"total":95995035648,"available":94272733184,"internally_reserved":0,"allocated":1057746944,"data_stored":1057478044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005351,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1411564,"kb_used_data":956932,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92333588,"statfs":{"total":95995035648,"available":94549594112,"internally_reserved":0,"allocated":979898368,"data_stored":979625916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1124660,"kb_used_data":562532,"kb_used_omap":41,"kb_used_meta":562070,"kb_avail":92620492,"statfs":{"total":95995035648,"available":94843383808,"internally_reserved":0,"allocated":576032768,"data_stored":575772144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":575559987},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495921,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":719708,"kb_used_data":247044,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93025444,"statfs":{"total":95995035648,"available":95258054656,"internally_reserved":0,"allocated":252973056,"data_stored":252700604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":483981766},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":729744,"kb_used_data":197372,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93015408,"statfs":{"total":95995035648,"available":95247777792,"internally_reserved":0,"allocated":202108928,"data_stored":201840028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":545126854},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986492,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1347092,"kb_used_data":886916,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92398060,"statfs":{"total":95995035648,"available":94615613440,"internally_reserved":0,"allocated":908201984,"data_stored":907929532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150014,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2484216,"kb_used_data":1808872,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":91260936,"statfs":{"total":95995035648,"available":93451198464,"internally_reserved":0,"allocated":1852284928,"data_stored":1852020232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":691534279},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411652,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1655476,"kb_used_data":1223076,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92089676,"statfs":{"total":95995035648,"available":94299828224,"internally_reserved":0,"allocated":1252429824,"data_stored":1252169200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575178,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193208,"kb_used_data":819112,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92551944,"statfs":{"total":95995035648,"available":94773190656,"internally_reserved":0,"allocated":838770688,"data_stored":838505992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738703,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1887444,"kb_used_data":1212032,"kb_used_omap":1,"kb_used_meta":675326,"kb_avail":91857708,"statfs":{"total":95995035648,"available":94062292992,"internally_reserved":0,"allocated":1241120768,"data_stored":1240847796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":691534277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1229590528,"data_stored":1229586456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":829100032,"data_stored":829095960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1246953472,"data_stored":1246953472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1838092288,"data_stored":1838088216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":904593408,"data_stored":904593408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":201392128,"data_stored":201392128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":250740736,"data_stored":250740736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":578093056,"data_stored":578093056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":977600512,"data_stored":977600512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1055981568,"data_stored":1055981568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1321336832,"data_stored":1321336832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":643170304,"data_stored":643170304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:33:09.374 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [6, 8, 0, 9, 2, 4] 2024-09-17T16:33:09.374 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.0', '6', '8', '0', '9', '2', '4'] 2024-09-17T16:33:09.374 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 1.0 6 8 0 9 2 4 2024-09-17T16:33:09.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:09.482 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:09.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:09.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:09.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:09.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:09.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:09.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:09.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:09.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:09.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:09 smithi017 ceph-mon[27571]: pgmap v1576: 27 pgs: 1 active+recovery_wait, 5 active+recovering+undersized+remapped, 21 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 11113/103116 objects misplaced (10.777%); 11 MiB/s, 193 objects/s recovering 2024-09-17T16:33:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:09 smithi017 ceph-mon[27571]: osdmap e708: 12 total, 12 up, 12 in 2024-09-17T16:33:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:09 smithi017 ceph-mon[27571]: 7.17 scrub starts 2024-09-17T16:33:09.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:09 smithi173 ceph-mon[30530]: pgmap v1576: 27 pgs: 1 active+recovery_wait, 5 active+recovering+undersized+remapped, 21 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 11113/103116 objects misplaced (10.777%); 11 MiB/s, 193 objects/s recovering 2024-09-17T16:33:09.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:09 smithi173 ceph-mon[30530]: osdmap e708: 12 total, 12 up, 12 in 2024-09-17T16:33:09.796 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:09 smithi173 ceph-mon[30530]: 7.17 scrub starts 2024-09-17T16:33:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:09 smithi071 ceph-mon[30002]: pgmap v1576: 27 pgs: 1 active+recovery_wait, 5 active+recovering+undersized+remapped, 21 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 715 op/s; 11113/103116 objects misplaced (10.777%); 11 MiB/s, 193 objects/s recovering 2024-09-17T16:33:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:09 smithi071 ceph-mon[30002]: osdmap e708: 12 total, 12 up, 12 in 2024-09-17T16:33:09.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:09 smithi071 ceph-mon[30002]: 7.17 scrub starts 2024-09-17T16:33:09.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:09.875 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:09.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:09.893 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:33:09.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:10.000 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:10.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:10.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:10.102 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:10.143 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:10.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:10.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:10.268 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:10.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:10.370 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:10.392 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:10.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:10.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:10.528 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:10.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:10.630 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:10.661 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:10.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:10.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:10.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:10.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:10.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:10.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:10.892 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:10.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:10.913 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:10.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:10.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:11.018 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:11.053 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:11.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:11.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:11.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:11.176 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:11.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:11.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:11.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:11.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:11.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:11.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:11.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:11.429 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:11.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:11.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:11.569 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:11.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:11.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:11.695 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:11.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:11.796 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:11.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:11 smithi071 ceph-mon[30002]: pgmap v1578: 27 pgs: 1 creating+peering, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 11105/101865 objects misplaced (10.902%); 7.8 MiB/s, 151 objects/s recovering 2024-09-17T16:33:11.818 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:11.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:11.842 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:11 smithi017 ceph-mon[27571]: pgmap v1578: 27 pgs: 1 creating+peering, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 11105/101865 objects misplaced (10.902%); 7.8 MiB/s, 151 objects/s recovering 2024-09-17T16:33:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:11 smithi173 ceph-mon[30530]: pgmap v1578: 27 pgs: 1 creating+peering, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 11105/101865 objects misplaced (10.902%); 7.8 MiB/s, 151 objects/s recovering 2024-09-17T16:33:11.920 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:11.943 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:11.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:12.045 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:12.068 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:12.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:12.170 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:12.205 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:12.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:12.307 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:12.330 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:12.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:12.431 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:12.452 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:12.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:12.556 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:12.578 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:12.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:12.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:12.726 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:12.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:12.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:12.868 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:12.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:12.970 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:13.014 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:13.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:13.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:13.138 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:13.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:13.240 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:13.269 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:13.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:13.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:13.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:13.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:13.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:13.494 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:13.517 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:13.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:13.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:13.650 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:13.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:13.752 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:13.781 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:13.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:13 smithi071 ceph-mon[30002]: pgmap v1579: 27 pgs: 1 creating+peering, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 9583/97587 objects misplaced (9.820%); 8.6 MiB/s, 164 objects/s recovering 2024-09-17T16:33:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:13 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:13 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:13.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:13.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:13 smithi173 ceph-mon[30530]: pgmap v1579: 27 pgs: 1 creating+peering, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 9583/97587 objects misplaced (9.820%); 8.6 MiB/s, 164 objects/s recovering 2024-09-17T16:33:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:13 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:13 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:13.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:13.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:13 smithi017 ceph-mon[27571]: pgmap v1579: 27 pgs: 1 creating+peering, 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 21 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 9583/97587 objects misplaced (9.820%); 8.6 MiB/s, 164 objects/s recovering 2024-09-17T16:33:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:13 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:13 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:14.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:14.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:14.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:14.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:14.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:14.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:14.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:14.295 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:33:14.295 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:33:14.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:14.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:14.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:14.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:14.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:14.430 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:14.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:14.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:14.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:14.656 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:14.693 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:14.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:14.795 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:14.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:14 smithi071 ceph-mon[30002]: osdmap e709: 12 total, 12 up, 12 in 2024-09-17T16:33:14.816 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:14.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:14 smithi173 ceph-mon[30530]: osdmap e709: 12 total, 12 up, 12 in 2024-09-17T16:33:14.918 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:14.940 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:14.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:14.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:14 smithi017 ceph-mon[27571]: osdmap e709: 12 total, 12 up, 12 in 2024-09-17T16:33:15.042 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:15.084 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:15.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:15.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:15.207 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:15.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:15.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:15.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:15.333 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:15.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:15.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:15.456 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:15.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:15.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:15.583 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:15.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:15.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:15.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:15.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:15.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:15.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:15.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:15.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:15 smithi173 ceph-mon[30530]: pgmap v1581: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 9069/93666 objects misplaced (9.682%); 6.5 MiB/s, 137 objects/s recovering 2024-09-17T16:33:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:15 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:15 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:15 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:15 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:15 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:15.954 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:15.976 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:15.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:15.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:15 smithi017 ceph-mon[27571]: pgmap v1581: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 9069/93666 objects misplaced (9.682%); 6.5 MiB/s, 137 objects/s recovering 2024-09-17T16:33:15.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:15 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:15.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:15 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:15.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:15 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:15.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:15 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:15.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:15 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:16.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:15 smithi071 ceph-mon[30002]: pgmap v1581: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 9069/93666 objects misplaced (9.682%); 6.5 MiB/s, 137 objects/s recovering 2024-09-17T16:33:16.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:15 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:16.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:15 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:16.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:15 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:16.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:15 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:16.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:15 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:16.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:16.117 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:16.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:16.218 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:16.257 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:16.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:16.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:16.394 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:16.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:16.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:16.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:16.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:16.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:16.645 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:16.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:16.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:16.773 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:16.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:16.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:16 smithi173 ceph-mon[30530]: osdmap e710: 12 total, 12 up, 12 in 2024-09-17T16:33:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:16.914 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:16.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:16.917 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:16 smithi017 ceph-mon[27571]: osdmap e710: 12 total, 12 up, 12 in 2024-09-17T16:33:17.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:17.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:17.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:17.043 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:16 smithi071 ceph-mon[30002]: osdmap e710: 12 total, 12 up, 12 in 2024-09-17T16:33:17.144 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:17.168 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:17.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:17.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:17.304 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:17.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:17.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:17.429 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:17.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:17.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:17.560 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:17.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:17.664 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:17.705 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:17.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:17.806 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:17.831 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:17.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:17 smithi173 ceph-mon[30530]: pgmap v1583: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 9069/91485 objects misplaced (9.913%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T16:33:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:17 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:17 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:17 smithi173 ceph-mon[30530]: osdmap e711: 12 total, 12 up, 12 in 2024-09-17T16:33:17.934 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:17.957 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:17.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:17.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:17 smithi017 ceph-mon[27571]: pgmap v1583: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 9069/91485 objects misplaced (9.913%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T16:33:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:17 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:17 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:17.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:17 smithi017 ceph-mon[27571]: osdmap e711: 12 total, 12 up, 12 in 2024-09-17T16:33:18.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:18.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:17 smithi071 ceph-mon[30002]: pgmap v1583: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 9069/91485 objects misplaced (9.913%); 5.6 MiB/s, 90 objects/s recovering 2024-09-17T16:33:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:17 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:17 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:17 smithi071 ceph-mon[30002]: osdmap e711: 12 total, 12 up, 12 in 2024-09-17T16:33:18.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:18.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:18.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:18.208 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:18.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:18.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:18.333 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:18.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:18.436 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:18.472 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:18.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:18.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:18.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:18.596 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:18.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:18.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:18.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:18.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:18.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:18.850 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:18.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:18.952 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:18.975 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:18.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:19.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:19.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:19.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:19.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:19.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:19.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:19.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:19.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:19.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:19.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:19.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:19.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:19.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:19.615 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:19.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:19.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:19.740 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:19.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:19.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:19.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:19.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:19 smithi173 ceph-mon[30530]: pgmap v1585: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 6842/83304 objects misplaced (8.213%); 9.5 MiB/s, 152 objects/s recovering 2024-09-17T16:33:19.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:19 smithi017 ceph-mon[27571]: pgmap v1585: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 6842/83304 objects misplaced (8.213%); 9.5 MiB/s, 152 objects/s recovering 2024-09-17T16:33:19.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:19.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:20.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:19 smithi071 ceph-mon[30002]: pgmap v1585: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 6842/83304 objects misplaced (8.213%); 9.5 MiB/s, 152 objects/s recovering 2024-09-17T16:33:20.095 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:20.118 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:20.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:20.220 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:20.241 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:20.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:20.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:20.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:20.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:20.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:33:20.460 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:20.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:20.542 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:20.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:20.639 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:20.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:20.665 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:20.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:20.714 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:20 smithi017 ceph-mon[27571]: 7.18 scrub starts 2024-09-17T16:33:20.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:20.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:20.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:20.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:20.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:20 smithi173 ceph-mon[30530]: 7.18 scrub starts 2024-09-17T16:33:20.923 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:20.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:21.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:21.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:21.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:20 smithi071 ceph-mon[30002]: 7.18 scrub starts 2024-09-17T16:33:21.151 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:21.179 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:21.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:21.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:21.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:21.304 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:21.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:21.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:21.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:21.431 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:21.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:21.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:21.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:21.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:21.663 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:21.701 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:21.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:21.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:21.803 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:21.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:21.838 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:21.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:21 smithi173 ceph-mon[30530]: pgmap v1586: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 6842/81933 objects misplaced (8.351%); 7.1 MiB/s, 114 objects/s recovering 2024-09-17T16:33:21.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:21.973 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:21.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:21.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:21 smithi017 ceph-mon[27571]: pgmap v1586: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 6842/81933 objects misplaced (8.351%); 7.1 MiB/s, 114 objects/s recovering 2024-09-17T16:33:22.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:21 smithi071 ceph-mon[30002]: pgmap v1586: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 6842/81933 objects misplaced (8.351%); 7.1 MiB/s, 114 objects/s recovering 2024-09-17T16:33:22.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:22.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:22.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:22.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:22.222 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:22.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:22.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:22.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:22.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:22.451 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:22.474 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:22.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:22.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:22.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:22.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:22.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:22.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:22.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:22.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:22.862 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:22 smithi017 ceph-mon[27571]: pgmap v1587: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 6842/80103 objects misplaced (8.542%); 5.8 MiB/s, 92 objects/s recovering 2024-09-17T16:33:22.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:22.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:22 smithi173 ceph-mon[30530]: pgmap v1587: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 6842/80103 objects misplaced (8.542%); 5.8 MiB/s, 92 objects/s recovering 2024-09-17T16:33:22.965 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:23.001 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:23.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:22 smithi071 ceph-mon[30002]: pgmap v1587: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 6842/80103 objects misplaced (8.542%); 5.8 MiB/s, 92 objects/s recovering 2024-09-17T16:33:23.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:23.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:23.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:23.228 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:23.251 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:23.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:23.352 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:23.390 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:23.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:23.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:23.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:23.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:23.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:23.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:23.654 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:23.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:23.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:23.777 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:23.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:23.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:23.879 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:23.902 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:23.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:23.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:24.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:24.027 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:24.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:24.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:24.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:24.164 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:24.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:24.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:24.289 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:24.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:24.391 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:24.431 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:24.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:24.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:24.554 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:24.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:24.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:24.656 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:24.700 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:24.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:24.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:24 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:24.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:24 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:24.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:24 smithi017 ceph-mon[27571]: osdmap e712: 12 total, 12 up, 12 in 2024-09-17T16:33:24.770 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:24 smithi017 ceph-mon[27571]: pgmap v1589: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 4610/71568 objects misplaced (6.441%); 12 MiB/s, 185 objects/s recovering 2024-09-17T16:33:24.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:24.843 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:24.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:24 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:24 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:24 smithi173 ceph-mon[30530]: osdmap e712: 12 total, 12 up, 12 in 2024-09-17T16:33:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:24 smithi173 ceph-mon[30530]: pgmap v1589: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 4610/71568 objects misplaced (6.441%); 12 MiB/s, 185 objects/s recovering 2024-09-17T16:33:24.945 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:24.986 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:24.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:24 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:24 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:24 smithi071 ceph-mon[30002]: osdmap e712: 12 total, 12 up, 12 in 2024-09-17T16:33:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:24 smithi071 ceph-mon[30002]: pgmap v1589: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 4610/71568 objects misplaced (6.441%); 12 MiB/s, 185 objects/s recovering 2024-09-17T16:33:25.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:25.131 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:25.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:25.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:25.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:25.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:25.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:25.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:25.399 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:25.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:25.501 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:25.524 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:25.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:25.626 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:25.650 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:25.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:25.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:25.752 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:25.786 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:25.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:25.888 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:25.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:33:25.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:33:25.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:33:25.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:33:25.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:33:25.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:33:25.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:33:25.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:33:25.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:33:25.914 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:33:25.914 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:33:25.914 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:33:25.914 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:33:25.914 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:33:25.914 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:33:25.914 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:33:25.914 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:33:25.914 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:33:25.915 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:33:25.915 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:33:25.915 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:33:25.915 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:33:25.915 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:33:25.915 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:33:25.915 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2530086591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:25.922 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:25.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:26.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:26.046 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:26.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:26.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2530086591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2530086591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:26.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:26.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:26.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:26.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:26.309 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:26.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:26.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:26.451 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:26.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:26.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:26.576 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:26.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:26.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:26.700 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:26.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:26.801 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:26.825 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:26.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:26.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:26.926 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:26.968 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:26.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:26 smithi017 ceph-mon[27571]: pgmap v1590: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 4610/70308 objects misplaced (6.557%); 9.8 MiB/s, 156 objects/s recovering 2024-09-17T16:33:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:26 smithi071 ceph-mon[30002]: pgmap v1590: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 4610/70308 objects misplaced (6.557%); 9.8 MiB/s, 156 objects/s recovering 2024-09-17T16:33:27.069 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:27.106 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:27.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:27.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:26 smithi173 ceph-mon[30530]: pgmap v1590: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 4610/70308 objects misplaced (6.557%); 9.8 MiB/s, 156 objects/s recovering 2024-09-17T16:33:27.207 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:27.243 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:27.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:27.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:27.369 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:27.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:27.471 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:27.505 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:27.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:27.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:27.607 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:27.629 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:27.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:27.731 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:27.755 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:27.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:27.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:27.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:27.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:27.982 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:28.005 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:28.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:28.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:28.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:28.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:28.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:28.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:28.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:28.364 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:28.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:28.405 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:28.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:28.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:28.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:28.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:28.636 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:28.671 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:28.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:28.773 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:28.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:28.813 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:28.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:28.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:28.961 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:28.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:28 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:28 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:28 smithi017 ceph-mon[27571]: osdmap e713: 12 total, 12 up, 12 in 2024-09-17T16:33:28.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:28 smithi017 ceph-mon[27571]: pgmap v1592: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 2/60939 objects degraded (0.003%); 2034/60939 objects misplaced (3.338%); 13 MiB/s, 215 objects/s recovering 2024-09-17T16:33:29.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:28 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:28 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:28 smithi071 ceph-mon[30002]: osdmap e713: 12 total, 12 up, 12 in 2024-09-17T16:33:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:28 smithi071 ceph-mon[30002]: pgmap v1592: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 2/60939 objects degraded (0.003%); 2034/60939 objects misplaced (3.338%); 13 MiB/s, 215 objects/s recovering 2024-09-17T16:33:29.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:29.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:28 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:28 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:28 smithi173 ceph-mon[30530]: osdmap e713: 12 total, 12 up, 12 in 2024-09-17T16:33:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:28 smithi173 ceph-mon[30530]: pgmap v1592: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 2/60939 objects degraded (0.003%); 2034/60939 objects misplaced (3.338%); 13 MiB/s, 215 objects/s recovering 2024-09-17T16:33:29.194 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:29.220 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:29.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:29.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:29.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:29.345 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:29.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:29.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:29.486 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:29.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:29.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:29.611 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:29.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:29.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:29.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:29.748 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:29.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:29.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:29.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:29.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:29.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:29.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:29 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2/60939 objects degraded (0.003%), 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:29.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:29 smithi017 ceph-mon[27571]: osdmap e714: 12 total, 12 up, 12 in 2024-09-17T16:33:30.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:30.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:29 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2/60939 objects degraded (0.003%), 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:29 smithi071 ceph-mon[30002]: osdmap e714: 12 total, 12 up, 12 in 2024-09-17T16:33:30.109 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:30.138 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:30.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:29 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2/60939 objects degraded (0.003%), 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:30.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:29 smithi173 ceph-mon[30530]: osdmap e714: 12 total, 12 up, 12 in 2024-09-17T16:33:30.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:30.260 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:30.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:30.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:30.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:30.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:30.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:30.487 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:30.512 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:30.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:30.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:30.638 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:30.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:30 smithi017 ceph-mon[27571]: 7.16 scrub starts 2024-09-17T16:33:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:30 smithi017 ceph-mon[27571]: 7.12 scrub starts 2024-09-17T16:33:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:30 smithi017 ceph-mon[27571]: 7.11 scrub starts 2024-09-17T16:33:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:30 smithi017 ceph-mon[27571]: pgmap v1594: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 2/59592 objects degraded (0.003%); 2034/59592 objects misplaced (3.413%); 8.3 MiB/s, 152 objects/s recovering 2024-09-17T16:33:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:30 smithi017 ceph-mon[27571]: osdmap e715: 12 total, 12 up, 12 in 2024-09-17T16:33:30.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:30.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:30.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:30.862 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:30.885 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:30.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:30.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:31.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:31.018 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:31.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:30 smithi071 ceph-mon[30002]: 7.16 scrub starts 2024-09-17T16:33:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:30 smithi071 ceph-mon[30002]: 7.12 scrub starts 2024-09-17T16:33:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:30 smithi071 ceph-mon[30002]: 7.11 scrub starts 2024-09-17T16:33:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:30 smithi071 ceph-mon[30002]: pgmap v1594: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 2/59592 objects degraded (0.003%); 2034/59592 objects misplaced (3.413%); 8.3 MiB/s, 152 objects/s recovering 2024-09-17T16:33:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:30 smithi071 ceph-mon[30002]: osdmap e715: 12 total, 12 up, 12 in 2024-09-17T16:33:31.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:31.143 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:31.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:31.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:30 smithi173 ceph-mon[30530]: 7.16 scrub starts 2024-09-17T16:33:31.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:30 smithi173 ceph-mon[30530]: 7.12 scrub starts 2024-09-17T16:33:31.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:30 smithi173 ceph-mon[30530]: 7.11 scrub starts 2024-09-17T16:33:31.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:30 smithi173 ceph-mon[30530]: pgmap v1594: 27 pgs: 1 active+recovery_wait, 4 active+recovering+undersized+remapped, 22 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 619 op/s; 2/59592 objects degraded (0.003%); 2034/59592 objects misplaced (3.413%); 8.3 MiB/s, 152 objects/s recovering 2024-09-17T16:33:31.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:30 smithi173 ceph-mon[30530]: osdmap e715: 12 total, 12 up, 12 in 2024-09-17T16:33:31.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:31.285 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:31.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:31.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:31.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:31.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:31.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:31.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:31.545 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:31.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:31.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:31.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:31.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:31.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:31.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:31.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:31.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:31.903 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:31.925 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:31.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:32.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:32.058 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:32.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:32.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:31 smithi173 ceph-mon[30530]: 7.13 scrub starts 2024-09-17T16:33:32.159 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:32.185 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:32.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:32.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:31 smithi017 ceph-mon[27571]: 7.13 scrub starts 2024-09-17T16:33:32.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:31 smithi071 ceph-mon[30002]: 7.13 scrub starts 2024-09-17T16:33:32.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:32.326 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:32.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:32.427 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:32.451 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:32.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:32.553 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:32.575 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:32.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:32.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:32.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:32.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:32.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:32.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:32.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:32.925 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:32.962 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:32.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:33.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:33.088 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:33.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:33.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:33.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:33.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:32 smithi017 ceph-mon[27571]: pgmap v1596: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 2/58116 objects degraded (0.003%); 2034/58116 objects misplaced (3.500%); 8.9 MiB/s, 163 objects/s recovering 2024-09-17T16:33:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:32 smithi071 ceph-mon[30002]: pgmap v1596: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 2/58116 objects degraded (0.003%); 2034/58116 objects misplaced (3.500%); 8.9 MiB/s, 163 objects/s recovering 2024-09-17T16:33:33.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:33.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:33.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:33.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:32 smithi173 ceph-mon[30530]: pgmap v1596: 27 pgs: 4 active+recovering+undersized+remapped, 23 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 2/58116 objects degraded (0.003%); 2034/58116 objects misplaced (3.500%); 8.9 MiB/s, 163 objects/s recovering 2024-09-17T16:33:33.447 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:33.469 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:33.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:33.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:33.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:33.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:33.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:33.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:33.720 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:33.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:33.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:33.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:33.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:33.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:33.951 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:33.973 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:33.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:34.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:34.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:34.121 INFO:teuthology.orchestra.run.smithi017.stderr:set 1.0 pg_upmap_items mapping to [6->8,0->9,2->4] 2024-09-17T16:33:34.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:34.235 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:34.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:34.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:34.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4161340186' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [6, 8, 0, 9, 2, 4]} : dispatch 2024-09-17T16:33:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4161340186' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [6, 8, 0, 9, 2, 4]} : dispatch 2024-09-17T16:33:34.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:34.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:34.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:34.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:34.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4161340186' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [6, 8, 0, 9, 2, 4]} : dispatch 2024-09-17T16:33:34.464 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:34.486 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:34.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:34.589 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:34.622 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:34.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:34.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:34.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:34.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:34.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:34.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:34.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:34.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:34.980 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:35.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:35.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:35.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:35.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:35.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4161340186' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [6, 8, 0, 9, 2, 4]}]': finished 2024-09-17T16:33:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:34 smithi017 ceph-mon[27571]: osdmap e716: 12 total, 12 up, 12 in 2024-09-17T16:33:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:34 smithi017 ceph-mon[27571]: pgmap v1598: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 2/49872 objects degraded (0.004%); 261/49872 objects misplaced (0.523%); 6.2 MiB/s, 129 objects/s recovering 2024-09-17T16:33:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:34 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2/58116 objects degraded (0.003%), 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:35.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:35.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:35.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4161340186' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [6, 8, 0, 9, 2, 4]}]': finished 2024-09-17T16:33:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:34 smithi071 ceph-mon[30002]: osdmap e716: 12 total, 12 up, 12 in 2024-09-17T16:33:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:34 smithi071 ceph-mon[30002]: pgmap v1598: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 2/49872 objects degraded (0.004%); 261/49872 objects misplaced (0.523%); 6.2 MiB/s, 129 objects/s recovering 2024-09-17T16:33:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:34 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2/58116 objects degraded (0.003%), 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:35.371 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4161340186' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [6, 8, 0, 9, 2, 4]}]': finished 2024-09-17T16:33:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:34 smithi173 ceph-mon[30530]: osdmap e716: 12 total, 12 up, 12 in 2024-09-17T16:33:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:34 smithi173 ceph-mon[30530]: pgmap v1598: 27 pgs: 3 active+recovering+undersized+remapped, 24 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 2/49872 objects degraded (0.004%); 261/49872 objects misplaced (0.523%); 6.2 MiB/s, 129 objects/s recovering 2024-09-17T16:33:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:34 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2/58116 objects degraded (0.003%), 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:35.396 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:35.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:35.498 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:35.538 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:35.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:35.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:35.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:35.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:35.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:35.803 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:35.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:35.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:35.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:35.937 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:35.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:36.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:36.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:36.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:36.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:36.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:36.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:36.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:36.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:36.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:36.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:36.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:36.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:36.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:36.557 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:36.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:36.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:36.686 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:36.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:36.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:36.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:36.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:36.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:36.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:36.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:36.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:36 smithi017 ceph-mon[27571]: osdmap e717: 12 total, 12 up, 12 in 2024-09-17T16:33:36.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:36 smithi017 ceph-mon[27571]: pgmap v1600: 27 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 24 active+clean; 979 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 261/46968 objects misplaced (0.556%); 6.2 MiB/s, 150 objects/s recovering 2024-09-17T16:33:37.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:37.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:37.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:37.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:37.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:36 smithi071 ceph-mon[30002]: osdmap e717: 12 total, 12 up, 12 in 2024-09-17T16:33:37.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:36 smithi071 ceph-mon[30002]: pgmap v1600: 27 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 24 active+clean; 979 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 261/46968 objects misplaced (0.556%); 6.2 MiB/s, 150 objects/s recovering 2024-09-17T16:33:37.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:36 smithi173 ceph-mon[30530]: osdmap e717: 12 total, 12 up, 12 in 2024-09-17T16:33:37.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:36 smithi173 ceph-mon[30530]: pgmap v1600: 27 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 24 active+clean; 979 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 261/46968 objects misplaced (0.556%); 6.2 MiB/s, 150 objects/s recovering 2024-09-17T16:33:37.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:37.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:37.284 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:37.305 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:37.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:37.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:37.409 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:37.431 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:37.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:37.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:37.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:37.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:37.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:37.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:37.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:37.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:37.781 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:37.813 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:37.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:37.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:37.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:37.936 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:37.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:37.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:37 smithi017 ceph-mon[27571]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:33:37.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:37 smithi017 ceph-mon[27571]: osdmap e718: 12 total, 12 up, 12 in 2024-09-17T16:33:37.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:37 smithi017 ceph-mon[27571]: 7.14 scrub starts 2024-09-17T16:33:38.039 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:38.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:37 smithi071 ceph-mon[30002]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:33:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:37 smithi071 ceph-mon[30002]: osdmap e718: 12 total, 12 up, 12 in 2024-09-17T16:33:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:37 smithi071 ceph-mon[30002]: 7.14 scrub starts 2024-09-17T16:33:38.064 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:38.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:38.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:38.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:37 smithi173 ceph-mon[30530]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:33:38.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:37 smithi173 ceph-mon[30530]: osdmap e718: 12 total, 12 up, 12 in 2024-09-17T16:33:38.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:37 smithi173 ceph-mon[30530]: 7.14 scrub starts 2024-09-17T16:33:38.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:38.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:38.289 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:38.311 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:38.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:38.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:38.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:38.435 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:38.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:38.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:38.557 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:38.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:38.659 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:38.680 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:38.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:38.716 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:38.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:38 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:38.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:38 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:38.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:38 smithi017 ceph-mon[27571]: osdmap e719: 12 total, 12 up, 12 in 2024-09-17T16:33:38.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:38 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:33:38.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:38 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:33:38.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:38 smithi017 ceph-mon[27571]: pgmap v1603: 27 pgs: 1 active+undersized+remapped, 1 creating+peering, 1 active+recovering+undersized+remapped, 24 active+clean; 865 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 27/41487 objects misplaced (0.065%); 1.2 MiB/s, 109 objects/s recovering 2024-09-17T16:33:38.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:38 smithi017 ceph-mon[27571]: 7.15 scrub starts 2024-09-17T16:33:38.782 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:38.805 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:38.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:38.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:38.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:38.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:39.037 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:39.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:38 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:39.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:38 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:39.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:38 smithi173 ceph-mon[30530]: osdmap e719: 12 total, 12 up, 12 in 2024-09-17T16:33:39.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:38 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:33:39.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:38 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:33:39.045 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:38 smithi173 ceph-mon[30530]: pgmap v1603: 27 pgs: 1 active+undersized+remapped, 1 creating+peering, 1 active+recovering+undersized+remapped, 24 active+clean; 865 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 27/41487 objects misplaced (0.065%); 1.2 MiB/s, 109 objects/s recovering 2024-09-17T16:33:39.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:38 smithi173 ceph-mon[30530]: 7.15 scrub starts 2024-09-17T16:33:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:39.061 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:39.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:38 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:38 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:38 smithi071 ceph-mon[30002]: osdmap e719: 12 total, 12 up, 12 in 2024-09-17T16:33:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:38 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:33:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:38 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:33:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:38 smithi071 ceph-mon[30002]: pgmap v1603: 27 pgs: 1 active+undersized+remapped, 1 creating+peering, 1 active+recovering+undersized+remapped, 24 active+clean; 865 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 27/41487 objects misplaced (0.065%); 1.2 MiB/s, 109 objects/s recovering 2024-09-17T16:33:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:38 smithi071 ceph-mon[30002]: 7.15 scrub starts 2024-09-17T16:33:39.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:39.186 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:39.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:39.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:39.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:39.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:39.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:39.415 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:39.452 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:39.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:39.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:39.577 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:39.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:39.678 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:39.696 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:33:39.716 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:39.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:39.818 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:39.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:33:39.849 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:39.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:39.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:39.964 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:39 smithi071 ceph-mon[30002]: osdmap e720: 12 total, 12 up, 12 in 2024-09-17T16:33:39.964 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:39 smithi071 ceph-mon[30002]: 7.19 scrub starts 2024-09-17T16:33:39.964 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2129399642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:33:39.964 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:39 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:39.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:39.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:39 smithi017 ceph-mon[27571]: osdmap e720: 12 total, 12 up, 12 in 2024-09-17T16:33:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:39 smithi017 ceph-mon[27571]: 7.19 scrub starts 2024-09-17T16:33:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2129399642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:33:39.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:39 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:40.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:40.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:40.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:39 smithi173 ceph-mon[30530]: osdmap e720: 12 total, 12 up, 12 in 2024-09-17T16:33:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:39 smithi173 ceph-mon[30530]: 7.19 scrub starts 2024-09-17T16:33:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2129399642' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:33:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:39 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:33:40.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:40.251 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:40.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:40.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:40.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:40.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:40.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:40.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:40.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:40.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:40.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:40.640 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:40.741 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:40.763 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:40.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:40.851 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:33:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:33:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:33:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:33:40.851 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:33:40.851 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:40 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:33:40.851 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2129399642' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:33:40.852 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:40 smithi017 ceph-mon[27571]: osdmap e721: 12 total, 12 up, 12 in 2024-09-17T16:33:40.852 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:40 smithi017 ceph-mon[27571]: 7.17 scrub starts 2024-09-17T16:33:40.852 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:40 smithi017 ceph-mon[27571]: pgmap v1606: 27 pgs: 1 active+undersized+remapped, 1 creating+peering, 25 active+clean; 802 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 707 op/s; 1.4 MiB/s, 134 objects/s recovering 2024-09-17T16:33:40.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:40.886 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:40.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:40.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:41.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:41.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:40 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:33:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2129399642' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:33:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:40 smithi071 ceph-mon[30002]: osdmap e721: 12 total, 12 up, 12 in 2024-09-17T16:33:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:40 smithi071 ceph-mon[30002]: 7.17 scrub starts 2024-09-17T16:33:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:40 smithi071 ceph-mon[30002]: pgmap v1606: 27 pgs: 1 active+undersized+remapped, 1 creating+peering, 25 active+clean; 802 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 707 op/s; 1.4 MiB/s, 134 objects/s recovering 2024-09-17T16:33:41.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:41.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:41.142 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:41.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:40 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:33:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2129399642' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:33:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:40 smithi173 ceph-mon[30530]: osdmap e721: 12 total, 12 up, 12 in 2024-09-17T16:33:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:40 smithi173 ceph-mon[30530]: 7.17 scrub starts 2024-09-17T16:33:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:40 smithi173 ceph-mon[30530]: pgmap v1606: 27 pgs: 1 active+undersized+remapped, 1 creating+peering, 25 active+clean; 802 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 707 op/s; 1.4 MiB/s, 134 objects/s recovering 2024-09-17T16:33:41.243 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:41.274 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:41.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:41.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:41.376 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:41.404 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:41.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:41.506 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:41.521 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:41.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:33:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:41.541 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:41.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:41.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:41.663 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:41.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:41.765 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:41.797 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:41.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:41.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:41.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:41.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:41.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:42.032 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:42.055 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:42.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:42.157 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:42.181 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:42.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:42.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:42.188 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:33:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:33:42.189 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:42.232 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:33:42.233 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:33:42.270 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:33:42.270 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:33:42.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:42.305 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:42.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:42.310 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:33:42.310 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:33:42.344 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:33:42.344 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:33:42.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:33:42.378 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:33:42.379 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:42 smithi017 ceph-mon[27571]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T16:33:42.379 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:42 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T16:33:42.379 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1046596708' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:42.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:42.413 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:33:42.413 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:33:42.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:42.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:42.445 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:33:42.446 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:33:42.482 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:33:42.482 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:33:42.517 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:33:42.518 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:33:42.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:42.560 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:42.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:42.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:42.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:42 smithi071 ceph-mon[30002]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T16:33:42.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:42 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T16:33:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1046596708' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:42.598 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:33:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:42 smithi173 ceph-mon[30530]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-17T16:33:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:42 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T16:33:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1046596708' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:42.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:42.702 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:42.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:42.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:42.762 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:33:42.762 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:33:42.804 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:33:42.804 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:33:42.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:42.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:42.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:42.868 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:33:42.868 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:33:42.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:42.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:42.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:43.056 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:43.079 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:43.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:43.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:43.211 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:43.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:43.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:43.340 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:43.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:43.415 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:43 smithi017 ceph-mon[27571]: pgmap v1607: 27 pgs: 27 active+clean; 762 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 1.0 MiB/s, 99 objects/s recovering 2024-09-17T16:33:43.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:43.463 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:43.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:43.491 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:33:43.492 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:33:43.492 INFO:tasks.thrashosds.thrasher:Reweighting osd 5 to 0.44201069285398775 2024-09-17T16:33:43.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 0.44201069285398775 2024-09-17T16:33:43.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:43 smithi071 ceph-mon[30002]: pgmap v1607: 27 pgs: 27 active+clean; 762 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 1.0 MiB/s, 99 objects/s recovering 2024-09-17T16:33:43.565 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:43.595 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:43.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:43 smithi173 ceph-mon[30530]: pgmap v1607: 27 pgs: 27 active+clean; 762 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 1.0 MiB/s, 99 objects/s recovering 2024-09-17T16:33:43.697 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:43.736 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:43.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:43.793 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:33:43.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:43.878 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:43.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:43.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:43.980 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:44.010 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:44.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:44.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:44.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:44.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:44.240 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:44.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:44.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:44.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:44.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:44.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:44.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:44.502 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:44.523 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:44.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:44.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:44.561 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:44.622 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:44.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:44.633 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:44.668 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:44.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:44.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:44.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:44.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:44.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:44.921 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:44.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:45.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:45.045 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:45.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:45.147 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:45.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:45.171 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:45.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:45.272 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:45.287 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:45 smithi017 ceph-mon[27571]: pgmap v1608: 27 pgs: 27 active+clean; 590 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 90 KiB/s, 34 objects/s recovering 2024-09-17T16:33:45.294 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:45.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:45.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:45.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:45.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:45.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:45.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:45.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:45.543 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:45.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:45 smithi071 ceph-mon[30002]: pgmap v1608: 27 pgs: 27 active+clean; 590 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 90 KiB/s, 34 objects/s recovering 2024-09-17T16:33:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:45 smithi173 ceph-mon[30530]: pgmap v1608: 27 pgs: 27 active+clean; 590 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 90 KiB/s, 34 objects/s recovering 2024-09-17T16:33:45.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:45.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:45.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:45.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:45.771 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:45.807 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:45.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:45.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:45.946 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:45.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:46.047 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:46.085 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:46.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:46.180 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 0.442011 (7127) 2024-09-17T16:33:46.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:46.209 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:46.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:46.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:46.345 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:46.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:46.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:46.475 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:46.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:46.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3307635915' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.44201069285398775} : dispatch 2024-09-17T16:33:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:46 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.44201069285398775} : dispatch 2024-09-17T16:33:46.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:33:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3307635915' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.44201069285398775} : dispatch 2024-09-17T16:33:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:46 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.44201069285398775} : dispatch 2024-09-17T16:33:46.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:46.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:46.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3307635915' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.44201069285398775} : dispatch 2024-09-17T16:33:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:46 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.44201069285398775} : dispatch 2024-09-17T16:33:46.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:46.724 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:46.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:46.777 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:33:46.777 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:33:46.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:46.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:46.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:46.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:46.951 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:46.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:46.973 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:46.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:47.075 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:47.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:47.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:47.182 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:33:47.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:47.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:47.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:47.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:47.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:47.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:47.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:47.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:47.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:47 smithi017 ceph-mon[27571]: pgmap v1609: 27 pgs: 27 active+clean; 517 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 72 KiB/s, 27 objects/s recovering 2024-09-17T16:33:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:47 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.44201069285398775}]': finished 2024-09-17T16:33:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:47 smithi017 ceph-mon[27571]: osdmap e722: 12 total, 12 up, 12 in 2024-09-17T16:33:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2380600092' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:33:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:47 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:33:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:47 smithi071 ceph-mon[30002]: pgmap v1609: 27 pgs: 27 active+clean; 517 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 72 KiB/s, 27 objects/s recovering 2024-09-17T16:33:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:47 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.44201069285398775}]': finished 2024-09-17T16:33:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:47 smithi071 ceph-mon[30002]: osdmap e722: 12 total, 12 up, 12 in 2024-09-17T16:33:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2380600092' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:33:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:47 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:33:47.566 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:33:47.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:47.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:47.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:47 smithi173 ceph-mon[30530]: pgmap v1609: 27 pgs: 27 active+clean; 517 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 72 KiB/s, 27 objects/s recovering 2024-09-17T16:33:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:47 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.44201069285398775}]': finished 2024-09-17T16:33:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:47 smithi173 ceph-mon[30530]: osdmap e722: 12 total, 12 up, 12 in 2024-09-17T16:33:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2380600092' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:33:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:47 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:33:47.695 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:47.717 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:47.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:47.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:47.841 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:47.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:47.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:47.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:47.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:48.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:48.089 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:48.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:48.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:48.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:48.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:48.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:48.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:48.337 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:48.439 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:48.461 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:48.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:48.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:48.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:48 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:33:48.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:48 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:33:48.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:48 smithi017 ceph-mon[27571]: osdmap e723: 12 total, 12 up, 12 in 2024-09-17T16:33:48.563 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:48 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:33:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:48 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:33:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:48 smithi071 ceph-mon[30002]: osdmap e723: 12 total, 12 up, 12 in 2024-09-17T16:33:48.586 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:48.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:48 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:33:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:48 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:33:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:48 smithi173 ceph-mon[30530]: osdmap e723: 12 total, 12 up, 12 in 2024-09-17T16:33:48.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:48.711 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:48.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:48.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:48.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:48.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:48.946 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:48.979 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:48.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:48.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:49.058 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:49.069 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:49.081 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:49.131 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:49.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:49.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:49.272 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:49.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:49.374 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:49.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:49 smithi017 ceph-mon[27571]: pgmap v1612: 27 pgs: 1 unknown, 26 active+clean; 373 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 858 op/s 2024-09-17T16:33:49.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:49 smithi017 ceph-mon[27571]: osdmap e724: 12 total, 12 up, 12 in 2024-09-17T16:33:49.396 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:49.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:49.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:49.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:49.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:49 smithi071 ceph-mon[30002]: pgmap v1612: 27 pgs: 1 unknown, 26 active+clean; 373 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 858 op/s 2024-09-17T16:33:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:49 smithi071 ceph-mon[30002]: osdmap e724: 12 total, 12 up, 12 in 2024-09-17T16:33:49.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:49.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:49 smithi173 ceph-mon[30530]: pgmap v1612: 27 pgs: 1 unknown, 26 active+clean; 373 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 858 op/s 2024-09-17T16:33:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:49 smithi173 ceph-mon[30530]: osdmap e724: 12 total, 12 up, 12 in 2024-09-17T16:33:49.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:49.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:49.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:49.825 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:49.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:49.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:49.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:49.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:50.031 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:33:50.031 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:33:50.059 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:50.094 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:50.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:50.196 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:50.218 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:50.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:50.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:50.355 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:50.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:50.457 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:50.485 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:50.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:50.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:50.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:50.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:50.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:50.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:50 smithi017 ceph-mon[27571]: 7.1 scrub starts 2024-09-17T16:33:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:50 smithi017 ceph-mon[27571]: 7.1 scrub ok 2024-09-17T16:33:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:50 smithi017 ceph-mon[27571]: 7.18 scrub starts 2024-09-17T16:33:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:50 smithi017 ceph-mon[27571]: 7.18 scrub ok 2024-09-17T16:33:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1187273168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:50.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:50.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:50.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:50 smithi071 ceph-mon[30002]: 7.1 scrub starts 2024-09-17T16:33:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:50 smithi071 ceph-mon[30002]: 7.1 scrub ok 2024-09-17T16:33:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:50 smithi071 ceph-mon[30002]: 7.18 scrub starts 2024-09-17T16:33:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:50 smithi071 ceph-mon[30002]: 7.18 scrub ok 2024-09-17T16:33:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1187273168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:50.818 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:33:50.818 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:33:50.842 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:50.855 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:33:50.856 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:33:50.865 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:50.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:50.888 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:33:50.888 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:33:50.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:50 smithi173 ceph-mon[30530]: 7.1 scrub starts 2024-09-17T16:33:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:50 smithi173 ceph-mon[30530]: 7.1 scrub ok 2024-09-17T16:33:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:50 smithi173 ceph-mon[30530]: 7.18 scrub starts 2024-09-17T16:33:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:50 smithi173 ceph-mon[30530]: 7.18 scrub ok 2024-09-17T16:33:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1187273168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:50.904 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:33:50.925 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:33:50.925 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:33:50.959 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:33:50.960 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:33:50.966 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:50.993 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:33:50.994 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:33:51.011 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:51.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:51.030 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:33:51.030 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:33:51.060 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:33:51.067 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:33:51.068 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:33:51.104 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:33:51.105 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:33:51.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:51.145 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:33:51.145 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:33:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:51.171 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:51.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:51.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:33:51.186 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:33:51.219 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:33:51.220 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:33:51.271 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:51.294 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:51.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:51.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:51.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:51.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:51.431 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:51.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:51.562 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:51.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:51.664 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:51.687 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:51.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:51 smithi017 ceph-mon[27571]: pgmap v1614: 27 pgs: 1 unknown, 26 active+clean; 343 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s 2024-09-17T16:33:51.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:51.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:51 smithi071 ceph-mon[30002]: pgmap v1614: 27 pgs: 1 unknown, 26 active+clean; 343 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s 2024-09-17T16:33:51.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:51.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:51 smithi173 ceph-mon[30530]: pgmap v1614: 27 pgs: 1 unknown, 26 active+clean; 343 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s 2024-09-17T16:33:51.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:51.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:51.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:52.044 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:52.078 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:52.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:52.180 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:52.209 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:52.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:52.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:52.333 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:52.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:52.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:52.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:52.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:52.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:52.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:52.591 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:52.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:52.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:52.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:52.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:52.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:52 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 16/14382 objects degraded (0.111%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:33:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:52 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 16/14382 objects degraded (0.111%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:33:52.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:52.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:52.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:52 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 16/14382 objects degraded (0.111%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:33:52.940 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:52.970 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:52.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:53.072 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:53.105 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:53.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:53.206 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:33:53.206 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:33:53.206 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-09-17T16:33:53.206 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T16:33:53.207 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T16:33:53.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-09-17T16:33:53.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:53.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:53.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:53.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:53.337 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:53.359 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:53.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:53.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:53.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:53.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:53.536 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:33:53.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:53.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:53.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:53.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:53 smithi173 ceph-mon[30530]: pgmap v1615: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 300 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 16/14382 objects degraded (0.111%); 1.5 MiB/s, 36 objects/s recovering 2024-09-17T16:33:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:53 smithi173 ceph-mon[30530]: 7.6 scrub starts 2024-09-17T16:33:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:53 smithi173 ceph-mon[30530]: 7.6 scrub ok 2024-09-17T16:33:53.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:53 smithi017 ceph-mon[27571]: pgmap v1615: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 300 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 16/14382 objects degraded (0.111%); 1.5 MiB/s, 36 objects/s recovering 2024-09-17T16:33:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:53 smithi017 ceph-mon[27571]: 7.6 scrub starts 2024-09-17T16:33:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:53 smithi017 ceph-mon[27571]: 7.6 scrub ok 2024-09-17T16:33:53.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:53.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:53.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:53 smithi071 ceph-mon[30002]: pgmap v1615: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 300 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 16/14382 objects degraded (0.111%); 1.5 MiB/s, 36 objects/s recovering 2024-09-17T16:33:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:53 smithi071 ceph-mon[30002]: 7.6 scrub starts 2024-09-17T16:33:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:53 smithi071 ceph-mon[30002]: 7.6 scrub ok 2024-09-17T16:33:53.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:53.904 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:53.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:54.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:54.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:54.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:54.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:54.164 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:54.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:54.264 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:54.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:54.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:54.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:54.388 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:54.422 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:54.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:54.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:54.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:54.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:54.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:54.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:54.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:54.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:54 smithi017 ceph-mon[27571]: 7.9 scrub starts 2024-09-17T16:33:54.717 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:54 smithi017 ceph-mon[27571]: 7.9 scrub ok 2024-09-17T16:33:54.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:54.792 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:54.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:54 smithi071 ceph-mon[30002]: 7.9 scrub starts 2024-09-17T16:33:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:54 smithi071 ceph-mon[30002]: 7.9 scrub ok 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26665.0:27646 6.5 e545/527 6:a767d192:::benchmark_data_smithi173_102017_object27645:head v 545'1660, mlcod=545'1657)", 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:32.707942+0000", 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 502.10623136999999, 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31267298700000001, 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.822 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.823 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.823 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.707942+0000", 2024-09-17T16:33:54.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.707942+0000", 2024-09-17T16:33:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.707943+0000", 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6200000000000006e-07 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.708050+0000", 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010678100000000001 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019547+0000", 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31149681200000001 2024-09-17T16:33:54.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019554+0000", 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2459999999999997e-06 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019609+0000", 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5305e-05 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019626+0000", 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6667000000000002e-05 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.020580+0000", 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00095388199999999997 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.020615+0000", 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5332e-05 2024-09-17T16:33:54.827 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26665.0:29370 6.0 e547/527 6:0f906142:::benchmark_data_smithi173_102017_object29369:head v 547'1848, mlcod=547'1845)", 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:35.556226+0000", 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 499.25794785199997, 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27913635799999997, 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.828 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.556226+0000", 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.556226+0000", 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.556226+0000", 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.556251+0000", 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4870000000000001e-05 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.833972+0000", 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27772044699999998 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.833981+0000", 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1989999999999995e-06 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.834052+0000", 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0758000000000003e-05 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.834077+0000", 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4850000000000001e-05 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.835298+0000", 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001221048 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.835362+0000", 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4550999999999999e-05 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:186285 6.f 6:ff66d09f:::benchmark_data_smithi173_102017_object93137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e621)", 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 277.27757540900001, 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53962611299999996, 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186285 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.833 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536599+0000", 2024-09-17T16:33:54.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4900000000000005e-07 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536603+0000", 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0380000000000001e-06 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536614+0000", 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1097000000000001e-05 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536621+0000", 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5560000000000004e-06 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540416+0000", 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037954429999999999 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540537+0000", 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012064499999999999 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540685+0000", 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014845900000000001 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.544648+0000", 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039628129999999999 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.800489+0000", 2024-09-17T16:33:54.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25584077599999999 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076152+0000", 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27566357899999999 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076194+0000", 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1468999999999999e-05 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076224+0000", 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0488999999999999e-05 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:26018 7.8 7:151f2f24:::benchmark_data_smithi173_116396_object26017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 230.77804379099999, 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35239512200000001, 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.839 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26018 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036131+0000", 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.029e-06 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036175+0000", 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4579000000000001e-05 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036188+0000", 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.218e-05 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036203+0000", 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5494000000000001e-05 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036337+0000", 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013415599999999999 2024-09-17T16:33:54.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036422+0000", 2024-09-17T16:33:54.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4427999999999999e-05 2024-09-17T16:33:54.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036500+0000", 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8009000000000005e-05 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037194+0000", 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00069466700000000005 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037673+0000", 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047842099999999998 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388425+0000", 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35075205700000001 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388482+0000", 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7630999999999998e-05 2024-09-17T16:33:54.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388525+0000", 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2471e-05 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:26018 7.8 e645/642 ondisk, result = 0)", 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 230.77677118899999, 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35113110600000003, 2024-09-17T16:33:54.846 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037403+0000", 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8700000000000001e-07 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037406+0000", 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8710000000000001e-06 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388344+0000", 2024-09-17T16:33:54.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35093866000000001 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388350+0000", 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388413+0000", 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2257999999999999e-05 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388420+0000", 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0450000000000003e-06 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388534+0000", 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011397599999999999 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:119650 7.8 7:17d95773:::benchmark_data_smithi173_116396_object24050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:58.385278+0000", 2024-09-17T16:33:54.850 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 116.428895178, 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30032905500000001, 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 119650 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385278+0000", 2024-09-17T16:33:54.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385278+0000", 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385280+0000", 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6300000000000001e-06 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385286+0000", 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2049999999999999e-06 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385298+0000", 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1477e-05 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385305+0000", 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9890000000000003e-06 2024-09-17T16:33:54.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.388017+0000", 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027118770000000001 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.388140+0000", 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012355600000000001 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.388258+0000", 2024-09-17T16:33:54.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118041 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.538443+0000", 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.150185233 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.546099+0000", 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007656014 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.685478+0000", 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.13937803200000001 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.685520+0000", 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2787000000000002e-05 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.685608+0000", 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7213999999999995e-05 2024-09-17T16:33:54.856 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27088.0:120213 7.3 e687/674 7:c59654a4:::benchmark_data_smithi173_116396_object24613:head v 687'6950, mlcod=686'6947)", 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.709364+0000", 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 115.10480967300001, 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28146429899999997, 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.857 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709364+0000", 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709364+0000", 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709365+0000", 2024-09-17T16:33:54.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8700000000000004e-07 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709373+0000", 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7330000000000003e-06 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989058+0000", 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.279684973 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989065+0000", 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1300000000000003e-06 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989111+0000", 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6569000000000001e-05 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989134+0000", 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2612e-05 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.990790+0000", 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001655807 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.990828+0000", 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8488000000000001e-05 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:120224 7.18 7:1c79604a:::benchmark_data_smithi173_116396_object24624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.730992+0000", 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 115.08318132700001, 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.274893622, 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 120224 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.730992+0000", 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.730992+0000", 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.730994+0000", 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8899999999999999e-06 2024-09-17T16:33:54.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731002+0000", 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.379e-06 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731015+0000", 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3892e-05 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731024+0000", 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9469999999999992e-06 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731315+0000", 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029021800000000002 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731451+0000", 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001359 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7,8", 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731569+0000", 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118727 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.751547+0000", 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019978124 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.751573+0000", 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5265e-05 2024-09-17T16:33:54.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989139+0000", 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23756648699999999 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.004357+0000", 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015217849 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.004409+0000", 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1891000000000002e-05 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.005886+0000", 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0014770530000000001 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(7.13 687/677 [PushReplyOp(7:cc214882:::benchmark_data_smithi173_116396_object24630:head)])", 2024-09-17T16:33:54.869 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.748767+0000", 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 115.065407032, 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.306875487, 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748767+0000", 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748767+0000", 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748768+0000", 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7499999999999998e-07 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748773+0000", 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4550000000000003e-06 2024-09-17T16:33:54.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.052878+0000", 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30410503 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.052883+0000", 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2650000000000004e-06 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.053064+0000", 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018071799999999999 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.055642+0000", 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0025780439999999998 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:120235 7.e 7:752ed815:::benchmark_data_smithi173_116396_object24635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-17T16:33:54.873 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.749714+0000", 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 115.064460118, 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27885106300000001, 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 120235 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749714+0000", 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749714+0000", 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749715+0000", 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-17T16:33:54.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749721+0000", 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9909999999999999e-06 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749732+0000", 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0806e-05 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749738+0000", 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.037e-06 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749792+0000", 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3906999999999997e-05 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749896+0000", 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103936 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749991+0000", 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4524000000000005e-05 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989005+0000", 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23901425800000001 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989092+0000", 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7128999999999999e-05 2024-09-17T16:33:54.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.028465+0000", 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.039372712999999997 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.028518+0000", 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3122999999999999e-05 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.028565+0000", 2024-09-17T16:33:54.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6808000000000003e-05 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:128523 7.e 7:78ef58c9:::benchmark_data_smithi173_116396_object32923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e693)", 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:16.133568+0000", 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 98.680605318999994, 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30194066800000002, 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:33:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 128523 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133568+0000", 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133568+0000", 2024-09-17T16:33:54.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133570+0000", 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5069999999999999e-06 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133576+0000", 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4320000000000004e-06 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133593+0000", 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6489000000000001e-05 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133602+0000", 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9889999999999998e-06 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.141144+0000", 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075426709999999999 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.141261+0000", 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011694299999999999 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.141386+0000", 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012499500000000001 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148436+0000", 2024-09-17T16:33:54.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070501180000000002 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.150261+0000", 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018242969999999999 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435410+0000", 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28514896200000001 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435468+0000", 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8149999999999997e-05 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435509+0000", 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1115000000000002e-05 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.886 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:128523 7.e e693/629 ondisk, result = 0)", 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 98.665328841000004, 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28666948399999997, 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3999999999999997e-07 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148847+0000", 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.575e-06 2024-09-17T16:33:54.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435339+0000", 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28649269199999999 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435344+0000", 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5809999999999998e-06 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435399+0000", 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4598999999999997e-05 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435405+0000", 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.049e-06 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435514+0000", 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000109648 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.890 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:130635 7.8 7:14a1b993:::benchmark_data_smithi173_116396_object35035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 94.681389695999997, 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.329086555, 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:33:54.891 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 130635 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132785+0000", 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5239999999999999e-06 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132792+0000", 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4860000000000002e-06 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132805+0000", 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2792999999999999e-05 2024-09-17T16:33:54.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132812+0000", 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1899999999999998e-06 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.133889+0000", 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001076881 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.134010+0000", 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120796 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.134124+0000", 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114875 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.136882+0000", 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027574539999999999 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.136893+0000", 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.154e-05 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461760+0000", 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32486675399999998 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461812+0000", 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2231000000000002e-05 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461870+0000", 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8031000000000001e-05 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:130635 7.8 e695/674 ondisk, result = 0)", 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:20.138684+0000", 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 94.675489814000002, 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32319844399999997, 2024-09-17T16:33:54.897 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138684+0000", 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138684+0000", 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138685+0000", 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138690+0000", 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4659999999999998e-06 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461660+0000", 2024-09-17T16:33:54.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.322969489 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461673+0000", 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3807e-05 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461745+0000", 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1297000000000005e-05 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461754+0000", 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0270000000000001e-06 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461882+0000", 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012858100000000001 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.901 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178141 7.e 7:7ed26a83:::benchmark_data_smithi173_116396_object82541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.064783+0000", 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.749390306999999, 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29001914499999998, 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178141 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064783+0000", 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064783+0000", 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064784+0000", 2024-09-17T16:33:54.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.128e-06 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064789+0000", 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.532e-06 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064800+0000", 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1229999999999999e-05 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064807+0000", 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0979999999999996e-06 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064882+0000", 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4942000000000002e-05 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.065014+0000", 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013223299999999999 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.065138+0000", 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012379200000000001 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.068993+0000", 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038547109999999998 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.069508+0000", 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051509400000000001 2024-09-17T16:33:54.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.354728+0000", 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28522005099999997 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.354766+0000", 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8275000000000002e-05 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.354802+0000", 2024-09-17T16:33:54.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6059000000000002e-05 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178146 7.8 7:14ce6dda:::benchmark_data_smithi173_116396_object82546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.072830+0000", 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.741343749999999, 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26665987099999999, 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.908 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:33:54.909 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:33:54.909 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:33:54.909 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178146 2024-09-17T16:33:54.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.909 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072830+0000", 2024-09-17T16:33:54.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072830+0000", 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072831+0000", 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7999999999999995e-07 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072834+0000", 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9140000000000001e-06 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072844+0000", 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0999999999999993e-06 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072848+0000", 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8250000000000004e-06 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.073968+0000", 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001119639 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.074037+0000", 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9041999999999994e-05 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.074098+0000", 2024-09-17T16:33:54.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0717999999999997e-05 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.075223+0000", 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001124702 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.075762+0000", 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00053989299999999995 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339392+0000", 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26362966900000001 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339433+0000", 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1085999999999999e-05 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339490+0000", 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6603e-05 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.914 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:178146 7.8 e717/674 ondisk, result = 0)", 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.076085+0000", 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.738088951999998, 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26341097200000002, 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.076085+0000", 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.076085+0000", 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.076085+0000", 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9299999999999999e-07 2024-09-17T16:33:54.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.076087+0000", 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9889999999999999e-06 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339303+0000", 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26321632499999997 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339310+0000", 2024-09-17T16:33:54.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8299999999999998e-06 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339380+0000", 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9505000000000006e-05 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339388+0000", 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339496+0000", 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010786 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27088.0:178656 7.17 e719/707 7:ecd8c6ba:::benchmark_data_smithi173_116396_object83056:head v 719'8589, mlcod=719'8586)", 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 16.781595945999999, 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27508470800000001, 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.919 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032584+0000", 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0349999999999998e-06 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305420+0000", 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27283574900000002 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305426+0000", 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9540000000000001e-06 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305471+0000", 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4836000000000002e-05 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305488+0000", 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7028e-05 2024-09-17T16:33:54.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.307628+0000", 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021396359999999999 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.307662+0000", 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4724999999999997e-05 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:189777 7.8 7:14f32b6c:::benchmark_data_smithi173_116396_object94177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:52.508349+0000", 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 2.3058245670000002, 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25456828199999998, 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 189777 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.924 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.508349+0000", 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.508349+0000", 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.508350+0000", 2024-09-17T16:33:54.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9999999999999997e-07 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.508353+0000", 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6229999999999999e-06 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.508365+0000", 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1885999999999999e-05 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.508372+0000", 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6390000000000003e-06 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.510037+0000", 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001664645 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.510151+0000", 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113979 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.510258+0000", 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010786 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.513135+0000", 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028765840000000002 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.513430+0000", 2024-09-17T16:33:54.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029507399999999999 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.762796+0000", 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.249365806 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.762850+0000", 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4466999999999999e-05 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:52.762917+0000", 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7019000000000003e-05 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27088.0:191193 7.c e724/629 7:3bfb41f2:::benchmark_data_smithi173_116396_object95593:head v 724'12078, mlcod=724'12075)", 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.451836+0000", 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.36233769599999999, 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.156618954, 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:33:54.930 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.451836+0000", 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.451836+0000", 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:33:54.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.451837+0000", 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1299999999999999e-07 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.451842+0000", 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6450000000000002e-06 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.603847+0000", 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.15200459999999999 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.603856+0000", 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1390000000000001e-06 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.603923+0000", 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6923999999999998e-05 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.603949+0000", 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6441e-05 2024-09-17T16:33:54.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.608410+0000", 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004460973 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.608455+0000", 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4518999999999998e-05 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:33:54.934 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:33:54.935 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:33:54.935 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:54.936 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:54 smithi173 ceph-mon[30530]: 7.9 scrub starts 2024-09-17T16:33:54.936 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:54 smithi173 ceph-mon[30530]: 7.9 scrub ok 2024-09-17T16:33:54.958 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:54.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:54.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:54.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 95597 objects 2024-09-17T16:33:54.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :164.044 2024-09-17T16:33:55.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:55.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:55.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:55.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:55.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:55.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:55.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:55.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:55.333 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:55.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:55.402 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:33:55.403 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T16:33:55.403 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:33:55.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:55.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:55.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:55.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:55 smithi173 ceph-mon[30530]: pgmap v1616: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 127 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 58 op/s; 16/6072 objects degraded (0.264%) 2024-09-17T16:33:55.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:55.572 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:55 smithi017 ceph-mon[27571]: pgmap v1616: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 127 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 58 op/s; 16/6072 objects degraded (0.264%) 2024-09-17T16:33:55.580 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:55.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:55.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:33:55.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:55.706 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:33:55.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:55.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:55 smithi071 ceph-mon[30002]: pgmap v1616: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 127 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 58 op/s; 16/6072 objects degraded (0.264%) 2024-09-17T16:33:55.832 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:55.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:55.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:55.944 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:33:55.957 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:55.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:56.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:56.084 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:56.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:56.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:56.137 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-17T16:33:56.138 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-17T16:33:56.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:56.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:56.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:56.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:33:56.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:56.309 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:56.335 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:56.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:56.437 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:56.460 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:56.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:56.477 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:33:56.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:56.585 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:56.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:56.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:56.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:56.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2425453092' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:33:56.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:56.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:56.810 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:56.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2425453092' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:33:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:56.838 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:56.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:56.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2425453092' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:33:56.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:56.961 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:56.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:57.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:57.089 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:57.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:57.155 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:33:57.155 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:33:57.155 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:33:57.155 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:33:57.191 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:57.216 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:57.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:57.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:57.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:57.375 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:57.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:57.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:57.477 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:57.500 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:57.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:57.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:57.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:57.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:57.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:57 smithi017 ceph-mon[27571]: pgmap v1617: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 77 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16/3663 objects degraded (0.437%) 2024-09-17T16:33:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:57 smithi017 ceph-mon[27571]: 7.4 scrub starts 2024-09-17T16:33:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:57 smithi017 ceph-mon[27571]: 7.4 scrub ok 2024-09-17T16:33:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:57 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:33:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2425453092' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:33:57.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:57 smithi017 ceph-mon[27571]: osdmap e725: 12 total, 12 up, 12 in 2024-09-17T16:33:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:57.750 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:57.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:57 smithi071 ceph-mon[30002]: pgmap v1617: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 77 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16/3663 objects degraded (0.437%) 2024-09-17T16:33:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:57 smithi071 ceph-mon[30002]: 7.4 scrub starts 2024-09-17T16:33:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:57 smithi071 ceph-mon[30002]: 7.4 scrub ok 2024-09-17T16:33:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:57 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:33:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2425453092' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:33:57.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:57 smithi071 ceph-mon[30002]: osdmap e725: 12 total, 12 up, 12 in 2024-09-17T16:33:57.842 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:57.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:57.875 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:57.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:57.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:57 smithi173 ceph-mon[30530]: pgmap v1617: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 77 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16/3663 objects degraded (0.437%) 2024-09-17T16:33:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:57 smithi173 ceph-mon[30530]: 7.4 scrub starts 2024-09-17T16:33:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:57 smithi173 ceph-mon[30530]: 7.4 scrub ok 2024-09-17T16:33:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:57 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:33:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2425453092' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:33:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:57 smithi173 ceph-mon[30530]: osdmap e725: 12 total, 12 up, 12 in 2024-09-17T16:33:57.977 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:33:57.999 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:58.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:58.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:33:58.101 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:58.122 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:58.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:58.224 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:33:58.247 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:58.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:33:58.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:58.374 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:58.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:58.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:33:58.477 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:33:58.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:58.503 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:58.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:33:58.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:58.627 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:58.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:58.647 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:58 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 16/396 objects degraded (4.040%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:33:58.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:58.752 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:58.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:58.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:58 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 16/396 objects degraded (4.040%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:33:58.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:58.876 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:58.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:58.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:58 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 16/396 objects degraded (4.040%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:33:58.944 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:58.944 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:33:58.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:33:59.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:59.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:33:59.102 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:33:59.128 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:59.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:33:59.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:33:59.252 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:59.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:33:59.354 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:33:59.380 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:59.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:33:59.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:33:59.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:59.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:33:59.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:33:59.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:33:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:59.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:33:59.728 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:33:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:59 smithi017 ceph-mon[27571]: pgmap v1619: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 8.6 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16/396 objects degraded (4.040%) 2024-09-17T16:33:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:33:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3126599949' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:59.745 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:33:59.746 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:33:59.753 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:33:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:59.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:33:59.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:33:59.768 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:33:59.789 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:33:59.790 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:33:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:59 smithi071 ceph-mon[30002]: pgmap v1619: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 8.6 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16/396 objects degraded (4.040%) 2024-09-17T16:33:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:33:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3126599949' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:59.827 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:33:59.827 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:33:59.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:33:59.851 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:33:59.851 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:33:59.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:33:59.877 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:33:59.877 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:33:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:33:59.883 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:33:59.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:33:59.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:59 smithi173 ceph-mon[30530]: pgmap v1619: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 8.6 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16/396 objects degraded (4.040%) 2024-09-17T16:33:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:33:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3126599949' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:33:59.899 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:33:59.900 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:33:59.915 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:33:59.915 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:33:59.926 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:33:59.926 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:33:59.958 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:33:59.958 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:33:59.979 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:33:59.979 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:33:59.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:00.000 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:34:00.001 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:34:00.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:00.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:00.022 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:00.023 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:00.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:00.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:00.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:00.231 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:00.258 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:00.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:00.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:00.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:00.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:00.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:00.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:00.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:00.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:00.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:00.630 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:00.677 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_5' removed 2024-09-17T16:34:00.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:00 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:34:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:00 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:34:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:00 smithi017 ceph-mon[27571]: 7.12 scrub starts 2024-09-17T16:34:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:00 smithi017 ceph-mon[27571]: 7.12 scrub ok 2024-09-17T16:34:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:00 smithi017 ceph-mon[27571]: 7.16 scrub starts 2024-09-17T16:34:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:00 smithi017 ceph-mon[27571]: 7.16 scrub ok 2024-09-17T16:34:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2780397607' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:34:00.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:00 smithi017 ceph-mon[27571]: pgmap v1620: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.5 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16/198 objects degraded (8.081%) 2024-09-17T16:34:00.754 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:00.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:00.856 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:00.881 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:00.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:00.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:01.004 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:01.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:01.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:00 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:34:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:00 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:34:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:00 smithi071 ceph-mon[30002]: 7.12 scrub starts 2024-09-17T16:34:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:00 smithi071 ceph-mon[30002]: 7.12 scrub ok 2024-09-17T16:34:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:00 smithi071 ceph-mon[30002]: 7.16 scrub starts 2024-09-17T16:34:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:00 smithi071 ceph-mon[30002]: 7.16 scrub ok 2024-09-17T16:34:01.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2780397607' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:34:01.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:00 smithi071 ceph-mon[30002]: pgmap v1620: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.5 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16/198 objects degraded (8.081%) 2024-09-17T16:34:01.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:01.131 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:01.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:00 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:34:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:00 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:34:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:00 smithi173 ceph-mon[30530]: 7.12 scrub starts 2024-09-17T16:34:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:00 smithi173 ceph-mon[30530]: 7.12 scrub ok 2024-09-17T16:34:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:00 smithi173 ceph-mon[30530]: 7.16 scrub starts 2024-09-17T16:34:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:00 smithi173 ceph-mon[30530]: 7.16 scrub ok 2024-09-17T16:34:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2780397607' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:34:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:00 smithi173 ceph-mon[30530]: pgmap v1620: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 26 active+clean; 4.5 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16/198 objects degraded (8.081%) 2024-09-17T16:34:01.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:01.254 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:01.255 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:01.356 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:01.378 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:01.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:01.433 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:34:01.434 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:34:01.434 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T16:34:01.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:34:01.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:01.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:01.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:01.604 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:01.629 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:01.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:01.690 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T16:34:01.691 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T16:34:01.691 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-09-17T16:34:01.692 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-09-17T16:34:01.730 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:01.755 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:01.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:01.857 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:01.880 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:01.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:01.982 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:01.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2780397607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:34:01.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:01 smithi017 ceph-mon[27571]: osdmap e726: 12 total, 12 up, 12 in 2024-09-17T16:34:02.008 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:02.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:02.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2780397607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:34:02.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:01 smithi071 ceph-mon[30002]: osdmap e726: 12 total, 12 up, 12 in 2024-09-17T16:34:02.110 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:02.133 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:02.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:02.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2780397607' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:34:02.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:01 smithi173 ceph-mon[30530]: osdmap e726: 12 total, 12 up, 12 in 2024-09-17T16:34:02.157 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:34:02.211 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:02.211 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:02.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:02.257 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:02.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:02.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:02.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:02.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:02.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:02.484 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:02.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:02.506 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:02.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:02.607 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:02.632 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:02.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:02.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:02.756 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:02.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:02.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:02.882 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:02.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:02.977 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:34:02.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:02 smithi017 ceph-mon[27571]: pgmap v1622: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:34:03.004 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:03.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:02 smithi071 ceph-mon[30002]: pgmap v1622: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:34:03.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:03.129 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:03.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:02 smithi173 ceph-mon[30530]: pgmap v1622: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:34:03.193 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:03.208 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:34:03.208 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:34:03.208 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 726, 2024-09-17T16:34:03.208 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:34:03.208 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:34:03.208 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:34:00.670117+0000", 2024-09-17T16:34:03.209 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:12:38.853431+0000", 2024-09-17T16:34:03.209 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:29:47.857157+0000", 2024-09-17T16:34:03.209 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:34:03.209 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5799936, 2024-09-17T16:34:03.209 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 7, 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:34:03.210 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-17T16:34:03.211 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:34:03.211 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:34:03.211 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:34:03.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.211 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:34:03.211 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:34:03.211 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:34:03.211 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:34:03.212 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:34:03.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:34:03.212 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:34:03.212 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:34:03.212 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:34:03.212 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:34:03.212 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:34:03.212 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:34:03.213 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:34:03.213 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:34:03.213 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:34:03.213 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:34:03.213 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:34:03.213 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:34:03.213 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:34:03.213 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:34:03.213 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "669", 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:34:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:34:03.215 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:34:03.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:34:03.217 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:34:03.218 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.219 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 718, 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:34:03.220 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:34:03.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.221 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.221 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:34:03.221 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:34:03.221 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.221 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:34:03.222 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.223 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.224 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.225 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 706, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.226 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.227 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.227 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.227 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.227 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:34:03.227 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:34:03.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.227 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:34:03.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:34:03.229 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.230 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:34:03.231 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:34:03.232 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 713, 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:34:03.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:34:03.234 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:34:03.235 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.236 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:34:03.237 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.238 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 702, 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:34:03.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.240 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.241 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.242 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:34:03.243 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.244 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 674, 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.245 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.246 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.247 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.248 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:34:03.249 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:03.250 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:34:03.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.250 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.250 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.251 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.4420013427734375, 2024-09-17T16:34:03.252 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:34:03.252 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.252 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.252 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:34:03.252 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 674, 2024-09-17T16:34:03.252 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.252 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.252 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.252 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.253 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.254 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.255 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.256 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.257 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.66790771484375, 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 719, 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:34:03.258 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:34:03.259 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:34:03.260 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.261 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:34:03.262 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.8061676025390625, 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.73492431640625, 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:34:03.263 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 718, 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.264 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.265 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:34:03.267 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:34:03.268 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:34:03.269 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 701, 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:34:03.270 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:34:03.271 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.272 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.273 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:34:03.274 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:34:03.275 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:34:03.275 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:34:03.275 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.275 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.275 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.275 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.275 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:34:03.275 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:34:03.276 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.276 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.276 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:34:03.276 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.2821197509765625, 2024-09-17T16:34:03.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 723, 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.277 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.278 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.279 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.280 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.281 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 717, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.282 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.285 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.286 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:34:03.287 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.588836669921875, 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 291, 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.288 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.289 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:34:03.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.292 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.293 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:01.084758+0000", 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.294 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:37.425764+0000", 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:16.538604+0000", 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:32:59.646034+0000", 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:34:03.296 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:37.701790+0000", 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.297 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:14.592129+0000", 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:12.692433+0000", 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:13.355540+0000", 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.299 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:23.445357+0000", 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.300 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:27.269836+0000", 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:14.888333+0000", 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:32:48.611806+0000", 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [ 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "1.0", 2024-09-17T16:34:03.302 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: 4, 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: 6, 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: 8 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [ 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "1.0", 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: "mappings": [ 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: "from": 6, 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: "to": 8 2024-09-17T16:34:03.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:34:03.304 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:34:03.305 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:34:03.306 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:03.307 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:34:03.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:03.308 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:03.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:03.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:03.431 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:03.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:03.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:03.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:03.555 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:03.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:03.578 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:03.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:03.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:03.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:03.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:03.782 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:03.803 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:03.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:03.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:03.928 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:03.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:03.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:03 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/198 objects degraded (8.081%), 1 pg degraded) 2024-09-17T16:34:03.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:03 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:34:03.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/155792197' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:34:03.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:34:03.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:34:03.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:34:03.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:34:03.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:34:03.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3797881218' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:04.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:04.052 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:04.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:03 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/198 objects degraded (8.081%), 1 pg degraded) 2024-09-17T16:34:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:03 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:34:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/155792197' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:34:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:34:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:34:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:34:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:34:04.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:34:04.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3797881218' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:03 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 16/198 objects degraded (8.081%), 1 pg degraded) 2024-09-17T16:34:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:03 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:34:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/155792197' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:34:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:34:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:34:04.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:34:04.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:34:04.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:34:04.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3797881218' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:04.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:04.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:04.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:04.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:04.285 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:34:04.307 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:04.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:04.410 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:04.434 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:04.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:04.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:04.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:04.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:04.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:04.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:04.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:04.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:04.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:04.792 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:04.792 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:04.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:04.816 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:04.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:04.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:04.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:04.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:04.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:04 smithi017 ceph-mon[27571]: pgmap v1623: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:04 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3797881218' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:34:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:04 smithi017 ceph-mon[27571]: osdmap e727: 12 total, 12 up, 12 in 2024-09-17T16:34:05.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:04 smithi071 ceph-mon[30002]: pgmap v1623: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:04 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3797881218' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:34:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:04 smithi071 ceph-mon[30002]: osdmap e727: 12 total, 12 up, 12 in 2024-09-17T16:34:05.068 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:05.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:04 smithi173 ceph-mon[30530]: pgmap v1623: 1 pgs: 1 active+clean; 577 KiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:04 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3797881218' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:34:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:04 smithi173 ceph-mon[30530]: osdmap e727: 12 total, 12 up, 12 in 2024-09-17T16:34:05.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:05.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:05.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:05.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:05.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:05.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:05.426 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:05.451 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:05.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:05.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:34:05.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:05.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:05.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:05.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:05.701 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_6' created 2024-09-17T16:34:05.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:05.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:05.787 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:05.812 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:05.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4120050288' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-17T16:34:05.836 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:05.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:05.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:05.960 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:05.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4120050288' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-17T16:34:06.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:06.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:06.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:06.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4120050288' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-17T16:34:06.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:06.214 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:06.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:06.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:06.317 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:06.340 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:06.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:06.351 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:06.351 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:06.409 INFO:tasks.thrashosds.thrasher:j is {'epoch': 726, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:34:00.670117+0000', 'last_up_change': '2024-09-17T16:12:38.853431+0000', 'last_in_change': '2024-09-17T16:29:47.857157+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '669', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 718, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 706, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 713, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 702, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 674, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 0.4420013427734375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 674, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 0.66790771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 719, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 0.8061676025390625, 'primary_affinity': 0.73492431640625, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 718, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 701, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0.2821197509765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 723, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 717, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 0.588836669921875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:01.084758+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:37.425764+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:16.538604+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:32:59.646034+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:37.701790+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:14.592129+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:12.692433+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:13.355540+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:23.445357+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:27.269836+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:14.888333+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:32:48.611806+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [4, 6, 8]}], 'pg_upmap_items': [{'pgid': '1.0', 'mappings': [{'from': 6, 'to': 8}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:34:06.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:34:06.410 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:06.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:06.475 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:06.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:06.576 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:06.601 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:06.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:06.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:06.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4120050288' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-17T16:34:06.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:06 smithi017 ceph-mon[27571]: osdmap e728: 12 total, 12 up, 12 in 2024-09-17T16:34:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:06 smithi017 ceph-mon[27571]: pgmap v1626: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:06 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:34:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:06 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:34:06.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:06.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:06.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:06.859 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:06.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:06.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:06.983 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:06.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4120050288' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-17T16:34:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:06 smithi071 ceph-mon[30002]: osdmap e728: 12 total, 12 up, 12 in 2024-09-17T16:34:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:06 smithi071 ceph-mon[30002]: pgmap v1626: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:06 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:34:07.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:06 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:34:07.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:07.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:07.107 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:07.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4120050288' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-17T16:34:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:06 smithi173 ceph-mon[30530]: osdmap e728: 12 total, 12 up, 12 in 2024-09-17T16:34:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:06 smithi173 ceph-mon[30530]: pgmap v1626: 17 pgs: 4 creating+peering, 12 unknown, 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:06 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:34:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:06 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:34:07.173 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:34:07.209 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:07.232 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:07.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:07.333 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:07.356 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:07.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:07.425 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:07.457 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:07.480 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:07.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:07.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:07.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:07.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:07.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-09-17T16:34:07.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:07.732 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:07.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:07.834 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:07.856 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:07.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:07.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:07.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:07 smithi017 ceph-mon[27571]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:34:07.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:07 smithi017 ceph-mon[27571]: osdmap e729: 12 total, 12 up, 12 in 2024-09-17T16:34:07.919 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:34:07.919 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:34:07.943 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:34:07.943 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:34:07.947 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:07.947 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:07.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:07.971 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:34:07.971 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:34:07.982 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:07.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:08.002 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:34:08.002 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:34:08.010 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:08.025 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:34:08.026 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:34:08.039 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:08.049 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:34:08.049 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:34:08.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:07 smithi071 ceph-mon[30002]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:34:08.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:07 smithi071 ceph-mon[30002]: osdmap e729: 12 total, 12 up, 12 in 2024-09-17T16:34:08.071 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:34:08.071 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:34:08.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:08.092 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:34:08.092 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:34:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:08.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:08.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:08.114 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:34:08.115 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:34:08.136 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:34:08.136 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:34:08.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:07 smithi173 ceph-mon[30530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:34:08.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:07 smithi173 ceph-mon[30530]: osdmap e729: 12 total, 12 up, 12 in 2024-09-17T16:34:08.158 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:34:08.158 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:34:08.181 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:08.181 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:08.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:08.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:08.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:08.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:08.362 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:08.364 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:08.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:08.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:08.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:08.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:08.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:08.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:08.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:34:08.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:08.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:08.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:08.840 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:08.862 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:08.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:08.873 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:08.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:34:08.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:08 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:34:08.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:08 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:34:08.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:08 smithi017 ceph-mon[27571]: osdmap e730: 12 total, 12 up, 12 in 2024-09-17T16:34:08.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2395911165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:08.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:08 smithi017 ceph-mon[27571]: pgmap v1629: 17 pgs: 4 creating+peering, 5 unknown, 8 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:08.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:08.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:08.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:08.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:08.993 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:08 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:34:08.993 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:08 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:34:08.993 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:08 smithi071 ceph-mon[30002]: osdmap e730: 12 total, 12 up, 12 in 2024-09-17T16:34:08.993 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2395911165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:08.993 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:08 smithi071 ceph-mon[30002]: pgmap v1629: 17 pgs: 4 creating+peering, 5 unknown, 8 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:09.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:08 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:34:09.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:08 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:34:09.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:08 smithi173 ceph-mon[30530]: osdmap e730: 12 total, 12 up, 12 in 2024-09-17T16:34:09.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2395911165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:09.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:08 smithi173 ceph-mon[30530]: pgmap v1629: 17 pgs: 4 creating+peering, 5 unknown, 8 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:09.089 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:09.111 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:09.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:09.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:09.213 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:09.236 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:09.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:09.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:09.359 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:09.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:09.371 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:34:09.371 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:34:09.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:34:09.371 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:34:09.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.371 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:17635 6.e 6.b9e5631e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:34:09.371 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:25:22.014210+0000", 2024-09-17T16:34:09.371 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 527.33614965100003, 2024-09-17T16:34:09.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.974852477, 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 17635 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.014210+0000", 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.014210+0000", 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.014212+0000", 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8530000000000001e-06 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.014272+0000", 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.033e-05 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.014288+0000", 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5577999999999999e-05 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.014297+0000", 2024-09-17T16:34:09.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0499999999999997e-06 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.014359+0000", 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2069000000000005e-05 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.014367+0000", 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7470000000000005e-06 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.989002+0000", 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.97463560100000002 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.989063+0000", 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0248999999999999e-05 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:166371 6.e 6:7d7d0369:::benchmark_data_smithi173_102017_object73223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:28:48.730674+0000", 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 320.61968558799998, 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52497791100000002, 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 166371 2024-09-17T16:34:09.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730674+0000", 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730674+0000", 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730676+0000", 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3850000000000001e-06 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730680+0000", 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3630000000000004e-06 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730691+0000", 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0937999999999999e-05 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730696+0000", 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1030000000000001e-06 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730733+0000", 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.714e-05 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730828+0000", 2024-09-17T16:34:09.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5094999999999997e-05 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,2", 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730904+0000", 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5702000000000001e-05 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.734707+0000", 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038030859999999998 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.735615+0000", 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00090786100000000004 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255546+0000", 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51993075499999997 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255605+0000", 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9404000000000002e-05 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255652+0000", 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7079000000000001e-05 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26665.0:166371 6.e e608/549 ondisk, result = 0)", 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:28:48.736512+0000", 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 320.61384763199999, 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51914618000000001, 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.383 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.736512+0000", 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.736512+0000", 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.736513+0000", 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9699999999999995e-07 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.736520+0000", 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0899999999999999e-06 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255442+0000", 2024-09-17T16:34:09.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51892181699999995 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255467+0000", 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5539000000000001e-05 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255527+0000", 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0044999999999999e-05 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255537+0000", 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5670000000000001e-06 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255658+0000", 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012132499999999999 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:95587 7.3 e671/640 7:c3123e23:::benchmark_data_smithi173_116396_object95586:head v 671'5966, mlcod=671'5964)", 2024-09-17T16:34:09.387 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:31:09.928995+0000", 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 179.42136516900001, 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.84880420199999995, 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:09.928995+0000", 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:09.928995+0000", 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:09.928996+0000", 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3799999999999996e-07 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:09.929044+0000", 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8328000000000002e-05 2024-09-17T16:34:09.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.775185+0000", 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.84614102099999999 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.775196+0000", 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0978999999999999e-05 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.775293+0000", 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6892000000000007e-05 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.775312+0000", 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9633e-05 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.777762+0000", 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024497569999999999 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.777799+0000", 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6653999999999998e-05 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:104052 7.1 7:823e012a:::benchmark_data_smithi173_116396_object8452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e678)", 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:31:25.992683+0000", 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 163.35767644500001, 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25560972900000001, 2024-09-17T16:34:09.392 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 104052 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:25.992683+0000", 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:25.992683+0000", 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:25.992685+0000", 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.491e-06 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:25.992691+0000", 2024-09-17T16:34:09.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4010000000000002e-06 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:25.992706+0000", 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4698999999999999e-05 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:25.992713+0000", 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5830000000000003e-06 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:25.992773+0000", 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0804999999999998e-05 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:25.992852+0000", 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8561999999999998e-05 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:25.992919+0000", 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6895000000000002e-05 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:25.994308+0000", 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013889600000000001 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:26.120074+0000", 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.125766709 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:26.248233+0000", 2024-09-17T16:34:09.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12815817700000001 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:26.248263+0000", 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9899e-05 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:26.248293+0000", 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0548e-05 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:119650 7.8 e686/674 7:17d95773:::benchmark_data_smithi173_116396_object24050:head v 686'6840, mlcod=685'6837)", 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:31:58.391430+0000", 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 130.95893000300001, 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.293510663, 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:58.391430+0000", 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:58.391430+0000", 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:58.391431+0000", 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.116e-06 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:58.391439+0000", 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9889999999999992e-06 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:58.683302+0000", 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29186353399999998 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:58.683307+0000", 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2459999999999997e-06 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:58.683355+0000", 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7978e-05 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:58.683376+0000", 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1506999999999999e-05 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:58.684896+0000", 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015194799999999999 2024-09-17T16:34:09.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:58.684940+0000", 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4812999999999999e-05 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:122592 7.3 e689/674 7:c667c471:::benchmark_data_smithi173_116396_object26992:head v 689'7021, mlcod=689'7019)", 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:04.943205+0000", 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 124.407155101, 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.161202225, 2024-09-17T16:34:09.403 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:04.943205+0000", 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:04.943205+0000", 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:04.943206+0000", 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1680000000000001e-06 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:04.943214+0000", 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.038e-06 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:05.102329+0000", 2024-09-17T16:34:09.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15911540900000001 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:05.102336+0000", 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8299999999999998e-06 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:05.102394+0000", 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7935e-05 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:05.102414+0000", 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9990999999999999e-05 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:05.104360+0000", 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001945773 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:05.104407+0000", 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7080999999999999e-05 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.407 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:123304 7.1 7:853adcb2:::benchmark_data_smithi173_116396_object27704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:06.139275+0000", 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 123.211084744, 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.149611296, 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:34:09.408 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 123304 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.139275+0000", 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.139275+0000", 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.139276+0000", 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2920000000000001e-06 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.139281+0000", 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.101e-06 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.139293+0000", 2024-09-17T16:34:09.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1545e-05 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.139315+0000", 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2123000000000001e-05 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.139367+0000", 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1415000000000002e-05 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.139491+0000", 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000124516 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.139622+0000", 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013085399999999999 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.141288+0000", 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001666388 2024-09-17T16:34:09.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.141863+0000", 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057514100000000004 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.288821+0000", 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.14695767800000001 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.288857+0000", 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5942000000000002e-05 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:06.288886+0000", 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9300999999999999e-05 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:124754 7.4 e690/674 7:248c4b03:::benchmark_data_smithi173_116396_object29154:head v 690'7149, mlcod=690'7146)", 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:08.917005+0000", 2024-09-17T16:34:09.414 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 120.433354868, 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19922224999999999, 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:08.917005+0000", 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:08.917005+0000", 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:08.917006+0000", 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6199999999999996e-07 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:08.917012+0000", 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4080000000000003e-06 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:09.114489+0000", 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.197476706 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:09.114511+0000", 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2323e-05 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:09.114578+0000", 2024-09-17T16:34:09.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6670999999999999e-05 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:09.114594+0000", 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6342999999999999e-05 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:09.116204+0000", 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016096859999999999 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:09.116227+0000", 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3251000000000002e-05 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:126622 7.1 7:81ccf7d0:::benchmark_data_smithi173_116396_object31022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e691)", 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:12.638165+0000", 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 116.712194586, 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.198864605, 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.419 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 126622 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.638165+0000", 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.638165+0000", 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.638167+0000", 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.255e-06 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.638177+0000", 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3789999999999995e-06 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.638193+0000", 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6285000000000001e-05 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.638200+0000", 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1099999999999997e-06 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.638252+0000", 2024-09-17T16:34:09.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1564000000000003e-05 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.638338+0000", 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6508000000000006e-05 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.638407+0000", 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8460999999999994e-05 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.640129+0000", 2024-09-17T16:34:09.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017219399999999999 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.641141+0000", 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010120210000000001 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.836951+0000", 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.195810399 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.836992+0000", 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0710999999999997e-05 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.837030+0000", 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7972000000000002e-05 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:126622 7.1 e691/674 ondisk, result = 0)", 2024-09-17T16:34:09.425 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:12.640432+0000", 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 116.709927339, 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19660245500000001, 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.640432+0000", 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.640432+0000", 2024-09-17T16:34:09.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.640433+0000", 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1999999999999998e-07 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.640438+0000", 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6110000000000004e-06 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.836828+0000", 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19639040399999999 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.836847+0000", 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8839999999999999e-05 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.836938+0000", 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1085e-05 2024-09-17T16:34:09.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.836946+0000", 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9400000000000002e-06 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:12.837035+0000", 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8654999999999997e-05 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.429 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:128521 7.3 e693/674 7:c7e1af8e:::benchmark_data_smithi173_116396_object32921:head v 693'7197, mlcod=693'7194)", 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:16.133648+0000", 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 113.216711503, 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.153210226, 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:16.133648+0000", 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:16.133648+0000", 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:16.133649+0000", 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.68e-07 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:16.133654+0000", 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2780000000000001e-06 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:16.285028+0000", 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15137368700000001 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:16.285053+0000", 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4788000000000001e-05 2024-09-17T16:34:09.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:16.285166+0000", 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000113001 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:16.285185+0000", 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9303000000000001e-05 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:16.286823+0000", 2024-09-17T16:34:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016377270000000001 2024-09-17T16:34:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:16.286858+0000", 2024-09-17T16:34:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5874000000000002e-05 2024-09-17T16:34:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.434 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:130925 7.1 7:87629740:::benchmark_data_smithi173_116396_object35325:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:20.674823+0000", 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 108.67553707499999, 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.177278506, 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:09.435 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 130925 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.674823+0000", 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.674823+0000", 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.674824+0000", 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.637e-06 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.674832+0000", 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1720000000000001e-06 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.674848+0000", 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6524999999999998e-05 2024-09-17T16:34:09.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.674858+0000", 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6819999999999998e-06 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.674927+0000", 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9193e-05 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.675021+0000", 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4543999999999995e-05 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.675107+0000", 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5784000000000004e-05 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.676816+0000", 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017088419999999999 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.681465+0000", 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0046489569999999996 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.851997+0000", 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17053226699999999 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.852055+0000", 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7213999999999998e-05 2024-09-17T16:34:09.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:20.852101+0000", 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6689e-05 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:131199 7.4 e695/674 7:217096b0:::benchmark_data_smithi173_116396_object35599:head v 695'7369, mlcod=695'7366)", 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:21.175409+0000", 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 108.17495042500001, 2024-09-17T16:34:09.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15546232800000001, 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:21.175409+0000", 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:21.175409+0000", 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:21.175411+0000", 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.207e-06 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:21.175419+0000", 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8720000000000008e-06 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:21.328771+0000", 2024-09-17T16:34:09.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.153351974 2024-09-17T16:34:09.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:21.328776+0000", 2024-09-17T16:34:09.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0329999999999999e-06 2024-09-17T16:34:09.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.445 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:21.328829+0000", 2024-09-17T16:34:09.445 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2441000000000002e-05 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:21.328847+0000", 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8603000000000001e-05 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:21.330842+0000", 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00199497 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.446 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:21.330872+0000", 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9227999999999999e-05 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:132692 7.1 7:81ba0d1f:::benchmark_data_smithi173_116396_object37092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:23.891089+0000", 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 105.45927059500001, 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26999491599999997, 2024-09-17T16:34:09.447 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 132692 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.891089+0000", 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.891089+0000", 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.891090+0000", 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0400000000000005e-07 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.891095+0000", 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7269999999999998e-06 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.891105+0000", 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0127e-05 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.891113+0000", 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.276e-06 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.891168+0000", 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5064e-05 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.891242+0000", 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3999999999999996e-05 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,10", 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.891304+0000", 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1772000000000004e-05 2024-09-17T16:34:09.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.892405+0000", 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011014390000000001 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:23.913622+0000", 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.021216373 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:09.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:24.160973+0000", 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.247351606 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:24.161033+0000", 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9704999999999998e-05 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:24.161084+0000", 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0923e-05 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.453 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:134413 7.4 e695/674 7:24726582:::benchmark_data_smithi173_116396_object38813:head v 695'7489, mlcod=695'7486)", 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:26.891895+0000", 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 102.45846482499999, 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20019772599999999, 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:26.891895+0000", 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:26.891895+0000", 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:26.891896+0000", 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2399999999999998e-07 2024-09-17T16:34:09.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:26.891901+0000", 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1139999999999997e-06 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.090263+0000", 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19836251399999999 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.090268+0000", 2024-09-17T16:34:09.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6789999999999996e-06 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.090332+0000", 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4268000000000002e-05 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.090348+0000", 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5954e-05 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.457 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.092051+0000", 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017025849999999999 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.092093+0000", 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1987999999999998e-05 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.458 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:134497 7.8 e695/674 7:12978d11:::benchmark_data_smithi173_116396_object38897:head v 695'7294, mlcod=695'7291)", 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:27.105577+0000", 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 102.244783265, 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17173216799999999, 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.105577+0000", 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.105577+0000", 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.105578+0000", 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1510000000000001e-06 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.105585+0000", 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2429999999999999e-06 2024-09-17T16:34:09.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.275557+0000", 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.169971712 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.275562+0000", 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3979999999999998e-06 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.275610+0000", 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8015e-05 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.275628+0000", 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8170000000000001e-05 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.277269+0000", 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001640615 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:27.277309+0000", 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9864000000000003e-05 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:171623 7.3 e712/674 7:c2274c33:::benchmark_data_smithi173_116396_object76023:head v 712'8534, mlcod=712'8530)", 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:27.596367+0000", 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 41.753992457999999, 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.158898703, 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.463 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:27.596367+0000", 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:27.596367+0000", 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:27.596368+0000", 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7700000000000005e-07 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:27.596371+0000", 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5370000000000002e-06 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:27.753054+0000", 2024-09-17T16:34:09.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15668257899999999 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:27.753059+0000", 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6129999999999997e-06 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:27.753106+0000", 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7639999999999998e-05 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:27.753120+0000", 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3569e-05 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:27.755238+0000", 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021182699999999998 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.467 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:27.755266+0000", 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8017999999999998e-05 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:173444 7.4 e714/674 7:209d45ce:::benchmark_data_smithi173_116396_object77844:head v 714'8713, mlcod=714'8710)", 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:30.075819+0000", 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 39.274540938000001, 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15867936299999999, 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.468 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.075819+0000", 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.075819+0000", 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.469 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.075820+0000", 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8400000000000002e-07 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.075827+0000", 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2230000000000001e-06 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.232382+0000", 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15655455800000001 2024-09-17T16:34:09.470 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.232386+0000", 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4000000000000002e-06 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.232433+0000", 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6826000000000002e-05 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.232452+0000", 2024-09-17T16:34:09.471 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9491e-05 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.234473+0000", 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020210359999999999 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.234498+0000", 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4845e-05 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.472 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:173615 7.4 e714/674 7:20d32a86:::benchmark_data_smithi173_116396_object78015:head v 714'8721, mlcod=714'8718)", 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:30.475593+0000", 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.874766858000001, 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22243810999999999, 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:09.473 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.475593+0000", 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.475593+0000", 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.475594+0000", 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.031e-06 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.474 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.475601+0000", 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1960000000000002e-06 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.694635+0000", 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21903430199999999 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.694642+0000", 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2140000000000001e-06 2024-09-17T16:34:09.475 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.694693+0000", 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1298000000000003e-05 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.694711+0000", 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7986e-05 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.697986+0000", 2024-09-17T16:34:09.476 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0032754590000000001 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:30.698031+0000", 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4623999999999998e-05 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:09.477 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:34:09.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:09.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:09.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:09.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:09.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:09.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:09.720 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-09-17T16:34:09.730 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:09.753 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:09.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:09.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:09.878 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:09.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:09.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:09.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/196825283' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:34:10.002 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:10.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:10.028 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:34:10.029 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:34:10.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/196825283' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:34:10.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:10.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:10.128 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:10.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:10.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/196825283' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:34:10.229 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:10.251 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:10.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:10.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:10.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:34:10.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:10.375 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:10.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:10.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:10.500 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:10.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:10.526 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:10.603 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:10.625 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:10.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:10.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:10.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:10.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:10.750 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:10.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:10.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:10.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:10.879 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:10.913 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/196825283' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:34:10.914 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:10 smithi071 ceph-mon[30002]: osdmap e731: 12 total, 12 up, 12 in 2024-09-17T16:34:10.914 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:10 smithi071 ceph-mon[30002]: from='client.17874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:34:10.914 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:10 smithi071 ceph-mon[30002]: pgmap v1631: 17 pgs: 4 creating+peering, 1 unknown, 12 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:10.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/196825283' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:34:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:10 smithi017 ceph-mon[27571]: osdmap e731: 12 total, 12 up, 12 in 2024-09-17T16:34:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:10 smithi017 ceph-mon[27571]: from='client.17874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:34:10.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:10 smithi017 ceph-mon[27571]: pgmap v1631: 17 pgs: 4 creating+peering, 1 unknown, 12 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:11.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:11.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:11.012 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:34:11.013 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:34:11.013 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:34:11.013 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:34:11.108 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:11.133 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:11.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:11.137 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1630,"stamp":"2024-09-17T16:34:09.719176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":7232500,"kb_used_data":733280,"kb_used_omap":58,"kb_used_meta":6498373,"kb_avail":1117709324,"statfs":{"total":1151940427776,"available":1144534347776,"internally_reserved":0,"allocated":750878720,"data_stored":747586405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":6654334115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.435185"},"pg_stats":[{"pgid":"8.f","version":"0'0","reported_seq":11,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-17T16:34:06.715857+0000","last_change":"2024-09-17T16:34:06.715555+0000","last_active":"2024-09-17T16:34:06.715857+0000","last_peered":"2024-09-17T16:34:06.715857+0000","last_clean":"2024-09-17T16:34:06.715857+0000","last_became_active":"2024-09-17T16:34:06.715239+0000","last_became_peered":"2024-09-17T16:34:06.715239+0000","last_unstale":"2024-09-17T16:34:06.715857+0000","last_undegraded":"2024-09-17T16:34:06.715857+0000","last_fullsized":"2024-09-17T16:34:06.715857+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:24.473561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,1],"acting":[2,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":11,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-17T16:34:06.729304+0000","last_change":"2024-09-17T16:34:06.728922+0000","last_active":"2024-09-17T16:34:06.729304+0000","last_peered":"2024-09-17T16:34:06.729304+0000","last_clean":"2024-09-17T16:34:06.729304+0000","last_became_active":"2024-09-17T16:34:06.728580+0000","last_became_peered":"2024-09-17T16:34:06.728580+0000","last_unstale":"2024-09-17T16:34:06.729304+0000","last_undegraded":"2024-09-17T16:34:06.729304+0000","last_fullsized":"2024-09-17T16:34:06.729304+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:19.809198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":11,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-17T16:34:06.715577+0000","last_change":"2024-09-17T16:34:06.715248+0000","last_active":"2024-09-17T16:34:06.715577+0000","last_peered":"2024-09-17T16:34:06.715577+0000","last_clean":"2024-09-17T16:34:06.715577+0000","last_became_active":"2024-09-17T16:34:06.714898+0000","last_became_peered":"2024-09-17T16:34:06.714898+0000","last_unstale":"2024-09-17T16:34:06.715577+0000","last_undegraded":"2024-09-17T16:34:06.715577+0000","last_fullsized":"2024-09-17T16:34:06.715577+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:15.263453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,2],"acting":[7,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-17T16:34:07.711561+0000","last_change":"2024-09-17T16:34:07.122358+0000","last_active":"2024-09-17T16:34:07.711561+0000","last_peered":"2024-09-17T16:34:07.711561+0000","last_clean":"2024-09-17T16:34:07.711561+0000","last_became_active":"2024-09-17T16:34:07.121994+0000","last_became_peered":"2024-09-17T16:34:07.121994+0000","last_unstale":"2024-09-17T16:34:07.711561+0000","last_undegraded":"2024-09-17T16:34:07.711561+0000","last_fullsized":"2024-09-17T16:34:07.711561+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:17.715303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":14,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-17T16:34:07.711551+0000","last_change":"2024-09-17T16:34:06.729072+0000","last_active":"2024-09-17T16:34:07.711551+0000","last_peered":"2024-09-17T16:34:07.711551+0000","last_clean":"2024-09-17T16:34:07.711551+0000","last_became_active":"2024-09-17T16:34:06.728735+0000","last_became_peered":"2024-09-17T16:34:06.728735+0000","last_unstale":"2024-09-17T16:34:07.711551+0000","last_undegraded":"2024-09-17T16:34:07.711551+0000","last_fullsized":"2024-09-17T16:34:07.711551+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:07.417365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-09-17T16:34:07.711569+0000","last_change":"2024-09-17T16:34:06.711362+0000","last_active":"2024-09-17T16:34:07.711569+0000","last_peered":"2024-09-17T16:34:07.711569+0000","last_clean":"2024-09-17T16:34:07.711569+0000","last_became_active":"2024-09-17T16:34:06.711038+0000","last_became_peered":"2024-09-17T16:34:06.711038+0000","last_unstale":"2024-09-17T16:34:07.711569+0000","last_undegraded":"2024-09-17T16:34:07.711569+0000","last_fullsized":"2024-09-17T16:34:07.711569+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:20.824604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,3],"acting":[8,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-17T16:34:07.122771+0000","last_change":"2024-09-17T16:34:07.122363+0000","last_active":"2024-09-17T16:34:07.122771+0000","last_peered":"2024-09-17T16:34:07.122771+0000","last_clean":"2024-09-17T16:34:07.122771+0000","last_became_active":"2024-09-17T16:34:07.121989+0000","last_became_peered":"2024-09-17T16:34:07.121989+0000","last_unstale":"2024-09-17T16:34:07.122771+0000","last_undegraded":"2024-09-17T16:34:07.122771+0000","last_fullsized":"2024-09-17T16:34:07.122771+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:12.969717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-17T16:34:06.715593+0000","last_change":"2024-09-17T16:34:06.715060+0000","last_active":"2024-09-17T16:34:06.715593+0000","last_peered":"2024-09-17T16:34:06.715593+0000","last_clean":"2024-09-17T16:34:06.715593+0000","last_became_active":"2024-09-17T16:34:06.714697+0000","last_became_peered":"2024-09-17T16:34:06.714697+0000","last_unstale":"2024-09-17T16:34:06.715593+0000","last_undegraded":"2024-09-17T16:34:06.715593+0000","last_fullsized":"2024-09-17T16:34:06.715593+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:06.036868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":728,"state":"creating+peering","last_fresh":"2024-09-17T16:34:05.707111+0000","last_change":"2024-09-17T16:34:05.707111+0000","last_active":"2024-09-17T16:34:05.693761+0000","last_peered":"2024-09-17T16:34:05.693761+0000","last_clean":"2024-09-17T16:34:05.693761+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:34:05.707111+0000","last_undegraded":"2024-09-17T16:34:05.707111+0000","last_fullsized":"2024-09-17T16:34:05.707111+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":728,"state":"creating+peering","last_fresh":"2024-09-17T16:34:05.706804+0000","last_change":"2024-09-17T16:34:05.706804+0000","last_active":"2024-09-17T16:34:05.693761+0000","last_peered":"2024-09-17T16:34:05.693761+0000","last_clean":"2024-09-17T16:34:05.693761+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:34:05.706804+0000","last_undegraded":"2024-09-17T16:34:05.706804+0000","last_fullsized":"2024-09-17T16:34:05.706804+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:34:05.693761+0000","last_change":"2024-09-17T16:34:05.693761+0000","last_active":"2024-09-17T16:34:05.693761+0000","last_peered":"2024-09-17T16:34:05.693761+0000","last_clean":"2024-09-17T16:34:05.693761+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:34:05.693761+0000","last_undegraded":"2024-09-17T16:34:05.693761+0000","last_fullsized":"2024-09-17T16:34:05.693761+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-17T16:34:06.715375+0000","last_change":"2024-09-17T16:34:06.714936+0000","last_active":"2024-09-17T16:34:06.715375+0000","last_peered":"2024-09-17T16:34:06.715375+0000","last_clean":"2024-09-17T16:34:06.715375+0000","last_became_active":"2024-09-17T16:34:06.714658+0000","last_became_peered":"2024-09-17T16:34:06.714658+0000","last_unstale":"2024-09-17T16:34:06.715375+0000","last_undegraded":"2024-09-17T16:34:06.715375+0000","last_fullsized":"2024-09-17T16:34:06.715375+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:11.487481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,6],"acting":[1,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":728,"state":"creating+peering","last_fresh":"2024-09-17T16:34:05.706906+0000","last_change":"2024-09-17T16:34:05.706906+0000","last_active":"2024-09-17T16:34:05.693761+0000","last_peered":"2024-09-17T16:34:05.693761+0000","last_clean":"2024-09-17T16:34:05.693761+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:34:05.706906+0000","last_undegraded":"2024-09-17T16:34:05.706906+0000","last_fullsized":"2024-09-17T16:34:05.706906+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-17T16:34:06.712578+0000","last_change":"2024-09-17T16:34:06.711356+0000","last_active":"2024-09-17T16:34:06.712578+0000","last_peered":"2024-09-17T16:34:06.712578+0000","last_clean":"2024-09-17T16:34:06.712578+0000","last_became_active":"2024-09-17T16:34:06.711027+0000","last_became_peered":"2024-09-17T16:34:06.711027+0000","last_unstale":"2024-09-17T16:34:06.712578+0000","last_undegraded":"2024-09-17T16:34:06.712578+0000","last_fullsized":"2024-09-17T16:34:06.712578+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:17.335212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,10],"acting":[1,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-09-17T16:34:07.122640+0000","last_change":"2024-09-17T16:34:07.122236+0000","last_active":"2024-09-17T16:34:07.122640+0000","last_peered":"2024-09-17T16:34:07.122640+0000","last_clean":"2024-09-17T16:34:07.122640+0000","last_became_active":"2024-09-17T16:34:07.121932+0000","last_became_peered":"2024-09-17T16:34:07.121932+0000","last_unstale":"2024-09-17T16:34:07.122640+0000","last_undegraded":"2024-09-17T16:34:07.122640+0000","last_fullsized":"2024-09-17T16:34:07.122640+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:18.812245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,4],"acting":[3,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":728,"state":"creating+peering","last_fresh":"2024-09-17T16:34:05.706836+0000","last_change":"2024-09-17T16:34:05.706836+0000","last_active":"2024-09-17T16:34:05.693761+0000","last_peered":"2024-09-17T16:34:05.693761+0000","last_clean":"2024-09-17T16:34:05.693761+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:34:05.706836+0000","last_undegraded":"2024-09-17T16:34:05.706836+0000","last_fullsized":"2024-09-17T16:34:05.706836+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3],"acting":[10,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":225,"reported_epoch":727,"state":"active+clean","last_fresh":"2024-09-17T16:34:04.287493+0000","last_change":"2024-09-17T16:33:59.364895+0000","last_active":"2024-09-17T16:34:04.287493+0000","last_peered":"2024-09-17T16:34:04.287493+0000","last_clean":"2024-09-17T16:34:04.287493+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:34:04.287493+0000","last_undegraded":"2024-09-17T16:34:04.287493+0000","last_fullsized":"2024-09-17T16:34:04.287493+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:13.261070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":45,"acting":45,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":607248,"kb_used_data":129020,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93137904,"statfs":{"total":95995035648,"available":95373213696,"internally_reserved":0,"allocated":132116480,"data_stored":131823575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":489683396},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547500,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":804380,"kb_used_data":285896,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92940772,"statfs":{"total":95995035648,"available":95171350528,"internally_reserved":0,"allocated":292757504,"data_stored":292459768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":530905536},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":677548,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93067604,"statfs":{"total":95995035648,"available":95301226496,"internally_reserved":0,"allocated":671744,"data_stored":417942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":692976064},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005367,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":465540,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93279612,"statfs":{"total":95995035648,"available":95518322688,"internally_reserved":0,"allocated":1228800,"data_stored":962939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963843,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638008,"kb_used_data":43804,"kb_used_omap":41,"kb_used_meta":594006,"kb_avail":93107144,"statfs":{"total":95995035648,"available":95341715456,"internally_reserved":0,"allocated":44855296,"data_stored":44597109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":608262451},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":510040,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235112,"statfs":{"total":95995035648,"available":95472754688,"internally_reserved":0,"allocated":1314816,"data_stored":1018379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":535900,"kb_used_data":3528,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93209252,"statfs":{"total":95995035648,"available":95446274048,"internally_reserved":0,"allocated":3612672,"data_stored":3314936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":545126854},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986507,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":469800,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93275352,"statfs":{"total":95995035648,"available":95513960448,"internally_reserved":0,"allocated":1327104,"data_stored":1025816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150030,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":935928,"kb_used_data":264604,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92809224,"statfs":{"total":95995035648,"available":95036645376,"internally_reserved":0,"allocated":270954496,"data_stored":270696309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":687274439},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491376,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93253776,"statfs":{"total":95995035648,"available":95491866624,"internally_reserved":0,"allocated":684032,"data_stored":425845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575194,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":388656,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93356496,"statfs":{"total":95995035648,"available":95597051904,"internally_reserved":0,"allocated":684032,"data_stored":425845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708076,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93037076,"statfs":{"total":95995035648,"available":95269965824,"internally_reserved":0,"allocated":671744,"data_stored":417942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:34:11.138 INFO:tasks.thrashosds.thrasher:No pools; doing nothing 2024-09-17T16:34:11.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/196825283' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:34:11.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:10 smithi173 ceph-mon[30530]: osdmap e731: 12 total, 12 up, 12 in 2024-09-17T16:34:11.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:10 smithi173 ceph-mon[30530]: from='client.17874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:34:11.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:10 smithi173 ceph-mon[30530]: pgmap v1631: 17 pgs: 4 creating+peering, 1 unknown, 12 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:11.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:11.238 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:11.257 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:11.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:11.359 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:11.382 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:11.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:11.478 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:34:11.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:11.508 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-09-17T16:34:11.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:11.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:11.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:11.512 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T16:34:11.512 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T16:34:11.612 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-09-17T16:34:11.612 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T16:34:11.612 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_130566 2024-09-17T16:34:11.612 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:11.613 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:34:11.613 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-09-17T16:34:11.647 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:11.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:11.696 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:34:11.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:11.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:11.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:11.834 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:11.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:11.901 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:11.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:11.952 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:11.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:11 smithi017 ceph-mon[27571]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:34:12.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:12.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:12.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:12.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:12.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:11 smithi071 ceph-mon[30002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:34:12.137 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:12.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:11 smithi173 ceph-mon[30530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:34:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:12.170 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:12.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:12.271 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:12.294 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:12.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:12.396 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:12.422 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:12.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:12.437 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:34:12.437 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:34:12.458 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:34:12.458 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:34:12.478 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:34:12.479 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:34:12.488 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:34:12.488 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:34:12.488 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:34:12.488 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:34:12.488 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:34:12.499 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:34:12.499 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:34:12.520 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:34:12.520 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:34:12.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:12.547 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:34:12.548 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:34:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:12.549 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:12.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:12.563 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:34:12.563 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:34:12.584 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:34:12.584 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:34:12.608 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:34:12.608 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:34:12.612 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 1483 1467 91.6825 91.6875 0.0748675 0.0104761 2024-09-17T16:34:12.636 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:34:12.636 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:34:12.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:12.664 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:34:12.664 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:34:12.682 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:12.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:12.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:12.696 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:12.736 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:34:12.784 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:12.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:12.839 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:12.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:12.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/186891680' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:12.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:12 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:12.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:12 smithi017 ceph-mon[27571]: pgmap v1632: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:12.929 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2072690800' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:12.940 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:12.966 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:12.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/186891680' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:12 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:12 smithi071 ceph-mon[30002]: pgmap v1632: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:13.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2072690800' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:13.068 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:13.092 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:13.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:13.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:34:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/186891680' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:12 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:12 smithi173 ceph-mon[30530]: pgmap v1632: 17 pgs: 17 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2072690800' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:13.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:13.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:13.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:13.317 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:13.343 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:13.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:13.359 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:13.446 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:13.468 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:13.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:13.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:13.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:13.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:13.612 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 3119 3103 96.9622 102.25 0.0095446 0.0102552 2024-09-17T16:34:13.697 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:13.734 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:13.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:13.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:13.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:13.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:13.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:13.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:13.943 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:13.943 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:13.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:13 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:13 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:13.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:13 smithi017 ceph-mon[27571]: osdmap e732: 12 total, 12 up, 12 in 2024-09-17T16:34:13.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:13.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:14.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:13 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:14.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:13 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:14.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:13 smithi071 ceph-mon[30002]: osdmap e732: 12 total, 12 up, 12 in 2024-09-17T16:34:14.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:14.120 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:14.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:14.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:13 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:14.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:13 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:14.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:13 smithi173 ceph-mon[30530]: osdmap e732: 12 total, 12 up, 12 in 2024-09-17T16:34:14.223 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:14.247 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:14.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:14.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:14.375 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:14.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:14.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:14.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:14.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:14.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:14.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:34:14.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:14.612 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 5024 5008 104.326 119.062 0.0017473 0.0095346 2024-09-17T16:34:14.658 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:14.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:14.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:14.786 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:14.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:14 smithi071 ceph-mon[30002]: pgmap v1634: 17 pgs: 17 active+clean; 44 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s wr, 109 op/s 2024-09-17T16:34:14.876 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:14.887 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:14.912 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:14.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:14.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:15.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:15.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:15.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:15.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:14 smithi173 ceph-mon[30530]: pgmap v1634: 17 pgs: 17 active+clean; 44 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s wr, 109 op/s 2024-09-17T16:34:15.145 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:15.170 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:15.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:15.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:14 smithi017 ceph-mon[27571]: pgmap v1634: 17 pgs: 17 active+clean; 44 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s wr, 109 op/s 2024-09-17T16:34:15.271 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:15.292 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:15.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:15.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:15.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:15.420 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:15.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:15.524 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:15.524 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:15.543 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:15.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:15.613 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 6731 6715 104.914 106.688 0.00157562 0.00944578 2024-09-17T16:34:15.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:15.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:15.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:15.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:34:15.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:15.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:15.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:15.899 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:15.920 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:15.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:16.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:16.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:16.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:16.088 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:16.139 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:34:16.140 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:34:16.140 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_6 2024-09-17T16:34:16.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:34:16.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:16.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:16.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:16.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:34:16.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:16.311 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:16.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:16.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:16.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:16.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:16.482 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:16.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:34:16.536 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:16.597 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:16.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:16.613 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 7936 7920 98.9925 75.3125 0.0629691 0.00986424 2024-09-17T16:34:16.660 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:34:16.660 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:34:16.686 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:16.688 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:34:16.689 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:34:16.698 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:16.715 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:34:16.715 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:34:16.728 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:16.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:16.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:16.739 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:34:16.740 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:34:16.761 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:34:16.761 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:34:16.787 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:34:16.788 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:34:16.811 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:34:16.811 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:34:16.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:16.832 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:34:16.832 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:34:16.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:16.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:16.853 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:34:16.854 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:34:16.877 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:34:16.877 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:34:16.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:34:16.898 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:34:16.919 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:16.919 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:16.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:16.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:16.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:16.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:17.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:17.101 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:17.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:17.117 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:34:17.117 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:34:17.117 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:34:17.117 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:34:17.117 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:34:17.203 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:17.227 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:17.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:17 smithi071 ceph-mon[30002]: pgmap v1635: 17 pgs: 17 active+clean; 235 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 468 op/s 2024-09-17T16:34:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3216829367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:17.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:17.350 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:17.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:17 smithi173 ceph-mon[30530]: pgmap v1635: 17 pgs: 17 active+clean; 235 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 468 op/s 2024-09-17T16:34:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3216829367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:17.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:17.474 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:17.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:17 smithi017 ceph-mon[27571]: pgmap v1635: 17 pgs: 17 active+clean; 235 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 468 op/s 2024-09-17T16:34:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3216829367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:17.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:17.599 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:17.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:17.613 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 8040 8024 83.5767 6.5 0.00145464 0.0104186 2024-09-17T16:34:17.701 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:17.724 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:17.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:17.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:34:17.826 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:17.848 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:17.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:17.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:17.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:17.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:17.991 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:18.077 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:18.101 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:18.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:18.135 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:18.203 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:34:18.217 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:34:18.225 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:18.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:18.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:18.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:18.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:18.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:18.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:18 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:34:18.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:18 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:34:18.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:18 smithi017 ceph-mon[27571]: osdmap e733: 12 total, 12 up, 12 in 2024-09-17T16:34:18.392 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:18 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:34:18.392 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:18 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:34:18.392 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:18 smithi071 ceph-mon[30002]: osdmap e733: 12 total, 12 up, 12 in 2024-09-17T16:34:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:18 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:34:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:18 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:34:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:18 smithi173 ceph-mon[30530]: osdmap e733: 12 total, 12 up, 12 in 2024-09-17T16:34:18.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:18.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:18.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:18.491 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:34:18.491 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:34:18.491 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:34:18.491 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:34:18.491 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.491 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:17638 6.a 6:5f824b0d:::benchmark_data_smithi173_102017_object17637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:34:18.491 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:25:22.116190+0000", 2024-09-17T16:34:18.491 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 536.37164890999998, 2024-09-17T16:34:18.491 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.402092703, 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 17638 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116190+0000", 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.492 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116190+0000", 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116191+0000", 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8599999999999998e-07 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.493 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116256+0000", 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4834999999999998e-05 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116264+0000", 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1329999999999999e-06 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116268+0000", 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2370000000000003e-06 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.494 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116296+0000", 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7438000000000001e-05 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116363+0000", 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7587000000000002e-05 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116436+0000", 2024-09-17T16:34:18.495 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2534999999999996e-05 2024-09-17T16:34:18.496 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.496 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.496 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.496 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.116929+0000", 2024-09-17T16:34:18.496 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049371199999999999 2024-09-17T16:34:18.496 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.496 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.496 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.117182+0000", 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000252141 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518211+0000", 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40102907199999999 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518251+0000", 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0295e-05 2024-09-17T16:34:18.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518283+0000", 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2032000000000001e-05 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26665.0:17638 6.a e538/527 ondisk, result = 0)", 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:25:22.117495+0000", 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 536.370343961, 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.400793913, 2024-09-17T16:34:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.499 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.499 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:34:18.499 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.499 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.499 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.499 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.117495+0000", 2024-09-17T16:34:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.117495+0000", 2024-09-17T16:34:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.117496+0000", 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4000000000000002e-07 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.117498+0000", 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8899999999999999e-06 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518017+0000", 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40051810799999998 2024-09-17T16:34:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518030+0000", 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3507e-05 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518173+0000", 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000142781 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518190+0000", 2024-09-17T16:34:18.502 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7368999999999999e-05 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:22.518289+0000", 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8818000000000002e-05 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:27647 6.a 6:52a81611:::benchmark_data_smithi173_102017_object27646:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:25:32.709430+0000", 2024-09-17T16:34:18.503 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 525.77840905100004, 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43289498700000001, 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 27647 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.504 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709430+0000", 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709430+0000", 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709432+0000", 2024-09-17T16:34:18.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5209999999999999e-06 2024-09-17T16:34:18.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709489+0000", 2024-09-17T16:34:18.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7837000000000002e-05 2024-09-17T16:34:18.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709506+0000", 2024-09-17T16:34:18.507 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6500000000000001e-05 2024-09-17T16:34:18.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.507 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.507 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709514+0000", 2024-09-17T16:34:18.507 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1899999999999995e-06 2024-09-17T16:34:18.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.508 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.508 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709575+0000", 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0844000000000002e-05 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709702+0000", 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012716499999999999 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.709798+0000", 2024-09-17T16:34:18.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5816000000000006e-05 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.710756+0000", 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00095821100000000005 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:32.968078+0000", 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25732213199999998 2024-09-17T16:34:18.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:33.142260+0000", 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.174181488 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:33.142294+0000", 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3816999999999998e-05 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:25:33.142325+0000", 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1466e-05 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:90558 6.a 6:50ab933e:::benchmark_data_smithi173_102017_object90557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected+supports_pool_eio e569)", 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:35.632133+0000", 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 462.855706043, 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36392078100000003, 2024-09-17T16:34:18.512 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 90558 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632133+0000", 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632133+0000", 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632135+0000", 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9769999999999999e-06 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632286+0000", 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000151342 2024-09-17T16:34:18.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632296+0000", 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6190000000000006e-06 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632303+0000", 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3690000000000001e-06 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632359+0000", 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5949000000000003e-05 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632372+0000", 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2429000000000001e-05 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.995955+0000", 2024-09-17T16:34:18.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36358281799999997 2024-09-17T16:34:18.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.996054+0000", 2024-09-17T16:34:18.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9277999999999994e-05 2024-09-17T16:34:18.517 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.517 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.517 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.518 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:90589 6.a 6:5bef2469:::benchmark_data_smithi173_102017_object90588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=5 ondisk+retry+write+known_if_redirected+supports_pool_eio e569)", 2024-09-17T16:34:18.518 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:35.632315+0000", 2024-09-17T16:34:18.518 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 462.85552397399999, 2024-09-17T16:34:18.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36740646999999999, 2024-09-17T16:34:18.518 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.518 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.518 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 90589 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632315+0000", 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632315+0000", 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632315+0000", 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2599999999999998e-07 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632389+0000", 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3570000000000002e-05 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632395+0000", 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2700000000000001e-06 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632399+0000", 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6940000000000001e-06 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632434+0000", 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5315999999999999e-05 2024-09-17T16:34:18.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632444+0000", 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7329999999999998e-06 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.998953+0000", 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.366508534 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.999722+0000", 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00076902700000000004 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:90598 6.a 6:5bf8700d:::benchmark_data_smithi173_102017_object90597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e569)", 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:35.632407+0000", 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 462.85543200400002, 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41091824900000001, 2024-09-17T16:34:18.523 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.524 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.524 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.524 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.524 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:18.524 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:18.524 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 90598 2024-09-17T16:34:18.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.524 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632407+0000", 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632407+0000", 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632407+0000", 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9999999999999999e-07 2024-09-17T16:34:18.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632464+0000", 2024-09-17T16:34:18.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6597e-05 2024-09-17T16:34:18.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632475+0000", 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.148e-05 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632479+0000", 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5200000000000002e-06 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632525+0000", 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5634999999999999e-05 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632530+0000", 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8109999999999999e-06 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.999777+0000", 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36724700100000002 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.999920+0000", 2024-09-17T16:34:18.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014235800000000001 2024-09-17T16:34:18.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:34:18.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.000074+0000", 2024-09-17T16:34:18.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015448400000000001 2024-09-17T16:34:18.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.011276+0000", 2024-09-17T16:34:18.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.011202118 2024-09-17T16:34:18.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.020160+0000", 2024-09-17T16:34:18.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0088838210000000001 2024-09-17T16:34:18.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.031139+0000", 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.010979067 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.031193+0000", 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.414e-05 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.043325+0000", 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.012131917000000001 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.531 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:90615 6.a 6:5a02cdcb:::benchmark_data_smithi173_102017_object90614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e569)", 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:35.632496+0000", 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 462.85534298099998, 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41453878900000002, 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:18.532 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 90615 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632496+0000", 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632496+0000", 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632496+0000", 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5999999999999999e-07 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632535+0000", 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8442000000000002e-05 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632540+0000", 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4210000000000003e-06 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632544+0000", 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.191e-06 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632575+0000", 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1727000000000003e-05 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632579+0000", 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7620000000000001e-06 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.002809+0000", 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37022986099999999 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.002947+0000", 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000137634 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.003094+0000", 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014770300000000001 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.017229+0000", 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.014134346000000001 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.043397+0000", 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.026168338999999999 2024-09-17T16:34:18.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.043501+0000", 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010389 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.043561+0000", 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0266000000000002e-05 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.047035+0000", 2024-09-17T16:34:18.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.003473847 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:90617 6.a 6:53fd4cb5:::benchmark_data_smithi173_102017_object90616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e569)", 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:35.632559+0000", 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 462.855280091, 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.418534719, 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:18.540 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:18.540 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 90617 2024-09-17T16:34:18.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.540 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632559+0000", 2024-09-17T16:34:18.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632559+0000", 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632559+0000", 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8999999999999998e-07 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632590+0000", 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0321e-05 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632595+0000", 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3469999999999998e-06 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632607+0000", 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1922e-05 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632630+0000", 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2827999999999999e-05 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632633+0000", 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6490000000000001e-06 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.005939+0000", 2024-09-17T16:34:18.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37330520699999997 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.006071+0000", 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013222700000000001 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.006226+0000", 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000155661 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.017234+0000", 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.011007551000000001 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.047110+0000", 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.029876296 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.047173+0000", 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2626999999999995e-05 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.047242+0000", 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9442000000000004e-05 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.051094+0000", 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038513509999999998 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.546 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:90622 6.a 6:546719eb:::benchmark_data_smithi173_102017_object90621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e569)", 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:35.632613+0000", 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 462.85522619800003, 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.42286261800000002, 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 90622 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632613+0000", 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632613+0000", 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632613+0000", 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1899999999999998e-07 2024-09-17T16:34:18.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632642+0000", 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8818000000000001e-05 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632647+0000", 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2129999999999999e-06 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632650+0000", 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.948e-06 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632679+0000", 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8411e-05 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632682+0000", 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5559999999999999e-06 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.008837+0000", 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37615481099999998 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.008933+0000", 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6272000000000003e-05 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.009028+0000", 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4365999999999997e-05 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.031097+0000", 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.022069247 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.051157+0000", 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.020060607000000001 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.051225+0000", 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7426000000000001e-05 2024-09-17T16:34:18.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.051289+0000", 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3905000000000006e-05 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.055476+0000", 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0041867190000000002 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:90631 6.a 6:56f536a5:::benchmark_data_smithi173_102017_object90630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e569)", 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:26:35.632664+0000", 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 462.85517540299998, 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.42675627599999999, 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 90631 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632664+0000", 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632664+0000", 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632664+0000", 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8999999999999998e-07 2024-09-17T16:34:18.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632683+0000", 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8851999999999998e-05 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632687+0000", 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1219999999999997e-06 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632692+0000", 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7500000000000003e-06 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632713+0000", 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1498e-05 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:35.632717+0000", 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7290000000000002e-06 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.014680+0000", 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38196269999999999 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.014796+0000", 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011678100000000001 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.014920+0000", 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000123435 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.043344+0000", 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.028424615 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.055547+0000", 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.01220278 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.055620+0000", 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2731000000000005e-05 2024-09-17T16:34:18.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.055678+0000", 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8276000000000002e-05 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:26:36.059420+0000", 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0037417169999999999 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:152017 6.a 6:5f28e5f6:::benchmark_data_smithi173_102017_object58869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e602)", 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:28:24.975823+0000", 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 353.51201652600002, 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56153204199999995, 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152017 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.561 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975823+0000", 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975823+0000", 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975824+0000", 2024-09-17T16:34:18.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6640000000000001e-06 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975830+0000", 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7450000000000001e-06 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975843+0000", 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3191e-05 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975849+0000", 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.251e-06 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975897+0000", 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8052e-05 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.976015+0000", 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011795599999999999 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.976112+0000", 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6572999999999998e-05 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.979946+0000", 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038342089999999999 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.980159+0000", 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00021301499999999999 2024-09-17T16:34:18.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534848+0000", 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55468829500000005 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534888+0000", 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0905999999999998e-05 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.537355+0000", 2024-09-17T16:34:18.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024661850000000001 2024-09-17T16:34:18.567 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.567 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.567 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.567 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26665.0:152017 6.a e602/568 ondisk, result = 0)", 2024-09-17T16:34:18.567 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:28:24.982375+0000", 2024-09-17T16:34:18.567 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 353.50546376, 2024-09-17T16:34:18.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55498762499999998, 2024-09-17T16:34:18.568 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.568 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.568 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:34:18.568 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.982375+0000", 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.982375+0000", 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.982376+0000", 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6400000000000002e-07 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.982380+0000", 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1389999999999997e-06 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534775+0000", 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55239448899999999 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534784+0000", 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9109999999999999e-06 2024-09-17T16:34:18.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534835+0000", 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1328000000000001e-05 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534843+0000", 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9289999999999998e-06 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.537363+0000", 2024-09-17T16:34:18.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002520165 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:170216 7.11 7:89b9c9a3:::benchmark_data_smithi173_116396_object74616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 52.948860557000003, 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.387146404, 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.572 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170216 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:34:18.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538980+0000", 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3030000000000001e-06 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538986+0000", 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6579999999999999e-06 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538998+0000", 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2903e-05 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539004+0000", 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9540000000000001e-06 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539051+0000", 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7067999999999997e-05 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539144+0000", 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2521000000000004e-05 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539219+0000", 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4791999999999998e-05 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541123+0000", 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019039650000000001 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541689+0000", 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00056619600000000004 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925945+0000", 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38425619700000002 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926009+0000", 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3516999999999995e-05 2024-09-17T16:34:18.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926125+0000", 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011633000000000001 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:170216 7.11 e712/703 ondisk, result = 0)", 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 52.946465754000002, 2024-09-17T16:34:18.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38475827200000001, 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541374+0000", 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541379+0000", 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6210000000000003e-06 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.923614+0000", 2024-09-17T16:34:18.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38223536499999999 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.923618+0000", 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7629999999999998e-06 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925930+0000", 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023126259999999999 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925939+0000", 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0070000000000003e-06 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926132+0000", 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019219300000000001 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:170289 7.11 7:8ff6a0b8:::benchmark_data_smithi173_116396_object74689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:34:18.582 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 52.863531940999998, 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30203754100000002, 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170289 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:34:18.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624309+0000", 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.136e-06 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624319+0000", 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9170000000000001e-06 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624348+0000", 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8368999999999999e-05 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624362+0000", 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4222999999999999e-05 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624491+0000", 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000129262 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624698+0000", 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000206546 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624844+0000", 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014652400000000001 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.626781+0000", 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019372759999999999 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627901+0000", 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011196260000000001 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926178+0000", 2024-09-17T16:34:18.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29827702699999997 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926231+0000", 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2892000000000001e-05 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926345+0000", 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000113743 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:170289 7.11 e712/703 ondisk, result = 0)", 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 52.860827778999997, 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29933906100000002, 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:34:18.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627012+0000", 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627016+0000", 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9809999999999997e-06 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925949+0000", 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29893330699999998 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925958+0000", 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926164+0000", 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000205942 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926171+0000", 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9500000000000001e-06 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926350+0000", 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017893299999999999 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.592 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDelete(7.19 e717,677 7:9da3d41f:::benchmark_data_smithi173_116396_object4781:head,673'6297)", 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 42.311901597000002, 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37136110999999999, 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:34:18.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175947+0000", 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3859999999999996e-06 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175959+0000", 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2269999999999999e-05 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547126+0000", 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37116651099999998 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547136+0000", 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0241000000000001e-05 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547216+0000", 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.051e-05 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547227+0000", 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0356999999999999e-05 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547299+0000", 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1835000000000006e-05 2024-09-17T16:34:18.596 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:191071 7.11 7:882b150b:::benchmark_data_smithi173_116396_object95471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:54.152057+0000", 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 24.335782012999999, 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25846097200000001, 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:34:18.597 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 191071 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.152057+0000", 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.152057+0000", 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.152058+0000", 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8899999999999998e-07 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.152062+0000", 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9990000000000002e-06 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.152070+0000", 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0700000000000007e-06 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.152074+0000", 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0230000000000003e-06 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.152123+0000", 2024-09-17T16:34:18.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8446999999999999e-05 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.152248+0000", 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012496799999999999 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.152363+0000", 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011497000000000001 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.154609+0000", 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022460919999999999 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.155107+0000", 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049845900000000003 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.410452+0000", 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25534515800000002 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.410487+0000", 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4248999999999997e-05 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.410518+0000", 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1547999999999997e-05 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.603 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:191171 7.11 7:8d50061b:::benchmark_data_smithi173_116396_object95571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 24.057273464000001, 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44239869199999998, 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 191171 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430568+0000", 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.897e-06 2024-09-17T16:34:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430578+0000", 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0149999999999999e-05 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430616+0000", 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8753e-05 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430622+0000", 2024-09-17T16:34:18.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3569999999999997e-06 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430666+0000", 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3831000000000001e-05 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430740+0000", 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4865999999999998e-05 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430817+0000", 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6490999999999997e-05 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433451+0000", 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0026335489999999998 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433803+0000", 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00035236799999999998 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872847+0000", 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43904401599999998 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872921+0000", 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3867000000000003e-05 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872964+0000", 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3547000000000002e-05 2024-09-17T16:34:18.609 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:191171 7.11 e724/703 ondisk, result = 0)", 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 24.054245233, 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43937563000000002, 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433595+0000", 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2900000000000002e-07 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433602+0000", 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9550000000000003e-06 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872622+0000", 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43902024000000001 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872639+0000", 2024-09-17T16:34:18.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7516e-05 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872820+0000", 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000180714 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872837+0000", 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7099999999999999e-05 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872970+0000", 2024-09-17T16:34:18.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013227599999999999 2024-09-17T16:34:18.614 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.614 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.614 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.614 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:34:18.614 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:34:18.614 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:34:18.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:18.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 8040 8024 71.6365 0 - 0.0104186 2024-09-17T16:34:18.637 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:18.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:18.739 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:18.760 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:18.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:18.862 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:18.886 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:18.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:18.987 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:19.010 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:19.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:19.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:19.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:19.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:19.216 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2024-09-17T16:34:18.063475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":442696224,"num_objects":6748,"num_object_clones":0,"num_object_copies":20244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6748,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":6869,"num_write_kb":433894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6838,"ondisk_log_size":6838,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8343072,"kb_used_data":1813912,"kb_used_omap":58,"kb_used_meta":6528901,"kb_avail":1116598752,"statfs":{"total":1151940427776,"available":1143397122048,"internally_reserved":0,"allocated":1857445888,"data_stored":1854183920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":6685594787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":442105856,"num_objects":6746,"num_object_clones":0,"num_object_copies":20238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6746,"num_write_kb":431744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001539"},"pg_stats":[{"pgid":"8.f","version":"733'523","reported_seq":544,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-17T16:34:17.403054+0000","last_change":"2024-09-17T16:34:06.715555+0000","last_active":"2024-09-17T16:34:17.403054+0000","last_peered":"2024-09-17T16:34:17.403054+0000","last_clean":"2024-09-17T16:34:17.403054+0000","last_became_active":"2024-09-17T16:34:06.715239+0000","last_became_peered":"2024-09-17T16:34:06.715239+0000","last_unstale":"2024-09-17T16:34:17.403054+0000","last_undegraded":"2024-09-17T16:34:17.403054+0000","last_fullsized":"2024-09-17T16:34:17.403054+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":523,"log_dups_size":0,"ondisk_log_size":523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:22.567660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34275328,"num_objects":523,"num_object_clones":0,"num_object_copies":1569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":523,"num_write_kb":33472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,1],"acting":[2,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"732'490","reported_seq":511,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-17T16:34:17.117420+0000","last_change":"2024-09-17T16:34:06.728922+0000","last_active":"2024-09-17T16:34:17.117420+0000","last_peered":"2024-09-17T16:34:17.117420+0000","last_clean":"2024-09-17T16:34:17.117420+0000","last_became_active":"2024-09-17T16:34:06.728580+0000","last_became_peered":"2024-09-17T16:34:06.728580+0000","last_unstale":"2024-09-17T16:34:17.117420+0000","last_undegraded":"2024-09-17T16:34:17.117420+0000","last_fullsized":"2024-09-17T16:34:17.117420+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:08.763600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32112640,"num_objects":490,"num_object_clones":0,"num_object_copies":1470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":490,"num_write_kb":31360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"732'181","reported_seq":199,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:13.075005+0000","last_change":"2024-09-17T16:34:06.715248+0000","last_active":"2024-09-17T16:34:13.075005+0000","last_peered":"2024-09-17T16:34:13.075005+0000","last_clean":"2024-09-17T16:34:13.075005+0000","last_became_active":"2024-09-17T16:34:06.714898+0000","last_became_peered":"2024-09-17T16:34:06.714898+0000","last_unstale":"2024-09-17T16:34:13.075005+0000","last_undegraded":"2024-09-17T16:34:13.075005+0000","last_fullsized":"2024-09-17T16:34:13.075005+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:24.841155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11862016,"num_objects":181,"num_object_clones":0,"num_object_copies":543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":181,"num_write_kb":11584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,2],"acting":[7,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"732'302","reported_seq":321,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:14.524551+0000","last_change":"2024-09-17T16:34:07.122358+0000","last_active":"2024-09-17T16:34:14.524551+0000","last_peered":"2024-09-17T16:34:14.524551+0000","last_clean":"2024-09-17T16:34:14.524551+0000","last_became_active":"2024-09-17T16:34:07.121994+0000","last_became_peered":"2024-09-17T16:34:07.121994+0000","last_unstale":"2024-09-17T16:34:14.524551+0000","last_undegraded":"2024-09-17T16:34:14.524551+0000","last_fullsized":"2024-09-17T16:34:14.524551+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:35.023476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19791872,"num_objects":302,"num_object_clones":0,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":302,"num_write_kb":19328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.b","version":"732'305","reported_seq":323,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:14.528551+0000","last_change":"2024-09-17T16:34:06.729072+0000","last_active":"2024-09-17T16:34:14.528551+0000","last_peered":"2024-09-17T16:34:14.528551+0000","last_clean":"2024-09-17T16:34:14.528551+0000","last_became_active":"2024-09-17T16:34:06.728735+0000","last_became_peered":"2024-09-17T16:34:06.728735+0000","last_unstale":"2024-09-17T16:34:14.528551+0000","last_undegraded":"2024-09-17T16:34:14.528551+0000","last_fullsized":"2024-09-17T16:34:14.528551+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:33.954619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19988480,"num_objects":305,"num_object_clones":0,"num_object_copies":915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":305,"num_write_kb":19520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.a","version":"732'306","reported_seq":325,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:14.535422+0000","last_change":"2024-09-17T16:34:06.711362+0000","last_active":"2024-09-17T16:34:14.535422+0000","last_peered":"2024-09-17T16:34:14.535422+0000","last_clean":"2024-09-17T16:34:14.535422+0000","last_became_active":"2024-09-17T16:34:06.711038+0000","last_became_peered":"2024-09-17T16:34:06.711038+0000","last_unstale":"2024-09-17T16:34:14.535422+0000","last_undegraded":"2024-09-17T16:34:14.535422+0000","last_fullsized":"2024-09-17T16:34:14.535422+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:29.726045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20054016,"num_objects":306,"num_object_clones":0,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":19584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,3],"acting":[8,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.8","version":"732'501","reported_seq":520,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:16.807186+0000","last_change":"2024-09-17T16:34:07.122363+0000","last_active":"2024-09-17T16:34:16.807186+0000","last_peered":"2024-09-17T16:34:16.807186+0000","last_clean":"2024-09-17T16:34:16.807186+0000","last_became_active":"2024-09-17T16:34:07.121989+0000","last_became_peered":"2024-09-17T16:34:07.121989+0000","last_unstale":"2024-09-17T16:34:16.807186+0000","last_undegraded":"2024-09-17T16:34:16.807186+0000","last_fullsized":"2024-09-17T16:34:16.807186+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":501,"log_dups_size":0,"ondisk_log_size":501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:22.783188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":501,"num_object_clones":0,"num_object_copies":1503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":501,"num_write_kb":32064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.7","version":"733'506","reported_seq":527,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-17T16:34:17.403260+0000","last_change":"2024-09-17T16:34:06.715060+0000","last_active":"2024-09-17T16:34:17.403260+0000","last_peered":"2024-09-17T16:34:17.403260+0000","last_clean":"2024-09-17T16:34:17.403260+0000","last_became_active":"2024-09-17T16:34:06.714697+0000","last_became_peered":"2024-09-17T16:34:06.714697+0000","last_unstale":"2024-09-17T16:34:17.403260+0000","last_undegraded":"2024-09-17T16:34:17.403260+0000","last_fullsized":"2024-09-17T16:34:17.403260+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:09.964072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33161216,"num_objects":506,"num_object_clones":0,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":506,"num_write_kb":32384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"732'400","reported_seq":418,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:15.751799+0000","last_change":"2024-09-17T16:34:06.712763+0000","last_active":"2024-09-17T16:34:15.751799+0000","last_peered":"2024-09-17T16:34:15.751799+0000","last_clean":"2024-09-17T16:34:15.751799+0000","last_became_active":"2024-09-17T16:34:06.712218+0000","last_became_peered":"2024-09-17T16:34:06.712218+0000","last_unstale":"2024-09-17T16:34:15.751799+0000","last_undegraded":"2024-09-17T16:34:15.751799+0000","last_fullsized":"2024-09-17T16:34:15.751799+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:18.607388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26214400,"num_objects":400,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":400,"num_write_kb":25600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.5","version":"732'438","reported_seq":457,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:15.763121+0000","last_change":"2024-09-17T16:34:06.730847+0000","last_active":"2024-09-17T16:34:15.763121+0000","last_peered":"2024-09-17T16:34:15.763121+0000","last_clean":"2024-09-17T16:34:15.763121+0000","last_became_active":"2024-09-17T16:34:06.730571+0000","last_became_peered":"2024-09-17T16:34:06.730571+0000","last_unstale":"2024-09-17T16:34:15.763121+0000","last_undegraded":"2024-09-17T16:34:15.763121+0000","last_fullsized":"2024-09-17T16:34:15.763121+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:19.270502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28704768,"num_objects":438,"num_object_clones":0,"num_object_copies":1314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":438,"num_write_kb":28032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.4","version":"732'417","reported_seq":436,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:15.743666+0000","last_change":"2024-09-17T16:34:07.122284+0000","last_active":"2024-09-17T16:34:15.743666+0000","last_peered":"2024-09-17T16:34:15.743666+0000","last_clean":"2024-09-17T16:34:15.743666+0000","last_became_active":"2024-09-17T16:34:07.121961+0000","last_became_peered":"2024-09-17T16:34:07.121961+0000","last_unstale":"2024-09-17T16:34:15.743666+0000","last_undegraded":"2024-09-17T16:34:15.743666+0000","last_fullsized":"2024-09-17T16:34:15.743666+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:19.750640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27328512,"num_objects":417,"num_object_clones":0,"num_object_copies":1251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":417,"num_write_kb":26688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"733'468","reported_seq":489,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-17T16:34:17.394705+0000","last_change":"2024-09-17T16:34:06.714936+0000","last_active":"2024-09-17T16:34:17.394705+0000","last_peered":"2024-09-17T16:34:17.394705+0000","last_clean":"2024-09-17T16:34:17.394705+0000","last_became_active":"2024-09-17T16:34:06.714658+0000","last_became_peered":"2024-09-17T16:34:06.714658+0000","last_unstale":"2024-09-17T16:34:17.394705+0000","last_undegraded":"2024-09-17T16:34:17.394705+0000","last_fullsized":"2024-09-17T16:34:17.394705+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:10.698855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30670848,"num_objects":468,"num_object_clones":0,"num_object_copies":1404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":468,"num_write_kb":29952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,6],"acting":[1,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"732'436","reported_seq":455,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:15.754798+0000","last_change":"2024-09-17T16:34:06.729979+0000","last_active":"2024-09-17T16:34:15.754798+0000","last_peered":"2024-09-17T16:34:15.754798+0000","last_clean":"2024-09-17T16:34:15.754798+0000","last_became_active":"2024-09-17T16:34:06.729705+0000","last_became_peered":"2024-09-17T16:34:06.729705+0000","last_unstale":"2024-09-17T16:34:15.754798+0000","last_undegraded":"2024-09-17T16:34:15.754798+0000","last_fullsized":"2024-09-17T16:34:15.754798+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:17.357430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28573696,"num_objects":436,"num_object_clones":0,"num_object_copies":1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":436,"num_write_kb":27904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.1","version":"733'534","reported_seq":555,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-17T16:34:17.401493+0000","last_change":"2024-09-17T16:34:06.711356+0000","last_active":"2024-09-17T16:34:17.401493+0000","last_peered":"2024-09-17T16:34:17.401493+0000","last_clean":"2024-09-17T16:34:17.401493+0000","last_became_active":"2024-09-17T16:34:06.711027+0000","last_became_peered":"2024-09-17T16:34:06.711027+0000","last_unstale":"2024-09-17T16:34:17.401493+0000","last_undegraded":"2024-09-17T16:34:17.401493+0000","last_fullsized":"2024-09-17T16:34:17.401493+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":534,"log_dups_size":0,"ondisk_log_size":534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:31.600661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34996224,"num_objects":534,"num_object_clones":0,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":534,"num_write_kb":34176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,10],"acting":[1,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"732'494","reported_seq":513,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:16.796466+0000","last_change":"2024-09-17T16:34:07.122236+0000","last_active":"2024-09-17T16:34:16.796466+0000","last_peered":"2024-09-17T16:34:16.796466+0000","last_clean":"2024-09-17T16:34:16.796466+0000","last_became_active":"2024-09-17T16:34:07.121932+0000","last_became_peered":"2024-09-17T16:34:07.121932+0000","last_unstale":"2024-09-17T16:34:16.796466+0000","last_undegraded":"2024-09-17T16:34:16.796466+0000","last_fullsized":"2024-09-17T16:34:16.796466+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:11.252507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32374784,"num_objects":494,"num_object_clones":0,"num_object_copies":1482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":494,"num_write_kb":31616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,4],"acting":[3,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"732'445","reported_seq":464,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:15.760462+0000","last_change":"2024-09-17T16:34:06.713785+0000","last_active":"2024-09-17T16:34:15.760462+0000","last_peered":"2024-09-17T16:34:15.760462+0000","last_clean":"2024-09-17T16:34:15.760462+0000","last_became_active":"2024-09-17T16:34:06.713432+0000","last_became_peered":"2024-09-17T16:34:06.713432+0000","last_unstale":"2024-09-17T16:34:15.760462+0000","last_undegraded":"2024-09-17T16:34:15.760462+0000","last_fullsized":"2024-09-17T16:34:15.760462+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":445,"log_dups_size":0,"ondisk_log_size":445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:15.024525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29163520,"num_objects":445,"num_object_clones":0,"num_object_copies":1335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":445,"num_write_kb":28480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3],"acting":[10,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":235,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-17T16:34:12.731494+0000","last_change":"2024-09-17T16:33:59.364895+0000","last_active":"2024-09-17T16:34:12.731494+0000","last_peered":"2024-09-17T16:34:12.731494+0000","last_clean":"2024-09-17T16:34:12.731494+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:34:12.731494+0000","last_undegraded":"2024-09-17T16:34:12.731494+0000","last_fullsized":"2024-09-17T16:34:12.731494+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:13.261070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":442105856,"num_objects":6746,"num_object_clones":0,"num_object_copies":20238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6746,"num_write_kb":431744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1359609856,"data_stored":1359609856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6746,"ondisk_log_size":6746,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285865,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":657428,"kb_used_data":175040,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93087724,"statfs":{"total":95995035648,"available":95321829376,"internally_reserved":0,"allocated":179240960,"data_stored":178969999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":493943236},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547502,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":959184,"kb_used_data":440704,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92785968,"statfs":{"total":95995035648,"available":95012831232,"internally_reserved":0,"allocated":451280896,"data_stored":451009955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":530905536},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743731,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":809744,"kb_used_data":124672,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92935408,"statfs":{"total":95995035648,"available":95165857792,"internally_reserved":0,"allocated":127664128,"data_stored":127393187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":701495744},"hb_peers":[0,1,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005368,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":524248,"kb_used_data":55752,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93220904,"statfs":{"total":95995035648,"available":95458205696,"internally_reserved":0,"allocated":57090048,"data_stored":56815555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646676,"kb_used_data":52608,"kb_used_omap":41,"kb_used_meta":594006,"kb_avail":93098476,"statfs":{"total":95995035648,"available":95332839424,"internally_reserved":0,"allocated":53870592,"data_stored":53599651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":608262451},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495938,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529176,"kb_used_data":20424,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93215976,"statfs":{"total":95995035648,"available":95453159424,"internally_reserved":0,"allocated":20914176,"data_stored":20639683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563408,"kb_used_data":26880,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93181744,"statfs":{"total":95995035648,"available":95418105856,"internally_reserved":0,"allocated":27525120,"data_stored":27254179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":549386694},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986509,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563736,"kb_used_data":91080,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93181416,"statfs":{"total":95995035648,"available":95417769984,"internally_reserved":0,"allocated":93265920,"data_stored":92991427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150032,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1007504,"kb_used_data":344640,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92737648,"statfs":{"total":95995035648,"available":94963351552,"internally_reserved":0,"allocated":352911360,"data_stored":352640419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":678754759},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681232,"kb_used_data":194688,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93063920,"statfs":{"total":95995035648,"available":95297454080,"internally_reserved":0,"allocated":199360512,"data_stored":199089571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575196,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609808,"kb_used_data":191296,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93135344,"statfs":{"total":95995035648,"available":95370592256,"internally_reserved":0,"allocated":195887104,"data_stored":195616163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738720,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790928,"kb_used_data":96128,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":92954224,"statfs":{"total":95995035648,"available":95185125376,"internally_reserved":0,"allocated":98435072,"data_stored":98164131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99811328,"data_stored":99811328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":197394432,"data_stored":197394432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200409088,"data_stored":200409088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":165675008,"data_stored":165675008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":118489088,"data_stored":118489088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27328512,"data_stored":27328512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25886720,"data_stored":25886720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":21626880,"data_stored":21626880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":59965440,"data_stored":59965440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":128253952,"data_stored":128253952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":225509376,"data_stored":225509376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":89260032,"data_stored":89260032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:34:19.218 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-17T16:34:19.218 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_6 pg_num 26 2024-09-17T16:34:19.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:19.262 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:19.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:19.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:19.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:19.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:19.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:19 smithi173 ceph-mon[30530]: pgmap v1637: 17 pgs: 17 active+clean; 422 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 843 op/s 2024-09-17T16:34:19.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:19 smithi173 ceph-mon[30530]: from='client.17907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:34:19.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:19 smithi017 ceph-mon[27571]: pgmap v1637: 17 pgs: 17 active+clean; 422 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 843 op/s 2024-09-17T16:34:19.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:19 smithi017 ceph-mon[27571]: from='client.17907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:34:19.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:19.515 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:19.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:19.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:19 smithi071 ceph-mon[30002]: pgmap v1637: 17 pgs: 17 active+clean; 422 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 843 op/s 2024-09-17T16:34:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:19 smithi071 ceph-mon[30002]: from='client.17907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:34:19.613 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 8040 8024 62.6814 0 - 0.0104186 2024-09-17T16:34:19.616 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:19.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:19.638 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:19.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:19.639 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:19.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:34:19.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:19.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:19.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:19.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:34:19.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:20.273 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:34:20.274 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:34:20.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:20.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:20.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:20.422 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:20.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:20.519 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:20.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:20.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:20.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:20.613 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 8040 8024 55.7167 0 - 0.0104186 2024-09-17T16:34:20.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:20.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:20.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:20.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:20.796 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:20.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:20.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:20 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:34:20.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:20 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:34:20.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:20 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:34:20.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:20 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:34:20.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3191343136' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:34:20.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:20 smithi173 ceph-mon[30530]: pgmap v1638: 17 pgs: 17 active+clean; 480 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 957 op/s 2024-09-17T16:34:20.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:20 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:20.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3191343136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:34:20.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:20 smithi173 ceph-mon[30530]: osdmap e734: 12 total, 12 up, 12 in 2024-09-17T16:34:20.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2941108402' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:34:20.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:20.912 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:34:20.913 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:34:20.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.913 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8032 8.c 8:3d0a817e:::benchmark_data_smithi173_130566_object8031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e733)", 2024-09-17T16:34:20.913 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.405332+0000", 2024-09-17T16:34:20.913 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.5045182000000001, 2024-09-17T16:34:20.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5045434790000001, 2024-09-17T16:34:20.913 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8032 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405332+0000", 2024-09-17T16:34:20.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:20.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405332+0000", 2024-09-17T16:34:20.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405334+0000", 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.897e-06 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405359+0000", 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5313999999999999e-05 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:20.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405376+0000", 2024-09-17T16:34:20.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7538999999999999e-05 2024-09-17T16:34:20.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:20.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405385+0000", 2024-09-17T16:34:20.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1710000000000002e-06 2024-09-17T16:34:20.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:20.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405454+0000", 2024-09-17T16:34:20.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9338999999999995e-05 2024-09-17T16:34:20.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:20.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405659+0000", 2024-09-17T16:34:20.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000204852 2024-09-17T16:34:20.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405792+0000", 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133187 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.406488+0000", 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069618000000000002 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.406840+0000", 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035165199999999999 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8037 8.c 8:3f3c220b:::benchmark_data_smithi173_130566_object8036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e733)", 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.409689+0000", 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.5001603370000001, 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5002573780000001, 2024-09-17T16:34:20.920 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:20.921 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:20.921 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:34:20.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:34:20.924 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:34:20.924 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:34:20.924 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8037 2024-09-17T16:34:20.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.925 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:20.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:20.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409689+0000", 2024-09-17T16:34:20.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:20.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409689+0000", 2024-09-17T16:34:20.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:20.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409691+0000", 2024-09-17T16:34:20.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.376e-06 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409713+0000", 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2589999999999999e-05 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409727+0000", 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4035e-05 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409736+0000", 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6470000000000004e-06 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409783+0000", 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6811000000000003e-05 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409909+0000", 2024-09-17T16:34:20.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012609599999999999 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.410018+0000", 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108467 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.410535+0000", 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051754199999999996 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.410949+0000", 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00041375800000000002 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8040 8.c 8:3ef6c0ff:::benchmark_data_smithi173_130566_object8039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e733)", 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.412931+0000", 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.4969188820000001, 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4970563210000001, 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:20.930 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8040 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412931+0000", 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:20.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412931+0000", 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412932+0000", 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5799999999999998e-07 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412952+0000", 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0636999999999999e-05 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:20.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412963+0000", 2024-09-17T16:34:20.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0993e-05 2024-09-17T16:34:20.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:20.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412969+0000", 2024-09-17T16:34:20.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.429e-06 2024-09-17T16:34:20.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:20.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413016+0000", 2024-09-17T16:34:20.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6777000000000003e-05 2024-09-17T16:34:20.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:20.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413102+0000", 2024-09-17T16:34:20.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6059999999999999e-05 2024-09-17T16:34:20.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:34:20.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413187+0000", 2024-09-17T16:34:20.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5408999999999995e-05 2024-09-17T16:34:20.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:34:20.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413596+0000", 2024-09-17T16:34:20.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040850499999999998 2024-09-17T16:34:20.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:20.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413999+0000", 2024-09-17T16:34:20.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040360899999999998 2024-09-17T16:34:20.941 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.941 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:20.941 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.942 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:7992 8.c 8:34878b4c:::benchmark_data_smithi173_130566_object7991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e732)", 2024-09-17T16:34:20.942 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.037799+0000", 2024-09-17T16:34:20.943 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8720510940000001, 2024-09-17T16:34:20.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8722285940000001, 2024-09-17T16:34:20.943 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:20.944 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:20.944 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:34:20.944 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:34:20.945 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:34:20.945 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:34:20.946 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 7992 2024-09-17T16:34:20.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.946 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:20.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:20.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037799+0000", 2024-09-17T16:34:20.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:20.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037799+0000", 2024-09-17T16:34:20.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:20.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037800+0000", 2024-09-17T16:34:20.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.054e-06 2024-09-17T16:34:20.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:20.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037828+0000", 2024-09-17T16:34:20.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8294e-05 2024-09-17T16:34:20.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:20.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037846+0000", 2024-09-17T16:34:20.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7649999999999999e-05 2024-09-17T16:34:20.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:20.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037854+0000", 2024-09-17T16:34:20.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5029999999999997e-06 2024-09-17T16:34:20.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037910+0000", 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5998000000000002e-05 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.038015+0000", 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010526000000000001 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:34:20.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.038108+0000", 2024-09-17T16:34:20.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2848000000000003e-05 2024-09-17T16:34:20.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:34:20.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.038630+0000", 2024-09-17T16:34:20.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00052117799999999996 2024-09-17T16:34:20.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:20.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.039017+0000", 2024-09-17T16:34:20.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038750600000000003 2024-09-17T16:34:20.955 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:7994 8.c 8:379b6ec9:::benchmark_data_smithi173_130566_object7993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e732)", 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.040560+0000", 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8692897149999999, 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8695060300000002, 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:34:20.956 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:34:20.957 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:34:20.957 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:34:20.957 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 7994 2024-09-17T16:34:20.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.957 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:20.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:20.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040560+0000", 2024-09-17T16:34:20.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:20.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040560+0000", 2024-09-17T16:34:20.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:20.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040561+0000", 2024-09-17T16:34:20.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3900000000000003e-07 2024-09-17T16:34:20.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:20.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040585+0000", 2024-09-17T16:34:20.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4145e-05 2024-09-17T16:34:20.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:20.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040599+0000", 2024-09-17T16:34:20.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3766e-05 2024-09-17T16:34:20.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:20.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040606+0000", 2024-09-17T16:34:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9990000000000002e-06 2024-09-17T16:34:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040650+0000", 2024-09-17T16:34:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4447999999999999e-05 2024-09-17T16:34:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040735+0000", 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4257000000000003e-05 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040821+0000", 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6341000000000006e-05 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.041276+0000", 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045481100000000001 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.041625+0000", 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034894899999999998 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8013 8.c 8:3c762cec:::benchmark_data_smithi173_130566_object8012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e732)", 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.082606+0000", 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8272436590000001, 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.827498904, 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:20.965 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8013 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082606+0000", 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082606+0000", 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082607+0000", 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5700000000000001e-07 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082627+0000", 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9565999999999999e-05 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082637+0000", 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0616e-05 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082645+0000", 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2960000000000001e-06 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082691+0000", 2024-09-17T16:34:20.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6681999999999998e-05 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082787+0000", 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6203e-05 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082873+0000", 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5687999999999997e-05 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:34:20.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.083367+0000", 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049404099999999997 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.083928+0000", 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056047200000000001 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 6 2024-09-17T16:34:20.970 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:34:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:20.971 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:20.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:20 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:34:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:20 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:34:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:20 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:34:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:20 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:34:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3191343136' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:34:20.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:20 smithi017 ceph-mon[27571]: pgmap v1638: 17 pgs: 17 active+clean; 480 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 957 op/s 2024-09-17T16:34:20.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:20 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:20.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3191343136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:34:20.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:20 smithi017 ceph-mon[27571]: osdmap e734: 12 total, 12 up, 12 in 2024-09-17T16:34:20.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2941108402' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:34:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:20 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:34:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:20 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:34:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:20 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:34:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:20 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:34:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3191343136' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:34:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:20 smithi071 ceph-mon[30002]: pgmap v1638: 17 pgs: 17 active+clean; 480 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 957 op/s 2024-09-17T16:34:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:20 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3191343136' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:34:21.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:20 smithi071 ceph-mon[30002]: osdmap e734: 12 total, 12 up, 12 in 2024-09-17T16:34:21.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2941108402' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:34:21.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:21.094 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:21.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:21.134 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 8 pg_num to 26 2024-09-17T16:34:21.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:21.222 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:21.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:21.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:21.324 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:21.350 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:21.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:21.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:21.474 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:21.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:21.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:21.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:21.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:21.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:21.613 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 8040 8024 50.1451 0 - 0.0104186 2024-09-17T16:34:21.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:21.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:21.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:21.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:34:21.825 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:21.848 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:21.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:21.893 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:21 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:34:21.893 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:21 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:34:21.893 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2941108402' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:34:21.893 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:21 smithi017 ceph-mon[27571]: osdmap e735: 12 total, 12 up, 12 in 2024-09-17T16:34:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:21 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:34:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:21 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:34:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2941108402' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:34:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:21 smithi173 ceph-mon[30530]: osdmap e735: 12 total, 12 up, 12 in 2024-09-17T16:34:21.952 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:21.978 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:21.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:22.058 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:34:22.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:21 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:34:22.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:21 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:34:22.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2941108402' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:34:22.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:21 smithi071 ceph-mon[30002]: osdmap e735: 12 total, 12 up, 12 in 2024-09-17T16:34:22.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:22.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:22.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:22.170 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:34:22.170 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:34:22.191 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:34:22.192 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:34:22.203 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:22.216 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:34:22.225 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:22.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:22.237 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:34:22.237 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:34:22.260 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:34:22.260 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:34:22.283 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:34:22.284 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:34:22.303 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:34:22.303 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:34:22.325 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:34:22.325 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:34:22.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:22.348 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:34:22.348 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:34:22.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:22.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:22.388 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:34:22.388 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:34:22.409 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:34:22.409 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:34:22.431 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:22.431 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:22.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:22.478 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:22.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:22.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:22.585 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:22.585 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:22.602 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:22.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:22.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:22.613 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 8040 8024 45.5863 0 - 0.0104186 2024-09-17T16:34:22.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:22.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:22.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:22.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:22.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:22.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:22.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:22 smithi173 ceph-mon[30530]: pgmap v1641: 17 pgs: 17 active+clean; 503 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 714 op/s 2024-09-17T16:34:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:22 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:34:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/408919329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:22.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:22.979 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:22.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:22 smithi017 ceph-mon[27571]: pgmap v1641: 17 pgs: 17 active+clean; 503 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 714 op/s 2024-09-17T16:34:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:22 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:34:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/408919329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:23.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:34:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:22 smithi071 ceph-mon[30002]: pgmap v1641: 17 pgs: 17 active+clean; 503 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 714 op/s 2024-09-17T16:34:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:22 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:34:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/408919329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:23.081 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:23.106 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:23.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:23.207 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:23.231 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:23.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:23.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:34:23.281 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:23.333 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:23.357 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:23.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:23.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:23.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:23.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:23.563 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:34:23.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:23.614 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:23.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:23.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 8040 8024 41.7872 0 - 0.0104186 2024-09-17T16:34:23.715 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:23.738 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:23.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:23.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:23.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:23.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:23.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:23.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:23.968 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:23 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:34:23.969 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:23 smithi173 ceph-mon[30530]: osdmap e736: 12 total, 12 up, 12 in 2024-09-17T16:34:23.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:23 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:34:23.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:23 smithi017 ceph-mon[27571]: osdmap e736: 12 total, 12 up, 12 in 2024-09-17T16:34:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:23.995 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:23.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:24.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:23 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:34:24.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:23 smithi071 ceph-mon[30002]: osdmap e736: 12 total, 12 up, 12 in 2024-09-17T16:34:24.069 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:24.069 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:24.097 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:24.118 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:24.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:24.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:24.243 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:24.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:24.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:24.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:24.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:24.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:24.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:24.494 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:24.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:24.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:24.614 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 8040 8024 38.5728 0 - 0.0104186 2024-09-17T16:34:24.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:24.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:24.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:34:24.703 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:34:24.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:24.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:24.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:24.852 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:24.876 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:24.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:24.933 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:34:24.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:24 smithi017 ceph-mon[27571]: osdmap e737: 12 total, 12 up, 12 in 2024-09-17T16:34:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:24 smithi017 ceph-mon[27571]: 8.8 scrub starts 2024-09-17T16:34:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:24 smithi017 ceph-mon[27571]: pgmap v1644: 27 pgs: 10 unknown, 17 active+clean; 503 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s wr, 93 op/s 2024-09-17T16:34:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:24 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:34:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3670839828' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:24 smithi017 ceph-mon[27571]: 8.6 scrub starts 2024-09-17T16:34:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:24 smithi017 ceph-mon[27571]: 8.6 scrub ok 2024-09-17T16:34:25.001 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:25.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:25.054 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:24 smithi173 ceph-mon[30530]: osdmap e737: 12 total, 12 up, 12 in 2024-09-17T16:34:25.054 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:24 smithi173 ceph-mon[30530]: 8.8 scrub starts 2024-09-17T16:34:25.054 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:24 smithi173 ceph-mon[30530]: pgmap v1644: 27 pgs: 10 unknown, 17 active+clean; 503 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s wr, 93 op/s 2024-09-17T16:34:25.054 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:24 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:34:25.054 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3670839828' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:25.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:24 smithi173 ceph-mon[30530]: 8.6 scrub starts 2024-09-17T16:34:25.055 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:24 smithi173 ceph-mon[30530]: 8.6 scrub ok 2024-09-17T16:34:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:24 smithi071 ceph-mon[30002]: osdmap e737: 12 total, 12 up, 12 in 2024-09-17T16:34:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:24 smithi071 ceph-mon[30002]: 8.8 scrub starts 2024-09-17T16:34:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:24 smithi071 ceph-mon[30002]: pgmap v1644: 27 pgs: 10 unknown, 17 active+clean; 503 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s wr, 93 op/s 2024-09-17T16:34:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:24 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:34:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3670839828' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:24 smithi071 ceph-mon[30002]: 8.6 scrub starts 2024-09-17T16:34:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:24 smithi071 ceph-mon[30002]: 8.6 scrub ok 2024-09-17T16:34:25.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:34:25.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:25.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:25.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:25.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:25.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:25.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:25.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:34:25.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:25.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:34:25.472 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:25.472 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:25.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:25.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:25.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:25.600 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:25.614 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 8040 8024 35.8174 0 - 0.0104186 2024-09-17T16:34:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:25.626 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:25.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:25.727 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:25.750 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:25.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:25.852 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:25.874 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:25.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:25.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:25.881 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:25.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: 8.1 scrub starts 2024-09-17T16:34:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: 8.1 scrub ok 2024-09-17T16:34:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:34:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:34:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: 8.8 scrub ok 2024-09-17T16:34:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: 8.4 scrub starts 2024-09-17T16:34:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: 8.4 scrub ok 2024-09-17T16:34:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:34:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:34:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:34:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3670839828' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: osdmap e738: 12 total, 12 up, 12 in 2024-09-17T16:34:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: 8.0 scrub starts 2024-09-17T16:34:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: 8.0 scrub ok 2024-09-17T16:34:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: 8.9 scrub starts 2024-09-17T16:34:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:25 smithi017 ceph-mon[27571]: 8.9 scrub ok 2024-09-17T16:34:26.003 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:26.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:26.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: 8.1 scrub starts 2024-09-17T16:34:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: 8.1 scrub ok 2024-09-17T16:34:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:34:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:34:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: 8.8 scrub ok 2024-09-17T16:34:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: 8.4 scrub starts 2024-09-17T16:34:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: 8.4 scrub ok 2024-09-17T16:34:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:34:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:34:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:34:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3670839828' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: osdmap e738: 12 total, 12 up, 12 in 2024-09-17T16:34:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: 8.0 scrub starts 2024-09-17T16:34:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: 8.0 scrub ok 2024-09-17T16:34:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: 8.9 scrub starts 2024-09-17T16:34:26.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:25 smithi071 ceph-mon[30002]: 8.9 scrub ok 2024-09-17T16:34:26.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:26.129 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:26.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: 8.1 scrub starts 2024-09-17T16:34:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: 8.1 scrub ok 2024-09-17T16:34:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:34:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:34:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: 8.8 scrub ok 2024-09-17T16:34:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: 8.4 scrub starts 2024-09-17T16:34:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: 8.4 scrub ok 2024-09-17T16:34:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:34:26.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:34:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:34:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3670839828' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: osdmap e738: 12 total, 12 up, 12 in 2024-09-17T16:34:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: 8.0 scrub starts 2024-09-17T16:34:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: 8.0 scrub ok 2024-09-17T16:34:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: 8.9 scrub starts 2024-09-17T16:34:26.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:25 smithi173 ceph-mon[30530]: 8.9 scrub ok 2024-09-17T16:34:26.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:26.253 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:26.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:26.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:26.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:26.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:26.481 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:26.506 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:26.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:26.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:26.614 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 8040 8024 33.4296 0 - 0.0104186 2024-09-17T16:34:26.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:26.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:26.734 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:26.766 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:26.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:26.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:26.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:26.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:26.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:34:26.922 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:34:26.922 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:34:26.922 INFO:tasks.thrashosds.thrasher:Killing osd 10, live_osds are [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 7] 2024-09-17T16:34:26.922 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-17T16:34:26.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.10 2024-09-17T16:34:26.983 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:26 smithi173 ceph-mon[30530]: 8.3 scrub starts 2024-09-17T16:34:26.983 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:26 smithi173 ceph-mon[30530]: 8.3 scrub ok 2024-09-17T16:34:26.983 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:26 smithi173 ceph-mon[30530]: osdmap e739: 12 total, 12 up, 12 in 2024-09-17T16:34:26.983 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:26 smithi173 ceph-mon[30530]: 8.7 scrub starts 2024-09-17T16:34:26.983 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:26 smithi173 ceph-mon[30530]: pgmap v1647: 27 pgs: 2 remapped+peering, 8 unknown, 17 active+clean; 457 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:26.984 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:26 smithi173 ceph-mon[30530]: 8.5 scrub starts 2024-09-17T16:34:26.984 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/144048250' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:26.984 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:26 smithi173 ceph-mon[30530]: 8.5 scrub ok 2024-09-17T16:34:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:26 smithi017 ceph-mon[27571]: 8.3 scrub starts 2024-09-17T16:34:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:26 smithi017 ceph-mon[27571]: 8.3 scrub ok 2024-09-17T16:34:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:26 smithi017 ceph-mon[27571]: osdmap e739: 12 total, 12 up, 12 in 2024-09-17T16:34:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:26 smithi017 ceph-mon[27571]: 8.7 scrub starts 2024-09-17T16:34:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:26 smithi017 ceph-mon[27571]: pgmap v1647: 27 pgs: 2 remapped+peering, 8 unknown, 17 active+clean; 457 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:26 smithi017 ceph-mon[27571]: 8.5 scrub starts 2024-09-17T16:34:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/144048250' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:26.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:26 smithi017 ceph-mon[27571]: 8.5 scrub ok 2024-09-17T16:34:26.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:27.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:27.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:27.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:26 smithi071 ceph-mon[30002]: 8.3 scrub starts 2024-09-17T16:34:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:26 smithi071 ceph-mon[30002]: 8.3 scrub ok 2024-09-17T16:34:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:26 smithi071 ceph-mon[30002]: osdmap e739: 12 total, 12 up, 12 in 2024-09-17T16:34:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:26 smithi071 ceph-mon[30002]: 8.7 scrub starts 2024-09-17T16:34:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:26 smithi071 ceph-mon[30002]: pgmap v1647: 27 pgs: 2 remapped+peering, 8 unknown, 17 active+clean; 457 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:34:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:26 smithi071 ceph-mon[30002]: 8.5 scrub starts 2024-09-17T16:34:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/144048250' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:27.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:26 smithi071 ceph-mon[30002]: 8.5 scrub ok 2024-09-17T16:34:27.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:27.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:27.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:27.267 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:34:27.268 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:34:27.281 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:27.289 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:34:27.290 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:34:27.291 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:26 smithi173 systemd[1]: Stopping Ceph osd.10 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:34:27.305 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:27.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:27.313 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:34:27.313 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:34:27.335 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:34:27.335 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:34:27.358 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:34:27.358 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:34:27.367 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:34:27.380 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:34:27.381 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:34:27.404 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:34:27.404 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:34:27.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:27.426 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:34:27.426 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:34:27.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:27.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:27.449 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:34:27.449 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:34:27.471 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:34:27.471 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:34:27.493 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:34:27.493 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:34:27.516 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:27.517 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:27.529 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:27.544 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:27 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[45258]: 2024-09-17T16:34:27.328+0000 7f16c9ffa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:34:27.544 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:27 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[45258]: 2024-09-17T16:34:27.328+0000 7f16c9ffa640 -1 osd.10 740 *** Got signal Terminated *** 2024-09-17T16:34:27.544 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:27 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[45258]: 2024-09-17T16:34:27.328+0000 7f16c9ffa640 -1 osd.10 740 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:34:27.552 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:27.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:27.614 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 8040 8024 31.3404 0 - 0.0104186 2024-09-17T16:34:27.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:27.676 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:27.678 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:27.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:27.786 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:34:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:27.810 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:27.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:27.874 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:27 smithi017 ceph-mon[27571]: 8.7 scrub ok 2024-09-17T16:34:27.875 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:27 smithi017 ceph-mon[27571]: osdmap e740: 12 total, 12 up, 12 in 2024-09-17T16:34:27.875 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:27 smithi017 ceph-mon[27571]: osd.10 marked itself down and dead 2024-09-17T16:34:27.875 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:27 smithi017 ceph-mon[27571]: 8.2 scrub starts 2024-09-17T16:34:27.875 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:27 smithi017 ceph-mon[27571]: 8.2 scrub ok 2024-09-17T16:34:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:27 smithi173 ceph-mon[30530]: 8.7 scrub ok 2024-09-17T16:34:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:27 smithi173 ceph-mon[30530]: osdmap e740: 12 total, 12 up, 12 in 2024-09-17T16:34:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:27 smithi173 ceph-mon[30530]: osd.10 marked itself down and dead 2024-09-17T16:34:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:27 smithi173 ceph-mon[30530]: 8.2 scrub starts 2024-09-17T16:34:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:27 smithi173 ceph-mon[30530]: 8.2 scrub ok 2024-09-17T16:34:27.911 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:27.938 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:27.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:28.006 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:34:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:34:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:34:28.006 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:34:28.006 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:34:28.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:27 smithi071 ceph-mon[30002]: 8.7 scrub ok 2024-09-17T16:34:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:27 smithi071 ceph-mon[30002]: osdmap e740: 12 total, 12 up, 12 in 2024-09-17T16:34:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:27 smithi071 ceph-mon[30002]: osd.10 marked itself down and dead 2024-09-17T16:34:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:27 smithi071 ceph-mon[30002]: 8.2 scrub starts 2024-09-17T16:34:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:27 smithi071 ceph-mon[30002]: 8.2 scrub ok 2024-09-17T16:34:28.065 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:28.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:28.168 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:28.191 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:28.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:28.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:28.325 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:28.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:28.427 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:28.455 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:28.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:28.490 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:28.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:28.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:28.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:28.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:28.614 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 8040 8024 29.4969 0 - 0.0104186 2024-09-17T16:34:28.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:28.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:28.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:28.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:28.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:34:28.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:28.840 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:28.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:28.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:28.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:28 smithi017 ceph-mon[27571]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T16:34:28.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:28 smithi017 ceph-mon[27571]: osdmap e741: 12 total, 11 up, 12 in 2024-09-17T16:34:28.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:28 smithi017 ceph-mon[27571]: pgmap v1650: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 4 stale+active+clean, 1 remapped+peering, 1 stale+remapped+peering, 13 active+clean; 501 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 251/24030 objects degraded (1.045%); 3500/24030 objects misplaced (14.565%); 7.5 MiB/s, 119 objects/s recovering 2024-09-17T16:34:28.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:28.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:28.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:29.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:28 smithi071 ceph-mon[30002]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T16:34:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:28 smithi071 ceph-mon[30002]: osdmap e741: 12 total, 11 up, 12 in 2024-09-17T16:34:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:28 smithi071 ceph-mon[30002]: pgmap v1650: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 4 stale+active+clean, 1 remapped+peering, 1 stale+remapped+peering, 13 active+clean; 501 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 251/24030 objects degraded (1.045%); 3500/24030 objects misplaced (14.565%); 7.5 MiB/s, 119 objects/s recovering 2024-09-17T16:34:29.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:29.087 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:29.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:29.090 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:28 smithi173 ceph-mon[30530]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T16:34:29.090 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:28 smithi173 ceph-mon[30530]: osdmap e741: 12 total, 11 up, 12 in 2024-09-17T16:34:29.090 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:28 smithi173 ceph-mon[30530]: pgmap v1650: 27 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 4 stale+active+clean, 1 remapped+peering, 1 stale+remapped+peering, 13 active+clean; 501 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 251/24030 objects degraded (1.045%); 3500/24030 objects misplaced (14.565%); 7.5 MiB/s, 119 objects/s recovering 2024-09-17T16:34:29.091 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:28 smithi173 podman[131850]: 2024-09-17 16:34:28.703663665 +0000 UTC m=+1.516811315 container died 98ea1bb1b26ba86329f5554fe5be0198aaa73def76fdb6528bfc527658bb542f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-17T16:34:29.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:29.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:29.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:29.213 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:29.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:29.338 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:29.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:29.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:29.395 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:29 smithi173 podman[131850]: 2024-09-17 16:34:29.196148032 +0000 UTC m=+2.009295678 container cleanup 98ea1bb1b26ba86329f5554fe5be0198aaa73def76fdb6528bfc527658bb542f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10, maintainer=Guillaume Abrioux , RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:34:29.395 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:29 smithi173 bash[131850]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10 2024-09-17T16:34:29.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:29.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:29.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:29.509 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:34:29.563 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:29.586 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:29.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:29.614 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 8040 8024 27.8581 0 - 0.0104186 2024-09-17T16:34:29.680 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:34:29.687 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:29.720 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:29.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:29.786 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:29 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:34:29.786 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:29 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:34:29.786 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:29 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 251/24030 objects degraded (1.045%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:34:29.786 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:29 smithi173 ceph-mon[30530]: osdmap e742: 12 total, 11 up, 12 in 2024-09-17T16:34:29.786 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2099160923' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:29.786 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:29 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:29.787 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2099160923' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:34:29.787 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:29 smithi173 ceph-mon[30530]: osdmap e743: 12 total, 11 up, 12 in 2024-09-17T16:34:29.787 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:29 smithi173 podman[131976]: 2024-09-17 16:34:29.47312784 +0000 UTC m=+1.098598072 container remove 98ea1bb1b26ba86329f5554fe5be0198aaa73def76fdb6528bfc527658bb542f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:34:29.825 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:29.849 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:29.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:29.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:29.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:29.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:29.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:29 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:34:29.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:29 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:34:29.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:29 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 251/24030 objects degraded (1.045%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:34:29.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:29 smithi017 ceph-mon[27571]: osdmap e742: 12 total, 11 up, 12 in 2024-09-17T16:34:29.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2099160923' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:29.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:29 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:29.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2099160923' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:34:29.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:29 smithi017 ceph-mon[27571]: osdmap e743: 12 total, 11 up, 12 in 2024-09-17T16:34:30.008 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:34:30.008 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:34:30.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:34:30.008 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:34:30.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.008 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:13492 6.4 e535/527 6:27b8e258:::benchmark_data_smithi173_102017_object13491:head v 535'822, mlcod=535'819)", 2024-09-17T16:34:30.008 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:18.460867+0000", 2024-09-17T16:34:30.008 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 551.54366420500003, 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23653454300000001, 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460867+0000", 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460867+0000", 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460868+0000", 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1699999999999998e-07 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.460931+0000", 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3336000000000005e-05 2024-09-17T16:34:30.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696639+0000", 2024-09-17T16:34:30.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23570720000000001 2024-09-17T16:34:30.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696643+0000", 2024-09-17T16:34:30.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7280000000000003e-06 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696694+0000", 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1141999999999999e-05 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.696718+0000", 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3328000000000001e-05 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.697364+0000", 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00064611300000000003 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:18.697402+0000", 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8078999999999999e-05 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.013 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:17620 6.5 e538/527 6:a62c44a9:::benchmark_data_smithi173_102017_object17619:head v 538'1058, mlcod=537'1054)", 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:21.992300+0000", 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 548.012231101, 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50495759100000004, 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992300+0000", 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992300+0000", 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992301+0000", 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8500000000000001e-07 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.992450+0000", 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149061 2024-09-17T16:34:30.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496350+0000", 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50389984499999996 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496361+0000", 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0807e-05 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496468+0000", 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106811 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.496508+0000", 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0079e-05 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.497225+0000", 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00071726500000000002 2024-09-17T16:34:30.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.497258+0000", 2024-09-17T16:34:30.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3037999999999998e-05 2024-09-17T16:34:30.018 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.018 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.018 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.019 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:17628 6.9 e538/527 6:9546f3b4:::benchmark_data_smithi173_102017_object17627:head v 538'1099, mlcod=537'1096)", 2024-09-17T16:34:30.019 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:21.999183+0000", 2024-09-17T16:34:30.019 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 548.00534828399998, 2024-09-17T16:34:30.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45068112500000002, 2024-09-17T16:34:30.019 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.020 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.020 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.020 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999183+0000", 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999183+0000", 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999184+0000", 2024-09-17T16:34:30.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3e-07 2024-09-17T16:34:30.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:21.999231+0000", 2024-09-17T16:34:30.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7206000000000002e-05 2024-09-17T16:34:30.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448811+0000", 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44957967100000001 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448826+0000", 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5480000000000001e-05 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448977+0000", 2024-09-17T16:34:30.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015076600000000001 2024-09-17T16:34:30.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.448996+0000", 2024-09-17T16:34:30.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9069999999999999e-05 2024-09-17T16:34:30.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.449834+0000", 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083746999999999999 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:22.449864+0000", 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0732000000000003e-05 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.025 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.026 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:27647 6.a e545/527 6:52a81611:::benchmark_data_smithi173_102017_object27646:head v 545'1782, mlcod=545'1779)", 2024-09-17T16:34:30.026 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:32.709980+0000", 2024-09-17T16:34:30.026 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 537.29455187500002, 2024-09-17T16:34:30.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43213955799999998, 2024-09-17T16:34:30.026 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.709980+0000", 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.709980+0000", 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.709981+0000", 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.23e-07 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:32.710035+0000", 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4273e-05 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141017+0000", 2024-09-17T16:34:30.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43098198300000001 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141021+0000", 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.515e-06 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141072+0000", 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0322999999999999e-05 2024-09-17T16:34:30.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.141086+0000", 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4409e-05 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.142088+0000", 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010016109999999999 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:33.142119+0000", 2024-09-17T16:34:30.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1621000000000001e-05 2024-09-17T16:34:30.031 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.031 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.031 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.032 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:85438 6.f e565/527 6:f29abae7:::benchmark_data_smithi173_102017_object85437:head v 565'5353, mlcod=565'5351)", 2024-09-17T16:34:30.032 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:26:27.791488+0000", 2024-09-17T16:34:30.032 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 482.21304369500001, 2024-09-17T16:34:30.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.243614578, 2024-09-17T16:34:30.032 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791488+0000", 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791488+0000", 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791489+0000", 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1900000000000001e-07 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:27.791572+0000", 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3040000000000005e-05 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032638+0000", 2024-09-17T16:34:30.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24106640200000001 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032645+0000", 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.55e-06 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032734+0000", 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9209000000000006e-05 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.032755+0000", 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0618000000000001e-05 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.035062+0000", 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023077140000000002 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:28.035102+0000", 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0126000000000002e-05 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.036 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:87629 6.9 e566/527 6:975355a4:::benchmark_data_smithi173_102017_object87628:head v 566'5443, mlcod=566'5441)", 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:26:30.268719+0000", 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 479.73581301600001, 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22957050300000001, 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268719+0000", 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268719+0000", 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268721+0000", 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.12e-06 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.268753+0000", 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2620999999999998e-05 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494429+0000", 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22567617500000001 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494433+0000", 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3000000000000002e-06 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494476+0000", 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2908999999999999e-05 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.494495+0000", 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9335999999999999e-05 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.498267+0000", 2024-09-17T16:34:30.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037720879999999998 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:26:30.498289+0000", 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1954e-05 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:114687 6.9 e582/527 6:96e26e86:::benchmark_data_smithi173_102017_object21539:head v 582'7080, mlcod=582'7078)", 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:12.569600+0000", 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 437.43493158199999, 2024-09-17T16:34:30.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24650798600000001, 2024-09-17T16:34:30.042 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.042 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.042 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.042 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569600+0000", 2024-09-17T16:34:30.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569600+0000", 2024-09-17T16:34:30.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569601+0000", 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.569999999999999e-07 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.569607+0000", 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2380000000000002e-06 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812466+0000", 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24285875200000001 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812471+0000", 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9960000000000001e-06 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812530+0000", 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9333999999999998e-05 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.812551+0000", 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1158999999999999e-05 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.816069+0000", 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035176840000000001 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:12.816108+0000", 2024-09-17T16:34:30.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8865999999999997e-05 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:115587 6.5 e582/527 6:ae192cb4:::benchmark_data_smithi173_102017_object22439:head v 582'7137, mlcod=582'7134)", 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:14.230950+0000", 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 435.77358168299997, 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29268783799999998, 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.047 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230950+0000", 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230950+0000", 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230951+0000", 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5700000000000001e-07 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.230956+0000", 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6210000000000001e-06 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520676+0000", 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.289719173 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520684+0000", 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.833e-06 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520770+0000", 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5866999999999996e-05 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.520790+0000", 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9266000000000002e-05 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.523606+0000", 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028161639999999999 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:14.523638+0000", 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2057000000000002e-05 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.051 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:116184 6.f e583/527 6:f2e9c488:::benchmark_data_smithi173_102017_object23036:head v 583'7277, mlcod=583'7273)", 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:29.905958+0000", 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 420.09857332500002, 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23584665899999999, 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905958+0000", 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905958+0000", 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905960+0000", 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.282e-06 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:29.905967+0000", 2024-09-17T16:34:30.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.61e-06 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136208+0000", 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23024043699999999 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136213+0000", 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.955e-06 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136316+0000", 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103074 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.136341+0000", 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5809e-05 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.141768+0000", 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054267589999999997 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:30.141805+0000", 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6733000000000002e-05 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:186285 6.f e621/527 6:ff66d09f:::benchmark_data_smithi173_102017_object93137:head v 621'11698, mlcod=621'11694)", 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:17.540727+0000", 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 312.46380477500003, 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53518362600000002, 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.056 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540727+0000", 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540727+0000", 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540728+0000", 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.361e-06 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540742+0000", 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4087000000000001e-05 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.071882+0000", 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53114022100000002 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.071897+0000", 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4039000000000001e-05 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.072016+0000", 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011962499999999999 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.072036+0000", 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0035999999999999e-05 2024-09-17T16:34:30.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.075868+0000", 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038314 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.075910+0000", 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2857e-05 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.061 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:21072 7.5 7:a6915c4f:::benchmark_data_smithi173_116396_object21071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-09-17T16:34:30.061 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:58.907216+0000", 2024-09-17T16:34:30.061 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 271.09731529599998, 2024-09-17T16:34:30.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32350631499999999, 2024-09-17T16:34:30.061 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.061 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.061 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.061 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:34:30.061 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21072 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907216+0000", 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907216+0000", 2024-09-17T16:34:30.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907218+0000", 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.638e-06 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907236+0000", 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7849000000000001e-05 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907248+0000", 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1933000000000001e-05 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907254+0000", 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0689999999999998e-06 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907308+0000", 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3940000000000002e-05 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907441+0000", 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133685 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907543+0000", 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101369 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.908430+0000", 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088757899999999999 2024-09-17T16:34:30.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.908686+0000", 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025581999999999997 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230610+0000", 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32192386200000001 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230673+0000", 2024-09-17T16:34:30.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2668000000000004e-05 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230723+0000", 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9903e-05 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:111379 7.0 e681/674 7:05ca7e9f:::benchmark_data_smithi173_116396_object15779:head v 681'6620, mlcod=681'6617)", 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:41.226937+0000", 2024-09-17T16:34:30.067 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.77759500799999, 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42183363800000001, 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.226937+0000", 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.226937+0000", 2024-09-17T16:34:30.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.226938+0000", 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2869999999999999e-06 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.226946+0000", 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4980000000000006e-06 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.647196+0000", 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42024978200000002 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.647206+0000", 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4180000000000003e-06 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.647303+0000", 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7071999999999995e-05 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.647321+0000", 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8780999999999999e-05 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.648742+0000", 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001420997 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.648770+0000", 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7803000000000001e-05 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.071 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:116084 7.5 7:a0912ad1:::benchmark_data_smithi173_116396_object20484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e685)", 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:50.990260+0000", 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 159.01427203700001, 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28407464199999999, 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:34:30.072 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 116084 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990260+0000", 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990260+0000", 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990261+0000", 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6670000000000001e-06 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990267+0000", 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5060000000000003e-06 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990279+0000", 2024-09-17T16:34:30.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2644e-05 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990286+0000", 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5549999999999999e-06 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990346+0000", 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0047999999999998e-05 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990482+0000", 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013580800000000001 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990620+0000", 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013782200000000001 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.992390+0000", 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001770647 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.993010+0000", 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062002699999999999 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274221+0000", 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28121024300000003 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274282+0000", 2024-09-17T16:34:30.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1865000000000005e-05 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274334+0000", 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.181e-05 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27088.0:116084 7.5 e685/674 ondisk, result = 0)", 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:50.992582+0000", 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 159.011949461, 2024-09-17T16:34:30.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28175829000000002, 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.992582+0000", 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.992582+0000", 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.992583+0000", 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.61e-07 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.992587+0000", 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.498e-06 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274134+0000", 2024-09-17T16:34:30.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28154638500000001 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274139+0000", 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7180000000000001e-06 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274200+0000", 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0126e-05 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274209+0000", 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1249999999999999e-06 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274340+0000", 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000131777 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:117077 7.5 7:a4e89f58:::benchmark_data_smithi173_116396_object21477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e685)", 2024-09-17T16:34:30.082 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:53.097682+0000", 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 156.90684933599999, 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.240555148, 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 117077 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097682+0000", 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097682+0000", 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097684+0000", 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8020000000000001e-06 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097691+0000", 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1720000000000001e-06 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097708+0000", 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6521999999999999e-05 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097715+0000", 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1130000000000003e-06 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097783+0000", 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7731999999999995e-05 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097884+0000", 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010187999999999999 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097963+0000", 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8838999999999995e-05 2024-09-17T16:34:30.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.099970+0000", 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020066760000000002 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.104300+0000", 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043297980000000002 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338164+0000", 2024-09-17T16:34:30.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23386404299999999 2024-09-17T16:34:30.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338198+0000", 2024-09-17T16:34:30.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4224999999999998e-05 2024-09-17T16:34:30.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338237+0000", 2024-09-17T16:34:30.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9345999999999999e-05 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27088.0:117077 7.5 e685/674 ondisk, result = 0)", 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:53.100127+0000", 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 156.90440499900001, 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.238116831, 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:34:30.089 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.100127+0000", 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.100127+0000", 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.100127+0000", 2024-09-17T16:34:30.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8299999999999997e-07 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.100130+0000", 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2959999999999999e-06 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338112+0000", 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23798161700000001 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338117+0000", 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5279999999999997e-06 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338154+0000", 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7812000000000003e-05 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338160+0000", 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4360000000000001e-06 2024-09-17T16:34:30.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338243+0000", 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3559000000000004e-05 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:170897 7.5 7:a2dbce43:::benchmark_data_smithi173_116396_object75297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:26.438350+0000", 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.566181944, 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34698544799999997, 2024-09-17T16:34:30.093 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 170897 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438350+0000", 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438350+0000", 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438351+0000", 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.815e-06 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438358+0000", 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8260000000000004e-06 2024-09-17T16:34:30.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438372+0000", 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4131000000000001e-05 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438380+0000", 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4449999999999999e-06 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438438+0000", 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7833e-05 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438537+0000", 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8987999999999995e-05 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438658+0000", 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121291 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.441438+0000", 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027800049999999999 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.453632+0000", 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012194439 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785264+0000", 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33163123999999999 2024-09-17T16:34:30.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785303+0000", 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8894999999999999e-05 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785335+0000", 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2539999999999997e-05 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.099 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(7.0 e724,723 7:00d6d267:::benchmark_data_smithi173_116396_object11771:head,679'6499)", 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:54.910007+0000", 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.094524624000002, 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71534004299999998, 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910007+0000", 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910007+0000", 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910009+0000", 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6169999999999999e-06 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910019+0000", 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0047000000000001e-05 2024-09-17T16:34:30.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.624856+0000", 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71483768199999997 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.624864+0000", 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5909999999999998e-06 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.624943+0000", 2024-09-17T16:34:30.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9234999999999996e-05 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.625347+0000", 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040387099999999998 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.15 740/739 [PushOp(8:a91ad1da:::benchmark_data_smithi173_130566_object2731:head, version: 732'202, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:a91ad1da:::benchmark_data_smithi173_130566_object2731:head@732'202, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.907347525, 2024-09-17T16:34:30.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448453453, 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097185+0000", 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2799999999999995e-07 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097245+0000", 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9749e-05 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545252+0000", 2024-09-17T16:34:30.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44800729700000003 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545267+0000", 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5017999999999999e-05 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545346+0000", 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9491000000000002e-05 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545357+0000", 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0491e-05 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545638+0000", 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028057899999999998 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.13 740/739 [PushOp(8:cce6ff3a:::benchmark_data_smithi173_130566_object3681:head, version: 732'214, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:cce6ff3a:::benchmark_data_smithi173_130566_object3681:head@732'214, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:34:30.107 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.9054050359999999, 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448568835, 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:30.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7899999999999997e-07 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099181+0000", 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3591999999999998e-05 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547347+0000", 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448166224 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547363+0000", 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5364000000000001e-05 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547441+0000", 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8873000000000002e-05 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547450+0000", 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5469999999999997e-06 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547695+0000", 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024535600000000001 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:30.111 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:34:30.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:30.112 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:29 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:34:30.113 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:29 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:34:30.113 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:29 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 251/24030 objects degraded (1.045%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:34:30.113 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:29 smithi071 ceph-mon[30002]: osdmap e742: 12 total, 11 up, 12 in 2024-09-17T16:34:30.113 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2099160923' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:30.113 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:29 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:30.113 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2099160923' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:34:30.113 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:29 smithi071 ceph-mon[30002]: osdmap e743: 12 total, 11 up, 12 in 2024-09-17T16:34:30.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:30.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:30.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:30.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:30.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:30.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:30.381 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:30.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:30.402 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:34:30.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:30.504 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:30.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:30.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:30.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 8040 8024 26.392 0 - 0.0104186 2024-09-17T16:34:30.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:30.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:30.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:30.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:30.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:30.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:30.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:30.874 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:30.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:30.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:34:30.934 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:30 smithi173 ceph-mon[30530]: 8.13 scrub starts 2024-09-17T16:34:30.934 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:30 smithi173 ceph-mon[30530]: pgmap v1653: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 7 active+recovering+undersized+remapped, 2 stale+active+clean, 1 active+undersized+degraded, 1 stale+remapped+peering, 13 active+clean; 502 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 499/24063 objects degraded (2.074%); 3438/24063 objects misplaced (14.287%); 8.5 MiB/s, 135 objects/s recovering 2024-09-17T16:34:30.934 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:30 smithi173 ceph-mon[30530]: osdmap e744: 12 total, 11 up, 12 in 2024-09-17T16:34:30.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:30 smithi017 ceph-mon[27571]: 8.13 scrub starts 2024-09-17T16:34:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:30 smithi017 ceph-mon[27571]: pgmap v1653: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 7 active+recovering+undersized+remapped, 2 stale+active+clean, 1 active+undersized+degraded, 1 stale+remapped+peering, 13 active+clean; 502 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 499/24063 objects degraded (2.074%); 3438/24063 objects misplaced (14.287%); 8.5 MiB/s, 135 objects/s recovering 2024-09-17T16:34:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:30 smithi017 ceph-mon[27571]: osdmap e744: 12 total, 11 up, 12 in 2024-09-17T16:34:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:30.997 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:30.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:30 smithi071 ceph-mon[30002]: 8.13 scrub starts 2024-09-17T16:34:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:30 smithi071 ceph-mon[30002]: pgmap v1653: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 7 active+recovering+undersized+remapped, 2 stale+active+clean, 1 active+undersized+degraded, 1 stale+remapped+peering, 13 active+clean; 502 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 499/24063 objects degraded (2.074%); 3438/24063 objects misplaced (14.287%); 8.5 MiB/s, 135 objects/s recovering 2024-09-17T16:34:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:30 smithi071 ceph-mon[30002]: osdmap e744: 12 total, 11 up, 12 in 2024-09-17T16:34:31.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:31.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:31.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:31.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:31.247 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:31.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:31.290 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:31.302 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:31 smithi173 podman[132363]: 2024-09-17 16:34:31.042522264 +0000 UTC m=+0.018026857 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:34:31.302 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:31 smithi173 podman[132363]: 2024-09-17 16:34:31.176824185 +0000 UTC m=+0.152328762 container create e3dbf31532385516cce2698a9d4dc5a093bb1a7cc312db3cb6933eaf104b6506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T16:34:31.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:31.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:31.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:31.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:31.476 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:31.499 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:31.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:31.588 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:34:31.601 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:31.614 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:34:31.613506+0000 min lat: 0.00112054 max lat: 15.0115 avg lat: 0.0380748 2024-09-17T16:34:31.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:34:31.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 8340 8324 26.0098 1.33929 0.0581951 0.0380748 2024-09-17T16:34:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:31.645 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:31.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:31.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:31 smithi173 podman[132363]: 2024-09-17 16:34:31.429891205 +0000 UTC m=+0.405395790 container init e3dbf31532385516cce2698a9d4dc5a093bb1a7cc312db3cb6933eaf104b6506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-17T16:34:31.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:31 smithi173 podman[132363]: 2024-09-17 16:34:31.434340429 +0000 UTC m=+0.409845005 container start e3dbf31532385516cce2698a9d4dc5a093bb1a7cc312db3cb6933eaf104b6506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16) 2024-09-17T16:34:31.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:31 smithi173 podman[132363]: 2024-09-17 16:34:31.53003241 +0000 UTC m=+0.505536995 container attach e3dbf31532385516cce2698a9d4dc5a093bb1a7cc312db3cb6933eaf104b6506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-17T16:34:31.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:31.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:31.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:31.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:31.832 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:31 smithi017 ceph-mon[27571]: 8.17 scrub starts 2024-09-17T16:34:31.832 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:31 smithi017 ceph-mon[27571]: osdmap e745: 12 total, 11 up, 12 in 2024-09-17T16:34:31.861 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:34:31.862 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:34:31.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:31.883 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:34:31.883 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:34:31.898 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:31.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:31.907 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:34:31.907 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:34:31.928 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:34:31.928 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:34:31.954 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:34:31.955 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:34:31.977 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:34:31.977 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:34:32.001 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:32.001 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:34:32.002 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:34:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:32.023 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:32.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:32.024 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:34:32.024 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:34:32.048 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:34:32.048 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:34:32.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:31 smithi071 ceph-mon[30002]: 8.17 scrub starts 2024-09-17T16:34:32.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:31 smithi071 ceph-mon[30002]: osdmap e745: 12 total, 11 up, 12 in 2024-09-17T16:34:32.070 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:34:32.071 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:34:32.071 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:34:32.092 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:32.092 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:32.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:32.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:31 smithi173 ceph-mon[30530]: 8.17 scrub starts 2024-09-17T16:34:32.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:31 smithi173 ceph-mon[30530]: osdmap e745: 12 total, 11 up, 12 in 2024-09-17T16:34:32.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:34:31 smithi173 podman[132363]: 2024-09-17 16:34:31.839945178 +0000 UTC m=+0.815449764 container died e3dbf31532385516cce2698a9d4dc5a093bb1a7cc312db3cb6933eaf104b6506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:34:32.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:32.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:32.248 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:32.270 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:32.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:32.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:32.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:32.395 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:32.496 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:32.518 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:32.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:32.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.10.service' 2024-09-17T16:34:32.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:34:32.606 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-17T16:34:32.606 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.10 2024-09-17T16:34:32.606 INFO:tasks.thrashosds.thrasher:remote for osd 10 is ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T16:34:32.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.10 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op list-pgs 2024-09-17T16:34:32.614 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 8818 8802 26.1937 29.875 0.001442 0.0369142 2024-09-17T16:34:32.620 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:32.642 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:32.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:32.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:32.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:32.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:32.843 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:32 smithi173 ceph-mon[30530]: 8.14 scrub starts 2024-09-17T16:34:32.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:32 smithi173 ceph-mon[30530]: 8.19 scrub starts 2024-09-17T16:34:32.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:32 smithi173 ceph-mon[30530]: 8.15 scrub starts 2024-09-17T16:34:32.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/275409083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:32.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:32 smithi173 ceph-mon[30530]: pgmap v1656: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 2 peering, 6 active+recovering+undersized+remapped, 2 stale+active+clean, 2 active+undersized+degraded, 13 active+clean; 502 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s wr, 59 op/s; 1009/24063 objects degraded (4.193%); 3320/24063 objects misplaced (13.797%); 4.7 MiB/s, 74 objects/s recovering 2024-09-17T16:34:32.844 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:32 smithi173 ceph-mon[30530]: osdmap e746: 12 total, 11 up, 12 in 2024-09-17T16:34:32.850 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.10/config 2024-09-17T16:34:32.876 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:32.877 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:32.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:32.909 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:32.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:32.937 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:32 smithi017 ceph-mon[27571]: 8.14 scrub starts 2024-09-17T16:34:32.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:32 smithi017 ceph-mon[27571]: 8.19 scrub starts 2024-09-17T16:34:32.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:32 smithi017 ceph-mon[27571]: 8.15 scrub starts 2024-09-17T16:34:32.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/275409083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:32.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:32 smithi017 ceph-mon[27571]: pgmap v1656: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 2 peering, 6 active+recovering+undersized+remapped, 2 stale+active+clean, 2 active+undersized+degraded, 13 active+clean; 502 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s wr, 59 op/s; 1009/24063 objects degraded (4.193%); 3320/24063 objects misplaced (13.797%); 4.7 MiB/s, 74 objects/s recovering 2024-09-17T16:34:32.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:32 smithi017 ceph-mon[27571]: osdmap e746: 12 total, 11 up, 12 in 2024-09-17T16:34:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:34:32.974 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:34:33.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:33.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:33.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:32 smithi071 ceph-mon[30002]: 8.14 scrub starts 2024-09-17T16:34:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:32 smithi071 ceph-mon[30002]: 8.19 scrub starts 2024-09-17T16:34:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:32 smithi071 ceph-mon[30002]: 8.15 scrub starts 2024-09-17T16:34:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/275409083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:32 smithi071 ceph-mon[30002]: pgmap v1656: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 2 peering, 6 active+recovering+undersized+remapped, 2 stale+active+clean, 2 active+undersized+degraded, 13 active+clean; 502 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s wr, 59 op/s; 1009/24063 objects degraded (4.193%); 3320/24063 objects misplaced (13.797%); 4.7 MiB/s, 74 objects/s recovering 2024-09-17T16:34:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:32 smithi071 ceph-mon[30002]: osdmap e746: 12 total, 11 up, 12 in 2024-09-17T16:34:33.137 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:33.168 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:33.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:33.255 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:34:33.271 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:33.332 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:33.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:33.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:33.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:33.463 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:33.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:33.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:34:33.566 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:33.597 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:33.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:33.598 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:33.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 10778 10762 30.5708 122.5 0.00235859 0.0326928 2024-09-17T16:34:33.699 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:33.721 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:33.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:33.822 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:33.843 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:33.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:33.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:33 smithi173 ceph-mon[30530]: 8.18 scrub starts 2024-09-17T16:34:33.851 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:33 smithi173 ceph-mon[30530]: 8.12 scrub starts 2024-09-17T16:34:33.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:33.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:33.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:33 smithi017 ceph-mon[27571]: 8.18 scrub starts 2024-09-17T16:34:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:33 smithi017 ceph-mon[27571]: 8.12 scrub starts 2024-09-17T16:34:34.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:33 smithi071 ceph-mon[30002]: 8.18 scrub starts 2024-09-17T16:34:34.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:33 smithi071 ceph-mon[30002]: 8.12 scrub starts 2024-09-17T16:34:34.071 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:34.093 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:34.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:34.196 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:34.218 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:34.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:34.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:34.285 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:34:34.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:34.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:34.343 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:34.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:34.445 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:34.469 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:34.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:34.572 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:34.594 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:34.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:34.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 11210 11194 30.4154 27 0.0627302 0.0323203 2024-09-17T16:34:34.681 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:34:34.696 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:34.723 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:34.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:34.799 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:34.799 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:34.825 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:34.857 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:34.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:34.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:34.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:34.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:34:34.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:34.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:34 smithi017 ceph-mon[27571]: 8.10 scrub starts 2024-09-17T16:34:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:34 smithi017 ceph-mon[27571]: osdmap e747: 12 total, 11 up, 12 in 2024-09-17T16:34:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:34 smithi017 ceph-mon[27571]: pgmap v1659: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovering+undersized+remapped, 8 active+undersized+degraded, 14 active+clean; 536 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 207 op/s; 2666/25719 objects degraded (10.366%); 1765/25719 objects misplaced (6.863%); 5.9 MiB/s, 94 objects/s recovering 2024-09-17T16:34:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1254530888' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:34 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1009/24063 objects degraded (4.193%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:34 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1254530888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:34.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:34 smithi017 ceph-mon[27571]: osdmap e748: 12 total, 11 up, 12 in 2024-09-17T16:34:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:34 smithi071 ceph-mon[30002]: 8.10 scrub starts 2024-09-17T16:34:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:34 smithi071 ceph-mon[30002]: osdmap e747: 12 total, 11 up, 12 in 2024-09-17T16:34:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:34 smithi071 ceph-mon[30002]: pgmap v1659: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovering+undersized+remapped, 8 active+undersized+degraded, 14 active+clean; 536 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 207 op/s; 2666/25719 objects degraded (10.366%); 1765/25719 objects misplaced (6.863%); 5.9 MiB/s, 94 objects/s recovering 2024-09-17T16:34:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1254530888' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:34 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1009/24063 objects degraded (4.193%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:34 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1254530888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:34 smithi071 ceph-mon[30002]: osdmap e748: 12 total, 11 up, 12 in 2024-09-17T16:34:35.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:35.106 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:35.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:35.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:35.129 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:34 smithi173 ceph-mon[30530]: 8.10 scrub starts 2024-09-17T16:34:35.130 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:34 smithi173 ceph-mon[30530]: osdmap e747: 12 total, 11 up, 12 in 2024-09-17T16:34:35.130 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:34 smithi173 ceph-mon[30530]: pgmap v1659: 27 pgs: 2 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovering+undersized+remapped, 8 active+undersized+degraded, 14 active+clean; 536 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 207 op/s; 2666/25719 objects degraded (10.366%); 1765/25719 objects misplaced (6.863%); 5.9 MiB/s, 94 objects/s recovering 2024-09-17T16:34:35.130 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1254530888' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:35.130 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:34 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1009/24063 objects degraded (4.193%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:35.130 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:34 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:35.130 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1254530888' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:35.130 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:34 smithi173 ceph-mon[30530]: osdmap e748: 12 total, 11 up, 12 in 2024-09-17T16:34:35.208 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:35.237 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:35.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:35.338 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:35.363 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:35.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:35.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:35.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:35.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:35.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:34:35.591 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:35.612 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:35.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:35.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 12923 12907 33.6086 107.062 0.00973475 0.0296894 2024-09-17T16:34:35.693 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:35.714 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:35.737 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:35.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:35.838 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:35.864 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:35.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:35.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:35 smithi017 ceph-mon[27571]: 8.11 scrub starts 2024-09-17T16:34:35.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:35.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:35.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:36.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:35 smithi173 ceph-mon[30530]: 8.11 scrub starts 2024-09-17T16:34:36.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:35 smithi071 ceph-mon[30002]: 8.11 scrub starts 2024-09-17T16:34:36.092 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:36.101 INFO:teuthology.orchestra.run.smithi173.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:34:36.121 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:36.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:36.145 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:34:36.223 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:36.250 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:36.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:36.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:36.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:36.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:36.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:36.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:36.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:36.609 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:36.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 14769 14753 36.8788 115.375 0.00447915 0.0271055 2024-09-17T16:34:36.642 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:36.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:36.723 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:34:36.723 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:34:36.745 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:36.757 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:34:36.757 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:34:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:36.778 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:36.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:36.786 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:34:36.786 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:34:36.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:34:36.816 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:34:36.840 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:34:36.840 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:34:36.867 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:34:36.867 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:34:36.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:36.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:34:36.893 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:34:36.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:36.908 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:36.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:36.921 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:34:36.921 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:34:36.947 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:34:36.947 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:34:36.980 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:34:36.981 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:34:36.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:36 smithi017 ceph-mon[27571]: pgmap v1661: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 644 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 583 op/s; 2454/30900 objects degraded (7.942%); 1164/30900 objects misplaced (3.767%); 6.8 MiB/s, 109 objects/s recovering 2024-09-17T16:34:36.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/858381288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:36.981 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:34:36.994 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:36 smithi173 ceph-mon[30530]: pgmap v1661: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 644 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 583 op/s; 2454/30900 objects degraded (7.942%); 1164/30900 objects misplaced (3.767%); 6.8 MiB/s, 109 objects/s recovering 2024-09-17T16:34:36.994 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/858381288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:37.009 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:37.012 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:37.013 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:37.033 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:37.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:37.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:37.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:36 smithi071 ceph-mon[30002]: pgmap v1661: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 644 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 583 op/s; 2454/30900 objects degraded (7.942%); 1164/30900 objects misplaced (3.767%); 6.8 MiB/s, 109 objects/s recovering 2024-09-17T16:34:37.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/858381288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:37.104 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:34:37.105 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:34:37.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:34:37.105 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:34:37.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.105 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:158720 5.f e512/481 5:f5ceb914:::benchmark_data_smithi173_90735_object75949:head v 512'9748, mlcod=512'9745)", 2024-09-17T16:34:37.105 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:34:37.105 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 598.15869807900003, 2024-09-17T16:34:37.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.233160072, 2024-09-17T16:34:37.105 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.105 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.106 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.106 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:34:37.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857995+0000", 2024-09-17T16:34:37.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.857996+0000", 2024-09-17T16:34:37.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7399999999999991e-07 2024-09-17T16:34:37.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:38.858002+0000", 2024-09-17T16:34:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5030000000000002e-06 2024-09-17T16:34:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087206+0000", 2024-09-17T16:34:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22920367999999999 2024-09-17T16:34:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087213+0000", 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0129999999999996e-06 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087267+0000", 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3978999999999999e-05 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.087289+0000", 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1597999999999999e-05 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.091102+0000", 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038130719999999998 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:39.091155+0000", 2024-09-17T16:34:37.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3253e-05 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26158.0:160454 5.17 e514/470 5:eb54973c:::benchmark_data_smithi173_90735_object77683:head v 514'6398, mlcod=514'6395)", 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 596.28605836999998, 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18539098100000001, 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.111 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730635+0000", 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2900000000000002e-07 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.730643+0000", 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3300000000000001e-06 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913880+0000", 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.183237224 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913887+0000", 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2400000000000001e-06 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913943+0000", 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5859999999999997e-05 2024-09-17T16:34:37.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.913958+0000", 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4854999999999999e-05 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.915994+0000", 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00203572 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:24:40.916026+0000", 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1822999999999997e-05 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:12912 6.d e535/527 6:ba9121f7:::benchmark_data_smithi173_102017_object12911:head v 535'804, mlcod=535'801)", 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 559.312862881, 2024-09-17T16:34:37.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27481599400000001, 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703830+0000", 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703831+0000", 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3340000000000001e-06 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.703859+0000", 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7898e-05 2024-09-17T16:34:37.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977854+0000", 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27399516899999998 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977863+0000", 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6309999999999992e-06 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977932+0000", 2024-09-17T16:34:37.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.923e-05 2024-09-17T16:34:37.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.977956+0000", 2024-09-17T16:34:37.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3606000000000001e-05 2024-09-17T16:34:37.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.978613+0000", 2024-09-17T16:34:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065716699999999995 2024-09-17T16:34:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:25:17.978646+0000", 2024-09-17T16:34:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2959000000000001e-05 2024-09-17T16:34:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.122 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:113603 6.8 e582/527 6:1d7478f4:::benchmark_data_smithi173_102017_object20455:head v 582'7045, mlcod=582'7042)", 2024-09-17T16:34:37.122 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:34:37.122 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.40696666999997, 2024-09-17T16:34:37.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.183501209, 2024-09-17T16:34:37.122 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.123 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.123 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.123 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:34:37.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:34:37.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609728+0000", 2024-09-17T16:34:37.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.176e-06 2024-09-17T16:34:37.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609742+0000", 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3703000000000001e-05 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790389+0000", 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18064696399999999 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790402+0000", 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3239e-05 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790530+0000", 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012778799999999999 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790567+0000", 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6999000000000003e-05 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.793200+0000", 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026325070000000001 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.793227+0000", 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7832999999999999e-05 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.128 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:29928 7.a e646/629 7:54337644:::benchmark_data_smithi173_116396_object29927:head v 646'1856, mlcod=646'1852)", 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 269.72649237500002, 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.232420081, 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290207+0000", 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6579999999999996e-06 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290242+0000", 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4696000000000001e-05 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521561+0000", 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23131880499999999 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521567+0000", 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.392e-06 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521628+0000", 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0473999999999997e-05 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521650+0000", 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2909e-05 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.522588+0000", 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093747100000000005 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.522621+0000", 2024-09-17T16:34:37.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2675999999999997e-05 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:46467 7.b e650/629 7:d098567c:::benchmark_data_smithi173_116396_object46466:head v 650'2942, mlcod=650'2939)", 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 255.853046554, 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.213043449, 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163647+0000", 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:34:37.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163708+0000", 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1152e-05 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.374934+0000", 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21122532999999999 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.374948+0000", 2024-09-17T16:34:37.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4756000000000001e-05 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.375068+0000", 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119771 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.375108+0000", 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9619000000000001e-05 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.376653+0000", 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001545416 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.376690+0000", 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6631000000000002e-05 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:103843 7.b e678/629 7:d0ef6818:::benchmark_data_smithi173_116396_object8243:head v 678'6575, mlcod=678'6572)", 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:25.488955+0000", 2024-09-17T16:34:37.137 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 191.52773824299999, 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.209073274, 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488955+0000", 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488955+0000", 2024-09-17T16:34:37.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488956+0000", 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4610000000000001e-06 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488964+0000", 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2910000000000007e-06 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695150+0000", 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20618561899999999 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695166+0000", 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5926e-05 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695324+0000", 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000157873 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695362+0000", 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8170000000000002e-05 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.697999+0000", 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026369560000000002 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.698028+0000", 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8977999999999999e-05 2024-09-17T16:34:37.141 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 685/677 [PushOp(7:2d0cb49e:::benchmark_data_smithi173_116396_object20958:head, version: 643'1334, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2d0cb49e:::benchmark_data_smithi173_116396_object20958:head@643'1334, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 164.99056880099999, 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25935365100000002, 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026125+0000", 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026176+0000", 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1581e-05 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283828+0000", 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25765147999999999 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283845+0000", 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7275e-05 2024-09-17T16:34:37.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283961+0000", 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011592899999999999 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283974+0000", 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2724e-05 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.285478+0000", 2024-09-17T16:34:37.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015038720000000001 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 685/677 [PushOp(7:295d1d25:::benchmark_data_smithi173_116396_object23683:head, version: 645'1499, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:295d1d25:::benchmark_data_smithi173_116396_object23683:head@645'1499, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 159.46532149800001, 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24965942299999999, 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551375+0000", 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9809999999999997e-06 2024-09-17T16:34:37.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551452+0000", 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7064999999999995e-05 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799411+0000", 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.247958129 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799426+0000", 2024-09-17T16:34:37.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5094e-05 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799521+0000", 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5449e-05 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799534+0000", 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3077e-05 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.801031+0000", 2024-09-17T16:34:37.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014966280000000001 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 687/677 [PushOp(7:2aa25eb8:::benchmark_data_smithi173_116396_object25172:head, version: 645'1590, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2aa25eb8:::benchmark_data_smithi173_116396_object25172:head@645'1590, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 155.86498220600001, 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25723122799999998, 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:34:37.150 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151712+0000", 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3030000000000001e-06 2024-09-17T16:34:37.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151805+0000", 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3330000000000003e-05 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.406885+0000", 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25507977999999998 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.406910+0000", 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4736000000000002e-05 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.407016+0000", 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106372 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.407029+0000", 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3032e-05 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.408942+0000", 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001912675 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 690/677 [PushOp(7:2d367e55:::benchmark_data_smithi173_116396_object28235:head, version: 646'1783, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2d367e55:::benchmark_data_smithi173_116396_object28235:head@646'1783, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.880022559, 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26142421399999999, 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.154 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136671+0000", 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210000000000001e-06 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136740+0000", 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8932999999999999e-05 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396250+0000", 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25950979000000002 2024-09-17T16:34:37.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396266+0000", 2024-09-17T16:34:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5874000000000001e-05 2024-09-17T16:34:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396362+0000", 2024-09-17T16:34:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5543000000000003e-05 2024-09-17T16:34:37.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396374+0000", 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2897999999999999e-05 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.398095+0000", 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017201549999999999 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:123977 7.14 e690/677 7:2b1a1b49:::benchmark_data_smithi173_116396_object28377:head v 690'7133, mlcod=646'1781)", 2024-09-17T16:34:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:34:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.57648831500001, 2024-09-17T16:34:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.206586876, 2024-09-17T16:34:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:34:37.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440212+0000", 2024-09-17T16:34:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2199999999999997e-06 2024-09-17T16:34:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.641936+0000", 2024-09-17T16:34:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20172389900000001 2024-09-17T16:34:37.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.641943+0000", 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1010000000000003e-06 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.641998+0000", 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5139999999999997e-05 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.642025+0000", 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7662999999999999e-05 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.646753+0000", 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047274800000000001 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.646791+0000", 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8574e-05 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.163 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:129321 7.b e695/629 7:d9dcd2c1:::benchmark_data_smithi173_116396_object33721:head v 695'8200, mlcod=695'8195)", 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:17.838831+0000", 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 139.17786174899999, 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18822972700000001, 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838831+0000", 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838831+0000", 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838832+0000", 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.054e-06 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838839+0000", 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7059999999999998e-06 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022112+0000", 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18327316599999999 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022120+0000", 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9039999999999992e-06 2024-09-17T16:34:37.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022180+0000", 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0446999999999999e-05 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022202+0000", 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1750999999999999e-05 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.027011+0000", 2024-09-17T16:34:37.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048087299999999998 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.027061+0000", 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9969000000000002e-05 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:129340 7.b e695/629 7:d6eb756c:::benchmark_data_smithi173_116396_object33740:head v 695'8201, mlcod=695'8197)", 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:34:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 139.14886587699999, 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.190853825, 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4700000000000002e-07 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867833+0000", 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2320000000000001e-06 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054604+0000", 2024-09-17T16:34:37.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18677139100000001 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054610+0000", 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1029999999999998e-06 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054667+0000", 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6928000000000001e-05 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054688+0000", 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0653e-05 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.058610+0000", 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039225379999999997 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.058681+0000", 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0532999999999997e-05 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 696/677 [PushOp(7:2bba9c3c:::benchmark_data_smithi173_116396_object40626:head, version: 650'2606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2bba9c3c:::benchmark_data_smithi173_116396_object40626:head@650'2606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:30.395026+0000", 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 126.62166718100001, 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19917216800000001, 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395026+0000", 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395026+0000", 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395027+0000", 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.421e-06 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395110+0000", 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2409000000000003e-05 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591821+0000", 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19671128299999999 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591844+0000", 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.34e-05 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591949+0000", 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010453700000000001 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591961+0000", 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2354000000000001e-05 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.594198+0000", 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002236764 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.176 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:170897 7.5 e712/674 7:a2dbce43:::benchmark_data_smithi173_116396_object75297:head v 712'8535, mlcod=712'8532)", 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 70.577950509999994, 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34638644600000001, 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438743+0000", 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0529999999999999e-06 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438750+0000", 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7989999999999996e-06 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.782943+0000", 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34419256799999998 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.782967+0000", 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3929000000000001e-05 2024-09-17T16:34:37.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.783013+0000", 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6400999999999998e-05 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.783027+0000", 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4337e-05 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785093+0000", 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002065271 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785129+0000", 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6087999999999997e-05 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:178680 7.13 e719/714 7:cae8152e:::benchmark_data_smithi173_116396_object83080:head v 719'8741, mlcod=719'8738)", 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:34:37.181 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.930635461999998, 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31485432099999999, 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:34:37.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086058+0000", 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.85e-07 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086063+0000", 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0050000000000004e-06 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398605+0000", 2024-09-17T16:34:37.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31254172400000002 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398610+0000", 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2789999999999998e-06 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398655+0000", 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4715000000000001e-05 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398670+0000", 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4895999999999999e-05 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.400870+0000", 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022001920000000001 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.400912+0000", 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1625000000000002e-05 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.185 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:191187 7.14 e724/717 7:2d119a99:::benchmark_data_smithi173_116396_object95587:head v 724'9235, mlcod=724'9232)", 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 42.569509252000003, 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48898925799999998, 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7599999999999996e-07 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447190+0000", 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9440000000000002e-06 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933730+0000", 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48653948800000002 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933739+0000", 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9199999999999993e-06 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933837+0000", 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8004999999999994e-05 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933856+0000", 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.891e-05 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.936137+0000", 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022811960000000001 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.936173+0000", 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6019000000000002e-05 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:1442 8.f e731/728 8:f297aed2:::benchmark_data_smithi173_130566_object1441:head v 731'97, mlcod=731'92)", 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:12.402004+0000", 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 24.61468889, 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.188874654, 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.190 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402004+0000", 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402004+0000", 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402005+0000", 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3200000000000004e-07 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402059+0000", 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4203000000000003e-05 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590370+0000", 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18831092399999999 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590374+0000", 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.177e-06 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590443+0000", 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8479000000000006e-05 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590466+0000", 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2901999999999999e-05 2024-09-17T16:34:37.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590851+0000", 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038551500000000001 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590879+0000", 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7622000000000001e-05 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.16 740/739 [PushOp(8:68fc3949:::benchmark_data_smithi173_130566_object1854:head, version: 732'115, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:68fc3949:::benchmark_data_smithi173_130566_object1854:head@732'115, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.117452193, 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.75943818799999996, 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5700000000000002e-07 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899299+0000", 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7451999999999998e-05 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658299+0000", 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.75900059799999997 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658317+0000", 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658403+0000", 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6600000000000004e-05 2024-09-17T16:34:37.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658413+0000", 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9340000000000001e-06 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658679+0000", 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026586099999999999 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:34:37.198 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:34:37.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:37.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:37.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:37.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:37.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:37.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:37.450 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:37.482 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:37.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:37.584 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:37.607 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:37.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:37.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 16435 16419 39.4649 104.125 0.00141247 0.0252908 2024-09-17T16:34:37.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:34:37.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:37.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:37.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:37.833 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:37.855 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:37.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:37.956 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:37.983 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:37.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:38.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:34:38.085 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:38.107 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:38.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:38.209 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:38.241 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:38.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:34:38.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:34:38.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:38.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:38.372 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:38.373 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:38.380 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:38.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:38.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:38.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:38.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:38.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 17775 17759 41.1048 83.75 0.00154573 0.024307 2024-09-17T16:34:38.621 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:38.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:38.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:38.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:38.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:38.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:38.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:38.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:38.907 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:34:38.907 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:34:38.908 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:34:38.908 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:34:38.972 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:38.996 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:38.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:39.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:39.128 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:39.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:39.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:39.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:39 smithi017 ceph-mon[27571]: pgmap v1662: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 826 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 912 op/s; 3575/39648 objects degraded (9.017%); 1164/39648 objects misplaced (2.936%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:34:39.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:39.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:39.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:39 smithi173 ceph-mon[30530]: pgmap v1662: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 826 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 912 op/s; 3575/39648 objects degraded (9.017%); 1164/39648 objects misplaced (2.936%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:34:39.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:39 smithi071 ceph-mon[30002]: pgmap v1662: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 826 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 912 op/s; 3575/39648 objects degraded (9.017%); 1164/39648 objects misplaced (2.936%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:34:39.360 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:39.390 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:39.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:39.491 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:39.520 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:39.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:39.581 INFO:teuthology.orchestra.run.smithi173.stdout:8.8 2024-09-17T16:34:39.581 INFO:teuthology.orchestra.run.smithi173.stdout:8.7 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.5 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.4 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.2 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:7.19 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.6 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.15 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.1 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.11 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:7.11 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.12 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.10 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.18 2024-09-17T16:34:39.582 INFO:teuthology.orchestra.run.smithi173.stdout:8.19 2024-09-17T16:34:39.583 INFO:teuthology.orchestra.run.smithi173.stdout:8.14 2024-09-17T16:34:39.583 INFO:teuthology.orchestra.run.smithi173.stdout:8.9 2024-09-17T16:34:39.583 INFO:teuthology.orchestra.run.smithi173.stdout:8.16 2024-09-17T16:34:39.583 INFO:teuthology.orchestra.run.smithi173.stdout:7.12 2024-09-17T16:34:39.583 INFO:teuthology.orchestra.run.smithi173.stdout:8.17 2024-09-17T16:34:39.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 19266 19250 42.9646 93.1875 0.00153078 0.023199 2024-09-17T16:34:39.622 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:39.653 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:39.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:39.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:39.758 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:34:39.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:39.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:39.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:39.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:39.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:40.008 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:40.034 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:40.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:40.045 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:40.127 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:34:40.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:40.161 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:40.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:40.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:40.263 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:40.289 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:40.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:40.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:34:40.390 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:40.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3950521731' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:34:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:40 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 3575/39648 objects degraded (9.017%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:40.417 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:40.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3950521731' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:34:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:40 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 3575/39648 objects degraded (9.017%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:40.519 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:40.542 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:40.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3950521731' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:34:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:40 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 3575/39648 objects degraded (9.017%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:40.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.10 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op export --pgid 8.9 --file /var/log/ceph/exp.8.9.10 2024-09-17T16:34:40.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 20281 20265 43.6704 63.4375 0.00134925 0.0228799 2024-09-17T16:34:40.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:40.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:40.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:40.773 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:40.804 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:40.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:40.824 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.10/config 2024-09-17T16:34:40.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:40.943 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:40.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:41.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:41.059 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:41.062 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:34:41.062 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:34:41.062 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:34:41.062 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:34:41.062 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:34:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:41.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:41.073 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:41.174 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:41.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:41 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:34:41.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:41 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:34:41.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:41 smithi017 ceph-mon[27571]: pgmap v1663: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s; 4041/50592 objects degraded (7.987%); 1397/50592 objects misplaced (2.761%); 1.5 MiB/s, 24 objects/s recovering 2024-09-17T16:34:41.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:41 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:41.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3950521731' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:34:41.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:41 smithi017 ceph-mon[27571]: osdmap e749: 12 total, 11 up, 12 in 2024-09-17T16:34:41.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:41 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:34:41.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:41 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:34:41.200 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:41.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:41.302 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:41.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:41 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:34:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:41 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:34:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:41 smithi071 ceph-mon[30002]: pgmap v1663: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s; 4041/50592 objects degraded (7.987%); 1397/50592 objects misplaced (2.761%); 1.5 MiB/s, 24 objects/s recovering 2024-09-17T16:34:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:41 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3950521731' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:34:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:41 smithi071 ceph-mon[30002]: osdmap e749: 12 total, 11 up, 12 in 2024-09-17T16:34:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:41 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:34:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:41 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:34:41.324 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:41.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:41.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:41 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:34:41.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:41 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:34:41.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:41 smithi173 ceph-mon[30530]: pgmap v1663: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.0 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s; 4041/50592 objects degraded (7.987%); 1397/50592 objects misplaced (2.761%); 1.5 MiB/s, 24 objects/s recovering 2024-09-17T16:34:41.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:41 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:41.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3950521731' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:34:41.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:41 smithi173 ceph-mon[30530]: osdmap e749: 12 total, 11 up, 12 in 2024-09-17T16:34:41.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:41 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:34:41.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:41 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:34:41.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:41.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:41.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:41.556 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:41.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:41.586 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:41.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:41.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 21901 21885 45.5894 101.25 0.0183657 0.0219151 2024-09-17T16:34:41.624 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:34:41.625 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:34:41.646 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:34:41.646 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:34:41.666 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:34:41.667 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:34:41.687 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:41.687 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:34:41.688 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:34:41.708 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:34:41.709 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:34:41.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:41.710 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:41.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:41.730 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:34:41.730 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:34:41.757 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:34:41.757 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:34:41.787 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:34:41.787 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:34:41.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:41.815 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:34:41.815 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:34:41.838 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:41.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:41.843 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:34:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:34:41.844 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:34:41.867 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:41.868 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:41.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:41.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:41.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:42.066 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:42.088 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:42.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:42.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:42.216 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:42.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:42 smithi071 ceph-mon[30002]: osdmap e750: 12 total, 11 up, 12 in 2024-09-17T16:34:42.430 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2930134274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:42.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:42.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:42 smithi173 ceph-mon[30530]: osdmap e750: 12 total, 11 up, 12 in 2024-09-17T16:34:42.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2930134274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:42.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:34:42.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:42.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:42.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:42.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:42 smithi017 ceph-mon[27571]: osdmap e750: 12 total, 11 up, 12 in 2024-09-17T16:34:42.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2930134274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:42.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:42.584 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:42.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:42.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:42.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 23053 23037 46.4412 72 0.0355171 0.021512 2024-09-17T16:34:42.686 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:34:42.705 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:34:42.709 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:42.710 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:34:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:42.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:42.801 INFO:teuthology.orchestra.run.smithi173.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:34:42.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:42.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:42.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:42.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:42.967 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:42.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:42.972 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:34:43.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:43.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:43.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:43.193 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:43.219 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:43.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:43.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: pgmap v1666: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 4260/53313 objects degraded (7.991%); 1861/53313 objects misplaced (3.491%); 2.8 MiB/s, 45 objects/s recovering 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1201475887,config=1059521453,health=2477371952,kv=1771054438}) 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826608060}) 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067923518}) 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1975243151}) 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2809517915}) 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970863646}) 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3298991146}) 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1146029010}) 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3559228341}) 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3111129397}) 2024-09-17T16:34:43.254 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3934460329}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=960354877}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1293665219}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=878935707}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2172947349}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80498996}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1811531488}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1870733702}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1106427131}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195725361}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256504340}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=753204095}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=474719501}) 2024-09-17T16:34:43.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1131504105}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982212099}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3587473857}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2555877704}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=744441552}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3361926316}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1591235066}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4148771126}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284892277}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644698401}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770508341}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=347982770}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1899107301}) 2024-09-17T16:34:43.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1867983821}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2019539857}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970525012}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1599762552}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988142845}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2845901325}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914204966}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=888956655}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2841094014}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124780751}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2665582643}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941117284}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=574068183}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1334754765}) 2024-09-17T16:34:43.257 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425388146}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1650062770}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2549426099}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3434447334}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1835756167}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=127038776}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=727983957}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196112442}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=2277253564,mdsmap=376187277,mgr=3228875837}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mgr=15,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=72} crc {mgr=2962277780,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=2802672430,mon_config_key=1371808592}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=814365744}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3059113080}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=13,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=66} crc {mon_config_key=1644109530,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=140031495,osdmap=1385027521}) 2024-09-17T16:34:43.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:43 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=2143000616}) 2024-09-17T16:34:43.271 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:43.321 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:43.346 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:43.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:43.405 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: pgmap v1666: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 4260/53313 objects degraded (7.991%); 1861/53313 objects misplaced (3.491%); 2.8 MiB/s, 45 objects/s recovering 2024-09-17T16:34:43.405 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1201475887,config=1059521453,health=2477371952,kv=1771054438}) 2024-09-17T16:34:43.405 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826608060}) 2024-09-17T16:34:43.405 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067923518}) 2024-09-17T16:34:43.405 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1975243151}) 2024-09-17T16:34:43.405 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2809517915}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970863646}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3298991146}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1146029010}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3559228341}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3111129397}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3934460329}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=960354877}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1293665219}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=878935707}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2172947349}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80498996}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1811531488}) 2024-09-17T16:34:43.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1870733702}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1106427131}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195725361}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256504340}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=753204095}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=474719501}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1131504105}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982212099}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3587473857}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2555877704}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=744441552}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3361926316}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1591235066}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4148771126}) 2024-09-17T16:34:43.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284892277}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644698401}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770508341}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=347982770}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1899107301}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1867983821}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2019539857}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970525012}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1599762552}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988142845}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2845901325}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914204966}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=888956655}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2841094014}) 2024-09-17T16:34:43.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124780751}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2665582643}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941117284}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=574068183}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1334754765}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425388146}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1650062770}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2549426099}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3434447334}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1835756167}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=127038776}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=727983957}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196112442}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=2277253564,mdsmap=376187277,mgr=3228875837}) 2024-09-17T16:34:43.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mgr=15,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=72} crc {mgr=2962277780,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=2802672430,mon_config_key=1371808592}) 2024-09-17T16:34:43.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=814365744}) 2024-09-17T16:34:43.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3059113080}) 2024-09-17T16:34:43.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=13,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=66} crc {mon_config_key=1644109530,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=140031495,osdmap=1385027521}) 2024-09-17T16:34:43.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:43 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=2143000616}) 2024-09-17T16:34:43.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:43.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:43.472 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:43.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: pgmap v1666: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 4260/53313 objects degraded (7.991%); 1861/53313 objects misplaced (3.491%); 2.8 MiB/s, 45 objects/s recovering 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1201475887,config=1059521453,health=2477371952,kv=1771054438}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826608060}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067923518}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1975243151}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2809517915}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970863646}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3298991146}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1146029010}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3559228341}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3111129397}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3934460329}) 2024-09-17T16:34:43.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=960354877}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1293665219}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=878935707}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2172947349}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80498996}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1811531488}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1870733702}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1106427131}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195725361}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256504340}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=753204095}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=474719501}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1131504105}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982212099}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3587473857}) 2024-09-17T16:34:43.566 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2555877704}) 2024-09-17T16:34:43.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=744441552}) 2024-09-17T16:34:43.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3361926316}) 2024-09-17T16:34:43.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1591235066}) 2024-09-17T16:34:43.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4148771126}) 2024-09-17T16:34:43.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3284892277}) 2024-09-17T16:34:43.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644698401}) 2024-09-17T16:34:43.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770508341}) 2024-09-17T16:34:43.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=347982770}) 2024-09-17T16:34:43.567 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1899107301}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1867983821}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2019539857}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970525012}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1599762552}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988142845}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2845901325}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=914204966}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=888956655}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2841094014}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124780751}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2665582643}) 2024-09-17T16:34:43.568 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=941117284}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=574068183}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1334754765}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3425388146}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1650062770}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2549426099}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3434447334}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1835756167}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=127038776}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=727983957}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196112442}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=2277253564,mdsmap=376187277,mgr=3228875837}) 2024-09-17T16:34:43.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mgr=15,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=72} crc {mgr=2962277780,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=2802672430,mon_config_key=1371808592}) 2024-09-17T16:34:43.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=814365744}) 2024-09-17T16:34:43.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3059113080}) 2024-09-17T16:34:43.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=13,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=66} crc {mon_config_key=1644109530,monmap=2143941632,nvmeofgw=598337686,osd_metadata=4094628057,osd_pg_creating=140031495,osdmap=1385027521}) 2024-09-17T16:34:43.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:43 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=75} crc {osdmap=2143000616}) 2024-09-17T16:34:43.572 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:43.602 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:43.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:43.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:43.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 24267 24251 47.3608 75.875 0.00716125 0.0211058 2024-09-17T16:34:43.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:43.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:43.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:43.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:43.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:43.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:43.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:43.976 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:43.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:43.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:44.078 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:44.083 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:44.083 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:44.104 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:44.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:44.206 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:44.229 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:44.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:44.330 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:44.356 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:44.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:44.458 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:44.485 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:44.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:44.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:44.610 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:44.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:44.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 25774 25758 48.7795 94.1875 0.00159272 0.0204863 2024-09-17T16:34:44.706 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:34:44.712 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:44.773 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:44.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:44.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:44.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:44.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:45.001 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:45.026 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:45.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:45.128 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:45.150 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:45.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:45.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:45.271 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:34:45.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:45.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:45.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:45.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:45.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:45 smithi173 ceph-mon[30530]: pgmap v1667: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s; 5454/64092 objects degraded (8.510%); 1861/64092 objects misplaced (2.904%) 2024-09-17T16:34:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:45 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 5454/64092 objects degraded (8.510%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:45.398 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:45.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:45.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:45 smithi017 ceph-mon[27571]: pgmap v1667: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s; 5454/64092 objects degraded (8.510%); 1861/64092 objects misplaced (2.904%) 2024-09-17T16:34:45.468 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:45 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 5454/64092 objects degraded (8.510%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:45.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:45.521 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:45.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:45.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:45 smithi071 ceph-mon[30002]: pgmap v1667: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s; 5454/64092 objects degraded (8.510%); 1861/64092 objects misplaced (2.904%) 2024-09-17T16:34:45.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:45 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 5454/64092 objects degraded (8.510%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:45.615 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 26827 26811 49.2803 65.8125 0.00384827 0.0202715 2024-09-17T16:34:45.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:45.651 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:45.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:45.753 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:45.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:45.784 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:45.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:45.872 INFO:teuthology.orchestra.run.smithi173.stderr:Exporting 8.9 info 8.9( v 733'523 (0'0,733'523] local-lis/les=736/737 n=279 ec=728/728 lis/c=736/736 les/c/f=737/737/0 sis=736) 2024-09-17T16:34:45.873 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9000f542:::benchmark_data_smithi173_130566_object696:head# 2024-09-17T16:34:45.874 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90065cee:::benchmark_data_smithi173_130566_object304:head# 2024-09-17T16:34:45.874 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:900c476d:::benchmark_data_smithi173_130566_object7779:head# 2024-09-17T16:34:45.875 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:900cfd0d:::benchmark_data_smithi173_130566_object6959:head# 2024-09-17T16:34:45.875 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:901a591d:::benchmark_data_smithi173_130566_object4853:head# 2024-09-17T16:34:45.875 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:902e275a:::benchmark_data_smithi173_130566_object7433:head# 2024-09-17T16:34:45.876 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:902f7b8b:::benchmark_data_smithi173_130566_object473:head# 2024-09-17T16:34:45.876 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9031940d:::benchmark_data_smithi173_130566_object5458:head# 2024-09-17T16:34:45.876 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9034cce4:::benchmark_data_smithi173_130566_object199:head# 2024-09-17T16:34:45.877 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9038159a:::benchmark_data_smithi173_130566_object3699:head# 2024-09-17T16:34:45.877 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90440db7:::benchmark_data_smithi173_130566_object1859:head# 2024-09-17T16:34:45.878 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90444b71:::benchmark_data_smithi173_130566_object2890:head# 2024-09-17T16:34:45.878 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9045562e:::benchmark_data_smithi173_130566_object5055:head# 2024-09-17T16:34:45.878 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90533b2b:::benchmark_data_smithi173_130566_object4937:head# 2024-09-17T16:34:45.879 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9059eb62:::benchmark_data_smithi173_130566_object5460:head# 2024-09-17T16:34:45.879 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9094c450:::benchmark_data_smithi173_130566_object5913:head# 2024-09-17T16:34:45.880 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:909d5f27:::benchmark_data_smithi173_130566_object3502:head# 2024-09-17T16:34:45.880 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:909f3c7f:::benchmark_data_smithi173_130566_object3749:head# 2024-09-17T16:34:45.881 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90a633ea:::benchmark_data_smithi173_130566_object6599:head# 2024-09-17T16:34:45.881 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90a7dd1a:::benchmark_data_smithi173_130566_object1053:head# 2024-09-17T16:34:45.881 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90bc7c08:::benchmark_data_smithi173_130566_object400:head# 2024-09-17T16:34:45.882 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90bfd592:::benchmark_data_smithi173_130566_object1259:head# 2024-09-17T16:34:45.884 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90c6d3e1:::benchmark_data_smithi173_130566_object3224:head# 2024-09-17T16:34:45.884 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90cb734f:::benchmark_data_smithi173_130566_object4895:head# 2024-09-17T16:34:45.884 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90d0cfd4:::benchmark_data_smithi173_130566_object4354:head# 2024-09-17T16:34:45.885 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90d0f081:::benchmark_data_smithi173_130566_object504:head# 2024-09-17T16:34:45.885 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90ec0701:::benchmark_data_smithi173_130566_object5919:head# 2024-09-17T16:34:45.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:45.886 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90eed5ed:::benchmark_data_smithi173_130566_object4784:head# 2024-09-17T16:34:45.886 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90eed61d:::benchmark_data_smithi173_130566_object6205:head# 2024-09-17T16:34:45.886 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90f204b8:::benchmark_data_smithi173_130566_object5989:head# 2024-09-17T16:34:45.888 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90f3c4fa:::benchmark_data_smithi173_130566_object6165:head# 2024-09-17T16:34:45.888 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90fb628b:::benchmark_data_smithi173_130566_object4833:head# 2024-09-17T16:34:45.888 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:90fcbef4:::benchmark_data_smithi173_130566_object1017:head# 2024-09-17T16:34:45.888 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9103c16a:::benchmark_data_smithi173_130566_object1574:head# 2024-09-17T16:34:45.889 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9107c432:::benchmark_data_smithi173_130566_object1378:head# 2024-09-17T16:34:45.889 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:910a08ef:::benchmark_data_smithi173_130566_object4889:head# 2024-09-17T16:34:45.889 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9111c9f3:::benchmark_data_smithi173_130566_object3256:head# 2024-09-17T16:34:45.890 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91140c99:::benchmark_data_smithi173_130566_object1297:head# 2024-09-17T16:34:45.890 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:911a896e:::benchmark_data_smithi173_130566_object3871:head# 2024-09-17T16:34:45.890 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:911daa4e:::benchmark_data_smithi173_130566_object6030:head# 2024-09-17T16:34:45.891 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:912258fa:::benchmark_data_smithi173_130566_object4777:head# 2024-09-17T16:34:45.891 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9122e2c8:::benchmark_data_smithi173_130566_object7277:head# 2024-09-17T16:34:45.892 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9133ffa1:::benchmark_data_smithi173_130566_object3584:head# 2024-09-17T16:34:45.892 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:913a5502:::benchmark_data_smithi173_130566_object312:head# 2024-09-17T16:34:45.893 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:913d4e17:::benchmark_data_smithi173_130566_object7330:head# 2024-09-17T16:34:45.893 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9140c358:::benchmark_data_smithi173_130566_object6343:head# 2024-09-17T16:34:45.893 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9142519e:::benchmark_data_smithi173_130566_object4378:head# 2024-09-17T16:34:45.894 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9153b175:::benchmark_data_smithi173_130566_object2939:head# 2024-09-17T16:34:45.894 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91577cd5:::benchmark_data_smithi173_130566_object5019:head# 2024-09-17T16:34:45.895 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91585782:::benchmark_data_smithi173_130566_object3816:head# 2024-09-17T16:34:45.895 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91599685:::benchmark_data_smithi173_130566_object2426:head# 2024-09-17T16:34:45.896 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:915ba07d:::benchmark_data_smithi173_130566_object4707:head# 2024-09-17T16:34:45.896 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9160d36f:::benchmark_data_smithi173_130566_object3621:head# 2024-09-17T16:34:45.896 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:916c6d86:::benchmark_data_smithi173_130566_object826:head# 2024-09-17T16:34:45.897 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:916c8e22:::benchmark_data_smithi173_130566_object7990:head# 2024-09-17T16:34:45.897 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9172b9dc:::benchmark_data_smithi173_130566_object4566:head# 2024-09-17T16:34:45.897 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9176e5ad:::benchmark_data_smithi173_130566_object5031:head# 2024-09-17T16:34:45.898 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:917c98b3:::benchmark_data_smithi173_130566_object1298:head# 2024-09-17T16:34:45.898 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91821118:::benchmark_data_smithi173_130566_object605:head# 2024-09-17T16:34:45.899 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:918a7ea8:::benchmark_data_smithi173_130566_object1301:head# 2024-09-17T16:34:45.899 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91932c14:::benchmark_data_smithi173_130566_object1110:head# 2024-09-17T16:34:45.900 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:919b24dd:::benchmark_data_smithi173_130566_object1205:head# 2024-09-17T16:34:45.900 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91a39ae6:::benchmark_data_smithi173_130566_object7820:head# 2024-09-17T16:34:45.900 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91b4ae05:::benchmark_data_smithi173_130566_object2402:head# 2024-09-17T16:34:45.901 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91b698ab:::benchmark_data_smithi173_130566_object4718:head# 2024-09-17T16:34:45.901 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91b927e7:::benchmark_data_smithi173_130566_object2795:head# 2024-09-17T16:34:45.902 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91bc039e:::benchmark_data_smithi173_130566_object2021:head# 2024-09-17T16:34:45.902 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91bcce82:::benchmark_data_smithi173_130566_object5965:head# 2024-09-17T16:34:45.902 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91c29b2f:::benchmark_data_smithi173_130566_object6594:head# 2024-09-17T16:34:45.903 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91c3a76e:::benchmark_data_smithi173_130566_object813:head# 2024-09-17T16:34:45.903 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91c67866:::benchmark_data_smithi173_130566_object786:head# 2024-09-17T16:34:45.903 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91c6b8bb:::benchmark_data_smithi173_130566_object4423:head# 2024-09-17T16:34:45.904 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91f01e97:::benchmark_data_smithi173_130566_object1541:head# 2024-09-17T16:34:45.905 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91f21c1e:::benchmark_data_smithi173_130566_object5136:head# 2024-09-17T16:34:45.905 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:91fc5a89:::benchmark_data_smithi173_130566_object2906:head# 2024-09-17T16:34:45.905 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9202cd53:::benchmark_data_smithi173_130566_object5937:head# 2024-09-17T16:34:45.906 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9205becd:::benchmark_data_smithi173_130566_object7461:head# 2024-09-17T16:34:45.906 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9205e63c:::benchmark_data_smithi173_130566_object7354:head# 2024-09-17T16:34:45.906 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:920676b5:::benchmark_data_smithi173_130566_object582:head# 2024-09-17T16:34:45.907 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9211482e:::benchmark_data_smithi173_130566_object1923:head# 2024-09-17T16:34:45.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:45.908 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9227df93:::benchmark_data_smithi173_130566_object3202:head# 2024-09-17T16:34:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:45.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:45.909 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:922ac0b7:::benchmark_data_smithi173_130566_object3549:head# 2024-09-17T16:34:45.909 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9233f812:::benchmark_data_smithi173_130566_object6091:head# 2024-09-17T16:34:45.910 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:923c4e0d:::benchmark_data_smithi173_130566_object6710:head# 2024-09-17T16:34:45.910 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9247da57:::benchmark_data_smithi173_130566_object1210:head# 2024-09-17T16:34:45.910 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:927258fd:::benchmark_data_smithi173_130566_object5027:head# 2024-09-17T16:34:45.910 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9284392c:::benchmark_data_smithi173_130566_object3534:head# 2024-09-17T16:34:45.910 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:928c3a93:::benchmark_data_smithi173_130566_object5000:head# 2024-09-17T16:34:45.911 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:929206fe:::benchmark_data_smithi173_130566_object854:head# 2024-09-17T16:34:45.911 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:929ad92d:::benchmark_data_smithi173_130566_object5946:head# 2024-09-17T16:34:45.912 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:929eb97d:::benchmark_data_smithi173_130566_object295:head# 2024-09-17T16:34:45.912 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:92a56f61:::benchmark_data_smithi173_130566_object1276:head# 2024-09-17T16:34:45.913 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:92abc0e3:::benchmark_data_smithi173_130566_object5448:head# 2024-09-17T16:34:45.914 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:92ade1ec:::benchmark_data_smithi173_130566_object4472:head# 2024-09-17T16:34:45.914 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:92b03dc7:::benchmark_data_smithi173_130566_object5159:head# 2024-09-17T16:34:45.914 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:92ca2570:::benchmark_data_smithi173_130566_object955:head# 2024-09-17T16:34:45.914 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:92d2eceb:::benchmark_data_smithi173_130566_object4015:head# 2024-09-17T16:34:45.915 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:92f6f67c:::benchmark_data_smithi173_130566_object7168:head# 2024-09-17T16:34:45.915 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9305660a:::benchmark_data_smithi173_130566_object1325:head# 2024-09-17T16:34:45.916 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:930ea188:::benchmark_data_smithi173_130566_object3414:head# 2024-09-17T16:34:45.916 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9327d755:::benchmark_data_smithi173_130566_object870:head# 2024-09-17T16:34:45.916 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:932a1564:::benchmark_data_smithi173_130566_object4198:head# 2024-09-17T16:34:45.917 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:932dffdc:::benchmark_data_smithi173_130566_object418:head# 2024-09-17T16:34:45.917 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:932fe421:::benchmark_data_smithi173_130566_object4708:head# 2024-09-17T16:34:45.917 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93324f6d:::benchmark_data_smithi173_130566_object220:head# 2024-09-17T16:34:45.918 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:934a6b3a:::benchmark_data_smithi173_130566_object5241:head# 2024-09-17T16:34:45.918 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:934e680e:::benchmark_data_smithi173_130566_object6423:head# 2024-09-17T16:34:45.919 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:935799aa:::benchmark_data_smithi173_130566_object3782:head# 2024-09-17T16:34:45.919 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:935d20f5:::benchmark_data_smithi173_130566_object1151:head# 2024-09-17T16:34:45.920 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93662906:::benchmark_data_smithi173_130566_object4074:head# 2024-09-17T16:34:45.920 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9367529f:::benchmark_data_smithi173_130566_object5006:head# 2024-09-17T16:34:45.921 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:936f4796:::benchmark_data_smithi173_130566_object5082:head# 2024-09-17T16:34:45.921 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93721618:::benchmark_data_smithi173_130566_object1426:head# 2024-09-17T16:34:45.922 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93768e31:::benchmark_data_smithi173_130566_object3435:head# 2024-09-17T16:34:45.922 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9384cc9f:::benchmark_data_smithi173_130566_object6674:head# 2024-09-17T16:34:45.923 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:938ab075:::benchmark_data_smithi173_130566_object6849:head# 2024-09-17T16:34:45.923 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9392e082:::benchmark_data_smithi173_130566_object3836:head# 2024-09-17T16:34:45.924 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9393489c:::benchmark_data_smithi173_130566_object6598:head# 2024-09-17T16:34:45.925 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93952705:::benchmark_data_smithi173_130566_object7450:head# 2024-09-17T16:34:45.925 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9396b81c:::benchmark_data_smithi173_130566_object3115:head# 2024-09-17T16:34:45.925 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9398a850:::benchmark_data_smithi173_130566_object3081:head# 2024-09-17T16:34:45.926 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93a94d82:::benchmark_data_smithi173_130566_object1956:head# 2024-09-17T16:34:45.926 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93ab35a2:::benchmark_data_smithi173_130566_object2950:head# 2024-09-17T16:34:45.927 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93b48f2d:::benchmark_data_smithi173_130566_object5208:head# 2024-09-17T16:34:45.927 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93b733dc:::benchmark_data_smithi173_130566_object3840:head# 2024-09-17T16:34:45.928 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93b88bab:::benchmark_data_smithi173_130566_object6441:head# 2024-09-17T16:34:45.928 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93c10f07:::benchmark_data_smithi173_130566_object1203:head# 2024-09-17T16:34:45.929 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93c92961:::benchmark_data_smithi173_130566_object2773:head# 2024-09-17T16:34:45.929 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93cc0c1d:::benchmark_data_smithi173_130566_object2661:head# 2024-09-17T16:34:45.930 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93cd4834:::benchmark_data_smithi173_130566_object7522:head# 2024-09-17T16:34:45.930 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93d42013:::benchmark_data_smithi173_130566_object1817:head# 2024-09-17T16:34:45.931 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93d6f7f7:::benchmark_data_smithi173_130566_object4347:head# 2024-09-17T16:34:45.931 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93db84ad:::benchmark_data_smithi173_130566_object7574:head# 2024-09-17T16:34:45.932 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93e4028b:::benchmark_data_smithi173_130566_object1815:head# 2024-09-17T16:34:45.932 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93efd0bf:::benchmark_data_smithi173_130566_object5976:head# 2024-09-17T16:34:45.933 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:93f27987:::benchmark_data_smithi173_130566_object6604:head# 2024-09-17T16:34:45.933 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:940bbe0a:::benchmark_data_smithi173_130566_object4726:head# 2024-09-17T16:34:45.934 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:940c200c:::benchmark_data_smithi173_130566_object5809:head# 2024-09-17T16:34:45.934 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:941770c8:::benchmark_data_smithi173_130566_object1811:head# 2024-09-17T16:34:45.935 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:941942a2:::benchmark_data_smithi173_130566_object840:head# 2024-09-17T16:34:45.936 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:942254a3:::benchmark_data_smithi173_130566_object2009:head# 2024-09-17T16:34:45.936 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9426733b:::benchmark_data_smithi173_130566_object3398:head# 2024-09-17T16:34:45.936 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94277dc3:::benchmark_data_smithi173_130566_object3969:head# 2024-09-17T16:34:45.937 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9429f6bb:::benchmark_data_smithi173_130566_object717:head# 2024-09-17T16:34:45.937 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:943026b7:::benchmark_data_smithi173_130566_object7583:head# 2024-09-17T16:34:45.938 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94435040:::benchmark_data_smithi173_130566_object4667:head# 2024-09-17T16:34:45.938 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94443a53:::benchmark_data_smithi173_130566_object4163:head# 2024-09-17T16:34:45.939 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:944a65e8:::benchmark_data_smithi173_130566_object4829:head# 2024-09-17T16:34:45.939 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9460a9e3:::benchmark_data_smithi173_130566_object2234:head# 2024-09-17T16:34:45.939 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:946899a5:::benchmark_data_smithi173_130566_object2362:head# 2024-09-17T16:34:45.940 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94702b3b:::benchmark_data_smithi173_130566_object4288:head# 2024-09-17T16:34:45.941 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94833472:::benchmark_data_smithi173_130566_object7669:head# 2024-09-17T16:34:45.942 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94a3419b:::benchmark_data_smithi173_130566_object5823:head# 2024-09-17T16:34:45.942 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94a53baa:::benchmark_data_smithi173_130566_object6129:head# 2024-09-17T16:34:45.942 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94b15fd9:::benchmark_data_smithi173_130566_object4883:head# 2024-09-17T16:34:45.942 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94b2e3da:::benchmark_data_smithi173_130566_object2424:head# 2024-09-17T16:34:45.943 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94b3f867:::benchmark_data_smithi173_130566_object4319:head# 2024-09-17T16:34:45.943 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94dc18f2:::benchmark_data_smithi173_130566_object4748:head# 2024-09-17T16:34:45.943 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94de4957:::benchmark_data_smithi173_130566_object7793:head# 2024-09-17T16:34:45.944 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94e030fc:::benchmark_data_smithi173_130566_object6602:head# 2024-09-17T16:34:45.944 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94e7f6cf:::benchmark_data_smithi173_130566_object7653:head# 2024-09-17T16:34:45.945 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:94fe0b78:::benchmark_data_smithi173_130566_object1302:head# 2024-09-17T16:34:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9506315a:::benchmark_data_smithi173_130566_object1492:head# 2024-09-17T16:34:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95101c9d:::benchmark_data_smithi173_130566_object3540:head# 2024-09-17T16:34:45.947 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95108746:::benchmark_data_smithi173_130566_object3978:head# 2024-09-17T16:34:45.947 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95146639:::benchmark_data_smithi173_130566_object7275:head# 2024-09-17T16:34:45.948 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:951fac3b:::benchmark_data_smithi173_130566_object7495:head# 2024-09-17T16:34:45.949 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95203a24:::benchmark_data_smithi173_130566_object4769:head# 2024-09-17T16:34:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95223cfb:::benchmark_data_smithi173_130566_object4147:head# 2024-09-17T16:34:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95268507:::benchmark_data_smithi173_130566_object3076:head# 2024-09-17T16:34:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95272860:::benchmark_data_smithi173_130566_object950:head# 2024-09-17T16:34:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:952a6969:::benchmark_data_smithi173_130566_object2388:head# 2024-09-17T16:34:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:952ae84d:::benchmark_data_smithi173_130566_object5988:head# 2024-09-17T16:34:45.951 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9532413d:::benchmark_data_smithi173_130566_object363:head# 2024-09-17T16:34:45.951 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95390790:::benchmark_data_smithi173_130566_object2241:head# 2024-09-17T16:34:45.951 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:953a8e06:::benchmark_data_smithi173_130566_object3475:head# 2024-09-17T16:34:45.952 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:954cf359:::benchmark_data_smithi173_130566_object2304:head# 2024-09-17T16:34:45.953 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:955141d6:::benchmark_data_smithi173_130566_object1557:head# 2024-09-17T16:34:45.953 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9551431e:::benchmark_data_smithi173_130566_object3023:head# 2024-09-17T16:34:45.953 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9555cfa7:::benchmark_data_smithi173_130566_object2715:head# 2024-09-17T16:34:45.954 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:955ca9b2:::benchmark_data_smithi173_130566_object5695:head# 2024-09-17T16:34:45.954 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:955e7c35:::benchmark_data_smithi173_130566_object358:head# 2024-09-17T16:34:45.954 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:955fcbb7:::benchmark_data_smithi173_130566_object7615:head# 2024-09-17T16:34:45.955 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:956097c2:::benchmark_data_smithi173_130566_object7470:head# 2024-09-17T16:34:45.955 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9567ac75:::benchmark_data_smithi173_130566_object7563:head# 2024-09-17T16:34:45.956 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:957452a4:::benchmark_data_smithi173_130566_object7697:head# 2024-09-17T16:34:45.956 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:958224f3:::benchmark_data_smithi173_130566_object2351:head# 2024-09-17T16:34:45.956 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9583c600:::benchmark_data_smithi173_130566_object2079:head# 2024-09-17T16:34:45.957 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95877418:::benchmark_data_smithi173_130566_object7871:head# 2024-09-17T16:34:45.957 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:958aff29:::benchmark_data_smithi173_130566_object55:head# 2024-09-17T16:34:45.958 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:958d2f4b:::benchmark_data_smithi173_130566_object7222:head# 2024-09-17T16:34:45.958 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:958eefd6:::benchmark_data_smithi173_130566_object655:head# 2024-09-17T16:34:45.959 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:959dab3c:::benchmark_data_smithi173_130566_object6479:head# 2024-09-17T16:34:45.959 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:959ff3fc:::benchmark_data_smithi173_130566_object7190:head# 2024-09-17T16:34:45.960 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95a36c91:::benchmark_data_smithi173_130566_object888:head# 2024-09-17T16:34:45.960 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95aa3975:::benchmark_data_smithi173_130566_object7331:head# 2024-09-17T16:34:45.961 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95aa8d85:::benchmark_data_smithi173_130566_object3080:head# 2024-09-17T16:34:45.961 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95b6c2c6:::benchmark_data_smithi173_130566_object3543:head# 2024-09-17T16:34:45.961 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95cab347:::benchmark_data_smithi173_130566_object132:head# 2024-09-17T16:34:45.962 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95cd6b7a:::benchmark_data_smithi173_130566_object5609:head# 2024-09-17T16:34:45.962 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95d1742a:::benchmark_data_smithi173_130566_object7943:head# 2024-09-17T16:34:45.963 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95d6e0a0:::benchmark_data_smithi173_130566_object3904:head# 2024-09-17T16:34:45.963 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95df2076:::benchmark_data_smithi173_130566_object7390:head# 2024-09-17T16:34:45.964 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95f21f8c:::benchmark_data_smithi173_130566_object6942:head# 2024-09-17T16:34:45.964 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95f389a5:::benchmark_data_smithi173_130566_object3993:head# 2024-09-17T16:34:45.964 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95fc745c:::benchmark_data_smithi173_130566_object3548:head# 2024-09-17T16:34:45.965 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:95fde072:::benchmark_data_smithi173_130566_object166:head# 2024-09-17T16:34:45.965 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9605c958:::benchmark_data_smithi173_130566_object4661:head# 2024-09-17T16:34:45.966 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96064b8a:::benchmark_data_smithi173_130566_object3326:head# 2024-09-17T16:34:45.966 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9606ffe0:::benchmark_data_smithi173_130566_object596:head# 2024-09-17T16:34:45.967 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:960af516:::benchmark_data_smithi173_130566_object5812:head# 2024-09-17T16:34:45.967 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:961151fe:::benchmark_data_smithi173_130566_object1743:head# 2024-09-17T16:34:45.968 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96190caf:::benchmark_data_smithi173_130566_object2154:head# 2024-09-17T16:34:45.968 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:962bbb6f:::benchmark_data_smithi173_130566_object7575:head# 2024-09-17T16:34:45.969 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:962d323c:::benchmark_data_smithi173_130566_object242:head# 2024-09-17T16:34:45.969 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96400829:::benchmark_data_smithi173_130566_object4007:head# 2024-09-17T16:34:45.970 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9641966d:::benchmark_data_smithi173_130566_object3557:head# 2024-09-17T16:34:45.970 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96450fe3:::benchmark_data_smithi173_130566_object6306:head# 2024-09-17T16:34:45.970 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:964e86f4:::benchmark_data_smithi173_130566_object3357:head# 2024-09-17T16:34:45.971 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9655c769:::benchmark_data_smithi173_130566_object5687:head# 2024-09-17T16:34:45.971 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9656b08e:::benchmark_data_smithi173_130566_object3390:head# 2024-09-17T16:34:45.972 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96577423:::benchmark_data_smithi173_130566_object7279:head# 2024-09-17T16:34:45.972 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:965d9baf:::benchmark_data_smithi173_130566_object519:head# 2024-09-17T16:34:45.973 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96646488:::benchmark_data_smithi173_130566_object2372:head# 2024-09-17T16:34:45.973 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:966e234d:::benchmark_data_smithi173_130566_object2688:head# 2024-09-17T16:34:45.973 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9674d2cc:::benchmark_data_smithi173_130566_object5012:head# 2024-09-17T16:34:45.974 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:967c61f9:::benchmark_data_smithi173_130566_object4374:head# 2024-09-17T16:34:45.974 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9698b093:::benchmark_data_smithi173_130566_object1280:head# 2024-09-17T16:34:45.975 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96999de1:::benchmark_data_smithi173_130566_object6648:head# 2024-09-17T16:34:45.975 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96a1233f:::benchmark_data_smithi173_130566_object4508:head# 2024-09-17T16:34:45.976 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96acd81e:::benchmark_data_smithi173_130566_object839:head# 2024-09-17T16:34:45.976 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96b10b0d:::benchmark_data_smithi173_130566_object5230:head# 2024-09-17T16:34:45.977 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96b97c5a:::benchmark_data_smithi173_130566_object1997:head# 2024-09-17T16:34:45.977 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96b9e363:::benchmark_data_smithi173_130566_object6388:head# 2024-09-17T16:34:45.977 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96c2fc4f:::benchmark_data_smithi173_130566_object5744:head# 2024-09-17T16:34:45.978 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96c9c428:::benchmark_data_smithi173_130566_object6470:head# 2024-09-17T16:34:45.978 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96ce405b:::benchmark_data_smithi173_130566_object7621:head# 2024-09-17T16:34:45.979 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96d86df5:::benchmark_data_smithi173_130566_object3356:head# 2024-09-17T16:34:45.979 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96fbf50d:::benchmark_data_smithi173_130566_object488:head# 2024-09-17T16:34:45.980 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:96fe284e:::benchmark_data_smithi173_130566_object1462:head# 2024-09-17T16:34:45.980 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9702cde2:::benchmark_data_smithi173_130566_object5383:head# 2024-09-17T16:34:45.980 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9710133c:::benchmark_data_smithi173_130566_object3741:head# 2024-09-17T16:34:45.981 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:971434e3:::benchmark_data_smithi173_130566_object2647:head# 2024-09-17T16:34:45.981 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:971a2592:::benchmark_data_smithi173_130566_object2013:head# 2024-09-17T16:34:45.982 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:971aa8ee:::benchmark_data_smithi173_130566_object4805:head# 2024-09-17T16:34:45.983 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:971d5e90:::benchmark_data_smithi173_130566_object4811:head# 2024-09-17T16:34:45.983 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:971ff081:::benchmark_data_smithi173_130566_object6514:head# 2024-09-17T16:34:45.983 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9727ec1c:::benchmark_data_smithi173_130566_object5811:head# 2024-09-17T16:34:45.984 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97294f27:::benchmark_data_smithi173_130566_object1473:head# 2024-09-17T16:34:45.984 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9731c02a:::benchmark_data_smithi173_130566_object123:head# 2024-09-17T16:34:45.984 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:973c9c89:::benchmark_data_smithi173_130566_object1283:head# 2024-09-17T16:34:45.985 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:974e8887:::benchmark_data_smithi173_130566_object2023:head# 2024-09-17T16:34:45.985 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9752ed91:::benchmark_data_smithi173_130566_object7564:head# 2024-09-17T16:34:45.986 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97577f17:::benchmark_data_smithi173_130566_object5881:head# 2024-09-17T16:34:45.986 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9762fe76:::benchmark_data_smithi173_130566_object7352:head# 2024-09-17T16:34:45.987 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9769d3c8:::benchmark_data_smithi173_130566_object5226:head# 2024-09-17T16:34:45.988 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:976a8e7f:::benchmark_data_smithi173_130566_object2171:head# 2024-09-17T16:34:45.988 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9772c3f0:::benchmark_data_smithi173_130566_object3944:head# 2024-09-17T16:34:45.988 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9776fdb5:::benchmark_data_smithi173_130566_object5703:head# 2024-09-17T16:34:45.989 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:977d2f80:::benchmark_data_smithi173_130566_object6103:head# 2024-09-17T16:34:45.989 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9781b034:::benchmark_data_smithi173_130566_object7764:head# 2024-09-17T16:34:45.990 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9782d320:::benchmark_data_smithi173_130566_object4129:head# 2024-09-17T16:34:45.991 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:9785fece:::benchmark_data_smithi173_130566_object6924:head# 2024-09-17T16:34:45.991 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97886e1d:::benchmark_data_smithi173_130566_object2643:head# 2024-09-17T16:34:45.992 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:978fc5cc:::benchmark_data_smithi173_130566_object6819:head# 2024-09-17T16:34:45.992 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97991a1f:::benchmark_data_smithi173_130566_object4337:head# 2024-09-17T16:34:45.993 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:979bb3fc:::benchmark_data_smithi173_130566_object3780:head# 2024-09-17T16:34:45.993 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97a12b58:::benchmark_data_smithi173_130566_object7403:head# 2024-09-17T16:34:45.994 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97a5ca5e:::benchmark_data_smithi173_130566_object7256:head# 2024-09-17T16:34:45.994 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97a69d07:::benchmark_data_smithi173_130566_object2864:head# 2024-09-17T16:34:45.995 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97ace031:::benchmark_data_smithi173_130566_object3006:head# 2024-09-17T16:34:45.995 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97c247f4:::benchmark_data_smithi173_130566_object7711:head# 2024-09-17T16:34:45.996 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97c93130:::benchmark_data_smithi173_130566_object7841:head# 2024-09-17T16:34:45.996 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97cb9e8b:::benchmark_data_smithi173_130566_object6270:head# 2024-09-17T16:34:45.997 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97de1bb9:::benchmark_data_smithi173_130566_object2872:head# 2024-09-17T16:34:45.997 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97e081d7:::benchmark_data_smithi173_130566_object752:head# 2024-09-17T16:34:45.998 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97f39fd6:::benchmark_data_smithi173_130566_object2981:head# 2024-09-17T16:34:45.998 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97f46cf8:::benchmark_data_smithi173_130566_object7219:head# 2024-09-17T16:34:45.999 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:97fc8546:::benchmark_data_smithi173_130566_object6498:head# 2024-09-17T16:34:46.000 INFO:teuthology.orchestra.run.smithi173.stderr:Export successful 2024-09-17T16:34:46.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:46.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:46.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:46.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:46.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:46.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:46.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:46.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:46.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:46.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:46.302 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:46.304 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:46.392 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:46.414 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:46.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:46.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:46.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:46.540 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:46.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:46.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 27999 27983 49.965 73.25 0.0029495 0.0200012 2024-09-17T16:34:46.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:46.664 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:46.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:46.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:46.791 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:46.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:46.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:46.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:46.925 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:46.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:47.027 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:47.049 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:47.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:47.148 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:34:47.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:47.184 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:47.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:47.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:47.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:47 smithi017 ceph-mon[27571]: pgmap v1668: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s; 5876/72999 objects degraded (8.049%); 2493/72999 objects misplaced (3.415%) 2024-09-17T16:34:47.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/611115988' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:47.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2957986593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:47.318 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:47.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:47 smithi173 ceph-mon[30530]: pgmap v1668: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s; 5876/72999 objects degraded (8.049%); 2493/72999 objects misplaced (3.415%) 2024-09-17T16:34:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/611115988' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2957986593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:47.419 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:47.436 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:34:47.436 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:34:47.436 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:34:47.436 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:34:47.436 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:34:47.436 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:34:47.437 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:34:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:34:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:34:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:34:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:34:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:34:47.438 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:47.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:47.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:47.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:47 smithi071 ceph-mon[30002]: pgmap v1668: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s; 5876/72999 objects degraded (8.049%); 2493/72999 objects misplaced (3.415%) 2024-09-17T16:34:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/611115988' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:34:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2957986593' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:47.577 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:47.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:47.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:47.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 29408 29392 51.0231 88.0625 0.00179246 0.019587 2024-09-17T16:34:47.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:47.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.10 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --force --op remove --pgid 8.9 2024-09-17T16:34:47.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:47.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:47.811 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:47.843 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:47.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:47.945 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:47.973 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:47.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:47.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:48.004 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.10/config 2024-09-17T16:34:48.060 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:34:48.060 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:34:48.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.060 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:28753 8.e 8:7c8951c8:::benchmark_data_smithi173_130566_object28752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e751)", 2024-09-17T16:34:48.060 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:47.165277+0000", 2024-09-17T16:34:48.060 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.0040248710000000002, 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040693300000000003, 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 28753 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.165277+0000", 2024-09-17T16:34:48.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:34:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:34:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.165277+0000", 2024-09-17T16:34:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:34:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:34:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.165283+0000", 2024-09-17T16:34:48.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9660000000000001e-06 2024-09-17T16:34:48.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:34:48.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.165339+0000", 2024-09-17T16:34:48.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6001999999999997e-05 2024-09-17T16:34:48.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.165365+0000", 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6041e-05 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.165377+0000", 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1316e-05 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.165449+0000", 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1784000000000003e-05 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.165608+0000", 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015915499999999999 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,4", 2024-09-17T16:34:48.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.165756+0000", 2024-09-17T16:34:48.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014840200000000001 2024-09-17T16:34:48.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:34:48.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.167377+0000", 2024-09-17T16:34:48.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016213309999999999 2024-09-17T16:34:48.066 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:48.066 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:48.066 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:28757 8.e 8:702eb48b:::benchmark_data_smithi173_130566_object28756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e751)", 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:47.168083+0000", 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.001219744, 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001352509, 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:34:48.067 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 28757 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.168083+0000", 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.168083+0000", 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:34:48.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.168084+0000", 2024-09-17T16:34:48.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7859999999999999e-06 2024-09-17T16:34:48.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:34:48.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.168168+0000", 2024-09-17T16:34:48.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4130999999999998e-05 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.168183+0000", 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4701e-05 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.168192+0000", 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0510000000000002e-06 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.168303+0000", 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000111257 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.168538+0000", 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023478799999999999 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,4", 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.168690+0000", 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015184899999999999 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:48.071 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:28714 8.9 8:917c2325:::benchmark_data_smithi173_130566_object28713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e750)", 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:47.136312+0000", 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.032990749999999999, 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.033197811000000001, 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:34:48.072 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 28714 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.136312+0000", 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.136312+0000", 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.136313+0000", 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.663e-06 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.136358+0000", 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4985999999999999e-05 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.136375+0000", 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232999999999999e-05 2024-09-17T16:34:48.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.136385+0000", 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2799999999999992e-06 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.136457+0000", 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2720999999999997e-05 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.136627+0000", 2024-09-17T16:34:48.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169498 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.136770+0000", 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014307600000000001 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:47.137839+0000", 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010687870000000001 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:34:48.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:48.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:34:48.077 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:34:48.077 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 3 2024-09-17T16:34:48.077 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:34:48.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:48.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:48.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:48.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:48.234 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:48.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:48.337 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:48.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:48.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:48.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:48 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:34:48.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/611115988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:48.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:48 smithi017 ceph-mon[27571]: osdmap e751: 12 total, 11 up, 12 in 2024-09-17T16:34:48.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:48 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:34:48.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/611115988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:48.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:48 smithi173 ceph-mon[30530]: osdmap e751: 12 total, 11 up, 12 in 2024-09-17T16:34:48.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:48.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:48.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:48 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:34:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/611115988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:34:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:48 smithi071 ceph-mon[30002]: osdmap e751: 12 total, 11 up, 12 in 2024-09-17T16:34:48.593 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:48.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 31115 31099 52.5273 106.688 0.0057242 0.0190304 2024-09-17T16:34:48.624 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:48.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:48.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:48.755 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:48.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:48.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:34:48.857 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:48.882 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:48.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:48.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:49.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:49.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:49.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:49.137 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:49.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:49.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:49.266 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:49.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:34:49.348 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:34:49.367 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:49.392 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:49.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:49.402 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:49 smithi173 ceph-mon[30530]: pgmap v1670: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 6856/80745 objects degraded (8.491%); 2493/80745 objects misplaced (3.087%) 2024-09-17T16:34:49.446 INFO:teuthology.orchestra.run.smithi173.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:34:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:49 smithi017 ceph-mon[27571]: pgmap v1670: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 6856/80745 objects degraded (8.491%); 2493/80745 objects misplaced (3.087%) 2024-09-17T16:34:49.494 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:49.522 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:49.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:49 smithi071 ceph-mon[30002]: pgmap v1670: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 6856/80745 objects degraded (8.491%); 2493/80745 objects misplaced (3.087%) 2024-09-17T16:34:49.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 32493 32477 53.4112 86.125 0.00377028 0.0187045 2024-09-17T16:34:49.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:49.646 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:49.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:49.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:49.771 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:49.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:49.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:49.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:49.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:50.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:50.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:50.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:50.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:50.101 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:50.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:50.165 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:50.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:50.266 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:50.293 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:50.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:50.385 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:50 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 6856/80745 objects degraded (8.491%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:50.385 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:50 smithi017 ceph-mon[27571]: osdmap e752: 12 total, 11 up, 12 in 2024-09-17T16:34:50.395 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:50.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:50 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 6856/80745 objects degraded (8.491%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:50.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:50 smithi173 ceph-mon[30530]: osdmap e752: 12 total, 11 up, 12 in 2024-09-17T16:34:50.418 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:50.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:50.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:50.551 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:50.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:50.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:50 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 6856/80745 objects degraded (8.491%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:50 smithi071 ceph-mon[30002]: osdmap e752: 12 total, 11 up, 12 in 2024-09-17T16:34:50.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 33888 33872 54.2771 87.1875 0.0201332 0.0184163 2024-09-17T16:34:50.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:50.686 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:50.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:50.691 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:34:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:34:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:34:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:34:50.691 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:34:50.788 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:50.817 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:50.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:50.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:50.941 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:50.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:51.043 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:51.067 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:51.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:51.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:51.169 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:51.194 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:51.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:51.297 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:51.321 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:51.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:51.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:51.350 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:34:51.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:51 smithi017 ceph-mon[27571]: pgmap v1672: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s; 7272/90333 objects degraded (8.050%); 2699/90333 objects misplaced (2.988%) 2024-09-17T16:34:51.423 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:51.448 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:51.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:51.451 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:34:51.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:51 smithi071 ceph-mon[30002]: pgmap v1672: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s; 7272/90333 objects degraded (8.050%); 2699/90333 objects misplaced (2.988%) 2024-09-17T16:34:51.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:51.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:51.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:34:51.615041+0000 min lat: 0.00112054 max lat: 15.0115 avg lat: 0.0180838 2024-09-17T16:34:51.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:34:51.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 35394 35378 55.2731 94.125 0.0188014 0.0180838 2024-09-17T16:34:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:51 smithi173 ceph-mon[30530]: pgmap v1672: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s; 7272/90333 objects degraded (8.050%); 2699/90333 objects misplaced (2.988%) 2024-09-17T16:34:51.678 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:51.693 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:51.701 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:51.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:51.803 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:51.824 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:51.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:51.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:51.950 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:51.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:51.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:52.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:52.074 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:52.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:52.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:52.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:34:52.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:52.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:52.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:52.298 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:34:52.298 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:34:52.301 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:52.333 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:34:52.333 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:34:52.339 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:52.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:52.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:52.363 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:52.369 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:34:52.369 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:34:52.403 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:34:52.404 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:34:52.432 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:34:52.432 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:34:52.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:34:52.460 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:34:52.461 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:34:52.465 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:52.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:34:52.489 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:34:52.490 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:34:52.516 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:34:52.516 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:34:52.544 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:34:52.545 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:34:52.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:52.568 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:34:52.569 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:34:52.569 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:34:52.590 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:52.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:52.597 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:34:52.598 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:34:52.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 36433 36417 55.5087 64.9375 0.0150773 0.018007 2024-09-17T16:34:52.693 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:52.723 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:52.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:52.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:52.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:52.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:52.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:52.953 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:52.984 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:52.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:53.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:53.113 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:53.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:53.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:53.242 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:53.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:53.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:53.367 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:53.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:53.468 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:53 smithi017 ceph-mon[27571]: pgmap v1673: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 7465/92865 objects degraded (8.039%); 3137/92865 objects misplaced (3.378%) 2024-09-17T16:34:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3235220854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:53.491 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:53.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:53 smithi071 ceph-mon[30002]: pgmap v1673: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 7465/92865 objects degraded (8.039%); 3137/92865 objects misplaced (3.378%) 2024-09-17T16:34:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3235220854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:53.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:53.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 37608 37592 55.9355 73.4375 0.0153661 0.0178687 2024-09-17T16:34:53.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:53.617 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:53.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:53 smithi173 ceph-mon[30530]: pgmap v1673: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 7465/92865 objects degraded (8.039%); 3137/92865 objects misplaced (3.378%) 2024-09-17T16:34:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3235220854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:34:53.718 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:53.740 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:53.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:53.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:53.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:53.867 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:53.968 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:54.002 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:54.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:54.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:54.127 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:54.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:54.229 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:54.260 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:54.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:54.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:54.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:54.385 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:54.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:54.487 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:54.520 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:54.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:54.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 39150 39134 56.8757 96.375 0.0161494 0.0175759 2024-09-17T16:34:54.623 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:54.633 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:54.633 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:54.650 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:54.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:54.729 INFO:teuthology.orchestra.run.smithi173.stdout: marking collection for removal 2024-09-17T16:34:54.729 INFO:teuthology.orchestra.run.smithi173.stdout:setting '_remove' omap key 2024-09-17T16:34:54.729 INFO:teuthology.orchestra.run.smithi173.stdout:finish_remove_pgs 8.9_head removing 8.9 2024-09-17T16:34:54.736 INFO:teuthology.orchestra.run.smithi173.stdout:Remove successful 2024-09-17T16:34:54.751 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:54.775 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:54.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:54.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:54.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:54.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:54.904 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:54.906 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:55.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:55.038 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:55.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:55.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:55.164 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:55.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:55.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:55.290 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:55.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:55.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:55.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:55.420 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:55.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:55.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:55 smithi017 ceph-mon[27571]: pgmap v1674: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s; 8780/104487 objects degraded (8.403%); 3137/104487 objects misplaced (3.002%) 2024-09-17T16:34:55.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:55 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 8780/104487 objects degraded (8.403%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:55.498 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:34:55.522 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:55 smithi071 ceph-mon[30002]: pgmap v1674: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s; 8780/104487 objects degraded (8.403%); 3137/104487 objects misplaced (3.002%) 2024-09-17T16:34:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:55 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 8780/104487 objects degraded (8.403%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:55.565 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:55.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:55.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 40468 40452 57.4551 82.375 0.0271759 0.0173812 2024-09-17T16:34:55.628 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:55 smithi173 ceph-mon[30530]: pgmap v1674: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s; 8780/104487 objects degraded (8.403%); 3137/104487 objects misplaced (3.002%) 2024-09-17T16:34:55.629 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:55 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 8780/104487 objects degraded (8.403%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:34:55.665 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:55.696 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:55.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:55.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:55.798 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:55.820 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:55.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:55.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:55.945 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:55.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:55.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.10 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.8.9.10 2024-09-17T16:34:56.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:56.070 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:56.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:56.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:34:56.200 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:56.201 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.10/config 2024-09-17T16:34:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:56.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:34:56.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:56.324 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:56.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:56.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:34:56.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:34:56.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:56.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:56.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:56.573 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:56.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:56.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 41117 41101 57.0797 40.5625 0.00205239 0.0174478 2024-09-17T16:34:56.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:56.685 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:56.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:56.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:56.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:56.805 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:56.834 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:56.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:56.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:56.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:56.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:57.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:57.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:57.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:57.174 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:34:57.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:57.216 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:57.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:57.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:57.264 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:34:57.264 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:34:57.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:57.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:57.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:57 smithi173 ceph-mon[30530]: pgmap v1675: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 9155/113154 objects degraded (8.091%); 3764/113154 objects misplaced (3.326%) 2024-09-17T16:34:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3671354804' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:57 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:57.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:57.478 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:57.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:57 smithi017 ceph-mon[27571]: pgmap v1675: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 9155/113154 objects degraded (8.091%); 3764/113154 objects misplaced (3.326%) 2024-09-17T16:34:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3671354804' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:57 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:57 smithi071 ceph-mon[30002]: pgmap v1675: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 9155/113154 objects degraded (8.091%); 3764/113154 objects misplaced (3.326%) 2024-09-17T16:34:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3671354804' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:57 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:34:57.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:34:57.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:57.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:34:57.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 42944 42928 58.3209 114.188 0.0107254 0.0171357 2024-09-17T16:34:57.705 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:57.728 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:57.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:57.830 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:57.853 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:57.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:57.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:57.980 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:57.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:58.024 INFO:teuthology.orchestra.run.smithi173.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:34:58.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:34:58.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:58.108 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:34:58.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:58.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:58.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:58.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:58.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:58 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:58.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:58 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:34:58.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:58 smithi017 ceph-mon[27571]: osdmap e753: 12 total, 11 up, 12 in 2024-09-17T16:34:58.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:58 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:58.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:58 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:34:58.322 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:58 smithi173 ceph-mon[30530]: osdmap e753: 12 total, 11 up, 12 in 2024-09-17T16:34:58.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:58.356 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:58.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:58.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:34:58.458 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:58.484 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:58.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:58 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:34:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:58 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:34:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:58 smithi071 ceph-mon[30002]: osdmap e753: 12 total, 11 up, 12 in 2024-09-17T16:34:58.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:58.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:58.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:58.616 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 44052 44036 58.5533 69.25 0.00204149 0.0170727 2024-09-17T16:34:58.705 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:34:58.715 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:58.783 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:58.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:58.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:58.848 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:34:58.886 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:34:58.928 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:58.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:58.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:34:59.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:59.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:59.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:59.088 INFO:teuthology.orchestra.run.smithi173.stdout:get_pg_num_history pg_num_history pg_num_history(e740 pg_nums {1={19=1},7={674=26},8={728=16,736=26}} deleted_pools 726,7) 2024-09-17T16:34:59.088 INFO:teuthology.orchestra.run.smithi173.stdout:Importing pgid 8.9 2024-09-17T16:34:59.088 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9000f542:::benchmark_data_smithi173_130566_object696:head# 2024-09-17T16:34:59.088 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.088 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90065cee:::benchmark_data_smithi173_130566_object304:head# 2024-09-17T16:34:59.088 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.088 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:900c476d:::benchmark_data_smithi173_130566_object7779:head# 2024-09-17T16:34:59.088 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.089 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:900cfd0d:::benchmark_data_smithi173_130566_object6959:head# 2024-09-17T16:34:59.090 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.090 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:901a591d:::benchmark_data_smithi173_130566_object4853:head# 2024-09-17T16:34:59.090 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.090 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:902e275a:::benchmark_data_smithi173_130566_object7433:head# 2024-09-17T16:34:59.090 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.090 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:902f7b8b:::benchmark_data_smithi173_130566_object473:head# 2024-09-17T16:34:59.090 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.090 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9031940d:::benchmark_data_smithi173_130566_object5458:head# 2024-09-17T16:34:59.090 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.096 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9034cce4:::benchmark_data_smithi173_130566_object199:head# 2024-09-17T16:34:59.096 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.096 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9038159a:::benchmark_data_smithi173_130566_object3699:head# 2024-09-17T16:34:59.096 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.096 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90440db7:::benchmark_data_smithi173_130566_object1859:head# 2024-09-17T16:34:59.096 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.096 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90444b71:::benchmark_data_smithi173_130566_object2890:head# 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9045562e:::benchmark_data_smithi173_130566_object5055:head# 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90533b2b:::benchmark_data_smithi173_130566_object4937:head# 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9059eb62:::benchmark_data_smithi173_130566_object5460:head# 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9094c450:::benchmark_data_smithi173_130566_object5913:head# 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:909d5f27:::benchmark_data_smithi173_130566_object3502:head# 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.097 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:909f3c7f:::benchmark_data_smithi173_130566_object3749:head# 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90a633ea:::benchmark_data_smithi173_130566_object6599:head# 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90a7dd1a:::benchmark_data_smithi173_130566_object1053:head# 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90bc7c08:::benchmark_data_smithi173_130566_object400:head# 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90bfd592:::benchmark_data_smithi173_130566_object1259:head# 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90c6d3e1:::benchmark_data_smithi173_130566_object3224:head# 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.098 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90cb734f:::benchmark_data_smithi173_130566_object4895:head# 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90d0cfd4:::benchmark_data_smithi173_130566_object4354:head# 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90d0f081:::benchmark_data_smithi173_130566_object504:head# 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90ec0701:::benchmark_data_smithi173_130566_object5919:head# 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90eed5ed:::benchmark_data_smithi173_130566_object4784:head# 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90eed61d:::benchmark_data_smithi173_130566_object6205:head# 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.099 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90f204b8:::benchmark_data_smithi173_130566_object5989:head# 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90f3c4fa:::benchmark_data_smithi173_130566_object6165:head# 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90fb628b:::benchmark_data_smithi173_130566_object4833:head# 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:90fcbef4:::benchmark_data_smithi173_130566_object1017:head# 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9103c16a:::benchmark_data_smithi173_130566_object1574:head# 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9107c432:::benchmark_data_smithi173_130566_object1378:head# 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.100 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:910a08ef:::benchmark_data_smithi173_130566_object4889:head# 2024-09-17T16:34:59.101 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.105 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9111c9f3:::benchmark_data_smithi173_130566_object3256:head# 2024-09-17T16:34:59.105 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.105 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91140c99:::benchmark_data_smithi173_130566_object1297:head# 2024-09-17T16:34:59.105 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.105 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:911a896e:::benchmark_data_smithi173_130566_object3871:head# 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:911daa4e:::benchmark_data_smithi173_130566_object6030:head# 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:912258fa:::benchmark_data_smithi173_130566_object4777:head# 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9122e2c8:::benchmark_data_smithi173_130566_object7277:head# 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9133ffa1:::benchmark_data_smithi173_130566_object3584:head# 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:913a5502:::benchmark_data_smithi173_130566_object312:head# 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:913d4e17:::benchmark_data_smithi173_130566_object7330:head# 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.106 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9140c358:::benchmark_data_smithi173_130566_object6343:head# 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9142519e:::benchmark_data_smithi173_130566_object4378:head# 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9153b175:::benchmark_data_smithi173_130566_object2939:head# 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91577cd5:::benchmark_data_smithi173_130566_object5019:head# 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91585782:::benchmark_data_smithi173_130566_object3816:head# 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91599685:::benchmark_data_smithi173_130566_object2426:head# 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:915ba07d:::benchmark_data_smithi173_130566_object4707:head# 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.107 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9160d36f:::benchmark_data_smithi173_130566_object3621:head# 2024-09-17T16:34:59.108 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.108 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:916c6d86:::benchmark_data_smithi173_130566_object826:head# 2024-09-17T16:34:59.108 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.108 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:916c8e22:::benchmark_data_smithi173_130566_object7990:head# 2024-09-17T16:34:59.108 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.108 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9172b9dc:::benchmark_data_smithi173_130566_object4566:head# 2024-09-17T16:34:59.108 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.108 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9176e5ad:::benchmark_data_smithi173_130566_object5031:head# 2024-09-17T16:34:59.108 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.108 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:917c98b3:::benchmark_data_smithi173_130566_object1298:head# 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91821118:::benchmark_data_smithi173_130566_object605:head# 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:918a7ea8:::benchmark_data_smithi173_130566_object1301:head# 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91932c14:::benchmark_data_smithi173_130566_object1110:head# 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:919b24dd:::benchmark_data_smithi173_130566_object1205:head# 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91a39ae6:::benchmark_data_smithi173_130566_object7820:head# 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91b4ae05:::benchmark_data_smithi173_130566_object2402:head# 2024-09-17T16:34:59.109 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91b698ab:::benchmark_data_smithi173_130566_object4718:head# 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91b927e7:::benchmark_data_smithi173_130566_object2795:head# 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91bc039e:::benchmark_data_smithi173_130566_object2021:head# 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91bcce82:::benchmark_data_smithi173_130566_object5965:head# 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91c29b2f:::benchmark_data_smithi173_130566_object6594:head# 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.112 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91c3a76e:::benchmark_data_smithi173_130566_object813:head# 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91c67866:::benchmark_data_smithi173_130566_object786:head# 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91c6b8bb:::benchmark_data_smithi173_130566_object4423:head# 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91f01e97:::benchmark_data_smithi173_130566_object1541:head# 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91f21c1e:::benchmark_data_smithi173_130566_object5136:head# 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:91fc5a89:::benchmark_data_smithi173_130566_object2906:head# 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9202cd53:::benchmark_data_smithi173_130566_object5937:head# 2024-09-17T16:34:59.113 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.114 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9205becd:::benchmark_data_smithi173_130566_object7461:head# 2024-09-17T16:34:59.114 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.114 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9205e63c:::benchmark_data_smithi173_130566_object7354:head# 2024-09-17T16:34:59.114 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.115 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:920676b5:::benchmark_data_smithi173_130566_object582:head# 2024-09-17T16:34:59.115 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.115 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9211482e:::benchmark_data_smithi173_130566_object1923:head# 2024-09-17T16:34:59.115 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.115 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9227df93:::benchmark_data_smithi173_130566_object3202:head# 2024-09-17T16:34:59.115 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.115 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:922ac0b7:::benchmark_data_smithi173_130566_object3549:head# 2024-09-17T16:34:59.115 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.115 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9233f812:::benchmark_data_smithi173_130566_object6091:head# 2024-09-17T16:34:59.115 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.116 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:923c4e0d:::benchmark_data_smithi173_130566_object6710:head# 2024-09-17T16:34:59.116 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.116 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9247da57:::benchmark_data_smithi173_130566_object1210:head# 2024-09-17T16:34:59.116 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.117 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:927258fd:::benchmark_data_smithi173_130566_object5027:head# 2024-09-17T16:34:59.117 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.118 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9284392c:::benchmark_data_smithi173_130566_object3534:head# 2024-09-17T16:34:59.118 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.118 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:928c3a93:::benchmark_data_smithi173_130566_object5000:head# 2024-09-17T16:34:59.118 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.118 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:929206fe:::benchmark_data_smithi173_130566_object854:head# 2024-09-17T16:34:59.119 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.120 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:929ad92d:::benchmark_data_smithi173_130566_object5946:head# 2024-09-17T16:34:59.120 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.121 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:929eb97d:::benchmark_data_smithi173_130566_object295:head# 2024-09-17T16:34:59.121 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.121 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:92a56f61:::benchmark_data_smithi173_130566_object1276:head# 2024-09-17T16:34:59.121 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.122 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:92abc0e3:::benchmark_data_smithi173_130566_object5448:head# 2024-09-17T16:34:59.122 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.122 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:92ade1ec:::benchmark_data_smithi173_130566_object4472:head# 2024-09-17T16:34:59.123 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.123 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:92b03dc7:::benchmark_data_smithi173_130566_object5159:head# 2024-09-17T16:34:59.123 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.123 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:92ca2570:::benchmark_data_smithi173_130566_object955:head# 2024-09-17T16:34:59.123 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.124 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:92d2eceb:::benchmark_data_smithi173_130566_object4015:head# 2024-09-17T16:34:59.124 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.124 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:92f6f67c:::benchmark_data_smithi173_130566_object7168:head# 2024-09-17T16:34:59.124 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.125 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9305660a:::benchmark_data_smithi173_130566_object1325:head# 2024-09-17T16:34:59.125 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.125 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:930ea188:::benchmark_data_smithi173_130566_object3414:head# 2024-09-17T16:34:59.125 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.126 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9327d755:::benchmark_data_smithi173_130566_object870:head# 2024-09-17T16:34:59.126 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.126 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:932a1564:::benchmark_data_smithi173_130566_object4198:head# 2024-09-17T16:34:59.126 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.127 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:932dffdc:::benchmark_data_smithi173_130566_object418:head# 2024-09-17T16:34:59.127 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.127 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:932fe421:::benchmark_data_smithi173_130566_object4708:head# 2024-09-17T16:34:59.127 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.127 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93324f6d:::benchmark_data_smithi173_130566_object220:head# 2024-09-17T16:34:59.127 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.127 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:934a6b3a:::benchmark_data_smithi173_130566_object5241:head# 2024-09-17T16:34:59.127 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.127 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:934e680e:::benchmark_data_smithi173_130566_object6423:head# 2024-09-17T16:34:59.128 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.128 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:935799aa:::benchmark_data_smithi173_130566_object3782:head# 2024-09-17T16:34:59.129 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.129 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:935d20f5:::benchmark_data_smithi173_130566_object1151:head# 2024-09-17T16:34:59.129 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.129 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93662906:::benchmark_data_smithi173_130566_object4074:head# 2024-09-17T16:34:59.129 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.129 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9367529f:::benchmark_data_smithi173_130566_object5006:head# 2024-09-17T16:34:59.129 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.129 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:936f4796:::benchmark_data_smithi173_130566_object5082:head# 2024-09-17T16:34:59.129 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.130 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93721618:::benchmark_data_smithi173_130566_object1426:head# 2024-09-17T16:34:59.130 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.130 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93768e31:::benchmark_data_smithi173_130566_object3435:head# 2024-09-17T16:34:59.130 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.130 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9384cc9f:::benchmark_data_smithi173_130566_object6674:head# 2024-09-17T16:34:59.130 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.133 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:938ab075:::benchmark_data_smithi173_130566_object6849:head# 2024-09-17T16:34:59.133 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.136 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9392e082:::benchmark_data_smithi173_130566_object3836:head# 2024-09-17T16:34:59.136 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.136 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9393489c:::benchmark_data_smithi173_130566_object6598:head# 2024-09-17T16:34:59.137 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.137 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93952705:::benchmark_data_smithi173_130566_object7450:head# 2024-09-17T16:34:59.137 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.137 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9396b81c:::benchmark_data_smithi173_130566_object3115:head# 2024-09-17T16:34:59.137 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.138 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9398a850:::benchmark_data_smithi173_130566_object3081:head# 2024-09-17T16:34:59.138 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.138 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93a94d82:::benchmark_data_smithi173_130566_object1956:head# 2024-09-17T16:34:59.138 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.139 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93ab35a2:::benchmark_data_smithi173_130566_object2950:head# 2024-09-17T16:34:59.139 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.139 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93b48f2d:::benchmark_data_smithi173_130566_object5208:head# 2024-09-17T16:34:59.139 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.139 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93b733dc:::benchmark_data_smithi173_130566_object3840:head# 2024-09-17T16:34:59.139 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.140 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93b88bab:::benchmark_data_smithi173_130566_object6441:head# 2024-09-17T16:34:59.140 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.140 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93c10f07:::benchmark_data_smithi173_130566_object1203:head# 2024-09-17T16:34:59.140 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.141 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93c92961:::benchmark_data_smithi173_130566_object2773:head# 2024-09-17T16:34:59.141 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.141 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93cc0c1d:::benchmark_data_smithi173_130566_object2661:head# 2024-09-17T16:34:59.141 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.142 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93cd4834:::benchmark_data_smithi173_130566_object7522:head# 2024-09-17T16:34:59.142 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.142 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93d42013:::benchmark_data_smithi173_130566_object1817:head# 2024-09-17T16:34:59.142 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.143 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93d6f7f7:::benchmark_data_smithi173_130566_object4347:head# 2024-09-17T16:34:59.143 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.143 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93db84ad:::benchmark_data_smithi173_130566_object7574:head# 2024-09-17T16:34:59.143 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.144 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93e4028b:::benchmark_data_smithi173_130566_object1815:head# 2024-09-17T16:34:59.144 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.144 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93efd0bf:::benchmark_data_smithi173_130566_object5976:head# 2024-09-17T16:34:59.144 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.145 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:93f27987:::benchmark_data_smithi173_130566_object6604:head# 2024-09-17T16:34:59.145 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.149 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:940bbe0a:::benchmark_data_smithi173_130566_object4726:head# 2024-09-17T16:34:59.149 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:940c200c:::benchmark_data_smithi173_130566_object5809:head# 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:941770c8:::benchmark_data_smithi173_130566_object1811:head# 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:941942a2:::benchmark_data_smithi173_130566_object840:head# 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:942254a3:::benchmark_data_smithi173_130566_object2009:head# 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9426733b:::benchmark_data_smithi173_130566_object3398:head# 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94277dc3:::benchmark_data_smithi173_130566_object3969:head# 2024-09-17T16:34:59.150 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.151 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9429f6bb:::benchmark_data_smithi173_130566_object717:head# 2024-09-17T16:34:59.151 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.151 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:943026b7:::benchmark_data_smithi173_130566_object7583:head# 2024-09-17T16:34:59.151 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.152 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94435040:::benchmark_data_smithi173_130566_object4667:head# 2024-09-17T16:34:59.152 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.152 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94443a53:::benchmark_data_smithi173_130566_object4163:head# 2024-09-17T16:34:59.152 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.153 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:944a65e8:::benchmark_data_smithi173_130566_object4829:head# 2024-09-17T16:34:59.153 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.153 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9460a9e3:::benchmark_data_smithi173_130566_object2234:head# 2024-09-17T16:34:59.153 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.154 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:946899a5:::benchmark_data_smithi173_130566_object2362:head# 2024-09-17T16:34:59.154 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.154 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94702b3b:::benchmark_data_smithi173_130566_object4288:head# 2024-09-17T16:34:59.154 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.155 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94833472:::benchmark_data_smithi173_130566_object7669:head# 2024-09-17T16:34:59.155 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.155 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94a3419b:::benchmark_data_smithi173_130566_object5823:head# 2024-09-17T16:34:59.155 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.155 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94a53baa:::benchmark_data_smithi173_130566_object6129:head# 2024-09-17T16:34:59.155 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:59.156 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94b15fd9:::benchmark_data_smithi173_130566_object4883:head# 2024-09-17T16:34:59.157 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.157 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94b2e3da:::benchmark_data_smithi173_130566_object2424:head# 2024-09-17T16:34:59.157 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.157 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94b3f867:::benchmark_data_smithi173_130566_object4319:head# 2024-09-17T16:34:59.157 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.157 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94dc18f2:::benchmark_data_smithi173_130566_object4748:head# 2024-09-17T16:34:59.157 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.158 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94de4957:::benchmark_data_smithi173_130566_object7793:head# 2024-09-17T16:34:59.158 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.158 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94e030fc:::benchmark_data_smithi173_130566_object6602:head# 2024-09-17T16:34:59.158 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.158 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94e7f6cf:::benchmark_data_smithi173_130566_object7653:head# 2024-09-17T16:34:59.158 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.159 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:94fe0b78:::benchmark_data_smithi173_130566_object1302:head# 2024-09-17T16:34:59.160 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.160 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9506315a:::benchmark_data_smithi173_130566_object1492:head# 2024-09-17T16:34:59.160 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.161 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95101c9d:::benchmark_data_smithi173_130566_object3540:head# 2024-09-17T16:34:59.161 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.163 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95108746:::benchmark_data_smithi173_130566_object3978:head# 2024-09-17T16:34:59.163 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.166 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95146639:::benchmark_data_smithi173_130566_object7275:head# 2024-09-17T16:34:59.166 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.166 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:951fac3b:::benchmark_data_smithi173_130566_object7495:head# 2024-09-17T16:34:59.166 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.167 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95203a24:::benchmark_data_smithi173_130566_object4769:head# 2024-09-17T16:34:59.167 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.167 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95223cfb:::benchmark_data_smithi173_130566_object4147:head# 2024-09-17T16:34:59.167 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.167 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95268507:::benchmark_data_smithi173_130566_object3076:head# 2024-09-17T16:34:59.167 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.168 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95272860:::benchmark_data_smithi173_130566_object950:head# 2024-09-17T16:34:59.168 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.168 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:952a6969:::benchmark_data_smithi173_130566_object2388:head# 2024-09-17T16:34:59.168 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.168 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:952ae84d:::benchmark_data_smithi173_130566_object5988:head# 2024-09-17T16:34:59.168 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.169 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9532413d:::benchmark_data_smithi173_130566_object363:head# 2024-09-17T16:34:59.169 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.169 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95390790:::benchmark_data_smithi173_130566_object2241:head# 2024-09-17T16:34:59.169 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.169 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:953a8e06:::benchmark_data_smithi173_130566_object3475:head# 2024-09-17T16:34:59.169 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.169 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:954cf359:::benchmark_data_smithi173_130566_object2304:head# 2024-09-17T16:34:59.170 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.170 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:955141d6:::benchmark_data_smithi173_130566_object1557:head# 2024-09-17T16:34:59.170 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.170 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9551431e:::benchmark_data_smithi173_130566_object3023:head# 2024-09-17T16:34:59.170 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.171 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9555cfa7:::benchmark_data_smithi173_130566_object2715:head# 2024-09-17T16:34:59.171 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.171 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:955ca9b2:::benchmark_data_smithi173_130566_object5695:head# 2024-09-17T16:34:59.171 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.171 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:955e7c35:::benchmark_data_smithi173_130566_object358:head# 2024-09-17T16:34:59.171 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.172 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:955fcbb7:::benchmark_data_smithi173_130566_object7615:head# 2024-09-17T16:34:59.172 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.172 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:956097c2:::benchmark_data_smithi173_130566_object7470:head# 2024-09-17T16:34:59.172 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.172 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9567ac75:::benchmark_data_smithi173_130566_object7563:head# 2024-09-17T16:34:59.172 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.172 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:957452a4:::benchmark_data_smithi173_130566_object7697:head# 2024-09-17T16:34:59.172 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.173 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:958224f3:::benchmark_data_smithi173_130566_object2351:head# 2024-09-17T16:34:59.173 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.174 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9583c600:::benchmark_data_smithi173_130566_object2079:head# 2024-09-17T16:34:59.174 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.175 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95877418:::benchmark_data_smithi173_130566_object7871:head# 2024-09-17T16:34:59.175 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.175 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:958aff29:::benchmark_data_smithi173_130566_object55:head# 2024-09-17T16:34:59.175 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.176 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:958d2f4b:::benchmark_data_smithi173_130566_object7222:head# 2024-09-17T16:34:59.176 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.176 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:958eefd6:::benchmark_data_smithi173_130566_object655:head# 2024-09-17T16:34:59.176 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.176 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:959dab3c:::benchmark_data_smithi173_130566_object6479:head# 2024-09-17T16:34:59.176 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.177 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:959ff3fc:::benchmark_data_smithi173_130566_object7190:head# 2024-09-17T16:34:59.177 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.177 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95a36c91:::benchmark_data_smithi173_130566_object888:head# 2024-09-17T16:34:59.177 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.177 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95aa3975:::benchmark_data_smithi173_130566_object7331:head# 2024-09-17T16:34:59.177 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.177 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95aa8d85:::benchmark_data_smithi173_130566_object3080:head# 2024-09-17T16:34:59.177 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.178 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95b6c2c6:::benchmark_data_smithi173_130566_object3543:head# 2024-09-17T16:34:59.178 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.178 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95cab347:::benchmark_data_smithi173_130566_object132:head# 2024-09-17T16:34:59.178 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.178 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95cd6b7a:::benchmark_data_smithi173_130566_object5609:head# 2024-09-17T16:34:59.178 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.179 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95d1742a:::benchmark_data_smithi173_130566_object7943:head# 2024-09-17T16:34:59.179 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.179 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95d6e0a0:::benchmark_data_smithi173_130566_object3904:head# 2024-09-17T16:34:59.179 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.179 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95df2076:::benchmark_data_smithi173_130566_object7390:head# 2024-09-17T16:34:59.179 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.180 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95f21f8c:::benchmark_data_smithi173_130566_object6942:head# 2024-09-17T16:34:59.180 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.180 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95f389a5:::benchmark_data_smithi173_130566_object3993:head# 2024-09-17T16:34:59.180 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:59.182 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:59.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:59.183 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95fc745c:::benchmark_data_smithi173_130566_object3548:head# 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:95fde072:::benchmark_data_smithi173_130566_object166:head# 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9605c958:::benchmark_data_smithi173_130566_object4661:head# 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96064b8a:::benchmark_data_smithi173_130566_object3326:head# 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9606ffe0:::benchmark_data_smithi173_130566_object596:head# 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:960af516:::benchmark_data_smithi173_130566_object5812:head# 2024-09-17T16:34:59.184 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:961151fe:::benchmark_data_smithi173_130566_object1743:head# 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96190caf:::benchmark_data_smithi173_130566_object2154:head# 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:962bbb6f:::benchmark_data_smithi173_130566_object7575:head# 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:962d323c:::benchmark_data_smithi173_130566_object242:head# 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96400829:::benchmark_data_smithi173_130566_object4007:head# 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9641966d:::benchmark_data_smithi173_130566_object3557:head# 2024-09-17T16:34:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.186 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96450fe3:::benchmark_data_smithi173_130566_object6306:head# 2024-09-17T16:34:59.186 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.186 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:964e86f4:::benchmark_data_smithi173_130566_object3357:head# 2024-09-17T16:34:59.186 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.186 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9655c769:::benchmark_data_smithi173_130566_object5687:head# 2024-09-17T16:34:59.186 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.188 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9656b08e:::benchmark_data_smithi173_130566_object3390:head# 2024-09-17T16:34:59.188 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.189 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96577423:::benchmark_data_smithi173_130566_object7279:head# 2024-09-17T16:34:59.189 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.189 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:965d9baf:::benchmark_data_smithi173_130566_object519:head# 2024-09-17T16:34:59.189 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.189 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96646488:::benchmark_data_smithi173_130566_object2372:head# 2024-09-17T16:34:59.189 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.190 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:966e234d:::benchmark_data_smithi173_130566_object2688:head# 2024-09-17T16:34:59.190 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.190 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9674d2cc:::benchmark_data_smithi173_130566_object5012:head# 2024-09-17T16:34:59.190 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.190 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:967c61f9:::benchmark_data_smithi173_130566_object4374:head# 2024-09-17T16:34:59.190 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.190 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9698b093:::benchmark_data_smithi173_130566_object1280:head# 2024-09-17T16:34:59.191 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.191 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96999de1:::benchmark_data_smithi173_130566_object6648:head# 2024-09-17T16:34:59.191 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.191 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96a1233f:::benchmark_data_smithi173_130566_object4508:head# 2024-09-17T16:34:59.191 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.191 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96acd81e:::benchmark_data_smithi173_130566_object839:head# 2024-09-17T16:34:59.192 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.192 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96b10b0d:::benchmark_data_smithi173_130566_object5230:head# 2024-09-17T16:34:59.192 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96b97c5a:::benchmark_data_smithi173_130566_object1997:head# 2024-09-17T16:34:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96b9e363:::benchmark_data_smithi173_130566_object6388:head# 2024-09-17T16:34:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96c2fc4f:::benchmark_data_smithi173_130566_object5744:head# 2024-09-17T16:34:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96c9c428:::benchmark_data_smithi173_130566_object6470:head# 2024-09-17T16:34:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96ce405b:::benchmark_data_smithi173_130566_object7621:head# 2024-09-17T16:34:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.194 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96d86df5:::benchmark_data_smithi173_130566_object3356:head# 2024-09-17T16:34:59.194 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.194 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96fbf50d:::benchmark_data_smithi173_130566_object488:head# 2024-09-17T16:34:59.194 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.194 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:96fe284e:::benchmark_data_smithi173_130566_object1462:head# 2024-09-17T16:34:59.194 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.195 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9702cde2:::benchmark_data_smithi173_130566_object5383:head# 2024-09-17T16:34:59.195 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.195 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9710133c:::benchmark_data_smithi173_130566_object3741:head# 2024-09-17T16:34:59.195 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.196 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:971434e3:::benchmark_data_smithi173_130566_object2647:head# 2024-09-17T16:34:59.196 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.196 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:971a2592:::benchmark_data_smithi173_130566_object2013:head# 2024-09-17T16:34:59.196 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.196 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:971aa8ee:::benchmark_data_smithi173_130566_object4805:head# 2024-09-17T16:34:59.196 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.197 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:971d5e90:::benchmark_data_smithi173_130566_object4811:head# 2024-09-17T16:34:59.197 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.197 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:971ff081:::benchmark_data_smithi173_130566_object6514:head# 2024-09-17T16:34:59.197 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.197 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9727ec1c:::benchmark_data_smithi173_130566_object5811:head# 2024-09-17T16:34:59.197 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.197 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97294f27:::benchmark_data_smithi173_130566_object1473:head# 2024-09-17T16:34:59.197 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.198 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9731c02a:::benchmark_data_smithi173_130566_object123:head# 2024-09-17T16:34:59.198 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.198 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:973c9c89:::benchmark_data_smithi173_130566_object1283:head# 2024-09-17T16:34:59.198 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.199 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:974e8887:::benchmark_data_smithi173_130566_object2023:head# 2024-09-17T16:34:59.199 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.199 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9752ed91:::benchmark_data_smithi173_130566_object7564:head# 2024-09-17T16:34:59.199 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.200 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97577f17:::benchmark_data_smithi173_130566_object5881:head# 2024-09-17T16:34:59.200 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.200 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9762fe76:::benchmark_data_smithi173_130566_object7352:head# 2024-09-17T16:34:59.200 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.200 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9769d3c8:::benchmark_data_smithi173_130566_object5226:head# 2024-09-17T16:34:59.200 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.201 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:976a8e7f:::benchmark_data_smithi173_130566_object2171:head# 2024-09-17T16:34:59.201 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.201 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9772c3f0:::benchmark_data_smithi173_130566_object3944:head# 2024-09-17T16:34:59.201 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.203 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9776fdb5:::benchmark_data_smithi173_130566_object5703:head# 2024-09-17T16:34:59.203 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:977d2f80:::benchmark_data_smithi173_130566_object6103:head# 2024-09-17T16:34:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9781b034:::benchmark_data_smithi173_130566_object7764:head# 2024-09-17T16:34:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9782d320:::benchmark_data_smithi173_130566_object4129:head# 2024-09-17T16:34:59.204 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.205 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:9785fece:::benchmark_data_smithi173_130566_object6924:head# 2024-09-17T16:34:59.205 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.205 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97886e1d:::benchmark_data_smithi173_130566_object2643:head# 2024-09-17T16:34:59.205 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.206 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:978fc5cc:::benchmark_data_smithi173_130566_object6819:head# 2024-09-17T16:34:59.206 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.206 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97991a1f:::benchmark_data_smithi173_130566_object4337:head# 2024-09-17T16:34:59.206 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.206 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:979bb3fc:::benchmark_data_smithi173_130566_object3780:head# 2024-09-17T16:34:59.206 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.207 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97a12b58:::benchmark_data_smithi173_130566_object7403:head# 2024-09-17T16:34:59.207 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.207 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97a5ca5e:::benchmark_data_smithi173_130566_object7256:head# 2024-09-17T16:34:59.207 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.207 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97a69d07:::benchmark_data_smithi173_130566_object2864:head# 2024-09-17T16:34:59.207 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.208 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97ace031:::benchmark_data_smithi173_130566_object3006:head# 2024-09-17T16:34:59.208 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.208 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97c247f4:::benchmark_data_smithi173_130566_object7711:head# 2024-09-17T16:34:59.208 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.208 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97c93130:::benchmark_data_smithi173_130566_object7841:head# 2024-09-17T16:34:59.208 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.209 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97cb9e8b:::benchmark_data_smithi173_130566_object6270:head# 2024-09-17T16:34:59.209 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.209 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97de1bb9:::benchmark_data_smithi173_130566_object2872:head# 2024-09-17T16:34:59.210 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.210 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97e081d7:::benchmark_data_smithi173_130566_object752:head# 2024-09-17T16:34:59.210 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.210 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97f39fd6:::benchmark_data_smithi173_130566_object2981:head# 2024-09-17T16:34:59.210 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.210 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97f46cf8:::benchmark_data_smithi173_130566_object7219:head# 2024-09-17T16:34:59.210 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.211 INFO:teuthology.orchestra.run.smithi173.stdout:Write #8:97fc8546:::benchmark_data_smithi173_130566_object6498:head# 2024-09-17T16:34:59.211 INFO:teuthology.orchestra.run.smithi173.stdout:snapset 0={} 2024-09-17T16:34:59.211 INFO:teuthology.orchestra.run.smithi173.stdout:write_pg epoch 738 info 8.9( v 733'523 (0'0,733'523] local-lis/les=736/737 n=279 ec=728/728 lis/c=736/736 les/c/f=737/737/0 sis=736) 2024-09-17T16:34:59.213 INFO:teuthology.orchestra.run.smithi173.stdout:Import successful 2024-09-17T16:34:59.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:34:59 smithi017 ceph-mon[27571]: pgmap v1677: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 10193/121407 objects degraded (8.396%); 3764/121407 objects misplaced (3.100%) 2024-09-17T16:34:59.283 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:34:59.313 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:59.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:34:59.416 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:34:59.449 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:34:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:59.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:34:59.551 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:34:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:34:59 smithi071 ceph-mon[30002]: pgmap v1677: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 10193/121407 objects degraded (8.396%); 3764/121407 objects misplaced (3.100%) 2024-09-17T16:34:59.573 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:59.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:34:59.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 45704 45688 59.4843 103.25 0.0157721 0.0168044 2024-09-17T16:34:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:34:59 smithi173 ceph-mon[30530]: pgmap v1677: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 10193/121407 objects degraded (8.396%); 3764/121407 objects misplaced (3.100%) 2024-09-17T16:34:59.674 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:34:59.700 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:59.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:34:59.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:34:59.823 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:34:59.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:59.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:34:59.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:34:59.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:34:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:34:59.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:34:59.993 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:35:00.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:00.076 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:00.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:00.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:00.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:00.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:00.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:00.307 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:00.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:00.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:00.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:00 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:00.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:00 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:00.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:00 smithi017 ceph-mon[27571]: osdmap e754: 12 total, 11 up, 12 in 2024-09-17T16:35:00.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:00 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 10193/121407 objects degraded (8.396%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:00.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:35:00.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:35:00.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:35:00.337 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:35:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:00 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:00 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:00 smithi173 ceph-mon[30530]: osdmap e754: 12 total, 11 up, 12 in 2024-09-17T16:35:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:00 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 10193/121407 objects degraded (8.396%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:35:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:35:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:35:00.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:35:00.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:00.463 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:00.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:00.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/exp.8.9.10 2024-09-17T16:35:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:00 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:00 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:00 smithi071 ceph-mon[30002]: osdmap e754: 12 total, 11 up, 12 in 2024-09-17T16:35:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:00 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 10193/121407 objects degraded (8.396%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:35:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:35:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:35:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:35:00.565 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:00.601 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:00.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:00.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 47022 47006 59.9514 82.375 0.00219038 0.0166748 2024-09-17T16:35:00.691 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:00.698 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:00.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:00.709 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:35:00.709 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:35:00.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:00.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:00.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:00.757 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:35:00.758 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:35:00.795 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:35:00.795 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:35:00.827 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:35:00.827 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:35:00.831 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:00.850 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:35:00.850 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:35:00.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:00.853 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:00.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:00.880 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:35:00.880 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:35:00.907 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:35:00.908 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:35:00.939 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:35:00.939 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:35:00.954 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:00.968 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:35:00.969 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:35:00.979 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:00.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:00.994 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:35:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:35:00.995 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:35:01.027 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:35:01.027 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:35:01.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:01.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:01.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:01.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:01.204 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:01.228 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:01.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:01.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:01.330 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:01.337 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:35:01.337 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:35:01.337 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:35:01.337 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:35:01.338 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:35:01.359 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:01.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:01.462 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:01.484 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:01.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:01.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:01.612 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:01.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:01.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 48104 48088 60.1047 67.625 0.00182176 0.0166316 2024-09-17T16:35:01.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:01.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:01.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: 8.13 scrub starts 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/460803104' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:35:01.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:35:01.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:35:01.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:01 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:01.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:01.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: 8.13 scrub starts 2024-09-17T16:35:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/460803104' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:35:01.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:01 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: 8.13 scrub starts 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/460803104' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.908 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.909 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:35:01.909 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:35:01.909 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.909 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:35:01.909 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:35:01.909 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:35:01.909 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:01 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:35:01.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:01.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:01.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:02.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:02.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:02.124 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:02.225 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:02.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:02.253 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:02.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:02.354 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:02.376 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:35:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:02.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:35:02.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:02.479 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:02.509 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:02.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:02.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:02.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 49600 49584 60.7594 93.5 0.027941 0.0164518 2024-09-17T16:35:02.624 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:02 smithi017 ceph-mon[27571]: 8.16 scrub starts 2024-09-17T16:35:02.624 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:02 smithi017 ceph-mon[27571]: 8.17 scrub starts 2024-09-17T16:35:02.624 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:02 smithi017 ceph-mon[27571]: 8.14 scrub starts 2024-09-17T16:35:02.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:02.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:02.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:02.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:02.763 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:02.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:02.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:02 smithi071 ceph-mon[30002]: 8.16 scrub starts 2024-09-17T16:35:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:02 smithi071 ceph-mon[30002]: 8.17 scrub starts 2024-09-17T16:35:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:02 smithi071 ceph-mon[30002]: 8.14 scrub starts 2024-09-17T16:35:02.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:02.887 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:02.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:02.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:02.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:02 smithi173 ceph-mon[30530]: 8.16 scrub starts 2024-09-17T16:35:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:02 smithi173 ceph-mon[30530]: 8.17 scrub starts 2024-09-17T16:35:02.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:02 smithi173 ceph-mon[30530]: 8.14 scrub starts 2024-09-17T16:35:02.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:03.011 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:03.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:03.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:03.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:03.138 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:03.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:35:03.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:03.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:03.261 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:03.362 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:03.384 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:03.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:03.486 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:03.500 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:03.524 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:03.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:03.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 50773 50757 61.0007 73.3125 0.00181756 0.0163861 2024-09-17T16:35:03.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:03.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:03.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:03.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:03.676 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:03 smithi017 ceph-mon[27571]: 8.19 scrub starts 2024-09-17T16:35:03.676 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:03 smithi017 ceph-mon[27571]: pgmap v1680: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 10741/132627 objects degraded (8.099%); 4410/132627 objects misplaced (3.325%) 2024-09-17T16:35:03.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:03 smithi017 ceph-mon[27571]: 8.15 scrub starts 2024-09-17T16:35:03.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:03 smithi017 ceph-mon[27571]: 8.18 scrub starts 2024-09-17T16:35:03.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:03 smithi017 ceph-mon[27571]: 8.12 scrub starts 2024-09-17T16:35:03.754 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:03.786 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:03.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:03 smithi071 ceph-mon[30002]: 8.19 scrub starts 2024-09-17T16:35:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:03 smithi071 ceph-mon[30002]: pgmap v1680: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 10741/132627 objects degraded (8.099%); 4410/132627 objects misplaced (3.325%) 2024-09-17T16:35:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:03 smithi071 ceph-mon[30002]: 8.15 scrub starts 2024-09-17T16:35:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:03 smithi071 ceph-mon[30002]: 8.18 scrub starts 2024-09-17T16:35:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:03 smithi071 ceph-mon[30002]: 8.12 scrub starts 2024-09-17T16:35:03.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:03.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:03 smithi173 ceph-mon[30530]: 8.19 scrub starts 2024-09-17T16:35:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:03 smithi173 ceph-mon[30530]: pgmap v1680: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 10741/132627 objects degraded (8.099%); 4410/132627 objects misplaced (3.325%) 2024-09-17T16:35:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:03 smithi173 ceph-mon[30530]: 8.15 scrub starts 2024-09-17T16:35:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:03 smithi173 ceph-mon[30530]: 8.18 scrub starts 2024-09-17T16:35:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:03 smithi173 ceph-mon[30530]: 8.12 scrub starts 2024-09-17T16:35:03.911 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:03.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:04.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:04.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:04.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:04.117 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:35:04.117 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:35:04.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17624 6.d 6.ff5d95fd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:21.994454+0000", 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 582.11947978800004, 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.992990979, 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:04.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17624 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.994454+0000", 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.994454+0000", 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.994456+0000", 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9429999999999999e-06 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.994519+0000", 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3274000000000004e-05 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.994533+0000", 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4504999999999999e-05 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.994542+0000", 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4100000000000008e-06 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.994601+0000", 2024-09-17T16:35:04.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9295000000000001e-05 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:21.994610+0000", 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0289999999999994e-06 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.987433+0000", 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.99282284099999996 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.987445+0000", 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1681999999999999e-05 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:17631 6.d 6.46dd643d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e538)", 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:22.000801+0000", 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 582.11313190099997, 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.98669106100000004, 2024-09-17T16:35:04.123 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17631 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000801+0000", 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000801+0000", 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000804+0000", 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3989999999999998e-06 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000883+0000", 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9149999999999999e-05 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000907+0000", 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3782e-05 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000918+0000", 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0841e-05 2024-09-17T16:35:04.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.000991+0000", 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3396e-05 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.001002+0000", 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1155000000000001e-05 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.987485+0000", 2024-09-17T16:35:04.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.98648253200000002 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:22.987493+0000", 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8059999999999995e-06 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90591 6.4 6.c1264da4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:33.605018+0000", 2024-09-17T16:35:04.128 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 510.508915409, 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.91834311899999999, 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90591 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605018+0000", 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605018+0000", 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605020+0000", 2024-09-17T16:35:04.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7099999999999999e-06 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605094+0000", 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4455000000000005e-05 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605104+0000", 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0220999999999999e-05 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605110+0000", 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7640000000000002e-06 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605166+0000", 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6301999999999997e-05 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605174+0000", 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2169999999999997e-06 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.523331+0000", 2024-09-17T16:35:04.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.91815760300000004 2024-09-17T16:35:04.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.523361+0000", 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9847000000000001e-05 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90612 6.4 6.5626e774 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:33.747552+0000", 2024-09-17T16:35:04.134 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 510.36638140000002, 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77617998600000004, 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90612 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747552+0000", 2024-09-17T16:35:04.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747552+0000", 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747554+0000", 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.66e-06 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747600+0000", 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5973000000000002e-05 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747616+0000", 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6330000000000001e-05 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747624+0000", 2024-09-17T16:35:04.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.61e-06 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747684+0000", 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730999999999998e-05 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747693+0000", 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3969999999999994e-06 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.523723+0000", 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77603073499999997 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.523732+0000", 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5499999999999995e-06 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90591 6.4 6:25b26483:::benchmark_data_smithi173_102017_object90590:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e568)", 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:35.553828+0000", 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 508.56010506199999, 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44443183400000003, 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90591 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.140 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553828+0000", 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553828+0000", 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553830+0000", 2024-09-17T16:35:04.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1670000000000002e-06 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553964+0000", 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000133585 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553985+0000", 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0899000000000001e-05 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553999+0000", 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4032000000000001e-05 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.557594+0000", 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035952169999999999 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.557612+0000", 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7753000000000001e-05 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.975806+0000", 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41819402700000002 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.975964+0000", 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015768999999999999 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,5", 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.976138+0000", 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017470499999999999 2024-09-17T16:35:04.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.986303+0000", 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010164289 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.986584+0000", 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00028114299999999997 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.993814+0000", 2024-09-17T16:35:04.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072301520000000001 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.993966+0000", 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015189000000000001 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.998260+0000", 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042942850000000001 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90612 6.4 6:2ee7646a:::benchmark_data_smithi173_102017_object90611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e568)", 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:35.554030+0000", 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 508.55990326900002, 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44921810899999998, 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90612 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554030+0000", 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554030+0000", 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554031+0000", 2024-09-17T16:35:04.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3200000000000005e-07 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554063+0000", 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1903999999999998e-05 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554072+0000", 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9310000000000003e-06 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554079+0000", 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1290000000000001e-06 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.557639+0000", 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035608100000000002 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.557646+0000", 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0559999999999999e-06 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.982926+0000", 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42527986299999998 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.983083+0000", 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015647299999999999 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,5", 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.983247+0000", 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016382500000000001 2024-09-17T16:35:04.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.986311+0000", 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030639030000000002 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.998542+0000", 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012231568999999999 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.998866+0000", 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032433599999999999 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.998917+0000", 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0952999999999998e-05 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:36.003248+0000", 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004330825 2024-09-17T16:35:04.153 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(6.4 582/568 [PushReplyOp(6:2ea410ca:::benchmark_data_smithi173_102017_object31074:head)])", 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:23.443416+0000", 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 460.67051686799999, 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44132895599999999, 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.443416+0000", 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.443416+0000", 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.443418+0000", 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-17T16:35:04.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.443423+0000", 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6659999999999996e-06 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.881101+0000", 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43767767299999999 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.881120+0000", 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8770000000000002e-05 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.881281+0000", 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000160939 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.884745+0000", 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003464894 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:95580 7.d 7:b4431bb8:::benchmark_data_smithi173_116396_object95579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e671)", 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:09.919360+0000", 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 234.19457310499999, 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38153392000000003, 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:04.158 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 95580 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919360+0000", 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919360+0000", 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919362+0000", 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.818e-06 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919391+0000", 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8687000000000002e-05 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919408+0000", 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7034e-05 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919415+0000", 2024-09-17T16:35:04.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4680000000000003e-06 2024-09-17T16:35:04.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919476+0000", 2024-09-17T16:35:04.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0708000000000002e-05 2024-09-17T16:35:04.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919603+0000", 2024-09-17T16:35:04.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012697000000000001 2024-09-17T16:35:04.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919734+0000", 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131531 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.922948+0000", 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003214008 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923916+0000", 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096724599999999995 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300815+0000", 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37689942500000001 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300859+0000", 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4095999999999999e-05 2024-09-17T16:35:04.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300894+0000", 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4928999999999997e-05 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:95580 7.d e671/629 ondisk, result = 0)", 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:09.923561+0000", 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 234.19037270300001, 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37734046100000002, 2024-09-17T16:35:04.165 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923561+0000", 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923561+0000", 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923562+0000", 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9500000000000001e-07 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923565+0000", 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.924e-06 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300712+0000", 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37714677099999999 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300721+0000", 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0140000000000004e-06 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300801+0000", 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9324000000000001e-05 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300809+0000", 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7290000000000006e-06 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300901+0000", 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1804000000000004e-05 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170891 7.b 7:d10181ca:::benchmark_data_smithi173_116396_object75291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.431248+0000", 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 97.682685547999995, 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37566875599999999, 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170891 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431248+0000", 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431248+0000", 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431250+0000", 2024-09-17T16:35:04.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.829e-06 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431256+0000", 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1700000000000002e-06 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431270+0000", 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4051e-05 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431278+0000", 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1939999999999998e-06 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431360+0000", 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2365000000000002e-05 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431498+0000", 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013799400000000001 2024-09-17T16:35:04.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431616+0000", 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117209 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.436464+0000", 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048481389999999996 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.439954+0000", 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034900729999999998 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.806805+0000", 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36685158699999998 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.806875+0000", 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9413999999999996e-05 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.806917+0000", 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1730999999999997e-05 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170904 7.d 7:b74ecf00:::benchmark_data_smithi173_116396_object75304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.444710+0000", 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 97.669223614000003, 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35556337700000001, 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.176 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170904 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444710+0000", 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:04.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444710+0000", 2024-09-17T16:35:04.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444712+0000", 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.874e-06 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444726+0000", 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4463e-05 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444743+0000", 2024-09-17T16:35:04.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7385e-05 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444751+0000", 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8760000000000005e-06 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444808+0000", 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7003999999999998e-05 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444935+0000", 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012665699999999999 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.445085+0000", 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000150008 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.454064+0000", 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089791750000000007 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457533+0000", 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034685369999999998 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.463684+0000", 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061514509999999996 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.800223+0000", 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.336538689 2024-09-17T16:35:04.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.800273+0000", 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0257999999999999e-05 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170913 7.d 7:b6585817:::benchmark_data_smithi173_116396_object75313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.457482+0000", 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 97.656451270999995, 2024-09-17T16:35:04.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35359675800000001, 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170913 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457482+0000", 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457482+0000", 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457484+0000", 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4890000000000001e-06 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457490+0000", 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2790000000000004e-06 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457502+0000", 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2573e-05 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457514+0000", 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1386000000000001e-05 2024-09-17T16:35:04.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457558+0000", 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4502000000000002e-05 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457681+0000", 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012229 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457796+0000", 2024-09-17T16:35:04.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114926 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.462659+0000", 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048635800000000002 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.807157+0000", 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34449779000000003 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.810897+0000", 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003739769 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.811012+0000", 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000115796 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.811079+0000", 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6377999999999994e-05 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170935 7.d 7:b3c7b477:::benchmark_data_smithi173_116396_object75335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.480767+0000", 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 97.633166485999993, 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.344245668, 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:04.191 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170935 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480767+0000", 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480767+0000", 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480769+0000", 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7770000000000001e-06 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480776+0000", 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7079999999999997e-06 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801464+0000", 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32068738499999999 2024-09-17T16:35:04.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801474+0000", 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8600000000000005e-06 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801522+0000", 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7987e-05 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801669+0000", 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014749699999999999 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801798+0000", 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012916300000000001 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.813516+0000", 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011717589 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.818920+0000", 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054044940000000001 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.824864+0000", 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005943444 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.824967+0000", 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103049 2024-09-17T16:35:04.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.825013+0000", 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5714999999999999e-05 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178134 7.d 7:b9b47827:::benchmark_data_smithi173_116396_object82534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.050884+0000", 2024-09-17T16:35:04.197 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 88.063049477999996, 2024-09-17T16:35:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48526529400000001, 2024-09-17T16:35:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178134 2024-09-17T16:35:04.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050884+0000", 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050884+0000", 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050885+0000", 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1299999999999998e-07 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050889+0000", 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7529999999999999e-06 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050899+0000", 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0837e-05 2024-09-17T16:35:04.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050905+0000", 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3229999999999997e-06 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050959+0000", 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4435000000000003e-05 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.051048+0000", 2024-09-17T16:35:04.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8609000000000005e-05 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.051121+0000", 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2887999999999997e-05 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.054958+0000", 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038371149999999999 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055583+0000", 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062568400000000005 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536053+0000", 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48047000499999998 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536105+0000", 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1311999999999999e-05 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536149+0000", 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4419999999999998e-05 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:178134 7.d e717/629 ondisk, result = 0)", 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.055085+0000", 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 88.058848330999993, 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48106911299999999, 2024-09-17T16:35:04.204 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055085+0000", 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055085+0000", 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055086+0000", 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.08e-06 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055091+0000", 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1989999999999997e-06 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.535995+0000", 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48090385899999999 2024-09-17T16:35:04.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.535999+0000", 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9899999999999999e-06 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536041+0000", 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1597999999999997e-05 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536048+0000", 2024-09-17T16:35:04.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3170000000000004e-06 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536154+0000", 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010607000000000001 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178372 7.b 7:d011cddd:::benchmark_data_smithi173_116396_object82772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e718)", 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.909320+0000", 2024-09-17T16:35:04.208 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 87.204612857000001, 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.399272617, 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178372 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909320+0000", 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909320+0000", 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909322+0000", 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6080000000000001e-06 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909329+0000", 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4300000000000003e-06 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909345+0000", 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6188000000000002e-05 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909352+0000", 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7719999999999994e-06 2024-09-17T16:35:04.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909421+0000", 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8571000000000005e-05 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909546+0000", 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012531700000000001 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909678+0000", 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013156999999999999 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.913604+0000", 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039264749999999996 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.915100+0000", 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001495461 2024-09-17T16:35:04.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308513+0000", 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39341283700000002 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308546+0000", 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3504999999999999e-05 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308593+0000", 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6882999999999998e-05 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:178372 7.b e718/629 ondisk, result = 0)", 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 87.199149770999995, 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.393815833, 2024-09-17T16:35:04.215 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9800000000000002e-07 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914790+0000", 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1270000000000002e-06 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308456+0000", 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39366633899999998 2024-09-17T16:35:04.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308463+0000", 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9480000000000002e-06 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308504+0000", 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1247e-05 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308509+0000", 2024-09-17T16:35:04.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2499999999999997e-06 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308599+0000", 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0024000000000004e-05 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(7.0 e725,723 7:04a95b88:::benchmark_data_smithi173_116396_object28410:head,690'6998)", 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 66.154322652999994, 2024-09-17T16:35:04.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89116632100000004, 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4199999999999995e-07 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959615+0000", 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0319999999999997e-06 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850585+0000", 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89097006499999998 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850600+0000", 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4747e-05 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850689+0000", 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8546000000000002e-05 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850701+0000", 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1878e-05 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850777+0000", 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6410999999999997e-05 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.223 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(7.0 e725,723 7:05ce1687:::benchmark_data_smithi173_116396_object29751:head,691'7038)", 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:59.220497+0000", 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.893435986, 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53176433000000001, 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220497+0000", 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220497+0000", 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220498+0000", 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8599999999999998e-07 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220501+0000", 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1209999999999998e-06 2024-09-17T16:35:04.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752133+0000", 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53163205499999999 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752141+0000", 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4950000000000002e-06 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752200+0000", 2024-09-17T16:35:04.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9954000000000002e-05 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752208+0000", 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9810000000000003e-06 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752262+0000", 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3238000000000001e-05 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.227 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.17 740/739 [PushReplyOp(8:ee7fa771:::benchmark_data_smithi173_130566_object904:head)])", 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.249952256999997, 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79826256500000004, 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:04.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.35e-07 2024-09-17T16:35:04.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863983+0000", 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.807e-06 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.661769+0000", 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79778559699999996 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.661776+0000", 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2660000000000004e-06 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.661830+0000", 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3662999999999998e-05 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.662244+0000", 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00041399699999999999 2024-09-17T16:35:04.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.232 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.232 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.232 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:04.232 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:04.232 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:35:04.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:04.256 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:04.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:04.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:04.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:35:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:04.394 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:04.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:04.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:04.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:04.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:04.535 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:35:04.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 52115 52099 61.4322 83.875 0.00623958 0.0162724 2024-09-17T16:35:04.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:04.655 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:04.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:04.729 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:04 smithi017 ceph-mon[27571]: 8.10 scrub starts 2024-09-17T16:35:04.729 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4157809453' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:04.756 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:04.779 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:04.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:04 smithi071 ceph-mon[30002]: 8.10 scrub starts 2024-09-17T16:35:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4157809453' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:04.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:04.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:04 smithi173 ceph-mon[30530]: 8.10 scrub starts 2024-09-17T16:35:04.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4157809453' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:04.903 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:04.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:05.005 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:05.027 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:05.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:05.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:05.163 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:05.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:05.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:05.295 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:05.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:05.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:05.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:05.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:05.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:05.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:05.530 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 7] 2024-09-17T16:35:05.530 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:35:05.530 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-17T16:35:05.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-17T16:35:05.533 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:05 smithi017 ceph-mon[27571]: pgmap v1681: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 12035/144165 objects degraded (8.348%); 4410/144165 objects misplaced (3.059%) 2024-09-17T16:35:05.533 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:05 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:05.533 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4157809453' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:05.534 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:05 smithi017 ceph-mon[27571]: osdmap e755: 12 total, 11 up, 12 in 2024-09-17T16:35:05.534 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:05 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 12035/144165 objects degraded (8.348%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:05.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:05.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:05.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 53506 53490 61.9043 86.9375 0.0179345 0.016148 2024-09-17T16:35:05.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:05.675 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:05.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:05.777 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:05.809 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:05.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:05.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:05 smithi071 ceph-mon[30002]: pgmap v1681: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 12035/144165 objects degraded (8.348%); 4410/144165 objects misplaced (3.059%) 2024-09-17T16:35:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:05 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4157809453' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:05 smithi071 ceph-mon[30002]: osdmap e755: 12 total, 11 up, 12 in 2024-09-17T16:35:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:05 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 12035/144165 objects degraded (8.348%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:05 smithi071 ceph-mon[30002]: osdmap e756: 12 total, 11 up, 12 in 2024-09-17T16:35:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:05 smithi173 ceph-mon[30530]: pgmap v1681: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 12035/144165 objects degraded (8.348%); 4410/144165 objects misplaced (3.059%) 2024-09-17T16:35:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:05 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4157809453' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:05 smithi173 ceph-mon[30530]: osdmap e755: 12 total, 11 up, 12 in 2024-09-17T16:35:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:05 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 12035/144165 objects degraded (8.348%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:05.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:05 smithi173 ceph-mon[30530]: osdmap e756: 12 total, 11 up, 12 in 2024-09-17T16:35:05.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:05.934 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:05.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:05 smithi017 ceph-mon[27571]: osdmap e756: 12 total, 11 up, 12 in 2024-09-17T16:35:06.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:06.060 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:06.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:06.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:06.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:06.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:06.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:06.288 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:06.310 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:06.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:06.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:06.433 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:06.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:06.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:06.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:06.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:06.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 54600 54584 62.0219 68.375 0.0290139 0.0161158 2024-09-17T16:35:06.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:06.691 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:06.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:06.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:06.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:06.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:06.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:06.945 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:06.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:06.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:07.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:07.073 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:07.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:07.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:07.206 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:07.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:07.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:07.307 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:35:07.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:07.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:07.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:07.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:07.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:07.458 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:07.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:07.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:07.584 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:07.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:07.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:07.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 56251 56235 62.7568 103.188 0.0617557 0.0159229 2024-09-17T16:35:07.686 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:07.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:07.709 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:07.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:07.811 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:07 smithi071 ceph-mon[30002]: 8.11 scrub starts 2024-09-17T16:35:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:07 smithi071 ceph-mon[30002]: pgmap v1684: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s; 12418/152613 objects degraded (8.137%); 5049/152613 objects misplaced (3.308%) 2024-09-17T16:35:07.843 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:07.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:07.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:07 smithi173 ceph-mon[30530]: 8.11 scrub starts 2024-09-17T16:35:07.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:07 smithi173 ceph-mon[30530]: pgmap v1684: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s; 12418/152613 objects degraded (8.137%); 5049/152613 objects misplaced (3.308%) 2024-09-17T16:35:07.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:07.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:07.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:07.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:07 smithi017 ceph-mon[27571]: 8.11 scrub starts 2024-09-17T16:35:07.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:07 smithi017 ceph-mon[27571]: pgmap v1684: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.38k op/s; 12418/152613 objects degraded (8.137%); 5049/152613 objects misplaced (3.308%) 2024-09-17T16:35:08.073 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:08.097 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:08.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:08.148 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:08.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:08.224 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:08.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:08.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:08.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:08.360 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:08.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:08.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:08.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:08.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:08.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:08.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:08.587 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:08.610 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:08.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:08.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 57499 57483 63.0241 78 0.0408223 0.015852 2024-09-17T16:35:08.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:08.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:08.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:08.742 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:08.767 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:35:08.768 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:35:08.798 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:35:08.799 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:35:08.823 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:35:08.824 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:35:08.835 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:08.847 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:35:08.848 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:35:08.861 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:08.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:08.873 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:35:08.873 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:35:08.905 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:35:08.905 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:35:08.939 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:35:08.939 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:35:08.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:08.972 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:35:08.972 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:35:08.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:08.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:08.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:09.002 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:35:09.003 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:35:09.037 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:35:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:35:09.037 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:35:09.072 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:35:09.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:35:09.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:09.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:09.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:09.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:09.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:09.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:09.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:09.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:09.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:09.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:09.489 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:09.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:09.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:09.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:09.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:09.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:09.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 58638 58622 63.1647 71.1875 0.0236976 0.0158219 2024-09-17T16:35:09.715 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:09.734 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:09 smithi017 ceph-mon[27571]: pgmap v1685: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 13463/160383 objects degraded (8.394%); 5049/160383 objects misplaced (3.148%) 2024-09-17T16:35:09.734 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1382923438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:09.743 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:09.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:09.801 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:35:09.801 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:35:09.801 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 756, 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "19341aec-750e-11ef-bceb-c7b262605968", 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-17T16:02:46.978716+0000", 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-17T16:35:04.687070+0000", 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-17T16:34:27.728666+0000", 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-17T16:29:47.857157+0000", 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5806080, 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "nodeep-scrub", 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "noscrub", 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-17T16:35:09.802 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 27, 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 8, 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 12, 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:35:09.803 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-17T16:35:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-17T16:35:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-17T16:35:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-17T16:35:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-17T16:35:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-17T16:35:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:06:06.088375+0000", 2024-09-17T16:35:09.804 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:35:09.805 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:35:09.805 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:35:09.805 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-17T16:35:09.806 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "669", 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:35:09.807 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:35:09.808 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:35:09.809 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-17T16:35:09.810 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 11.109999656677246, 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 11.109999656677246, 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.27000001072883606, 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 3, 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 3, 2024-09-17T16:35:09.811 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:35:09.812 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:35:09.812 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:35:09.812 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 8, 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "unique_pool_6", 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-17T16:34:04.745563+0000", 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 3, 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 2, 2024-09-17T16:35:09.813 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 26, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 26, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 26, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 26, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 26, 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-17T16:35:09.814 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "738", 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "736", 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-17T16:35:09.815 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:35:09.816 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:35:09.817 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "rados": {} 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 1.690000057220459, 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 1.690000057220459, 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1, 2024-09-17T16:35:09.818 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 1.690000057220459, 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 1.690000057220459, 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 0.83475363254547119, 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 0.9100000262260437, 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "139c41b7-e38b-42d2-ae0b-70b94041dbfb", 2024-09-17T16:35:09.819 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 8, 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 743, 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.820 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-17T16:35:09.821 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:35:09.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.821 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.821 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-17T16:35:09.821 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:35:09.821 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.821 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.822 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:09 smithi071 ceph-mon[30002]: pgmap v1685: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 13463/160383 objects degraded (8.394%); 5049/160383 objects misplaced (3.148%) 2024-09-17T16:35:09.822 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1382923438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:09.822 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:35:09.824 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-17T16:35:09.825 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3081446730 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6803/3081446730", 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6805/3081446730", 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6809/3081446730", 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6807/3081446730", 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1", 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 13, 2024-09-17T16:35:09.827 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 742, 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-17T16:35:09.828 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.829 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.830 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.831 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 603840826 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6811/603840826", 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6813/603840826", 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6817/603840826", 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6815/603840826", 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.832 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "efb6c3f1-6157-4581-a39d-bda978aea6b2", 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 18, 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 743, 2024-09-17T16:35:09.833 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:35:09.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-17T16:35:09.835 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6824", 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.836 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6825", 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:35:09.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2706983780 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6819/2706983780", 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6821/2706983780", 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6825/2706983780", 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6823/2706983780", 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.838 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37609e4c-d0cb-464f-a3c6-2d6e530149e1", 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 26, 2024-09-17T16:35:09.839 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 742, 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6826", 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.840 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6827", 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6828", 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.841 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6829", 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6832", 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.842 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6833", 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6830", 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6831", 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3512825275 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6827/3512825275", 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6829/3512825275", 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6833/3512825275", 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6831/3512825275", 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.844 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "bd9e473c-bf74-4621-92d0-7031d7db10e5", 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 31, 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 745, 2024-09-17T16:35:09.845 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6800", 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6801", 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.846 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6802", 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6803", 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:35:09.847 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6806", 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6807", 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:35:09.848 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6804", 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6805", 2024-09-17T16:35:09.849 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 146827020 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6801/146827020", 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6803/146827020", 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6807/146827020", 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6805/146827020", 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f08464f8-610f-4775-87d0-803b569eb211", 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.4420013427734375, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 39, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 747, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6808", 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6809", 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.852 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6810", 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6811", 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6814", 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6815", 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.854 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6812", 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6813", 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1837749307 2024-09-17T16:35:09.855 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6809/1837749307", 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6811/1837749307", 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6815/1837749307", 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6813/1837749307", 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "d05170c6-0258-43da-b81b-cc9cc64b5d31", 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.66790771484375, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 46, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 744, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.857 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6816", 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6817", 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.858 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6818", 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6819", 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.859 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6822", 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6823", 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.860 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.861 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.861 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.861 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6820", 2024-09-17T16:35:09.861 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:35:09.861 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:09.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.862 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.862 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6821", 2024-09-17T16:35:09.862 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2384780628 2024-09-17T16:35:09.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6817/2384780628", 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6819/2384780628", 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6823/2384780628", 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6821/2384780628", 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "0d34db33-5d4d-4761-887d-cf1a63dbf3d2", 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.8061676025390625, 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.73492431640625, 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 54, 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 132, 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 154, 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 743, 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 133, 2024-09-17T16:35:09.864 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6824", 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6825", 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.865 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6826", 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6827", 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:35:09.866 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6830", 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6831", 2024-09-17T16:35:09.867 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6828", 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.868 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.71:6829", 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3827414222 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.71:6825/3827414222", 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.71:6827/3827414222", 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.71:6831/3827414222", 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.71:6829/3827414222", 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.869 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "f419836d-0416-4789-80ad-a45ae2ccf035", 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.870 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 61, 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 728, 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6800", 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6801", 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.872 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6802", 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6803", 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.873 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6806", 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6807", 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.874 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6804", 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6805", 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 527945865 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6801/527945865", 2024-09-17T16:35:09.875 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6803/527945865", 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6807/527945865", 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6805/527945865", 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "9dcaa1d8-e921-427c-8a2c-a368658a5bfe", 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.876 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.95001220703125, 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0.2821197509765625, 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 69, 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 723, 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.877 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6808", 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6809", 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6810", 2024-09-17T16:35:09.878 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6811", 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6814", 2024-09-17T16:35:09.879 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6815", 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6812", 2024-09-17T16:35:09.880 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6813", 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3589162352 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6809/3589162352", 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6811/3589162352", 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6815/3589162352", 2024-09-17T16:35:09.881 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6813/3589162352", 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ee08b73f-f02b-4795-87bc-ab559b7faf61", 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 0, 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.882 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 75, 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 739, 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 741, 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6816", 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6817", 2024-09-17T16:35:09.883 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6818", 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6819", 2024-09-17T16:35:09.884 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6822", 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6823", 2024-09-17T16:35:09.885 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6820", 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.886 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6821", 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2428879719 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6817/2428879719", 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6819/2428879719", 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6823/2428879719", 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6821/2428879719", 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: "exists" 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ce178d21-377e-40ac-90fa-46009901114d", 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0.588836669921875, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 0, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 83, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 291, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-17T16:35:09.888 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6824", 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6825", 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.889 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6826", 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6827", 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:35:09.890 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6830", 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6831", 2024-09-17T16:35:09.891 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6828", 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-17T16:35:09.892 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.173:6829", 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1731820377 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.173:6825/1731820377", 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.173:6827/1731820377", 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.173:6831/1731820377", 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.173:6829/1731820377", 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.893 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:24.002604+0000", 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:58.913457+0000", 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.895 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:20.670109+0000", 2024-09-17T16:35:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:35:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-17T16:35:09.896 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:07.664959+0000", 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.897 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:39.473335+0000", 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:19.516336+0000", 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:35:09.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 6, 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:16.514971+0000", 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 7, 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.899 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:19.472096+0000", 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 132 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 8, 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.900 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:28.438089+0000", 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 9, 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:40.277532+0000", 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:35:09.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 10, 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:33:14.888333+0000", 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 740 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 11, 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-17T16:35:09.902 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540719139924082687, 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-17T16:34:06.447156+0000", 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [ 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "1.0", 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: 4, 2024-09-17T16:35:09.903 INFO:teuthology.orchestra.run.smithi017.stdout: 6, 2024-09-17T16:35:09.904 INFO:teuthology.orchestra.run.smithi017.stdout: 8 2024-09-17T16:35:09.904 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.904 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.904 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:35:09.904 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [ 2024-09-17T16:35:09.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.904 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "1.0", 2024-09-17T16:35:09.904 INFO:teuthology.orchestra.run.smithi017.stdout: "mappings": [ 2024-09-17T16:35:09.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.904 INFO:teuthology.orchestra.run.smithi017.stdout: "from": 6, 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: "to": 8 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [ 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "8.10", 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: 4, 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: 7, 2024-09-17T16:35:09.905 INFO:teuthology.orchestra.run.smithi017.stdout: 3 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "8.11", 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: 5, 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: 7, 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: 1 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "8.12", 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-17T16:35:09.906 INFO:teuthology.orchestra.run.smithi017.stdout: 6, 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: 7, 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: 0 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": { 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3205448177": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/250768364": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/394168293": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/4225461221": "2024-09-18T16:03:49.423969+0000", 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/3672343901": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:35:09.907 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/419827434": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6801/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:6800/365103448": "2024-09-18T16:03:21.713930+0000", 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "172.21.15.17:0/2688765754": "2024-09-18T16:03:21.713930+0000" 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-17T16:35:09.908 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-17T16:35:09.909 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:09.910 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:35:09.910 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:09 smithi173 ceph-mon[30530]: pgmap v1685: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 13463/160383 objects degraded (8.394%); 5049/160383 objects misplaced (3.148%) 2024-09-17T16:35:09.910 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1382923438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:09.933 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:09.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:10.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:10.058 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:10.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:10.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:35:10.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:10.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:10.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:10.285 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:10.306 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:10.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:10.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:10.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:10.431 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:10.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:10.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:10.560 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:10.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:10.617 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 59608 59592 63.1216 60.625 0.00695695 0.0158358 2024-09-17T16:35:10.662 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:10.692 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:10.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:10.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:10.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 13463/160383 objects degraded (8.394%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:10.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2457306143' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:35:10.824 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:10.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:10.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 13463/160383 objects degraded (8.394%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:10.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2457306143' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:35:10.896 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 13463/160383 objects degraded (8.394%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:10.896 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2457306143' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-17T16:35:10.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:10.938 INFO:tasks.thrashosds.thrasher:j is {'epoch': 756, 'fsid': '19341aec-750e-11ef-bceb-c7b262605968', 'created': '2024-09-17T16:02:46.978716+0000', 'modified': '2024-09-17T16:35:04.687070+0000', 'last_up_change': '2024-09-17T16:34:27.728666+0000', 'last_in_change': '2024-09-17T16:29:47.857157+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:06:06.088375+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '669', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.109999656677246, 'score_stable': 11.109999656677246, 'optimal_score': 0.27000001072883606, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-09-17T16:34:04.745563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '738', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '736', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.8347536325454712, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': '139c41b7-e38b-42d2-ae0b-70b94041dbfb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 743, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 3081446730}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 3081446730}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 3081446730}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 3081446730}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 3081446730}]}, 'public_addr': '172.21.15.17:6803/3081446730', 'cluster_addr': '172.21.15.17:6805/3081446730', 'heartbeat_back_addr': '172.21.15.17:6809/3081446730', 'heartbeat_front_addr': '172.21.15.17:6807/3081446730', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 742, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 603840826}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 603840826}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 603840826}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 603840826}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 603840826}]}, 'public_addr': '172.21.15.17:6811/603840826', 'cluster_addr': '172.21.15.17:6813/603840826', 'heartbeat_back_addr': '172.21.15.17:6817/603840826', 'heartbeat_front_addr': '172.21.15.17:6815/603840826', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'efb6c3f1-6157-4581-a39d-bda978aea6b2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 743, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 2706983780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 2706983780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6824', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6825', 'nonce': 2706983780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 2706983780}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 2706983780}]}, 'public_addr': '172.21.15.17:6819/2706983780', 'cluster_addr': '172.21.15.17:6821/2706983780', 'heartbeat_back_addr': '172.21.15.17:6825/2706983780', 'heartbeat_front_addr': '172.21.15.17:6823/2706983780', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37609e4c-d0cb-464f-a3c6-2d6e530149e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 742, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6826', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6827', 'nonce': 3512825275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6828', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6829', 'nonce': 3512825275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6832', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6833', 'nonce': 3512825275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6830', 'nonce': 3512825275}, {'type': 'v1', 'addr': '172.21.15.17:6831', 'nonce': 3512825275}]}, 'public_addr': '172.21.15.17:6827/3512825275', 'cluster_addr': '172.21.15.17:6829/3512825275', 'heartbeat_back_addr': '172.21.15.17:6833/3512825275', 'heartbeat_front_addr': '172.21.15.17:6831/3512825275', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'bd9e473c-bf74-4621-92d0-7031d7db10e5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 745, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6800', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6801', 'nonce': 146827020}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6802', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6803', 'nonce': 146827020}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6806', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6807', 'nonce': 146827020}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6804', 'nonce': 146827020}, {'type': 'v1', 'addr': '172.21.15.71:6805', 'nonce': 146827020}]}, 'public_addr': '172.21.15.71:6801/146827020', 'cluster_addr': '172.21.15.71:6803/146827020', 'heartbeat_back_addr': '172.21.15.71:6807/146827020', 'heartbeat_front_addr': '172.21.15.71:6805/146827020', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f08464f8-610f-4775-87d0-803b569eb211', 'up': 1, 'in': 1, 'weight': 0.4420013427734375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 747, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6808', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6809', 'nonce': 1837749307}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6810', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6811', 'nonce': 1837749307}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6814', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6815', 'nonce': 1837749307}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6812', 'nonce': 1837749307}, {'type': 'v1', 'addr': '172.21.15.71:6813', 'nonce': 1837749307}]}, 'public_addr': '172.21.15.71:6809/1837749307', 'cluster_addr': '172.21.15.71:6811/1837749307', 'heartbeat_back_addr': '172.21.15.71:6815/1837749307', 'heartbeat_front_addr': '172.21.15.71:6813/1837749307', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'd05170c6-0258-43da-b81b-cc9cc64b5d31', 'up': 1, 'in': 1, 'weight': 0.66790771484375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 744, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6816', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6817', 'nonce': 2384780628}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6818', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6819', 'nonce': 2384780628}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6822', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6823', 'nonce': 2384780628}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6820', 'nonce': 2384780628}, {'type': 'v1', 'addr': '172.21.15.71:6821', 'nonce': 2384780628}]}, 'public_addr': '172.21.15.71:6817/2384780628', 'cluster_addr': '172.21.15.71:6819/2384780628', 'heartbeat_back_addr': '172.21.15.71:6823/2384780628', 'heartbeat_front_addr': '172.21.15.71:6821/2384780628', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0d34db33-5d4d-4761-887d-cf1a63dbf3d2', 'up': 1, 'in': 1, 'weight': 0.8061676025390625, 'primary_affinity': 0.73492431640625, 'last_clean_begin': 54, 'last_clean_end': 132, 'up_from': 154, 'up_thru': 743, 'down_at': 133, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6824', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6825', 'nonce': 3827414222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6826', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6827', 'nonce': 3827414222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6830', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6831', 'nonce': 3827414222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6828', 'nonce': 3827414222}, {'type': 'v1', 'addr': '172.21.15.71:6829', 'nonce': 3827414222}]}, 'public_addr': '172.21.15.71:6825/3827414222', 'cluster_addr': '172.21.15.71:6827/3827414222', 'heartbeat_back_addr': '172.21.15.71:6831/3827414222', 'heartbeat_front_addr': '172.21.15.71:6829/3827414222', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'f419836d-0416-4789-80ad-a45ae2ccf035', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 728, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 527945865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 527945865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 527945865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 527945865}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 527945865}]}, 'public_addr': '172.21.15.173:6801/527945865', 'cluster_addr': '172.21.15.173:6803/527945865', 'heartbeat_back_addr': '172.21.15.173:6807/527945865', 'heartbeat_front_addr': '172.21.15.173:6805/527945865', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9dcaa1d8-e921-427c-8a2c-a368658a5bfe', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 0.2821197509765625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 723, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3589162352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3589162352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3589162352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3589162352}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 3589162352}]}, 'public_addr': '172.21.15.173:6809/3589162352', 'cluster_addr': '172.21.15.173:6811/3589162352', 'heartbeat_back_addr': '172.21.15.173:6815/3589162352', 'heartbeat_front_addr': '172.21.15.173:6813/3589162352', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'ee08b73f-f02b-4795-87bc-ab559b7faf61', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 75, 'up_thru': 739, 'down_at': 741, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 2428879719}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 2428879719}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 2428879719}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 2428879719}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 2428879719}]}, 'public_addr': '172.21.15.173:6817/2428879719', 'cluster_addr': '172.21.15.173:6819/2428879719', 'heartbeat_back_addr': '172.21.15.173:6823/2428879719', 'heartbeat_front_addr': '172.21.15.173:6821/2428879719', 'state': ['exists']}, {'osd': 11, 'uuid': 'ce178d21-377e-40ac-90fa-46009901114d', 'up': 1, 'in': 1, 'weight': 0.588836669921875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 83, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 1731820377}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 1731820377}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 1731820377}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 1731820377}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 1731820377}]}, 'public_addr': '172.21.15.173:6825/1731820377', 'cluster_addr': '172.21.15.173:6827/1731820377', 'heartbeat_back_addr': '172.21.15.173:6831/1731820377', 'heartbeat_front_addr': '172.21.15.173:6829/1731820377', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:24.002604+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:58.913457+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:20.670109+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:07.664959+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:39.473335+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:19.516336+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:16.514971+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:19.472096+0000', 'dead_epoch': 132}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:28.438089+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:40.277532+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:33:14.888333+0000', 'dead_epoch': 740}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:34:06.447156+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [4, 6, 8]}], 'pg_upmap_items': [{'pgid': '1.0', 'mappings': [{'from': 6, 'to': 8}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.10', 'osds': [4, 7, 3]}, {'pgid': '8.11', 'osds': [5, 7, 1]}, {'pgid': '8.12', 'osds': [6, 7, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.17:0/3205448177': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6801/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:6800/250768364': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/394168293': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/4225461221': '2024-09-18T16:03:49.423969+0000', '172.21.15.17:0/3672343901': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/419827434': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6801/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:6800/365103448': '2024-09-18T16:03:21.713930+0000', '172.21.15.17:0/2688765754': '2024-09-18T16:03:21.713930+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:35:10.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:35:10.951 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:10.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:11.053 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:11.093 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:11.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:11.191 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:35:11.191 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:35:11.195 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:11.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:11.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:11.335 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:11.362 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:11.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:11.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:11.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:11.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:11.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:11.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:11.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:11.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:11.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:35:11.616675+0000 min lat: 0.00112054 max lat: 15.0115 avg lat: 0.0157006 2024-09-17T16:35:11.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:35:11.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 61141 61125 63.6663 95.8125 0.00523047 0.0157006 2024-09-17T16:35:11.716 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:11.731 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:35:11.732 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:35:11.742 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:11.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:11.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:11 smithi071 ceph-mon[30002]: pgmap v1686: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s; 13887/169659 objects degraded (8.185%); 5277/169659 objects misplaced (3.110%) 2024-09-17T16:35:11.843 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:11.868 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:11.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:11.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:11 smithi173 ceph-mon[30530]: pgmap v1686: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s; 13887/169659 objects degraded (8.185%); 5277/169659 objects misplaced (3.110%) 2024-09-17T16:35:11.970 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:11.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:11 smithi017 ceph-mon[27571]: pgmap v1686: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s; 13887/169659 objects degraded (8.185%); 5277/169659 objects misplaced (3.110%) 2024-09-17T16:35:11.998 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:12.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:12.102 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:12.127 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:12.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:12.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:12.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:12.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:12.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:12.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:12.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:12.486 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:12.510 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:12.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:12.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:12.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 62370 62354 63.8817 76.8125 0.00198855 0.01563 2024-09-17T16:35:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:12.634 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:12.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:12.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:12.770 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:12.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:12.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:35:12.800 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:12.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:12.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:12.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:12.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:12.999 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:13.031 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:13.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:13.133 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:13.169 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:13.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:13.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:13.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:13.300 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:13.363 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:13.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:13.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:13.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:13.527 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:13.555 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:13.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:13.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 63841 63825 64.334 91.9375 0.00296899 0.0155309 2024-09-17T16:35:13.624 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:13 smithi017 ceph-mon[27571]: pgmap v1687: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s; 14062/171876 objects degraded (8.181%); 5652/171876 objects misplaced (3.288%) 2024-09-17T16:35:13.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:13.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:13.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:13.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:35:13.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:13.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:13.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:13.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:13.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:13 smithi071 ceph-mon[30002]: pgmap v1687: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s; 14062/171876 objects degraded (8.181%); 5652/171876 objects misplaced (3.288%) 2024-09-17T16:35:13.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:13 smithi173 ceph-mon[30530]: pgmap v1687: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s; 14062/171876 objects degraded (8.181%); 5652/171876 objects misplaced (3.288%) 2024-09-17T16:35:13.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:13.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:14.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:14.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:14.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:14.157 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:14.179 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:14.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:14.263 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:14.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:14.335 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:14.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:14.437 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:14.461 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:14.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:14.552 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:35:14.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:14.584 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:14.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:14.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 64612 64596 64.0777 48.1875 0.00249642 0.0155995 2024-09-17T16:35:14.686 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:14.718 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:14.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:14.804 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:35:14.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4141619545' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:14.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:14 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:14.805 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:35:14.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4141619545' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:14.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:14 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:14.820 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:14.849 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:14.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:14.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4141619545' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:14 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:14.952 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:14.979 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:14.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:15.081 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:15.105 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:15.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:15.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:15.229 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:15.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:15.331 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:15.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:15.354 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:15.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:15.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:15.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:15.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:15.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:15.582 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:15.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 15 65881 65866 64.3166 79.375 0.0082376 0.0155432 2024-09-17T16:35:15.619 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:15.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:15.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:15.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:15.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:15 smithi071 ceph-mon[30002]: pgmap v1688: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s; 15245/182382 objects degraded (8.359%); 5652/182382 objects misplaced (3.099%) 2024-09-17T16:35:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:15 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:15 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:35:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:15 smithi071 ceph-mon[30002]: osdmap e757: 12 total, 11 up, 12 in 2024-09-17T16:35:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:15 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 15245/182382 objects degraded (8.359%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:15 smithi071 ceph-mon[30002]: osdmap e758: 12 total, 11 up, 12 in 2024-09-17T16:35:15.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:15.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:15.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:15 smithi173 ceph-mon[30530]: pgmap v1688: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s; 15245/182382 objects degraded (8.359%); 5652/182382 objects misplaced (3.099%) 2024-09-17T16:35:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:15 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:15 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:35:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:15 smithi173 ceph-mon[30530]: osdmap e757: 12 total, 11 up, 12 in 2024-09-17T16:35:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:15 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 15245/182382 objects degraded (8.359%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:15.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:15 smithi173 ceph-mon[30530]: osdmap e758: 12 total, 11 up, 12 in 2024-09-17T16:35:15.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:15 smithi017 ceph-mon[27571]: pgmap v1688: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s; 15245/182382 objects degraded (8.359%); 5652/182382 objects misplaced (3.099%) 2024-09-17T16:35:15.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:15 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:15.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:15 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:35:15.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:15 smithi017 ceph-mon[27571]: osdmap e757: 12 total, 11 up, 12 in 2024-09-17T16:35:15.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:15 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 15245/182382 objects degraded (8.359%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:15.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:15 smithi017 ceph-mon[27571]: osdmap e758: 12 total, 11 up, 12 in 2024-09-17T16:35:15.977 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:16.006 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:16.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:16.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:16.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:16.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:16.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:16.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:16.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:16.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:16.334 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1690,"stamp":"2024-09-17T16:35:14.692626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4060381728,"num_objects":61951,"num_object_clones":0,"num_object_copies":185853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15245,"num_objects_misplaced":5652,"num_objects_unfound":0,"num_objects_dirty":61951,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":60615,"num_write_kb":3873542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":83436608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67091,"ondisk_log_size":67091,"up":70,"acting":73,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":18125064,"kb_used_data":11168320,"kb_used_omap":58,"kb_used_meta":6956485,"kb_avail":1106816760,"statfs":{"total":1151940427776,"available":1133380362240,"internally_reserved":0,"allocated":11436359680,"data_stored":11431944848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60253,"internal_metadata":7123440803},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":726138880,"num_objects":11080,"num_object_clones":0,"num_object_copies":33240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2827,"num_objects_misplaced":603,"num_objects_unfound":0,"num_objects_dirty":11080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11080,"num_write_kb":709120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.619078"},"pg_stats":[{"pgid":"8.19","version":"756'2182","reported_seq":2246,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:10.704623+0000","last_change":"2024-09-17T16:35:01.875330+0000","last_active":"2024-09-17T16:35:10.704623+0000","last_peered":"2024-09-17T16:35:10.704623+0000","last_clean":"2024-09-17T16:35:10.704623+0000","last_became_active":"2024-09-17T16:34:30.685252+0000","last_became_peered":"2024-09-17T16:34:30.685252+0000","last_unstale":"2024-09-17T16:35:10.704623+0000","last_undegraded":"2024-09-17T16:35:10.704623+0000","last_fullsized":"2024-09-17T16:35:10.704623+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":744,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":2182,"log_dups_size":0,"ondisk_log_size":2182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:31.868062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125861888,"num_objects":1921,"num_object_clones":0,"num_object_copies":5763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1921,"num_write_kb":122912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":15990784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,8],"acting":[5,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.18","version":"756'2139","reported_seq":2219,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:09.941766+0000","last_change":"2024-09-17T16:35:02.547224+0000","last_active":"2024-09-17T16:35:09.941766+0000","last_peered":"2024-09-17T16:35:09.941766+0000","last_clean":"2024-09-17T16:35:09.941766+0000","last_became_active":"2024-09-17T16:34:30.684458+0000","last_became_peered":"2024-09-17T16:34:30.684458+0000","last_unstale":"2024-09-17T16:35:09.941766+0000","last_undegraded":"2024-09-17T16:35:09.941766+0000","last_fullsized":"2024-09-17T16:35:09.941766+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":744,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":2139,"log_dups_size":0,"ondisk_log_size":2139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:32.513136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123699200,"num_objects":1888,"num_object_clones":0,"num_object_copies":5664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1888,"num_write_kb":120800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,5],"acting":[4,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.14","version":"756'2118","reported_seq":2200,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:09.899364+0000","last_change":"2024-09-17T16:35:01.573326+0000","last_active":"2024-09-17T16:35:09.899364+0000","last_peered":"2024-09-17T16:35:09.899364+0000","last_clean":"2024-09-17T16:35:09.899364+0000","last_became_active":"2024-09-17T16:34:30.684351+0000","last_became_peered":"2024-09-17T16:34:30.684351+0000","last_unstale":"2024-09-17T16:35:09.899364+0000","last_undegraded":"2024-09-17T16:35:09.899364+0000","last_fullsized":"2024-09-17T16:35:09.899364+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":744,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":2118,"log_dups_size":0,"ondisk_log_size":2118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:31.495130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122781696,"num_objects":1874,"num_object_clones":0,"num_object_copies":5622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1874,"num_write_kb":119904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.13","version":"756'2109","reported_seq":2193,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:12.079128+0000","last_change":"2024-09-17T16:35:00.644309+0000","last_active":"2024-09-17T16:35:12.079128+0000","last_peered":"2024-09-17T16:35:12.079128+0000","last_clean":"2024-09-17T16:35:12.079128+0000","last_became_active":"2024-09-17T16:34:29.684658+0000","last_became_peered":"2024-09-17T16:34:29.684658+0000","last_unstale":"2024-09-17T16:35:12.079128+0000","last_undegraded":"2024-09-17T16:35:12.079128+0000","last_fullsized":"2024-09-17T16:35:12.079128+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":2109,"log_dups_size":0,"ondisk_log_size":2109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:30.632635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122880000,"num_objects":1875,"num_object_clones":0,"num_object_copies":5625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1875,"num_write_kb":120000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":15597568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,9],"acting":[3,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.12","version":"756'2100","reported_seq":2159,"reported_epoch":756,"state":"active+clean+remapped","last_fresh":"2024-09-17T16:35:10.643146+0000","last_change":"2024-09-17T16:35:02.704886+0000","last_active":"2024-09-17T16:35:10.643146+0000","last_peered":"2024-09-17T16:35:10.643146+0000","last_clean":"2024-09-17T16:35:10.643146+0000","last_became_active":"2024-09-17T16:34:31.685021+0000","last_became_peered":"2024-09-17T16:34:31.685021+0000","last_unstale":"2024-09-17T16:35:10.643146+0000","last_undegraded":"2024-09-17T16:35:10.643146+0000","last_fullsized":"2024-09-17T16:35:10.643146+0000","mapping_epoch":744,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":745,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:32.695324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120881152,"num_objects":1845,"num_object_clones":0,"num_object_copies":5535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1845,"num_objects_unfound":0,"num_objects_dirty":1845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1845,"num_write_kb":118048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7,0],"avail_no_missing":["6","0","7"],"object_location_counts":[{"shards":"0,6,7","objects":1845}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"756'2194","reported_seq":3087,"reported_epoch":756,"state":"active+clean+remapped","last_fresh":"2024-09-17T16:35:10.735378+0000","last_change":"2024-09-17T16:35:05.864318+0000","last_active":"2024-09-17T16:35:10.735378+0000","last_peered":"2024-09-17T16:35:10.735378+0000","last_clean":"2024-09-17T16:35:10.735378+0000","last_became_active":"2024-09-17T16:34:34.683700+0000","last_became_peered":"2024-09-17T16:34:34.683700+0000","last_unstale":"2024-09-17T16:35:10.735378+0000","last_undegraded":"2024-09-17T16:35:10.735378+0000","last_fullsized":"2024-09-17T16:35:10.735378+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":2194,"log_dups_size":0,"ondisk_log_size":2194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:35.856921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126287872,"num_objects":1927,"num_object_clones":0,"num_object_copies":5781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1927,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1927,"num_write_kb":123328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":17891328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,1],"avail_no_missing":["5","1","7"],"object_location_counts":[{"shards":"1,5,7","objects":1927}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"756'2127","reported_seq":2699,"reported_epoch":756,"state":"active+clean+remapped","last_fresh":"2024-09-17T16:35:09.832522+0000","last_change":"2024-09-17T16:35:03.511281+0000","last_active":"2024-09-17T16:35:09.832522+0000","last_peered":"2024-09-17T16:35:09.832522+0000","last_clean":"2024-09-17T16:35:09.832522+0000","last_became_active":"2024-09-17T16:34:32.686875+0000","last_became_peered":"2024-09-17T16:34:32.686875+0000","last_unstale":"2024-09-17T16:35:09.832522+0000","last_undegraded":"2024-09-17T16:35:09.832522+0000","last_fullsized":"2024-09-17T16:35:09.832522+0000","mapping_epoch":745,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":746,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":2127,"log_dups_size":0,"ondisk_log_size":2127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:33.480689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123174912,"num_objects":1880,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1880,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1880,"num_write_kb":120288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,3],"avail_no_missing":["4","3","7"],"object_location_counts":[{"shards":"3,4,7","objects":1880}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"756'3898","reported_seq":3967,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:12.400657+0000","last_change":"2024-09-17T16:34:06.715555+0000","last_active":"2024-09-17T16:35:12.400657+0000","last_peered":"2024-09-17T16:35:12.400657+0000","last_clean":"2024-09-17T16:35:12.400657+0000","last_became_active":"2024-09-17T16:34:06.715239+0000","last_became_peered":"2024-09-17T16:34:06.715239+0000","last_unstale":"2024-09-17T16:35:12.400657+0000","last_undegraded":"2024-09-17T16:35:12.400657+0000","last_fullsized":"2024-09-17T16:35:12.400657+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255459328,"num_objects":3898,"num_object_clones":0,"num_object_copies":11694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":249472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,1],"acting":[2,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"756'3833","reported_seq":3903,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:12.135729+0000","last_change":"2024-09-17T16:34:06.728922+0000","last_active":"2024-09-17T16:35:12.135729+0000","last_peered":"2024-09-17T16:35:12.135729+0000","last_clean":"2024-09-17T16:35:12.135729+0000","last_became_active":"2024-09-17T16:34:06.728580+0000","last_became_peered":"2024-09-17T16:34:06.728580+0000","last_unstale":"2024-09-17T16:35:12.135729+0000","last_undegraded":"2024-09-17T16:35:12.135729+0000","last_fullsized":"2024-09-17T16:35:12.135729+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:31.232509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251199488,"num_objects":3833,"num_object_clones":0,"num_object_copies":11499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3833,"num_write_kb":245312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"756'3974","reported_seq":4044,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:13.110677+0000","last_change":"2024-09-17T16:34:06.715248+0000","last_active":"2024-09-17T16:35:13.110677+0000","last_peered":"2024-09-17T16:35:13.110677+0000","last_clean":"2024-09-17T16:35:13.110677+0000","last_became_active":"2024-09-17T16:34:06.714898+0000","last_became_peered":"2024-09-17T16:34:06.714898+0000","last_unstale":"2024-09-17T16:35:13.110677+0000","last_undegraded":"2024-09-17T16:35:13.110677+0000","last_fullsized":"2024-09-17T16:35:13.110677+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":3974,"log_dups_size":0,"ondisk_log_size":3974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:36:11.478619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260440064,"num_objects":3974,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":254336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,2],"acting":[7,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"756'3922","reported_seq":3991,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:14.520882+0000","last_change":"2024-09-17T16:34:07.122358+0000","last_active":"2024-09-17T16:35:14.520882+0000","last_peered":"2024-09-17T16:35:14.520882+0000","last_clean":"2024-09-17T16:35:14.520882+0000","last_became_active":"2024-09-17T16:34:07.121994+0000","last_became_peered":"2024-09-17T16:34:07.121994+0000","last_unstale":"2024-09-17T16:35:14.520882+0000","last_undegraded":"2024-09-17T16:35:14.520882+0000","last_fullsized":"2024-09-17T16:35:14.520882+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":3922,"log_dups_size":0,"ondisk_log_size":3922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:24.673106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257032192,"num_objects":3922,"num_object_clones":0,"num_object_copies":11766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3922,"num_write_kb":251008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.b","version":"756'4123","reported_seq":4192,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:14.537989+0000","last_change":"2024-09-17T16:34:06.729072+0000","last_active":"2024-09-17T16:35:14.537989+0000","last_peered":"2024-09-17T16:35:14.537989+0000","last_clean":"2024-09-17T16:35:14.537989+0000","last_became_active":"2024-09-17T16:34:06.728735+0000","last_became_peered":"2024-09-17T16:34:06.728735+0000","last_unstale":"2024-09-17T16:35:14.537989+0000","last_undegraded":"2024-09-17T16:35:14.537989+0000","last_fullsized":"2024-09-17T16:35:14.537989+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":4123,"log_dups_size":0,"ondisk_log_size":4123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:31.259161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270204928,"num_objects":4123,"num_object_clones":0,"num_object_copies":12369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4123,"num_write_kb":263872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.17","version":"756'2235","reported_seq":2317,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:13.080245+0000","last_change":"2024-09-17T16:35:01.556037+0000","last_active":"2024-09-17T16:35:13.080245+0000","last_peered":"2024-09-17T16:35:13.080245+0000","last_clean":"2024-09-17T16:35:13.080245+0000","last_became_active":"2024-09-17T16:34:30.685417+0000","last_became_peered":"2024-09-17T16:34:30.685417+0000","last_unstale":"2024-09-17T16:35:13.080245+0000","last_undegraded":"2024-09-17T16:35:13.080245+0000","last_fullsized":"2024-09-17T16:35:13.080245+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":744,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":2235,"log_dups_size":0,"ondisk_log_size":2235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:31.413422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129892352,"num_objects":1982,"num_object_clones":0,"num_object_copies":5946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1982,"num_write_kb":126848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"756'4157","reported_seq":4226,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:14.523915+0000","last_change":"2024-09-17T16:34:06.711362+0000","last_active":"2024-09-17T16:35:14.523915+0000","last_peered":"2024-09-17T16:35:14.523915+0000","last_clean":"2024-09-17T16:35:14.523915+0000","last_became_active":"2024-09-17T16:34:06.711038+0000","last_became_peered":"2024-09-17T16:34:06.711038+0000","last_unstale":"2024-09-17T16:35:14.523915+0000","last_undegraded":"2024-09-17T16:35:14.523915+0000","last_fullsized":"2024-09-17T16:35:14.523915+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":4157,"log_dups_size":0,"ondisk_log_size":4157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:22.044429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272433152,"num_objects":4157,"num_object_clones":0,"num_object_copies":12471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4157,"num_write_kb":266048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,3],"acting":[8,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.15","version":"756'2148","reported_seq":2212,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:13.092687+0000","last_change":"2024-09-17T16:35:02.473476+0000","last_active":"2024-09-17T16:35:13.092687+0000","last_peered":"2024-09-17T16:35:13.092687+0000","last_clean":"2024-09-17T16:35:13.092687+0000","last_became_active":"2024-09-17T16:34:30.686385+0000","last_became_peered":"2024-09-17T16:34:30.686385+0000","last_unstale":"2024-09-17T16:35:13.092687+0000","last_undegraded":"2024-09-17T16:35:13.092687+0000","last_fullsized":"2024-09-17T16:35:13.092687+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":744,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:32.461449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124289024,"num_objects":1897,"num_object_clones":0,"num_object_copies":5691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1897,"num_write_kb":121376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.8","version":"756'2205","reported_seq":2296,"reported_epoch":756,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:35:12.100166+0000","last_change":"2024-09-17T16:34:28.739653+0000","last_active":"2024-09-17T16:35:12.100166+0000","last_peered":"2024-09-17T16:35:12.100166+0000","last_clean":"2024-09-17T16:34:26.723877+0000","last_became_active":"2024-09-17T16:34:28.739653+0000","last_became_peered":"2024-09-17T16:34:28.739653+0000","last_unstale":"2024-09-17T16:35:12.100166+0000","last_undegraded":"2024-09-17T16:34:28.736513+0000","last_fullsized":"2024-09-17T16:34:28.736407+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'503","last_scrub_stamp":"2024-09-17T16:34:24.399773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:24.399773+0000","objects_scrubbed":237,"log_size":2205,"log_dups_size":0,"ondisk_log_size":2205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:25.111104+0000","scrub_duration":584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127074304,"num_objects":1939,"num_object_clones":0,"num_object_copies":5817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1702,"num_write_kb":108928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":["3","9"],"object_location_counts":[{"shards":"3,9","objects":1939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.7","version":"756'2186","reported_seq":2277,"reported_epoch":756,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:35:12.517091+0000","last_change":"2024-09-17T16:34:28.746844+0000","last_active":"2024-09-17T16:35:12.517091+0000","last_peered":"2024-09-17T16:35:12.517091+0000","last_clean":"2024-09-17T16:34:26.712015+0000","last_became_active":"2024-09-17T16:34:28.746844+0000","last_became_peered":"2024-09-17T16:34:28.746844+0000","last_unstale":"2024-09-17T16:35:12.517091+0000","last_undegraded":"2024-09-17T16:34:28.738236+0000","last_fullsized":"2024-09-17T16:34:28.738124+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'506","last_scrub_stamp":"2024-09-17T16:34:25.830136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:25.830136+0000","objects_scrubbed":242,"log_size":2186,"log_dups_size":0,"ondisk_log_size":2186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:38.641513+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125960192,"num_objects":1922,"num_object_clones":0,"num_object_copies":5766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1680,"num_write_kb":107520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","11"],"object_location_counts":[{"shards":"1,11","objects":1922}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.6","version":"756'2196","reported_seq":2253,"reported_epoch":756,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:35:13.063723+0000","last_change":"2024-09-17T16:34:29.165721+0000","last_active":"2024-09-17T16:35:13.063723+0000","last_peered":"2024-09-17T16:35:13.063723+0000","last_clean":"2024-09-17T16:34:17.399008+0000","last_became_active":"2024-09-17T16:34:29.165721+0000","last_became_peered":"2024-09-17T16:34:29.165721+0000","last_unstale":"2024-09-17T16:35:13.063723+0000","last_undegraded":"2024-09-17T16:34:28.736698+0000","last_fullsized":"2024-09-17T16:34:28.736582+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'479","last_scrub_stamp":"2024-09-17T16:34:24.454489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:24.454489+0000","objects_scrubbed":0,"log_size":2196,"log_dups_size":0,"ondisk_log_size":2196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:29.862687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128221184,"num_objects":1956,"num_object_clones":0,"num_object_copies":5868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1956,"num_write_kb":125216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1956}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"756'2053","reported_seq":2110,"reported_epoch":756,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:35:09.792911+0000","last_change":"2024-09-17T16:34:28.740238+0000","last_active":"2024-09-17T16:35:09.792911+0000","last_peered":"2024-09-17T16:35:09.792911+0000","last_clean":"2024-09-17T16:34:17.385589+0000","last_became_active":"2024-09-17T16:34:28.740238+0000","last_became_peered":"2024-09-17T16:34:28.740238+0000","last_unstale":"2024-09-17T16:35:09.792911+0000","last_undegraded":"2024-09-17T16:34:28.735423+0000","last_fullsized":"2024-09-17T16:34:28.735306+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'503","last_scrub_stamp":"2024-09-17T16:34:26.428661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:26.428661+0000","objects_scrubbed":0,"log_size":2053,"log_dups_size":0,"ondisk_log_size":2053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:29.393103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118063104,"num_objects":1801,"num_object_clones":0,"num_object_copies":5403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1801,"num_write_kb":115296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1801}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"756'2021","reported_seq":2111,"reported_epoch":756,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:35:10.723002+0000","last_change":"2024-09-17T16:34:28.740724+0000","last_active":"2024-09-17T16:35:10.723002+0000","last_peered":"2024-09-17T16:35:10.723002+0000","last_clean":"2024-09-17T16:34:26.718664+0000","last_became_active":"2024-09-17T16:34:28.740724+0000","last_became_peered":"2024-09-17T16:34:28.740724+0000","last_unstale":"2024-09-17T16:35:10.723002+0000","last_undegraded":"2024-09-17T16:34:28.738338+0000","last_fullsized":"2024-09-17T16:34:28.738222+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'489","last_scrub_stamp":"2024-09-17T16:34:24.567630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:24.567630+0000","objects_scrubbed":243,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:25.041805+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116326400,"num_objects":1775,"num_object_clones":0,"num_object_copies":5325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1532,"num_write_kb":98048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":["5","9"],"object_location_counts":[{"shards":"5,9","objects":1775}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3","version":"756'2156","reported_seq":2243,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:12.551984+0000","last_change":"2024-09-17T16:34:24.839944+0000","last_active":"2024-09-17T16:35:12.551984+0000","last_peered":"2024-09-17T16:35:12.551984+0000","last_clean":"2024-09-17T16:35:12.551984+0000","last_became_active":"2024-09-17T16:34:23.702789+0000","last_became_peered":"2024-09-17T16:34:23.702789+0000","last_unstale":"2024-09-17T16:35:12.551984+0000","last_undegraded":"2024-09-17T16:35:12.551984+0000","last_fullsized":"2024-09-17T16:35:12.551984+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'468","last_scrub_stamp":"2024-09-17T16:34:24.839844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:24.839844+0000","objects_scrubbed":230,"log_size":2156,"log_dups_size":0,"ondisk_log_size":2156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:30.429356+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125698048,"num_objects":1918,"num_object_clones":0,"num_object_copies":5754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1688,"num_write_kb":108032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,6],"acting":[1,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"756'2113","reported_seq":2168,"reported_epoch":756,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:35:12.116663+0000","last_change":"2024-09-17T16:34:28.738542+0000","last_active":"2024-09-17T16:35:12.116663+0000","last_peered":"2024-09-17T16:35:12.116663+0000","last_clean":"2024-09-17T16:34:17.020462+0000","last_became_active":"2024-09-17T16:34:28.738542+0000","last_became_peered":"2024-09-17T16:34:28.738542+0000","last_unstale":"2024-09-17T16:35:12.116663+0000","last_undegraded":"2024-09-17T16:34:28.736302+0000","last_fullsized":"2024-09-17T16:34:28.736173+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'511","last_scrub_stamp":"2024-09-17T16:34:27.458705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:27.458705+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:29.245201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121733120,"num_objects":1857,"num_object_clones":0,"num_object_copies":5571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1857,"num_write_kb":118880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":1857}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"756'2270","reported_seq":2361,"reported_epoch":756,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:35:12.542188+0000","last_change":"2024-09-17T16:34:28.751837+0000","last_active":"2024-09-17T16:35:12.542188+0000","last_peered":"2024-09-17T16:35:12.542188+0000","last_clean":"2024-09-17T16:34:26.706822+0000","last_became_active":"2024-09-17T16:34:28.751837+0000","last_became_peered":"2024-09-17T16:34:28.751837+0000","last_unstale":"2024-09-17T16:35:12.542188+0000","last_undegraded":"2024-09-17T16:34:28.737641+0000","last_fullsized":"2024-09-17T16:34:28.737529+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'534","last_scrub_stamp":"2024-09-17T16:34:23.918428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:23.918428+0000","objects_scrubbed":261,"log_size":2270,"log_dups_size":0,"ondisk_log_size":2270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:45.789487+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130875392,"num_objects":1997,"num_object_clones":0,"num_object_copies":5991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1736,"num_write_kb":111104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1997}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"756'2169","reported_seq":2256,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:12.092133+0000","last_change":"2024-09-17T16:34:24.868144+0000","last_active":"2024-09-17T16:35:12.092133+0000","last_peered":"2024-09-17T16:35:12.092133+0000","last_clean":"2024-09-17T16:35:12.092133+0000","last_became_active":"2024-09-17T16:34:23.706504+0000","last_became_peered":"2024-09-17T16:34:23.706504+0000","last_unstale":"2024-09-17T16:35:12.092133+0000","last_undegraded":"2024-09-17T16:35:12.092133+0000","last_fullsized":"2024-09-17T16:35:12.092133+0000","mapping_epoch":728,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'495","last_scrub_stamp":"2024-09-17T16:34:24.868013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:24.868013+0000","objects_scrubbed":244,"log_size":2169,"log_dups_size":0,"ondisk_log_size":2169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:53.327398+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125698048,"num_objects":1918,"num_object_clones":0,"num_object_copies":5754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1674,"num_write_kb":107136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,4],"acting":[3,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"756'2111","reported_seq":2941,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:12.448567+0000","last_change":"2024-09-17T16:35:01.018433+0000","last_active":"2024-09-17T16:35:12.448567+0000","last_peered":"2024-09-17T16:35:12.448567+0000","last_clean":"2024-09-17T16:35:12.448567+0000","last_became_active":"2024-09-17T16:34:30.685054+0000","last_became_peered":"2024-09-17T16:34:30.685054+0000","last_unstale":"2024-09-17T16:35:12.448567+0000","last_undegraded":"2024-09-17T16:35:12.448567+0000","last_fullsized":"2024-09-17T16:35:12.448567+0000","mapping_epoch":743,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":744,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:05.693761+0000","objects_scrubbed":0,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:31.012085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122650624,"num_objects":1872,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1872,"num_write_kb":119776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,5],"acting":[2,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"756'2260","reported_seq":2317,"reported_epoch":756,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:35:12.142913+0000","last_change":"2024-09-17T16:34:28.741847+0000","last_active":"2024-09-17T16:35:12.142913+0000","last_peered":"2024-09-17T16:35:12.142913+0000","last_clean":"2024-09-17T16:34:17.385513+0000","last_became_active":"2024-09-17T16:34:28.741847+0000","last_became_peered":"2024-09-17T16:34:28.741847+0000","last_unstale":"2024-09-17T16:35:12.142913+0000","last_undegraded":"2024-09-17T16:34:28.736676+0000","last_fullsized":"2024-09-17T16:34:28.736142+0000","mapping_epoch":741,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'523","last_scrub_stamp":"2024-09-17T16:34:25.467380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:34:05.693761+0000","last_clean_scrub_stamp":"2024-09-17T16:34:25.467380+0000","objects_scrubbed":0,"log_size":2260,"log_dups_size":0,"ondisk_log_size":2260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:35:30.257977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130973696,"num_objects":1998,"num_object_clones":0,"num_object_copies":5994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1998,"num_write_kb":127904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1998}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":283,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-09-17T16:35:04.690278+0000","last_change":"2024-09-17T16:33:59.364895+0000","last_active":"2024-09-17T16:35:04.690278+0000","last_peered":"2024-09-17T16:35:04.690278+0000","last_clean":"2024-09-17T16:35:04.690278+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:35:04.690278+0000","last_undegraded":"2024-09-17T16:35:04.690278+0000","last_fullsized":"2024-09-17T16:35:04.690278+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:35:13.261070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":4059791360,"num_objects":61949,"num_object_clones":0,"num_object_copies":185847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15245,"num_objects_misplaced":5652,"num_objects_unfound":0,"num_objects_dirty":61949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60492,"num_write_kb":3871392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1257,"num_bytes_recovered":82378752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11347492864,"data_stored":11347492864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66999,"ondisk_log_size":66999,"up":67,"acting":70,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285877,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1429872,"kb_used_data":921120,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92315280,"statfs":{"total":95995035648,"available":94530846720,"internally_reserved":0,"allocated":943226880,"data_stored":942853441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":520944068},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547504,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948256,"kb_used_data":429768,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92796896,"statfs":{"total":95995035648,"available":95024021504,"internally_reserved":0,"allocated":440082432,"data_stored":439785885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":530905536},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743742,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1531760,"kb_used_data":828640,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92213392,"statfs":{"total":95995035648,"available":94426513408,"internally_reserved":0,"allocated":848527360,"data_stored":848153941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":719976896},"hb_peers":[0,1,2,3,4,5,6,7,8,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005380,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1685176,"kb_used_data":1176424,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92059976,"statfs":{"total":95995035648,"available":94269415424,"internally_reserved":0,"allocated":1204658176,"data_stored":1204281205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1667824,"kb_used_data":1001568,"kb_used_omap":41,"kb_used_meta":666198,"kb_avail":92077328,"statfs":{"total":95995035648,"available":94287183872,"internally_reserved":0,"allocated":1025605632,"data_stored":1025232213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":682187059},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495949,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747064,"kb_used_data":238312,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92998088,"statfs":{"total":95995035648,"available":95230042112,"internally_reserved":0,"allocated":244031488,"data_stored":243654517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724880,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1167088,"kb_used_data":590304,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92578064,"statfs":{"total":95995035648,"available":94799937536,"internally_reserved":0,"allocated":604471296,"data_stored":604097877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590608838},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986520,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1833400,"kb_used_data":1292712,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91911752,"statfs":{"total":95995035648,"available":94117634048,"internally_reserved":0,"allocated":1323737088,"data_stored":1323360117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150043,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1805104,"kb_used_data":1106144,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":91940048,"statfs":{"total":95995035648,"available":94146609152,"internally_reserved":0,"allocated":1132691456,"data_stored":1132318037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":715717063},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2062320,"kb_used_data":1485536,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91682832,"statfs":{"total":95995035648,"available":93883219968,"internally_reserved":0,"allocated":1521188864,"data_stored":1520815445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575207,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1669424,"kb_used_data":1232864,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92075728,"statfs":{"total":95995035648,"available":94285545472,"internally_reserved":0,"allocated":1262452736,"data_stored":1262079317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738731,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1577776,"kb_used_data":864928,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92167376,"statfs":{"total":95995035648,"available":94379393024,"internally_reserved":0,"allocated":885686272,"data_stored":885312853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":886439936,"data_stored":886439936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1279983616,"data_stored":1279983616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1538260992,"data_stored":1538260992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1140588544,"data_stored":1140588544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1332019200,"data_stored":1332019200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":613351424,"data_stored":613351424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":243466240,"data_stored":243466240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1036451840,"data_stored":1036451840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1203240960,"data_stored":1203240960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":851509248,"data_stored":851509248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":265289728,"data_stored":265289728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":956891136,"data_stored":956891136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:35:16.336 INFO:tasks.thrashosds.thrasher:Setting 8.8 to [8, 10, 9, 0, 4, 11] 2024-09-17T16:35:16.336 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '8.8', '8', '10', '9', '0', '4', '11'] 2024-09-17T16:35:16.336 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pg-upmap-items 8.8 8 10 9 0 4 11 2024-09-17T16:35:16.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:16.380 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:16.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:16.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:16.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:16.509 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:16.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:16.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:16.555 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:16.602 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:35:16.602 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:35:16.611 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:16.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 67346 67330 64.7347 91.5 0.00398535 0.0154362 2024-09-17T16:35:16.636 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:16.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:16.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:16.761 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:16.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:16.844 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:16 smithi017 ceph-mon[27571]: from='client.18036 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:35:16.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:16.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:16.890 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:16.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:16 smithi173 ceph-mon[30530]: from='client.18036 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:35:16.991 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:17.017 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:17.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:17.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:16 smithi071 ceph-mon[30002]: from='client.18036 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:35:17.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:17.145 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:17.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:17.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:17.281 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:17.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:17.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:17.406 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:35:17.406 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:35:17.423 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:17.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:17.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:17.441 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:35:17.442 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:35:17.476 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:35:17.476 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:35:17.508 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:35:17.508 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:35:17.525 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:17.540 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:35:17.541 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:35:17.555 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:17.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:17.572 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:35:17.572 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:35:17.606 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:35:17.606 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:35:17.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 68886 68870 65.2121 96.25 0.00639285 0.0153301 2024-09-17T16:35:17.639 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:35:17.639 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:35:17.656 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:35:17.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:17.664 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:35:17.664 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:35:17.680 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:17.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:17.686 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:35:17.686 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:35:17.687 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:35:17.708 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:35:17.708 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:35:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:17 smithi017 ceph-mon[27571]: pgmap v1691: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 15754/190620 objects degraded (8.265%); 6041/190620 objects misplaced (3.169%) 2024-09-17T16:35:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1463684888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:17.783 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:17.810 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:17.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:17.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:17 smithi173 ceph-mon[30530]: pgmap v1691: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 15754/190620 objects degraded (8.265%); 6041/190620 objects misplaced (3.169%) 2024-09-17T16:35:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1463684888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:17.911 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:17.948 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:17.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:18.050 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:18.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:17 smithi071 ceph-mon[30002]: pgmap v1691: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 15754/190620 objects degraded (8.265%); 6041/190620 objects misplaced (3.169%) 2024-09-17T16:35:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1463684888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:18.083 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:18.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:18.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:18.185 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:18.209 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:18.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:18.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:18.341 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:18.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:18.443 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:18.473 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:18.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:18.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:18.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:18.598 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:18.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:18.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 70264 70248 65.5241 86.125 0.00234109 0.0152553 2024-09-17T16:35:18.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:18.732 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:18.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:18.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:18.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:18.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:18.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:18.990 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:18.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:19.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:19.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:19.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:19.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:19.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:19.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:19.345 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:19.371 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:19.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:19.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:19.496 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:19.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:19.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:19.598 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:19.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 71263 71247 65.4786 62.4375 0.00772589 0.0152602 2024-09-17T16:35:19.623 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:19.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:19.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:19.747 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:19.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:19.849 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:19.873 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:19.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:19 smithi173 ceph-mon[30530]: pgmap v1692: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 16580/197850 objects degraded (8.380%); 6242/197850 objects misplaced (3.155%) 2024-09-17T16:35:19.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:19.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:19 smithi017 ceph-mon[27571]: pgmap v1692: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 16580/197850 objects degraded (8.380%); 6242/197850 objects misplaced (3.155%) 2024-09-17T16:35:20.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:20.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:20.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:20.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:19 smithi071 ceph-mon[30002]: pgmap v1692: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 16580/197850 objects degraded (8.380%); 6242/197850 objects misplaced (3.155%) 2024-09-17T16:35:20.080 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:35:20.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:20.135 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:20.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:20.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:20.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:20.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:20.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:20.350 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:20.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:20.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:20.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:20.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:20.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:20.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:20.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:20.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 72764 72748 65.8892 93.8125 0.00440259 0.0151726 2024-09-17T16:35:20.636 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:20.670 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:20.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:20 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:20 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:20 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 16580/197850 objects degraded (8.380%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:20.772 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:20.796 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:20.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:20 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:20 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:20 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 16580/197850 objects degraded (8.380%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:20 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:20 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:20 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 16580/197850 objects degraded (8.380%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:20.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:20.923 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:20.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:20.956 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:35:20.956 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:35:21.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:21.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:21.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:21.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:21.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:21.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:21.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:21.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:21.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:21.311 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:21.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:21.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:21.437 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:21.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:21.484 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:21.537 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:21.538 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:21.568 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:21.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:21.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 74147 74131 66.1826 86.4375 0.0243425 0.015097 2024-09-17T16:35:21.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:21.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:21.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:21 smithi017 ceph-mon[27571]: pgmap v1693: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s; 16999/207108 objects degraded (8.208%); 6484/207108 objects misplaced (3.131%) 2024-09-17T16:35:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:21 smithi017 ceph-mon[27571]: osdmap e759: 12 total, 11 up, 12 in 2024-09-17T16:35:21.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:21.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:21.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:21 smithi173 ceph-mon[30530]: pgmap v1693: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s; 16999/207108 objects degraded (8.208%); 6484/207108 objects misplaced (3.131%) 2024-09-17T16:35:21.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:21 smithi173 ceph-mon[30530]: osdmap e759: 12 total, 11 up, 12 in 2024-09-17T16:35:21.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:21.959 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:21.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:22.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:22.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:21 smithi071 ceph-mon[30002]: pgmap v1693: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s; 16999/207108 objects degraded (8.208%); 6484/207108 objects misplaced (3.131%) 2024-09-17T16:35:22.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:21 smithi071 ceph-mon[30002]: osdmap e759: 12 total, 11 up, 12 in 2024-09-17T16:35:22.094 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:22.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:22.196 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:22.220 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:22.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:22.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:22.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:22.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:22.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:22.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:22.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:22.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:22.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:22.584 INFO:teuthology.orchestra.run.smithi017.stderr:set 8.8 pg_upmap_items mapping to [8->10,9->0,4->11] 2024-09-17T16:35:22.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:22.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:22.618 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 75504 75488 66.4449 84.8125 0.00918633 0.0150442 2024-09-17T16:35:22.648 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2286673085' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [8, 10, 9, 0, 4, 11]} : dispatch 2024-09-17T16:35:22.649 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:22 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [8, 10, 9, 0, 4, 11]} : dispatch 2024-09-17T16:35:22.649 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:22 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:22.649 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:22 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:22.702 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:22.726 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:22.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:22.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:22.855 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:22.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:22.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2286673085' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [8, 10, 9, 0, 4, 11]} : dispatch 2024-09-17T16:35:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:22 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [8, 10, 9, 0, 4, 11]} : dispatch 2024-09-17T16:35:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:22 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:22.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:22 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:22.957 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:22.958 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:35:22.982 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:22.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:23.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2286673085' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [8, 10, 9, 0, 4, 11]} : dispatch 2024-09-17T16:35:23.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:22 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [8, 10, 9, 0, 4, 11]} : dispatch 2024-09-17T16:35:23.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:22 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:23.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:22 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:23.083 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:23.108 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:23.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:23.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:23.178 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26665.0:27646 6.5 e545/527 6:a767d192:::benchmark_data_smithi173_102017_object27645:head v 545'1660, mlcod=545'1657)", 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:32.707942+0000", 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 590.45197634700003, 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31267298700000001, 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.179 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.707942+0000", 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.707942+0000", 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.707943+0000", 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6200000000000006e-07 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:32.708050+0000", 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010678100000000001 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019547+0000", 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.31149681200000001 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019554+0000", 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2459999999999997e-06 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019609+0000", 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5305e-05 2024-09-17T16:35:23.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.019626+0000", 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6667000000000002e-05 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.020580+0000", 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00095388199999999997 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:33.020615+0000", 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5332e-05 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.26665.0:29370 6.0 e547/527 6:0f906142:::benchmark_data_smithi173_102017_object29369:head v 547'1848, mlcod=547'1845)", 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:25:35.556226+0000", 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 587.60369282900001, 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27913635799999997, 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.184 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.556226+0000", 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.556226+0000", 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.556226+0000", 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.556251+0000", 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4870000000000001e-05 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.833972+0000", 2024-09-17T16:35:23.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27772044699999998 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.833981+0000", 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1989999999999995e-06 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.834052+0000", 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0758000000000003e-05 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.834077+0000", 2024-09-17T16:35:23.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4850000000000001e-05 2024-09-17T16:35:23.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.835298+0000", 2024-09-17T16:35:23.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001221048 2024-09-17T16:35:23.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:25:35.835362+0000", 2024-09-17T16:35:23.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4550999999999999e-05 2024-09-17T16:35:23.189 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.189 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.189 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.189 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:186285 6.f 6:ff66d09f:::benchmark_data_smithi173_102017_object93137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e621)", 2024-09-17T16:35:23.189 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:35:23.189 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 365.62332038599999, 2024-09-17T16:35:23.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53962611299999996, 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186285 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536599+0000", 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4900000000000005e-07 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536603+0000", 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0380000000000001e-06 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536614+0000", 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1097000000000001e-05 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536621+0000", 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5560000000000004e-06 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540416+0000", 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037954429999999999 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540537+0000", 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012064499999999999 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540685+0000", 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014845900000000001 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.544648+0000", 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039628129999999999 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.800489+0000", 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25584077599999999 2024-09-17T16:35:23.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076152+0000", 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27566357899999999 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076194+0000", 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1468999999999999e-05 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076224+0000", 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0488999999999999e-05 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:26018 7.8 7:151f2f24:::benchmark_data_smithi173_116396_object26017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 319.123788768, 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35239512200000001, 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26018 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036131+0000", 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.029e-06 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036175+0000", 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4579000000000001e-05 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036188+0000", 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.218e-05 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036203+0000", 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5494000000000001e-05 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036337+0000", 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013415599999999999 2024-09-17T16:35:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036422+0000", 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4427999999999999e-05 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036500+0000", 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8009000000000005e-05 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037194+0000", 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00069466700000000005 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037673+0000", 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047842099999999998 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388425+0000", 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35075205700000001 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388482+0000", 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7630999999999998e-05 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388525+0000", 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2471e-05 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.202 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:26018 7.8 e645/642 ondisk, result = 0)", 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 319.12251616600003, 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35113110600000003, 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037403+0000", 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8700000000000001e-07 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037406+0000", 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8710000000000001e-06 2024-09-17T16:35:23.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388344+0000", 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35093866000000001 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388350+0000", 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388413+0000", 2024-09-17T16:35:23.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2257999999999999e-05 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388420+0000", 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0450000000000003e-06 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388534+0000", 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011397599999999999 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:119650 7.8 7:17d95773:::benchmark_data_smithi173_116396_object24050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:58.385278+0000", 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 204.77464015499999, 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30032905500000001, 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 119650 2024-09-17T16:35:23.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385278+0000", 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385278+0000", 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385280+0000", 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6300000000000001e-06 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385286+0000", 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2049999999999999e-06 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385298+0000", 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1477e-05 2024-09-17T16:35:23.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385305+0000", 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9890000000000003e-06 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.388017+0000", 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027118770000000001 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.388140+0000", 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012355600000000001 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.388258+0000", 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118041 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.538443+0000", 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.150185233 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.546099+0000", 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007656014 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.685478+0000", 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.13937803200000001 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.685520+0000", 2024-09-17T16:35:23.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2787000000000002e-05 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.685608+0000", 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7213999999999995e-05 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27088.0:120213 7.3 e687/674 7:c59654a4:::benchmark_data_smithi173_116396_object24613:head v 687'6950, mlcod=686'6947)", 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.709364+0000", 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 203.45055464999999, 2024-09-17T16:35:23.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28146429899999997, 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709364+0000", 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709364+0000", 2024-09-17T16:35:23.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709365+0000", 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8700000000000004e-07 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709373+0000", 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7330000000000003e-06 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989058+0000", 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.279684973 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989065+0000", 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1300000000000003e-06 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989111+0000", 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6569000000000001e-05 2024-09-17T16:35:23.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989134+0000", 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2612e-05 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.990790+0000", 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001655807 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.990828+0000", 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8488000000000001e-05 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:120224 7.18 7:1c79604a:::benchmark_data_smithi173_116396_object24624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.730992+0000", 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 203.42892630399999, 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.274893622, 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.218 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 120224 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.730992+0000", 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.730992+0000", 2024-09-17T16:35:23.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.730994+0000", 2024-09-17T16:35:23.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8899999999999999e-06 2024-09-17T16:35:23.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731002+0000", 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.379e-06 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731015+0000", 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3892e-05 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731024+0000", 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9469999999999992e-06 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731315+0000", 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029021800000000002 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731451+0000", 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001359 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7,8", 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731569+0000", 2024-09-17T16:35:23.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118727 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.751547+0000", 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019978124 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.751573+0000", 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5265e-05 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989139+0000", 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23756648699999999 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.004357+0000", 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015217849 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.004409+0000", 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1891000000000002e-05 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.005886+0000", 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0014770530000000001 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(7.13 687/677 [PushReplyOp(7:cc214882:::benchmark_data_smithi173_116396_object24630:head)])", 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.748767+0000", 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 203.41115200900001, 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.306875487, 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.225 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748767+0000", 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748767+0000", 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748768+0000", 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7499999999999998e-07 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748773+0000", 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4550000000000003e-06 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.052878+0000", 2024-09-17T16:35:23.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30410503 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.052883+0000", 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2650000000000004e-06 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.053064+0000", 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018071799999999999 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.055642+0000", 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0025780439999999998 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:120235 7.e 7:752ed815:::benchmark_data_smithi173_116396_object24635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.749714+0000", 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 203.41020509500001, 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27885106300000001, 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.229 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 120235 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749714+0000", 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749714+0000", 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749715+0000", 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749721+0000", 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9909999999999999e-06 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749732+0000", 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0806e-05 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749738+0000", 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.037e-06 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749792+0000", 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3906999999999997e-05 2024-09-17T16:35:23.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749896+0000", 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103936 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749991+0000", 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4524000000000005e-05 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989005+0000", 2024-09-17T16:35:23.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23901425800000001 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989092+0000", 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7128999999999999e-05 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.028465+0000", 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.039372712999999997 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.028518+0000", 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3122999999999999e-05 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.028565+0000", 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6808000000000003e-05 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:128523 7.e 7:78ef58c9:::benchmark_data_smithi173_116396_object32923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e693)", 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:16.133568+0000", 2024-09-17T16:35:23.235 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 187.026350296, 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30194066800000002, 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 128523 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133568+0000", 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133568+0000", 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133570+0000", 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5069999999999999e-06 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133576+0000", 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4320000000000004e-06 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133593+0000", 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6489000000000001e-05 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133602+0000", 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9889999999999998e-06 2024-09-17T16:35:23.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.141144+0000", 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075426709999999999 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.141261+0000", 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011694299999999999 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:35:23.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.141386+0000", 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012499500000000001 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148436+0000", 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070501180000000002 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.150261+0000", 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018242969999999999 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435410+0000", 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28514896200000001 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435468+0000", 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8149999999999997e-05 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435509+0000", 2024-09-17T16:35:23.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1115000000000002e-05 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:128523 7.e e693/629 ondisk, result = 0)", 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 187.011073818, 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28666948399999997, 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:35:23.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3999999999999997e-07 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148847+0000", 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.575e-06 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435339+0000", 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28649269199999999 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435344+0000", 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5809999999999998e-06 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435399+0000", 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4598999999999997e-05 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435405+0000", 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.049e-06 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435514+0000", 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000109648 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:130635 7.8 7:14a1b993:::benchmark_data_smithi173_116396_object35035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 183.02713467300001, 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.329086555, 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.246 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 130635 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132785+0000", 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5239999999999999e-06 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132792+0000", 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4860000000000002e-06 2024-09-17T16:35:23.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132805+0000", 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2792999999999999e-05 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132812+0000", 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1899999999999998e-06 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.133889+0000", 2024-09-17T16:35:23.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001076881 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.134010+0000", 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120796 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.134124+0000", 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114875 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.136882+0000", 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027574539999999999 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.136893+0000", 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.154e-05 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461760+0000", 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32486675399999998 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461812+0000", 2024-09-17T16:35:23.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2231000000000002e-05 2024-09-17T16:35:23.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:23.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461870+0000", 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8031000000000001e-05 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:130635 7.8 e695/674 ondisk, result = 0)", 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:20.138684+0000", 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 183.02123479100001, 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32319844399999997, 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.254 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138684+0000", 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138684+0000", 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138685+0000", 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138690+0000", 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4659999999999998e-06 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461660+0000", 2024-09-17T16:35:23.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.322969489 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461673+0000", 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3807e-05 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461745+0000", 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1297000000000005e-05 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461754+0000", 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0270000000000001e-06 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461882+0000", 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012858100000000001 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178141 7.e 7:7ed26a83:::benchmark_data_smithi173_116396_object82541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-09-17T16:35:23.258 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.064783+0000", 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 107.09513528399999, 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29001914499999998, 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178141 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064783+0000", 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064783+0000", 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064784+0000", 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.128e-06 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064789+0000", 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.532e-06 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064800+0000", 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1229999999999999e-05 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064807+0000", 2024-09-17T16:35:23.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0979999999999996e-06 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064882+0000", 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4942000000000002e-05 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.065014+0000", 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013223299999999999 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:35:23.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.065138+0000", 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012379200000000001 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.068993+0000", 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038547109999999998 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.069508+0000", 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051509400000000001 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.354728+0000", 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28522005099999997 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.354766+0000", 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8275000000000002e-05 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.354802+0000", 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6059000000000002e-05 2024-09-17T16:35:23.264 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.265 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.265 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178146 7.8 7:14ce6dda:::benchmark_data_smithi173_116396_object82546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-09-17T16:35:23.265 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.072830+0000", 2024-09-17T16:35:23.265 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 107.08708872699999, 2024-09-17T16:35:23.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26665987099999999, 2024-09-17T16:35:23.265 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178146 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072830+0000", 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072830+0000", 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072831+0000", 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7999999999999995e-07 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072834+0000", 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9140000000000001e-06 2024-09-17T16:35:23.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072844+0000", 2024-09-17T16:35:23.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0999999999999993e-06 2024-09-17T16:35:23.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.072848+0000", 2024-09-17T16:35:23.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8250000000000004e-06 2024-09-17T16:35:23.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.073968+0000", 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001119639 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.074037+0000", 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9041999999999994e-05 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.074098+0000", 2024-09-17T16:35:23.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0717999999999997e-05 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.075223+0000", 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001124702 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.075762+0000", 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00053989299999999995 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339392+0000", 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26362966900000001 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339433+0000", 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1085999999999999e-05 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.339490+0000", 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6603e-05 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.271 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27088.0:178656 7.17 e719/707 7:ecd8c6ba:::benchmark_data_smithi173_116396_object83056:head v 719'8589, mlcod=719'8586)", 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 105.12734092300001, 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27508470800000001, 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032584+0000", 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0349999999999998e-06 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305420+0000", 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27283574900000002 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305426+0000", 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9540000000000001e-06 2024-09-17T16:35:23.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305471+0000", 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4836000000000002e-05 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305488+0000", 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7028e-05 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.307628+0000", 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021396359999999999 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.307662+0000", 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4724999999999997e-05 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:191190 7.e 7:7081439d:::benchmark_data_smithi173_116396_object95590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:35:23.276 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 88.706833685000007, 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.428509103, 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 191190 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453086+0000", 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3510000000000001e-06 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453092+0000", 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6010000000000002e-06 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453103+0000", 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.065e-05 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453110+0000", 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1160000000000001e-06 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453277+0000", 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016785300000000001 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453458+0000", 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018056699999999999 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453577+0000", 2024-09-17T16:35:23.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118678 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.458776+0000", 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051995009999999996 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.458855+0000", 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8597999999999995e-05 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:23.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881478+0000", 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42262325899999997 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881537+0000", 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9354999999999997e-05 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881594+0000", 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6573999999999997e-05 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.282 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:191190 7.e e724/629 ondisk, result = 0)", 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 88.701940635, 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42362284100000003, 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:35:23.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457979+0000", 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457985+0000", 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9460000000000003e-06 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881412+0000", 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42342706200000002 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881419+0000", 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.525e-06 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881467+0000", 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7667999999999998e-05 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881474+0000", 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9489999999999999e-06 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881601+0000", 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126797 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.286 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.16 741/739 [PushReplyOp(8:6b780127:::benchmark_data_smithi173_130566_object3478:head)])", 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.369245389, 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49350836100000001, 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790674+0000", 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4700000000000001e-07 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790677+0000", 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6720000000000002e-06 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283734+0000", 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49305611500000002 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283742+0000", 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7960000000000001e-06 2024-09-17T16:35:23.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283814+0000", 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1419000000000002e-05 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.284182+0000", 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00036781200000000002 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.290 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:23.291 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:23.291 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:35:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:23.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:23.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:23.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:23.418 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:23.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:23.520 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:23.542 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:23.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:23.592 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:35:23.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 76006 75990 65.9578 31.375 0.0805956 0.0150292 2024-09-17T16:35:23.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:23.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:23.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:23.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:23.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:23.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:23.890 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:23.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:23 smithi173 ceph-mon[30530]: pgmap v1695: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 17661/212319 objects degraded (8.318%); 6882/212319 objects misplaced (3.241%) 2024-09-17T16:35:23.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:23 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [8, 10, 9, 0, 4, 11]}]': finished 2024-09-17T16:35:23.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:23 smithi173 ceph-mon[30530]: osdmap e760: 12 total, 11 up, 12 in 2024-09-17T16:35:23.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3244476792' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:23.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:23 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:23.911 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:23.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:23 smithi017 ceph-mon[27571]: pgmap v1695: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 17661/212319 objects degraded (8.318%); 6882/212319 objects misplaced (3.241%) 2024-09-17T16:35:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:23 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [8, 10, 9, 0, 4, 11]}]': finished 2024-09-17T16:35:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:23 smithi017 ceph-mon[27571]: osdmap e760: 12 total, 11 up, 12 in 2024-09-17T16:35:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3244476792' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:23.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:23 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:24.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:24.037 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:24.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:24.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:23 smithi071 ceph-mon[30002]: pgmap v1695: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 17661/212319 objects degraded (8.318%); 6882/212319 objects misplaced (3.241%) 2024-09-17T16:35:24.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:23 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [8, 10, 9, 0, 4, 11]}]': finished 2024-09-17T16:35:24.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:23 smithi071 ceph-mon[30002]: osdmap e760: 12 total, 11 up, 12 in 2024-09-17T16:35:24.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3244476792' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:24.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:23 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:24.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:24.160 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:24.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:24.262 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:24.285 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:24.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:24.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:24.408 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:24.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:24.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:24.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:24.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:24.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 76037 76021 65.0807 1.9375 0.00197698 0.0150267 2024-09-17T16:35:24.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:24.658 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:24.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:24.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:24.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:24.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:24.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:24.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:24.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:24 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:35:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:24 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:24.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:24 smithi173 ceph-mon[30530]: osdmap e761: 12 total, 11 up, 12 in 2024-09-17T16:35:24.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:24.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:24 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:35:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:24 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:24 smithi017 ceph-mon[27571]: osdmap e761: 12 total, 11 up, 12 in 2024-09-17T16:35:25.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:25.040 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:25.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:24 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:35:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:24 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:24 smithi071 ceph-mon[30002]: osdmap e761: 12 total, 11 up, 12 in 2024-09-17T16:35:25.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:25.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:25.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:25.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:25.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:25.295 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:25.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:25.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:25.397 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:25.423 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:25.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:25.526 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:25.563 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:25.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:25.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 77480 77464 65.4199 90.1875 0.00894744 0.015279 2024-09-17T16:35:25.665 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:25.693 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:25.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:25.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:25.795 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:25.819 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:25.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:25.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:25 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:25.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:25 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:25.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:25 smithi173 ceph-mon[30530]: pgmap v1698: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s; 18433/220926 objects degraded (8.344%); 6882/220926 objects misplaced (3.115%) 2024-09-17T16:35:25.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:25 smithi173 ceph-mon[30530]: osdmap e762: 12 total, 11 up, 12 in 2024-09-17T16:35:25.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:25 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 18433/220926 objects degraded (8.344%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:25.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:25.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:25.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:25 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:25 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:25.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:25 smithi017 ceph-mon[27571]: pgmap v1698: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s; 18433/220926 objects degraded (8.344%); 6882/220926 objects misplaced (3.115%) 2024-09-17T16:35:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:25 smithi017 ceph-mon[27571]: osdmap e762: 12 total, 11 up, 12 in 2024-09-17T16:35:25.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:25 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 18433/220926 objects degraded (8.344%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:26.047 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:25 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:25 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:25 smithi071 ceph-mon[30002]: pgmap v1698: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s; 18433/220926 objects degraded (8.344%); 6882/220926 objects misplaced (3.115%) 2024-09-17T16:35:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:25 smithi071 ceph-mon[30002]: osdmap e762: 12 total, 11 up, 12 in 2024-09-17T16:35:26.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:25 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 18433/220926 objects degraded (8.344%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:26.071 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:26.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:26.125 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:35:26.172 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:26.195 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:26.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:26.235 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:26.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:26.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:26.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:26.421 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:26.443 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:26.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:26.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:26.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:26.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:26.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 15 78324 78309 65.2518 52.8125 0.00251132 0.0153205 2024-09-17T16:35:26.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:26.703 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:26.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:26.805 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:26.830 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:26.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:26.882 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:35:26.883 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:35:26.885 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:26.920 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:35:26.920 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:35:26.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:26.942 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:35:26.942 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:35:26.954 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:26.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:26.965 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:35:26.965 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:35:26.993 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:35:26.993 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:35:27.016 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:35:27.017 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:35:27.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:35:27.042 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:35:27.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:27.072 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:35:27.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:35:27.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:27.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:27.098 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:35:27.098 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:35:27.124 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:35:27.125 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:35:27.125 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:35:27.151 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:35:27.151 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:35:27.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:27.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:27.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:27.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:27.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:27.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:27.428 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:27.451 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:27.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:27.553 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:27.576 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:27.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:27.584 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:35:27.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 79241 79225 65.1464 57.25 0.0201192 0.0153418 2024-09-17T16:35:27.678 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:27.710 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:27.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:27.745 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:35:27.745 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:35:27.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:27.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:27.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:27 smithi173 ceph-mon[30530]: pgmap v1700: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s; 18970/228762 objects degraded (8.292%); 7328/228762 objects misplaced (3.203%) 2024-09-17T16:35:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:27 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:27 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:27.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2164172618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:27.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:27.959 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:27.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:27.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:27 smithi017 ceph-mon[27571]: pgmap v1700: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s; 18970/228762 objects degraded (8.292%); 7328/228762 objects misplaced (3.203%) 2024-09-17T16:35:27.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:27 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:27.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:27 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:27.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2164172618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:28.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:27 smithi071 ceph-mon[30002]: pgmap v1700: 27 pgs: 3 active+clean+remapped, 8 active+undersized+degraded, 16 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s; 18970/228762 objects degraded (8.292%); 7328/228762 objects misplaced (3.203%) 2024-09-17T16:35:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:27 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:27 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2164172618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:28.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:28.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:28.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:28.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:28.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:28.317 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:28.351 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:28.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:28.453 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:28.479 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:28.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:28.581 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:28.615 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:28.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:28.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 80520 80504 65.3384 79.9375 0.0171532 0.0153 2024-09-17T16:35:28.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:28.742 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:28.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:28.843 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:28.866 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:28.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:28 smithi173 ceph-mon[30530]: 8.a scrub starts 2024-09-17T16:35:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:28 smithi173 ceph-mon[30530]: osdmap e763: 12 total, 11 up, 12 in 2024-09-17T16:35:28.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:28 smithi017 ceph-mon[27571]: 8.a scrub starts 2024-09-17T16:35:28.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:28 smithi017 ceph-mon[27571]: osdmap e763: 12 total, 11 up, 12 in 2024-09-17T16:35:28.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:28.993 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:28.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:28.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:35:29.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:35:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:28 smithi071 ceph-mon[30002]: 8.a scrub starts 2024-09-17T16:35:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:28 smithi071 ceph-mon[30002]: osdmap e763: 12 total, 11 up, 12 in 2024-09-17T16:35:29.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:29.123 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:29.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:29.225 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:29.270 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:29.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:29.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:29.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:29.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:29.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:29.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:29.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:29.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:29.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 81725 81709 65.4662 75.3125 0.00644159 0.015268 2024-09-17T16:35:29.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:29.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:29.647 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:29.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:29.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:29.772 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:29.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:29.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:35:29.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:29 smithi173 ceph-mon[30530]: pgmap v1702: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 19303/231933 objects degraded (8.323%); 9525/231933 objects misplaced (4.107%); 1.8 MiB/s, 28 objects/s recovering 2024-09-17T16:35:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:29 smithi173 ceph-mon[30530]: 8.c scrub starts 2024-09-17T16:35:29.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:29 smithi173 ceph-mon[30530]: 8.a scrub ok 2024-09-17T16:35:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:29.896 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:29.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:29.933 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:29.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:29 smithi017 ceph-mon[27571]: pgmap v1702: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 19303/231933 objects degraded (8.323%); 9525/231933 objects misplaced (4.107%); 1.8 MiB/s, 28 objects/s recovering 2024-09-17T16:35:29.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:29 smithi017 ceph-mon[27571]: 8.c scrub starts 2024-09-17T16:35:29.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:29 smithi017 ceph-mon[27571]: 8.a scrub ok 2024-09-17T16:35:29.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:30.019 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:30.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:30.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:29 smithi071 ceph-mon[30002]: pgmap v1702: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 19303/231933 objects degraded (8.323%); 9525/231933 objects misplaced (4.107%); 1.8 MiB/s, 28 objects/s recovering 2024-09-17T16:35:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:29 smithi071 ceph-mon[30002]: 8.c scrub starts 2024-09-17T16:35:30.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:29 smithi071 ceph-mon[30002]: 8.a scrub ok 2024-09-17T16:35:30.101 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 7] 2024-09-17T16:35:30.101 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:35:30.101 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T16:35:30.101 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.023167 2024-09-17T16:35:30.101 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 0.023166919749457282 2024-09-17T16:35:30.121 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:30.146 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:30.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:30.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:30.309 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:30.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:30.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:30.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:30.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:30.539 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:30.574 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:30.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:30.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 83084 83068 65.7126 84.9375 0.00412691 0.0152125 2024-09-17T16:35:30.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:30.700 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:30.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:30.803 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:30.826 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:30.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:30.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:30 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:30 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:30 smithi173 ceph-mon[30530]: 8.c scrub ok 2024-09-17T16:35:30.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:30 smithi173 ceph-mon[30530]: osdmap e764: 12 total, 11 up, 12 in 2024-09-17T16:35:30.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:30 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 19303/231933 objects degraded (8.323%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:30.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:30 smithi173 ceph-mon[30530]: pgmap v1704: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s; 19597/239001 objects degraded (8.200%); 9707/239001 objects misplaced (4.061%); 1.6 MiB/s, 26 objects/s recovering 2024-09-17T16:35:30.928 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:30.952 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:30.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:30.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:30 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:30 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:30 smithi017 ceph-mon[27571]: 8.c scrub ok 2024-09-17T16:35:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:30 smithi017 ceph-mon[27571]: osdmap e764: 12 total, 11 up, 12 in 2024-09-17T16:35:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:30 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 19303/231933 objects degraded (8.323%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:30.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:30 smithi017 ceph-mon[27571]: pgmap v1704: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s; 19597/239001 objects degraded (8.200%); 9707/239001 objects misplaced (4.061%); 1.6 MiB/s, 26 objects/s recovering 2024-09-17T16:35:31.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:35:31.053 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:30 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:30 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:30 smithi071 ceph-mon[30002]: 8.c scrub ok 2024-09-17T16:35:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:30 smithi071 ceph-mon[30002]: osdmap e764: 12 total, 11 up, 12 in 2024-09-17T16:35:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:30 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 19303/231933 objects degraded (8.323%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:35:31.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:30 smithi071 ceph-mon[30002]: pgmap v1704: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s; 19597/239001 objects degraded (8.200%); 9707/239001 objects misplaced (4.061%); 1.6 MiB/s, 26 objects/s recovering 2024-09-17T16:35:31.092 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:31.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:31.194 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:31.228 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:31.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:31.330 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:31.355 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:31.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:31.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:31.482 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:31.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:31.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:31.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:31.611 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:31.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:31.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:35:31.618412+0000 min lat: 0.00112054 max lat: 15.0115 avg lat: 0.015173 2024-09-17T16:35:31.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:35:31.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 84350 84334 65.8802 79.125 0.00336247 0.015173 2024-09-17T16:35:31.713 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:31.736 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:31.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:31.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:31.861 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:31.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:31.963 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:31.987 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:31.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:31.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:32.089 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:32.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:32.119 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:32.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:32.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:31 smithi173 ceph-mon[30530]: 8.3 scrub starts 2024-09-17T16:35:32.220 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:32.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:31 smithi017 ceph-mon[27571]: 8.3 scrub starts 2024-09-17T16:35:32.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:32.248 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:32.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:31 smithi071 ceph-mon[30002]: 8.3 scrub starts 2024-09-17T16:35:32.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:32.359 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:32.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:32.433 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:32.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:32.528 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:32.534 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:32.572 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:32.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:32.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 85132 85116 65.6702 48.875 0.00758046 0.0152227 2024-09-17T16:35:32.674 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:32.709 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:32.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:32.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:32.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:32.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:32.891 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:35:32.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:32.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:32.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:33.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:33.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:33.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:32 smithi173 ceph-mon[30530]: 8.16 scrub starts 2024-09-17T16:35:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:32 smithi173 ceph-mon[30530]: 8.13 scrub starts 2024-09-17T16:35:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3432875104' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:35:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:32 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:35:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:32 smithi173 ceph-mon[30530]: 8.17 scrub starts 2024-09-17T16:35:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:32 smithi173 ceph-mon[30530]: 8.3 scrub ok 2024-09-17T16:35:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:32 smithi173 ceph-mon[30530]: pgmap v1705: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 879 op/s; 20149/243336 objects degraded (8.280%); 10082/243336 objects misplaced (4.143%); 1.3 MiB/s, 20 objects/s recovering 2024-09-17T16:35:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:32 smithi173 ceph-mon[30530]: 8.19 scrub starts 2024-09-17T16:35:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:32 smithi173 ceph-mon[30530]: 8.19 scrub ok 2024-09-17T16:35:33.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:33.219 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:33.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:33.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:32 smithi017 ceph-mon[27571]: 8.16 scrub starts 2024-09-17T16:35:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:32 smithi017 ceph-mon[27571]: 8.13 scrub starts 2024-09-17T16:35:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3432875104' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:35:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:32 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:35:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:32 smithi017 ceph-mon[27571]: 8.17 scrub starts 2024-09-17T16:35:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:32 smithi017 ceph-mon[27571]: 8.3 scrub ok 2024-09-17T16:35:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:32 smithi017 ceph-mon[27571]: pgmap v1705: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 879 op/s; 20149/243336 objects degraded (8.280%); 10082/243336 objects misplaced (4.143%); 1.3 MiB/s, 20 objects/s recovering 2024-09-17T16:35:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:32 smithi017 ceph-mon[27571]: 8.19 scrub starts 2024-09-17T16:35:33.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:32 smithi017 ceph-mon[27571]: 8.19 scrub ok 2024-09-17T16:35:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:32 smithi071 ceph-mon[30002]: 8.16 scrub starts 2024-09-17T16:35:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:32 smithi071 ceph-mon[30002]: 8.13 scrub starts 2024-09-17T16:35:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3432875104' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:35:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:32 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:35:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:32 smithi071 ceph-mon[30002]: 8.17 scrub starts 2024-09-17T16:35:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:32 smithi071 ceph-mon[30002]: 8.3 scrub ok 2024-09-17T16:35:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:32 smithi071 ceph-mon[30002]: pgmap v1705: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 879 op/s; 20149/243336 objects degraded (8.280%); 10082/243336 objects misplaced (4.143%); 1.3 MiB/s, 20 objects/s recovering 2024-09-17T16:35:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:32 smithi071 ceph-mon[30002]: 8.19 scrub starts 2024-09-17T16:35:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:32 smithi071 ceph-mon[30002]: 8.19 scrub ok 2024-09-17T16:35:33.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:33.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:33.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:33.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:33.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:33.486 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:33.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:33.588 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:33.619 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 86380 86364 65.8205 78 0.0054558 0.0151882 2024-09-17T16:35:33.621 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:33.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:33.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:33.723 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:33.746 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:33.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:33.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:33.877 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:33.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:33.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:33.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:34.004 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:34.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:34.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:34.107 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:34.138 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:34.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:33 smithi173 ceph-mon[30530]: 8.17 scrub ok 2024-09-17T16:35:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:33 smithi173 ceph-mon[30530]: 8.14 scrub starts 2024-09-17T16:35:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:33 smithi173 ceph-mon[30530]: 8.16 scrub ok 2024-09-17T16:35:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:33 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:33 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:35:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:33 smithi173 ceph-mon[30530]: 8.13 scrub ok 2024-09-17T16:35:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:33 smithi173 ceph-mon[30530]: osdmap e765: 12 total, 11 up, 12 in 2024-09-17T16:35:34.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:33 smithi173 ceph-mon[30530]: 8.15 scrub starts 2024-09-17T16:35:34.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:33 smithi017 ceph-mon[27571]: 8.17 scrub ok 2024-09-17T16:35:34.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:33 smithi017 ceph-mon[27571]: 8.14 scrub starts 2024-09-17T16:35:34.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:33 smithi017 ceph-mon[27571]: 8.16 scrub ok 2024-09-17T16:35:34.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:33 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:34.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:33 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:35:34.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:33 smithi017 ceph-mon[27571]: 8.13 scrub ok 2024-09-17T16:35:34.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:33 smithi017 ceph-mon[27571]: osdmap e765: 12 total, 11 up, 12 in 2024-09-17T16:35:34.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:33 smithi017 ceph-mon[27571]: 8.15 scrub starts 2024-09-17T16:35:34.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:34.276 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:34.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:33 smithi071 ceph-mon[30002]: 8.17 scrub ok 2024-09-17T16:35:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:33 smithi071 ceph-mon[30002]: 8.14 scrub starts 2024-09-17T16:35:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:33 smithi071 ceph-mon[30002]: 8.16 scrub ok 2024-09-17T16:35:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:33 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:33 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:35:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:33 smithi071 ceph-mon[30002]: 8.13 scrub ok 2024-09-17T16:35:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:33 smithi071 ceph-mon[30002]: osdmap e765: 12 total, 11 up, 12 in 2024-09-17T16:35:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:33 smithi071 ceph-mon[30002]: 8.15 scrub starts 2024-09-17T16:35:34.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:34.405 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:34.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:34.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:34.508 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:34.545 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:34.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:34.620 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 87692 87676 66.0153 82 0.00231535 0.015143 2024-09-17T16:35:34.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:34.670 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:34.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:34.782 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:34.805 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:34.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:34.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:34.907 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:34.941 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:34.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:34.970 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:35:34.971 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:35:35.004 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:35:35.004 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:35:35.004 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:35:35.004 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:35:35.005 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:35:35.005 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:35:35.005 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:35:35.005 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:35:35.021 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:35:35.021 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:35:35.043 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:35.056 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:35.056 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:35.056 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:34 smithi017 ceph-mon[27571]: 8.12 scrub starts 2024-09-17T16:35:35.056 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:34 smithi017 ceph-mon[27571]: osdmap e766: 12 total, 11 up, 12 in 2024-09-17T16:35:35.056 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:34 smithi017 ceph-mon[27571]: pgmap v1708: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 15 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 195 op/s; 20872/251274 objects degraded (8.306%); 9697/251274 objects misplaced (3.859%); 1.4 MiB/s, 22 objects/s recovering 2024-09-17T16:35:35.056 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:34 smithi017 ceph-mon[27571]: 8.10 scrub starts 2024-09-17T16:35:35.057 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:34 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 20149/243336 objects degraded (8.280%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:35.057 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1097557688' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:35.057 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:35:35.057 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:35:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:35.078 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:35.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:35.089 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:35:35.089 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:35:35.119 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:35:35.120 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:35:35.148 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:35:35.148 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:35:35.176 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:35:35.176 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:35:35.176 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:35:35.180 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:35.208 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:35:35.209 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:35:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:35.210 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:35.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:35.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:34 smithi071 ceph-mon[30002]: 8.12 scrub starts 2024-09-17T16:35:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:34 smithi071 ceph-mon[30002]: osdmap e766: 12 total, 11 up, 12 in 2024-09-17T16:35:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:34 smithi071 ceph-mon[30002]: pgmap v1708: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 15 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 195 op/s; 20872/251274 objects degraded (8.306%); 9697/251274 objects misplaced (3.859%); 1.4 MiB/s, 22 objects/s recovering 2024-09-17T16:35:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:34 smithi071 ceph-mon[30002]: 8.10 scrub starts 2024-09-17T16:35:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:34 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 20149/243336 objects degraded (8.280%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:35.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1097557688' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:35.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:35.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:35.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:34 smithi173 ceph-mon[30530]: 8.12 scrub starts 2024-09-17T16:35:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:34 smithi173 ceph-mon[30530]: osdmap e766: 12 total, 11 up, 12 in 2024-09-17T16:35:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:34 smithi173 ceph-mon[30530]: pgmap v1708: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 15 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 195 op/s; 20872/251274 objects degraded (8.306%); 9697/251274 objects misplaced (3.859%); 1.4 MiB/s, 22 objects/s recovering 2024-09-17T16:35:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:34 smithi173 ceph-mon[30530]: 8.10 scrub starts 2024-09-17T16:35:35.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:34 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 20149/243336 objects degraded (8.280%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:35.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1097557688' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:35.437 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:35.467 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:35.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:35.497 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:35.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:35.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:35.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:35.620 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 88815 88799 66.0649 70.1875 0.0188228 0.0151283 2024-09-17T16:35:35.693 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:35.718 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:35.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:35.820 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:35.850 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:35.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:35.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:35.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:35.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:35.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:36.079 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:36.102 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:36.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:36.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:36.205 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:36.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:36.229 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:36.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:36.331 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:36.362 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:36.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:36.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:36.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:36.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:35:36.559 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:35:36.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:36.620 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 89901 89885 66.0862 67.875 0.0233755 0.0151262 2024-09-17T16:35:36.625 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:36.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:36.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:36.749 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:36.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:36.851 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:36.873 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:36.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:36.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:36.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:36.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:36.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:37.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:35:37.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:37.122 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:37.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:37.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:37.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:37.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:37.351 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:37.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:37 smithi017 ceph-mon[27571]: pgmap v1709: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 15 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s wr, 4 op/s; 21569/260892 objects degraded (8.267%); 10261/260892 objects misplaced (3.933%) 2024-09-17T16:35:37.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:37 smithi017 ceph-mon[27571]: 8.11 scrub starts 2024-09-17T16:35:37.374 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:37.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:37 smithi173 ceph-mon[30530]: pgmap v1709: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 15 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s wr, 4 op/s; 21569/260892 objects degraded (8.267%); 10261/260892 objects misplaced (3.933%) 2024-09-17T16:35:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:37 smithi173 ceph-mon[30530]: 8.11 scrub starts 2024-09-17T16:35:37.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:37.508 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:37.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:37 smithi071 ceph-mon[30002]: pgmap v1709: 27 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 15 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s wr, 4 op/s; 21569/260892 objects degraded (8.267%); 10261/260892 objects misplaced (3.933%) 2024-09-17T16:35:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:37 smithi071 ceph-mon[30002]: 8.11 scrub starts 2024-09-17T16:35:37.611 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:37.620 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 91047 91031 66.1505 71.625 0.00399022 0.0151118 2024-09-17T16:35:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:37.638 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:37.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:37.739 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:37.768 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:37.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:37.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:37.871 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:37.895 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:37.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:37.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:38.021 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:38.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:38.109 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:38.117 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:35:38.122 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:38.146 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:38.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:38.248 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:38.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:38.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:38.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:38.416 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:38.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:38.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:38.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:38.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:38.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:35:38.620 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 15 92368 92353 66.3398 82.625 0.00654551 0.0150693 2024-09-17T16:35:38.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:38.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:38.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:38.773 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:38.793 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:35:38.793 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:35:38.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:35:38.794 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:35:38.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.794 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90607 6.c 6.63d0154c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-09-17T16:35:38.794 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:33.678865+0000", 2024-09-17T16:35:38.794 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 545.10762145700005, 2024-09-17T16:35:38.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.84340117199999998, 2024-09-17T16:35:38.795 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.795 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.795 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:35:38.795 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.796 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.796 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.796 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90607 2024-09-17T16:35:38.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.797 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678865+0000", 2024-09-17T16:35:38.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678865+0000", 2024-09-17T16:35:38.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678866+0000", 2024-09-17T16:35:38.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3600000000000003e-07 2024-09-17T16:35:38.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678922+0000", 2024-09-17T16:35:38.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6008999999999999e-05 2024-09-17T16:35:38.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678931+0000", 2024-09-17T16:35:38.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1060000000000006e-06 2024-09-17T16:35:38.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678935+0000", 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2250000000000002e-06 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678988+0000", 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2604e-05 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:35:38.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.678995+0000", 2024-09-17T16:35:38.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2230000000000001e-06 2024-09-17T16:35:38.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.522239+0000", 2024-09-17T16:35:38.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.84324350999999997 2024-09-17T16:35:38.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.522266+0000", 2024-09-17T16:35:38.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7759e-05 2024-09-17T16:35:38.804 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.804 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.804 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.805 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115483 6.b 6:d60f373c:::benchmark_data_smithi173_102017_object22335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.805 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.040530+0000", 2024-09-17T16:35:38.805 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.74595621399999, 2024-09-17T16:35:38.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.027404495000001, 2024-09-17T16:35:38.805 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.805 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.805 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.805 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115483 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040530+0000", 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040530+0000", 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040531+0000", 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5000000000000002e-07 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040535+0000", 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6260000000000001e-06 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040545+0000", 2024-09-17T16:35:38.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0001e-05 2024-09-17T16:35:38.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.040551+0000", 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2210000000000002e-06 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.044294+0000", 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037428040000000002 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.044410+0000", 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011616999999999999 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.044515+0000", 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104713 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.052307+0000", 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0077919399999999998 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.054367+0000", 2024-09-17T16:35:38.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020602149999999998 2024-09-17T16:35:38.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067817+0000", 2024-09-17T16:35:38.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.013449883 2024-09-17T16:35:38.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067881+0000", 2024-09-17T16:35:38.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4331000000000005e-05 2024-09-17T16:35:38.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067935+0000", 2024-09-17T16:35:38.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3940999999999997e-05 2024-09-17T16:35:38.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.814 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.815 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115488 6.b 6:d53f7bd0:::benchmark_data_smithi173_102017_object22340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.048310+0000", 2024-09-17T16:35:38.816 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.73817709500003, 2024-09-17T16:35:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.019735976, 2024-09-17T16:35:38.817 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.818 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115488 2024-09-17T16:35:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048310+0000", 2024-09-17T16:35:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048310+0000", 2024-09-17T16:35:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048311+0000", 2024-09-17T16:35:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.393e-06 2024-09-17T16:35:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048326+0000", 2024-09-17T16:35:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5081e-05 2024-09-17T16:35:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048336+0000", 2024-09-17T16:35:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7140000000000005e-06 2024-09-17T16:35:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048342+0000", 2024-09-17T16:35:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4110000000000001e-06 2024-09-17T16:35:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048387+0000", 2024-09-17T16:35:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5049000000000002e-05 2024-09-17T16:35:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048484+0000", 2024-09-17T16:35:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6362999999999999e-05 2024-09-17T16:35:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.048584+0000", 2024-09-17T16:35:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100679 2024-09-17T16:35:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.052398+0000", 2024-09-17T16:35:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038139070000000001 2024-09-17T16:35:38.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.054404+0000", 2024-09-17T16:35:38.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002006056 2024-09-17T16:35:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.067973+0000", 2024-09-17T16:35:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.013568521 2024-09-17T16:35:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.068008+0000", 2024-09-17T16:35:38.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5404000000000002e-05 2024-09-17T16:35:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.068046+0000", 2024-09-17T16:35:38.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7397999999999997e-05 2024-09-17T16:35:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115496 6.b 6:d757f72e:::benchmark_data_smithi173_102017_object22348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.060444+0000", 2024-09-17T16:35:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.72604229299998, 2024-09-17T16:35:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.011022887999999, 2024-09-17T16:35:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.835 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115496 2024-09-17T16:35:38.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060444+0000", 2024-09-17T16:35:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060444+0000", 2024-09-17T16:35:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060446+0000", 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6589999999999999e-06 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060451+0000", 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.429e-06 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060462+0000", 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0485999999999999e-05 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060469+0000", 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8759999999999999e-06 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060543+0000", 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4586999999999996e-05 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060653+0000", 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000109873 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.060753+0000", 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000100114 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.064621+0000", 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038672899999999998 2024-09-17T16:35:38.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.075055+0000", 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010434218 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071408+0000", 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.99635303 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071443+0000", 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4671000000000001e-05 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071467+0000", 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4655e-05 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115501 6.b 6:d2a1b698:::benchmark_data_smithi173_102017_object22353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.843 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.067982+0000", 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.71850435900001, 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 15.003590427000001, 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115501 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067982+0000", 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067982+0000", 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067984+0000", 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.067989+0000", 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5439999999999998e-06 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068001+0000", 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1437000000000001e-05 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068008+0000", 2024-09-17T16:35:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2629999999999997e-06 2024-09-17T16:35:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068575+0000", 2024-09-17T16:35:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00056690400000000002 2024-09-17T16:35:38.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068713+0000", 2024-09-17T16:35:38.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013839599999999999 2024-09-17T16:35:38.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.068972+0000", 2024-09-17T16:35:38.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025855700000000003 2024-09-17T16:35:38.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.075175+0000", 2024-09-17T16:35:38.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0062034830000000001 2024-09-17T16:35:38.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.080658+0000", 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054827449999999998 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071502+0000", 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.990844352 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071534+0000", 2024-09-17T16:35:38.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2150000000000002e-05 2024-09-17T16:35:38.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.071573+0000", 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8297999999999998e-05 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115511 6.b 6:dffba458:::benchmark_data_smithi173_102017_object22363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.086508+0000", 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.69997854500002, 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.993681085, 2024-09-17T16:35:38.853 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.854 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.854 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.854 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.854 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.854 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.854 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115511 2024-09-17T16:35:38.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.855 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086508+0000", 2024-09-17T16:35:38.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086508+0000", 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086510+0000", 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1509999999999998e-06 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086519+0000", 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1570000000000006e-06 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086535+0000", 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5914e-05 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086544+0000", 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6670000000000002e-06 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086616+0000", 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1768999999999997e-05 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086756+0000", 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013999699999999999 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.086877+0000", 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000121729 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092902+0000", 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060247219999999997 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092914+0000", 2024-09-17T16:35:38.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1815e-05 2024-09-17T16:35:38.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080089+0000", 2024-09-17T16:35:38.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.987174871000001 2024-09-17T16:35:38.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080145+0000", 2024-09-17T16:35:38.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5692000000000001e-05 2024-09-17T16:35:38.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080189+0000", 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4601000000000002e-05 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115517 6.b 6:dccfd4a2:::benchmark_data_smithi173_102017_object22369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.092979+0000", 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.69350752899999, 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.987374223, 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.862 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115517 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092979+0000", 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092979+0000", 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092980+0000", 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.82e-07 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092982+0000", 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5509999999999998e-06 2024-09-17T16:35:38.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092990+0000", 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8089999999999993e-06 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.092995+0000", 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2460000000000003e-06 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.093046+0000", 2024-09-17T16:35:38.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0569000000000003e-05 2024-09-17T16:35:38.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.093179+0000", 2024-09-17T16:35:38.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013340499999999999 2024-09-17T16:35:38.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.093304+0000", 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012481600000000001 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.097499+0000", 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041949659999999996 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.100969+0000", 2024-09-17T16:35:38.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034705370000000001 2024-09-17T16:35:38.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.868 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:38.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:38.870 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:38.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:38.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080244+0000", 2024-09-17T16:35:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.979274189 2024-09-17T16:35:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080301+0000", 2024-09-17T16:35:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7593000000000002e-05 2024-09-17T16:35:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.080353+0000", 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.206e-05 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115529 6.b 6:d3cef629:::benchmark_data_smithi173_102017_object22381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.122349+0000", 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.66413741399998, 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.959155643999999, 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.873 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115529 2024-09-17T16:35:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122349+0000", 2024-09-17T16:35:38.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122349+0000", 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122351+0000", 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9599999999999999e-06 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122358+0000", 2024-09-17T16:35:38.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1400000000000002e-06 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122371+0000", 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2791e-05 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122379+0000", 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8450000000000003e-06 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122456+0000", 2024-09-17T16:35:38.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7334000000000002e-05 2024-09-17T16:35:38.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122594+0000", 2024-09-17T16:35:38.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013791399999999999 2024-09-17T16:35:38.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.122736+0000", 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014201799999999999 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.126464+0000", 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037274119999999998 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.126918+0000", 2024-09-17T16:35:38.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00045421399999999998 2024-09-17T16:35:38.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.081386+0000", 2024-09-17T16:35:38.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.954468022 2024-09-17T16:35:38.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.081467+0000", 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0822999999999995e-05 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.081505+0000", 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8170999999999998e-05 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.880 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115584 6.b 6:d2471696:::benchmark_data_smithi173_102017_object22436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.222584+0000", 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.56390298999997, 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.861712117, 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115584 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.881 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222584+0000", 2024-09-17T16:35:38.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222584+0000", 2024-09-17T16:35:38.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222585+0000", 2024-09-17T16:35:38.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:35:38.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222590+0000", 2024-09-17T16:35:38.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:35:38.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222599+0000", 2024-09-17T16:35:38.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0569999999999999e-06 2024-09-17T16:35:38.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222606+0000", 2024-09-17T16:35:38.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0040000000000002e-06 2024-09-17T16:35:38.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222665+0000", 2024-09-17T16:35:38.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8761999999999997e-05 2024-09-17T16:35:38.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222796+0000", 2024-09-17T16:35:38.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131798 2024-09-17T16:35:38.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.222915+0000", 2024-09-17T16:35:38.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011872800000000001 2024-09-17T16:35:38.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226472+0000", 2024-09-17T16:35:38.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003557218 2024-09-17T16:35:38.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226919+0000", 2024-09-17T16:35:38.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00044620499999999997 2024-09-17T16:35:38.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084197+0000", 2024-09-17T16:35:38.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.857278443 2024-09-17T16:35:38.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.891 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084256+0000", 2024-09-17T16:35:38.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8581e-05 2024-09-17T16:35:38.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.892 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.892 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084296+0000", 2024-09-17T16:35:38.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0209000000000001e-05 2024-09-17T16:35:38.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.892 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.893 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115586 6.b 6:defa5317:::benchmark_data_smithi173_102017_object22438:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.894 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.226877+0000", 2024-09-17T16:35:38.894 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.559609583, 2024-09-17T16:35:38.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.857535156000001, 2024-09-17T16:35:38.894 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.894 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.894 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.895 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.895 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.895 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.895 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115586 2024-09-17T16:35:38.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.896 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226877+0000", 2024-09-17T16:35:38.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.898 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226877+0000", 2024-09-17T16:35:38.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226879+0000", 2024-09-17T16:35:38.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5409999999999999e-06 2024-09-17T16:35:38.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226885+0000", 2024-09-17T16:35:38.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.759e-06 2024-09-17T16:35:38.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226898+0000", 2024-09-17T16:35:38.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3103e-05 2024-09-17T16:35:38.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226907+0000", 2024-09-17T16:35:38.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2060000000000007e-06 2024-09-17T16:35:38.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.226955+0000", 2024-09-17T16:35:38.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7991999999999997e-05 2024-09-17T16:35:38.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.227075+0000", 2024-09-17T16:35:38.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012014 2024-09-17T16:35:38.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.227184+0000", 2024-09-17T16:35:38.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010894400000000001 2024-09-17T16:35:38.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.230180+0000", 2024-09-17T16:35:38.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002995841 2024-09-17T16:35:38.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.230383+0000", 2024-09-17T16:35:38.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000203043 2024-09-17T16:35:38.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084340+0000", 2024-09-17T16:35:38.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.853957060000001 2024-09-17T16:35:38.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084380+0000", 2024-09-17T16:35:38.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0500000000000002e-05 2024-09-17T16:35:38.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084412+0000", 2024-09-17T16:35:38.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2026999999999997e-05 2024-09-17T16:35:38.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.911 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.911 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.912 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115600 6.b 6:d47603e1:::benchmark_data_smithi173_102017_object22452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.912 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.270754+0000", 2024-09-17T16:35:38.912 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.51573250299998, 2024-09-17T16:35:38.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.813756322, 2024-09-17T16:35:38.912 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.912 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.913 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.913 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.913 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.913 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.913 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115600 2024-09-17T16:35:38.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.913 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270754+0000", 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270754+0000", 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270756+0000", 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7129999999999999e-06 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270763+0000", 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0990000000000001e-06 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270778+0000", 2024-09-17T16:35:38.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5316e-05 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270787+0000", 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6300000000000004e-06 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270852+0000", 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5432000000000006e-05 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.270992+0000", 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013987099999999999 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.271065+0000", 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3156000000000002e-05 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.274226+0000", 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031605219999999998 2024-09-17T16:35:38.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.274496+0000", 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000270492 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084443+0000", 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.809946601 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084482+0000", 2024-09-17T16:35:38.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8633000000000001e-05 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084510+0000", 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8856999999999998e-05 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115609 6.b 6:d5c07821:::benchmark_data_smithi173_102017_object22461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.328401+0000", 2024-09-17T16:35:38.919 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.458085305, 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.756214807999999, 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115609 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328401+0000", 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328401+0000", 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328403+0000", 2024-09-17T16:35:38.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9010000000000001e-06 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328409+0000", 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5509999999999999e-06 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328423+0000", 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4318000000000001e-05 2024-09-17T16:35:38.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328430+0000", 2024-09-17T16:35:38.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0210000000000002e-06 2024-09-17T16:35:38.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328486+0000", 2024-09-17T16:35:38.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6168999999999997e-05 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328615+0000", 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012836600000000001 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.328743+0000", 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012822699999999999 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.332302+0000", 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035596270000000001 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.332400+0000", 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7771999999999998e-05 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084544+0000", 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.752143610999999 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084585+0000", 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1352e-05 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.084616+0000", 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0892999999999997e-05 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115610 6.b 6:d3fb650b:::benchmark_data_smithi173_102017_object22462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.524259+0000", 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.26222736400001, 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.567226706, 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.927 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115610 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524259+0000", 2024-09-17T16:35:38.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524259+0000", 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524262+0000", 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6259999999999999e-06 2024-09-17T16:35:38.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524273+0000", 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1076e-05 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524294+0000", 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1111000000000002e-05 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524303+0000", 2024-09-17T16:35:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6920000000000008e-06 2024-09-17T16:35:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524368+0000", 2024-09-17T16:35:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5178999999999994e-05 2024-09-17T16:35:38.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524462+0000", 2024-09-17T16:35:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4438999999999995e-05 2024-09-17T16:35:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.524539+0000", 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6746e-05 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.527502+0000", 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029623760000000001 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.528273+0000", 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077131499999999996 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.091400+0000", 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.563127508999999 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.091448+0000", 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8084999999999998e-05 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.091486+0000", 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7552000000000002e-05 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115634 6.b 6:d0fb3e7a:::benchmark_data_smithi173_102017_object22486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.738056+0000", 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.04843048800001, 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.356902781000001, 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.935 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115634 2024-09-17T16:35:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738056+0000", 2024-09-17T16:35:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738056+0000", 2024-09-17T16:35:38.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738058+0000", 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5969999999999999e-06 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738064+0000", 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5740000000000001e-06 2024-09-17T16:35:38.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738080+0000", 2024-09-17T16:35:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5451000000000002e-05 2024-09-17T16:35:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738088+0000", 2024-09-17T16:35:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1000000000000004e-06 2024-09-17T16:35:38.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738152+0000", 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4474999999999995e-05 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738284+0000", 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131435 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.738414+0000", 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000130068 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.741749+0000", 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033351320000000002 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.741984+0000", 2024-09-17T16:35:38.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023470500000000001 2024-09-17T16:35:38.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094858+0000", 2024-09-17T16:35:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.352874224000001 2024-09-17T16:35:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094915+0000", 2024-09-17T16:35:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6904999999999998e-05 2024-09-17T16:35:38.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094959+0000", 2024-09-17T16:35:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4115e-05 2024-09-17T16:35:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.944 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115641 6.b 6:d9f11449:::benchmark_data_smithi173_102017_object22493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.022141987, 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.330730932, 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.945 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115641 2024-09-17T16:35:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:35:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:35:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764345+0000", 2024-09-17T16:35:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-17T16:35:38.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764348+0000", 2024-09-17T16:35:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0539999999999999e-06 2024-09-17T16:35:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764355+0000", 2024-09-17T16:35:38.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2089999999999999e-06 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764362+0000", 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.533e-06 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764422+0000", 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0105000000000001e-05 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764557+0000", 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013532 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.764684+0000", 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126761 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.768205+0000", 2024-09-17T16:35:38.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035209350000000002 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.768669+0000", 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046402699999999999 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.094999+0000", 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.326329940000001 2024-09-17T16:35:38.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095044+0000", 2024-09-17T16:35:38.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-17T16:35:38.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095076+0000", 2024-09-17T16:35:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1754999999999997e-05 2024-09-17T16:35:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:115643 6.b 6:d327719e:::benchmark_data_smithi173_102017_object22495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-09-17T16:35:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:14.775625+0000", 2024-09-17T16:35:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 504.01086149999998, 2024-09-17T16:35:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.319555412, 2024-09-17T16:35:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.954 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:35:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:35:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 115643 2024-09-17T16:35:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775625+0000", 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775625+0000", 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775627+0000", 2024-09-17T16:35:38.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.671e-06 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775634+0000", 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2370000000000003e-06 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775650+0000", 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6436999999999999e-05 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775660+0000", 2024-09-17T16:35:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4639999999999995e-06 2024-09-17T16:35:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775720+0000", 2024-09-17T16:35:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9599000000000003e-05 2024-09-17T16:35:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775841+0000", 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000121878 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1,6", 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.775957+0000", 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001158 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.779558+0000", 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036009620000000001 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:14.779623+0000", 2024-09-17T16:35:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4332999999999995e-05 2024-09-17T16:35:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095106+0000", 2024-09-17T16:35:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 14.315483789 2024-09-17T16:35:38.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095149+0000", 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2670999999999999e-05 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:29.095181+0000", 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1571e-05 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.961 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:95587 7.3 7:c3123e23:::benchmark_data_smithi173_116396_object95586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e671)", 2024-09-17T16:35:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:09.928298+0000", 2024-09-17T16:35:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 268.85818866400001, 2024-09-17T16:35:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.84987048899999995, 2024-09-17T16:35:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.962 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 95587 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928298+0000", 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928298+0000", 2024-09-17T16:35:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928300+0000", 2024-09-17T16:35:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8419999999999999e-06 2024-09-17T16:35:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928356+0000", 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6459999999999998e-05 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928372+0000", 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6030000000000001e-05 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928380+0000", 2024-09-17T16:35:38.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3010000000000001e-06 2024-09-17T16:35:38.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.966 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.966 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928498+0000", 2024-09-17T16:35:38.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118095 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928616+0000", 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011820999999999999 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,4", 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928769+0000", 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015275799999999999 2024-09-17T16:35:38.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.932423+0000", 2024-09-17T16:35:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036541949999999998 2024-09-17T16:35:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933233+0000", 2024-09-17T16:35:38.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00081015899999999997 2024-09-17T16:35:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.778093+0000", 2024-09-17T16:35:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.84486028400000002 2024-09-17T16:35:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.778136+0000", 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2456000000000001e-05 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.778168+0000", 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2699e-05 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.18 741/739 [PushReplyOp(8:1f8b3d36:::benchmark_data_smithi173_130566_object2490:head)])", 2024-09-17T16:35:38.970 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:27.809184+0000", 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 70.977303054000004, 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58530468800000002, 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.809184+0000", 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.809184+0000", 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.809184+0000", 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6599999999999995e-07 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.809189+0000", 2024-09-17T16:35:38.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7849999999999999e-06 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.394061+0000", 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58487161399999998 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.394069+0000", 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7870000000000002e-06 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.394126+0000", 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.702e-05 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.394488+0000", 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00036271600000000001 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:78176 8.8 8:12eb1bb9:::benchmark_data_smithi173_130566_object78175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:26.350904+0000", 2024-09-17T16:35:38.974 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 12.435582658, 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.216627872, 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 78176 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.350904+0000", 2024-09-17T16:35:38.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.350904+0000", 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.350914+0000", 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0487e-05 2024-09-17T16:35:38.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.350954+0000", 2024-09-17T16:35:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9406000000000002e-05 2024-09-17T16:35:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.350973+0000", 2024-09-17T16:35:38.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8774999999999999e-05 2024-09-17T16:35:38.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.350981+0000", 2024-09-17T16:35:38.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0509999999999997e-06 2024-09-17T16:35:38.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.352645+0000", 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00166395 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.352765+0000", 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012012500000000001 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:35:38.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.352936+0000", 2024-09-17T16:35:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017173700000000001 2024-09-17T16:35:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.355897+0000", 2024-09-17T16:35:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0029600120000000001 2024-09-17T16:35:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.355944+0000", 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7883000000000002e-05 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.565928+0000", 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.20998397999999999 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.566011+0000", 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2193000000000004e-05 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.567532+0000", 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015212730000000001 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:78178 8.8 8:13a4b4ba:::benchmark_data_smithi173_130566_object78177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:26.354228+0000", 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 12.432258586, 2024-09-17T16:35:38.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.221423971, 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 78178 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.354228+0000", 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.354228+0000", 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.354230+0000", 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7349999999999999e-06 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:35:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.354261+0000", 2024-09-17T16:35:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1083e-05 2024-09-17T16:35:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:35:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.354274+0000", 2024-09-17T16:35:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3555e-05 2024-09-17T16:35:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:35:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.354281+0000", 2024-09-17T16:35:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2360000000000001e-06 2024-09-17T16:35:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:35:38.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.354399+0000", 2024-09-17T16:35:38.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118515 2024-09-17T16:35:38.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.354543+0000", 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000143802 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.354702+0000", 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015928599999999999 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.356124+0000", 2024-09-17T16:35:38.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00142143 2024-09-17T16:35:38.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.399825+0000", 2024-09-17T16:35:38.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.043701006000000001 2024-09-17T16:35:38.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:35:38.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.573974+0000", 2024-09-17T16:35:38.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.17414882400000001 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.574088+0000", 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114364 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:26.575652+0000", 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015641349999999999 2024-09-17T16:35:38.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.991 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:35:38.991 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:35:38.991 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:35:38.992 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:39.015 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:39.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:39.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:39.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:39.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:39.145 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.3 primary-affinity to 0.0231669 (815182) 2024-09-17T16:35:39.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:39.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:39.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:39.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:39.365 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:39.386 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:39.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:39 smithi173 ceph-mon[30530]: pgmap v1710: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22093/265770 objects degraded (8.313%); 9934/265770 objects misplaced (3.738%) 2024-09-17T16:35:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1080092343' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.023166919749457282} : dispatch 2024-09-17T16:35:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:39 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.023166919749457282} : dispatch 2024-09-17T16:35:39.488 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:39 smithi017 ceph-mon[27571]: pgmap v1710: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22093/265770 objects degraded (8.313%); 9934/265770 objects misplaced (3.738%) 2024-09-17T16:35:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1080092343' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.023166919749457282} : dispatch 2024-09-17T16:35:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:39 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.023166919749457282} : dispatch 2024-09-17T16:35:39.513 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:39.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:39 smithi071 ceph-mon[30002]: pgmap v1710: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 22093/265770 objects degraded (8.313%); 9934/265770 objects misplaced (3.738%) 2024-09-17T16:35:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1080092343' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.023166919749457282} : dispatch 2024-09-17T16:35:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:39 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 0.023166919749457282} : dispatch 2024-09-17T16:35:39.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:39.620 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 93004 92988 66.0369 39.6875 0.00255419 0.0150927 2024-09-17T16:35:39.636 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:39.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:39.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:39.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:39.766 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:39.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:39.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:39.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:39.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:40.011 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:40.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:40.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:40.154 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:40.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:40.256 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:40.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:40.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:40.393 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:40 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.023166919749457282}]': finished 2024-09-17T16:35:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:40 smithi173 ceph-mon[30530]: osdmap e767: 12 total, 11 up, 12 in 2024-09-17T16:35:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:40 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 22093/265770 objects degraded (8.313%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:40 smithi173 ceph-mon[30530]: osdmap e768: 12 total, 11 up, 12 in 2024-09-17T16:35:40.417 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:40.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:40 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.023166919749457282}]': finished 2024-09-17T16:35:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:40 smithi017 ceph-mon[27571]: osdmap e767: 12 total, 11 up, 12 in 2024-09-17T16:35:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:40 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 22093/265770 objects degraded (8.313%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:40 smithi017 ceph-mon[27571]: osdmap e768: 12 total, 11 up, 12 in 2024-09-17T16:35:40.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:40.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:40.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:40.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:40 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.023166919749457282}]': finished 2024-09-17T16:35:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:40 smithi071 ceph-mon[30002]: osdmap e767: 12 total, 11 up, 12 in 2024-09-17T16:35:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:40 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 22093/265770 objects degraded (8.313%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:40.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:40 smithi071 ceph-mon[30002]: osdmap e768: 12 total, 11 up, 12 in 2024-09-17T16:35:40.620 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 93970 93954 65.9732 60.375 0.0173211 0.0151532 2024-09-17T16:35:40.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:40.679 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:40.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:40.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:40.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:40.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:40.808 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:40.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:40.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:40.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:40.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:41.036 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:41.058 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:41.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:41.149 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:35:41.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:41.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:41.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:41.289 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:41.293 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:35:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:41.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:41.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:41.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:41.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:41 smithi017 ceph-mon[27571]: pgmap v1713: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 363 op/s; 22429/273915 objects degraded (8.188%); 10111/273915 objects misplaced (3.691%) 2024-09-17T16:35:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2205174247' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:41.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:41 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:41.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:41.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:41 smithi071 ceph-mon[30002]: pgmap v1713: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 363 op/s; 22429/273915 objects degraded (8.188%); 10111/273915 objects misplaced (3.691%) 2024-09-17T16:35:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2205174247' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:41 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:41.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:41.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:41.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:41.577 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:35:41.620 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 12 94972 94960 65.9387 62.875 0.00711055 0.0151593 2024-09-17T16:35:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:41 smithi173 ceph-mon[30530]: pgmap v1713: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 363 op/s; 22429/273915 objects degraded (8.188%); 10111/273915 objects misplaced (3.691%) 2024-09-17T16:35:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2205174247' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:41 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:41.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:41.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:41.794 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:41.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:41.816 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:41.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:41.918 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:41.941 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:41.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:42.041 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:42.064 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:42.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:42.166 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:42.187 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:42.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:42 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:42 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:42 smithi017 ceph-mon[27571]: osdmap e769: 12 total, 11 up, 12 in 2024-09-17T16:35:42.289 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:42.310 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:42.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:42.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:42.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:42.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:42.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:42 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:42.466 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:42 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:42.466 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:42 smithi071 ceph-mon[30002]: osdmap e769: 12 total, 11 up, 12 in 2024-09-17T16:35:42.507 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:35:42.507 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:35:42.507 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:35:42.507 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:35:42.511 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:42.538 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:42.562 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:42.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:42.620 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.7806 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 94972 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 65.3856 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 35.6312 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 122.5 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1046 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 570.108 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 1960 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 0 2024-09-17T16:35:42.621 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.015208 2024-09-17T16:35:42.622 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.190638 2024-09-17T16:35:42.622 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 15.0115 2024-09-17T16:35:42.622 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00112054 2024-09-17T16:35:42.629 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T16:35:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:42 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:42 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:42.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:42 smithi173 ceph-mon[30530]: osdmap e769: 12 total, 11 up, 12 in 2024-09-17T16:35:42.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:42.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:42.694 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:42.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:42.829 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:42.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:42.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:42.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:42.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:43.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:43.092 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:43.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:43.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:35:43.177 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:35:43.195 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:43.206 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:35:43.206 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:35:43.217 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:43.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:43.237 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:35:43.237 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:35:43.269 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:35:43.269 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:35:43.269 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:43 smithi017 ceph-mon[27571]: pgmap v1715: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 501 op/s; 22927/277695 objects degraded (8.256%); 10407/277695 objects misplaced (3.748%); 960 KiB/s, 14 objects/s recovering 2024-09-17T16:35:43.269 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3459709621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:43.300 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:35:43.300 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:35:43.319 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:43.330 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:35:43.331 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:35:43.344 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:35:43.349 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:43.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:43.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:43.358 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:35:43.359 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:35:43.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:35:43.390 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:35:43.415 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:35:43.416 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:35:43.442 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:35:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:35:43.442 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:35:43.450 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:43.472 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:35:43.473 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:35:43.478 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:43.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:43 smithi071 ceph-mon[30002]: pgmap v1715: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 501 op/s; 22927/277695 objects degraded (8.256%); 10407/277695 objects misplaced (3.748%); 960 KiB/s, 14 objects/s recovering 2024-09-17T16:35:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3459709621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:43.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:43.604 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:43.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:43.614 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:35:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:43 smithi173 ceph-mon[30530]: pgmap v1715: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 501 op/s; 22927/277695 objects degraded (8.256%); 10407/277695 objects misplaced (3.748%); 960 KiB/s, 14 objects/s recovering 2024-09-17T16:35:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3459709621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:43.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:43.731 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:43.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:43.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:43.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:43.861 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:43.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:43.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:43.963 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:43.993 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:43.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:44.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:44.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:44.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:44.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:44.173 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:35:44.173 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:35:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:35:44.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:35:44.222 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:44.246 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:44.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:44.348 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:44.373 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:44.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:44.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:35:44.475 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:44.498 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:44.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:44.600 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:44.695 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:44.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:44.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:44.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:44.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:44.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:44.830 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:44.862 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:35:44.921 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:44.945 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:44.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:45.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:45.074 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:45.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:45.138 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:35:45.176 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:45.197 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:45.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:45.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:45 smithi017 ceph-mon[27571]: pgmap v1716: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 57 MiB/s wr, 938 op/s; 23382/281919 objects degraded (8.294%); 10103/281919 objects misplaced (3.584%); 3.2 MiB/s, 50 objects/s recovering 2024-09-17T16:35:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:45 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 23382/281919 objects degraded (8.294%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:45.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:45 smithi017 ceph-mon[27571]: osdmap e770: 12 total, 11 up, 12 in 2024-09-17T16:35:45.299 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:45.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:45.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:45.422 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:45 smithi071 ceph-mon[30002]: pgmap v1716: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 57 MiB/s wr, 938 op/s; 23382/281919 objects degraded (8.294%); 10103/281919 objects misplaced (3.584%); 3.2 MiB/s, 50 objects/s recovering 2024-09-17T16:35:45.423 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:45 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 23382/281919 objects degraded (8.294%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:45.423 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:45 smithi071 ceph-mon[30002]: osdmap e770: 12 total, 11 up, 12 in 2024-09-17T16:35:45.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:45.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:45.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:45.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:45.577 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:45.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:45 smithi173 ceph-mon[30530]: pgmap v1716: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 57 MiB/s wr, 938 op/s; 23382/281919 objects degraded (8.294%); 10103/281919 objects misplaced (3.584%); 3.2 MiB/s, 50 objects/s recovering 2024-09-17T16:35:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:45 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 23382/281919 objects degraded (8.294%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:45 smithi173 ceph-mon[30530]: osdmap e770: 12 total, 11 up, 12 in 2024-09-17T16:35:45.663 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:35:45.663 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:35:45.679 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:45.709 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:45.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:45.811 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:45.841 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:45.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:45.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:45.943 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:45.968 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:45.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:45.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:46.070 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:46.107 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:46.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:46.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:46.177 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:35:46.202 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:35:46.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:46.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:46.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:46.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:46.370 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:46.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:46.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:46.469 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2306601515' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:46.469 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:46 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:46.472 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2306601515' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:46 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:46.500 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:46.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:46.509 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:35:46.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:46.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:46.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2306601515' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:46 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:35:46.727 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:46.756 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:46.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:46.849 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:35:46.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:46.885 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:46.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:46.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:46.987 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:35:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:47.013 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:47.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:35:47.014 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 7] 2024-09-17T16:35:47.014 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:35:47.014 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 7] 2024-09-17T16:35:47.014 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-17T16:35:47.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.5 2024-09-17T16:35:47.115 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:47.142 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:47.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:47.245 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:47.246 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:47 smithi071 systemd[1]: Stopping Ceph osd.5 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:35:47.247 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:47 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:47.247 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:47 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:47.247 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:47 smithi071 ceph-mon[30002]: pgmap v1718: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s rd, 43 MiB/s wr, 1k op/s; 23042/278862 objects degraded (8.263%); 9902/278862 objects misplaced (3.551%); 3.0 MiB/s, 47 objects/s recovering 2024-09-17T16:35:47.247 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:47 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:47.247 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:47 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:35:47.247 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:47 smithi071 ceph-mon[30002]: osdmap e771: 12 total, 11 up, 12 in 2024-09-17T16:35:47.275 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:47.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:47.360 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:35:47.360 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:35:47.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:47.403 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:47.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:47 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:47 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:47 smithi017 ceph-mon[27571]: pgmap v1718: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s rd, 43 MiB/s wr, 1k op/s; 23042/278862 objects degraded (8.263%); 9902/278862 objects misplaced (3.551%); 3.0 MiB/s, 47 objects/s recovering 2024-09-17T16:35:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:47 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:47 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:35:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:47 smithi017 ceph-mon[27571]: osdmap e771: 12 total, 11 up, 12 in 2024-09-17T16:35:47.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:47.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:47.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:47.626 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:47 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:47 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:47 smithi173 ceph-mon[30530]: pgmap v1718: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 160 B/s rd, 43 MiB/s wr, 1k op/s; 23042/278862 objects degraded (8.263%); 9902/278862 objects misplaced (3.551%); 3.0 MiB/s, 47 objects/s recovering 2024-09-17T16:35:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:47 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:35:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:47 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:35:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:47 smithi173 ceph-mon[30530]: osdmap e771: 12 total, 11 up, 12 in 2024-09-17T16:35:47.655 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:47.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:47.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:47.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:47.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:47.881 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:47.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:47.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:48.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:48.028 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:48.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:48.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:48.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:48.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:48.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:48.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:48.283 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:48.384 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:48.413 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:48.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:48 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:48 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:48 smithi017 ceph-mon[27571]: osdmap e772: 12 total, 11 up, 12 in 2024-09-17T16:35:48.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:48.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:48.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:48 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:48 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:48 smithi071 ceph-mon[30002]: osdmap e772: 12 total, 11 up, 12 in 2024-09-17T16:35:48.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:48.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:48 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:48 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:48 smithi173 ceph-mon[30530]: osdmap e772: 12 total, 11 up, 12 in 2024-09-17T16:35:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:48.665 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:48.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:48.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:48.790 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:48.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:48.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:48.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:48.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:49.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:49.042 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:49.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:49.144 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:49.169 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:49.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:49.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:49.297 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:49.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:49.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:49.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:49.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:49.424 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:49.445 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:49 smithi071 ceph-mon[30002]: pgmap v1721: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 25 MiB/s wr, 960 op/s; 22530/274677 objects degraded (8.202%); 9391/274677 objects misplaced (3.419%); 8.5 MiB/s, 135 objects/s recovering 2024-09-17T16:35:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:49 smithi017 ceph-mon[27571]: pgmap v1721: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 25 MiB/s wr, 960 op/s; 22530/274677 objects degraded (8.202%); 9391/274677 objects misplaced (3.419%); 8.5 MiB/s, 135 objects/s recovering 2024-09-17T16:35:49.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:49.555 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:49.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:49.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:49 smithi173 ceph-mon[30530]: pgmap v1721: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 25 MiB/s wr, 960 op/s; 22530/274677 objects degraded (8.202%); 9391/274677 objects misplaced (3.419%); 8.5 MiB/s, 135 objects/s recovering 2024-09-17T16:35:49.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:49.682 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:49.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:49.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:49.784 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:49.813 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:49.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:49.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:49.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:49.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:50.038 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:50.063 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:50.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:50.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:50.189 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:50.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:50.291 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:50.319 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:50.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:50.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:50.446 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:50.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:50.455 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:50 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 22530/274677 objects degraded (8.202%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:50.488 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:35:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:50 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 22530/274677 objects degraded (8.202%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:50.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:50.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:50.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:50.608 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:50 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 22530/274677 objects degraded (8.202%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:50.674 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:50.702 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:50.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:50.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:50.813 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[39093]: 2024-09-17T16:35:50.634+0000 7f1ab3a5f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:35:50.814 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[39093]: 2024-09-17T16:35:50.634+0000 7f1ab3a5f640 -1 osd.5 772 *** Got signal Terminated *** 2024-09-17T16:35:50.814 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[39093]: 2024-09-17T16:35:50.634+0000 7f1ab3a5f640 -1 osd.5 772 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:35:50.828 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:35:50.830 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:50.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:50.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:50.957 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:50.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:51.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:51.083 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:51.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:51.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:51.194 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:35:51.194 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:35:51.215 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:51.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:51.221 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:35:51.221 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:35:51.241 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:35:51.241 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:35:51.261 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:35:51.262 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:35:51.284 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:35:51.285 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:35:51.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:51.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:51.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:51.353 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:35:51.353 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:35:51.441 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:51.462 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:51.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:51 smithi017 ceph-mon[27571]: pgmap v1722: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s wr, 1.12k op/s; 22237/266796 objects degraded (8.335%); 9215/266796 objects misplaced (3.454%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T16:35:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:51 smithi017 ceph-mon[27571]: osd.5 marked itself down and dead 2024-09-17T16:35:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:51 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:51 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2823647783' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:51 smithi071 ceph-mon[30002]: pgmap v1722: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s wr, 1.12k op/s; 22237/266796 objects degraded (8.335%); 9215/266796 objects misplaced (3.454%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T16:35:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:51 smithi071 ceph-mon[30002]: osd.5 marked itself down and dead 2024-09-17T16:35:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:51 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:51 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2823647783' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:51.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:51.590 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:51.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:51 smithi173 ceph-mon[30530]: pgmap v1722: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 7 active+undersized+degraded, 16 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s wr, 1.12k op/s; 22237/266796 objects degraded (8.335%); 9215/266796 objects misplaced (3.454%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T16:35:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:51 smithi173 ceph-mon[30530]: osd.5 marked itself down and dead 2024-09-17T16:35:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:51 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:51 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2823647783' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:51.657 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:35:51.682 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:35:51.683 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:35:51.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:51.704 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:35:51.705 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:35:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:51.714 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:51.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:51.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:35:51.728 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:35:51.750 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:35:51.751 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:35:51.751 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:35:51.776 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:35:51.777 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:35:51.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:51.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:51.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:51.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:51.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:51.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:51.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:52.034 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:35:52.062 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:52.087 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:52.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:52.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:52.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:52.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:52.215 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:52 smithi071 ceph-mon[30002]: Health check update: 2 osds down (OSD_DOWN) 2024-09-17T16:35:52.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:52 smithi071 ceph-mon[30002]: osdmap e773: 12 total, 10 up, 12 in 2024-09-17T16:35:52.314 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:52 smithi071 podman[138735]: 2024-09-17 16:35:52.090135511 +0000 UTC m=+2.744577255 container died 45ac68a2cd9070e3ab7b91e5b6772f9b1d93000751cd51847ab555965e3c9c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:35:52.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:52.349 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:52.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:52.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:52.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:52.473 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:52.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:52 smithi017 ceph-mon[27571]: Health check update: 2 osds down (OSD_DOWN) 2024-09-17T16:35:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:52 smithi017 ceph-mon[27571]: osdmap e773: 12 total, 10 up, 12 in 2024-09-17T16:35:52.542 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:35:52.542 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:35:52.551 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:35:52.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:52.598 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:52.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:52 smithi173 ceph-mon[30530]: Health check update: 2 osds down (OSD_DOWN) 2024-09-17T16:35:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:52 smithi173 ceph-mon[30530]: osdmap e773: 12 total, 10 up, 12 in 2024-09-17T16:35:52.695 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:52 smithi071 podman[138735]: 2024-09-17 16:35:52.475082218 +0000 UTC m=+3.129523961 container cleanup 45ac68a2cd9070e3ab7b91e5b6772f9b1d93000751cd51847ab555965e3c9c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-17T16:35:52.695 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:52 smithi071 bash[138735]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5 2024-09-17T16:35:52.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:52.723 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:52.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:52.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:52.759 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:35:52.816 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:52.825 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:52.847 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:52.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:52.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:52.974 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:52.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:53.001 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:52 smithi071 podman[138935]: 2024-09-17 16:35:52.693328028 +0000 UTC m=+1.007134830 container remove 45ac68a2cd9070e3ab7b91e5b6772f9b1d93000751cd51847ab555965e3c9c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T16:35:53.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:53.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:53.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:53.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:53.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:53.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:53.226 INFO:teuthology.orchestra.run.smithi071.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-17T16:35:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:53 smithi071 ceph-mon[30002]: pgmap v1724: 27 pgs: 1 peering, 1 stale+active+clean+remapped, 1 stale+active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 6 active+undersized+degraded, 15 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 19136/262731 objects degraded (7.283%); 8843/262731 objects misplaced (3.366%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T16:35:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:53 smithi071 ceph-mon[30002]: osdmap e774: 12 total, 10 up, 12 in 2024-09-17T16:35:53.323 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:53.349 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:53.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:53.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:53.451 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:53.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:53.473 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:53.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:53 smithi017 ceph-mon[27571]: pgmap v1724: 27 pgs: 1 peering, 1 stale+active+clean+remapped, 1 stale+active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 6 active+undersized+degraded, 15 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 19136/262731 objects degraded (7.283%); 8843/262731 objects misplaced (3.366%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T16:35:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:53 smithi017 ceph-mon[27571]: osdmap e774: 12 total, 10 up, 12 in 2024-09-17T16:35:53.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:53.598 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:53.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:53 smithi173 ceph-mon[30530]: pgmap v1724: 27 pgs: 1 peering, 1 stale+active+clean+remapped, 1 stale+active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 6 active+undersized+degraded, 15 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 19136/262731 objects degraded (7.283%); 8843/262731 objects misplaced (3.366%); 5.3 MiB/s, 85 objects/s recovering 2024-09-17T16:35:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:53 smithi173 ceph-mon[30530]: osdmap e774: 12 total, 10 up, 12 in 2024-09-17T16:35:53.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:53.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:53.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:53.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:35:53.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:35:53.847 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:53.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:53.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:53.973 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:53.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:54.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:54 smithi071 podman[139219]: 2024-09-17 16:35:53.92663187 +0000 UTC m=+0.015735065 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:35:54.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:54 smithi071 podman[139219]: 2024-09-17 16:35:54.034450784 +0000 UTC m=+0.123554018 container create f7e4358b41a67efd3f2f1923ae644bedd48287ee0b7bf1411781cc458c033fba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:35:54.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:54.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:54.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:54.115 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:35:54.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:54.203 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:35:54.221 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:54.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:54.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:54.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:54.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:54.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:54.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:54.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:54.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:54.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:54.472 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:54 smithi173 ceph-mon[30530]: osdmap e775: 12 total, 10 up, 12 in 2024-09-17T16:35:54.472 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3560156131' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:54.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:54 smithi017 ceph-mon[27571]: osdmap e775: 12 total, 10 up, 12 in 2024-09-17T16:35:54.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3560156131' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:54.544 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:35:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:54 smithi071 ceph-mon[30002]: osdmap e775: 12 total, 10 up, 12 in 2024-09-17T16:35:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3560156131' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:35:54.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:54 smithi071 podman[139219]: 2024-09-17 16:35:54.286885646 +0000 UTC m=+0.375988843 container init f7e4358b41a67efd3f2f1923ae644bedd48287ee0b7bf1411781cc458c033fba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T16:35:54.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:54 smithi071 podman[139219]: 2024-09-17 16:35:54.2903026 +0000 UTC m=+0.379405789 container start f7e4358b41a67efd3f2f1923ae644bedd48287ee0b7bf1411781cc458c033fba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-17T16:35:54.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:54 smithi071 podman[139219]: 2024-09-17 16:35:54.395277086 +0000 UTC m=+0.484380288 container attach f7e4358b41a67efd3f2f1923ae644bedd48287ee0b7bf1411781cc458c033fba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:35:54.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:54.573 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:35:54.573 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:35:54.573 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:35:54.573 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:35:54.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:54.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:54.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:54.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:54.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:54.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:54.818 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:54.843 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:54.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:54.890 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:35:54 smithi071 podman[139219]: 2024-09-17 16:35:54.576451341 +0000 UTC m=+0.665554545 container died f7e4358b41a67efd3f2f1923ae644bedd48287ee0b7bf1411781cc458c033fba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T16:35:54.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:54.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:54.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:55.071 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:55.094 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:55.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:55.197 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:55.220 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:55.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:55.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.5.service' 2024-09-17T16:35:55.311 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:35:55.311 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-17T16:35:55.311 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-09-17T16:35:55.311 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi071.front.sepia.ceph.com 2024-09-17T16:35:55.312 INFO:tasks.thrashosds.thrasher:remote for exp osd 10 is ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T16:35:55.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.10 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op list-pgs 2024-09-17T16:35:55.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:55.371 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:55.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:55.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:55 smithi017 ceph-mon[27571]: pgmap v1727: 27 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 stale+active+clean, 2 active+clean+remapped, 7 active+undersized+degraded, 14 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 926 op/s; 21281/257991 objects degraded (8.249%); 10897/257991 objects misplaced (4.224%); 6.0 MiB/s, 95 objects/s recovering 2024-09-17T16:35:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:55 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:35:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3560156131' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:55 smithi017 ceph-mon[27571]: osdmap e776: 12 total, 10 up, 12 in 2024-09-17T16:35:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:55 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 21281/257991 objects degraded (8.249%), 8 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:55.494 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:55.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:55 smithi071 ceph-mon[30002]: pgmap v1727: 27 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 stale+active+clean, 2 active+clean+remapped, 7 active+undersized+degraded, 14 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 926 op/s; 21281/257991 objects degraded (8.249%); 10897/257991 objects misplaced (4.224%); 6.0 MiB/s, 95 objects/s recovering 2024-09-17T16:35:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:55 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:35:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3560156131' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:55 smithi071 ceph-mon[30002]: osdmap e776: 12 total, 10 up, 12 in 2024-09-17T16:35:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:55 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 21281/257991 objects degraded (8.249%), 8 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:55.597 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:55.619 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:55.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:55 smithi173 ceph-mon[30530]: pgmap v1727: 27 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 stale+active+clean, 2 active+clean+remapped, 7 active+undersized+degraded, 14 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 926 op/s; 21281/257991 objects degraded (8.249%); 10897/257991 objects misplaced (4.224%); 6.0 MiB/s, 95 objects/s recovering 2024-09-17T16:35:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:55 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:35:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3560156131' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:35:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:55 smithi173 ceph-mon[30530]: osdmap e776: 12 total, 10 up, 12 in 2024-09-17T16:35:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:55 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 21281/257991 objects degraded (8.249%), 8 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:35:55.672 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:55.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:55.745 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:55.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:55.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:35:55.835 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.10/config 2024-09-17T16:35:55.845 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:55.870 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:55.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:55.971 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:55.997 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:55.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:56.099 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:56.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:56.122 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:56.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:56.204 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:35:56.204 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:35:56.223 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:56.226 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:35:56.226 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:35:56.246 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:35:56.246 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:35:56.247 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:56.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:56.267 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:35:56.267 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:35:56.289 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:35:56.289 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:35:56.289 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:35:56.310 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:35:56.311 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:35:56.332 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:35:56.332 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:35:56.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:56.353 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:35:56.354 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:35:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:56.374 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:56.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:56.375 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:35:56.375 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:35:56.375 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:35:56.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/611426449' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:56.397 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:35:56.397 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:35:56.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:56.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/611426449' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:56.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:56.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/611426449' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:35:56.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:56.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:56.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:56.723 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:56.746 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:56.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:56.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:35:56.871 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:56.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:56.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:35:56.969 INFO:teuthology.orchestra.run.smithi173.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:35:56.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:56.980 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:35:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:56.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:56.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:57.094 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:35:57.095 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:35:57.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:57.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:57.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:57.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:57.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:57.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:57.377 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:57.402 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:57.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:57.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:57 smithi017 ceph-mon[27571]: pgmap v1729: 27 pgs: 1 active+undersized+remapped, 1 peering, 3 active+clean+remapped, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 23901/255675 objects degraded (9.348%); 13390/255675 objects misplaced (5.237%); 8.5 MiB/s, 136 objects/s recovering 2024-09-17T16:35:57.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:57.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:57.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:57.527 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:57 smithi173 ceph-mon[30530]: pgmap v1729: 27 pgs: 1 active+undersized+remapped, 1 peering, 3 active+clean+remapped, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 23901/255675 objects degraded (9.348%); 13390/255675 objects misplaced (5.237%); 8.5 MiB/s, 136 objects/s recovering 2024-09-17T16:35:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:57 smithi071 ceph-mon[30002]: pgmap v1729: 27 pgs: 1 active+undersized+remapped, 1 peering, 3 active+clean+remapped, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 23901/255675 objects degraded (9.348%); 13390/255675 objects misplaced (5.237%); 8.5 MiB/s, 136 objects/s recovering 2024-09-17T16:35:57.627 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:35:57.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:57.654 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:57.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:57.753 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:35:57.753 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:35:57.756 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:57.779 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:57.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:57.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:57.878 INFO:teuthology.orchestra.run.smithi173.stdout:8.8 2024-09-17T16:35:57.878 INFO:teuthology.orchestra.run.smithi173.stdout:8.7 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.5 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.4 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.2 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:7.19 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.6 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.15 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.1 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.11 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:7.11 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.12 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.10 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.18 2024-09-17T16:35:57.879 INFO:teuthology.orchestra.run.smithi173.stdout:8.19 2024-09-17T16:35:57.880 INFO:teuthology.orchestra.run.smithi173.stdout:8.14 2024-09-17T16:35:57.880 INFO:teuthology.orchestra.run.smithi173.stdout:8.9 2024-09-17T16:35:57.880 INFO:teuthology.orchestra.run.smithi173.stdout:8.16 2024-09-17T16:35:57.880 INFO:teuthology.orchestra.run.smithi173.stdout:7.12 2024-09-17T16:35:57.880 INFO:teuthology.orchestra.run.smithi173.stdout:8.17 2024-09-17T16:35:57.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:35:57.895 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:35:57.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:57.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:57.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:35:58.004 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:35:58.026 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:58.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:35:58.128 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:58.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:58.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:58.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:58.277 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:58.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:58.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:35:58.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:35:58.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:35:58.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:58.402 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:58.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:35:58.482 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:35:58.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:35:58.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:58.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:35:58.530 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:58.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:35:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:35:58.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:58.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:58.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:58.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:58.755 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:58.776 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:58.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:58.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:35:58.901 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:58.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:35:59.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:35:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:59.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:59.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:35:59.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:35:59.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:35:59.162 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:35:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:59.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:35:59.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.10 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op export --pgid 8.7 --file /var/log/ceph/exp.8.7.10 2024-09-17T16:35:59.194 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:35:59.229 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:35:59.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:59.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:59.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:59.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:35:59.412 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:59.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:35:59.465 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.10/config 2024-09-17T16:35:59.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:59 smithi173 ceph-mon[30530]: pgmap v1730: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%); 7.0 MiB/s, 111 objects/s recovering 2024-09-17T16:35:59.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:59 smithi173 ceph-mon[30530]: osdmap e777: 12 total, 10 up, 12 in 2024-09-17T16:35:59.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:59 smithi173 ceph-mon[30530]: 8.e scrub starts 2024-09-17T16:35:59.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/679261936' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:35:59.467 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:35:59 smithi173 ceph-mon[30530]: 8.e scrub ok 2024-09-17T16:35:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:59 smithi017 ceph-mon[27571]: pgmap v1730: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%); 7.0 MiB/s, 111 objects/s recovering 2024-09-17T16:35:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:59 smithi017 ceph-mon[27571]: osdmap e777: 12 total, 10 up, 12 in 2024-09-17T16:35:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:59 smithi017 ceph-mon[27571]: 8.e scrub starts 2024-09-17T16:35:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/679261936' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:35:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:35:59 smithi017 ceph-mon[27571]: 8.e scrub ok 2024-09-17T16:35:59.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:59.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:59.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:59 smithi071 ceph-mon[30002]: pgmap v1730: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%); 7.0 MiB/s, 111 objects/s recovering 2024-09-17T16:35:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:59 smithi071 ceph-mon[30002]: osdmap e777: 12 total, 10 up, 12 in 2024-09-17T16:35:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:59 smithi071 ceph-mon[30002]: 8.e scrub starts 2024-09-17T16:35:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/679261936' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:35:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:35:59 smithi071 ceph-mon[30002]: 8.e scrub ok 2024-09-17T16:35:59.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:35:59.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:35:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:59.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:35:59.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:59.783 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:59.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:59.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:35:59.907 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:35:59.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:35:59.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:35:59.950 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:35:59.950 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:36:00.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:00.032 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:00.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:00.135 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:00.157 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:00.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:00.189 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:00.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:00.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:00.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:00.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:00 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:00.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/679261936' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:36:00.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:00 smithi017 ceph-mon[27571]: osdmap e778: 12 total, 10 up, 12 in 2024-09-17T16:36:00.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:00 smithi017 ceph-mon[27571]: 8.8 scrub starts 2024-09-17T16:36:00.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:00 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 29096/255615 objects degraded (11.383%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:36:00.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:36:00.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:36:00.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:36:00.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:00 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:36:00.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:00.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:00.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:00.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:00.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:00.530 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:00.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:00.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:00 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/679261936' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:36:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:00 smithi071 ceph-mon[30002]: osdmap e778: 12 total, 10 up, 12 in 2024-09-17T16:36:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:00 smithi071 ceph-mon[30002]: 8.8 scrub starts 2024-09-17T16:36:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:00 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 29096/255615 objects degraded (11.383%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:36:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:36:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:36:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:36:00.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:00 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:36:00.631 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:00 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/679261936' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:36:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:00 smithi173 ceph-mon[30530]: osdmap e778: 12 total, 10 up, 12 in 2024-09-17T16:36:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:00 smithi173 ceph-mon[30530]: 8.8 scrub starts 2024-09-17T16:36:00.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:00 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 29096/255615 objects degraded (11.383%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:36:00.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:36:00.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:36:00.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:36:00.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:00 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:36:00.652 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:00.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:00.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:00.757 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:36:00.757 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:36:00.778 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:36:00.778 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:36:00.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:00.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:00.798 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:36:00.799 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:36:00.819 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:36:00.819 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:36:00.840 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:36:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:36:00.840 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:36:00.862 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:36:00.862 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:36:00.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:00.883 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:36:00.883 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:36:00.904 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:36:00.904 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:36:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:00.905 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:00.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:00.938 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:36:00.938 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:36:00.938 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:36:00.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:36:00.960 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:36:01.006 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:01.029 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:01.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:01.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:01.153 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:01.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:01.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:01.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:01.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:01.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:36:01.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:01.404 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:01.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:01.506 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:01.532 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:01.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:01.590 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:01.590 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:01 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:01.590 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3361102971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:01.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:01.655 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:01.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:01 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:01.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3361102971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:01.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:01.781 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:01.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:01 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3361102971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:36:01.825 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:36:01.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:01.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:01.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:02.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:02.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:02.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:02.095 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:36:02.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:02.161 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:02.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:02.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:02.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:02.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:02.332 INFO:teuthology.orchestra.run.smithi173.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:36:02.352 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:02.386 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:02.408 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:02.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:02.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:02.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:02.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:02.586 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:02.586 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:02.586 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:36:02.586 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:36:02.586 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:02 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:02.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:02.656 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:02.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:36:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:36:02.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:02 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:02.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:02.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:02.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:02.802 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:36:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:36:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:36:02.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:02 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:02.881 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:02.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:02.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:03.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:03.010 INFO:teuthology.orchestra.run.smithi173.stderr:Exporting 8.7 info 8.7( v 733'506 (0'0,733'506] local-lis/les=736/737 n=253 ec=728/728 lis/c=736/736 les/c/f=737/737/0 sis=736) 2024-09-17T16:36:03.014 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e013f008:::benchmark_data_smithi173_130566_object2492:head# 2024-09-17T16:36:03.015 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e02436fb:::benchmark_data_smithi173_130566_object3392:head# 2024-09-17T16:36:03.015 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e02a5e53:::benchmark_data_smithi173_130566_object4279:head# 2024-09-17T16:36:03.016 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e02b4741:::benchmark_data_smithi173_130566_object4826:head# 2024-09-17T16:36:03.016 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e02ee428:::benchmark_data_smithi173_130566_object5894:head# 2024-09-17T16:36:03.016 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e039a39d:::benchmark_data_smithi173_130566_object6377:head# 2024-09-17T16:36:03.017 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e05561a0:::benchmark_data_smithi173_130566_object6908:head# 2024-09-17T16:36:03.017 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0612afd:::benchmark_data_smithi173_130566_object4432:head# 2024-09-17T16:36:03.018 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e067fd00:::benchmark_data_smithi173_130566_object5984:head# 2024-09-17T16:36:03.018 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e069acc8:::benchmark_data_smithi173_130566_object6050:head# 2024-09-17T16:36:03.019 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e06b1702:::benchmark_data_smithi173_130566_object3057:head# 2024-09-17T16:36:03.019 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e078b797:::benchmark_data_smithi173_130566_object2532:head# 2024-09-17T16:36:03.020 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e086b588:::benchmark_data_smithi173_130566_object443:head# 2024-09-17T16:36:03.020 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0a138f0:::benchmark_data_smithi173_130566_object3292:head# 2024-09-17T16:36:03.021 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0bad6c3:::benchmark_data_smithi173_130566_object1271:head# 2024-09-17T16:36:03.021 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0bdfd05:::benchmark_data_smithi173_130566_object5633:head# 2024-09-17T16:36:03.022 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0caed98:::benchmark_data_smithi173_130566_object3120:head# 2024-09-17T16:36:03.022 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0cc00f3:::benchmark_data_smithi173_130566_object4858:head# 2024-09-17T16:36:03.022 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0eb3aa5:::benchmark_data_smithi173_130566_object5587:head# 2024-09-17T16:36:03.023 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0f117c7:::benchmark_data_smithi173_130566_object7509:head# 2024-09-17T16:36:03.023 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0f7e64c:::benchmark_data_smithi173_130566_object4557:head# 2024-09-17T16:36:03.024 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0f9d939:::benchmark_data_smithi173_130566_object7243:head# 2024-09-17T16:36:03.024 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0fa5920:::benchmark_data_smithi173_130566_object2047:head# 2024-09-17T16:36:03.025 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e0fdc8f0:::benchmark_data_smithi173_130566_object5888:head# 2024-09-17T16:36:03.025 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1020f8d:::benchmark_data_smithi173_130566_object2178:head# 2024-09-17T16:36:03.025 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1022f9b:::benchmark_data_smithi173_130566_object1484:head# 2024-09-17T16:36:03.026 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1069b27:::benchmark_data_smithi173_130566_object2316:head# 2024-09-17T16:36:03.027 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e110922f:::benchmark_data_smithi173_130566_object4862:head# 2024-09-17T16:36:03.027 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e11d4c10:::benchmark_data_smithi173_130566_object7163:head# 2024-09-17T16:36:03.027 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1236541:::benchmark_data_smithi173_130566_object4232:head# 2024-09-17T16:36:03.028 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e133bb03:::benchmark_data_smithi173_130566_object4024:head# 2024-09-17T16:36:03.028 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e13a7b9f:::benchmark_data_smithi173_130566_object80:head# 2024-09-17T16:36:03.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:03.029 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e13e1c40:::benchmark_data_smithi173_130566_object6667:head# 2024-09-17T16:36:03.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:03.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:03.031 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e147b17c:::benchmark_data_smithi173_130566_object5258:head# 2024-09-17T16:36:03.031 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e149f0d3:::benchmark_data_smithi173_130566_object2165:head# 2024-09-17T16:36:03.032 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e14d87e8:::benchmark_data_smithi173_130566_object1485:head# 2024-09-17T16:36:03.032 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e15058e3:::benchmark_data_smithi173_130566_object6664:head# 2024-09-17T16:36:03.032 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e15b999c:::benchmark_data_smithi173_130566_object5565:head# 2024-09-17T16:36:03.032 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e15e8f66:::benchmark_data_smithi173_130566_object5750:head# 2024-09-17T16:36:03.032 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1691a85:::benchmark_data_smithi173_130566_object6097:head# 2024-09-17T16:36:03.033 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1691ee1:::benchmark_data_smithi173_130566_object1561:head# 2024-09-17T16:36:03.033 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e171e2ac:::benchmark_data_smithi173_130566_object1510:head# 2024-09-17T16:36:03.033 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e177c881:::benchmark_data_smithi173_130566_object3720:head# 2024-09-17T16:36:03.033 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e188d7dd:::benchmark_data_smithi173_130566_object7964:head# 2024-09-17T16:36:03.034 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e18d2a5d:::benchmark_data_smithi173_130566_object1355:head# 2024-09-17T16:36:03.034 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1997ee6:::benchmark_data_smithi173_130566_object2242:head# 2024-09-17T16:36:03.035 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e19c43de:::benchmark_data_smithi173_130566_object557:head# 2024-09-17T16:36:03.035 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1a1047f:::benchmark_data_smithi173_130566_object7716:head# 2024-09-17T16:36:03.036 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1a557b0:::benchmark_data_smithi173_130566_object7102:head# 2024-09-17T16:36:03.036 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1b8cffb:::benchmark_data_smithi173_130566_object7038:head# 2024-09-17T16:36:03.037 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1b8fe49:::benchmark_data_smithi173_130566_object1481:head# 2024-09-17T16:36:03.037 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1bd3cf9:::benchmark_data_smithi173_130566_object4075:head# 2024-09-17T16:36:03.038 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1c0fd10:::benchmark_data_smithi173_130566_object7947:head# 2024-09-17T16:36:03.038 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1c6ff69:::benchmark_data_smithi173_130566_object924:head# 2024-09-17T16:36:03.038 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1d6a6ee:::benchmark_data_smithi173_130566_object4825:head# 2024-09-17T16:36:03.039 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1dc146e:::benchmark_data_smithi173_130566_object755:head# 2024-09-17T16:36:03.040 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1dcd0be:::benchmark_data_smithi173_130566_object6109:head# 2024-09-17T16:36:03.040 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1e418f4:::benchmark_data_smithi173_130566_object986:head# 2024-09-17T16:36:03.040 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1e5f890:::benchmark_data_smithi173_130566_object7771:head# 2024-09-17T16:36:03.041 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1ed7c5c:::benchmark_data_smithi173_130566_object6414:head# 2024-09-17T16:36:03.041 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1ee682b:::benchmark_data_smithi173_130566_object2292:head# 2024-09-17T16:36:03.042 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e1fe9b48:::benchmark_data_smithi173_130566_object2971:head# 2024-09-17T16:36:03.042 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2109253:::benchmark_data_smithi173_130566_object5734:head# 2024-09-17T16:36:03.043 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e219e45a:::benchmark_data_smithi173_130566_object6923:head# 2024-09-17T16:36:03.043 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2284f8c:::benchmark_data_smithi173_130566_object7059:head# 2024-09-17T16:36:03.044 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e22ad34b:::benchmark_data_smithi173_130566_object49:head# 2024-09-17T16:36:03.044 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e22d5025:::benchmark_data_smithi173_130566_object1951:head# 2024-09-17T16:36:03.045 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e22f876a:::benchmark_data_smithi173_130566_object5113:head# 2024-09-17T16:36:03.045 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e22f976e:::benchmark_data_smithi173_130566_object5787:head# 2024-09-17T16:36:03.046 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e23796c9:::benchmark_data_smithi173_130566_object6862:head# 2024-09-17T16:36:03.046 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2384b73:::benchmark_data_smithi173_130566_object4972:head# 2024-09-17T16:36:03.047 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e23a4098:::benchmark_data_smithi173_130566_object6443:head# 2024-09-17T16:36:03.047 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e246c764:::benchmark_data_smithi173_130566_object3212:head# 2024-09-17T16:36:03.047 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e249798b:::benchmark_data_smithi173_130566_object3677:head# 2024-09-17T16:36:03.047 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e24c9a4e:::benchmark_data_smithi173_130566_object2035:head# 2024-09-17T16:36:03.048 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e253df38:::benchmark_data_smithi173_130566_object7224:head# 2024-09-17T16:36:03.048 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e25d3657:::benchmark_data_smithi173_130566_object5311:head# 2024-09-17T16:36:03.049 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2787418:::benchmark_data_smithi173_130566_object7496:head# 2024-09-17T16:36:03.049 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e281c15b:::benchmark_data_smithi173_130566_object322:head# 2024-09-17T16:36:03.049 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2a56fd7:::benchmark_data_smithi173_130566_object7733:head# 2024-09-17T16:36:03.050 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2b4d4b2:::benchmark_data_smithi173_130566_object1527:head# 2024-09-17T16:36:03.050 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2c001f8:::benchmark_data_smithi173_130566_object5386:head# 2024-09-17T16:36:03.051 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2c42bcb:::benchmark_data_smithi173_130566_object1032:head# 2024-09-17T16:36:03.051 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2c7034e:::benchmark_data_smithi173_130566_object5901:head# 2024-09-17T16:36:03.052 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2da8537:::benchmark_data_smithi173_130566_object19:head# 2024-09-17T16:36:03.052 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2e46e22:::benchmark_data_smithi173_130566_object3126:head# 2024-09-17T16:36:03.053 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2eb3c3a:::benchmark_data_smithi173_130566_object7140:head# 2024-09-17T16:36:03.053 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e2f8d8e4:::benchmark_data_smithi173_130566_object1315:head# 2024-09-17T16:36:03.053 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e301d688:::benchmark_data_smithi173_130566_object972:head# 2024-09-17T16:36:03.054 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e30a232c:::benchmark_data_smithi173_130566_object4619:head# 2024-09-17T16:36:03.054 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e31498c6:::benchmark_data_smithi173_130566_object1622:head# 2024-09-17T16:36:03.055 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e314aceb:::benchmark_data_smithi173_130566_object5246:head# 2024-09-17T16:36:03.055 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e31984cd:::benchmark_data_smithi173_130566_object7104:head# 2024-09-17T16:36:03.055 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3236d8f:::benchmark_data_smithi173_130566_object5425:head# 2024-09-17T16:36:03.056 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e32f974b:::benchmark_data_smithi173_130566_object7481:head# 2024-09-17T16:36:03.056 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e33fc415:::benchmark_data_smithi173_130566_object3199:head# 2024-09-17T16:36:03.057 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e342426b:::benchmark_data_smithi173_130566_object5316:head# 2024-09-17T16:36:03.057 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e345e6df:::benchmark_data_smithi173_130566_object6089:head# 2024-09-17T16:36:03.058 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e34f2152:::benchmark_data_smithi173_130566_object6553:head# 2024-09-17T16:36:03.058 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3637997:::benchmark_data_smithi173_130566_object5916:head# 2024-09-17T16:36:03.059 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e364185b:::benchmark_data_smithi173_130566_object2411:head# 2024-09-17T16:36:03.059 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3824d09:::benchmark_data_smithi173_130566_object4602:head# 2024-09-17T16:36:03.059 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3881772:::benchmark_data_smithi173_130566_object7544:head# 2024-09-17T16:36:03.060 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e38d3755:::benchmark_data_smithi173_130566_object7244:head# 2024-09-17T16:36:03.060 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3a213e6:::benchmark_data_smithi173_130566_object6579:head# 2024-09-17T16:36:03.061 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3a2d1a2:::benchmark_data_smithi173_130566_object3516:head# 2024-09-17T16:36:03.061 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3a95960:::benchmark_data_smithi173_130566_object3767:head# 2024-09-17T16:36:03.062 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3b5a797:::benchmark_data_smithi173_130566_object7201:head# 2024-09-17T16:36:03.062 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3c071a1:::benchmark_data_smithi173_130566_object2806:head# 2024-09-17T16:36:03.063 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3c4e113:::benchmark_data_smithi173_130566_object6439:head# 2024-09-17T16:36:03.063 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3cdf39a:::benchmark_data_smithi173_130566_object5172:head# 2024-09-17T16:36:03.064 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3d2e447:::benchmark_data_smithi173_130566_object6223:head# 2024-09-17T16:36:03.064 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3efe9ed:::benchmark_data_smithi173_130566_object7127:head# 2024-09-17T16:36:03.065 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3f15394:::benchmark_data_smithi173_130566_object7866:head# 2024-09-17T16:36:03.065 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e3faa680:::benchmark_data_smithi173_130566_object4415:head# 2024-09-17T16:36:03.066 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e401c53e:::benchmark_data_smithi173_130566_object3573:head# 2024-09-17T16:36:03.066 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e41d24ac:::benchmark_data_smithi173_130566_object4848:head# 2024-09-17T16:36:03.067 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4246fe4:::benchmark_data_smithi173_130566_object2498:head# 2024-09-17T16:36:03.067 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e426e512:::benchmark_data_smithi173_130566_object3772:head# 2024-09-17T16:36:03.068 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e42d0498:::benchmark_data_smithi173_130566_object525:head# 2024-09-17T16:36:03.068 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e434b224:::benchmark_data_smithi173_130566_object1160:head# 2024-09-17T16:36:03.069 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4381f3f:::benchmark_data_smithi173_130566_object5996:head# 2024-09-17T16:36:03.069 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e443d89f:::benchmark_data_smithi173_130566_object5185:head# 2024-09-17T16:36:03.069 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e451b21d:::benchmark_data_smithi173_130566_object6601:head# 2024-09-17T16:36:03.070 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4550051:::benchmark_data_smithi173_130566_object276:head# 2024-09-17T16:36:03.070 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4604412:::benchmark_data_smithi173_130566_object4033:head# 2024-09-17T16:36:03.070 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4654205:::benchmark_data_smithi173_130566_object2069:head# 2024-09-17T16:36:03.071 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4660ad6:::benchmark_data_smithi173_130566_object3269:head# 2024-09-17T16:36:03.071 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e46a9c0b:::benchmark_data_smithi173_130566_object3193:head# 2024-09-17T16:36:03.072 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4760a33:::benchmark_data_smithi173_130566_object1230:head# 2024-09-17T16:36:03.072 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e476670b:::benchmark_data_smithi173_130566_object4151:head# 2024-09-17T16:36:03.073 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4777771:::benchmark_data_smithi173_130566_object394:head# 2024-09-17T16:36:03.073 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e47fa40b:::benchmark_data_smithi173_130566_object7501:head# 2024-09-17T16:36:03.073 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e493d931:::benchmark_data_smithi173_130566_object6583:head# 2024-09-17T16:36:03.074 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4979842:::benchmark_data_smithi173_130566_object6720:head# 2024-09-17T16:36:03.074 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4ad4c7b:::benchmark_data_smithi173_130566_object3297:head# 2024-09-17T16:36:03.075 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4b2d15e:::benchmark_data_smithi173_130566_object3188:head# 2024-09-17T16:36:03.075 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4c154cd:::benchmark_data_smithi173_130566_object7768:head# 2024-09-17T16:36:03.076 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4c75e95:::benchmark_data_smithi173_130566_object6418:head# 2024-09-17T16:36:03.076 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4cb7016:::benchmark_data_smithi173_130566_object135:head# 2024-09-17T16:36:03.077 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4ceb052:::benchmark_data_smithi173_130566_object3410:head# 2024-09-17T16:36:03.077 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4d85f2a:::benchmark_data_smithi173_130566_object1049:head# 2024-09-17T16:36:03.078 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4e3bf60:::benchmark_data_smithi173_130566_object6938:head# 2024-09-17T16:36:03.078 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4e86e25:::benchmark_data_smithi173_130566_object6110:head# 2024-09-17T16:36:03.078 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4e8c708:::benchmark_data_smithi173_130566_object3590:head# 2024-09-17T16:36:03.079 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4ec5dd2:::benchmark_data_smithi173_130566_object4306:head# 2024-09-17T16:36:03.079 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e4f1c430:::benchmark_data_smithi173_130566_object5780:head# 2024-09-17T16:36:03.079 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e50deee5:::benchmark_data_smithi173_130566_object3771:head# 2024-09-17T16:36:03.080 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e50f7fb0:::benchmark_data_smithi173_130566_object5183:head# 2024-09-17T16:36:03.080 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e515c860:::benchmark_data_smithi173_130566_object2366:head# 2024-09-17T16:36:03.081 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e51655bb:::benchmark_data_smithi173_130566_object2832:head# 2024-09-17T16:36:03.081 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5168b2a:::benchmark_data_smithi173_130566_object2271:head# 2024-09-17T16:36:03.081 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e53ac44e:::benchmark_data_smithi173_130566_object2649:head# 2024-09-17T16:36:03.082 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5447035:::benchmark_data_smithi173_130566_object398:head# 2024-09-17T16:36:03.082 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e54c0118:::benchmark_data_smithi173_130566_object1479:head# 2024-09-17T16:36:03.083 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5515e32:::benchmark_data_smithi173_130566_object3809:head# 2024-09-17T16:36:03.083 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e55c5e26:::benchmark_data_smithi173_130566_object6657:head# 2024-09-17T16:36:03.084 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e55f7180:::benchmark_data_smithi173_130566_object3689:head# 2024-09-17T16:36:03.084 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e562eb35:::benchmark_data_smithi173_130566_object1252:head# 2024-09-17T16:36:03.084 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5660350:::benchmark_data_smithi173_130566_object6837:head# 2024-09-17T16:36:03.085 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e58d7961:::benchmark_data_smithi173_130566_object2314:head# 2024-09-17T16:36:03.085 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e590f721:::benchmark_data_smithi173_130566_object781:head# 2024-09-17T16:36:03.086 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5928211:::benchmark_data_smithi173_130566_object4997:head# 2024-09-17T16:36:03.086 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e59a351b:::benchmark_data_smithi173_130566_object3794:head# 2024-09-17T16:36:03.086 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e59b39db:::benchmark_data_smithi173_130566_object3351:head# 2024-09-17T16:36:03.087 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5a4e826:::benchmark_data_smithi173_130566_object2380:head# 2024-09-17T16:36:03.088 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5a6e395:::benchmark_data_smithi173_130566_object3216:head# 2024-09-17T16:36:03.089 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5aa3a87:::benchmark_data_smithi173_130566_object5151:head# 2024-09-17T16:36:03.089 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5aaa26f:::benchmark_data_smithi173_130566_object4414:head# 2024-09-17T16:36:03.089 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5b9ddeb:::benchmark_data_smithi173_130566_object6128:head# 2024-09-17T16:36:03.090 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5c379ac:::benchmark_data_smithi173_130566_object3734:head# 2024-09-17T16:36:03.090 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5c9cc81:::benchmark_data_smithi173_130566_object205:head# 2024-09-17T16:36:03.091 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5d848f4:::benchmark_data_smithi173_130566_object7188:head# 2024-09-17T16:36:03.091 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5dae115:::benchmark_data_smithi173_130566_object3844:head# 2024-09-17T16:36:03.091 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5dff33a:::benchmark_data_smithi173_130566_object534:head# 2024-09-17T16:36:03.092 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5eaba9b:::benchmark_data_smithi173_130566_object4289:head# 2024-09-17T16:36:03.092 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5ee483e:::benchmark_data_smithi173_130566_object4699:head# 2024-09-17T16:36:03.093 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e5ff1dfc:::benchmark_data_smithi173_130566_object6813:head# 2024-09-17T16:36:03.093 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e60932c6:::benchmark_data_smithi173_130566_object2678:head# 2024-09-17T16:36:03.093 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e60cca13:::benchmark_data_smithi173_130566_object6211:head# 2024-09-17T16:36:03.094 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e612e5d6:::benchmark_data_smithi173_130566_object1933:head# 2024-09-17T16:36:03.094 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e61860d3:::benchmark_data_smithi173_130566_object5597:head# 2024-09-17T16:36:03.095 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e61a6da4:::benchmark_data_smithi173_130566_object1782:head# 2024-09-17T16:36:03.095 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e62a11f9:::benchmark_data_smithi173_130566_object1352:head# 2024-09-17T16:36:03.096 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6391825:::benchmark_data_smithi173_130566_object3070:head# 2024-09-17T16:36:03.096 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e63fb09e:::benchmark_data_smithi173_130566_object4515:head# 2024-09-17T16:36:03.096 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e643712e:::benchmark_data_smithi173_130566_object442:head# 2024-09-17T16:36:03.097 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e646da6c:::benchmark_data_smithi173_130566_object4814:head# 2024-09-17T16:36:03.097 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6520bc8:::benchmark_data_smithi173_130566_object5689:head# 2024-09-17T16:36:03.098 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6587638:::benchmark_data_smithi173_130566_object6605:head# 2024-09-17T16:36:03.098 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e664f82f:::benchmark_data_smithi173_130566_object7725:head# 2024-09-17T16:36:03.098 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6786f7a:::benchmark_data_smithi173_130566_object4093:head# 2024-09-17T16:36:03.098 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e67b2e3a:::benchmark_data_smithi173_130566_object192:head# 2024-09-17T16:36:03.099 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e67bbd7c:::benchmark_data_smithi173_130566_object4864:head# 2024-09-17T16:36:03.099 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6911040:::benchmark_data_smithi173_130566_object7245:head# 2024-09-17T16:36:03.100 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6939b5b:::benchmark_data_smithi173_130566_object6802:head# 2024-09-17T16:36:03.100 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6a0de3c:::benchmark_data_smithi173_130566_object751:head# 2024-09-17T16:36:03.100 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6b64760:::benchmark_data_smithi173_130566_object2230:head# 2024-09-17T16:36:03.101 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6c0ff5c:::benchmark_data_smithi173_130566_object6297:head# 2024-09-17T16:36:03.101 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6dbf7ff:::benchmark_data_smithi173_130566_object5740:head# 2024-09-17T16:36:03.102 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6debbd7:::benchmark_data_smithi173_130566_object1828:head# 2024-09-17T16:36:03.102 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6e4e99b:::benchmark_data_smithi173_130566_object1512:head# 2024-09-17T16:36:03.102 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6edcbe0:::benchmark_data_smithi173_130566_object670:head# 2024-09-17T16:36:03.103 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6eea838:::benchmark_data_smithi173_130566_object1851:head# 2024-09-17T16:36:03.103 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6ef97f5:::benchmark_data_smithi173_130566_object3313:head# 2024-09-17T16:36:03.104 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6f44ece:::benchmark_data_smithi173_130566_object2121:head# 2024-09-17T16:36:03.104 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6f55464:::benchmark_data_smithi173_130566_object7442:head# 2024-09-17T16:36:03.104 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e6ff1ad0:::benchmark_data_smithi173_130566_object6200:head# 2024-09-17T16:36:03.105 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e70066fc:::benchmark_data_smithi173_130566_object6480:head# 2024-09-17T16:36:03.105 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7010a58:::benchmark_data_smithi173_130566_object2295:head# 2024-09-17T16:36:03.106 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7050b9f:::benchmark_data_smithi173_130566_object956:head# 2024-09-17T16:36:03.106 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e707218a:::benchmark_data_smithi173_130566_object2936:head# 2024-09-17T16:36:03.106 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7122980:::benchmark_data_smithi173_130566_object4488:head# 2024-09-17T16:36:03.107 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e73050d7:::benchmark_data_smithi173_130566_object4534:head# 2024-09-17T16:36:03.107 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7352532:::benchmark_data_smithi173_130566_object4346:head# 2024-09-17T16:36:03.108 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7440a17:::benchmark_data_smithi173_130566_object2775:head# 2024-09-17T16:36:03.108 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7458ab3:::benchmark_data_smithi173_130566_object3954:head# 2024-09-17T16:36:03.109 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e746d936:::benchmark_data_smithi173_130566_object4318:head# 2024-09-17T16:36:03.109 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e74857ab:::benchmark_data_smithi173_130566_object7332:head# 2024-09-17T16:36:03.109 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e748e0d7:::benchmark_data_smithi173_130566_object2615:head# 2024-09-17T16:36:03.110 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7509560:::benchmark_data_smithi173_130566_object3309:head# 2024-09-17T16:36:03.110 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e75891df:::benchmark_data_smithi173_130566_object6898:head# 2024-09-17T16:36:03.111 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e763011e:::benchmark_data_smithi173_130566_object5399:head# 2024-09-17T16:36:03.111 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e766a366:::benchmark_data_smithi173_130566_object1013:head# 2024-09-17T16:36:03.111 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e76d72ad:::benchmark_data_smithi173_130566_object701:head# 2024-09-17T16:36:03.112 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e76e20bf:::benchmark_data_smithi173_130566_object570:head# 2024-09-17T16:36:03.112 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e76ef68b:::benchmark_data_smithi173_130566_object4775:head# 2024-09-17T16:36:03.113 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e76f5c2b:::benchmark_data_smithi173_130566_object5936:head# 2024-09-17T16:36:03.113 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7750079:::benchmark_data_smithi173_130566_object3592:head# 2024-09-17T16:36:03.113 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e78beab2:::benchmark_data_smithi173_130566_object1193:head# 2024-09-17T16:36:03.114 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e78c4cda:::benchmark_data_smithi173_130566_object2176:head# 2024-09-17T16:36:03.114 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e79c7a5d:::benchmark_data_smithi173_130566_object7901:head# 2024-09-17T16:36:03.115 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7a6e4aa:::benchmark_data_smithi173_130566_object5158:head# 2024-09-17T16:36:03.115 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7af7a73:::benchmark_data_smithi173_130566_object6606:head# 2024-09-17T16:36:03.116 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7c199cf:::benchmark_data_smithi173_130566_object7577:head# 2024-09-17T16:36:03.116 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7cae102:::benchmark_data_smithi173_130566_object7893:head# 2024-09-17T16:36:03.117 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7e06317:::benchmark_data_smithi173_130566_object2961:head# 2024-09-17T16:36:03.117 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7e383ce:::benchmark_data_smithi173_130566_object2219:head# 2024-09-17T16:36:03.117 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7e7772d:::benchmark_data_smithi173_130566_object6627:head# 2024-09-17T16:36:03.117 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7e9f195:::benchmark_data_smithi173_130566_object6901:head# 2024-09-17T16:36:03.118 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7ef617b:::benchmark_data_smithi173_130566_object3014:head# 2024-09-17T16:36:03.118 INFO:teuthology.orchestra.run.smithi173.stderr:Read #8:e7ff4392:::benchmark_data_smithi173_130566_object3242:head# 2024-09-17T16:36:03.119 INFO:teuthology.orchestra.run.smithi173.stderr:Export successful 2024-09-17T16:36:03.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:03.153 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:03.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:03.256 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:03.278 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:03.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:03.352 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:36:03.352 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:36:03.352 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:36:03.352 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:36:03.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:03.406 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:03.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:03.508 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:03.531 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:03.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:03.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:03.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:03.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:03.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:03.657 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:03 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:03 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:03 smithi017 ceph-mon[27571]: pgmap v1734: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+undersized+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%); 763 KiB/s, 12 objects/s recovering 2024-09-17T16:36:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:03 smithi017 ceph-mon[27571]: osdmap e779: 12 total, 10 up, 12 in 2024-09-17T16:36:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2773473883' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:03.758 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:03.784 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:03.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:03 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:03 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:03 smithi071 ceph-mon[30002]: pgmap v1734: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+undersized+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%); 763 KiB/s, 12 objects/s recovering 2024-09-17T16:36:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:03 smithi071 ceph-mon[30002]: osdmap e779: 12 total, 10 up, 12 in 2024-09-17T16:36:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2773473883' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:03.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:36:03.886 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:03 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:03 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:03 smithi173 ceph-mon[30530]: pgmap v1734: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+undersized+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 100 op/s; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%); 763 KiB/s, 12 objects/s recovering 2024-09-17T16:36:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:03 smithi173 ceph-mon[30530]: osdmap e779: 12 total, 10 up, 12 in 2024-09-17T16:36:03.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2773473883' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:03.910 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:03.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:04.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:04.016 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:36:04.038 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:04.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:04.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:04.140 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:04.162 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:04.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:04.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:04.288 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:04.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:04.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.10 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --force --op remove --pgid 8.7 2024-09-17T16:36:04.390 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:04.416 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:04.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:04.518 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:04.541 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:04.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:04.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:04.656 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.10/config 2024-09-17T16:36:04.669 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:04.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:04.771 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:04.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:36:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:04.798 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:04.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:04.863 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:04.899 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:04.925 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:04.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:05.029 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:05.054 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:05.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:05 smithi173 ceph-mon[30530]: 8.14 scrub starts 2024-09-17T16:36:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:05 smithi173 ceph-mon[30530]: 8.12 scrub starts 2024-09-17T16:36:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:05 smithi173 ceph-mon[30530]: 8.15 scrub starts 2024-09-17T16:36:05.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2773473883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:05.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:05 smithi173 ceph-mon[30530]: osdmap e780: 12 total, 10 up, 12 in 2024-09-17T16:36:05.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:05 smithi173 ceph-mon[30530]: pgmap v1737: 27 pgs: 1 active+undersized+remapped, 1 active+clean+remapped+scrubbing, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:05.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:05 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:05.154 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:05.178 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:05.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:05.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:05.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:05 smithi017 ceph-mon[27571]: 8.14 scrub starts 2024-09-17T16:36:05.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:05 smithi017 ceph-mon[27571]: 8.12 scrub starts 2024-09-17T16:36:05.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:05 smithi017 ceph-mon[27571]: 8.15 scrub starts 2024-09-17T16:36:05.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2773473883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:05.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:05 smithi017 ceph-mon[27571]: osdmap e780: 12 total, 10 up, 12 in 2024-09-17T16:36:05.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:05 smithi017 ceph-mon[27571]: pgmap v1737: 27 pgs: 1 active+undersized+remapped, 1 active+clean+remapped+scrubbing, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:05.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:05 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:05.305 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:05.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:05.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:05 smithi071 ceph-mon[30002]: 8.14 scrub starts 2024-09-17T16:36:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:05 smithi071 ceph-mon[30002]: 8.12 scrub starts 2024-09-17T16:36:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:05 smithi071 ceph-mon[30002]: 8.15 scrub starts 2024-09-17T16:36:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2773473883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:05 smithi071 ceph-mon[30002]: osdmap e780: 12 total, 10 up, 12 in 2024-09-17T16:36:05.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:05 smithi071 ceph-mon[30002]: pgmap v1737: 27 pgs: 1 active+undersized+remapped, 1 active+clean+remapped+scrubbing, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:05.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:05 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:05.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:05.410 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:36:05.411 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:36:05.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:05.433 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:36:05.433 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:36:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:05.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:05.452 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:36:05.453 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:36:05.473 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:36:05.473 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:36:05.495 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:36:05.495 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:36:05.495 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:36:05.516 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:36:05.517 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:36:05.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:05.538 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:36:05.538 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:36:05.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:05.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:36:05.560 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:36:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:05.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:05.581 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:36:05.582 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:36:05.582 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:36:05.603 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:36:05.603 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:36:05.662 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:05.684 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:05.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:05.787 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:05.810 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:05.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:05.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:05.937 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:05.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:05.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:06.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:06.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:06.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:06 smithi173 ceph-mon[30530]: 8.10 scrub starts 2024-09-17T16:36:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4096851760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:06.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:06.171 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:36:06.190 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:06.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:06.214 INFO:teuthology.orchestra.run.smithi173.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:36:06.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:06 smithi017 ceph-mon[27571]: 8.10 scrub starts 2024-09-17T16:36:06.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4096851760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:06.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:06.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:06.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:06.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:06.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:06 smithi071 ceph-mon[30002]: 8.10 scrub starts 2024-09-17T16:36:06.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4096851760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:36:06.379 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:36:06.415 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:06.441 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:06.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:06.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:06.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:06.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:06.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:06.694 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:06.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:06.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:36:06.796 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:06.825 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:06.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:06.927 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:06.952 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:06.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:07.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:07.063 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:07.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:07.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:07.116 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:07 smithi173 ceph-mon[30530]: pgmap v1738: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:07.116 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:36:07.117 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:36:07.180 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:07.184 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:07 smithi017 ceph-mon[27571]: pgmap v1738: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:07.204 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:07.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:07.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:07.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:07 smithi071 ceph-mon[30002]: pgmap v1738: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:07.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:07.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:07.432 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:07.456 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:07.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:07.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:07.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:07.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:07.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:07.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:07.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: marking collection for removal 2024-09-17T16:36:07.725 INFO:teuthology.orchestra.run.smithi173.stdout:setting '_remove' omap key 2024-09-17T16:36:07.725 INFO:teuthology.orchestra.run.smithi173.stdout:finish_remove_pgs 8.7_head removing 8.7 2024-09-17T16:36:07.730 INFO:teuthology.orchestra.run.smithi173.stdout:Remove successful 2024-09-17T16:36:07.804 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:07.830 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:07.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:07.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:36:07.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:07.955 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:07.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:08.031 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:36:08.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:08.080 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:08.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:08.150 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:36:08.182 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:08.208 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:08.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:08.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3882067481' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:36:08.310 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:08.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3882067481' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:36:08.335 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:08.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:08.380 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:36:08.396 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3882067481' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:36:08.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:08.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:08.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:08.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:08.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:08.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:08.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:08.624 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:36:08.624 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:36:08.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:08.707 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:08.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:08.809 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:08.834 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:08.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:08.868 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:08.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:08.961 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:08.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:08.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:09.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:09.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:09.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:09.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:09.209 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:09.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:09.300 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3882067481' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:36:09.300 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:09 smithi173 ceph-mon[30530]: osdmap e781: 12 total, 10 up, 12 in 2024-09-17T16:36:09.300 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:09 smithi173 ceph-mon[30530]: pgmap v1740: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:09.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3882067481' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:36:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:09 smithi071 ceph-mon[30002]: osdmap e781: 12 total, 10 up, 12 in 2024-09-17T16:36:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:09 smithi071 ceph-mon[30002]: pgmap v1740: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:09.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3882067481' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:36:09.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:09 smithi017 ceph-mon[27571]: osdmap e781: 12 total, 10 up, 12 in 2024-09-17T16:36:09.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:09 smithi017 ceph-mon[27571]: pgmap v1740: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:09.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:09.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:09.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:09.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:36:09.436 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:09.459 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:09.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:09.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:09.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:09.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:09.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:09.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:09.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:09.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:09.811 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:09.833 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:09.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:09.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:09.955 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:09.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:10.020 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.5 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-09-17T16:36:10.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:10.059 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:36:10.083 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:10.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:10.185 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:10.210 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:10.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:10.212 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.5/config 2024-09-17T16:36:10.256 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:36:10.257 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:36:10.279 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:36:10.279 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:36:10.288 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2954191387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:10.288 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:10 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:10.300 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:36:10.301 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:36:10.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2954191387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:10.301 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:10 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:10.312 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2954191387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:10.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:10 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:10.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:36:10.322 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:36:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:10.336 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:10.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:10.344 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:36:10.345 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:36:10.345 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:36:10.367 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:36:10.367 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:36:10.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:36:10.389 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:36:10.410 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:36:10.410 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:36:10.430 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:36:10.431 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:36:10.431 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:36:10.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:10.453 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:36:10.453 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:36:10.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:10.461 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:10.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:10.584 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:10.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:10.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:10.703 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:36:10.703 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:36:10.703 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:36:10.703 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:36:10.707 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:10.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:10.791 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:36:10.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:10.855 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:10.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:10.958 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:10.979 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:10.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:11.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:11.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:11.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:11.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:11.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:11.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:11.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:11 smithi017 ceph-mon[27571]: pgmap v1741: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:36:11.299 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:36:11.303 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:11 smithi173 ceph-mon[30530]: pgmap v1741: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:11.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:11 smithi071 ceph-mon[30002]: pgmap v1741: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:11.327 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:11.350 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:11.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:11.377 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:11.421 INFO:teuthology.orchestra.run.smithi071.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:36:11.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:36:11.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:11.476 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:11.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:11.578 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:11.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:11.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:11.677 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:36:11.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:11.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:11.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:11.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:11.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:11.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:11.950 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:11.972 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:11.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:12.074 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:12.096 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:12.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:12.130 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:36:12.162 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:36:12.162 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:36:12.162 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:36:12.162 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:36:12.162 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:36:12.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:12.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:12.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:12.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:12.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:12.345 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2229960073' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:12.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:12.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2229960073' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:12.469 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:12.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2229960073' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:12.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:12.592 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:12.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:12.694 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:12.718 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:12.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:12.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:36:12.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:12.873 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:12.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:12.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:12.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:12.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:12.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:12.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:13.052 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:36:13.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:13.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:13.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:13.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:13.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:13.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:13.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:13.300 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:36:13.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:13.371 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:13.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:13 smithi173 ceph-mon[30530]: pgmap v1742: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:13 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:36:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2229960073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:13 smithi173 ceph-mon[30530]: osdmap e782: 12 total, 10 up, 12 in 2024-09-17T16:36:13.472 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:13 smithi017 ceph-mon[27571]: pgmap v1742: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:13 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:36:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2229960073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:13 smithi017 ceph-mon[27571]: osdmap e782: 12 total, 10 up, 12 in 2024-09-17T16:36:13.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:13.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:13.534 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:36:13.534 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:36:13.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:36:13.534 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:36:13.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.534 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:113603 6.8 e582/527 6:1d7478f4:::benchmark_data_smithi173_102017_object20455:head v 582'7045, mlcod=582'7042)", 2024-09-17T16:36:13.534 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:36:13.535 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 542.92131949700001, 2024-09-17T16:36:13.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.183501209, 2024-09-17T16:36:13.535 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.535 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.535 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.535 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609726+0000", 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609728+0000", 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.176e-06 2024-09-17T16:36:13.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.609742+0000", 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3703000000000001e-05 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790389+0000", 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18064696399999999 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790402+0000", 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3239e-05 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790530+0000", 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012778799999999999 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.790567+0000", 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6999000000000003e-05 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.793200+0000", 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026325070000000001 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:27:10.793227+0000", 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7832999999999999e-05 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.539 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.540 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:29928 7.a e646/629 7:54337644:::benchmark_data_smithi173_116396_object29927:head v 646'1856, mlcod=646'1852)", 2024-09-17T16:36:13.540 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:36:13.540 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 366.240845202, 2024-09-17T16:36:13.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.232420081, 2024-09-17T16:36:13.540 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:36:13.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290207+0000", 2024-09-17T16:36:13.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6579999999999996e-06 2024-09-17T16:36:13.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290242+0000", 2024-09-17T16:36:13.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4696000000000001e-05 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521561+0000", 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23131880499999999 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521567+0000", 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.392e-06 2024-09-17T16:36:13.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521628+0000", 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0473999999999997e-05 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521650+0000", 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2909e-05 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.522588+0000", 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093747100000000005 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.522621+0000", 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2675999999999997e-05 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:46467 7.b e650/629 7:d098567c:::benchmark_data_smithi173_116396_object46466:head v 650'2942, mlcod=650'2939)", 2024-09-17T16:36:13.545 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 352.36739938099998, 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.213043449, 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:36:13.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163647+0000", 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163708+0000", 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1152e-05 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.374934+0000", 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21122532999999999 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.374948+0000", 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4756000000000001e-05 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.375068+0000", 2024-09-17T16:36:13.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119771 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.375108+0000", 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9619000000000001e-05 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.376653+0000", 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001545416 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.376690+0000", 2024-09-17T16:36:13.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6631000000000002e-05 2024-09-17T16:36:13.550 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.550 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.550 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.550 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:103843 7.b e678/629 7:d0ef6818:::benchmark_data_smithi173_116396_object8243:head v 678'6575, mlcod=678'6572)", 2024-09-17T16:36:13.550 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:25.488955+0000", 2024-09-17T16:36:13.550 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 288.04209107000003, 2024-09-17T16:36:13.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.209073274, 2024-09-17T16:36:13.551 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.551 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.551 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488955+0000", 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488955+0000", 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488956+0000", 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4610000000000001e-06 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488964+0000", 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2910000000000007e-06 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695150+0000", 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20618561899999999 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695166+0000", 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5926e-05 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695324+0000", 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000157873 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695362+0000", 2024-09-17T16:36:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8170000000000002e-05 2024-09-17T16:36:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.697999+0000", 2024-09-17T16:36:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026369560000000002 2024-09-17T16:36:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.698028+0000", 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8977999999999999e-05 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 685/677 [PushOp(7:2d0cb49e:::benchmark_data_smithi173_116396_object20958:head, version: 643'1334, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2d0cb49e:::benchmark_data_smithi173_116396_object20958:head@643'1334, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 261.50492162799998, 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25935365100000002, 2024-09-17T16:36:13.556 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:36:13.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026125+0000", 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026176+0000", 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1581e-05 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283828+0000", 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25765147999999999 2024-09-17T16:36:13.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283845+0000", 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7275e-05 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283961+0000", 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011592899999999999 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283974+0000", 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2724e-05 2024-09-17T16:36:13.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.285478+0000", 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015038720000000001 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 685/677 [PushOp(7:295d1d25:::benchmark_data_smithi173_116396_object23683:head, version: 645'1499, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:295d1d25:::benchmark_data_smithi173_116396_object23683:head@645'1499, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 255.97967432499999, 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24965942299999999, 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.560 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551375+0000", 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9809999999999997e-06 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551452+0000", 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7064999999999995e-05 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799411+0000", 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.247958129 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799426+0000", 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5094e-05 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799521+0000", 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5449e-05 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799534+0000", 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3077e-05 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.801031+0000", 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014966280000000001 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 687/677 [PushOp(7:2aa25eb8:::benchmark_data_smithi173_116396_object25172:head, version: 645'1590, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2aa25eb8:::benchmark_data_smithi173_116396_object25172:head@645'1590, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 252.37933503299999, 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25723122799999998, 2024-09-17T16:36:13.564 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:36:13.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151712+0000", 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3030000000000001e-06 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151805+0000", 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3330000000000003e-05 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.406885+0000", 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25507977999999998 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.406910+0000", 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4736000000000002e-05 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.407016+0000", 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106372 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.407029+0000", 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3032e-05 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.408942+0000", 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001912675 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 690/677 [PushOp(7:2d367e55:::benchmark_data_smithi173_116396_object28235:head, version: 646'1783, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2d367e55:::benchmark_data_smithi173_116396_object28235:head@646'1783, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 246.39437538600001, 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26142421399999999, 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:36:13.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136671+0000", 2024-09-17T16:36:13.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210000000000001e-06 2024-09-17T16:36:13.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136740+0000", 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8932999999999999e-05 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396250+0000", 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25950979000000002 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396266+0000", 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5874000000000001e-05 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396362+0000", 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5543000000000003e-05 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396374+0000", 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2897999999999999e-05 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.398095+0000", 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017201549999999999 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:123977 7.14 e690/677 7:2b1a1b49:::benchmark_data_smithi173_116396_object28377:head v 690'7133, mlcod=646'1781)", 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 246.09084114199999, 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.206586876, 2024-09-17T16:36:13.573 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.574 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.574 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.574 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:36:13.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440212+0000", 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2199999999999997e-06 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.641936+0000", 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20172389900000001 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.641943+0000", 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1010000000000003e-06 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.641998+0000", 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5139999999999997e-05 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.642025+0000", 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7662999999999999e-05 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.646753+0000", 2024-09-17T16:36:13.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047274800000000001 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.646791+0000", 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8574e-05 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:129321 7.b e695/629 7:d9dcd2c1:::benchmark_data_smithi173_116396_object33721:head v 695'8200, mlcod=695'8195)", 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:17.838831+0000", 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 235.692214576, 2024-09-17T16:36:13.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18822972700000001, 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838831+0000", 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838831+0000", 2024-09-17T16:36:13.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838832+0000", 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.054e-06 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838839+0000", 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7059999999999998e-06 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022112+0000", 2024-09-17T16:36:13.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18327316599999999 2024-09-17T16:36:13.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022120+0000", 2024-09-17T16:36:13.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9039999999999992e-06 2024-09-17T16:36:13.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022180+0000", 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0446999999999999e-05 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022202+0000", 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1750999999999999e-05 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.027011+0000", 2024-09-17T16:36:13.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048087299999999998 2024-09-17T16:36:13.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.027061+0000", 2024-09-17T16:36:13.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9969000000000002e-05 2024-09-17T16:36:13.588 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.589 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.589 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.589 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:129340 7.b e695/629 7:d6eb756c:::benchmark_data_smithi173_116396_object33740:head v 695'8201, mlcod=695'8197)", 2024-09-17T16:36:13.589 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:36:13.589 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 235.663218704, 2024-09-17T16:36:13.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.190853825, 2024-09-17T16:36:13.589 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.589 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.590 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.590 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:36:13.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:36:13.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:36:13.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4700000000000002e-07 2024-09-17T16:36:13.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867833+0000", 2024-09-17T16:36:13.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2320000000000001e-06 2024-09-17T16:36:13.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054604+0000", 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18677139100000001 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054610+0000", 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1029999999999998e-06 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054667+0000", 2024-09-17T16:36:13.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6928000000000001e-05 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054688+0000", 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0653e-05 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.058610+0000", 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039225379999999997 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.058681+0000", 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0532999999999997e-05 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 696/677 [PushOp(7:2bba9c3c:::benchmark_data_smithi173_116396_object40626:head, version: 650'2606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2bba9c3c:::benchmark_data_smithi173_116396_object40626:head@650'2606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:30.395026+0000", 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 223.136020008, 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19917216800000001, 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:36:13.595 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395026+0000", 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395026+0000", 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395027+0000", 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.421e-06 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395110+0000", 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2409000000000003e-05 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591821+0000", 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19671128299999999 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591844+0000", 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.34e-05 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591949+0000", 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010453700000000001 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591961+0000", 2024-09-17T16:36:13.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2354000000000001e-05 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.594198+0000", 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002236764 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:170897 7.5 e712/674 7:a2dbce43:::benchmark_data_smithi173_116396_object75297:head v 712'8535, mlcod=712'8532)", 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:36:13.599 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 167.092303337, 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34638644600000001, 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438743+0000", 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0529999999999999e-06 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438750+0000", 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7989999999999996e-06 2024-09-17T16:36:13.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.782943+0000", 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34419256799999998 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.782967+0000", 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3929000000000001e-05 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.783013+0000", 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6400999999999998e-05 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.783027+0000", 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4337e-05 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785093+0000", 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002065271 2024-09-17T16:36:13.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785129+0000", 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6087999999999997e-05 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:178680 7.13 e719/714 7:cae8152e:::benchmark_data_smithi173_116396_object83080:head v 719'8741, mlcod=719'8738)", 2024-09-17T16:36:13.604 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 155.44498828900001, 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31485432099999999, 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086058+0000", 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.85e-07 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086063+0000", 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0050000000000004e-06 2024-09-17T16:36:13.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398605+0000", 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31254172400000002 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398610+0000", 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2789999999999998e-06 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398655+0000", 2024-09-17T16:36:13.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4715000000000001e-05 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398670+0000", 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4895999999999999e-05 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.400870+0000", 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022001920000000001 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.400912+0000", 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1625000000000002e-05 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:191187 7.14 e724/717 7:2d119a99:::benchmark_data_smithi173_116396_object95587:head v 724'9235, mlcod=724'9232)", 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 139.083862079, 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48898925799999998, 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.609 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7599999999999996e-07 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447190+0000", 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9440000000000002e-06 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933730+0000", 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48653948800000002 2024-09-17T16:36:13.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933739+0000", 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9199999999999993e-06 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933837+0000", 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8004999999999994e-05 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933856+0000", 2024-09-17T16:36:13.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.891e-05 2024-09-17T16:36:13.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.936137+0000", 2024-09-17T16:36:13.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022811960000000001 2024-09-17T16:36:13.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.936173+0000", 2024-09-17T16:36:13.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6019000000000002e-05 2024-09-17T16:36:13.614 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.614 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.614 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:13.615 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:13.616 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:13 smithi071 ceph-mon[30002]: pgmap v1742: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:13.616 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:13 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:36:13.616 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2229960073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:13.616 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:13 smithi071 ceph-mon[30002]: osdmap e782: 12 total, 10 up, 12 in 2024-09-17T16:36:13.616 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:1442 8.f e731/728 8:f297aed2:::benchmark_data_smithi173_130566_object1441:head v 731'97, mlcod=731'92)", 2024-09-17T16:36:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:12.402004+0000", 2024-09-17T16:36:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.12904171700001, 2024-09-17T16:36:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.188874654, 2024-09-17T16:36:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.617 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402004+0000", 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402004+0000", 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402005+0000", 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3200000000000004e-07 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402059+0000", 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4203000000000003e-05 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590370+0000", 2024-09-17T16:36:13.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18831092399999999 2024-09-17T16:36:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590374+0000", 2024-09-17T16:36:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.177e-06 2024-09-17T16:36:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590443+0000", 2024-09-17T16:36:13.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8479000000000006e-05 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590466+0000", 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2901999999999999e-05 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590851+0000", 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038551500000000001 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590879+0000", 2024-09-17T16:36:13.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7622000000000001e-05 2024-09-17T16:36:13.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.622 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.622 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.16 740/739 [PushOp(8:68fc3949:::benchmark_data_smithi173_130566_object1854:head, version: 732'115, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:68fc3949:::benchmark_data_smithi173_130566_object1854:head@732'115, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:36:13.622 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 106.63180502, 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.75943818799999996, 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5700000000000002e-07 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899299+0000", 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7451999999999998e-05 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658299+0000", 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.75900059799999997 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658317+0000", 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:36:13.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658403+0000", 2024-09-17T16:36:13.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6600000000000004e-05 2024-09-17T16:36:13.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658413+0000", 2024-09-17T16:36:13.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9340000000000001e-06 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658679+0000", 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026586099999999999 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:35399 8.3 e752/736 8:c662ba6f:::benchmark_data_smithi173_130566_object35398:head v 752'1323, mlcod=752'1320)", 2024-09-17T16:36:13.627 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:51.618198+0000", 2024-09-17T16:36:13.628 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 81.912847489000001, 2024-09-17T16:36:13.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.163220328, 2024-09-17T16:36:13.628 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.628 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.628 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.628 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:51.618198+0000", 2024-09-17T16:36:13.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:51.618198+0000", 2024-09-17T16:36:13.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:51.618199+0000", 2024-09-17T16:36:13.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1689999999999999e-06 2024-09-17T16:36:13.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:51.618253+0000", 2024-09-17T16:36:13.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3193999999999999e-05 2024-09-17T16:36:13.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:51.780324+0000", 2024-09-17T16:36:13.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.162071466 2024-09-17T16:36:13.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:51.780331+0000", 2024-09-17T16:36:13.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2919999999999998e-06 2024-09-17T16:36:13.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:51.780378+0000", 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6696999999999997e-05 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:51.780393+0000", 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4601e-05 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:51.781363+0000", 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096990100000000001 2024-09-17T16:36:13.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:51.781419+0000", 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6008000000000003e-05 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:54419 8.3 e756/736 8:c6586a74:::benchmark_data_smithi173_130566_object54418:head v 756'1909, mlcod=756'1907)", 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:35:06.395468+0000", 2024-09-17T16:36:13.633 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 67.135577497, 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15362371899999999, 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:06.395468+0000", 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:06.395468+0000", 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:06.395470+0000", 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8959999999999999e-06 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:06.395523+0000", 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2703e-05 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:06.547820+0000", 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15229742600000001 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:06.547828+0000", 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7719999999999994e-06 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:06.547883+0000", 2024-09-17T16:36:13.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4786e-05 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:06.547917+0000", 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3779000000000003e-05 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:06.549050+0000", 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011333529999999999 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:06.549092+0000", 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2004e-05 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:78174 8.3 e762/736 8:c5a7ef5a:::benchmark_data_smithi173_130566_object78173:head v 762'2677, mlcod=762'2674)", 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:35:26.349148+0000", 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 47.181897530999997, 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.175846063, 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:13.638 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:26.349148+0000", 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:26.349148+0000", 2024-09-17T16:36:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:26.349149+0000", 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:26.349212+0000", 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3331999999999996e-05 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:26.523668+0000", 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17445534200000001 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:26.523675+0000", 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3409999999999997e-06 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:26.523730+0000", 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5238000000000002e-05 2024-09-17T16:36:13.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:26.523746+0000", 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5432999999999999e-05 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:26.524958+0000", 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012127780000000001 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:26.524994+0000", 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5809000000000002e-05 2024-09-17T16:36:13.642 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.643 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.643 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.643 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:13.643 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:13.643 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:36:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:13.644 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:13.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:13.746 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:13.767 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:13.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:13.804 INFO:teuthology.orchestra.run.smithi071.stdout:8.4 2024-09-17T16:36:13.804 INFO:teuthology.orchestra.run.smithi071.stdout:8.19 2024-09-17T16:36:13.804 INFO:teuthology.orchestra.run.smithi071.stdout:8.18 2024-09-17T16:36:13.804 INFO:teuthology.orchestra.run.smithi071.stdout:8.16 2024-09-17T16:36:13.804 INFO:teuthology.orchestra.run.smithi071.stdout:8.11 2024-09-17T16:36:13.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:13.891 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:13.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:13.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:14.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:14.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:14.084 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:36:14.084 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:36:14.105 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:36:14.105 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:36:14.118 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:14.126 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:36:14.126 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:36:14.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:14.142 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:14.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:14.147 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:36:14.147 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:36:14.168 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:36:14.169 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:36:14.169 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:36:14.190 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:36:14.190 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:36:14.212 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:36:14.213 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:36:14.234 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:36:14.235 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:36:14.235 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:14 smithi017 ceph-mon[27571]: 8.d scrub starts 2024-09-17T16:36:14.235 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1631071442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:14.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:14.256 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:36:14.256 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:36:14.256 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:36:14.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:14.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:14.279 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:36:14.280 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:36:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:14 smithi173 ceph-mon[30530]: 8.d scrub starts 2024-09-17T16:36:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1631071442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:14.351 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:36:14.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:14.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:14.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:14.489 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:14 smithi071 ceph-mon[30002]: 8.d scrub starts 2024-09-17T16:36:14.490 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1631071442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:14.490 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:14.513 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:14.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:14.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:14.637 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:14.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:14.739 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:14.760 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:14.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:14.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:36:14.862 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:14.883 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:14.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:14.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:14.915 INFO:tasks.thrashosds.thrasher:Moving pg 8.7 from osd.10 to osd.5 2024-09-17T16:36:14.915 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@smithi173.front.sepia.ceph.com to ubuntu@smithi071.front.sepia.ceph.com 2024-09-17T16:36:14.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /var/log/ceph 2024-09-17T16:36:14.942 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2024-09-17T16:36:14.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:14.991 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-09-17T16:36:15.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:15.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:15.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:15.022 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.abVuoqMi6G 2024-09-17T16:36:15.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/exp.8.7.10 /tmp/tmp.abVuoqMi6G 2024-09-17T16:36:15.032 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:36:15.109 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:15.132 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:15.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:15.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.abVuoqMi6G 2024-09-17T16:36:15.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:15.251 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.abVuoqMi6G is 15MB 2024-09-17T16:36:15.256 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:15.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:15.332 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:36:15.333 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:36:15.369 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:15.407 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:15 smithi173 ceph-mon[30530]: pgmap v1744: 27 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:15.407 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:15 smithi173 ceph-mon[30530]: 8.d scrub ok 2024-09-17T16:36:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:15.409 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:15.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:15.448 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:15.483 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:15 smithi071 ceph-mon[30002]: pgmap v1744: 27 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:15.484 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:15 smithi071 ceph-mon[30002]: 8.d scrub ok 2024-09-17T16:36:15.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:15 smithi017 ceph-mon[27571]: pgmap v1744: 27 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:15.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:15 smithi017 ceph-mon[27571]: 8.d scrub ok 2024-09-17T16:36:15.492 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:36:15.492 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:36:15.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:15.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:15.588 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:15.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:15.689 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.abVuoqMi6G 2024-09-17T16:36:15.689 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:15.712 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:15.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:15.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:15.864 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:15.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:15.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:16.043 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:16.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:16.092 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv /var/log/ceph/exp.8.7.10 /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/exp.8.7.10 2024-09-17T16:36:16.130 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -n osd.5 --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.8.7.10 2024-09-17T16:36:16.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:16.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:16.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:16.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:16.293 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:16.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:16.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:16 smithi071 ceph-mon[30002]: 8.f scrub starts 2024-09-17T16:36:16.395 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:16.418 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:16.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:16.459 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/osd.5/config 2024-09-17T16:36:16.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:36:16.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:16 smithi017 ceph-mon[27571]: 8.f scrub starts 2024-09-17T16:36:16.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:16.542 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:16.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:16.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:16 smithi173 ceph-mon[30530]: 8.f scrub starts 2024-09-17T16:36:16.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:16.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:16.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:16.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:16.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:16.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:16.913 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:16.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:17.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:17.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:17.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:17.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:17.138 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:17.152 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:36:17.164 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:17.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:17.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:17.290 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:17.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:17.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:17.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:36:17.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:17.414 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:17.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:17.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:17 smithi017 ceph-mon[27571]: pgmap v1745: 27 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3814776862' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:36:17.516 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:17.538 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:17 smithi071 ceph-mon[30002]: pgmap v1745: 27 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:17.538 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3814776862' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:36:17.539 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:17.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:17.552 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:36:17.588 INFO:teuthology.orchestra.run.smithi071.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-17T16:36:17.640 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:17 smithi173 ceph-mon[30530]: pgmap v1745: 27 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3814776862' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:36:17.662 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:17.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:17.764 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:17.786 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:17.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:17.888 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:17.910 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:17.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:18.012 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:18.020 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:18.034 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:18.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:18.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:18.158 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:18.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:18 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3814776862' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:36:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:18 smithi017 ceph-mon[27571]: osdmap e783: 12 total, 10 up, 12 in 2024-09-17T16:36:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:18 smithi017 ceph-mon[27571]: 8.f scrub ok 2024-09-17T16:36:18.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:18.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:18.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:18.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:18.293 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:36:18.293 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:36:18.388 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:18.410 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:18.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:18.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:18.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:18.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:18.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:18 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:18.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3814776862' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:36:18.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:18 smithi071 ceph-mon[30002]: osdmap e783: 12 total, 10 up, 12 in 2024-09-17T16:36:18.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:18 smithi071 ceph-mon[30002]: 8.f scrub ok 2024-09-17T16:36:18.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:18 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3814776862' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:36:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:18 smithi173 ceph-mon[30530]: osdmap e783: 12 total, 10 up, 12 in 2024-09-17T16:36:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:18 smithi173 ceph-mon[30530]: 8.f scrub ok 2024-09-17T16:36:18.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:18.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:18.768 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:18.790 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:18.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:18.892 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:36:18.893 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:18.916 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:18.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:19.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:19.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:19.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:19.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:19.140 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:36:19.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:19.169 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:19.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:19.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:19.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:19.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:19.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:19.362 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:36:19.362 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:36:19.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:19.425 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:19.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:19.468 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:19 smithi071 ceph-mon[30002]: pgmap v1747: 27 pgs: 1 active+undersized+remapped, 2 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 11 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:19.468 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2601607819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:19.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:19 smithi017 ceph-mon[27571]: pgmap v1747: 27 pgs: 1 active+undersized+remapped, 2 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 11 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:19.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2601607819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:19.527 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:19.551 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:19.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:19.570 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:36:19.570 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:36:19.588 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:36:19.588 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:36:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:19 smithi173 ceph-mon[30530]: pgmap v1747: 27 pgs: 1 active+undersized+remapped, 2 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 11 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2601607819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:19.653 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:19.680 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:19.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:19.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:19.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:19.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:19.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:19.811 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:36:19.812 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:36:19.833 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:36:19.833 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:36:19.855 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:36:19.856 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:36:19.856 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:36:19.878 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:36:19.878 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:36:19.899 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:36:19.900 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:36:19.910 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:19.922 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:36:19.922 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:36:19.933 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:19.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:19.944 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:36:19.944 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:36:19.945 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:36:19.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:36:19.967 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:36:19.968 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:36:20.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:20.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:20.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:20.163 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:20.185 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:20.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:20.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:36:20.288 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:20.311 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:20.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:20.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:20.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:20.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:20.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:20.537 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:20.559 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:20.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:20.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:20.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:20.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:20.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:20.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:20.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:36:20.890 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:36:20.920 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:20.924 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:20.961 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:20.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:21.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:21.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:21.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:21.198 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:21.229 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:21.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:21.330 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:21.358 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:21.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:21.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:21.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:21.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:21 smithi017 ceph-mon[27571]: pgmap v1748: 27 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:21.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:21 smithi071 ceph-mon[30002]: pgmap v1748: 27 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:21.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:21.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:21.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:21.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:21 smithi173 ceph-mon[30530]: pgmap v1748: 27 pgs: 1 active+undersized+remapped, 1 active+clean+scrubbing, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 12 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:21.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:21.736 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:21.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:21.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:21.861 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:21.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:21.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:21.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:21.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:22.059 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:22.092 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:22.115 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:22.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:22.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:22.221 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-17T16:36:22.218+0000 7f25fe8df640 0 --2- 172.21.15.17:0/3018655677 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f25f8108030 0x7f25f810e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T16:36:22.241 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:22.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:22.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:22.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:22.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:22.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:22.472 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:22.493 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:22.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:22.564 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:36:22.564 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:36:22.564 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:36:22.564 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:36:22.591 INFO:teuthology.orchestra.run.smithi071.stdout:get_pg_num_history pg_num_history pg_num_history(e772 pg_nums {1={19=1},7={674=26},8={728=16,736=26}} deleted_pools 726,7) 2024-09-17T16:36:22.595 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:22.598 INFO:teuthology.orchestra.run.smithi071.stdout:Importing pgid 8.7 2024-09-17T16:36:22.617 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:22.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:22.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:22.628 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e013f008:::benchmark_data_smithi173_130566_object2492:head# 2024-09-17T16:36:22.628 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.630 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e02436fb:::benchmark_data_smithi173_130566_object3392:head# 2024-09-17T16:36:22.630 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.630 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e02a5e53:::benchmark_data_smithi173_130566_object4279:head# 2024-09-17T16:36:22.630 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e02b4741:::benchmark_data_smithi173_130566_object4826:head# 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e02ee428:::benchmark_data_smithi173_130566_object5894:head# 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e039a39d:::benchmark_data_smithi173_130566_object6377:head# 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e05561a0:::benchmark_data_smithi173_130566_object6908:head# 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0612afd:::benchmark_data_smithi173_130566_object4432:head# 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e067fd00:::benchmark_data_smithi173_130566_object5984:head# 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.631 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e069acc8:::benchmark_data_smithi173_130566_object6050:head# 2024-09-17T16:36:22.632 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.634 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e06b1702:::benchmark_data_smithi173_130566_object3057:head# 2024-09-17T16:36:22.634 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.634 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e078b797:::benchmark_data_smithi173_130566_object2532:head# 2024-09-17T16:36:22.634 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e086b588:::benchmark_data_smithi173_130566_object443:head# 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0a138f0:::benchmark_data_smithi173_130566_object3292:head# 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0bad6c3:::benchmark_data_smithi173_130566_object1271:head# 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0bdfd05:::benchmark_data_smithi173_130566_object5633:head# 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0caed98:::benchmark_data_smithi173_130566_object3120:head# 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.635 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0cc00f3:::benchmark_data_smithi173_130566_object4858:head# 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0eb3aa5:::benchmark_data_smithi173_130566_object5587:head# 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0f117c7:::benchmark_data_smithi173_130566_object7509:head# 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0f7e64c:::benchmark_data_smithi173_130566_object4557:head# 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0f9d939:::benchmark_data_smithi173_130566_object7243:head# 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0fa5920:::benchmark_data_smithi173_130566_object2047:head# 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e0fdc8f0:::benchmark_data_smithi173_130566_object5888:head# 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.636 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1020f8d:::benchmark_data_smithi173_130566_object2178:head# 2024-09-17T16:36:22.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.637 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1022f9b:::benchmark_data_smithi173_130566_object1484:head# 2024-09-17T16:36:22.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.637 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1069b27:::benchmark_data_smithi173_130566_object2316:head# 2024-09-17T16:36:22.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.637 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e110922f:::benchmark_data_smithi173_130566_object4862:head# 2024-09-17T16:36:22.637 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.647 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e11d4c10:::benchmark_data_smithi173_130566_object7163:head# 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1236541:::benchmark_data_smithi173_130566_object4232:head# 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e133bb03:::benchmark_data_smithi173_130566_object4024:head# 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e13a7b9f:::benchmark_data_smithi173_130566_object80:head# 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e13e1c40:::benchmark_data_smithi173_130566_object6667:head# 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e147b17c:::benchmark_data_smithi173_130566_object5258:head# 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.648 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e149f0d3:::benchmark_data_smithi173_130566_object2165:head# 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e14d87e8:::benchmark_data_smithi173_130566_object1485:head# 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e15058e3:::benchmark_data_smithi173_130566_object6664:head# 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e15b999c:::benchmark_data_smithi173_130566_object5565:head# 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e15e8f66:::benchmark_data_smithi173_130566_object5750:head# 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1691a85:::benchmark_data_smithi173_130566_object6097:head# 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1691ee1:::benchmark_data_smithi173_130566_object1561:head# 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.649 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e171e2ac:::benchmark_data_smithi173_130566_object1510:head# 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e177c881:::benchmark_data_smithi173_130566_object3720:head# 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e188d7dd:::benchmark_data_smithi173_130566_object7964:head# 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e18d2a5d:::benchmark_data_smithi173_130566_object1355:head# 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1997ee6:::benchmark_data_smithi173_130566_object2242:head# 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e19c43de:::benchmark_data_smithi173_130566_object557:head# 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1a1047f:::benchmark_data_smithi173_130566_object7716:head# 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.650 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1a557b0:::benchmark_data_smithi173_130566_object7102:head# 2024-09-17T16:36:22.651 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.651 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1b8cffb:::benchmark_data_smithi173_130566_object7038:head# 2024-09-17T16:36:22.651 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.651 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1b8fe49:::benchmark_data_smithi173_130566_object1481:head# 2024-09-17T16:36:22.651 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.651 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1bd3cf9:::benchmark_data_smithi173_130566_object4075:head# 2024-09-17T16:36:22.651 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.651 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1c0fd10:::benchmark_data_smithi173_130566_object7947:head# 2024-09-17T16:36:22.652 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.652 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1c6ff69:::benchmark_data_smithi173_130566_object924:head# 2024-09-17T16:36:22.652 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.652 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1d6a6ee:::benchmark_data_smithi173_130566_object4825:head# 2024-09-17T16:36:22.652 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.652 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1dc146e:::benchmark_data_smithi173_130566_object755:head# 2024-09-17T16:36:22.652 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.652 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1dcd0be:::benchmark_data_smithi173_130566_object6109:head# 2024-09-17T16:36:22.652 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.652 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1e418f4:::benchmark_data_smithi173_130566_object986:head# 2024-09-17T16:36:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1e5f890:::benchmark_data_smithi173_130566_object7771:head# 2024-09-17T16:36:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1ed7c5c:::benchmark_data_smithi173_130566_object6414:head# 2024-09-17T16:36:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1ee682b:::benchmark_data_smithi173_130566_object2292:head# 2024-09-17T16:36:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e1fe9b48:::benchmark_data_smithi173_130566_object2971:head# 2024-09-17T16:36:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.653 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2109253:::benchmark_data_smithi173_130566_object5734:head# 2024-09-17T16:36:22.654 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.654 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e219e45a:::benchmark_data_smithi173_130566_object6923:head# 2024-09-17T16:36:22.654 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.654 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2284f8c:::benchmark_data_smithi173_130566_object7059:head# 2024-09-17T16:36:22.654 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.654 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e22ad34b:::benchmark_data_smithi173_130566_object49:head# 2024-09-17T16:36:22.654 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.654 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e22d5025:::benchmark_data_smithi173_130566_object1951:head# 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e22f876a:::benchmark_data_smithi173_130566_object5113:head# 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e22f976e:::benchmark_data_smithi173_130566_object5787:head# 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e23796c9:::benchmark_data_smithi173_130566_object6862:head# 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2384b73:::benchmark_data_smithi173_130566_object4972:head# 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e23a4098:::benchmark_data_smithi173_130566_object6443:head# 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e246c764:::benchmark_data_smithi173_130566_object3212:head# 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.655 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e249798b:::benchmark_data_smithi173_130566_object3677:head# 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e24c9a4e:::benchmark_data_smithi173_130566_object2035:head# 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e253df38:::benchmark_data_smithi173_130566_object7224:head# 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e25d3657:::benchmark_data_smithi173_130566_object5311:head# 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2787418:::benchmark_data_smithi173_130566_object7496:head# 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e281c15b:::benchmark_data_smithi173_130566_object322:head# 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2a56fd7:::benchmark_data_smithi173_130566_object7733:head# 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2b4d4b2:::benchmark_data_smithi173_130566_object1527:head# 2024-09-17T16:36:22.656 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2c001f8:::benchmark_data_smithi173_130566_object5386:head# 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2c42bcb:::benchmark_data_smithi173_130566_object1032:head# 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2c7034e:::benchmark_data_smithi173_130566_object5901:head# 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2da8537:::benchmark_data_smithi173_130566_object19:head# 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2e46e22:::benchmark_data_smithi173_130566_object3126:head# 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2eb3c3a:::benchmark_data_smithi173_130566_object7140:head# 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e2f8d8e4:::benchmark_data_smithi173_130566_object1315:head# 2024-09-17T16:36:22.657 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.658 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e301d688:::benchmark_data_smithi173_130566_object972:head# 2024-09-17T16:36:22.658 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.658 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e30a232c:::benchmark_data_smithi173_130566_object4619:head# 2024-09-17T16:36:22.658 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.681 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e31498c6:::benchmark_data_smithi173_130566_object1622:head# 2024-09-17T16:36:22.681 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.681 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e314aceb:::benchmark_data_smithi173_130566_object5246:head# 2024-09-17T16:36:22.681 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.681 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e31984cd:::benchmark_data_smithi173_130566_object7104:head# 2024-09-17T16:36:22.681 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.681 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3236d8f:::benchmark_data_smithi173_130566_object5425:head# 2024-09-17T16:36:22.681 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.681 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e32f974b:::benchmark_data_smithi173_130566_object7481:head# 2024-09-17T16:36:22.681 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e33fc415:::benchmark_data_smithi173_130566_object3199:head# 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e342426b:::benchmark_data_smithi173_130566_object5316:head# 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e345e6df:::benchmark_data_smithi173_130566_object6089:head# 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e34f2152:::benchmark_data_smithi173_130566_object6553:head# 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3637997:::benchmark_data_smithi173_130566_object5916:head# 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e364185b:::benchmark_data_smithi173_130566_object2411:head# 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3824d09:::benchmark_data_smithi173_130566_object4602:head# 2024-09-17T16:36:22.682 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3881772:::benchmark_data_smithi173_130566_object7544:head# 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e38d3755:::benchmark_data_smithi173_130566_object7244:head# 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3a213e6:::benchmark_data_smithi173_130566_object6579:head# 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3a2d1a2:::benchmark_data_smithi173_130566_object3516:head# 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3a95960:::benchmark_data_smithi173_130566_object3767:head# 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3b5a797:::benchmark_data_smithi173_130566_object7201:head# 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3c071a1:::benchmark_data_smithi173_130566_object2806:head# 2024-09-17T16:36:22.683 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3c4e113:::benchmark_data_smithi173_130566_object6439:head# 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3cdf39a:::benchmark_data_smithi173_130566_object5172:head# 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3d2e447:::benchmark_data_smithi173_130566_object6223:head# 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3efe9ed:::benchmark_data_smithi173_130566_object7127:head# 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3f15394:::benchmark_data_smithi173_130566_object7866:head# 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e3faa680:::benchmark_data_smithi173_130566_object4415:head# 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e401c53e:::benchmark_data_smithi173_130566_object3573:head# 2024-09-17T16:36:22.684 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e41d24ac:::benchmark_data_smithi173_130566_object4848:head# 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4246fe4:::benchmark_data_smithi173_130566_object2498:head# 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e426e512:::benchmark_data_smithi173_130566_object3772:head# 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e42d0498:::benchmark_data_smithi173_130566_object525:head# 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e434b224:::benchmark_data_smithi173_130566_object1160:head# 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4381f3f:::benchmark_data_smithi173_130566_object5996:head# 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e443d89f:::benchmark_data_smithi173_130566_object5185:head# 2024-09-17T16:36:22.685 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e451b21d:::benchmark_data_smithi173_130566_object6601:head# 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4550051:::benchmark_data_smithi173_130566_object276:head# 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4604412:::benchmark_data_smithi173_130566_object4033:head# 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4654205:::benchmark_data_smithi173_130566_object2069:head# 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4660ad6:::benchmark_data_smithi173_130566_object3269:head# 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.686 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e46a9c0b:::benchmark_data_smithi173_130566_object3193:head# 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4760a33:::benchmark_data_smithi173_130566_object1230:head# 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e476670b:::benchmark_data_smithi173_130566_object4151:head# 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4777771:::benchmark_data_smithi173_130566_object394:head# 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e47fa40b:::benchmark_data_smithi173_130566_object7501:head# 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e493d931:::benchmark_data_smithi173_130566_object6583:head# 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4979842:::benchmark_data_smithi173_130566_object6720:head# 2024-09-17T16:36:22.687 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.688 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4ad4c7b:::benchmark_data_smithi173_130566_object3297:head# 2024-09-17T16:36:22.688 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.688 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4b2d15e:::benchmark_data_smithi173_130566_object3188:head# 2024-09-17T16:36:22.688 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.688 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4c154cd:::benchmark_data_smithi173_130566_object7768:head# 2024-09-17T16:36:22.688 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.688 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4c75e95:::benchmark_data_smithi173_130566_object6418:head# 2024-09-17T16:36:22.688 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4cb7016:::benchmark_data_smithi173_130566_object135:head# 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4ceb052:::benchmark_data_smithi173_130566_object3410:head# 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4d85f2a:::benchmark_data_smithi173_130566_object1049:head# 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4e3bf60:::benchmark_data_smithi173_130566_object6938:head# 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4e86e25:::benchmark_data_smithi173_130566_object6110:head# 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4e8c708:::benchmark_data_smithi173_130566_object3590:head# 2024-09-17T16:36:22.689 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4ec5dd2:::benchmark_data_smithi173_130566_object4306:head# 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e4f1c430:::benchmark_data_smithi173_130566_object5780:head# 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e50deee5:::benchmark_data_smithi173_130566_object3771:head# 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e50f7fb0:::benchmark_data_smithi173_130566_object5183:head# 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e515c860:::benchmark_data_smithi173_130566_object2366:head# 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.690 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e51655bb:::benchmark_data_smithi173_130566_object2832:head# 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5168b2a:::benchmark_data_smithi173_130566_object2271:head# 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e53ac44e:::benchmark_data_smithi173_130566_object2649:head# 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5447035:::benchmark_data_smithi173_130566_object398:head# 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e54c0118:::benchmark_data_smithi173_130566_object1479:head# 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5515e32:::benchmark_data_smithi173_130566_object3809:head# 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e55c5e26:::benchmark_data_smithi173_130566_object6657:head# 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.691 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e55f7180:::benchmark_data_smithi173_130566_object3689:head# 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e562eb35:::benchmark_data_smithi173_130566_object1252:head# 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5660350:::benchmark_data_smithi173_130566_object6837:head# 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e58d7961:::benchmark_data_smithi173_130566_object2314:head# 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e590f721:::benchmark_data_smithi173_130566_object781:head# 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5928211:::benchmark_data_smithi173_130566_object4997:head# 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e59a351b:::benchmark_data_smithi173_130566_object3794:head# 2024-09-17T16:36:22.692 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e59b39db:::benchmark_data_smithi173_130566_object3351:head# 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5a4e826:::benchmark_data_smithi173_130566_object2380:head# 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5a6e395:::benchmark_data_smithi173_130566_object3216:head# 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5aa3a87:::benchmark_data_smithi173_130566_object5151:head# 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5aaa26f:::benchmark_data_smithi173_130566_object4414:head# 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5b9ddeb:::benchmark_data_smithi173_130566_object6128:head# 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.693 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5c379ac:::benchmark_data_smithi173_130566_object3734:head# 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5c9cc81:::benchmark_data_smithi173_130566_object205:head# 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5d848f4:::benchmark_data_smithi173_130566_object7188:head# 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5dae115:::benchmark_data_smithi173_130566_object3844:head# 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5dff33a:::benchmark_data_smithi173_130566_object534:head# 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5eaba9b:::benchmark_data_smithi173_130566_object4289:head# 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5ee483e:::benchmark_data_smithi173_130566_object4699:head# 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.694 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e5ff1dfc:::benchmark_data_smithi173_130566_object6813:head# 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e60932c6:::benchmark_data_smithi173_130566_object2678:head# 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e60cca13:::benchmark_data_smithi173_130566_object6211:head# 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e612e5d6:::benchmark_data_smithi173_130566_object1933:head# 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e61860d3:::benchmark_data_smithi173_130566_object5597:head# 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e61a6da4:::benchmark_data_smithi173_130566_object1782:head# 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.695 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e62a11f9:::benchmark_data_smithi173_130566_object1352:head# 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6391825:::benchmark_data_smithi173_130566_object3070:head# 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e63fb09e:::benchmark_data_smithi173_130566_object4515:head# 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e643712e:::benchmark_data_smithi173_130566_object442:head# 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e646da6c:::benchmark_data_smithi173_130566_object4814:head# 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6520bc8:::benchmark_data_smithi173_130566_object5689:head# 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.696 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6587638:::benchmark_data_smithi173_130566_object6605:head# 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e664f82f:::benchmark_data_smithi173_130566_object7725:head# 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6786f7a:::benchmark_data_smithi173_130566_object4093:head# 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e67b2e3a:::benchmark_data_smithi173_130566_object192:head# 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e67bbd7c:::benchmark_data_smithi173_130566_object4864:head# 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6911040:::benchmark_data_smithi173_130566_object7245:head# 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6939b5b:::benchmark_data_smithi173_130566_object6802:head# 2024-09-17T16:36:22.697 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6a0de3c:::benchmark_data_smithi173_130566_object751:head# 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6b64760:::benchmark_data_smithi173_130566_object2230:head# 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6c0ff5c:::benchmark_data_smithi173_130566_object6297:head# 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6dbf7ff:::benchmark_data_smithi173_130566_object5740:head# 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6debbd7:::benchmark_data_smithi173_130566_object1828:head# 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6e4e99b:::benchmark_data_smithi173_130566_object1512:head# 2024-09-17T16:36:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6edcbe0:::benchmark_data_smithi173_130566_object670:head# 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6eea838:::benchmark_data_smithi173_130566_object1851:head# 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6ef97f5:::benchmark_data_smithi173_130566_object3313:head# 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6f44ece:::benchmark_data_smithi173_130566_object2121:head# 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6f55464:::benchmark_data_smithi173_130566_object7442:head# 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.699 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e6ff1ad0:::benchmark_data_smithi173_130566_object6200:head# 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e70066fc:::benchmark_data_smithi173_130566_object6480:head# 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7010a58:::benchmark_data_smithi173_130566_object2295:head# 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7050b9f:::benchmark_data_smithi173_130566_object956:head# 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e707218a:::benchmark_data_smithi173_130566_object2936:head# 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7122980:::benchmark_data_smithi173_130566_object4488:head# 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.700 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e73050d7:::benchmark_data_smithi173_130566_object4534:head# 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7352532:::benchmark_data_smithi173_130566_object4346:head# 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7440a17:::benchmark_data_smithi173_130566_object2775:head# 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7458ab3:::benchmark_data_smithi173_130566_object3954:head# 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e746d936:::benchmark_data_smithi173_130566_object4318:head# 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e74857ab:::benchmark_data_smithi173_130566_object7332:head# 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.701 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e748e0d7:::benchmark_data_smithi173_130566_object2615:head# 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7509560:::benchmark_data_smithi173_130566_object3309:head# 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e75891df:::benchmark_data_smithi173_130566_object6898:head# 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e763011e:::benchmark_data_smithi173_130566_object5399:head# 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e766a366:::benchmark_data_smithi173_130566_object1013:head# 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e76d72ad:::benchmark_data_smithi173_130566_object701:head# 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e76e20bf:::benchmark_data_smithi173_130566_object570:head# 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e76ef68b:::benchmark_data_smithi173_130566_object4775:head# 2024-09-17T16:36:22.702 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e76f5c2b:::benchmark_data_smithi173_130566_object5936:head# 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7750079:::benchmark_data_smithi173_130566_object3592:head# 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e78beab2:::benchmark_data_smithi173_130566_object1193:head# 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e78c4cda:::benchmark_data_smithi173_130566_object2176:head# 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e79c7a5d:::benchmark_data_smithi173_130566_object7901:head# 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7a6e4aa:::benchmark_data_smithi173_130566_object5158:head# 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7af7a73:::benchmark_data_smithi173_130566_object6606:head# 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7c199cf:::benchmark_data_smithi173_130566_object7577:head# 2024-09-17T16:36:22.703 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7cae102:::benchmark_data_smithi173_130566_object7893:head# 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7e06317:::benchmark_data_smithi173_130566_object2961:head# 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7e383ce:::benchmark_data_smithi173_130566_object2219:head# 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7e7772d:::benchmark_data_smithi173_130566_object6627:head# 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7e9f195:::benchmark_data_smithi173_130566_object6901:head# 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7ef617b:::benchmark_data_smithi173_130566_object3014:head# 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:Write #8:e7ff4392:::benchmark_data_smithi173_130566_object3242:head# 2024-09-17T16:36:22.704 INFO:teuthology.orchestra.run.smithi071.stdout:snapset 0={} 2024-09-17T16:36:22.705 INFO:teuthology.orchestra.run.smithi071.stdout:write_pg epoch 739 info 8.7( v 733'506 (0'0,733'506] local-lis/les=736/737 n=253 ec=728/728 lis/c=736/736 les/c/f=737/737/0 sis=736) 2024-09-17T16:36:22.705 INFO:teuthology.orchestra.run.smithi071.stdout:Import successful 2024-09-17T16:36:22.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:22.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:22.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:22.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:22.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:22.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:22.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:36:22.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:23.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:23.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:23.103 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:23.124 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:23.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:23.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:23.164 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:36:23.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:23.248 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:23.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:23.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:23.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:36:23.350 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:23.375 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:23.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:23.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:23 smithi017 ceph-mon[27571]: pgmap v1749: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/747726508' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:23.500 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:23.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:23.524 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:23 smithi071 ceph-mon[30002]: pgmap v1749: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:23.525 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/747726508' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:23.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:23.624 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:23.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:23 smithi173 ceph-mon[30530]: pgmap v1749: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/747726508' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:23.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:23.748 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:23.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:23.839 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/exp.8.7.10 2024-09-17T16:36:23.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:23.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/exp.8.7.10 2024-09-17T16:36:23.873 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:23.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:23.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:24.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:24.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:24.016 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:24.019 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:24.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:24.128 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:24.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:24.231 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:24.254 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:24.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:24.356 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:24.378 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:24.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:24.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:24.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:24 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:24.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/747726508' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:24.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:24 smithi017 ceph-mon[27571]: osdmap e784: 12 total, 10 up, 12 in 2024-09-17T16:36:24.480 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:24.502 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:24.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:24.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:24.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:24 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:24.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/747726508' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:24.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:24 smithi071 ceph-mon[30002]: osdmap e784: 12 total, 10 up, 12 in 2024-09-17T16:36:24.599 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:36:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:36:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:36:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:36:24.599 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:36:24.604 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:24.625 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:24.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:24 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/747726508' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:24 smithi173 ceph-mon[30530]: osdmap e784: 12 total, 10 up, 12 in 2024-09-17T16:36:24.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:24.752 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:24.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:24.853 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:24.878 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:24.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:24.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:25.002 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:25.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:25.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:25.127 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:25.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:25.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:25.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:36:25.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:25.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:25.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:25.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:25.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:25.484 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:25 smithi017 ceph-mon[27571]: pgmap v1751: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:25.509 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:25.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:25.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:25 smithi071 ceph-mon[30002]: pgmap v1751: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:25.612 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:25.635 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:25.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:25 smithi173 ceph-mon[30530]: pgmap v1751: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:25.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:25.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:25.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:25.784 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:25.786 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:36:25.787 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:36:25.810 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:36:25.811 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:36:25.831 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:36:25.831 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:36:25.854 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:36:25.854 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:36:25.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:25.876 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:36:25.876 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:36:25.877 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:36:25.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:25.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:25.899 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:36:25.899 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:36:25.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:36:25.921 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:36:25.921 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:36:25.943 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:36:25.944 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:36:25.965 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:36:25.966 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:36:25.966 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:36:25.987 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:25.988 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:36:25.989 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:36:26.011 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:26.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:26.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:26.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:26.140 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:26.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:26.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:26.266 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:26.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:26.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:26.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4017469221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:26.393 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:26.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:26.471 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:36:26.471 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:36:26.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:26.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:26.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:26.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:26.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4017469221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:26.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4017469221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:26.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:26.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:26.747 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:26.770 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:26.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:26.872 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:26.895 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:26.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:26.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:26.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:27.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:27.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:27.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:27.149 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:27.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:27.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:27.277 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:27.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:27.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:27.405 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:27.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:27.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:27.432 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:27 smithi017 ceph-mon[27571]: pgmap v1752: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:27.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:27.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:27.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:27 smithi071 ceph-mon[30002]: pgmap v1752: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:27.635 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:27 smithi173 ceph-mon[30530]: pgmap v1752: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:27.662 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:27.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:27.764 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:27.770 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:36:27.790 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:27.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:27.892 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:27.917 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:27.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:27.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:28.019 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:28.043 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:28.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:28.145 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:28.169 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:28.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:36:28.183 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:36:28.196 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:28.224 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:28.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:28.293 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:28.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:28.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:28.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:28.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:28.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:28.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:28.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:28.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:28.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:28.674 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:28.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:28.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:28.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:28.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:28.903 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [] dead_osds: [10, 5] live_osds: [0, 1, 2, 3, 4, 6, 8, 9, 11, 7] 2024-09-17T16:36:28.904 INFO:tasks.thrashosds.thrasher:Reviving osd 10 2024-09-17T16:36:28.904 INFO:tasks.cephadm.osd.10:Restarting osd.10 (starting--it wasn't running)... 2024-09-17T16:36:28.904 DEBUG:teuthology.orchestra.run.smithi173:osd.10> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.10.service 2024-09-17T16:36:28.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:28.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl start ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.10 2024-09-17T16:36:28.927 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:28.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:29.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:29.032 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:36:29.032 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:36:29.055 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:29.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:29.157 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:29.182 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:29.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:29.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:29.185 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:28 smithi173 systemd[1]: Starting Ceph osd.10 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:36:29.284 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:29.310 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:29.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:29.351 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:29 smithi017 ceph-mon[27571]: pgmap v1753: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:29.412 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:29.438 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:29.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:29.451 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:29 smithi173 ceph-mon[30530]: pgmap v1753: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:29.452 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:29 smithi173 podman[141418]: 2024-09-17 16:36:29.121542134 +0000 UTC m=+0.015972929 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:36:29.452 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:29 smithi173 podman[141418]: 2024-09-17 16:36:29.22615219 +0000 UTC m=+0.120582968 container create 919647968b19c131e6af730a2484e65e6183adda982f5b69d68cc65dc027ddaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T16:36:29.452 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:29 smithi173 podman[141418]: 2024-09-17 16:36:29.450473282 +0000 UTC m=+0.344904064 container init 919647968b19c131e6af730a2484e65e6183adda982f5b69d68cc65dc027ddaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T16:36:29.540 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:29 smithi071 ceph-mon[30002]: pgmap v1753: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:29.565 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:29.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:29.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:29.690 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:29.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:29.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:29.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:29.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:29.818 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:29 smithi173 podman[141418]: 2024-09-17 16:36:29.453965086 +0000 UTC m=+0.348395864 container start 919647968b19c131e6af730a2484e65e6183adda982f5b69d68cc65dc027ddaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16) 2024-09-17T16:36:29.818 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:29 smithi173 podman[141418]: 2024-09-17 16:36:29.533873788 +0000 UTC m=+0.428304567 container attach 919647968b19c131e6af730a2484e65e6183adda982f5b69d68cc65dc027ddaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T16:36:29.818 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:29 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate[141434]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:36:29.818 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:29 smithi173 bash[141418]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:36:29.818 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:29 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate[141434]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:36:29.818 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:29 smithi173 bash[141418]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:36:29.918 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:29.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:36:29.941 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:29.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:29.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:30.042 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:30.069 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:30.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:30.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate[141434]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-17T16:36:30.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 bash[141418]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-17T16:36:30.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate[141434]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-10 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T16:36:30.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 bash[141418]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-10 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T16:36:30.171 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:30.184 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:36:30.191 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:36:30.199 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:30.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:30.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:30.205 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:30.301 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:30.324 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:30.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:30.427 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:30.450 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:30.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:30.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:30 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:30 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:30 smithi017 ceph-mon[27571]: osdmap e785: 12 total, 10 up, 12 in 2024-09-17T16:36:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3640879079' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:36:30.551 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:30.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:30 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:30 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:30 smithi071 ceph-mon[30002]: osdmap e785: 12 total, 10 up, 12 in 2024-09-17T16:36:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3640879079' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:36:30.577 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:30.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-mon[30530]: osdmap e785: 12 total, 10 up, 12 in 2024-09-17T16:36:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3640879079' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:36:30.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate[141434]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T16:36:30.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 bash[141418]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T16:36:30.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate[141434]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T16:36:30.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 bash[141418]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T16:36:30.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate[141434]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-10/block 2024-09-17T16:36:30.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 bash[141418]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-10/block 2024-09-17T16:36:30.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate[141434]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-17T16:36:30.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 bash[141418]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-17T16:36:30.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate[141434]: --> ceph-volume raw activate successful for osd ID: 10 2024-09-17T16:36:30.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 bash[141418]: --> ceph-volume raw activate successful for osd ID: 10 2024-09-17T16:36:30.647 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:30 smithi173 podman[141418]: 2024-09-17 16:36:30.392750139 +0000 UTC m=+1.287180920 container died 919647968b19c131e6af730a2484e65e6183adda982f5b69d68cc65dc027ddaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True) 2024-09-17T16:36:30.679 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:30.702 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:30.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:30.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:30.805 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:30.827 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:30.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:30.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:30.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:30.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:31.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:31.081 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:31.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:31.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:31.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:31.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:31.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:31.298 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:31 smithi173 podman[141595]: 2024-09-17 16:36:31.062835916 +0000 UTC m=+0.659723916 container remove 919647968b19c131e6af730a2484e65e6183adda982f5b69d68cc65dc027ddaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-activate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:36:31.299 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:31 smithi173 podman[141672]: 2024-09-17 16:36:31.171411017 +0000 UTC m=+0.016179687 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:36:31.299 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:31 smithi173 podman[141672]: 2024-09-17 16:36:31.297112012 +0000 UTC m=+0.141880683 container create e338f206d52c72e1fc2370b12ff67b2f97256f11cd12cb03d56a25d222b5a05d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-17T16:36:31.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:31 smithi173 ceph-mon[30530]: pgmap v1755: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:31.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:31 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:31.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3640879079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:36:31.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:31 smithi173 ceph-mon[30530]: osdmap e786: 12 total, 10 up, 12 in 2024-09-17T16:36:31.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:31 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:31.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:31 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:31.307 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:31.333 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:31.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:31.384 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:36:31.385 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:36:31.385 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:31.395 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:31 smithi017 ceph-mon[27571]: pgmap v1755: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:31.395 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:31 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:31.395 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3640879079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:36:31.395 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:31 smithi017 ceph-mon[27571]: osdmap e786: 12 total, 10 up, 12 in 2024-09-17T16:36:31.396 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:31 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:31.396 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:31 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:31.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:31.457 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:31.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:31.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:31.562 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:31 smithi173 podman[141672]: 2024-09-17 16:36:31.533264821 +0000 UTC m=+0.378033492 container init e338f206d52c72e1fc2370b12ff67b2f97256f11cd12cb03d56a25d222b5a05d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-09-17T16:36:31.562 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:31 smithi173 podman[141672]: 2024-09-17 16:36:31.536549463 +0000 UTC m=+0.381318135 container start e338f206d52c72e1fc2370b12ff67b2f97256f11cd12cb03d56a25d222b5a05d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-17T16:36:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:31 smithi071 ceph-mon[30002]: pgmap v1755: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:31 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3640879079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:36:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:31 smithi071 ceph-mon[30002]: osdmap e786: 12 total, 10 up, 12 in 2024-09-17T16:36:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:31 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:31 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:31.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:31.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:31.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:36:31.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:31.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:31.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:31.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:31.827 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:31.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:31.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:31 smithi173 ceph-osd[141708]: -- 172.21.15.173:0/3763442841 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b955373a40 con 0x55b956142000 2024-09-17T16:36:31.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:31 smithi173 bash[141672]: e338f206d52c72e1fc2370b12ff67b2f97256f11cd12cb03d56a25d222b5a05d 2024-09-17T16:36:31.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:31 smithi173 systemd[1]: Started Ceph osd.10 for 19341aec-750e-11ef-bceb-c7b262605968. 2024-09-17T16:36:31.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:31.932 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:36:31.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:31.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:31.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:32.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:32.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:32.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:32.176 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:32.202 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:32.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:32.304 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:32.330 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:32.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:36:32.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:32.338 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:32 smithi173 ceph-mon[30530]: osdmap e787: 12 total, 10 up, 12 in 2024-09-17T16:36:32.432 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:32.434 INFO:teuthology.orchestra.run.smithi173.stderr:no valid command found; 10 closest matches: 2024-09-17T16:36:32.434 INFO:teuthology.orchestra.run.smithi173.stderr:0 2024-09-17T16:36:32.434 INFO:teuthology.orchestra.run.smithi173.stderr:1 2024-09-17T16:36:32.434 INFO:teuthology.orchestra.run.smithi173.stderr:2 2024-09-17T16:36:32.434 INFO:teuthology.orchestra.run.smithi173.stderr:abort 2024-09-17T16:36:32.434 INFO:teuthology.orchestra.run.smithi173.stderr:assert 2024-09-17T16:36:32.435 INFO:teuthology.orchestra.run.smithi173.stderr:config diff 2024-09-17T16:36:32.435 INFO:teuthology.orchestra.run.smithi173.stderr:config diff get 2024-09-17T16:36:32.435 INFO:teuthology.orchestra.run.smithi173.stderr:config get 2024-09-17T16:36:32.435 INFO:teuthology.orchestra.run.smithi173.stderr:config help [] 2024-09-17T16:36:32.435 INFO:teuthology.orchestra.run.smithi173.stderr:config set ... 2024-09-17T16:36:32.435 INFO:teuthology.orchestra.run.smithi173.stderr:admin_socket: invalid command 2024-09-17T16:36:32.457 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:32.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:32.470 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:32 smithi017 ceph-mon[27571]: osdmap e787: 12 total, 10 up, 12 in 2024-09-17T16:36:32.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:32 smithi071 ceph-mon[30002]: osdmap e787: 12 total, 10 up, 12 in 2024-09-17T16:36:32.582 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:36:32.583 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:36:32.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:32.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:32.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:32.606 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:36:32.606 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:36:32.621 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:32.627 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:36:32.627 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:36:32.649 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:36:32.649 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:36:32.671 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:36:32.672 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:36:32.672 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:36:32.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:32.694 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:36:32.694 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:36:32.711 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:32.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:32.716 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:36:32.716 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:36:32.737 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:36:32.737 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:36:32.758 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:36:32.759 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:36:32.759 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:36:32.779 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:36:32.780 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:36:32.813 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:32.835 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:32.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:32.862 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:36:32.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:32.961 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:32.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:32.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:33.063 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:33.087 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:33.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:36:33.127 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-10, ['dump_ops_in_flight'] 2024-09-17T16:36:33.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:33.215 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:33.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:33.317 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:33.339 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:33.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:33.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:33.342 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:36:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:36:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:36:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:36:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90591 6.4 6.c1264da4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-09-17T16:36:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:33.605018+0000", 2024-09-17T16:36:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 599.73267282999996, 2024-09-17T16:36:33.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.91834311899999999, 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90591 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605018+0000", 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605018+0000", 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605020+0000", 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7099999999999999e-06 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605094+0000", 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4455000000000005e-05 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605104+0000", 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0220999999999999e-05 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605110+0000", 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7640000000000002e-06 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605166+0000", 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6301999999999997e-05 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.605174+0000", 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2169999999999997e-06 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.523331+0000", 2024-09-17T16:36:33.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.91815760300000004 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.523361+0000", 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9847000000000001e-05 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90612 6.4 6.5626e774 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:33.747552+0000", 2024-09-17T16:36:33.347 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 599.59013882099998, 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77617998600000004, 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90612 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747552+0000", 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747552+0000", 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747554+0000", 2024-09-17T16:36:33.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.66e-06 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747600+0000", 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5973000000000002e-05 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747616+0000", 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6330000000000001e-05 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747624+0000", 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.61e-06 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747684+0000", 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730999999999998e-05 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:33.747693+0000", 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3969999999999994e-06 2024-09-17T16:36:33.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.523723+0000", 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77603073499999997 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:34.523732+0000", 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5499999999999995e-06 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90591 6.4 6:25b26483:::benchmark_data_smithi173_102017_object90590:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e568)", 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:35.553828+0000", 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 597.78386248300001, 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44443183400000003, 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90591 2024-09-17T16:36:33.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553828+0000", 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553828+0000", 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553830+0000", 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1670000000000002e-06 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553964+0000", 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000133585 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553985+0000", 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0899000000000001e-05 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.553999+0000", 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4032000000000001e-05 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.557594+0000", 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035952169999999999 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.557612+0000", 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7753000000000001e-05 2024-09-17T16:36:33.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.975806+0000", 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.41819402700000002 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.975964+0000", 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015768999999999999 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,5", 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.976138+0000", 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017470499999999999 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.986303+0000", 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010164289 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.986584+0000", 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00028114299999999997 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.993814+0000", 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072301520000000001 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.993966+0000", 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015189000000000001 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.998260+0000", 2024-09-17T16:36:33.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042942850000000001 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:90612 6.4 6:2ee7646a:::benchmark_data_smithi173_102017_object90611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e568)", 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:26:35.554030+0000", 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 597.78366069000003, 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44921810899999998, 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.360 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 90612 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554030+0000", 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554030+0000", 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554031+0000", 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3200000000000005e-07 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554063+0000", 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1903999999999998e-05 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554072+0000", 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9310000000000003e-06 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.554079+0000", 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1290000000000001e-06 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.557639+0000", 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035608100000000002 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.557646+0000", 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0559999999999999e-06 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.982926+0000", 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42527986299999998 2024-09-17T16:36:33.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.983083+0000", 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015647299999999999 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,5", 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.983247+0000", 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016382500000000001 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:33.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.986311+0000", 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030639030000000002 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.998542+0000", 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012231568999999999 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.998866+0000", 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032433599999999999 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:35.998917+0000", 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0952999999999998e-05 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:26:36.003248+0000", 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004330825 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.367 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(6.4 582/568 [PushReplyOp(6:2ea410ca:::benchmark_data_smithi173_102017_object31074:head)])", 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:27:23.443416+0000", 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 549.89427428900001, 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44132895599999999, 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.443416+0000", 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.443416+0000", 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.443418+0000", 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.443423+0000", 2024-09-17T16:36:33.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6659999999999996e-06 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.881101+0000", 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43767767299999999 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.881120+0000", 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8770000000000002e-05 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.881281+0000", 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000160939 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:27:23.884745+0000", 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003464894 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.372 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:95580 7.d 7:b4431bb8:::benchmark_data_smithi173_116396_object95579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e671)", 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:09.919360+0000", 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 323.41833052599998, 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38153392000000003, 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 95580 2024-09-17T16:36:33.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919360+0000", 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919360+0000", 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919362+0000", 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.818e-06 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919391+0000", 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8687000000000002e-05 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919408+0000", 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7034e-05 2024-09-17T16:36:33.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919415+0000", 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4680000000000003e-06 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919476+0000", 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0708000000000002e-05 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919603+0000", 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012697000000000001 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919734+0000", 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131531 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.922948+0000", 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003214008 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923916+0000", 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096724599999999995 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300815+0000", 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37689942500000001 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300859+0000", 2024-09-17T16:36:33.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4095999999999999e-05 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300894+0000", 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4928999999999997e-05 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:95580 7.d e671/629 ondisk, result = 0)", 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:09.923561+0000", 2024-09-17T16:36:33.379 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 323.414130124, 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37734046100000002, 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923561+0000", 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923561+0000", 2024-09-17T16:36:33.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923562+0000", 2024-09-17T16:36:33.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9500000000000001e-07 2024-09-17T16:36:33.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923565+0000", 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.924e-06 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300712+0000", 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37714677099999999 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300721+0000", 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0140000000000004e-06 2024-09-17T16:36:33.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300801+0000", 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9324000000000001e-05 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300809+0000", 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7290000000000006e-06 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300901+0000", 2024-09-17T16:36:33.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1804000000000004e-05 2024-09-17T16:36:33.384 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.384 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.384 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.384 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170891 7.b 7:d10181ca:::benchmark_data_smithi173_116396_object75291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:36:33.384 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.431248+0000", 2024-09-17T16:36:33.384 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 186.90644296900001, 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37566875599999999, 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170891 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431248+0000", 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431248+0000", 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431250+0000", 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.829e-06 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431256+0000", 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1700000000000002e-06 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431270+0000", 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4051e-05 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431278+0000", 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1939999999999998e-06 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431360+0000", 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2365000000000002e-05 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431498+0000", 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013799400000000001 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431616+0000", 2024-09-17T16:36:33.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117209 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.436464+0000", 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048481389999999996 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.439954+0000", 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034900729999999998 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.806805+0000", 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36685158699999998 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.806875+0000", 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9413999999999996e-05 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.806917+0000", 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1730999999999997e-05 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.390 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170904 7.d 7:b74ecf00:::benchmark_data_smithi173_116396_object75304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.444710+0000", 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 186.89298103499999, 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35556337700000001, 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170904 2024-09-17T16:36:33.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444710+0000", 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444710+0000", 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444712+0000", 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.874e-06 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444726+0000", 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4463e-05 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444743+0000", 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7385e-05 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444751+0000", 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8760000000000005e-06 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444808+0000", 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7003999999999998e-05 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444935+0000", 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012665699999999999 2024-09-17T16:36:33.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.445085+0000", 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000150008 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.454064+0000", 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089791750000000007 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457533+0000", 2024-09-17T16:36:33.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034685369999999998 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.463684+0000", 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061514509999999996 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.800223+0000", 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.336538689 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.800273+0000", 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0257999999999999e-05 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170913 7.d 7:b6585817:::benchmark_data_smithi173_116396_object75313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.457482+0000", 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 186.880208692, 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35359675800000001, 2024-09-17T16:36:33.397 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.399 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.399 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.399 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170913 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457482+0000", 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457482+0000", 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457484+0000", 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4890000000000001e-06 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457490+0000", 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2790000000000004e-06 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457502+0000", 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2573e-05 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457514+0000", 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1386000000000001e-05 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457558+0000", 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4502000000000002e-05 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457681+0000", 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012229 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457796+0000", 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114926 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.462659+0000", 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048635800000000002 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.807157+0000", 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34449779000000003 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.810897+0000", 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003739769 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.811012+0000", 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000115796 2024-09-17T16:36:33.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.811079+0000", 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6377999999999994e-05 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170935 7.d 7:b3c7b477:::benchmark_data_smithi173_116396_object75335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.480767+0000", 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 186.85692390700001, 2024-09-17T16:36:33.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.344245668, 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170935 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480767+0000", 2024-09-17T16:36:33.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480767+0000", 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480769+0000", 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7770000000000001e-06 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480776+0000", 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7079999999999997e-06 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801464+0000", 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32068738499999999 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801474+0000", 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8600000000000005e-06 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801522+0000", 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7987e-05 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801669+0000", 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014749699999999999 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801798+0000", 2024-09-17T16:36:33.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012916300000000001 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.813516+0000", 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011717589 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.818920+0000", 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054044940000000001 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.824864+0000", 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005943444 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.824967+0000", 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103049 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.825013+0000", 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5714999999999999e-05 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.411 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178134 7.d 7:b9b47827:::benchmark_data_smithi173_116396_object82534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.050884+0000", 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 177.286806899, 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48526529400000001, 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.412 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178134 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050884+0000", 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050884+0000", 2024-09-17T16:36:33.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050885+0000", 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1299999999999998e-07 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050889+0000", 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7529999999999999e-06 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050899+0000", 2024-09-17T16:36:33.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0837e-05 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050905+0000", 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3229999999999997e-06 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050959+0000", 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4435000000000003e-05 2024-09-17T16:36:33.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.051048+0000", 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8609000000000005e-05 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.051121+0000", 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2887999999999997e-05 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.054958+0000", 2024-09-17T16:36:33.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038371149999999999 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055583+0000", 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062568400000000005 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536053+0000", 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48047000499999998 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536105+0000", 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1311999999999999e-05 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536149+0000", 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4419999999999998e-05 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:178134 7.d e717/629 ondisk, result = 0)", 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.055085+0000", 2024-09-17T16:36:33.419 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 177.28260575199999, 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48106911299999999, 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055085+0000", 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055085+0000", 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055086+0000", 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.08e-06 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055091+0000", 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1989999999999997e-06 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.535995+0000", 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48090385899999999 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.535999+0000", 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9899999999999999e-06 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536041+0000", 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1597999999999997e-05 2024-09-17T16:36:33.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536048+0000", 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3170000000000004e-06 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536154+0000", 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010607000000000001 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178372 7.b 7:d011cddd:::benchmark_data_smithi173_116396_object82772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e718)", 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.909320+0000", 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 176.42837027799999, 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.399272617, 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178372 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.424 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909320+0000", 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909320+0000", 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909322+0000", 2024-09-17T16:36:33.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6080000000000001e-06 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909329+0000", 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4300000000000003e-06 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909345+0000", 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6188000000000002e-05 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909352+0000", 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7719999999999994e-06 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909421+0000", 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8571000000000005e-05 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909546+0000", 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012531700000000001 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909678+0000", 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013156999999999999 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.913604+0000", 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039264749999999996 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.915100+0000", 2024-09-17T16:36:33.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001495461 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308513+0000", 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39341283700000002 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308546+0000", 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3504999999999999e-05 2024-09-17T16:36:33.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308593+0000", 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6882999999999998e-05 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:178372 7.b e718/629 ondisk, result = 0)", 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 176.422907192, 2024-09-17T16:36:33.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.393815833, 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9800000000000002e-07 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914790+0000", 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1270000000000002e-06 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308456+0000", 2024-09-17T16:36:33.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39366633899999998 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308463+0000", 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9480000000000002e-06 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308504+0000", 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1247e-05 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308509+0000", 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2499999999999997e-06 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308599+0000", 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0024000000000004e-05 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(7.0 e725,723 7:04a95b88:::benchmark_data_smithi173_116396_object28410:head,690'6998)", 2024-09-17T16:36:33.434 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 155.378080074, 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89116632100000004, 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4199999999999995e-07 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959615+0000", 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0319999999999997e-06 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.436 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850585+0000", 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89097006499999998 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850600+0000", 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4747e-05 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850689+0000", 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8546000000000002e-05 2024-09-17T16:36:33.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850701+0000", 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1878e-05 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850777+0000", 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6410999999999997e-05 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(7.0 e725,723 7:05ce1687:::benchmark_data_smithi173_116396_object29751:head,691'7038)", 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:59.220497+0000", 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 154.117193407, 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53176433000000001, 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220497+0000", 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220497+0000", 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.440 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220498+0000", 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8599999999999998e-07 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220501+0000", 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1209999999999998e-06 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752133+0000", 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53163205499999999 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752141+0000", 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4950000000000002e-06 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752200+0000", 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9954000000000002e-05 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752208+0000", 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9810000000000003e-06 2024-09-17T16:36:33.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752262+0000", 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3238000000000001e-05 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.17 740/739 [PushReplyOp(8:ee7fa771:::benchmark_data_smithi173_130566_object904:head)])", 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 126.47370967800001, 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79826256500000004, 2024-09-17T16:36:33.443 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.35e-07 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863983+0000", 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.807e-06 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.661769+0000", 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79778559699999996 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.661776+0000", 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2660000000000004e-06 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.661830+0000", 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3662999999999998e-05 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.662244+0000", 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00041399699999999999 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:94950 8.15 e769/743 8:aa553242:::benchmark_data_smithi173_130566_object94949:head v 769'3213, mlcod=769'3210)", 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:41.590971+0000", 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 51.746720250999999, 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47015781000000001, 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.447 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.590971+0000", 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.590971+0000", 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.590971+0000", 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8000000000000004e-07 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.591068+0000", 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6348999999999995e-05 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.059385+0000", 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46831688199999999 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.059396+0000", 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1151999999999999e-05 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.059468+0000", 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2408000000000002e-05 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.059499+0000", 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0930999999999999e-05 2024-09-17T16:36:33.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.061083+0000", 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001583568 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.061128+0000", 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5639999999999997e-05 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:94971 8.b e769/728 8:db7e09c1:::benchmark_data_smithi173_130566_object94970:head v 769'5977, mlcod=769'5974)", 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:41.611309+0000", 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 51.726381390999997, 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.780065801, 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.611309+0000", 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.611309+0000", 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.611310+0000", 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0899999999999999e-06 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.611418+0000", 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010796799999999999 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.388780+0000", 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77736194300000006 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.388788+0000", 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7999999999999999e-06 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.388850+0000", 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2075000000000005e-05 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.388869+0000", 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8453000000000001e-05 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.391341+0000", 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0024723890000000002 2024-09-17T16:36:33.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:33.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:33.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:33.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.391375+0000", 2024-09-17T16:36:33.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4082999999999999e-05 2024-09-17T16:36:33.456 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.456 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.456 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.456 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:33.456 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:33.456 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:36:33.456 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:33.457 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:33 smithi173 ceph-mon[30530]: pgmap v1758: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:33.457 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1556565996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:33.457 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:33 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:33.458 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:33 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:33.458 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:33 smithi173 ceph-mon[30530]: 8.8 scrub starts 2024-09-17T16:36:33.483 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:33.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:33 smithi017 ceph-mon[27571]: pgmap v1758: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1556565996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:33 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:33 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:33 smithi017 ceph-mon[27571]: 8.8 scrub starts 2024-09-17T16:36:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:33 smithi071 ceph-mon[30002]: pgmap v1758: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1556565996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:33 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:33 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:33 smithi071 ceph-mon[30002]: 8.8 scrub starts 2024-09-17T16:36:33.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:33.610 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:33.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:33.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:33.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:33.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:33.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:33.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:33.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:33.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:33.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:36:33.414+0000 7ff0be236640 -1 Falling back to public interface 2024-09-17T16:36:33.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:33.987 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:33.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:34.089 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:34.111 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:34.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:34.213 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:34.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:36:34.239 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:34.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:34.343 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:34.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:34.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:34.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:34.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:34 smithi017 ceph-mon[27571]: osdmap e788: 12 total, 10 up, 12 in 2024-09-17T16:36:34.494 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:34.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:34.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:34 smithi071 ceph-mon[30002]: osdmap e788: 12 total, 10 up, 12 in 2024-09-17T16:36:34.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:34.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:34.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:34 smithi173 ceph-mon[30530]: osdmap e788: 12 total, 10 up, 12 in 2024-09-17T16:36:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:36:34.712 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:36:34.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:34.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:34.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:34.787 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:34.821 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:34.849 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:34.872 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:34.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:34.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:34.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:34.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:34.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:35.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:35.121 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:35.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:35.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:35.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:35 smithi017 ceph-mon[27571]: 8.8 scrub ok 2024-09-17T16:36:35.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:35 smithi017 ceph-mon[27571]: pgmap v1760: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:35.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:35 smithi017 ceph-mon[27571]: 8.15 scrub starts 2024-09-17T16:36:35.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:35 smithi017 ceph-mon[27571]: 8.15 scrub ok 2024-09-17T16:36:35.245 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:35.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:35.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:35.347 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:35.369 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:35.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:35.471 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:35.492 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:35.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:35 smithi071 ceph-mon[30002]: 8.8 scrub ok 2024-09-17T16:36:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:35 smithi071 ceph-mon[30002]: pgmap v1760: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:35 smithi071 ceph-mon[30002]: 8.15 scrub starts 2024-09-17T16:36:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:35 smithi071 ceph-mon[30002]: 8.15 scrub ok 2024-09-17T16:36:35.595 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:35.617 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:35.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:35 smithi173 ceph-mon[30530]: 8.8 scrub ok 2024-09-17T16:36:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:35 smithi173 ceph-mon[30530]: pgmap v1760: 27 pgs: 1 active+undersized+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255615 objects degraded (11.383%); 13390/255615 objects misplaced (5.238%) 2024-09-17T16:36:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:35 smithi173 ceph-mon[30530]: 8.15 scrub starts 2024-09-17T16:36:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:35 smithi173 ceph-mon[30530]: 8.15 scrub ok 2024-09-17T16:36:35.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:35.740 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:35.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:35.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:35.842 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:35.863 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:35.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:35.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:35.989 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:35.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:35.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:36.001 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:36:36.001 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:36:36.001 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:36:36.001 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:36:36.092 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:36.114 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:36.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:36.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:36.213 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:36:36.216 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:36.238 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:36.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:36.340 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:36.362 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:36.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:36.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:36.487 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:36.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:36 smithi017 ceph-mon[27571]: 8.14 scrub starts 2024-09-17T16:36:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:36 smithi017 ceph-mon[27571]: 8.12 scrub starts 2024-09-17T16:36:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:36 smithi017 ceph-mon[27571]: 8.14 scrub ok 2024-09-17T16:36:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3000832861' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:36 smithi071 ceph-mon[30002]: 8.14 scrub starts 2024-09-17T16:36:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:36 smithi071 ceph-mon[30002]: 8.12 scrub starts 2024-09-17T16:36:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:36 smithi071 ceph-mon[30002]: 8.14 scrub ok 2024-09-17T16:36:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3000832861' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:36.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:36.616 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:36.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:36.617 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:36 smithi173 ceph-mon[30530]: 8.14 scrub starts 2024-09-17T16:36:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:36 smithi173 ceph-mon[30530]: 8.12 scrub starts 2024-09-17T16:36:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:36 smithi173 ceph-mon[30530]: 8.14 scrub ok 2024-09-17T16:36:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3000832861' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:36.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:36:36.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:36.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:36.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:36.843 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:36.865 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:36.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:36.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:36.989 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:36.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:37.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:37.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:37.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:37.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:36 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:36:36.838+0000 7ff0be236640 -1 osd.10 740 log_to_monitors true 2024-09-17T16:36:37.215 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:37.239 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:37.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:37.340 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:37.362 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:37.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:37.462 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:36:37.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:37.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: 8.10 scrub starts 2024-09-17T16:36:37.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: 8.12 scrub ok 2024-09-17T16:36:37.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: 8.10 scrub ok 2024-09-17T16:36:37.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: pgmap v1761: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+undersized+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255618 objects degraded (11.383%); 13390/255618 objects misplaced (5.238%) 2024-09-17T16:36:37.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 29096/255618 objects degraded (11.383%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:36:37.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:36:37.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3000832861' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:37.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: osdmap e789: 12 total, 10 up, 12 in 2024-09-17T16:36:37.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: 8.17 scrub starts 2024-09-17T16:36:37.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: from='osd.10 [v2:172.21.15.173:6816/2988338661,v1:172.21.15.173:6817/2988338661]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:36:37.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:36:37.486 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:37 smithi017 ceph-mon[27571]: 8.17 scrub ok 2024-09-17T16:36:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:37.487 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:37.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:37.497 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: 8.10 scrub starts 2024-09-17T16:36:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: 8.12 scrub ok 2024-09-17T16:36:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: 8.10 scrub ok 2024-09-17T16:36:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: pgmap v1761: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+undersized+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255618 objects degraded (11.383%); 13390/255618 objects misplaced (5.238%) 2024-09-17T16:36:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 29096/255618 objects degraded (11.383%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:36:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:36:37.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3000832861' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:37.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: osdmap e789: 12 total, 10 up, 12 in 2024-09-17T16:36:37.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: 8.17 scrub starts 2024-09-17T16:36:37.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: from='osd.10 [v2:172.21.15.173:6816/2988338661,v1:172.21.15.173:6817/2988338661]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:36:37.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:36:37.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:37 smithi071 ceph-mon[30002]: 8.17 scrub ok 2024-09-17T16:36:37.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:37.610 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:37.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: 8.10 scrub starts 2024-09-17T16:36:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: 8.12 scrub ok 2024-09-17T16:36:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: 8.10 scrub ok 2024-09-17T16:36:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: pgmap v1761: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+undersized+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255618 objects degraded (11.383%); 13390/255618 objects misplaced (5.238%) 2024-09-17T16:36:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 29096/255618 objects degraded (11.383%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:36:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:36:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3000832861' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:37.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: osdmap e789: 12 total, 10 up, 12 in 2024-09-17T16:36:37.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: 8.17 scrub starts 2024-09-17T16:36:37.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: from='osd.10 [v2:172.21.15.173:6816/2988338661,v1:172.21.15.173:6817/2988338661]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:36:37.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-17T16:36:37.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-mon[30530]: 8.17 scrub ok 2024-09-17T16:36:37.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:37.734 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:37.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:37.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:37.858 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:37.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:37.960 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:37.982 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:37.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:38.011 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:36:38.012 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:36:38.034 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:36:38.035 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:36:38.047 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:38.057 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:36:38.057 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:36:38.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:36:38.081 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:36:38.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:38.103 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:36:38.104 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:36:38.104 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:36:38.107 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:38.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:38.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:38.125 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:36:38.125 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:36:38.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:36:38.144 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:36:37 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:36:37.743+0000 7ff0b1fd8640 -1 osd.10 790 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:36:38.146 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:36:38.147 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:36:38.209 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:38.233 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:38.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:38.265 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:36:38.265 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: osdmap e790: 12 total, 10 up, 12 in 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: from='osd.10 [v2:172.21.15.173:6816/2988338661,v1:172.21.15.173:6817/2988338661]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: 8.b scrub starts 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: from='osd.10 ' entity='osd.10' 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:36:38.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3578190590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:38.286 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:36:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:36:38.287 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-17T16:36:38.309 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:36:38.309 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:36:38.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:38.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:38.361 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:38.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:38.377 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:36:38.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:38.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:38.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:38.498 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-17T16:36:38.498 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: osdmap e790: 12 total, 10 up, 12 in 2024-09-17T16:36:38.498 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: from='osd.10 [v2:172.21.15.173:6816/2988338661,v1:172.21.15.173:6817/2988338661]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:36:38.498 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:36:38.498 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: 8.b scrub starts 2024-09-17T16:36:38.498 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: from='osd.10 ' entity='osd.10' 2024-09-17T16:36:38.498 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:36:38.498 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:36:38.499 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:36:38.499 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:36:38.499 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3578190590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-17T16:36:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: osdmap e790: 12 total, 10 up, 12 in 2024-09-17T16:36:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: from='osd.10 [v2:172.21.15.173:6816/2988338661,v1:172.21.15.173:6817/2988338661]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:36:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-09-17T16:36:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: 8.b scrub starts 2024-09-17T16:36:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: from='osd.10 ' entity='osd.10' 2024-09-17T16:36:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:36:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:36:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:36:38.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:36:38.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3578190590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:38.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:38.611 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:38.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:38.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:38.741 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:38.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:38.743 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:36:38.743 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:36:38.843 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:38.867 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:38.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:38.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:38.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:36:38.982 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:36:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:36:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104264 8.4 8.cdf0f904 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958458+0000", 2024-09-17T16:36:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020526461999999999, 2024-09-17T16:36:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020557071, 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104264 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958458+0000", 2024-09-17T16:36:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958458+0000", 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958460+0000", 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1150000000000001e-06 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958467+0000", 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.776e-06 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958476+0000", 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3300000000000005e-06 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958485+0000", 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5250000000000005e-06 2024-09-17T16:36:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958553+0000", 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8027000000000004e-05 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958563+0000", 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210999999999999e-05 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104266 8.4 8.df640d24 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958502+0000", 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020482033, 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020593612000000001, 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104266 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958502+0000", 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958502+0000", 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958503+0000", 2024-09-17T16:36:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3500000000000002e-07 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958505+0000", 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.446e-06 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958510+0000", 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3950000000000003e-06 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958513+0000", 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0790000000000001e-06 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958585+0000", 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2260000000000003e-05 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:36:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2109999999999997e-06 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104310 8.4 8.3aa61324 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020448323000000001, 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020618572000000002, 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104310 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1899999999999998e-07 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958538+0000", 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.818e-06 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958545+0000", 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4300000000000003e-06 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958548+0000", 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8279999999999996e-06 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958613+0000", 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4791999999999999e-05 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958620+0000", 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6980000000000001e-06 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104365 8.4 8.98b36004 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020425382999999998, 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.02064976, 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104365 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:36:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1399999999999998e-07 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958561+0000", 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.793e-06 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958565+0000", 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4790000000000001e-06 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958568+0000", 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9409999999999999e-06 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958647+0000", 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9058000000000001e-05 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958650+0000", 2024-09-17T16:36:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.422e-06 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104510 8.4 8.71f883e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020406372999999998, 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020686685999999999, 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104510 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0899999999999997e-07 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958580+0000", 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3149999999999999e-06 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958583+0000", 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7900000000000001e-06 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958586+0000", 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5639999999999999e-06 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958683+0000", 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6762999999999995e-05 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958689+0000", 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2349999999999996e-06 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104538 8.4 8.69da0a4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.02039417, 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020724237999999999, 2024-09-17T16:36:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104538 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:36:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.29e-07 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958592+0000", 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4929999999999999e-06 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958595+0000", 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0369999999999999e-06 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958598+0000", 2024-09-17T16:36:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.932e-06 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958715+0000", 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011665 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958718+0000", 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.591e-06 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104542 8.4 8.5948f164 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020365548000000001, 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020724659999999999, 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104542 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8900000000000001e-07 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958621+0000", 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.978e-06 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958625+0000", 2024-09-17T16:36:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4599999999999999e-06 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958627+0000", 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7240000000000001e-06 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958747+0000", 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012020599999999999 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958753+0000", 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9800000000000003e-06 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104571 8.4 8.f7b7a3a4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020341604999999999, 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020730135, 2024-09-17T16:36:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104571 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9999999999999999e-07 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958644+0000", 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2529999999999999e-06 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958648+0000", 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3620000000000002e-06 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958650+0000", 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.514e-06 2024-09-17T16:36:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958778+0000", 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012807300000000001 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958781+0000", 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.039e-06 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104651 8.4 8.2da1dbe4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020322864999999999, 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020738691, 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104651 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8999999999999998e-07 2024-09-17T16:36:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958663+0000", 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2759999999999999e-06 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958667+0000", 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2880000000000003e-06 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958670+0000", 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4650000000000001e-06 2024-09-17T16:36:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958811+0000", 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014136500000000001 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958817+0000", 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1160000000000004e-06 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104652 8.4 8.77678704 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958673+0000", 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020310948999999998, 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020754659000000002, 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104652 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958673+0000", 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958673+0000", 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958674+0000", 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.28e-07 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958675+0000", 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.141e-06 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958678+0000", 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0010000000000002e-06 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958682+0000", 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2869999999999998e-06 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958843+0000", 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001607 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958846+0000", 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0419999999999998e-06 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104673 8.4 8.63bb1ee4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958685+0000", 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020299052000000001, 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020769935, 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104673 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958685+0000", 2024-09-17T16:36:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:39.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:39.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958685+0000", 2024-09-17T16:36:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958686+0000", 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2600000000000001e-07 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958687+0000", 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2580000000000001e-06 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958690+0000", 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9909999999999998e-06 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958692+0000", 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4360000000000001e-06 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958875+0000", 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000182327 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958881+0000", 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0410000000000003e-06 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104674 8.4 8.5e810684 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020273349999999999, 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020771685000000002, 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104674 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9299999999999999e-07 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958715+0000", 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7869999999999999e-06 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958719+0000", 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5980000000000001e-06 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958721+0000", 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6589999999999999e-06 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958920+0000", 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000198789 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958924+0000", 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4379999999999999e-06 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104695 8.4 8.bc0ebd64 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020259282, 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020784315000000001, 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104695 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4400000000000001e-07 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958727+0000", 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.657e-06 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958730+0000", 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8480000000000001e-06 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958732+0000", 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4200000000000001e-06 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958955+0000", 2024-09-17T16:36:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022305200000000001 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958962+0000", 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0369999999999997e-06 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104721 8.4 8.83410e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020243483, 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020795195999999998, 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104721 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:36:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0199999999999998e-07 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958742+0000", 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.26e-06 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958746+0000", 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8319999999999999e-06 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958749+0000", 2024-09-17T16:36:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7980000000000001e-06 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958987+0000", 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023819600000000001 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958990+0000", 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0110000000000001e-06 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104726 8.4 8.bcfb8bc4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958752+0000", 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020231927, 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020814836999999999, 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104726 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958752+0000", 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958752+0000", 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958753+0000", 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.29e-07 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958754+0000", 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3799999999999996e-07 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958756+0000", 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8329999999999998e-06 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958759+0000", 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7049999999999999e-06 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959018+0000", 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025930899999999999 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959025+0000", 2024-09-17T16:36:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1990000000000003e-06 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104729 8.4 8.1aa6e3c4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.020209297000000001, 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020819588, 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for peered", 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:36:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104729 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:36:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9499999999999998e-07 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958777+0000", 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.184e-06 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958780+0000", 2024-09-17T16:36:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7440000000000001e-06 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958783+0000", 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0910000000000001e-06 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959039+0000", 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000255133 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:36:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959042+0000", 2024-09-17T16:36:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6049999999999998e-06 2024-09-17T16:36:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:36:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:36:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:36:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 16 2024-09-17T16:36:39.052 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:36:39.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:39.152 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:39.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:39.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:39.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:39 smithi017 ceph-mon[27571]: pgmap v1764: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+undersized+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255618 objects degraded (11.383%); 13390/255618 objects misplaced (5.238%) 2024-09-17T16:36:39.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:39 smithi017 ceph-mon[27571]: Health check update: 1 osds down (OSD_DOWN) 2024-09-17T16:36:39.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:39 smithi017 ceph-mon[27571]: osd.10 [v2:172.21.15.173:6816/2988338661,v1:172.21.15.173:6817/2988338661] boot 2024-09-17T16:36:39.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:39 smithi017 ceph-mon[27571]: osdmap e791: 12 total, 11 up, 12 in 2024-09-17T16:36:39.266 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:39 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:36:39.278 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:39.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:39.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:39.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:39.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:39.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:39.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:39.504 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:39.527 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:39.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:39.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:39 smithi173 ceph-mon[30530]: pgmap v1764: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+undersized+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255618 objects degraded (11.383%); 13390/255618 objects misplaced (5.238%) 2024-09-17T16:36:39.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:39 smithi173 ceph-mon[30530]: Health check update: 1 osds down (OSD_DOWN) 2024-09-17T16:36:39.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:39 smithi173 ceph-mon[30530]: osd.10 [v2:172.21.15.173:6816/2988338661,v1:172.21.15.173:6817/2988338661] boot 2024-09-17T16:36:39.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:39 smithi173 ceph-mon[30530]: osdmap e791: 12 total, 11 up, 12 in 2024-09-17T16:36:39.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:39 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:36:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:39 smithi071 ceph-mon[30002]: pgmap v1764: 27 pgs: 1 active+clean+remapped+scrubbing, 1 active+undersized+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 9 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 29096/255618 objects degraded (11.383%); 13390/255618 objects misplaced (5.238%) 2024-09-17T16:36:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:39 smithi071 ceph-mon[30002]: Health check update: 1 osds down (OSD_DOWN) 2024-09-17T16:36:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:39 smithi071 ceph-mon[30002]: osd.10 [v2:172.21.15.173:6816/2988338661,v1:172.21.15.173:6817/2988338661] boot 2024-09-17T16:36:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:39 smithi071 ceph-mon[30002]: osdmap e791: 12 total, 11 up, 12 in 2024-09-17T16:36:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:39 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-17T16:36:39.629 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:39.652 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:39.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:36:39.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:36:39.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:39.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:39.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:39.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:39.900 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:39.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:40.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:40.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:40.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:40.035 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-09-17T16:36:40.035 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-09-17T16:36:40.036 DEBUG:teuthology.orchestra.run.smithi071:osd.5> sudo journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.5.service 2024-09-17T16:36:40.038 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl start ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.5 2024-09-17T16:36:40.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:40.153 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:40.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:40.255 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:40.277 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:40.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:40.314 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:40 smithi071 systemd[1]: Starting Ceph osd.5 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:36:40.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:40.401 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:40.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:40.503 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:40.524 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:40.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:40.616 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:40 smithi071 podman[142587]: 2024-09-17 16:36:40.265478293 +0000 UTC m=+0.019187586 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:36:40.617 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:40 smithi071 podman[142587]: 2024-09-17 16:36:40.382558486 +0000 UTC m=+0.136267780 container create 96d67861aa0b7acc9fb36d3991b90714a2c6b78d4447e316879a94841fd7b0bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:36:40.617 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:40 smithi071 podman[142587]: 2024-09-17 16:36:40.61040783 +0000 UTC m=+0.364117136 container init 96d67861aa0b7acc9fb36d3991b90714a2c6b78d4447e316879a94841fd7b0bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T16:36:40.626 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:40.652 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:40.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:40.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:40.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:40.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:40.886 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:40.909 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:40.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:40.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:40 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:40.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:40 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:40.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:40 smithi017 ceph-mon[27571]: 8.b scrub ok 2024-09-17T16:36:40.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:40 smithi017 ceph-mon[27571]: osdmap e792: 12 total, 11 up, 12 in 2024-09-17T16:36:40.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:40 smithi017 ceph-mon[27571]: pgmap v1767: 27 pgs: 3 remapped+peering, 1 active+undersized+remapped, 1 active+clean+remapped+scrubbing, 1 active+clean+remapped, 1 undersized+degraded+peered, 7 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 23717/255624 objects degraded (9.278%); 10595/255624 objects misplaced (4.145%) 2024-09-17T16:36:41.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:41.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:41.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:40 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:40 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:40 smithi071 ceph-mon[30002]: 8.b scrub ok 2024-09-17T16:36:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:40 smithi071 ceph-mon[30002]: osdmap e792: 12 total, 11 up, 12 in 2024-09-17T16:36:41.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:40 smithi071 ceph-mon[30002]: pgmap v1767: 27 pgs: 3 remapped+peering, 1 active+undersized+remapped, 1 active+clean+remapped+scrubbing, 1 active+clean+remapped, 1 undersized+degraded+peered, 7 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 23717/255624 objects degraded (9.278%); 10595/255624 objects misplaced (4.145%) 2024-09-17T16:36:41.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:40 smithi071 podman[142587]: 2024-09-17 16:36:40.614899203 +0000 UTC m=+0.368608497 container start 96d67861aa0b7acc9fb36d3991b90714a2c6b78d4447e316879a94841fd7b0bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T16:36:41.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:40 smithi071 podman[142587]: 2024-09-17 16:36:40.719538749 +0000 UTC m=+0.473248047 container attach 96d67861aa0b7acc9fb36d3991b90714a2c6b78d4447e316879a94841fd7b0bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True) 2024-09-17T16:36:41.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:40 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate[142602]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:36:41.065 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:40 smithi071 bash[142587]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:36:41.065 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:40 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate[142602]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:36:41.065 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:40 smithi071 bash[142587]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T16:36:41.104 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:41.138 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:36:41.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:41.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:40 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:40 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:40 smithi173 ceph-mon[30530]: 8.b scrub ok 2024-09-17T16:36:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:40 smithi173 ceph-mon[30530]: osdmap e792: 12 total, 11 up, 12 in 2024-09-17T16:36:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:40 smithi173 ceph-mon[30530]: pgmap v1767: 27 pgs: 3 remapped+peering, 1 active+undersized+remapped, 1 active+clean+remapped+scrubbing, 1 active+clean+remapped, 1 undersized+degraded+peered, 7 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 23717/255624 objects degraded (9.278%); 10595/255624 objects misplaced (4.145%) 2024-09-17T16:36:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:41.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:41.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:41.313 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:41.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:41.416 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:41.444 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:41.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:41.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:41.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate[142602]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T16:36:41.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 bash[142587]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T16:36:41.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate[142602]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T16:36:41.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 bash[142587]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T16:36:41.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:41.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:41.671 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:41.694 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:41.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:41.736 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:36:41.796 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:41.841 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:41.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:41.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:41.965 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:41 smithi071 ceph-mon[30002]: 8.a scrub starts 2024-09-17T16:36:41.965 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:41 smithi071 ceph-mon[30002]: osdmap e793: 12 total, 11 up, 12 in 2024-09-17T16:36:41.965 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate[142602]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T16:36:41.965 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 bash[142587]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T16:36:41.966 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate[142602]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T16:36:41.966 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 bash[142587]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T16:36:41.966 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate[142602]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T16:36:41.966 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 bash[142587]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T16:36:41.966 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate[142602]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T16:36:41.966 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 bash[142587]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T16:36:41.966 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate[142602]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T16:36:41.966 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 bash[142587]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T16:36:41.966 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:41 smithi071 podman[142587]: 2024-09-17 16:36:41.601793935 +0000 UTC m=+1.355503229 container died 96d67861aa0b7acc9fb36d3991b90714a2c6b78d4447e316879a94841fd7b0bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T16:36:41.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:41.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:41.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:41 smithi017 ceph-mon[27571]: 8.a scrub starts 2024-09-17T16:36:41.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:41 smithi017 ceph-mon[27571]: osdmap e793: 12 total, 11 up, 12 in 2024-09-17T16:36:42.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:42.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:42.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:42.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:42.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:41 smithi173 ceph-mon[30530]: 8.a scrub starts 2024-09-17T16:36:42.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:41 smithi173 ceph-mon[30530]: osdmap e793: 12 total, 11 up, 12 in 2024-09-17T16:36:42.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:42.251 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:42.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:42.294 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:42.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:42.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:42.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:42.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:42.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:42.527 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:42.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:42.628 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:42.660 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:42.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:42.729 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:36:42.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:42.776 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:42 smithi071 podman[142724]: 2024-09-17 16:36:42.523329359 +0000 UTC m=+0.914278813 container remove 96d67861aa0b7acc9fb36d3991b90714a2c6b78d4447e316879a94841fd7b0bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-activate, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-17T16:36:42.776 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:42 smithi071 podman[142779]: 2024-09-17 16:36:42.658275819 +0000 UTC m=+0.019301914 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:36:42.776 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:42 smithi071 ceph-mon[30002]: pgmap v1769: 27 pgs: 1 activating+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 1 active+undersized+remapped, 3 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30457/255453 objects degraded (11.923%); 5298/255453 objects misplaced (2.074%) 2024-09-17T16:36:42.776 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3335017308' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:36:42.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:42.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:42.898 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:42.923 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:42.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:42.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:42 smithi017 ceph-mon[27571]: pgmap v1769: 27 pgs: 1 activating+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 1 active+undersized+remapped, 3 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30457/255453 objects degraded (11.923%); 5298/255453 objects misplaced (2.074%) 2024-09-17T16:36:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3335017308' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:36:43.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:43.028 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:42 smithi071 podman[142779]: 2024-09-17 16:36:42.774458503 +0000 UTC m=+0.135484582 container create d46e44097d19934d74526cd6eab922679ad9de482202c482162f11b2a64d0478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T16:36:43.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:43.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:42 smithi173 ceph-mon[30530]: pgmap v1769: 27 pgs: 1 activating+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 1 active+undersized+remapped, 3 active+undersized+degraded, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 30457/255453 objects degraded (11.923%); 5298/255453 objects misplaced (2.074%) 2024-09-17T16:36:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3335017308' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:36:43.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:43.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:43.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:43.228 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:36:43.298 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:43.314 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:43 smithi071 podman[142779]: 2024-09-17 16:36:43.027066027 +0000 UTC m=+0.388092115 container init d46e44097d19934d74526cd6eab922679ad9de482202c482162f11b2a64d0478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:36:43.314 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:43 smithi071 podman[142779]: 2024-09-17 16:36:43.030991597 +0000 UTC m=+0.392017677 container start d46e44097d19934d74526cd6eab922679ad9de482202c482162f11b2a64d0478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-17T16:36:43.314 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:43 smithi071 ceph-osd[142816]: -- 172.21.15.71:0/3832205500 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5618fdffc1a0 con 0x5618fd1b9c00 2024-09-17T16:36:43.314 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:43 smithi071 ceph-osd[142816]: -- 172.21.15.71:0/3832205500 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5618fd1dba40 con 0x5618fd1b9c00 2024-09-17T16:36:43.314 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:43 smithi071 bash[142779]: d46e44097d19934d74526cd6eab922679ad9de482202c482162f11b2a64d0478 2024-09-17T16:36:43.314 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:43 smithi071 systemd[1]: Started Ceph osd.5 for 19341aec-750e-11ef-bceb-c7b262605968. 2024-09-17T16:36:43.327 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:43.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:43.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:43.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:43.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:43.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:43.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:43.549 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:36:43.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:43.600 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:43.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:43.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:43.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:43.728 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:43.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:43.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:43.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:43.859 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:43.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:43.967 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:43 smithi071 ceph-mon[30002]: 8.a scrub ok 2024-09-17T16:36:43.967 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:43 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:43.967 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:43 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:43.967 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:43 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 30457/255453 objects degraded (11.923%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:36:43.967 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:43 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:43.967 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3335017308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:36:43.967 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:43 smithi071 ceph-mon[30002]: osdmap e794: 12 total, 11 up, 12 in 2024-09-17T16:36:43.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:43 smithi017 ceph-mon[27571]: 8.a scrub ok 2024-09-17T16:36:43.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:43 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:43.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:43 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:43.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:43 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 30457/255453 objects degraded (11.923%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:36:43.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:43 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:43.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3335017308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:36:43.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:43 smithi017 ceph-mon[27571]: osdmap e794: 12 total, 11 up, 12 in 2024-09-17T16:36:43.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:43.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:44.090 INFO:teuthology.orchestra.run.smithi071.stderr:no valid command found; 10 closest matches: 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:0 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:1 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:2 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:abort 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:assert 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:config diff 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:config diff get 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:config get 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:config help [] 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:config set ... 2024-09-17T16:36:44.091 INFO:teuthology.orchestra.run.smithi071.stderr:admin_socket: invalid command 2024-09-17T16:36:44.098 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:44.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:44.128 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:44.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:43 smithi173 ceph-mon[30530]: 8.a scrub ok 2024-09-17T16:36:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:43 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:43 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:43 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 30457/255453 objects degraded (11.923%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:36:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:43 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3335017308' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:36:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:43 smithi173 ceph-mon[30530]: osdmap e794: 12 total, 11 up, 12 in 2024-09-17T16:36:44.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:44.256 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:44.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:44.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:44.384 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:44.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:44.473 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:44.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:44.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:44.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:44.519 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:36:44.520 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:36:44.546 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:36:44.547 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:36:44.595 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:36:44.596 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:36:44.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:44.630 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:36:44.630 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:36:44.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:44.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:44.665 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:36:44.665 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:36:44.665 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-17T16:36:44.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:36:44.693 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:36:44.723 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:36:44.724 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:36:44.747 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:44.757 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:36:44.757 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:36:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:36:44.772 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-09-17T16:36:44.775 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:44.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:44.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:44.785 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:36:44.785 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:36:44.815 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:36:44.815 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:36:44.839 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:36:44.839 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:36:44.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:44.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:44.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:44 smithi017 ceph-mon[27571]: pgmap v1771: 27 pgs: 1 activating+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33987/254025 objects degraded (13.379%); 2393/254025 objects misplaced (0.942%); 1.0 MiB/s, 16 objects/s recovering 2024-09-17T16:36:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3011544996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:45.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:45.028 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:45.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:45.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:44 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:36:44.962+0000 7f1f9cd45640 -1 Falling back to public interface 2024-09-17T16:36:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:44 smithi071 ceph-mon[30002]: pgmap v1771: 27 pgs: 1 activating+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33987/254025 objects degraded (13.379%); 2393/254025 objects misplaced (0.942%); 1.0 MiB/s, 16 objects/s recovering 2024-09-17T16:36:45.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3011544996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:45.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:45.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:44 smithi173 ceph-mon[30530]: pgmap v1771: 27 pgs: 1 activating+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 13 active+clean; 5.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 33987/254025 objects degraded (13.379%); 2393/254025 objects misplaced (0.942%); 1.0 MiB/s, 16 objects/s recovering 2024-09-17T16:36:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3011544996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:36:45.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:45.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:45.258 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:45.282 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:45.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:45.384 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:45.413 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:45.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:45.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:45.539 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:45.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:45.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:45.666 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:45.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:45.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:45.794 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:45.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:45.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:45.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:36:45.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:45.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:46.021 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:186285 6.f 6:ff66d09f:::benchmark_data_smithi173_102017_object93137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e621)", 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 448.48088096200001, 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53962611299999996, 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186285 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536599+0000", 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4900000000000005e-07 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536603+0000", 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0380000000000001e-06 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536614+0000", 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1097000000000001e-05 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536621+0000", 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5560000000000004e-06 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540416+0000", 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037954429999999999 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540537+0000", 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012064499999999999 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540685+0000", 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014845900000000001 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.544648+0000", 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039628129999999999 2024-09-17T16:36:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.800489+0000", 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25584077599999999 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076152+0000", 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27566357899999999 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076194+0000", 2024-09-17T16:36:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1468999999999999e-05 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076224+0000", 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0488999999999999e-05 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:26018 7.8 7:151f2f24:::benchmark_data_smithi173_116396_object26017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-17T16:36:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 401.98134934400002, 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35239512200000001, 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26018 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:36:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036131+0000", 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.029e-06 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036175+0000", 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4579000000000001e-05 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036188+0000", 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.218e-05 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036203+0000", 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5494000000000001e-05 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036337+0000", 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013415599999999999 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036422+0000", 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4427999999999999e-05 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036500+0000", 2024-09-17T16:36:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8009000000000005e-05 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037194+0000", 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00069466700000000005 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037673+0000", 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047842099999999998 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388425+0000", 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35075205700000001 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388482+0000", 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7630999999999998e-05 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388525+0000", 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2471e-05 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:26018 7.8 e645/642 ondisk, result = 0)", 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 401.98007674199999, 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35113110600000003, 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:36:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:36:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037403+0000", 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8700000000000001e-07 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037406+0000", 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8710000000000001e-06 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388344+0000", 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35093866000000001 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388350+0000", 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388413+0000", 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2257999999999999e-05 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388420+0000", 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0450000000000003e-06 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388534+0000", 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011397599999999999 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:119650 7.8 7:17d95773:::benchmark_data_smithi173_116396_object24050:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:58.385278+0000", 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 287.63220073100001, 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30032905500000001, 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 119650 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385278+0000", 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385278+0000", 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385280+0000", 2024-09-17T16:36:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6300000000000001e-06 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385286+0000", 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2049999999999999e-06 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385298+0000", 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1477e-05 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.385305+0000", 2024-09-17T16:36:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9890000000000003e-06 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.388017+0000", 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027118770000000001 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.388140+0000", 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012355600000000001 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.388258+0000", 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118041 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.538443+0000", 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.150185233 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.546099+0000", 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007656014 2024-09-17T16:36:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.685478+0000", 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.13937803200000001 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.685520+0000", 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2787000000000002e-05 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:58.685608+0000", 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7213999999999995e-05 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27088.0:120213 7.3 e687/674 7:c59654a4:::benchmark_data_smithi173_116396_object24613:head v 687'6950, mlcod=686'6947)", 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.709364+0000", 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 286.30811522599998, 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28146429899999997, 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709364+0000", 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709364+0000", 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709365+0000", 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8700000000000004e-07 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.709373+0000", 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7330000000000003e-06 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989058+0000", 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.279684973 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989065+0000", 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1300000000000003e-06 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989111+0000", 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6569000000000001e-05 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989134+0000", 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2612e-05 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.990790+0000", 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001655807 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.990828+0000", 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8488000000000001e-05 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:120224 7.18 7:1c79604a:::benchmark_data_smithi173_116396_object24624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-17T16:36:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.730992+0000", 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 286.28648687999998, 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.274893622, 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 120224 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.730992+0000", 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.730992+0000", 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.730994+0000", 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8899999999999999e-06 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731002+0000", 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.379e-06 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731015+0000", 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3892e-05 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731024+0000", 2024-09-17T16:36:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9469999999999992e-06 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731315+0000", 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029021800000000002 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731451+0000", 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001359 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,7,8", 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.731569+0000", 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118727 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.751547+0000", 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019978124 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.751573+0000", 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5265e-05 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989139+0000", 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23756648699999999 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.004357+0000", 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015217849 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.004409+0000", 2024-09-17T16:36:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1891000000000002e-05 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.005886+0000", 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0014770530000000001 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(7.13 687/677 [PushReplyOp(7:cc214882:::benchmark_data_smithi173_116396_object24630:head)])", 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.748767+0000", 2024-09-17T16:36:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 286.268712585, 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.306875487, 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748767+0000", 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748767+0000", 2024-09-17T16:36:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748768+0000", 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7499999999999998e-07 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.748773+0000", 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4550000000000003e-06 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.052878+0000", 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30410503 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.052883+0000", 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2650000000000004e-06 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.053064+0000", 2024-09-17T16:36:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018071799999999999 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.055642+0000", 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0025780439999999998 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:120235 7.e 7:752ed815:::benchmark_data_smithi173_116396_object24635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:59.749714+0000", 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 286.26776567100001, 2024-09-17T16:36:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27885106300000001, 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 120235 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749714+0000", 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749714+0000", 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749715+0000", 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749721+0000", 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9909999999999999e-06 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749732+0000", 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0806e-05 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749738+0000", 2024-09-17T16:36:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.037e-06 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749792+0000", 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3906999999999997e-05 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749896+0000", 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103936 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.749991+0000", 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4524000000000005e-05 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989005+0000", 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23901425800000001 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:59.989092+0000", 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7128999999999999e-05 2024-09-17T16:36:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.028465+0000", 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.039372712999999997 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.028518+0000", 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3122999999999999e-05 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:00.028565+0000", 2024-09-17T16:36:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6808000000000003e-05 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:128523 7.e 7:78ef58c9:::benchmark_data_smithi173_116396_object32923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e693)", 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:16.133568+0000", 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 269.883910872, 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30194066800000002, 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 128523 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133568+0000", 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133568+0000", 2024-09-17T16:36:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133570+0000", 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5069999999999999e-06 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133576+0000", 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4320000000000004e-06 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133593+0000", 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6489000000000001e-05 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.133602+0000", 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9889999999999998e-06 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.141144+0000", 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075426709999999999 2024-09-17T16:36:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.141261+0000", 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011694299999999999 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.141386+0000", 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012499500000000001 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148436+0000", 2024-09-17T16:36:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070501180000000002 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.150261+0000", 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018242969999999999 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435410+0000", 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28514896200000001 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435468+0000", 2024-09-17T16:36:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8149999999999997e-05 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435509+0000", 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1115000000000002e-05 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:128523 7.e e693/629 ondisk, result = 0)", 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:36:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 269.86863439400003, 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28666948399999997, 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:36:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148845+0000", 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3999999999999997e-07 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.148847+0000", 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.575e-06 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435339+0000", 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28649269199999999 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435344+0000", 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5809999999999998e-06 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435399+0000", 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4598999999999997e-05 2024-09-17T16:36:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435405+0000", 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.049e-06 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:16.435514+0000", 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000109648 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:130635 7.8 7:14a1b993:::benchmark_data_smithi173_116396_object35035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-09-17T16:36:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:36:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 265.884695249, 2024-09-17T16:36:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.329086555, 2024-09-17T16:36:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 130635 2024-09-17T16:36:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132785+0000", 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5239999999999999e-06 2024-09-17T16:36:46.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132792+0000", 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4860000000000002e-06 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132805+0000", 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2792999999999999e-05 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132812+0000", 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1899999999999998e-06 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.133889+0000", 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001076881 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.134010+0000", 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120796 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.134124+0000", 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114875 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.136882+0000", 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027574539999999999 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.136893+0000", 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.154e-05 2024-09-17T16:36:46.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461760+0000", 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32486675399999998 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461812+0000", 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2231000000000002e-05 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461870+0000", 2024-09-17T16:36:46.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8031000000000001e-05 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:130635 7.8 e695/674 ondisk, result = 0)", 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:20.138684+0000", 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 265.87879536700001, 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32319844399999997, 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138684+0000", 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138684+0000", 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138685+0000", 2024-09-17T16:36:46.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.138690+0000", 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4659999999999998e-06 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461660+0000", 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.322969489 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461673+0000", 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3807e-05 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461745+0000", 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1297000000000005e-05 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461754+0000", 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0270000000000001e-06 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461882+0000", 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012858100000000001 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178141 7.e 7:7ed26a83:::benchmark_data_smithi173_116396_object82541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.064783+0000", 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 189.95269586000001, 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29001914499999998, 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178141 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064783+0000", 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064783+0000", 2024-09-17T16:36:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064784+0000", 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.128e-06 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064789+0000", 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.532e-06 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064800+0000", 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1229999999999999e-05 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064807+0000", 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0979999999999996e-06 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.064882+0000", 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4942000000000002e-05 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.065014+0000", 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013223299999999999 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.065138+0000", 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012379200000000001 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.068993+0000", 2024-09-17T16:36:46.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038547109999999998 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.069508+0000", 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00051509400000000001 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.354728+0000", 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.28522005099999997 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.354766+0000", 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8275000000000002e-05 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.354802+0000", 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6059000000000002e-05 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27088.0:178656 7.17 e719/707 7:ecd8c6ba:::benchmark_data_smithi173_116396_object83056:head v 719'8589, mlcod=719'8586)", 2024-09-17T16:36:46.098 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 187.98490149899999, 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27508470800000001, 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032578+0000", 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.032584+0000", 2024-09-17T16:36:46.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0349999999999998e-06 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305420+0000", 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27283574900000002 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305426+0000", 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9540000000000001e-06 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305471+0000", 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4836000000000002e-05 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.305488+0000", 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7028e-05 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.307628+0000", 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021396359999999999 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:38.307662+0000", 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4724999999999997e-05 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:191190 7.e 7:7081439d:::benchmark_data_smithi173_116396_object95590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 171.56439426099999, 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.428509103, 2024-09-17T16:36:46.103 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 191190 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453086+0000", 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3510000000000001e-06 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453092+0000", 2024-09-17T16:36:46.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6010000000000002e-06 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453103+0000", 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.065e-05 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453110+0000", 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1160000000000001e-06 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453277+0000", 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016785300000000001 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453458+0000", 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018056699999999999 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453577+0000", 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118678 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.458776+0000", 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051995009999999996 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.458855+0000", 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8597999999999995e-05 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:36:46.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881478+0000", 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42262325899999997 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881537+0000", 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9354999999999997e-05 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881594+0000", 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6573999999999997e-05 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:191190 7.e e724/629 ondisk, result = 0)", 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 171.559501211, 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42362284100000003, 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:36:46.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457979+0000", 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457985+0000", 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9460000000000003e-06 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881412+0000", 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42342706200000002 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881419+0000", 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.525e-06 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881467+0000", 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7667999999999998e-05 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881474+0000", 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9489999999999999e-06 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881601+0000", 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126797 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.113 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.16 741/739 [PushReplyOp(8:6b780127:::benchmark_data_smithi173_130566_object3478:head)])", 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 138.22680596500001, 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49350836100000001, 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790674+0000", 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4700000000000001e-07 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790677+0000", 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6720000000000002e-06 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283734+0000", 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49305611500000002 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283742+0000", 2024-09-17T16:36:46.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7960000000000001e-06 2024-09-17T16:36:46.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283814+0000", 2024-09-17T16:36:46.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1419000000000002e-05 2024-09-17T16:36:46.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.284182+0000", 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00036781200000000002 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:104220 8.f e772/728 ondisk, result = 0)", 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:51.169917+0000", 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.847562486000001, 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.30926205600000001, 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.118 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.169917+0000", 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.169917+0000", 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.169917+0000", 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1699999999999998e-07 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.169921+0000", 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5200000000000002e-06 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.479017+0000", 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.309096437 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.479023+0000", 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3820000000000003e-06 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.479097+0000", 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3962999999999996e-05 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.479105+0000", 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2479999999999996e-06 2024-09-17T16:36:46.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.479179+0000", 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3888999999999997e-05 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:104296 8.d e773/728 8:bcfac3b0:::benchmark_data_smithi173_130566_object9321:head v 773'6646, mlcod=773'6643)", 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:51.519367+0000", 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.498112536000001, 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27254341300000001, 2024-09-17T16:36:46.122 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.519367+0000", 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.519367+0000", 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.519367+0000", 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4799999999999998e-07 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.519371+0000", 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8750000000000002e-06 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.788347+0000", 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26897626800000002 2024-09-17T16:36:46.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.788357+0000", 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7450000000000007e-06 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.788450+0000", 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3158000000000005e-05 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.788476+0000", 2024-09-17T16:36:46.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5420999999999999e-05 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.791870+0000", 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033945099999999999 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.791910+0000", 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9888000000000002e-05 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:104694 8.a e774/728 8:5a8d8c06:::benchmark_data_smithi173_130566_object9719:head v 774'6677, mlcod=774'6674)", 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:52.642152+0000", 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 53.375326925000003, 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27929387300000003, 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:52.642152+0000", 2024-09-17T16:36:46.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:52.642152+0000", 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:52.642153+0000", 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1399999999999996e-07 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:36:46.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:52.642159+0000", 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.852e-06 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:52.918431+0000", 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27627217900000001 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:52.918447+0000", 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.552e-05 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:36:46.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:52.918579+0000", 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013251199999999999 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:52.918610+0000", 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0707000000000003e-05 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:52.921408+0000", 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002797852 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:36:46.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:36:46.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:52.921446+0000", 2024-09-17T16:36:46.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8436999999999998e-05 2024-09-17T16:36:46.131 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.131 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.131 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.131 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:36:46.131 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:36:46.131 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:36:46.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:46.132 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:46.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:46.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:46.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:46.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:46.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:46.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:46.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:46.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:46.523 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:46.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:46.625 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:46.652 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:46.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:36:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:36:46.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:46 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:46.754 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:46.779 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:46.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:46.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:46.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:36:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:36:46.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:46 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:46.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:36:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:36:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:46 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:36:46.905 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:46.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:47.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:47.039 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:47.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:47.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:47.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:47.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:47.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:47.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:47.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:47.405 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:47.434 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:47.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:47.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:47.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:47.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:47.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:47.694 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:47.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:47.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:47.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:47 smithi017 ceph-mon[27571]: pgmap v1772: 27 pgs: 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42213/247893 objects degraded (17.029%); 2393/247893 objects misplaced (0.965%); 3.3 MiB/s, 53 objects/s recovering 2024-09-17T16:36:47.796 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:47.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:47 smithi071 ceph-mon[30002]: pgmap v1772: 27 pgs: 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42213/247893 objects degraded (17.029%); 2393/247893 objects misplaced (0.965%); 3.3 MiB/s, 53 objects/s recovering 2024-09-17T16:36:47.818 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:47.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:47 smithi173 ceph-mon[30530]: pgmap v1772: 27 pgs: 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 42213/247893 objects degraded (17.029%); 2393/247893 objects misplaced (0.965%); 3.3 MiB/s, 53 objects/s recovering 2024-09-17T16:36:47.897 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:36:47.898 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:36:47.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:47.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:47.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:48.055 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:48.084 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:48.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:48.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:48.210 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:48.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:48.312 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:48.341 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:48.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:48.420 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:36:48.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:48.506 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:48.535 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:48.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:48.638 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:48.668 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:48.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:48.669 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:48 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 41620/243519 objects degraded (17.091%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T16:36:48.770 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:48.794 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:48.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:48.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:48.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:48 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 41620/243519 objects degraded (17.091%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T16:36:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:48 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 41620/243519 objects degraded (17.091%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T16:36:48.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:48.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:48.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:49.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:49.052 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:49.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:49.063 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:48 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:36:48.808+0000 7f1f9cd45640 -1 osd.5 772 log_to_monitors true 2024-09-17T16:36:49.154 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:49.178 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:49.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:49.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:49.302 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:49.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:49.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:49.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:49.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:49.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:49.560 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:49.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:49.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:49.652 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:36:49.662 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:49.686 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:49.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:49.773 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:36:49.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:49.794 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:49 smithi071 ceph-mon[30002]: pgmap v1773: 27 pgs: 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 95 op/s; 41620/243519 objects degraded (17.091%); 2393/243519 objects misplaced (0.983%); 9.7 MiB/s, 156 objects/s recovering 2024-09-17T16:36:49.794 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:49 smithi071 ceph-mon[30002]: from='osd.5 [v2:172.21.15.71:6808/501265341,v1:172.21.15.71:6809/501265341]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:36:49.794 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:49 smithi071 ceph-mon[30002]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:36:49.794 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:49 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:49.794 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:49 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:49.795 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1038265827' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:49.795 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:49 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:49.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:49.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:49 smithi173 ceph-mon[30530]: pgmap v1773: 27 pgs: 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 95 op/s; 41620/243519 objects degraded (17.091%); 2393/243519 objects misplaced (0.983%); 9.7 MiB/s, 156 objects/s recovering 2024-09-17T16:36:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:49 smithi173 ceph-mon[30530]: from='osd.5 [v2:172.21.15.71:6808/501265341,v1:172.21.15.71:6809/501265341]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:36:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:49 smithi173 ceph-mon[30530]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:36:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:49 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:49 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1038265827' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:49 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:49.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:36:49.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:49.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:49 smithi017 ceph-mon[27571]: pgmap v1773: 27 pgs: 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 95 op/s; 41620/243519 objects degraded (17.091%); 2393/243519 objects misplaced (0.983%); 9.7 MiB/s, 156 objects/s recovering 2024-09-17T16:36:49.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:49 smithi017 ceph-mon[27571]: from='osd.5 [v2:172.21.15.71:6808/501265341,v1:172.21.15.71:6809/501265341]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:36:49.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:49 smithi017 ceph-mon[27571]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-17T16:36:49.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:49 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:49.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:49 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:49.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1038265827' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:49.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:49 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:36:49.940 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:49.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:50.042 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:50.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:36:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:36:50.028+0000 7f1f90ae7640 -1 osd.5 795 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:36:50.074 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:50.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:50.176 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:50.189 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:36:50.203 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:50.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:50.305 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:50.330 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:50.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:50.432 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:50.453 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:50.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:50.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:50.582 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:50.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:50.684 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:50.714 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:50.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:50 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:50 smithi071 ceph-mon[30002]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T16:36:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:50 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:50 smithi071 ceph-mon[30002]: from='osd.5 [v2:172.21.15.71:6808/501265341,v1:172.21.15.71:6809/501265341]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:36:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:50 smithi071 ceph-mon[30002]: osdmap e795: 12 total, 11 up, 12 in 2024-09-17T16:36:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:50 smithi071 ceph-mon[30002]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:36:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:50 smithi071 ceph-mon[30002]: from='osd.5 ' entity='osd.5' 2024-09-17T16:36:50.816 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:50.843 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:50.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:50 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:50 smithi173 ceph-mon[30530]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T16:36:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:50 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:50 smithi173 ceph-mon[30530]: from='osd.5 [v2:172.21.15.71:6808/501265341,v1:172.21.15.71:6809/501265341]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:36:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:50 smithi173 ceph-mon[30530]: osdmap e795: 12 total, 11 up, 12 in 2024-09-17T16:36:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:50 smithi173 ceph-mon[30530]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:36:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:50 smithi173 ceph-mon[30530]: from='osd.5 ' entity='osd.5' 2024-09-17T16:36:50.945 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:50.967 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:50.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:50 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:36:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:50 smithi017 ceph-mon[27571]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T16:36:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:50 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:36:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:50 smithi017 ceph-mon[27571]: from='osd.5 [v2:172.21.15.71:6808/501265341,v1:172.21.15.71:6809/501265341]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:36:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:50 smithi017 ceph-mon[27571]: osdmap e795: 12 total, 11 up, 12 in 2024-09-17T16:36:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:50 smithi017 ceph-mon[27571]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]} : dispatch 2024-09-17T16:36:50.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:50 smithi017 ceph-mon[27571]: from='osd.5 ' entity='osd.5' 2024-09-17T16:36:51.068 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:51.097 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:51.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:51.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:51.220 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:51.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:51.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:51.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:51.344 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:51.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:51.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:51.468 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:51.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:51.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:51.570 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:51.586 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:51.592 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:51.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:51.694 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:51.719 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:51.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:51 smithi071 ceph-mon[30002]: pgmap v1775: 27 pgs: 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 41204/236910 objects degraded (17.392%); 1991/236910 objects misplaced (0.840%); 14 MiB/s, 230 objects/s recovering 2024-09-17T16:36:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:51 smithi071 ceph-mon[30002]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T16:36:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:51 smithi071 ceph-mon[30002]: osd.5 [v2:172.21.15.71:6808/501265341,v1:172.21.15.71:6809/501265341] boot 2024-09-17T16:36:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:51 smithi071 ceph-mon[30002]: osdmap e796: 12 total, 12 up, 12 in 2024-09-17T16:36:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:36:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:51 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:51 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:51.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:51.844 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:51.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:51 smithi173 ceph-mon[30530]: pgmap v1775: 27 pgs: 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 41204/236910 objects degraded (17.392%); 1991/236910 objects misplaced (0.840%); 14 MiB/s, 230 objects/s recovering 2024-09-17T16:36:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:51 smithi173 ceph-mon[30530]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T16:36:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:51 smithi173 ceph-mon[30530]: osd.5 [v2:172.21.15.71:6808/501265341,v1:172.21.15.71:6809/501265341] boot 2024-09-17T16:36:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:51 smithi173 ceph-mon[30530]: osdmap e796: 12 total, 12 up, 12 in 2024-09-17T16:36:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:36:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:51 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:51 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:51.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:51.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:51.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:51 smithi017 ceph-mon[27571]: pgmap v1775: 27 pgs: 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 772 op/s; 41204/236910 objects degraded (17.392%); 1991/236910 objects misplaced (0.840%); 14 MiB/s, 230 objects/s recovering 2024-09-17T16:36:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:51 smithi017 ceph-mon[27571]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T16:36:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:51 smithi017 ceph-mon[27571]: osd.5 [v2:172.21.15.71:6808/501265341,v1:172.21.15.71:6809/501265341] boot 2024-09-17T16:36:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:51 smithi017 ceph-mon[27571]: osdmap e796: 12 total, 12 up, 12 in 2024-09-17T16:36:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-17T16:36:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:51 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:51 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:52.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:52.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:52.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:52.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:52.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:52.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:52.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:52.301 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:36:52.302 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:36:52.302 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:36:52.302 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:36:52.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:52.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:52.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:52.438 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:52.460 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:52.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:52.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:52.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:52.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:52.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:52.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:52.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:52.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:52 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:52 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:52 smithi071 ceph-mon[30002]: osdmap e797: 12 total, 12 up, 12 in 2024-09-17T16:36:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:52 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:52.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:52 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:52.836 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:52.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:52 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:52 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:52 smithi173 ceph-mon[30530]: osdmap e797: 12 total, 12 up, 12 in 2024-09-17T16:36:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:52 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:52.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:52 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:52.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:52.964 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:52.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:52.969 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T16:36:52.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:52 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:52.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:52 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:52.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:52 smithi017 ceph-mon[27571]: osdmap e797: 12 total, 12 up, 12 in 2024-09-17T16:36:52.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:52 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:52.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:52 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:53.066 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:53.099 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:53.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:53.202 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:53.244 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:53.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:53.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:53.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:53.379 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:53.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:53.481 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:53.512 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:53.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:53 smithi071 ceph-mon[30002]: pgmap v1778: 27 pgs: 1 remapped, 1 remapped+peering, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 33698/232956 objects degraded (14.465%); 1991/232956 objects misplaced (0.855%); 22 MiB/s, 351 objects/s recovering 2024-09-17T16:36:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:53 smithi071 ceph-mon[30002]: osdmap e798: 12 total, 12 up, 12 in 2024-09-17T16:36:53.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:53.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:53.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:53.746 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:53.782 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:53.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:53.884 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:53 smithi173 ceph-mon[30530]: pgmap v1778: 27 pgs: 1 remapped, 1 remapped+peering, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 33698/232956 objects degraded (14.465%); 1991/232956 objects misplaced (0.855%); 22 MiB/s, 351 objects/s recovering 2024-09-17T16:36:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:53 smithi173 ceph-mon[30530]: osdmap e798: 12 total, 12 up, 12 in 2024-09-17T16:36:53.921 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:53.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:53.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:53 smithi017 ceph-mon[27571]: pgmap v1778: 27 pgs: 1 remapped, 1 remapped+peering, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 33698/232956 objects degraded (14.465%); 1991/232956 objects misplaced (0.855%); 22 MiB/s, 351 objects/s recovering 2024-09-17T16:36:53.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:53 smithi017 ceph-mon[27571]: osdmap e798: 12 total, 12 up, 12 in 2024-09-17T16:36:54.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:54.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:54.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:54.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:54.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:54.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:54.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:36:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:54.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:54.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:36:54.432 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:54.458 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:54.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:54.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:54.587 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:54.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:54.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:54.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:54.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:54.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:54 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 33113/230643 objects degraded (14.357%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T16:36:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:54 smithi071 ceph-mon[30002]: osdmap e799: 12 total, 12 up, 12 in 2024-09-17T16:36:54.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:54.840 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:54.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:54.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:54 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 33113/230643 objects degraded (14.357%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T16:36:54.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:54 smithi173 ceph-mon[30530]: osdmap e799: 12 total, 12 up, 12 in 2024-09-17T16:36:54.900 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:54 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 33113/230643 objects degraded (14.357%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T16:36:54.900 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:54 smithi017 ceph-mon[27571]: osdmap e799: 12 total, 12 up, 12 in 2024-09-17T16:36:54.910 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:36:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:36:54.910 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:36:54.910 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:36:54.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:54.966 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:54.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:55.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:55.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:55.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:55.142 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:55.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:55.216 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:55.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:55.318 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:55.342 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:55.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:55.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:55.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:55.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:55.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:55.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:55.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:55.595 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:55.632 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:36:55.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:55.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:55.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:55.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:55.774 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:55 smithi017 ceph-mon[27571]: pgmap v1780: 27 pgs: 1 remapped, 1 remapped+peering, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 33113/230643 objects degraded (14.357%); 20 MiB/s, 319 objects/s recovering 2024-09-17T16:36:55.774 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:55 smithi017 ceph-mon[27571]: osdmap e800: 12 total, 12 up, 12 in 2024-09-17T16:36:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:55 smithi071 ceph-mon[30002]: pgmap v1780: 27 pgs: 1 remapped, 1 remapped+peering, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 33113/230643 objects degraded (14.357%); 20 MiB/s, 319 objects/s recovering 2024-09-17T16:36:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:55 smithi071 ceph-mon[30002]: osdmap e800: 12 total, 12 up, 12 in 2024-09-17T16:36:55.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:55.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:55.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:55 smithi173 ceph-mon[30530]: pgmap v1780: 27 pgs: 1 remapped, 1 remapped+peering, 2 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 769 op/s; 33113/230643 objects degraded (14.357%); 20 MiB/s, 319 objects/s recovering 2024-09-17T16:36:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:55 smithi173 ceph-mon[30530]: osdmap e800: 12 total, 12 up, 12 in 2024-09-17T16:36:55.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:55.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:55.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:56.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:56.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:56.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:56.107 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:56.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:56.231 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:56.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:56.276 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:36:56.333 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:56.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:36:56.357 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:56.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:56.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:56.459 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:56.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:56.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:56.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:56.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:56.715 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:56.739 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:56.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:56.743 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T16:36:56.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:56.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:56.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:56.966 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:56 smithi017 ceph-mon[27571]: osdmap e801: 12 total, 12 up, 12 in 2024-09-17T16:36:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:56 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:36:56.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:56 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:36:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:56 smithi017 ceph-mon[27571]: pgmap v1784: 27 pgs: 1 active+recovering+undersized+remapped, 1 remapped, 1 remapped+peering, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 507 op/s; 30555/226863 objects degraded (13.468%); 9.4 MiB/s, 176 objects/s recovering 2024-09-17T16:36:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/427966622' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:36:56.990 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:56 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:36:56.998 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:57.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:56 smithi071 ceph-mon[30002]: osdmap e801: 12 total, 12 up, 12 in 2024-09-17T16:36:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:56 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:36:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:56 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:36:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:56 smithi071 ceph-mon[30002]: pgmap v1784: 27 pgs: 1 active+recovering+undersized+remapped, 1 remapped, 1 remapped+peering, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 507 op/s; 30555/226863 objects degraded (13.468%); 9.4 MiB/s, 176 objects/s recovering 2024-09-17T16:36:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/427966622' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:36:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:56 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:36:57.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:36:57.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:57.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:57.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:36:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:56 smithi173 ceph-mon[30530]: osdmap e801: 12 total, 12 up, 12 in 2024-09-17T16:36:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:56 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:36:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:56 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:36:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:56 smithi173 ceph-mon[30530]: pgmap v1784: 27 pgs: 1 active+recovering+undersized+remapped, 1 remapped, 1 remapped+peering, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 13 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 507 op/s; 30555/226863 objects degraded (13.468%); 9.4 MiB/s, 176 objects/s recovering 2024-09-17T16:36:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/427966622' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:36:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:56 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:36:57.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:57.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:57.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:57.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:36:57.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:57.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:57.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:36:57.477 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:36:57.498 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:57.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:57.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:36:57.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:57.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:57.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:57.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:57.751 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:57.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:57.853 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:57.879 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:57.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:57.981 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:57.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:57 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:36:57.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:57 smithi017 ceph-mon[27571]: osdmap e802: 12 total, 12 up, 12 in 2024-09-17T16:36:58.006 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:58.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:57 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:36:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:57 smithi071 ceph-mon[30002]: osdmap e802: 12 total, 12 up, 12 in 2024-09-17T16:36:58.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:58.133 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:58.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:57 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:36:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:57 smithi173 ceph-mon[30530]: osdmap e802: 12 total, 12 up, 12 in 2024-09-17T16:36:58.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:36:58.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:58.263 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:58.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:36:58.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:58.387 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:58.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:58.489 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:36:58.520 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:58.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:58.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:36:58.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:36:58.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:58.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:36:58.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:58.774 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:58.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:58.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:58.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:58.904 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:58.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:59.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:59.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:59.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:58 smithi071 ceph-mon[30002]: osdmap e803: 12 total, 12 up, 12 in 2024-09-17T16:36:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:58 smithi071 ceph-mon[30002]: pgmap v1787: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 14 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s; 31689/224151 objects degraded (14.137%); 16 MiB/s, 421 objects/s recovering 2024-09-17T16:36:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:58 smithi071 ceph-mon[30002]: 8.18 scrub starts 2024-09-17T16:36:59.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:58 smithi017 ceph-mon[27571]: osdmap e803: 12 total, 12 up, 12 in 2024-09-17T16:36:59.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:58 smithi017 ceph-mon[27571]: pgmap v1787: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 14 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s; 31689/224151 objects degraded (14.137%); 16 MiB/s, 421 objects/s recovering 2024-09-17T16:36:59.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:58 smithi017 ceph-mon[27571]: 8.18 scrub starts 2024-09-17T16:36:59.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:59.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:58 smithi173 ceph-mon[30530]: osdmap e803: 12 total, 12 up, 12 in 2024-09-17T16:36:59.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:58 smithi173 ceph-mon[30530]: pgmap v1787: 27 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 14 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s; 31689/224151 objects degraded (14.137%); 16 MiB/s, 421 objects/s recovering 2024-09-17T16:36:59.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:58 smithi173 ceph-mon[30530]: 8.18 scrub starts 2024-09-17T16:36:59.171 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:59.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:59.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:36:59.298 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:59.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:36:59.400 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:36:59.421 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:59.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:36:59.523 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:36:59.546 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:59.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:36:59.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:36:59.673 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:36:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:59.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:36:59.775 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:36:59.806 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:36:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:59.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:36:59.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:36:59.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:36:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:36:59.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:36:59.992 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:36:59.993 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:37:00.023 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2163033592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:00.024 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:36:59 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 31689/224151 objects degraded (14.137%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:00.024 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:37:00.024 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:37:00.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:00.053 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:37:00.053 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:37:00.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2163033592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:00.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:36:59 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 31689/224151 objects degraded (14.137%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:00.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:00.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:00.083 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:37:00.083 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:37:00.112 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:37:00.113 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:37:00.141 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:37:00.141 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:37:00.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2163033592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:00.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:36:59 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 31689/224151 objects degraded (14.137%), 12 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:00.171 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:00.172 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:37:00.173 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:37:00.202 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:00.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:00.204 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:37:00.204 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:37:00.229 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:37:00.229 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:37:00.255 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:37:00.255 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:37:00.280 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:37:00.280 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:37:00.304 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:00.305 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:37:00.305 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:37:00.328 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:00.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:00.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:00.430 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:00.461 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:00.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:00.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:00.592 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:00.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:00.694 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:00.716 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:00.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:00.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:00.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:00.841 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:37:00.841 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:37:00.842 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:00 smithi017 ceph-mon[27571]: pgmap v1788: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 29214/218433 objects degraded (13.374%); 11 MiB/s, 377 objects/s recovering 2024-09-17T16:37:00.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:00.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:00.946 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:00.974 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:00.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:00 smithi071 ceph-mon[30002]: pgmap v1788: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 29214/218433 objects degraded (13.374%); 11 MiB/s, 377 objects/s recovering 2024-09-17T16:37:01.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:01.097 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:01.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:00 smithi173 ceph-mon[30530]: pgmap v1788: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 29214/218433 objects degraded (13.374%); 11 MiB/s, 377 objects/s recovering 2024-09-17T16:37:01.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:01.229 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:01.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:01.331 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:01.354 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:01.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:01.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:01.457 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:01.479 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:01.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:01.581 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:01.616 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:01.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:01.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:01.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:01.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:01.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:01.875 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:01.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:01.977 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:02.008 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:02.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:02.110 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:02.132 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:02.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:02.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:02.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:02.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:02.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:02.362 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:02.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:02.385 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:02.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:02.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:02.507 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:02.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:02.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:02.633 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:02.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:02.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:02.735 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:02.757 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:02.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:02.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:02.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:02.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:02.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:03.012 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:03.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:03.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:03.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:03.142 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:03.243 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:03.270 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:03.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:03.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:03.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:03.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:03.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:03 smithi017 ceph-mon[27571]: pgmap v1789: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 28272/214524 objects degraded (13.179%); 18 MiB/s, 456 objects/s recovering 2024-09-17T16:37:03.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:03.530 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:03.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:03.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:03 smithi071 ceph-mon[30002]: pgmap v1789: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 28272/214524 objects degraded (13.179%); 18 MiB/s, 456 objects/s recovering 2024-09-17T16:37:03.632 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:03 smithi173 ceph-mon[30530]: pgmap v1789: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 28272/214524 objects degraded (13.179%); 18 MiB/s, 456 objects/s recovering 2024-09-17T16:37:03.666 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:03.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:03.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:03.801 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:03.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:03.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:03.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:03.929 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:04.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:04.054 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:04.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:04.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:04.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:04.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:04.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:04.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:04.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:04.415 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:04.436 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:04.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:04 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:37:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:04 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:37:04.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:04 smithi017 ceph-mon[27571]: osdmap e804: 12 total, 12 up, 12 in 2024-09-17T16:37:04.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:04.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:04 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:37:04.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:04 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:37:04.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:04 smithi071 ceph-mon[30002]: osdmap e804: 12 total, 12 up, 12 in 2024-09-17T16:37:04.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:04.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:04.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:04 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:37:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:04 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:37:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:04 smithi173 ceph-mon[30530]: osdmap e804: 12 total, 12 up, 12 in 2024-09-17T16:37:04.672 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:04.693 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:04.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:04.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:04.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:04.820 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:04.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:04.923 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:04.949 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:04.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:04.996 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:37:05.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:05.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:05.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:05.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:05.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:05.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:05.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:05.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:05.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:05.432 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:05.464 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:05.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:05.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:05 smithi017 ceph-mon[27571]: pgmap v1791: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 660 op/s; 27857/210987 objects degraded (13.203%); 15 MiB/s, 311 objects/s recovering 2024-09-17T16:37:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:05 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 27857/210987 objects degraded (13.203%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:05.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:05 smithi017 ceph-mon[27571]: osdmap e805: 12 total, 12 up, 12 in 2024-09-17T16:37:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:05 smithi071 ceph-mon[30002]: pgmap v1791: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 660 op/s; 27857/210987 objects degraded (13.203%); 15 MiB/s, 311 objects/s recovering 2024-09-17T16:37:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:05 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 27857/210987 objects degraded (13.203%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:05 smithi071 ceph-mon[30002]: osdmap e805: 12 total, 12 up, 12 in 2024-09-17T16:37:05.566 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:05.599 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:05.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:05 smithi173 ceph-mon[30530]: pgmap v1791: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 660 op/s; 27857/210987 objects degraded (13.203%); 15 MiB/s, 311 objects/s recovering 2024-09-17T16:37:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:05 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 27857/210987 objects degraded (13.203%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:05 smithi173 ceph-mon[30530]: osdmap e805: 12 total, 12 up, 12 in 2024-09-17T16:37:05.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:05.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:05.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:05.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:05.830 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T16:37:05.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:05.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:05.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:05.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:05.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:05.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:05.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:06.090 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:06.115 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:06.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:06.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:06.237 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:06.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:06.266 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:06.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:06.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:06.365 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:06.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:06.502 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:06.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:06.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:06.521 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:06.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:06.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:06.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:06.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:06.763 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:06.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:06.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:06.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:06.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:06.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:06.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:07.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:07.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:07.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:07.151 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:07.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:07.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:07.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:07.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:07.385 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:07.411 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:07.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:07 smithi017 ceph-mon[27571]: pgmap v1793: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 27856/204975 objects degraded (13.590%); 11 MiB/s, 241 objects/s recovering 2024-09-17T16:37:07.512 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:07.541 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:07.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:07 smithi071 ceph-mon[30002]: pgmap v1793: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 27856/204975 objects degraded (13.590%); 11 MiB/s, 241 objects/s recovering 2024-09-17T16:37:07.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:07 smithi173 ceph-mon[30530]: pgmap v1793: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 27856/204975 objects degraded (13.590%); 11 MiB/s, 241 objects/s recovering 2024-09-17T16:37:07.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:07.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:07.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:07.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:07.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:07.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:07.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:07.933 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:07.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:08.035 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:08.070 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:08.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:08.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:08.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:08.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:08.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:08.262 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:37:08.304 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:08.326 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:08.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:08.428 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:08.450 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:08.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:08.552 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:08.581 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:08.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:08.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:08.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:08.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:08.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:08.842 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:08.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:08.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:08.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:08.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:09.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:09.106 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:09.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:09.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:09.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:09.232 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:09.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:09.360 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:09.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:09.446 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:09 smithi173 ceph-mon[30530]: pgmap v1794: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s; 26336/201282 objects degraded (13.084%); 22 MiB/s, 366 objects/s recovering 2024-09-17T16:37:09.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:09.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:09.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:09.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:09 smithi017 ceph-mon[27571]: pgmap v1794: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s; 26336/201282 objects degraded (13.084%); 22 MiB/s, 366 objects/s recovering 2024-09-17T16:37:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:09 smithi071 ceph-mon[30002]: pgmap v1794: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s; 26336/201282 objects degraded (13.084%); 22 MiB/s, 366 objects/s recovering 2024-09-17T16:37:09.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:09.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:09.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:09.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:09.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:09.739 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:09.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:09.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:09.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:09.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:09.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:09.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:09.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:09.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:10.099 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:10.131 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:10.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:10.233 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:10.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:10.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:10.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:10.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:10.388 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:10.488 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 26336/201282 objects degraded (13.084%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:10.519 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:10.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:10.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 26336/201282 objects degraded (13.084%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:10.621 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:10.643 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:10.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:10.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 26336/201282 objects degraded (13.084%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:10.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:10.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:10.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:10.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:10.896 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:10.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:10.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:11.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:11.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:11.112 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:11.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:11.178 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:11.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:11.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:11.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:11.313 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:11.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:11.414 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:11.439 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:11.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:11.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:11 smithi017 ceph-mon[27571]: pgmap v1795: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s; 26334/194982 objects degraded (13.506%); 15 MiB/s, 242 objects/s recovering 2024-09-17T16:37:11.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1058939749' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:37:11.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:11 smithi071 ceph-mon[30002]: pgmap v1795: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s; 26334/194982 objects degraded (13.506%); 15 MiB/s, 242 objects/s recovering 2024-09-17T16:37:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1058939749' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:37:11.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:11.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:11.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:11 smithi173 ceph-mon[30530]: pgmap v1795: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s; 26334/194982 objects degraded (13.506%); 15 MiB/s, 242 objects/s recovering 2024-09-17T16:37:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1058939749' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:37:11.670 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:11.693 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:11.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:11.715 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:11.739 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:37:11.740 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T16:37:11.750 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:37:11.750 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:37:11.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:11.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:11.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:11.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:11.955 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:11.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:12.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:12.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:12.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:12.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:12.219 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:12.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:12.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:12.345 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:12.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:12.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:12.483 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:12.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:12.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3214468357' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:37:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:12 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:37:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:12 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:37:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1058939749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:37:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:12 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:37:12.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:12 smithi017 ceph-mon[27571]: osdmap e806: 12 total, 12 up, 12 in 2024-09-17T16:37:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3214468357' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:37:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:12 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:37:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:12 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:37:12.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1058939749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:37:12.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:12 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:37:12.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:12 smithi071 ceph-mon[30002]: osdmap e806: 12 total, 12 up, 12 in 2024-09-17T16:37:12.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:12.608 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:12.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3214468357' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:37:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:12 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:37:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:12 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:37:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1058939749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:37:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:12 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:37:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:12 smithi173 ceph-mon[30530]: osdmap e806: 12 total, 12 up, 12 in 2024-09-17T16:37:12.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:12.742 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:12.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:12.844 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:12.874 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:12.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:12.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:13.001 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:13.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:13.104 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:13.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:13.127 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:13.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:13.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:13.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:13.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:13.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:13.390 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:13.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:13.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:13.470 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:13 smithi017 ceph-mon[27571]: pgmap v1797: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s; 25580/190971 objects degraded (13.395%); 18 MiB/s, 284 objects/s recovering 2024-09-17T16:37:13.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:13.513 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:13.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:13.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:13 smithi071 ceph-mon[30002]: pgmap v1797: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s; 25580/190971 objects degraded (13.395%); 18 MiB/s, 284 objects/s recovering 2024-09-17T16:37:13.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:13.626 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:37:13.627 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:37:13.638 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:13.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:13.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:13 smithi173 ceph-mon[30530]: pgmap v1797: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 833 op/s; 25580/190971 objects degraded (13.395%); 18 MiB/s, 284 objects/s recovering 2024-09-17T16:37:13.657 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:37:13.657 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:37:13.685 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:37:13.686 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:37:13.717 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:37:13.717 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:37:13.741 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:13.748 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:37:13.748 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:37:13.766 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:13.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:13.781 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:37:13.781 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:37:13.810 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:37:13.811 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:37:13.840 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:37:13.841 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:37:13.865 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:37:13.865 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:37:13.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:13.894 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:37:13.894 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:37:13.896 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:13.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:13.928 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:37:13.928 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:37:13.958 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:37:13.959 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:37:13.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:14.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:14.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:14.133 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:14.155 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:14.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:14.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:14.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:14.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:14.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:14.409 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:14.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:14.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:37:14.474 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/415197122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/415197122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:14.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:14.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:14.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/415197122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:14.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:14.678 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:14.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:14.751 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:14.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:14.808 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:14.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:14.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:14.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:15.043 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:15.067 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:15.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:15.086 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-17T16:37:15.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:15.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:15.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:15.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:15.300 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:15.325 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:15.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:15.341 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:37:15.383 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:15 smithi017 ceph-mon[27571]: pgmap v1798: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 25252/187029 objects degraded (13.502%); 17 MiB/s, 277 objects/s recovering 2024-09-17T16:37:15.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:15 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 25252/187029 objects degraded (13.502%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:15.384 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:15 smithi017 ceph-mon[27571]: osdmap e807: 12 total, 12 up, 12 in 2024-09-17T16:37:15.425 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:15 smithi071 ceph-mon[30002]: pgmap v1798: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 25252/187029 objects degraded (13.502%); 17 MiB/s, 277 objects/s recovering 2024-09-17T16:37:15.425 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:15 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 25252/187029 objects degraded (13.502%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:15.425 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:15 smithi071 ceph-mon[30002]: osdmap e807: 12 total, 12 up, 12 in 2024-09-17T16:37:15.427 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:15.449 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:37:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:37:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:131474 8.0 8:07d44f96:::benchmark_data_smithi173_130566_object36499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e807)", 2024-09-17T16:37:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:15.422584+0000", 2024-09-17T16:37:15.449 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.022901076999999999, 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.022927284999999999, 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 131474 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.450 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.422584+0000", 2024-09-17T16:37:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.422584+0000", 2024-09-17T16:37:15.451 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.422585+0000", 2024-09-17T16:37:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3069999999999999e-06 2024-09-17T16:37:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.422590+0000", 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8810000000000004e-06 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.422602+0000", 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2632999999999999e-05 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.422610+0000", 2024-09-17T16:37:15.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1670000000000002e-06 2024-09-17T16:37:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.422665+0000", 2024-09-17T16:37:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5603999999999998e-05 2024-09-17T16:37:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:15.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.422798+0000", 2024-09-17T16:37:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000132869 2024-09-17T16:37:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:37:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.422886+0000", 2024-09-17T16:37:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7855000000000005e-05 2024-09-17T16:37:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:15.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.424934+0000", 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002047751 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.425539+0000", 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00060485499999999995 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.17883.0:131490 8.b e807/728 8:df995e49:::benchmark_data_smithi173_130566_object36515:head v 807'8314, mlcod=807'8311)", 2024-09-17T16:37:15.456 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:15.438762+0000", 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0067229100000000003, 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0068170720000000004, 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.438762+0000", 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.457 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.438762+0000", 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.438763+0000", 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9599999999999998e-07 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.438769+0000", 2024-09-17T16:37:15.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.438e-06 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.438809+0000", 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9595000000000003e-05 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.438817+0000", 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7159999999999992e-06 2024-09-17T16:37:15.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.438861+0000", 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3479999999999997e-05 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.438878+0000", 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7326999999999999e-05 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:15.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.17883.0:131496 8.e e807/728 8:7b0f005d:::benchmark_data_smithi173_130566_object36521:head v 807'8141, mlcod=807'8138)", 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:15.443992+0000", 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0014927219999999999, 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016293740000000001, 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.443992+0000", 2024-09-17T16:37:15.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.443992+0000", 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.443993+0000", 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.124e-06 2024-09-17T16:37:15.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444001+0000", 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4749999999999996e-06 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444060+0000", 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9147000000000001e-05 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:15.463 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444067+0000", 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7109999999999995e-06 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444201+0000", 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013332599999999999 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444220+0000", 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9352e-05 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:15.464 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.17883.0:131498 8.b e807/728 8:dd7b0419:::benchmark_data_smithi173_130566_object36523:head v 807'8315, mlcod=807'8312)", 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:15.444010+0000", 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.001474424, 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016542989999999999, 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444010+0000", 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444010+0000", 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444011+0000", 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3699999999999999e-07 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444014+0000", 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9919999999999999e-06 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444094+0000", 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9977999999999999e-05 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444098+0000", 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6380000000000003e-06 2024-09-17T16:37:15.467 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444140+0000", 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1588999999999998e-05 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:15.444159+0000", 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9559999999999999e-05 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:15.468 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:37:15.469 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 4 2024-09-17T16:37:15.469 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:37:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:15.470 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:15.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:15.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:15.601 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:15.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:15 smithi173 ceph-mon[30530]: pgmap v1798: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 757 op/s; 25252/187029 objects degraded (13.502%); 17 MiB/s, 277 objects/s recovering 2024-09-17T16:37:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:15 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 25252/187029 objects degraded (13.502%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:15 smithi173 ceph-mon[30530]: osdmap e807: 12 total, 12 up, 12 in 2024-09-17T16:37:15.671 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:15.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:15.726 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:15.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:15.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:15.862 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:15.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:15.963 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:15.994 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:15.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:16.096 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:16.120 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:16.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:16.191 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:37:16.222 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:16.256 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:16.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:16.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:16.334 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:16.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:16.381 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:16.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:16.481 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:16.483 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:16.511 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:16.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:16.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:16.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:16.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:16.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:16.773 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:16.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:16.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:16.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:16.918 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:16.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:17.020 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:17.048 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:17.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:17.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:17.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:17.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:17.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:17.203 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T16:37:17.224 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:37:17.224 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:37:17.224 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:37:17.225 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:37:17.225 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:37:17.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:17.317 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:17.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:17.419 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:17.446 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:17.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:17 smithi017 ceph-mon[27571]: pgmap v1800: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 25245/179751 objects degraded (14.044%); 8.4 MiB/s, 136 objects/s recovering 2024-09-17T16:37:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3499408219' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:37:17.548 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:17.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:17 smithi071 ceph-mon[30002]: pgmap v1800: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 25245/179751 objects degraded (14.044%); 8.4 MiB/s, 136 objects/s recovering 2024-09-17T16:37:17.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3499408219' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:37:17.571 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:17.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:17.600 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:37:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:17 smithi173 ceph-mon[30530]: pgmap v1800: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 25245/179751 objects degraded (14.044%); 8.4 MiB/s, 136 objects/s recovering 2024-09-17T16:37:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3499408219' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:37:17.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:17.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:17.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:17.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:17.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:17.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:17.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:17.928 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:17.958 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:17.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:17.996 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T16:37:18.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:18.060 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:18.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:18.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:18.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:18.209 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:37:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:18.212 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:18.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:18.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3499408219' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:37:18.221 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:37:18.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:18.349 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:18.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:18.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:18.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:18.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:18 smithi017 ceph-mon[27571]: osdmap e808: 12 total, 12 up, 12 in 2024-09-17T16:37:18.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2132285231' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:37:18.500 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:18.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3499408219' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:37:18.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:18 smithi071 ceph-mon[30002]: osdmap e808: 12 total, 12 up, 12 in 2024-09-17T16:37:18.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2132285231' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:37:18.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:18.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:18.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3499408219' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:37:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:18 smithi173 ceph-mon[30530]: osdmap e808: 12 total, 12 up, 12 in 2024-09-17T16:37:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2132285231' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:37:18.710 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:18.732 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:18.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:18.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:18.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:18.857 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:18.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:18.959 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:18.998 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:18.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:19.030 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:37:19.030 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:37:19.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:37:19.030 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:37:19.030 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.030 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:166371 6.e 6:7d7d0369:::benchmark_data_smithi173_102017_object73223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-17T16:37:19.030 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:28:48.730674+0000", 2024-09-17T16:37:19.030 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 510.29577722499999, 2024-09-17T16:37:19.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52497791100000002, 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 166371 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730674+0000", 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730674+0000", 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730676+0000", 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3850000000000001e-06 2024-09-17T16:37:19.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730680+0000", 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3630000000000004e-06 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730691+0000", 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0937999999999999e-05 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730696+0000", 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1030000000000001e-06 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730733+0000", 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.714e-05 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730828+0000", 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5094999999999997e-05 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,2", 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.730904+0000", 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5702000000000001e-05 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.734707+0000", 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038030859999999998 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.735615+0000", 2024-09-17T16:37:19.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00090786100000000004 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255546+0000", 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51993075499999997 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255605+0000", 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9404000000000002e-05 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255652+0000", 2024-09-17T16:37:19.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7079000000000001e-05 2024-09-17T16:37:19.037 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.037 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.037 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.037 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26665.0:166371 6.e e608/549 ondisk, result = 0)", 2024-09-17T16:37:19.037 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:28:48.736512+0000", 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 510.289939269, 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51914618000000001, 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.736512+0000", 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.736512+0000", 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.736513+0000", 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9699999999999995e-07 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:48.736520+0000", 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0899999999999999e-06 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255442+0000", 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51892181699999995 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255467+0000", 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5539000000000001e-05 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255527+0000", 2024-09-17T16:37:19.040 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0044999999999999e-05 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255537+0000", 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5670000000000001e-06 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:49.255658+0000", 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012132499999999999 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.041 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:95587 7.3 e671/640 7:c3123e23:::benchmark_data_smithi173_116396_object95586:head v 671'5966, mlcod=671'5964)", 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:31:09.928995+0000", 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 369.09745680600003, 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.84880420199999995, 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:09.928995+0000", 2024-09-17T16:37:19.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:09.928995+0000", 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:09.928996+0000", 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3799999999999996e-07 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:09.929044+0000", 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8328000000000002e-05 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.775185+0000", 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.84614102099999999 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.775196+0000", 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0978999999999999e-05 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.044 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.775293+0000", 2024-09-17T16:37:19.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6892000000000007e-05 2024-09-17T16:37:19.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.775312+0000", 2024-09-17T16:37:19.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9633e-05 2024-09-17T16:37:19.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:19.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.777762+0000", 2024-09-17T16:37:19.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024497569999999999 2024-09-17T16:37:19.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:10.777799+0000", 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6653999999999998e-05 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(8.19 741/739 [PushOp(8:9b4c75cc:::benchmark_data_smithi173_130566_object6584:head, version: 732'423, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:9b4c75cc:::benchmark_data_smithi173_130566_object6584:head@732'423, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:28.096915+0000", 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 170.92953680799999, 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.64022687700000003, 2024-09-17T16:37:19.047 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.096915+0000", 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.096915+0000", 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.096915+0000", 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6400000000000002e-07 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.097032+0000", 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000116988 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736654+0000", 2024-09-17T16:37:19.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.63962142799999999 2024-09-17T16:37:19.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.050 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736671+0000", 2024-09-17T16:37:19.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7062999999999999e-05 2024-09-17T16:37:19.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.050 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736829+0000", 2024-09-17T16:37:19.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000158808 2024-09-17T16:37:19.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736842+0000", 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3021e-05 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.737141+0000", 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00029900499999999997 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(8.19 741/739 [PushOp(8:98915622:::benchmark_data_smithi173_130566_object6588:head, version: 732'424, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:98915622:::benchmark_data_smithi173_130566_object6588:head@732'424, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:28.098526+0000", 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 170.927925472, 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.63899293999999995, 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.098526+0000", 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.098526+0000", 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.098526+0000", 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5300000000000004e-07 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.098638+0000", 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000111109 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736712+0000", 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.63807422999999996 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736726+0000", 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4058e-05 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.737217+0000", 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049099300000000001 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.737229+0000", 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2534999999999999e-05 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.737519+0000", 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00028946199999999999 2024-09-17T16:37:19.056 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(8.14 742/739 [PushOp(8:29340722:::benchmark_data_smithi173_130566_object8035:head, version: 733'489, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:29340722:::benchmark_data_smithi173_130566_object8035:head@733'489, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:28.785447+0000", 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 170.24100485100001, 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.873767925, 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.785447+0000", 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.785447+0000", 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.785447+0000", 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.785506+0000", 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8687999999999998e-05 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.658772+0000", 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87326642799999998 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.658800+0000", 2024-09-17T16:37:19.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7483000000000001e-05 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.658964+0000", 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016395400000000001 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.658998+0000", 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4059e-05 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.659214+0000", 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00021660399999999999 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(8.18 742/739 [PushOp(8:1d7992ea:::benchmark_data_smithi173_130566_object4669:head, version: 732'295, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:1d7992ea:::benchmark_data_smithi173_130566_object4669:head@732'295, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:28.879770+0000", 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 170.146681508, 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.370364148, 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.061 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.879770+0000", 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.879770+0000", 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.879770+0000", 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2300000000000002e-07 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.879915+0000", 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000144661 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.249482+0000", 2024-09-17T16:37:19.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36956730700000001 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.249512+0000", 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9955999999999999e-05 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.249764+0000", 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025143600000000001 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.249786+0000", 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2719999999999999e-05 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.250134+0000", 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000347645 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(8.10 743/742 [PushReplyOp(8:0bc28d4c:::benchmark_data_smithi173_130566_object2319:head)])", 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:29.824885+0000", 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 169.20156626900001, 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86261194500000005, 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.824885+0000", 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.824885+0000", 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.824886+0000", 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.13e-07 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.824889+0000", 2024-09-17T16:37:19.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6689999999999999e-06 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.684407+0000", 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.85951759500000002 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.684415+0000", 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1089999999999998e-06 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.686916+0000", 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002500979 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.687497+0000", 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058107999999999999 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:94963 8.14 8:294a30cb:::benchmark_data_smithi173_130566_object94962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e769)", 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:41.603167+0000", 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 97.423284577999993, 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45312667699999998, 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 94963 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.070 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.603167+0000", 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.603167+0000", 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.603168+0000", 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6589999999999999e-06 2024-09-17T16:37:19.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.603223+0000", 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4175000000000002e-05 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.603236+0000", 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3611000000000001e-05 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.603242+0000", 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6400000000000002e-06 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.603295+0000", 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3449000000000003e-05 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.603435+0000", 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000139658 2024-09-17T16:37:19.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.603559+0000", 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012430399999999999 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.604914+0000", 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001354566 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.606287+0000", 2024-09-17T16:37:19.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013730940000000001 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.056216+0000", 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.449929207 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.056260+0000", 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3794000000000001e-05 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.056294+0000", 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3519999999999998e-05 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:94963 8.14 e769/743 ondisk, result = 0)", 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:41.609943+0000", 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 97.416508183000005, 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44635628799999999, 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.076 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.609943+0000", 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.609943+0000", 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.609944+0000", 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.203e-06 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.609953+0000", 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0709999999999995e-06 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.056135+0000", 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44618263200000002 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.056142+0000", 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7279999999999998e-06 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.056203+0000", 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1370000000000004e-05 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.056211+0000", 2024-09-17T16:37:19.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3309999999999998e-06 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.056300+0000", 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8953e-05 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113329 8.19 8.cd7b9579 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.083476+0000", 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.942975623999999, 2024-09-17T16:37:19.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93999810299999997, 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113329 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083476+0000", 2024-09-17T16:37:19.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083476+0000", 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083476+0000", 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.75e-07 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083483+0000", 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3199999999999996e-06 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083493+0000", 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9580000000000002e-06 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083498+0000", 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9629999999999997e-06 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083558+0000", 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9917999999999998e-05 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083570+0000", 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1942e-05 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023439+0000", 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93986991600000003 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023474+0000", 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4411e-05 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113344 8.19 8.40c48eb9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.120379+0000", 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.906072289000001, 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.90320710199999998, 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.085 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113344 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120379+0000", 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120379+0000", 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120380+0000", 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.54e-07 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120387+0000", 2024-09-17T16:37:19.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0949999999999998e-06 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120396+0000", 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1980000000000007e-06 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120402+0000", 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2719999999999997e-06 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120449+0000", 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7304e-05 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120456+0000", 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0090000000000001e-06 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023577+0000", 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.903121226 2024-09-17T16:37:19.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023586+0000", 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1439999999999992e-06 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113366 8.19 8.da4bff99 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.226795+0000", 2024-09-17T16:37:19.090 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.799655959999999, 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79682678600000001, 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113366 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226795+0000", 2024-09-17T16:37:19.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226795+0000", 2024-09-17T16:37:19.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226799+0000", 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.163e-06 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226814+0000", 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5837000000000001e-05 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226854+0000", 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9303e-05 2024-09-17T16:37:19.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226877+0000", 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3516e-05 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.227001+0000", 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012416099999999999 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.227011+0000", 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6439999999999995e-06 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023615+0000", 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79660426600000001 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023622+0000", 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8959999999999997e-06 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.095 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113385 8.10 8:0d6b7f85:::benchmark_data_smithi173_130566_object18410:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.458365+0000", 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.568086598000001, 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36643920299999999, 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:19.096 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113385 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458365+0000", 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458365+0000", 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458366+0000", 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2999999999999999e-07 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458370+0000", 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9620000000000004e-06 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458379+0000", 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3370000000000006e-06 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458384+0000", 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9389999999999996e-06 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458453+0000", 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8963999999999999e-05 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458546+0000", 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3283999999999997e-05 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458629+0000", 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3102999999999994e-05 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.460329+0000", 2024-09-17T16:37:19.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016996229999999999 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.462031+0000", 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001702393 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.823107+0000", 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36107610699999998 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:19.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.823154+0000", 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6480000000000002e-05 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.824804+0000", 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016501809999999999 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.102 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:19.104 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113400 8.19 8.6f899219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:37:19.104 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.512591+0000", 2024-09-17T16:37:19.104 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.513860251000001, 2024-09-17T16:37:19.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51106722999999998, 2024-09-17T16:37:19.104 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.104 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.104 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113400 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512591+0000", 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512591+0000", 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512593+0000", 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0669999999999999e-06 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512604+0000", 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0526000000000001e-05 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512632+0000", 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8595999999999999e-05 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512653+0000", 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0299999999999999e-05 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512799+0000", 2024-09-17T16:37:19.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014620099999999999 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512817+0000", 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7683e-05 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023651+0000", 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51083464700000003 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023658+0000", 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2099999999999996e-06 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113401 8.19 8.6a0bcc79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.512908+0000", 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.513543324, 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51079329600000001, 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.109 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113401 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512908+0000", 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.110 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512908+0000", 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512909+0000", 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8400000000000004e-07 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512912+0000", 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0309999999999999e-06 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512919+0000", 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0360000000000001e-06 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512925+0000", 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0680000000000002e-06 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512969+0000", 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4558999999999998e-05 2024-09-17T16:37:19.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512977+0000", 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1849999999999998e-06 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023693+0000", 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51071658499999995 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023701+0000", 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2479999999999996e-06 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:115081 8.0 8:02125afa:::benchmark_data_smithi173_130566_object20106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:54.801418+0000", 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 24.225033093, 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.67051243999999999, 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:19.114 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 115081 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801418+0000", 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801418+0000", 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801420+0000", 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3710000000000001e-06 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801424+0000", 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.724e-06 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801437+0000", 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2724999999999999e-05 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801445+0000", 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4789999999999999e-06 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801495+0000", 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0609000000000003e-05 2024-09-17T16:37:19.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801598+0000", 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000103088 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801699+0000", 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100719 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.804026+0000", 2024-09-17T16:37:19.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023265709999999999 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.805302+0000", 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001276358 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.471864+0000", 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.66656182399999997 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.471901+0000", 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7225999999999999e-05 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.471931+0000", 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9745999999999999e-05 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:115157 8.10 8:0c8466af:::benchmark_data_smithi173_130566_object20182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:54.975177+0000", 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 24.051273949999999, 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.35659368600000002, 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 115157 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.121 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.975177+0000", 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.975177+0000", 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.975179+0000", 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6589999999999999e-06 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.975184+0000", 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3260000000000003e-06 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.975197+0000", 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3032e-05 2024-09-17T16:37:19.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.975203+0000", 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3290000000000001e-06 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.975280+0000", 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6967999999999999e-05 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.975388+0000", 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000108522 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.975477+0000", 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8805000000000001e-05 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.977210+0000", 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017324560000000001 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.978784+0000", 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001574799 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.330402+0000", 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.35161716100000001 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.330440+0000", 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8243e-05 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.331771+0000", 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013313859999999999 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:115157 8.10 e800/792 ondisk, result = 0)", 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:54.977272+0000", 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 24.049179201000001, 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.35450469800000001, 2024-09-17T16:37:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.977272+0000", 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.977272+0000", 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.128 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.977273+0000", 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4000000000000001e-07 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.977277+0000", 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9949999999999999e-06 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.330220+0000", 2024-09-17T16:37:19.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.35294272700000001 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.330232+0000", 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2245999999999999e-05 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.330379+0000", 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014734000000000001 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:19.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.330397+0000", 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7819e-05 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.331777+0000", 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001379831 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(8.18 e800,797 8:1d5619a6:::benchmark_data_smithi173_130566_object13907:head,794'3698)", 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:55.127854+0000", 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 23.898597612, 2024-09-17T16:37:19.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40147669699999999, 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.127854+0000", 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.127854+0000", 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:19.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.127854+0000", 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5300000000000004e-07 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.127858+0000", 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1970000000000001e-06 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.529076+0000", 2024-09-17T16:37:19.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40121873200000002 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.529082+0000", 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8259999999999998e-06 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.529179+0000", 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7188999999999994e-05 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:19.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.529330+0000", 2024-09-17T16:37:19.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015119999999999999 2024-09-17T16:37:19.135 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.135 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.135 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.135 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:19.135 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:19.135 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:37:19.136 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:19.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:19.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:19.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:19.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:19.364 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:19.391 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:19.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:19.398 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:37:19.398 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:37:19.412 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:19.422 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:37:19.422 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:37:19.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:37:19.455 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:37:19.487 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:37:19.487 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:37:19.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:19 smithi017 ceph-mon[27571]: pgmap v1802: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s; 23901/175317 objects degraded (13.633%); 16 MiB/s, 309 objects/s recovering 2024-09-17T16:37:19.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:19 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:37:19.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2132285231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:37:19.487 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:19 smithi017 ceph-mon[27571]: osdmap e809: 12 total, 12 up, 12 in 2024-09-17T16:37:19.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:19.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:19.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:19.522 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:37:19.522 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:37:19.562 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:37:19.562 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:37:19.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:19 smithi071 ceph-mon[30002]: pgmap v1802: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s; 23901/175317 objects degraded (13.633%); 16 MiB/s, 309 objects/s recovering 2024-09-17T16:37:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:19 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:37:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2132285231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:37:19.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:19 smithi071 ceph-mon[30002]: osdmap e809: 12 total, 12 up, 12 in 2024-09-17T16:37:19.589 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:37:19.590 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:37:19.620 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:19.625 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:37:19.625 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:37:19.641 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:19.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:19 smithi173 ceph-mon[30530]: pgmap v1802: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 819 op/s; 23901/175317 objects degraded (13.633%); 16 MiB/s, 309 objects/s recovering 2024-09-17T16:37:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:19 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:37:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2132285231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:37:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:19 smithi173 ceph-mon[30530]: osdmap e809: 12 total, 12 up, 12 in 2024-09-17T16:37:19.662 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:37:19.663 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:37:19.690 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:37:19.691 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:37:19.725 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:37:19.725 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:37:19.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:19.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:19.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:19.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:37:19.783 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:37:19.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:19.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:19.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:19.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:37:19.995 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:20.016 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:20.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:20.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:20.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:20.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:20.190 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:20.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:20.263 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:20.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:20.364 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:20.391 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:20.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:20.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:20 smithi017 ceph-mon[27571]: osdmap e810: 12 total, 12 up, 12 in 2024-09-17T16:37:20.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1982169306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:20.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:20 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 23901/175317 objects degraded (13.633%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:20.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:20 smithi017 ceph-mon[27571]: osdmap e811: 12 total, 12 up, 12 in 2024-09-17T16:37:20.481 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:20 smithi071 ceph-mon[30002]: osdmap e810: 12 total, 12 up, 12 in 2024-09-17T16:37:20.481 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1982169306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:20.481 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:20 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 23901/175317 objects degraded (13.633%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:20.481 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:20 smithi071 ceph-mon[30002]: osdmap e811: 12 total, 12 up, 12 in 2024-09-17T16:37:20.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:20.517 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:20.518 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:20.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:20.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:20.642 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:20.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:20 smithi173 ceph-mon[30530]: osdmap e810: 12 total, 12 up, 12 in 2024-09-17T16:37:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1982169306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:20 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 23901/175317 objects degraded (13.633%), 11 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:20 smithi173 ceph-mon[30530]: osdmap e811: 12 total, 12 up, 12 in 2024-09-17T16:37:20.700 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:37:20.700 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:37:20.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:20.748 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T16:37:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:20.765 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:20.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:20.867 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:20.891 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:20.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:20.981 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:37:20.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:21.017 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:21.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:21.119 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:21.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:21.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:21.257 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:21.278 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:21.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:21.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:21 smithi071 ceph-mon[30002]: pgmap v1806: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 23737/168573 objects degraded (14.081%); 23 MiB/s, 521 objects/s recovering 2024-09-17T16:37:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/317709953' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:37:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:21 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/317709953' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:37:21.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:21 smithi071 ceph-mon[30002]: osdmap e812: 12 total, 12 up, 12 in 2024-09-17T16:37:21.367 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:37:21.380 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:21.407 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:21.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:21 smithi017 ceph-mon[27571]: pgmap v1806: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 23737/168573 objects degraded (14.081%); 23 MiB/s, 521 objects/s recovering 2024-09-17T16:37:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/317709953' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:37:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:21 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/317709953' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:37:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:21 smithi017 ceph-mon[27571]: osdmap e812: 12 total, 12 up, 12 in 2024-09-17T16:37:21.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:21.533 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:21.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:21.635 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:21.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:21 smithi173 ceph-mon[30530]: pgmap v1806: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 931 op/s; 23737/168573 objects degraded (14.081%); 23 MiB/s, 521 objects/s recovering 2024-09-17T16:37:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/317709953' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:37:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:21 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/317709953' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:37:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:21 smithi173 ceph-mon[30530]: osdmap e812: 12 total, 12 up, 12 in 2024-09-17T16:37:21.660 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:21.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:21.662 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:21.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:21.793 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:21.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:21.895 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:21.918 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:21.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:21.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:22.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:22.058 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:22.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:22.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:22.196 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:22.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:22.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:22.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:22.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:22.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:22.452 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:37:22.453 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:37:22.457 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:22.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:22.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:22.593 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:22.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:22.678 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:37:22.695 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:22.766 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:22.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:22.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:22.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:22.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:22.899 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:22 smithi017 ceph-mon[27571]: osdmap e813: 12 total, 12 up, 12 in 2024-09-17T16:37:22.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:22 smithi017 ceph-mon[27571]: pgmap v1809: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 23355/165426 objects degraded (14.118%); 3.2 MiB/s, 126 objects/s recovering 2024-09-17T16:37:22.999 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:23.030 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:23.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:23.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:22 smithi071 ceph-mon[30002]: osdmap e813: 12 total, 12 up, 12 in 2024-09-17T16:37:23.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:22 smithi071 ceph-mon[30002]: pgmap v1809: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 23355/165426 objects degraded (14.118%); 3.2 MiB/s, 126 objects/s recovering 2024-09-17T16:37:23.132 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:22 smithi173 ceph-mon[30530]: osdmap e813: 12 total, 12 up, 12 in 2024-09-17T16:37:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:22 smithi173 ceph-mon[30530]: pgmap v1809: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 16 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 23355/165426 objects degraded (14.118%); 3.2 MiB/s, 126 objects/s recovering 2024-09-17T16:37:23.158 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:23.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:23.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:23.206 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T16:37:23.259 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:23.266 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:23.329 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:23.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:23.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:23.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:23.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:23.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:23.556 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:23.577 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:23.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:23.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:23.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:23.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:23.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:23.808 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:23.840 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:23.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:23.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:23.942 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:23.963 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:23.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:23.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:23 smithi017 ceph-mon[27571]: 8.6 scrub starts 2024-09-17T16:37:23.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:23 smithi017 ceph-mon[27571]: 8.6 scrub ok 2024-09-17T16:37:24.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:23 smithi071 ceph-mon[30002]: 8.6 scrub starts 2024-09-17T16:37:24.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:23 smithi071 ceph-mon[30002]: 8.6 scrub ok 2024-09-17T16:37:24.065 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:24.097 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:24.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:23 smithi173 ceph-mon[30530]: 8.6 scrub starts 2024-09-17T16:37:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:23 smithi173 ceph-mon[30530]: 8.6 scrub ok 2024-09-17T16:37:24.200 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:24.234 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:24.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:24.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:24.363 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:24.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:24.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:24.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:24.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:24.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:24.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:24.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:24.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:24.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:24.746 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:24.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:24.759 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:37:24.820 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:24.822 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:37:24.822 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:37:24.848 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:24.863 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:37:24.863 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:37:24.869 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:24.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:24.871 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:24.900 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:37:24.900 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:37:24.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:24 smithi017 ceph-mon[27571]: pgmap v1810: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 18 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21474/162150 objects degraded (13.243%) 2024-09-17T16:37:24.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:24 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 23355/165426 objects degraded (14.118%), 10 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:24.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:24.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:24 smithi071 ceph-mon[30002]: pgmap v1810: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 18 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21474/162150 objects degraded (13.243%) 2024-09-17T16:37:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:24 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 23355/165426 objects degraded (14.118%), 10 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:25.085 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:25.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:25.124 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:25.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:25.129 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:37:25.129 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:37:25.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:24 smithi173 ceph-mon[30530]: pgmap v1810: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 18 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21474/162150 objects degraded (13.243%) 2024-09-17T16:37:25.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:24 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 23355/165426 objects degraded (14.118%), 10 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:25.154 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:37:25.154 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:37:25.179 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:37:25.179 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:37:25.207 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:37:25.207 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:37:25.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:25.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:25.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:25.259 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:37:25.259 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:37:25.295 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:37:25.296 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:37:25.330 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:37:25.331 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:37:25.347 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:25.362 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:37:25.362 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:37:25.375 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:25.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:25.392 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:37:25.393 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:37:25.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:25.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:25.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:25.606 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:25.638 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:25.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:25.741 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:25.765 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:25.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:25.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:25.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3076484971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:25.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:25.889 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:25.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:25.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:25.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:26.027 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:26.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.26665.0:152017 6.a 6:5f28e5f6:::benchmark_data_smithi173_102017_object58869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e602)", 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:28:24.975823+0000", 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 541.05220133900002, 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56153204199999995, 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.032 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.26665", 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152017 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975823+0000", 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975823+0000", 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975824+0000", 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6640000000000001e-06 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975830+0000", 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7450000000000001e-06 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975843+0000", 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3191e-05 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975849+0000", 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.251e-06 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.975897+0000", 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8052e-05 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.976015+0000", 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011795599999999999 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,5", 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.976112+0000", 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6572999999999998e-05 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.979946+0000", 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038342089999999999 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.980159+0000", 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00021301499999999999 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534848+0000", 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55468829500000005 2024-09-17T16:37:26.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534888+0000", 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0905999999999998e-05 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.537355+0000", 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024661850000000001 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.26665.0:152017 6.a e602/568 ondisk, result = 0)", 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:28:24.982375+0000", 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 541.04564857299999, 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55498762499999998, 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.982375+0000", 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.982375+0000", 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.982376+0000", 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6400000000000002e-07 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.040 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:24.982380+0000", 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1389999999999997e-06 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534775+0000", 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55239448899999999 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534784+0000", 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9109999999999999e-06 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534835+0000", 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1328000000000001e-05 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.534843+0000", 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9289999999999998e-06 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:28:25.537363+0000", 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002520165 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:170216 7.11 7:89b9c9a3:::benchmark_data_smithi173_116396_object74616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 240.48904537000001, 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.387146404, 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.043 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:37:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:37:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170216 2024-09-17T16:37:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.044 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538980+0000", 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3030000000000001e-06 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538986+0000", 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6579999999999999e-06 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538998+0000", 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2903e-05 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539004+0000", 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9540000000000001e-06 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539051+0000", 2024-09-17T16:37:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7067999999999997e-05 2024-09-17T16:37:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539144+0000", 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2521000000000004e-05 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539219+0000", 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4791999999999998e-05 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541123+0000", 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019039650000000001 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541689+0000", 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00056619600000000004 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925945+0000", 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38425619700000002 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926009+0000", 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3516999999999995e-05 2024-09-17T16:37:26.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926125+0000", 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011633000000000001 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:170216 7.11 e712/703 ondisk, result = 0)", 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 240.486650567, 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38475827200000001, 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.050 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541374+0000", 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541379+0000", 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6210000000000003e-06 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.923614+0000", 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38223536499999999 2024-09-17T16:37:26.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.923618+0000", 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7629999999999998e-06 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925930+0000", 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023126259999999999 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925939+0000", 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0070000000000003e-06 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926132+0000", 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019219300000000001 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:170289 7.11 7:8ff6a0b8:::benchmark_data_smithi173_116396_object74689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:37:26.054 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 240.40371675399999, 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30203754100000002, 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170289 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:37:26.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624309+0000", 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.136e-06 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624319+0000", 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9170000000000001e-06 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624348+0000", 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8368999999999999e-05 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624362+0000", 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4222999999999999e-05 2024-09-17T16:37:26.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624491+0000", 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000129262 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624698+0000", 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000206546 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624844+0000", 2024-09-17T16:37:26.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014652400000000001 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.626781+0000", 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019372759999999999 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627901+0000", 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011196260000000001 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926178+0000", 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29827702699999997 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926231+0000", 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2892000000000001e-05 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926345+0000", 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000113743 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.060 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:170289 7.11 e712/703 ondisk, result = 0)", 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 240.401012592, 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29933906100000002, 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.061 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627012+0000", 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627016+0000", 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9809999999999997e-06 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925949+0000", 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29893330699999998 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925958+0000", 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926164+0000", 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000205942 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926171+0000", 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9500000000000001e-06 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926350+0000", 2024-09-17T16:37:26.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017893299999999999 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDelete(7.19 e717,677 7:9da3d41f:::benchmark_data_smithi173_116396_object4781:head,673'6297)", 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 229.85208641, 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37136110999999999, 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.065 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175947+0000", 2024-09-17T16:37:26.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3859999999999996e-06 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175959+0000", 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2269999999999999e-05 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547126+0000", 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37116651099999998 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547136+0000", 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0241000000000001e-05 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547216+0000", 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.051e-05 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547227+0000", 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0356999999999999e-05 2024-09-17T16:37:26.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547299+0000", 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1835000000000006e-05 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:191171 7.11 7:8d50061b:::benchmark_data_smithi173_116396_object95571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 211.59745827699999, 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44239869199999998, 2024-09-17T16:37:26.069 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 191171 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430568+0000", 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.897e-06 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430578+0000", 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0149999999999999e-05 2024-09-17T16:37:26.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430616+0000", 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8753e-05 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430622+0000", 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3569999999999997e-06 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430666+0000", 2024-09-17T16:37:26.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3831000000000001e-05 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430740+0000", 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4865999999999998e-05 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430817+0000", 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6490999999999997e-05 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433451+0000", 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0026335489999999998 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433803+0000", 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00035236799999999998 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872847+0000", 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43904401599999998 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872921+0000", 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3867000000000003e-05 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872964+0000", 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3547000000000002e-05 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.075 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:191171 7.11 e724/703 ondisk, result = 0)", 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 211.59443004600001, 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43937563000000002, 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433595+0000", 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2900000000000002e-07 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433602+0000", 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9550000000000003e-06 2024-09-17T16:37:26.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872622+0000", 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43902024000000001 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872639+0000", 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7516e-05 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872820+0000", 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000180714 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872837+0000", 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7099999999999999e-05 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872970+0000", 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013227599999999999 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(8.12 743/742 [PushOp(8:4d9389a5:::benchmark_data_smithi173_130566_object4465:head, version: 732'276, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:4d9389a5:::benchmark_data_smithi173_130566_object4465:head@732'276, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:29.740078+0000", 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 176.28794587900001, 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32815092099999998, 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740078+0000", 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740078+0000", 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740079+0000", 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0900000000000001e-07 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740180+0000", 2024-09-17T16:37:26.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010132900000000001 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067836+0000", 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32765588099999998 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067848+0000", 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1754e-05 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067920+0000", 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2813999999999998e-05 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067929+0000", 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1939999999999998e-06 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.068229+0000", 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030034000000000001 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.083 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:78112 8.12 8:4c421870:::benchmark_data_smithi173_130566_object78111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:26.226347+0000", 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 119.801676927, 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.305366213, 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:26.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 78112 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226347+0000", 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226347+0000", 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226348+0000", 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.435e-06 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226401+0000", 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2386000000000003e-05 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226415+0000", 2024-09-17T16:37:26.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4095e-05 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226421+0000", 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1240000000000002e-06 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226475+0000", 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4376999999999998e-05 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226606+0000", 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013063500000000001 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226759+0000", 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015287100000000001 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228063+0000", 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013040199999999999 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228274+0000", 2024-09-17T16:37:26.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000211201 2024-09-17T16:37:26.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531553+0000", 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30327892299999998 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531612+0000", 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8836999999999999e-05 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531713+0000", 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000101309 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:78112 8.12 e762/744 ondisk, result = 0)", 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:26.228645+0000", 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 119.79937882500001, 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30307498500000002, 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.091 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228645+0000", 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228645+0000", 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228646+0000", 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7999999999999995e-07 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228649+0000", 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6370000000000001e-06 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531475+0000", 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.302826076 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531482+0000", 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7490000000000001e-06 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531537+0000", 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4731000000000002e-05 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531547+0000", 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0200000000000001e-05 2024-09-17T16:37:26.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531720+0000", 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000173012 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "rep_scrubmap(8.12 e744 from shard 0)", 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:34.607872+0000", 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 51.420152246999997, 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26429114300000001, 2024-09-17T16:37:26.095 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.607872+0000", 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.607872+0000", 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.607872+0000", 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2300000000000001e-07 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.607897+0000", 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4831e-05 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.871871+0000", 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26397431999999998 2024-09-17T16:37:26.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.871878+0000", 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7839999999999998e-06 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.871942+0000", 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3478000000000004e-05 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.871957+0000", 2024-09-17T16:37:26.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5006e-05 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.872163+0000", 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00020610100000000001 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:112090 8.12 8:4f9a8053:::benchmark_data_smithi173_130566_object17115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e794)", 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:49.085350+0000", 2024-09-17T16:37:26.099 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 36.942673493999997, 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25109838400000001, 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 112090 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.085350+0000", 2024-09-17T16:37:26.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.085350+0000", 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.085352+0000", 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0860000000000001e-06 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.085356+0000", 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7690000000000004e-06 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.085367+0000", 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1195999999999999e-05 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.085373+0000", 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2750000000000003e-06 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.085429+0000", 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6413999999999999e-05 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.085568+0000", 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013860700000000001 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.085659+0000", 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0845000000000002e-05 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.087324+0000", 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001665478 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.333337+0000", 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.246013124 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.334552+0000", 2024-09-17T16:37:26.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001214555 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.334614+0000", 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.198e-05 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.336449+0000", 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018350549999999999 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:112090 8.12 e794/792 ondisk, result = 0)", 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:49.093820+0000", 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 36.934203738999997, 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24263654900000001, 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.093820+0000", 2024-09-17T16:37:26.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.093820+0000", 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.093821+0000", 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0499999999999999e-06 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.093830+0000", 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9099999999999994e-06 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.334481+0000", 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24065130700000001 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.334488+0000", 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1990000000000003e-06 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.334535+0000", 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7073000000000001e-05 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.334544+0000", 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6399999999999992e-06 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.336457+0000", 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00191237 2024-09-17T16:37:26.109 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:112337 8.12 8:48434714:::benchmark_data_smithi173_130566_object17362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e795)", 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:49.532649+0000", 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 36.495374783999999, 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20270577400000001, 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:26.110 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 112337 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.532649+0000", 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.532649+0000", 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.532651+0000", 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9149999999999999e-06 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.532658+0000", 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0779999999999998e-06 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.532673+0000", 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5234e-05 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.532682+0000", 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5709999999999998e-06 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.532756+0000", 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3878e-05 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:37:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.532765+0000", 2024-09-17T16:37:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7520000000000002e-06 2024-09-17T16:37:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.565007+0000", 2024-09-17T16:37:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.032242522000000003 2024-09-17T16:37:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.565194+0000", 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018695800000000001 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.565367+0000", 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017306900000000001 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.568436+0000", 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0030693489999999999 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.598928+0000", 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.030491936000000001 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.733743+0000", 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13481509899999999 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.733792+0000", 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8967000000000001e-05 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:49.735355+0000", 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015624460000000001 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:118510 8.12 8:4951923c:::benchmark_data_smithi173_130566_object23535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-09-17T16:37:26.117 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:59.744124+0000", 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.283900358, 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22734859399999999, 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 118510 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.744124+0000", 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.744124+0000", 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.744125+0000", 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.531e-06 2024-09-17T16:37:26.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.744130+0000", 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0560000000000004e-06 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.744145+0000", 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4741e-05 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.744153+0000", 2024-09-17T16:37:26.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6690000000000006e-06 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.745189+0000", 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010364549999999999 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.745300+0000", 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000110815 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:37:26.121 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.745404+0000", 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010377100000000001 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.747346+0000", 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001942561 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.754067+0000", 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0067211529999999997 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.969930+0000", 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.215862308 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.969966+0000", 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6695e-05 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.971472+0000", 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015058389999999999 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.123 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:118510 8.12 e803/792 ondisk, result = 0)", 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:59.747183+0000", 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.280840809000001, 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.224294726, 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.747183+0000", 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.747183+0000", 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.747184+0000", 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.47e-07 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.747189+0000", 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6419999999999998e-06 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.969874+0000", 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.222685038 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.969880+0000", 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5189999999999998e-06 2024-09-17T16:37:26.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.969918+0000", 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7539000000000001e-05 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.969924+0000", 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5769999999999999e-06 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:59.971478+0000", 2024-09-17T16:37:26.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015534640000000001 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:133564 8.12 8:4eb7d47c:::benchmark_data_smithi173_130566_object38589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e808)", 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:17.681433+0000", 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.3465909469999993, 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25490686699999998, 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:26.128 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 133564 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.681433+0000", 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.681433+0000", 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.681434+0000", 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5090000000000001e-06 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.681440+0000", 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6890000000000001e-06 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.681454+0000", 2024-09-17T16:37:26.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4242e-05 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.681461+0000", 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0199999999999997e-06 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.681510+0000", 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8266999999999997e-05 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.131 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.681621+0000", 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000111358 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.681715+0000", 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3640999999999993e-05 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.683588+0000", 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018737230000000001 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.683745+0000", 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015670300000000001 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.934644+0000", 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25089896299999997 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.934712+0000", 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7600000000000003e-05 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.936340+0000", 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016281519999999999 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:133564 8.12 e808/792 ondisk, result = 0)", 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:17.685242+0000", 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 8.3427822309999993, 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25110464199999999, 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:26.134 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.685242+0000", 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.685242+0000", 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.685242+0000", 2024-09-17T16:37:26.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5400000000000002e-07 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.685245+0000", 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6170000000000001e-06 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.934568+0000", 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.2493233 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:26.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.934572+0000", 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2250000000000002e-06 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.934624+0000", 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1527000000000003e-05 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.934635+0000", 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1440999999999999e-05 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:26.137 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:26.138 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.936346+0000", 2024-09-17T16:37:26.138 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001711078 2024-09-17T16:37:26.138 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.138 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.138 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.138 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:26.138 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:26.138 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:37:26.138 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:26.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3076484971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:26.162 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:26.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:26.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:26.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3076484971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:26.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:26.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:26.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:26.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:26.387 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:26.422 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:26.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:26.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:26.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:26.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:26.648 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:26.680 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:26.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:26.782 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:26.806 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:26.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:26.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:26.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:26.937 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:26 smithi071 ceph-mon[30002]: pgmap v1811: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 68 op/s; 21191/156258 objects degraded (13.562%) 2024-09-17T16:37:26.943 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:26.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:26.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:37:27.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:27.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:27.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:27.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:26 smithi173 ceph-mon[30530]: pgmap v1811: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 68 op/s; 21191/156258 objects degraded (13.562%) 2024-09-17T16:37:27.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:27.209 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:27.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:26 smithi017 ceph-mon[27571]: pgmap v1811: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 68 op/s; 21191/156258 objects degraded (13.562%) 2024-09-17T16:37:27.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:27.311 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:27.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:27.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:27.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:27.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:27.456 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:27.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:27.558 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:27.594 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:27.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:27.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:27.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:27.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:27.821 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:27.844 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:27.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:27.884 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:37:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:37:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:140879 8.a 8:5b9b0b9e:::benchmark_data_smithi173_130566_object45904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e814)", 2024-09-17T16:37:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:27.846983+0000", 2024-09-17T16:37:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0041925219999999997, 2024-09-17T16:37:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042205369999999999, 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 140879 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:27.846983+0000", 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:27.846983+0000", 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:27.846984+0000", 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.542e-06 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:27.846990+0000", 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0900000000000001e-06 2024-09-17T16:37:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:27.847007+0000", 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6249e-05 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:27.847013+0000", 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5889999999999999e-06 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:27.847071+0000", 2024-09-17T16:37:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8155000000000001e-05 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:27.847176+0000", 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010456200000000001 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:27.847247+0000", 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1134e-05 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:27.851210+0000", 2024-09-17T16:37:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039630940000000003 2024-09-17T16:37:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:37:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-09-17T16:37:27.889 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:37:27.947 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:27.978 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:27.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:28.010 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:37:28.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1461539393' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:37:28.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:28.101 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:28.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:28.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1461539393' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:37:28.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1461539393' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:37:28.203 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:28.236 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:28.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:28.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:28.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:28.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:28.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:28.491 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:28.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:28.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:28.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:28.617 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:28.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:28.719 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:28.748 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:28.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:28.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:28.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:28.874 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:28.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:28.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:29.004 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:29.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:29.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:28 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:37:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1461539393' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:37:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:28 smithi071 ceph-mon[30002]: osdmap e814: 12 total, 12 up, 12 in 2024-09-17T16:37:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:28 smithi071 ceph-mon[30002]: pgmap v1813: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 20573/152751 objects degraded (13.468%) 2024-09-17T16:37:29.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:28 smithi071 ceph-mon[30002]: 8.18 scrub starts 2024-09-17T16:37:29.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:29.127 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:29.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:29.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:28 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:37:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1461539393' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:37:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:28 smithi173 ceph-mon[30530]: osdmap e814: 12 total, 12 up, 12 in 2024-09-17T16:37:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:28 smithi173 ceph-mon[30530]: pgmap v1813: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 20573/152751 objects degraded (13.468%) 2024-09-17T16:37:29.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:28 smithi173 ceph-mon[30530]: 8.18 scrub starts 2024-09-17T16:37:29.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:28 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:37:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1461539393' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:37:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:28 smithi017 ceph-mon[27571]: osdmap e814: 12 total, 12 up, 12 in 2024-09-17T16:37:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:28 smithi017 ceph-mon[27571]: pgmap v1813: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 131 op/s; 20573/152751 objects degraded (13.468%) 2024-09-17T16:37:29.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:28 smithi017 ceph-mon[27571]: 8.18 scrub starts 2024-09-17T16:37:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:29.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:29.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:29.358 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:29.393 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:29.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:29.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:29.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:37:29.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:29.517 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:29.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:29.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:29.642 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:29.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:29.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:29.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:29.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:29.797 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:29.878 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:29.901 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:29.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:30.005 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:30.027 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:30.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:30.092 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:29 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 20573/152751 objects degraded (13.468%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:30.093 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:29 smithi173 ceph-mon[30530]: osdmap e815: 12 total, 12 up, 12 in 2024-09-17T16:37:30.128 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:30.163 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:30.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:30.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:29 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 20573/152751 objects degraded (13.468%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:30.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:29 smithi017 ceph-mon[27571]: osdmap e815: 12 total, 12 up, 12 in 2024-09-17T16:37:30.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:30.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:30.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:30.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:29 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 20573/152751 objects degraded (13.468%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:30.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:29 smithi071 ceph-mon[30002]: osdmap e815: 12 total, 12 up, 12 in 2024-09-17T16:37:30.397 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:37:30.398 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:37:30.398 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:37:30.398 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:37:30.398 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:37:30.405 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:30.437 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:30.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:30.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:30.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:30.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:30.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:30.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:30.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:30.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:30.824 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:30.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:30.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:30.948 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:30.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:30.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:31.050 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:31.073 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:31.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:31.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:37:31.121 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:37:31.122 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:37:31.122 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:37:31.122 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:37:31.166 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:37:31.167 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:37:31.167 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:37:31.167 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:37:31.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:31.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:31.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:31.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:31.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:37:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:37:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:37:31.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:37:31.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:31.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:31.426 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:31.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:31.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:31.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:31.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:31.597 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:31.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:31.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:31.734 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:31.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:31.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:31.873 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:31.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:31.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:32.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:32.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:32.020 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:37:32.020 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:37:32.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:37:32.020 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:37:32.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.020 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:10904 7.7 7:e822bbe0:::benchmark_data_smithi173_116396_object10903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e636)", 2024-09-17T16:37:32.020 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:47.859010+0000", 2024-09-17T16:37:32.020 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 464.15610147199999, 2024-09-17T16:37:32.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34506130800000001, 2024-09-17T16:37:32.020 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 10904 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.859010+0000", 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.859010+0000", 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.859011+0000", 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2309999999999999e-06 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.859037+0000", 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5318000000000001e-05 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.859053+0000", 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6067e-05 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.859061+0000", 2024-09-17T16:37:32.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1040000000000007e-06 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.859112+0000", 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0852e-05 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.859199+0000", 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7183999999999997e-05 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.859351+0000", 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000152774 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.861113+0000", 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017613069999999999 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.861403+0000", 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029052800000000002 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.203956+0000", 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34255243600000002 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.204022+0000", 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5816999999999997e-05 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.204071+0000", 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.969e-05 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27088.0:10904 7.7 e636/629 ondisk, result = 0)", 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:47.863513+0000", 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 464.15159827899998, 2024-09-17T16:37:32.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34056465600000002, 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.863513+0000", 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.863513+0000", 2024-09-17T16:37:32.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.863514+0000", 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7599999999999997e-07 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.863517+0000", 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4910000000000001e-06 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.203742+0000", 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34022485499999999 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.203759+0000", 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7096e-05 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.203931+0000", 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000171547 2024-09-17T16:37:32.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.203947+0000", 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5990000000000001e-05 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.204078+0000", 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013120100000000001 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.031 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:10908 7.9 e637/629 7:980f6f60:::benchmark_data_smithi173_116396_object10907:head v 637'707, mlcod=636'704)", 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:47.868238+0000", 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 464.14687323499999, 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39738201000000001, 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.868238+0000", 2024-09-17T16:37:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.868238+0000", 2024-09-17T16:37:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.868238+0000", 2024-09-17T16:37:32.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.91e-07 2024-09-17T16:37:32.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:47.868280+0000", 2024-09-17T16:37:32.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1544000000000001e-05 2024-09-17T16:37:32.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.264882+0000", 2024-09-17T16:37:32.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39660245 2024-09-17T16:37:32.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.264890+0000", 2024-09-17T16:37:32.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0700000000000001e-06 2024-09-17T16:37:32.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.264956+0000", 2024-09-17T16:37:32.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6849999999999999e-05 2024-09-17T16:37:32.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.264981+0000", 2024-09-17T16:37:32.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4320000000000001e-05 2024-09-17T16:37:32.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.265584+0000", 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00060362800000000004 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:48.265620+0000", 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5856999999999999e-05 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.038 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:103863 7.7 7:e5793b30:::benchmark_data_smithi173_116396_object8263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e678)", 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:25.525545+0000", 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 366.48956651700001, 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39959506700000003, 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 103863 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.039 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.525545+0000", 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.525545+0000", 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.525546+0000", 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4160000000000001e-06 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.525552+0000", 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2240000000000003e-06 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.525565+0000", 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3399e-05 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.525571+0000", 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6799999999999998e-06 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.525616+0000", 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5074000000000002e-05 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.525698+0000", 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2692e-05 2024-09-17T16:37:32.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.525765+0000", 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6969000000000002e-05 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.527216+0000", 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014511070000000001 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.530368+0000", 2024-09-17T16:37:32.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031511690000000001 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.925032+0000", 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39466482000000003 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.925092+0000", 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9213999999999999e-05 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.925140+0000", 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8303000000000002e-05 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:178376 7.16 7:6e17509c:::benchmark_data_smithi173_116396_object82776:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e718)", 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:36.932136+0000", 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 235.08297525399999, 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.346692789, 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.045 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 178376 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932136+0000", 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932136+0000", 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932137+0000", 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6399999999999998e-07 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932138+0000", 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.827e-06 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932142+0000", 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0500000000000002e-06 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932146+0000", 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4570000000000001e-06 2024-09-17T16:37:32.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932233+0000", 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6953999999999995e-05 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932318+0000", 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4808000000000005e-05 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932395+0000", 2024-09-17T16:37:32.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7099000000000002e-05 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.934437+0000", 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00204199 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.934635+0000", 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019845 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:37.278717+0000", 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34408162399999997 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:37.278780+0000", 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3583000000000004e-05 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:37.278829+0000", 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8582999999999999e-05 2024-09-17T16:37:32.051 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:7992 8.c 8:34878b4c:::benchmark_data_smithi173_130566_object7991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e732)", 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.037799+0000", 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 194.977312697, 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.26676612, 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.052 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 7992 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037799+0000", 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037799+0000", 2024-09-17T16:37:32.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037800+0000", 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.054e-06 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037828+0000", 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8294e-05 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037846+0000", 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7649999999999999e-05 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037854+0000", 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5029999999999997e-06 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037910+0000", 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5998000000000002e-05 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.038015+0000", 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010526000000000001 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.038108+0000", 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2848000000000003e-05 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.038630+0000", 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00052117799999999996 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.039017+0000", 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038750600000000003 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304468+0000", 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.265451467 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304528+0000", 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9901000000000001e-05 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304565+0000", 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6461000000000002e-05 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.059 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:7994 8.c 8:379b6ec9:::benchmark_data_smithi173_130566_object7993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e732)", 2024-09-17T16:37:32.059 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.040560+0000", 2024-09-17T16:37:32.059 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 194.97455131800001, 2024-09-17T16:37:32.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.264147548, 2024-09-17T16:37:32.059 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.059 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.059 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.059 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.059 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 7994 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040560+0000", 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040560+0000", 2024-09-17T16:37:32.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040561+0000", 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3900000000000003e-07 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040585+0000", 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4145e-05 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040599+0000", 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3766e-05 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040606+0000", 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9990000000000002e-06 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040650+0000", 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4447999999999999e-05 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040735+0000", 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4257000000000003e-05 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040821+0000", 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6341000000000006e-05 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.041276+0000", 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045481100000000001 2024-09-17T16:37:32.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.041625+0000", 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034894899999999998 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304619+0000", 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.262994183 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304666+0000", 2024-09-17T16:37:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7079000000000001e-05 2024-09-17T16:37:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304708+0000", 2024-09-17T16:37:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1631000000000002e-05 2024-09-17T16:37:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8013 8.c 8:3c762cec:::benchmark_data_smithi173_130566_object8012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e732)", 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.082606+0000", 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 194.93250526200001, 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.222249886, 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.066 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.067 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8013 2024-09-17T16:37:32.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.067 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082606+0000", 2024-09-17T16:37:32.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082606+0000", 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082607+0000", 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5700000000000001e-07 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082627+0000", 2024-09-17T16:37:32.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9565999999999999e-05 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082637+0000", 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0616e-05 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082645+0000", 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2960000000000001e-06 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082691+0000", 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6681999999999998e-05 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082787+0000", 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6203e-05 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082873+0000", 2024-09-17T16:37:32.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5687999999999997e-05 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.083367+0000", 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049404099999999997 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.083928+0000", 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056047200000000001 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304759+0000", 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.22083164 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304817+0000", 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7763000000000002e-05 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304856+0000", 2024-09-17T16:37:32.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9062e-05 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8032 8.c 8:3d0a817e:::benchmark_data_smithi173_130566_object8031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e733)", 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.405332+0000", 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 194.60977980300001, 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.899664348, 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.073 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8032 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405332+0000", 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405332+0000", 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405334+0000", 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.897e-06 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405359+0000", 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5313999999999999e-05 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405376+0000", 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7538999999999999e-05 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405385+0000", 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1710000000000002e-06 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405454+0000", 2024-09-17T16:37:32.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9338999999999995e-05 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405659+0000", 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000204852 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405792+0000", 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133187 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.406488+0000", 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069618000000000002 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.406840+0000", 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035165199999999999 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304910+0000", 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.898070153999999 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304960+0000", 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0421999999999999e-05 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304996+0000", 2024-09-17T16:37:32.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5641e-05 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8037 8.c 8:3f3c220b:::benchmark_data_smithi173_130566_object8036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e733)", 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.409689+0000", 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 194.60542194000001, 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.895437986999999, 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.080 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8037 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409689+0000", 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409689+0000", 2024-09-17T16:37:32.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409691+0000", 2024-09-17T16:37:32.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.376e-06 2024-09-17T16:37:32.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409713+0000", 2024-09-17T16:37:32.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2589999999999999e-05 2024-09-17T16:37:32.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409727+0000", 2024-09-17T16:37:32.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4035e-05 2024-09-17T16:37:32.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409736+0000", 2024-09-17T16:37:32.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6470000000000004e-06 2024-09-17T16:37:32.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409783+0000", 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6811000000000003e-05 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409909+0000", 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012609599999999999 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.410018+0000", 2024-09-17T16:37:32.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108467 2024-09-17T16:37:32.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.410535+0000", 2024-09-17T16:37:32.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051754199999999996 2024-09-17T16:37:32.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.410949+0000", 2024-09-17T16:37:32.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00041375800000000002 2024-09-17T16:37:32.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305039+0000", 2024-09-17T16:37:32.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.894089841 2024-09-17T16:37:32.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305091+0000", 2024-09-17T16:37:32.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2429000000000002e-05 2024-09-17T16:37:32.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305127+0000", 2024-09-17T16:37:32.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6395e-05 2024-09-17T16:37:32.089 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.089 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.089 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8040 8.c 8:3ef6c0ff:::benchmark_data_smithi173_130566_object8039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e733)", 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.412931+0000", 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 194.60218048499999, 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.892416228, 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8040 2024-09-17T16:37:32.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412931+0000", 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412931+0000", 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412932+0000", 2024-09-17T16:37:32.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5799999999999998e-07 2024-09-17T16:37:32.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412952+0000", 2024-09-17T16:37:32.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0636999999999999e-05 2024-09-17T16:37:32.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412963+0000", 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0993e-05 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412969+0000", 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.429e-06 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413016+0000", 2024-09-17T16:37:32.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6777000000000003e-05 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413102+0000", 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6059999999999999e-05 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413187+0000", 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5408999999999995e-05 2024-09-17T16:37:32.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413596+0000", 2024-09-17T16:37:32.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040850499999999998 2024-09-17T16:37:32.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413999+0000", 2024-09-17T16:37:32.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040360899999999998 2024-09-17T16:37:32.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305263+0000", 2024-09-17T16:37:32.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.891264103999999 2024-09-17T16:37:32.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305314+0000", 2024-09-17T16:37:32.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0522999999999997e-05 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305347+0000", 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3324000000000001e-05 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.17 741/739 [PushOp(8:ed657130:::benchmark_data_smithi173_130566_object3105:head, version: 732'197, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ed657130:::benchmark_data_smithi173_130566_object3105:head@732'197, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:27.949079+0000", 2024-09-17T16:37:32.097 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 184.066032436, 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34690885100000002, 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.949079+0000", 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.949079+0000", 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.949086+0000", 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4749999999999996e-06 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.949139+0000", 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2955999999999999e-05 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:28.295598+0000", 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34645822399999998 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:28.295611+0000", 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3377000000000001e-05 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:28.295697+0000", 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5765000000000003e-05 2024-09-17T16:37:32.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:28.295706+0000", 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5840000000000001e-06 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:28.295988+0000", 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028146999999999997 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:92948 8.a 8:5c06d358:::benchmark_data_smithi173_130566_object92947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:35:39.305492+0000", 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 112.70961973, 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42912947499999998, 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 92948 2024-09-17T16:37:32.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305492+0000", 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305492+0000", 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305493+0000", 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4449999999999999e-06 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305518+0000", 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.442e-05 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305530+0000", 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2873e-05 2024-09-17T16:37:32.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305536+0000", 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9329999999999998e-06 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305587+0000", 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0291999999999999e-05 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305707+0000", 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119984 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305797+0000", 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0766000000000005e-05 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.308288+0000", 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002490361 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.309496+0000", 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001208104 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734535+0000", 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42503903700000001 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734580+0000", 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4672000000000002e-05 2024-09-17T16:37:32.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734621+0000", 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1588000000000002e-05 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.17883.0:92948 8.a e767/728 ondisk, result = 0)", 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:35:39.308966+0000", 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 112.70614566899999, 2024-09-17T16:37:32.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42566467299999999, 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.308966+0000", 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.308966+0000", 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.308967+0000", 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4600000000000003e-07 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.308970+0000", 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6109999999999998e-06 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734474+0000", 2024-09-17T16:37:32.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42550323899999998 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734479+0000", 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.023e-06 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734524+0000", 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5732999999999997e-05 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734531+0000", 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1959999999999996e-06 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734630+0000", 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9925000000000003e-05 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.112 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:94971 8.b 8:db7e09c1:::benchmark_data_smithi173_130566_object94970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e769)", 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:35:41.610934+0000", 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 110.404177035, 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.78101871599999995, 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 94971 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.610934+0000", 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.610934+0000", 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.610935+0000", 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5099999999999999e-07 2024-09-17T16:37:32.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.610982+0000", 2024-09-17T16:37:32.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7377999999999999e-05 2024-09-17T16:37:32.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.610992+0000", 2024-09-17T16:37:32.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5109999999999992e-06 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.611000+0000", 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8120000000000008e-06 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.611059+0000", 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9290999999999999e-05 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.611147+0000", 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7466000000000006e-05 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.611227+0000", 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0873999999999999e-05 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.613680+0000", 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024524719999999998 2024-09-17T16:37:32.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.614006+0000", 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032607600000000001 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:42.391836+0000", 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77782961699999997 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:42.391912+0000", 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6388000000000001e-05 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:42.391953+0000", 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1180000000000002e-05 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:118372 8.a 8:529743c5:::benchmark_data_smithi173_130566_object23397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-09-17T16:37:32.119 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:59.334736+0000", 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 32.68037503, 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34404921399999999, 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 118372 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.120 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.334736+0000", 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.334736+0000", 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.334738+0000", 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.308e-06 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.334742+0000", 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4889999999999997e-06 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.334751+0000", 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9619999999999999e-06 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.334756+0000", 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7620000000000003e-06 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.334809+0000", 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2876e-05 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.334940+0000", 2024-09-17T16:37:32.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013094800000000001 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.335050+0000", 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110278 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.345880+0000", 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010829719 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.345933+0000", 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3699000000000002e-05 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.678647+0000", 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.332713643 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.678739+0000", 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1915000000000003e-05 2024-09-17T16:37:32.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.678786+0000", 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6615e-05 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:118374 8.a 8:5bb28042:::benchmark_data_smithi173_130566_object23399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:59.336268+0000", 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 32.678843555, 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34283517800000002, 2024-09-17T16:37:32.126 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 118374 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.336268+0000", 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.336268+0000", 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.336269+0000", 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1900000000000001e-07 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.336273+0000", 2024-09-17T16:37:32.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3229999999999999e-06 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.336285+0000", 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2088e-05 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.336292+0000", 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2219999999999996e-06 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.338540+0000", 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002247583 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.338664+0000", 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000124215 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.338769+0000", 2024-09-17T16:37:32.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104874 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.345927+0000", 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0071577170000000001 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.345937+0000", 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9499999999999996e-06 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.679025+0000", 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33308800999999999 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.679070+0000", 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5176999999999998e-05 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.679103+0000", 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3099999999999998e-05 2024-09-17T16:37:32.132 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:118375 8.a 8:54083f79:::benchmark_data_smithi173_130566_object23400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:59.337033+0000", 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 32.678078055, 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34225655999999999, 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.133 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 118375 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.337033+0000", 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.337033+0000", 2024-09-17T16:37:32.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.337034+0000", 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6799999999999999e-07 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.337039+0000", 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4020000000000004e-06 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.337049+0000", 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0258000000000001e-05 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.337054+0000", 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.451e-06 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.342247+0000", 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0051930689999999998 2024-09-17T16:37:32.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.342378+0000", 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013030999999999999 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.342478+0000", 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010084100000000001 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.345943+0000", 2024-09-17T16:37:32.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003464328 2024-09-17T16:37:32.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.346872+0000", 2024-09-17T16:37:32.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00092888300000000003 2024-09-17T16:37:32.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.679135+0000", 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.332262907 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.679256+0000", 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012170700000000001 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:59.679290+0000", 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3636000000000001e-05 2024-09-17T16:37:32.139 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:134919 8.c 8:3b552185:::benchmark_data_smithi173_130566_object39944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:19.240157+0000", 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 12.774954038000001, 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23803208000000001, 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.140 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 134919 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240157+0000", 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240157+0000", 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240159+0000", 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0279999999999999e-06 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240167+0000", 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8730000000000007e-06 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240189+0000", 2024-09-17T16:37:32.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2079e-05 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240200+0000", 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0658e-05 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240271+0000", 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1200999999999998e-05 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240412+0000", 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014078300000000001 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240541+0000", 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012865500000000001 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.244318+0000", 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037768839999999999 2024-09-17T16:37:32.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.244381+0000", 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3064000000000005e-05 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.478088+0000", 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.233706894 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.478146+0000", 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8187999999999999e-05 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.478189+0000", 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3773000000000003e-05 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:141534 8.b 8:d610c369:::benchmark_data_smithi173_130566_object46559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e814)", 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:28.648975+0000", 2024-09-17T16:37:32.146 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.366136547, 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20427547300000001, 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 141534 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.648975+0000", 2024-09-17T16:37:32.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.648975+0000", 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.648975+0000", 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.84e-07 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.648977+0000", 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.035e-06 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.648983+0000", 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9120000000000003e-06 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.648987+0000", 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3739999999999998e-06 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:32.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.649098+0000", 2024-09-17T16:37:32.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111597 2024-09-17T16:37:32.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:32.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.649177+0000", 2024-09-17T16:37:32.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8685999999999998e-05 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.649245+0000", 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8190999999999998e-05 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.652872+0000", 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003627384 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.763054+0000", 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11018122700000001 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.853134+0000", 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.090080714000000006 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.853197+0000", 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2589000000000007e-05 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:32.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:32.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:32.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:28.853250+0000", 2024-09-17T16:37:32.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3380000000000001e-05 2024-09-17T16:37:32.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.154 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:32.154 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:32.154 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:37:32.154 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:32.155 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:31 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:37:32.156 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:31 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:37:32.156 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:31 smithi173 ceph-mon[30530]: osdmap e816: 12 total, 12 up, 12 in 2024-09-17T16:37:32.156 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:31 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:37:32.156 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:31 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:37:32.177 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:32.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:32.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:32.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:31 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:37:32.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:31 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:37:32.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:31 smithi017 ceph-mon[27571]: osdmap e816: 12 total, 12 up, 12 in 2024-09-17T16:37:32.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:31 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:37:32.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:31 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:37:32.278 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:32.314 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:31 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:37:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:31 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:37:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:31 smithi071 ceph-mon[30002]: osdmap e816: 12 total, 12 up, 12 in 2024-09-17T16:37:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:31 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:37:32.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:31 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:37:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:32.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:32.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:32.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:32.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:32.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:32.573 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:32.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:32.675 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:32.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:32.718 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:32.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:32.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:32.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:32.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:32 smithi173 ceph-mon[30530]: osdmap e817: 12 total, 12 up, 12 in 2024-09-17T16:37:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:32 smithi173 ceph-mon[30530]: pgmap v1818: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 20283/143739 objects degraded (14.111%); 4.2 MiB/s, 67 objects/s recovering 2024-09-17T16:37:32.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:32.979 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:32.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:32.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:32 smithi017 ceph-mon[27571]: osdmap e817: 12 total, 12 up, 12 in 2024-09-17T16:37:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:32 smithi017 ceph-mon[27571]: pgmap v1818: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 20283/143739 objects degraded (14.111%); 4.2 MiB/s, 67 objects/s recovering 2024-09-17T16:37:33.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:33.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:33.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:33.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:33.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:37:33.212 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:33.233 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:33.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:32 smithi071 ceph-mon[30002]: osdmap e817: 12 total, 12 up, 12 in 2024-09-17T16:37:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:32 smithi071 ceph-mon[30002]: pgmap v1818: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 20283/143739 objects degraded (14.111%); 4.2 MiB/s, 67 objects/s recovering 2024-09-17T16:37:33.335 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:33.360 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:33.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:33.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:33.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:33.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:33.539 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:33.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:33.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:33.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:33.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:33.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:33.766 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:33.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:33.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:33.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:33.901 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:33.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:34.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:34.025 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:34.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:34.126 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:34.148 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:34.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:34.250 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:34.273 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:34.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:34.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:34.399 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:34.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:34.453 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:37:34.453 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:37:34.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:34.454 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:146212 8.c e817/728 8:323bb84d:::benchmark_data_smithi173_130566_object51237:head v 817'8822, mlcod=817'8820)", 2024-09-17T16:37:34.454 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:34.378379+0000", 2024-09-17T16:37:34.454 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0019511019999999999, 2024-09-17T16:37:34.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019822099999999999, 2024-09-17T16:37:34.454 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:34.454 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:34.454 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.378379+0000", 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.378379+0000", 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:34.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.378380+0000", 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.082e-06 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.378388+0000", 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6249999999999998e-06 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.378428+0000", 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9728000000000003e-05 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:34.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.378435+0000", 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7719999999999994e-06 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.378513+0000", 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7589999999999994e-05 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.378533+0000", 2024-09-17T16:37:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9636999999999999e-05 2024-09-17T16:37:34.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:34.458 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:34.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:34.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:34.458 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:37:34.458 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-09-17T16:37:34.458 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:37:34.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:34.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:34.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:34.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:34.660 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:34.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:34.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:34.793 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:34.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:34.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:34.931 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:34.932 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:35.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:35.070 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:35.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:35.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:35.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:35.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:35.304 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:35.333 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:35.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:35.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:35.458 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:35.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:35 smithi017 ceph-mon[27571]: pgmap v1819: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 726 op/s; 19495/139680 objects degraded (13.957%); 11 MiB/s, 179 objects/s recovering 2024-09-17T16:37:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:35 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 19495/139680 objects degraded (13.957%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:35.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:35 smithi071 ceph-mon[30002]: pgmap v1819: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 726 op/s; 19495/139680 objects degraded (13.957%); 11 MiB/s, 179 objects/s recovering 2024-09-17T16:37:35.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:35 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 19495/139680 objects degraded (13.957%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:35.582 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:35.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:35.594 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:35 smithi173 ceph-mon[30530]: pgmap v1819: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 726 op/s; 19495/139680 objects degraded (13.957%); 11 MiB/s, 179 objects/s recovering 2024-09-17T16:37:35.594 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:35 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 19495/139680 objects degraded (13.957%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:35.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:37:35.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:35.707 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:35.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:35.809 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:35.848 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:35.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:35.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:35.982 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:35.983 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:35.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:36.085 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:36.123 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:36.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:36.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:36.249 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:36.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:36.352 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:36.384 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:36.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:36.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:36 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:37:36.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:36 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:37:36.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:36 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:37:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:36 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:37:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:36.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:36.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:36.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:36 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:37:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:36 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:37:36.573 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:37:36.573 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:37:36.573 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:37:36.574 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:37:36.574 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:37:36.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:36.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:36.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:36.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:36.785 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:36.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:36.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:36.887 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:36.910 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:36.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:37.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:37.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:37.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:37.149 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:37.181 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:37.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:37.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:37:37.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:37.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:37.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:37.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:37.439 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:37.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:37 smithi017 ceph-mon[27571]: pgmap v1820: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 19150/131955 objects degraded (14.513%); 14 MiB/s, 222 objects/s recovering 2024-09-17T16:37:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:37 smithi017 ceph-mon[27571]: osdmap e818: 12 total, 12 up, 12 in 2024-09-17T16:37:37.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:37 smithi071 ceph-mon[30002]: pgmap v1820: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 19150/131955 objects degraded (14.513%); 14 MiB/s, 222 objects/s recovering 2024-09-17T16:37:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:37 smithi071 ceph-mon[30002]: osdmap e818: 12 total, 12 up, 12 in 2024-09-17T16:37:37.570 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:37.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:37.571 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:37 smithi173 ceph-mon[30530]: pgmap v1820: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 848 op/s; 19150/131955 objects degraded (14.513%); 14 MiB/s, 222 objects/s recovering 2024-09-17T16:37:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:37 smithi173 ceph-mon[30530]: osdmap e818: 12 total, 12 up, 12 in 2024-09-17T16:37:37.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:37.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:37.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:37.799 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:37.825 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:37.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:37.927 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:37.966 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:37.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:38.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:38.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:38.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:38.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:38.148 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:37:38.148 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:37:38.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:37:38.148 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:37:38.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.148 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26665.0:186285 6.f e621/527 6:ff66d09f:::benchmark_data_smithi173_102017_object93137:head v 621'11698, mlcod=621'11694)", 2024-09-17T16:37:38.148 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:17.540727+0000", 2024-09-17T16:37:38.148 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 500.603064276, 2024-09-17T16:37:38.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53518362600000002, 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540727+0000", 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540727+0000", 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540728+0000", 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.361e-06 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:17.540742+0000", 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4087000000000001e-05 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.071882+0000", 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53114022100000002 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.071897+0000", 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4039000000000001e-05 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.072016+0000", 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011962499999999999 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.072036+0000", 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0035999999999999e-05 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.075868+0000", 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038314 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:18.075910+0000", 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2857e-05 2024-09-17T16:37:38.152 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:21072 7.5 7:a6915c4f:::benchmark_data_smithi173_116396_object21071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e643)", 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:29:58.907216+0000", 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 459.236574797, 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32350631499999999, 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:38.153 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21072 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907216+0000", 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907216+0000", 2024-09-17T16:37:38.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907218+0000", 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.638e-06 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907236+0000", 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7849000000000001e-05 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907248+0000", 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1933000000000001e-05 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907254+0000", 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0689999999999998e-06 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907308+0000", 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3940000000000002e-05 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907441+0000", 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133685 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.907543+0000", 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101369 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.908430+0000", 2024-09-17T16:37:38.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088757899999999999 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:58.908686+0000", 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025581999999999997 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230610+0000", 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32192386200000001 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230673+0000", 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2668000000000004e-05 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:29:59.230723+0000", 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9903e-05 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:111379 7.0 e681/674 7:05ca7e9f:::benchmark_data_smithi173_116396_object15779:head v 681'6620, mlcod=681'6617)", 2024-09-17T16:37:38.159 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:41.226937+0000", 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 356.91685450900002, 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42183363800000001, 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.226937+0000", 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.226937+0000", 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.226938+0000", 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2869999999999999e-06 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.226946+0000", 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4980000000000006e-06 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.647196+0000", 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42024978200000002 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.647206+0000", 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4180000000000003e-06 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.647303+0000", 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7071999999999995e-05 2024-09-17T16:37:38.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.647321+0000", 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8780999999999999e-05 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.648742+0000", 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001420997 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:41.648770+0000", 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7803000000000001e-05 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:116084 7.5 7:a0912ad1:::benchmark_data_smithi173_116396_object20484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e685)", 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:50.990260+0000", 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 347.15353153799998, 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28407464199999999, 2024-09-17T16:37:38.164 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 116084 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990260+0000", 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990260+0000", 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990261+0000", 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6670000000000001e-06 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990267+0000", 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5060000000000003e-06 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990279+0000", 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2644e-05 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990286+0000", 2024-09-17T16:37:38.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5549999999999999e-06 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990346+0000", 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0047999999999998e-05 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990482+0000", 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013580800000000001 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:37:38.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.990620+0000", 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013782200000000001 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.992390+0000", 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001770647 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.993010+0000", 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062002699999999999 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274221+0000", 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28121024300000003 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274282+0000", 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1865000000000005e-05 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274334+0000", 2024-09-17T16:37:38.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.181e-05 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27088.0:116084 7.5 e685/674 ondisk, result = 0)", 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:50.992582+0000", 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 347.151208962, 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28175829000000002, 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.992582+0000", 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.992582+0000", 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.992583+0000", 2024-09-17T16:37:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.61e-07 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:50.992587+0000", 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.498e-06 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274134+0000", 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28154638500000001 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274139+0000", 2024-09-17T16:37:38.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7180000000000001e-06 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274200+0000", 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0126e-05 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274209+0000", 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1249999999999999e-06 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:51.274340+0000", 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000131777 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:117077 7.5 7:a4e89f58:::benchmark_data_smithi173_116396_object21477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e685)", 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:53.097682+0000", 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 345.04610883700002, 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.240555148, 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.175 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 117077 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097682+0000", 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097682+0000", 2024-09-17T16:37:38.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097684+0000", 2024-09-17T16:37:38.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8020000000000001e-06 2024-09-17T16:37:38.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097691+0000", 2024-09-17T16:37:38.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1720000000000001e-06 2024-09-17T16:37:38.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097708+0000", 2024-09-17T16:37:38.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6521999999999999e-05 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097715+0000", 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1130000000000003e-06 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097783+0000", 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7731999999999995e-05 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097884+0000", 2024-09-17T16:37:38.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010187999999999999 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.097963+0000", 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8838999999999995e-05 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.099970+0000", 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020066760000000002 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.104300+0000", 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043297980000000002 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338164+0000", 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23386404299999999 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338198+0000", 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4224999999999998e-05 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338237+0000", 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9345999999999999e-05 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27088.0:117077 7.5 e685/674 ondisk, result = 0)", 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:53.100127+0000", 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 345.04366449999998, 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.238116831, 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.182 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.100127+0000", 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.100127+0000", 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.100127+0000", 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8299999999999997e-07 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.100130+0000", 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2959999999999999e-06 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338112+0000", 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23798161700000001 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338117+0000", 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5279999999999997e-06 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338154+0000", 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7812000000000003e-05 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338160+0000", 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4360000000000001e-06 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:53.338243+0000", 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3559000000000004e-05 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:170897 7.5 7:a2dbce43:::benchmark_data_smithi173_116396_object75297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:37:38.186 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:26.438350+0000", 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 251.70544144499999, 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34698544799999997, 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 170897 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438350+0000", 2024-09-17T16:37:38.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438350+0000", 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438351+0000", 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.815e-06 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438358+0000", 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8260000000000004e-06 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438372+0000", 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4131000000000001e-05 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438380+0000", 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4449999999999999e-06 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438438+0000", 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7833e-05 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438537+0000", 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8987999999999995e-05 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438658+0000", 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121291 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.441438+0000", 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027800049999999999 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.453632+0000", 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012194439 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:38.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785264+0000", 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33163123999999999 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785303+0000", 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8894999999999999e-05 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785335+0000", 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2539999999999997e-05 2024-09-17T16:37:38.193 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(7.0 e724,723 7:00d6d267:::benchmark_data_smithi173_116396_object11771:head,679'6499)", 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:54.910007+0000", 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 223.233784125, 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71534004299999998, 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910007+0000", 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910007+0000", 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910009+0000", 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6169999999999999e-06 2024-09-17T16:37:38.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910019+0000", 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0047000000000001e-05 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.624856+0000", 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71483768199999997 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.624864+0000", 2024-09-17T16:37:38.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5909999999999998e-06 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.624943+0000", 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9234999999999996e-05 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.625347+0000", 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040387099999999998 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.197 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.15 740/739 [PushOp(8:a91ad1da:::benchmark_data_smithi173_130566_object2731:head, version: 732'202, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:a91ad1da:::benchmark_data_smithi173_130566_object2731:head@732'202, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 191.046607026, 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448453453, 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:37:38.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:37:38.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097185+0000", 2024-09-17T16:37:38.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2799999999999995e-07 2024-09-17T16:37:38.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097245+0000", 2024-09-17T16:37:38.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9749e-05 2024-09-17T16:37:38.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545252+0000", 2024-09-17T16:37:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44800729700000003 2024-09-17T16:37:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545267+0000", 2024-09-17T16:37:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5017999999999999e-05 2024-09-17T16:37:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545346+0000", 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9491000000000002e-05 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545357+0000", 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0491e-05 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545638+0000", 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028057899999999998 2024-09-17T16:37:38.202 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.13 740/739 [PushOp(8:cce6ff3a:::benchmark_data_smithi173_130566_object3681:head, version: 732'214, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:cce6ff3a:::benchmark_data_smithi173_130566_object3681:head@732'214, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 191.04466453699999, 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448568835, 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:37:38.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7899999999999997e-07 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099181+0000", 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3591999999999998e-05 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547347+0000", 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448166224 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547363+0000", 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5364000000000001e-05 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547441+0000", 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8873000000000002e-05 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547450+0000", 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5469999999999997e-06 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547695+0000", 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024535600000000001 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:113416 8.4 8.9400fec4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:51.833544+0000", 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.310247234000002, 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19042836099999999, 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.207 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 113416 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.833544+0000", 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.833544+0000", 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.833545+0000", 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4899999999999999e-06 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.833551+0000", 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1229999999999997e-06 2024-09-17T16:37:38.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.833565+0000", 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3157e-05 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.833572+0000", 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1350000000000003e-06 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.833626+0000", 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3983999999999997e-05 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.833633+0000", 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1609999999999997e-06 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:52.023939+0000", 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19030571099999999 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:52.023972+0000", 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3599999999999997e-05 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:115081 8.0 e800/767 8:02125afa:::benchmark_data_smithi173_130566_object20106:head v 800'3851, mlcod=800'3848)", 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:54.801782+0000", 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.342008610000001, 2024-09-17T16:37:38.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66971172400000001, 2024-09-17T16:37:38.213 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.213 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.213 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.213 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801782+0000", 2024-09-17T16:37:38.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801782+0000", 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801783+0000", 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0500000000000002e-07 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801788+0000", 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1939999999999997e-06 2024-09-17T16:37:38.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469802+0000", 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66801375100000004 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469811+0000", 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.473e-06 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469859+0000", 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7874999999999998e-05 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469877+0000", 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8321e-05 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.471459+0000", 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00158165 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.471494+0000", 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5655000000000003e-05 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(8.4 e800,797 8:22e2a6d6:::benchmark_data_smithi173_130566_object14161:head,794'3581)", 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:55.001177+0000", 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.142613744000002, 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38830608500000002, 2024-09-17T16:37:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.001177+0000", 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.001177+0000", 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.001178+0000", 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0500000000000003e-07 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.001183+0000", 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8130000000000003e-06 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.387655+0000", 2024-09-17T16:37:38.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.386472549 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.387680+0000", 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4977000000000002e-05 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.387871+0000", 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019087999999999999 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.389483+0000", 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001612161 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:115204 8.4 8:225dd052:::benchmark_data_smithi173_130566_object20229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:55.198272+0000", 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 42.945519007999998, 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19615502500000001, 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.221 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 115204 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.198272+0000", 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.198272+0000", 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.198273+0000", 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3400000000000001e-06 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.198280+0000", 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1840000000000004e-06 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.198291+0000", 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1275e-05 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.198297+0000", 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6509999999999998e-06 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.198340+0000", 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3844000000000002e-05 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.198442+0000", 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101532 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,10", 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.198524+0000", 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2589000000000005e-05 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.200249+0000", 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017240280000000001 2024-09-17T16:37:38.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.200549+0000", 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030036100000000002 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.393805+0000", 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19325637900000001 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.393873+0000", 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7243000000000007e-05 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.394427+0000", 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00055459900000000002 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:116728 8.15 e802/743 8:a96131de:::benchmark_data_smithi173_130566_object21753:head v 802'3888, mlcod=802'3885)", 2024-09-17T16:37:38.227 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:57.515606+0000", 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.628185293999998, 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.344180555, 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.515606+0000", 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.515606+0000", 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.515607+0000", 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8000000000000004e-07 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.515613+0000", 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0410000000000003e-06 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.857627+0000", 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34201457499999999 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.857635+0000", 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8029999999999997e-06 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.857731+0000", 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6113000000000006e-05 2024-09-17T16:37:38.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.857751+0000", 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9983000000000002e-05 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.859742+0000", 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019903719999999998 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.859786+0000", 2024-09-17T16:37:38.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4787999999999999e-05 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(8.4 807/797 [PushReplyOp(8:20aecd0b:::benchmark_data_smithi173_130566_object88362:head)])", 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:15.867816+0000", 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 22.275974744999999, 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.196850048, 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:15.867816+0000", 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:15.867816+0000", 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:15.867818+0000", 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2550000000000001e-06 2024-09-17T16:37:38.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:15.867824+0000", 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5339999999999996e-06 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:16.064190+0000", 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19636610700000001 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:16.064198+0000", 2024-09-17T16:37:38.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6020000000000002e-06 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:16.064262+0000", 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4319000000000005e-05 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:16.064666+0000", 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040423100000000001 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:38.237 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:134919 8.c e810/728 8:3b552185:::benchmark_data_smithi173_130566_object39944:head v 810'8168, mlcod=809'8165)", 2024-09-17T16:37:38.237 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:19.240659+0000", 2024-09-17T16:37:38.237 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 18.903131607999999, 2024-09-17T16:37:38.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.233729831, 2024-09-17T16:37:38.237 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.238 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.238 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.238 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240659+0000", 2024-09-17T16:37:38.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240659+0000", 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240661+0000", 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3e-06 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.240670+0000", 2024-09-17T16:37:38.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3090000000000001e-06 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.467013+0000", 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.226343348 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.467019+0000", 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3340000000000001e-06 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.467074+0000", 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5356000000000003e-05 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.467098+0000", 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3938999999999999e-05 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.474343+0000", 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0072447029999999999 2024-09-17T16:37:38.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:19.474389+0000", 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6542000000000003e-05 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:138881 8.13 e813/742 8:cc7a69da:::benchmark_data_smithi173_130566_object43906:head v 813'4461, mlcod=813'4458)", 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:25.399836+0000", 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 12.74395528, 2024-09-17T16:37:38.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22007708000000001, 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:25.399836+0000", 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:25.399836+0000", 2024-09-17T16:37:38.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:25.399837+0000", 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8700000000000004e-07 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:25.399844+0000", 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0589999999999997e-06 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:25.617976+0000", 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21813231999999999 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:25.617983+0000", 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.379e-06 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:25.618041+0000", 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7800000000000002e-05 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:25.618063+0000", 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1690000000000001e-05 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:25.619867+0000", 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018044339999999999 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:25.619913+0000", 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5510999999999998e-05 2024-09-17T16:37:38.246 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:146559 8.c e817/728 8:39bf53b9:::benchmark_data_smithi173_130566_object51584:head v 817'8848, mlcod=817'8845)", 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:34.689242+0000", 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.4545487110000002, 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11875828099999999, 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.689242+0000", 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.689242+0000", 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.689244+0000", 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3170000000000001e-06 2024-09-17T16:37:38.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.689253+0000", 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6239999999999997e-06 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.803288+0000", 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.114034776 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.803298+0000", 2024-09-17T16:37:38.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8619999999999998e-06 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.803359+0000", 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0616999999999999e-05 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.803381+0000", 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2785e-05 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:38.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.807953+0000", 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045715139999999996 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:34.808001+0000", 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7786e-05 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:38.251 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:37:38.282 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:38.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:38.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:38.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:38 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:37:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:38 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:37:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:38 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:37:38.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:38.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:38.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:38 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:37:38.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:38 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:37:38.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:38 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:37:38.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:38.541 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:38.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:38.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:38 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:37:38.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:38 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:37:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:38 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:37:38.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:38.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:38.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:38.770 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:38.792 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:38.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:38.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:38.917 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:38.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:39.019 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:39.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:39.049 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:39.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:39.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:37:39.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:39.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:39.173 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:39.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:39.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:39.298 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:39.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:39.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:39 smithi173 ceph-mon[30530]: pgmap v1822: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 18292/128229 objects degraded (14.265%); 20 MiB/s, 313 objects/s recovering 2024-09-17T16:37:39.400 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:39.422 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:39.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:39.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:39.464 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:39 smithi071 ceph-mon[30002]: pgmap v1822: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 18292/128229 objects degraded (14.265%); 20 MiB/s, 313 objects/s recovering 2024-09-17T16:37:39.466 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:39 smithi017 ceph-mon[27571]: pgmap v1822: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 836 op/s; 18292/128229 objects degraded (14.265%); 20 MiB/s, 313 objects/s recovering 2024-09-17T16:37:39.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:39.563 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:39.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:39.665 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:39.689 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:39.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:39.792 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:39.817 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:39.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:39.920 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:39.952 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:39.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:40.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:40.078 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:40.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:40.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:40.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:40.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:40.307 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:40.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:40 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 18292/128229 objects degraded (14.265%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:40.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:40 smithi071 ceph-mon[30002]: osdmap e819: 12 total, 12 up, 12 in 2024-09-17T16:37:40.332 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:40.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:40.352 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:37:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:37:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:151115 8.d e819/728 8:b4bd6e99:::benchmark_data_smithi173_130566_object56140:head v 819'9562, mlcod=819'9560)", 2024-09-17T16:37:40.352 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:40.346911+0000", 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.002291941, 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023197080000000002, 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.346911+0000", 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:40.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.346911+0000", 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.346912+0000", 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1800000000000004e-07 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.346919+0000", 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5569999999999998e-06 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:40.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.346951+0000", 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1433999999999998e-05 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.346958+0000", 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4259999999999997e-06 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.347012+0000", 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3693000000000002e-05 2024-09-17T16:37:40.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.347026+0000", 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4385e-05 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-09-17T16:37:40.356 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:37:40.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:40 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 18292/128229 objects degraded (14.265%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:40.366 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:40 smithi173 ceph-mon[30530]: osdmap e819: 12 total, 12 up, 12 in 2024-09-17T16:37:40.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:40.466 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:40.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:40.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:40 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 18292/128229 objects degraded (14.265%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T16:37:40.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:40 smithi017 ceph-mon[27571]: osdmap e819: 12 total, 12 up, 12 in 2024-09-17T16:37:40.547 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:37:40.568 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:40.612 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:40.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:40.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:40.714 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:40.742 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:40.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:40.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:40.844 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:40.876 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:40.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:40.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:40.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:40.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:37:41.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:41.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:41.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:41.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:41.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:41.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:41.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:41.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:41.332 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:41.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:41.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:41.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:41.422 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:37:41.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:41 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:37:41.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:41 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:37:41.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:41 smithi017 ceph-mon[27571]: pgmap v1824: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 938 op/s; 18288/121218 objects degraded (15.087%); 16 MiB/s, 249 objects/s recovering 2024-09-17T16:37:41.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:41 smithi017 ceph-mon[27571]: osdmap e820: 12 total, 12 up, 12 in 2024-09-17T16:37:41.454 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:41 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:37:41.454 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:41 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:37:41.454 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:41 smithi173 ceph-mon[30530]: pgmap v1824: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 938 op/s; 18288/121218 objects degraded (15.087%); 16 MiB/s, 249 objects/s recovering 2024-09-17T16:37:41.455 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:41 smithi173 ceph-mon[30530]: osdmap e820: 12 total, 12 up, 12 in 2024-09-17T16:37:41.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:41.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:41.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:41 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:37:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:41 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:37:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:41 smithi071 ceph-mon[30002]: pgmap v1824: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 938 op/s; 18288/121218 objects degraded (15.087%); 16 MiB/s, 249 objects/s recovering 2024-09-17T16:37:41.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:41 smithi071 ceph-mon[30002]: osdmap e820: 12 total, 12 up, 12 in 2024-09-17T16:37:41.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:41.638 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:41.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:41.740 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:41.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:41.765 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:41.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:41.866 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:41.904 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:41.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:41.927 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:37:41.927 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:37:41.927 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:37:41.927 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:37:41.927 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:37:42.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:42.039 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:42.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:42.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:42.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:42.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:42.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:42.298 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:42.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:42.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:42.427 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:42.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:42.529 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:42.552 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:42.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:42.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:37:42.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:42.674 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:42.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:42.777 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:42.799 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:42.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:42.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:42.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:42.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:42.969 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:43.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:43.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:43.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:43.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:43.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:43.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:43.298 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:43.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:43.330 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:43.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:43.373 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:43 smithi173 ceph-mon[30530]: pgmap v1826: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 17927/118119 objects degraded (15.177%); 13 MiB/s, 203 objects/s recovering 2024-09-17T16:37:43.373 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2846236132' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:37:43.373 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:43 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:37:43.432 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:43.469 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:43.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:43 smithi017 ceph-mon[27571]: pgmap v1826: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 17927/118119 objects degraded (15.177%); 13 MiB/s, 203 objects/s recovering 2024-09-17T16:37:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2846236132' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:37:43.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:43 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:37:43.522 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:37:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:37:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:37:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:37:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:29928 7.a e646/629 7:54337644:::benchmark_data_smithi173_116396_object29927:head v 646'1856, mlcod=646'1852)", 2024-09-17T16:37:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:37:43.522 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 456.22790203199997, 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.232420081, 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:37:43.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290207+0000", 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6579999999999996e-06 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290242+0000", 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4696000000000001e-05 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521561+0000", 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23131880499999999 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521567+0000", 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.392e-06 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521628+0000", 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0473999999999997e-05 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521650+0000", 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2909e-05 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.522588+0000", 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093747100000000005 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.522621+0000", 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2675999999999997e-05 2024-09-17T16:37:43.526 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:46467 7.b e650/629 7:d098567c:::benchmark_data_smithi173_116396_object46466:head v 650'2942, mlcod=650'2939)", 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 442.35445621100001, 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.213043449, 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:37:43.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163646+0000", 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163647+0000", 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.163708+0000", 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1152e-05 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.374934+0000", 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21122532999999999 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.374948+0000", 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4756000000000001e-05 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.375068+0000", 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119771 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.375108+0000", 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9619000000000001e-05 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.376653+0000", 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001545416 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:21.376690+0000", 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6631000000000002e-05 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:103843 7.b e678/629 7:d0ef6818:::benchmark_data_smithi173_116396_object8243:head v 678'6575, mlcod=678'6572)", 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:25.488955+0000", 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 378.0291479, 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.209073274, 2024-09-17T16:37:43.531 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488955+0000", 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488955+0000", 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488956+0000", 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4610000000000001e-06 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.488964+0000", 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2910000000000007e-06 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695150+0000", 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20618561899999999 2024-09-17T16:37:43.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695166+0000", 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5926e-05 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695324+0000", 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000157873 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.695362+0000", 2024-09-17T16:37:43.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8170000000000002e-05 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.697999+0000", 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026369560000000002 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:25.698028+0000", 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8977999999999999e-05 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.536 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 685/677 [PushOp(7:2d0cb49e:::benchmark_data_smithi173_116396_object20958:head, version: 643'1334, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2d0cb49e:::benchmark_data_smithi173_116396_object20958:head@643'1334, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:43.536 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:37:43.536 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 351.49197845800001, 2024-09-17T16:37:43.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25935365100000002, 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026125+0000", 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026176+0000", 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1581e-05 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283828+0000", 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25765147999999999 2024-09-17T16:37:43.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283845+0000", 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7275e-05 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283961+0000", 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011592899999999999 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283974+0000", 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2724e-05 2024-09-17T16:37:43.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.285478+0000", 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015038720000000001 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 685/677 [PushOp(7:295d1d25:::benchmark_data_smithi173_116396_object23683:head, version: 645'1499, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:295d1d25:::benchmark_data_smithi173_116396_object23683:head@645'1499, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 345.96673115499999, 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24965942299999999, 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.540 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551375+0000", 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9809999999999997e-06 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551452+0000", 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7064999999999995e-05 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799411+0000", 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.247958129 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799426+0000", 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5094e-05 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799521+0000", 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5449e-05 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799534+0000", 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3077e-05 2024-09-17T16:37:43.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.801031+0000", 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014966280000000001 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 687/677 [PushOp(7:2aa25eb8:::benchmark_data_smithi173_116396_object25172:head, version: 645'1590, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2aa25eb8:::benchmark_data_smithi173_116396_object25172:head@645'1590, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 342.36639186299999, 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25723122799999998, 2024-09-17T16:37:43.544 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151712+0000", 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3030000000000001e-06 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151805+0000", 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3330000000000003e-05 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.406885+0000", 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25507977999999998 2024-09-17T16:37:43.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.406910+0000", 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4736000000000002e-05 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.407016+0000", 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106372 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.407029+0000", 2024-09-17T16:37:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3032e-05 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.408942+0000", 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001912675 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 690/677 [PushOp(7:2d367e55:::benchmark_data_smithi173_116396_object28235:head, version: 646'1783, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2d367e55:::benchmark_data_smithi173_116396_object28235:head@646'1783, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:37:43.548 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 336.38143221600001, 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26142421399999999, 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:37:43.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136671+0000", 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210000000000001e-06 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136740+0000", 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8932999999999999e-05 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396250+0000", 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25950979000000002 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396266+0000", 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5874000000000001e-05 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396362+0000", 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5543000000000003e-05 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396374+0000", 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2897999999999999e-05 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.398095+0000", 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017201549999999999 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:123977 7.14 e690/677 7:2b1a1b49:::benchmark_data_smithi173_116396_object28377:head v 690'7133, mlcod=646'1781)", 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:37:43.552 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 336.07789797200002, 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.206586876, 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440205+0000", 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.440212+0000", 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2199999999999997e-06 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.641936+0000", 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20172389900000001 2024-09-17T16:37:43.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.641943+0000", 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1010000000000003e-06 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.641998+0000", 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5139999999999997e-05 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.642025+0000", 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7662999999999999e-05 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.646753+0000", 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047274800000000001 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.646791+0000", 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8574e-05 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.556 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:129321 7.b e695/629 7:d9dcd2c1:::benchmark_data_smithi173_116396_object33721:head v 695'8200, mlcod=695'8195)", 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:17.838831+0000", 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 325.679271406, 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18822972700000001, 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838831+0000", 2024-09-17T16:37:43.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838831+0000", 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838832+0000", 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.054e-06 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.838839+0000", 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7059999999999998e-06 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022112+0000", 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18327316599999999 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022120+0000", 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9039999999999992e-06 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022180+0000", 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0446999999999999e-05 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.022202+0000", 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1750999999999999e-05 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.027011+0000", 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048087299999999998 2024-09-17T16:37:43.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.027061+0000", 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9969000000000002e-05 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:129340 7.b e695/629 7:d6eb756c:::benchmark_data_smithi173_116396_object33740:head v 695'8201, mlcod=695'8197)", 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 325.650275534, 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.190853825, 2024-09-17T16:37:43.561 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867827+0000", 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4700000000000002e-07 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:17.867833+0000", 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2320000000000001e-06 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054604+0000", 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18677139100000001 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054610+0000", 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1029999999999998e-06 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054667+0000", 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6928000000000001e-05 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.054688+0000", 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0653e-05 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.058610+0000", 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039225379999999997 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:18.058681+0000", 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0532999999999997e-05 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.565 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 696/677 [PushOp(7:2bba9c3c:::benchmark_data_smithi173_116396_object40626:head, version: 650'2606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2bba9c3c:::benchmark_data_smithi173_116396_object40626:head@650'2606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:30.395026+0000", 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 313.12307683799997, 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19917216800000001, 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395026+0000", 2024-09-17T16:37:43.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395026+0000", 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395027+0000", 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.421e-06 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.395110+0000", 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2409000000000003e-05 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591821+0000", 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19671128299999999 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591844+0000", 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.34e-05 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591949+0000", 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010453700000000001 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.591961+0000", 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2354000000000001e-05 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:30.594198+0000", 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002236764 2024-09-17T16:37:43.569 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:170897 7.5 e712/674 7:a2dbce43:::benchmark_data_smithi173_116396_object75297:head v 712'8535, mlcod=712'8532)", 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 257.079360167, 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34638644600000001, 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438743+0000", 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0529999999999999e-06 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438750+0000", 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7989999999999996e-06 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.782943+0000", 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34419256799999998 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.782967+0000", 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3929000000000001e-05 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.783013+0000", 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6400999999999998e-05 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.783027+0000", 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4337e-05 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:43.574 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:43 smithi071 ceph-mon[30002]: pgmap v1826: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 17927/118119 objects degraded (15.177%); 13 MiB/s, 203 objects/s recovering 2024-09-17T16:37:43.575 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2846236132' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:37:43.575 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:43 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:37:43.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785093+0000", 2024-09-17T16:37:43.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002065271 2024-09-17T16:37:43.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785129+0000", 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6087999999999997e-05 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:178680 7.13 e719/714 7:cae8152e:::benchmark_data_smithi173_116396_object83080:head v 719'8741, mlcod=719'8738)", 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 245.43204511900001, 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31485432099999999, 2024-09-17T16:37:43.576 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086058+0000", 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.85e-07 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086063+0000", 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0050000000000004e-06 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398605+0000", 2024-09-17T16:37:43.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31254172400000002 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398610+0000", 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2789999999999998e-06 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398655+0000", 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4715000000000001e-05 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398670+0000", 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4895999999999999e-05 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.400870+0000", 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022001920000000001 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.400912+0000", 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1625000000000002e-05 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.580 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:191187 7.14 e724/717 7:2d119a99:::benchmark_data_smithi173_116396_object95587:head v 724'9235, mlcod=724'9232)", 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 229.070918909, 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48898925799999998, 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7599999999999996e-07 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447190+0000", 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9440000000000002e-06 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933730+0000", 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48653948800000002 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933739+0000", 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9199999999999993e-06 2024-09-17T16:37:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933837+0000", 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8004999999999994e-05 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933856+0000", 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.891e-05 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.936137+0000", 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022811960000000001 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.936173+0000", 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6019000000000002e-05 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:1442 8.f e731/728 8:f297aed2:::benchmark_data_smithi173_130566_object1441:head v 731'97, mlcod=731'92)", 2024-09-17T16:37:43.585 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:12.402004+0000", 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 211.11609854700001, 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.188874654, 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402004+0000", 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402004+0000", 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402005+0000", 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3200000000000004e-07 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.402059+0000", 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4203000000000003e-05 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590370+0000", 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18831092399999999 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590374+0000", 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.177e-06 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590443+0000", 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8479000000000006e-05 2024-09-17T16:37:43.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590466+0000", 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2901999999999999e-05 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590851+0000", 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038551500000000001 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:12.590879+0000", 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7622000000000001e-05 2024-09-17T16:37:43.589 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.16 740/739 [PushOp(8:68fc3949:::benchmark_data_smithi173_130566_object1854:head, version: 732'115, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:68fc3949:::benchmark_data_smithi173_130566_object1854:head@732'115, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 196.61886185, 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.75943818799999996, 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5700000000000002e-07 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899299+0000", 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7451999999999998e-05 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658299+0000", 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.75900059799999997 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658317+0000", 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:37:43.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658403+0000", 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6600000000000004e-05 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658413+0000", 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9340000000000001e-06 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658679+0000", 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026586099999999999 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "replica_scrub(pg: 8.d,from:0'0,to:772'6583,epoch:782/728,start:8:b03f31a8:::benchmark_data_smithi173_130566_object64387:0,end:8:b08c319a:::benchmark_data_smithi173_130566_object64164:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:12.777592+0000", 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 90.740510622000002, 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55434039499999999, 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.594 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:12.777592+0000", 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:12.777592+0000", 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:12.777598+0000", 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3629999999999999e-06 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:12.777609+0000", 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0655999999999999e-05 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:13.331787+0000", 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55417758299999997 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:13.331797+0000", 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0525e-05 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:13.331876+0000", 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9132000000000001e-05 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:13.331898+0000", 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1534000000000001e-05 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:13.331932+0000", 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4601999999999998e-05 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:113357 8.f e796/728 8:fcae9aeb:::benchmark_data_smithi173_130566_object18382:head v 796'7102, mlcod=795'7099)", 2024-09-17T16:37:43.598 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:51.182041+0000", 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 52.336061874000002, 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.203503397, 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.182041+0000", 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.182041+0000", 2024-09-17T16:37:43.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.182042+0000", 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1799999999999999e-06 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.182050+0000", 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6380000000000004e-06 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.382551+0000", 2024-09-17T16:37:43.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20050184200000001 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.382559+0000", 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3069999999999997e-06 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.382619+0000", 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0316999999999999e-05 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.382637+0000", 2024-09-17T16:37:43.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7563000000000001e-05 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.385516+0000", 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028793809999999999 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:51.385544+0000", 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8169000000000001e-05 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:151722 8.f e820/728 8:ff37f022:::benchmark_data_smithi173_130566_object56747:head v 820'9476, mlcod=820'9472)", 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:41.326544+0000", 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.1915582119999999, 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22418427199999999, 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:41.326544+0000", 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:41.326544+0000", 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:41.326546+0000", 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3459999999999999e-06 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:41.326555+0000", 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0019999999999995e-06 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:41.547078+0000", 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.220523309 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:41.547083+0000", 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8600000000000001e-06 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:41.547126+0000", 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3176000000000001e-05 2024-09-17T16:37:43.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:41.547142+0000", 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5610000000000001e-05 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:41.550698+0000", 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003556629 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:41.550729+0000", 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0340000000000001e-05 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:152548 8.3 e820/736 8:c1e3f3aa:::benchmark_data_smithi173_130566_object57573:head v 820'4995, mlcod=820'4992)", 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:42.473616+0000", 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.044486933, 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22429043800000001, 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:43.607 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.473616+0000", 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.473616+0000", 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.473617+0000", 2024-09-17T16:37:43.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.034e-06 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.473624+0000", 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3560000000000004e-06 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.690685+0000", 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21706067200000001 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:43.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.690693+0000", 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3909999999999998e-06 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.690771+0000", 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7776999999999998e-05 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.690791+0000", 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9731000000000001e-05 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.697825+0000", 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0070339369999999997 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.697906+0000", 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1539999999999995e-05 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:43.611 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:37:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:43.613 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:43.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:43.621 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T16:37:43.714 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:43.737 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:43.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:43.839 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:43.859 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:43.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:43.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:43.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:43.983 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:43.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:44.085 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:44.118 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:44.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:44.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:44.255 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:44.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:44.357 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:44.392 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:44.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:37:44.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:44 smithi017 ceph-mon[27571]: osdmap e821: 12 total, 12 up, 12 in 2024-09-17T16:37:44.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:44.523 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:44.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:37:44.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:44 smithi071 ceph-mon[30002]: osdmap e821: 12 total, 12 up, 12 in 2024-09-17T16:37:44.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:37:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:44 smithi173 ceph-mon[30530]: osdmap e821: 12 total, 12 up, 12 in 2024-09-17T16:37:44.651 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:44.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:44.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:44.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:44.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:44.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:44.908 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:44.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:45.010 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:45.032 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:45.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:45.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:37:45.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:45.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:45.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:45.259 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:45.261 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:45.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:45.300 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:45.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:45 smithi173 ceph-mon[30530]: pgmap v1828: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 17228/114543 objects degraded (15.041%); 9.6 MiB/s, 229 objects/s recovering 2024-09-17T16:37:45.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:45 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 17228/114543 objects degraded (15.041%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:37:45.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:45.405 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:45.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:45.425 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:45.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:45 smithi017 ceph-mon[27571]: pgmap v1828: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 17228/114543 objects degraded (15.041%); 9.6 MiB/s, 229 objects/s recovering 2024-09-17T16:37:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:45 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 17228/114543 objects degraded (15.041%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:37:45.526 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:45.529 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:45 smithi071 ceph-mon[30002]: pgmap v1828: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 17228/114543 objects degraded (15.041%); 9.6 MiB/s, 229 objects/s recovering 2024-09-17T16:37:45.529 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:45 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 17228/114543 objects degraded (15.041%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:37:45.548 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:45.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:45.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:45.681 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:45.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:45.783 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:45.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:45.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:45.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:45.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:46.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:46.068 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:37:46.069 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:37:46.069 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:37:46.069 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:37:46.082 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:46.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:46.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:46.211 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:46.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:46.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:46.345 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:46.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:46.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:46.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:46.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:46.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:46.604 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:46.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:46.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:46.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:46.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:46.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:46.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:46.832 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:46.855 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:46.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:46.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:46.981 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:46.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:47.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:47.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:47.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:47.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:47.237 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:47.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:47.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:47.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:47.372 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:47.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:47.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:47 smithi017 ceph-mon[27571]: pgmap v1829: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s; 16933/108786 objects degraded (15.565%); 12 MiB/s, 261 objects/s recovering 2024-09-17T16:37:47.497 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:47.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:47.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:47 smithi071 ceph-mon[30002]: pgmap v1829: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s; 16933/108786 objects degraded (15.565%); 12 MiB/s, 261 objects/s recovering 2024-09-17T16:37:47.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:47.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:47.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:47.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:47 smithi173 ceph-mon[30530]: pgmap v1829: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s; 16933/108786 objects degraded (15.565%); 12 MiB/s, 261 objects/s recovering 2024-09-17T16:37:47.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:47.759 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:47.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:47.861 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:47.890 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:47.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:47.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:48.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:48.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:48.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:37:48.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:48.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:48.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:48.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:48.334 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:48.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:48.398 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:48.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:48.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:48.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:48.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:48.587 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:48.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:48.689 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:48.718 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:48.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:48.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:48.842 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:48.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:48.909 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:37:48.909 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:37:48.909 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:37:48.909 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:37:48.909 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:37:48.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:48.967 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:48.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:49.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:49.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:49.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:49.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:49.193 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:49.217 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:49.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:49.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:49.355 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:49.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:49.457 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:49.483 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:49.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:49.489 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-17T16:37:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:49 smithi017 ceph-mon[27571]: pgmap v1830: 27 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 15847/105375 objects degraded (15.039%); 9.5 MiB/s, 384 objects/s recovering 2024-09-17T16:37:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:49 smithi017 ceph-mon[27571]: osdmap e822: 12 total, 12 up, 12 in 2024-09-17T16:37:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:49 smithi071 ceph-mon[30002]: pgmap v1830: 27 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 15847/105375 objects degraded (15.039%); 9.5 MiB/s, 384 objects/s recovering 2024-09-17T16:37:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:49 smithi071 ceph-mon[30002]: osdmap e822: 12 total, 12 up, 12 in 2024-09-17T16:37:49.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:49.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:37:49.610 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:49.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:49 smithi173 ceph-mon[30530]: pgmap v1830: 27 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 15847/105375 objects degraded (15.039%); 9.5 MiB/s, 384 objects/s recovering 2024-09-17T16:37:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:49 smithi173 ceph-mon[30530]: osdmap e822: 12 total, 12 up, 12 in 2024-09-17T16:37:49.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:49.742 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:49.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:49.844 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:49.877 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:49.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:49.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:49.913 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:49.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:50.001 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:50.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:50.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:50.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:50.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:50.229 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:50.255 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:50.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:50.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:50 smithi071 ceph-mon[30002]: osdmap e823: 12 total, 12 up, 12 in 2024-09-17T16:37:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:50 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 15847/105375 objects degraded (15.039%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:37:50.356 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:50.391 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:50.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:50.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:50.509 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(7.17 684/677 [PushOp(7:ef0cacf0:::benchmark_data_smithi173_116396_object24669:head, version: 645'1495, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:ef0cacf0:::benchmark_data_smithi173_116396_object24669:head@645'1495, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:31:44.909938+0000", 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 365.577040173, 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43308435099999998, 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:44.909938+0000", 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:44.909938+0000", 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:44.909939+0000", 2024-09-17T16:37:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1600000000000001e-07 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:44.909991+0000", 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2293000000000002e-05 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.341690+0000", 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43169834499999998 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.341718+0000", 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7912999999999999e-05 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.341900+0000", 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000182163 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.341909+0000", 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1989999999999995e-06 2024-09-17T16:37:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.343023+0000", 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001113822 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27088.0:113132 7.17 e684/677 7:ebb4b373:::benchmark_data_smithi173_116396_object17532:head v 684'6605, mlcod=645'1494)", 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:31:45.046789+0000", 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 365.440189218, 2024-09-17T16:37:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30454746300000002, 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.046789+0000", 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.046789+0000", 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.046799+0000", 2024-09-17T16:37:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6330000000000008e-06 2024-09-17T16:37:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.046811+0000", 2024-09-17T16:37:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2359e-05 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.348890+0000", 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30207848999999998 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.348896+0000", 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4239999999999998e-06 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.348948+0000", 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2167999999999999e-05 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.348968+0000", 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9457999999999999e-05 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.351302+0000", 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023337449999999999 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:45.351337+0000", 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5185999999999998e-05 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(7.17 685/677 [PushOp(7:eb5f58fe:::benchmark_data_smithi173_116396_object30697:head, version: 646'1864, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:eb5f58fe:::benchmark_data_smithi173_116396_object30697:head@646'1864, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:31:53.921113+0000", 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 356.56586586100002, 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29376142799999999, 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:53.921113+0000", 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:53.921113+0000", 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:53.921114+0000", 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5199999999999996e-07 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:53.921142+0000", 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8212e-05 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:54.213553+0000", 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29241161399999999 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:54.213563+0000", 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0117e-05 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:54.213651+0000", 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7732000000000006e-05 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:54.213664+0000", 2024-09-17T16:37:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2738999999999999e-05 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:31:54.214874+0000", 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012102619999999999 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(7.17 693/677 [PushOp(7:e8031d16:::benchmark_data_smithi173_116396_object47389:head, version: 650'2900, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:e8031d16:::benchmark_data_smithi173_116396_object47389:head@650'2900, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:32:15.125787+0000", 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 335.36119195499998, 2024-09-17T16:37:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38280529899999999, 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:15.125787+0000", 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:15.125787+0000", 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:15.125787+0000", 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1800000000000005e-07 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:15.125845+0000", 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8001000000000003e-05 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:15.507019+0000", 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38117327899999998 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:15.507030+0000", 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0951000000000001e-05 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:15.507126+0000", 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5947999999999997e-05 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:15.507137+0000", 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1748e-05 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:32:15.508592+0000", 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014546540000000001 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:57462 8.17 8:eee33a5a:::benchmark_data_smithi173_130566_object57461:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e756)", 2024-09-17T16:37:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:08.525217+0000", 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 161.96176200900001, 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.369186334, 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 57462 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.525217+0000", 2024-09-17T16:37:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.525217+0000", 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.525217+0000", 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.06e-07 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.525313+0000", 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5653000000000001e-05 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.525321+0000", 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5159999999999994e-06 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.525326+0000", 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8999999999999997e-06 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.525363+0000", 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6152000000000002e-05 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.525443+0000", 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0377999999999995e-05 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.525523+0000", 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9618999999999998e-05 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.526219+0000", 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00069665799999999996 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.526790+0000", 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057046400000000002 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.894291+0000", 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36750110800000002 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.894346+0000", 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5671000000000003e-05 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.894403+0000", 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6508999999999997e-05 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:57462 8.17 e756/743 ondisk, result = 0)", 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:08.526518+0000", 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 161.96046047199999, 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36789475599999999, 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.526518+0000", 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.526518+0000", 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.526519+0000", 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5000000000000003e-07 2024-09-17T16:37:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.526521+0000", 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.401e-06 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.894215+0000", 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36769376100000001 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.894221+0000", 2024-09-17T16:37:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.268e-06 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.894275+0000", 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3757e-05 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.894283+0000", 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9889999999999992e-06 2024-09-17T16:37:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:08.894413+0000", 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013003 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:78044 8.17 8:e9e46858:::benchmark_data_smithi173_130566_object78043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:26.155763+0000", 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 144.33121511600001, 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36153681999999998, 2024-09-17T16:37:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 78044 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.155763+0000", 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.155763+0000", 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.155764+0000", 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3300000000000001e-07 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.155806+0000", 2024-09-17T16:37:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1662000000000002e-05 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.155814+0000", 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1289999999999996e-06 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.155819+0000", 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5190000000000003e-06 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.155869+0000", 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0578999999999998e-05 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.155992+0000", 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012269099999999999 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,8", 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.156109+0000", 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011696699999999999 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.157444+0000", 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00133562 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.157936+0000", 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049176700000000005 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.517235+0000", 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.35929865300000002 2024-09-17T16:37:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.517273+0000", 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8120000000000001e-05 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.517300+0000", 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7379999999999999e-05 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:94950 8.15 8:aa553242:::benchmark_data_smithi173_130566_object94949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e769)", 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:41.590545+0000", 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 128.89643325500001, 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.470882835, 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 94950 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590545+0000", 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590545+0000", 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590547+0000", 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4330000000000001e-06 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590601+0000", 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4546000000000003e-05 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590614+0000", 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2944e-05 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590622+0000", 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9109999999999993e-06 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590681+0000", 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8641999999999999e-05 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590803+0000", 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000121885 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590902+0000", 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9684000000000003e-05 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.592520+0000", 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001617261 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.599662+0000", 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.007142425 2024-09-17T16:37:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.061357+0000", 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.461694565 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.061398+0000", 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0868999999999998e-05 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.061428+0000", 2024-09-17T16:37:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0670000000000003e-05 2024-09-17T16:37:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.17883.0:94955 8.10 e769/745 8:0a94a0a3:::benchmark_data_smithi173_130566_object94954:head v 769'3338, mlcod=769'3334)", 2024-09-17T16:37:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:41.598365+0000", 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 128.88861315, 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34446574600000002, 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.598365+0000", 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.598365+0000", 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.598366+0000", 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0599999999999999e-07 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.598432+0000", 2024-09-17T16:37:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5698999999999996e-05 2024-09-17T16:37:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.941093+0000", 2024-09-17T16:37:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34266139099999998 2024-09-17T16:37:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.941102+0000", 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3699999999999995e-06 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.941192+0000", 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0453999999999998e-05 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.941215+0000", 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3173999999999999e-05 2024-09-17T16:37:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.942796+0000", 2024-09-17T16:37:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015810500000000001 2024-09-17T16:37:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.942831+0000", 2024-09-17T16:37:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4802000000000003e-05 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "rep_scrubmap(8.d e728 from shard 11)", 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:13.537886+0000", 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 96.949092906000004, 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.66040258699999999, 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537886+0000", 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537886+0000", 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537887+0000", 2024-09-17T16:37:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2410000000000001e-06 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537958+0000", 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0797999999999995e-05 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.197838+0000", 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.65987990699999999 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.197850+0000", 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2785999999999999e-05 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.197989+0000", 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013900800000000001 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.198027+0000", 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7422999999999997e-05 2024-09-17T16:37:50.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.198288+0000", 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026142400000000001 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113330 8.15 8:acf5d31c:::benchmark_data_smithi173_130566_object18355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.084420+0000", 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.402558634000002, 2024-09-17T16:37:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39774510699999999, 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113330 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.084420+0000", 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.084420+0000", 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.084421+0000", 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.145e-06 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.084425+0000", 2024-09-17T16:37:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.54e-06 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.084433+0000", 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7900000000000005e-06 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.084438+0000", 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5519999999999998e-06 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.084476+0000", 2024-09-17T16:37:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7796000000000002e-05 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.084557+0000", 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1155000000000005e-05 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.084660+0000", 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000103483 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.086313+0000", 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016525240000000001 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.086675+0000", 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036162299999999999 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.482095+0000", 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39542043399999999 2024-09-17T16:37:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.482141+0000", 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5701999999999997e-05 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.482165+0000", 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4363e-05 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:113330 8.15 e796/743 ondisk, result = 0)", 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.086876+0000", 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.400102541999999, 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.395292423, 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.086876+0000", 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.086876+0000", 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.086877+0000", 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4600000000000004e-07 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.086881+0000", 2024-09-17T16:37:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9149999999999998e-06 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.481992+0000", 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39511136400000002 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.482015+0000", 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2909999999999999e-05 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.482081+0000", 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5672000000000004e-05 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.482089+0000", 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2570000000000007e-06 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.482168+0000", 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9559000000000002e-05 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.17883.0:113352 8.10 e796/792 8:087be7bc:::benchmark_data_smithi173_130566_object18377:head v 796'3878, mlcod=748'756)", 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.165342+0000", 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.32163671, 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29515114999999997, 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.165342+0000", 2024-09-17T16:37:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.165342+0000", 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.165343+0000", 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6400000000000001e-07 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.165350+0000", 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8630000000000002e-06 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458770+0000", 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.293420607 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458775+0000", 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1070000000000004e-06 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458827+0000", 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1205999999999997e-05 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458850+0000", 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3869000000000001e-05 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.460461+0000", 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016104629999999999 2024-09-17T16:37:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.460493+0000", 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2171000000000001e-05 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113356 8.15 8:abda444a:::benchmark_data_smithi173_130566_object18381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.179326+0000", 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.307652617000002, 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33412438700000002, 2024-09-17T16:37:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113356 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.179326+0000", 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.179326+0000", 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.179327+0000", 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3340000000000001e-06 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.179332+0000", 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0490000000000003e-06 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.179344+0000", 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1513e-05 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.179349+0000", 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3279999999999996e-06 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.179429+0000", 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9575999999999998e-05 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.179509+0000", 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0511000000000002e-05 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.179577+0000", 2024-09-17T16:37:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7960000000000007e-05 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.181215+0000", 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001637498 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.181610+0000", 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00039487499999999999 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.513382+0000", 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33177262200000002 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.513421+0000", 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8507999999999998e-05 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.513450+0000", 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9612999999999999e-05 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:113356 8.15 e796/743 ondisk, result = 0)", 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.181446+0000", 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.305532669000002, 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33200914999999998, 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.181446+0000", 2024-09-17T16:37:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.181446+0000", 2024-09-17T16:37:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.181447+0000", 2024-09-17T16:37:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3099999999999997e-07 2024-09-17T16:37:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.181450+0000", 2024-09-17T16:37:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5319999999999998e-06 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.513291+0000", 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33184054499999999 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.513315+0000", 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4073000000000001e-05 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.513371+0000", 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6118e-05 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.513377+0000", 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.511e-06 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.513455+0000", 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7639999999999995e-05 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.17883.0:113385 8.10 e796/792 8:0d6b7f85:::benchmark_data_smithi173_130566_object18410:head v 796'3879, mlcod=748'756)", 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.458739+0000", 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.028239694, 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36422911200000002, 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458739+0000", 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458739+0000", 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458740+0000", 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1600000000000001e-07 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.458754+0000", 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.394e-05 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.820975+0000", 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36222094999999999 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.820984+0000", 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8919999999999995e-06 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.821044+0000", 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9755e-05 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.821070+0000", 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.55e-05 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.822933+0000", 2024-09-17T16:37:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018637739999999999 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.822968+0000", 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4585000000000002e-05 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:115141 8.11 8:8f69026c:::benchmark_data_smithi173_130566_object20166:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:54.888941+0000", 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 55.598037783000002, 2024-09-17T16:37:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28301732699999999, 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 115141 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.888941+0000", 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.888941+0000", 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.888942+0000", 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4729999999999999e-06 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.888949+0000", 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2809999999999997e-06 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.888962+0000", 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3256e-05 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.888968+0000", 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0719999999999996e-06 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.889026+0000", 2024-09-17T16:37:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7846000000000001e-05 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.889168+0000", 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000142378 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 5,10", 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.889323+0000", 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015441700000000001 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.890893+0000", 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015701770000000001 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.890949+0000", 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6128999999999997e-05 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.170556+0000", 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27960674899999999 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.170715+0000", 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015981900000000001 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.171958+0000", 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00124273 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:116728 8.15 8:a96131de:::benchmark_data_smithi173_130566_object21753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e802)", 2024-09-17T16:37:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:57.515131+0000", 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 52.971848025, 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.345008282, 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 116728 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.515131+0000", 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.515131+0000", 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.515133+0000", 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.12e-06 2024-09-17T16:37:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.515142+0000", 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6169999999999996e-06 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.515170+0000", 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7809000000000001e-05 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.515184+0000", 2024-09-17T16:37:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3463e-05 2024-09-17T16:37:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.515278+0000", 2024-09-17T16:37:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4252000000000005e-05 2024-09-17T16:37:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.515449+0000", 2024-09-17T16:37:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017086299999999999 2024-09-17T16:37:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:37:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.515515+0000", 2024-09-17T16:37:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6307000000000001e-05 2024-09-17T16:37:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.517007+0000", 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001491806 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.517819+0000", 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00081172000000000004 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.859986+0000", 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34216785500000002 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.860087+0000", 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100581 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:57.860139+0000", 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1888999999999997e-05 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:151039 8.d 8:baaadabc:::benchmark_data_smithi173_130566_object56064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e819)", 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:39.967546+0000", 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.519433065999999, 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33117150099999998, 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 151039 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.967546+0000", 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.967546+0000", 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.967547+0000", 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3459999999999999e-06 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.967551+0000", 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.631e-06 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.967564+0000", 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3712e-05 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.967570+0000", 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2020000000000001e-06 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.968423+0000", 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000852434 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.968539+0000", 2024-09-17T16:37:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000115912 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,11", 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.968636+0000", 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6865000000000002e-05 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.972869+0000", 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.004233466 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:39.973428+0000", 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055900500000000001 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.298649+0000", 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.325220762 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.298687+0000", 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8211999999999999e-05 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.298717+0000", 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9954000000000002e-05 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:157575 8.d 8:b9a23873:::benchmark_data_smithi173_130566_object62600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e822)", 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:49.015112+0000", 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.471867045, 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26665923000000002, 2024-09-17T16:37:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 157575 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.015112+0000", 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.015112+0000", 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.015112+0000", 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2999999999999999e-07 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.015116+0000", 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.76e-06 2024-09-17T16:37:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:37:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.015127+0000", 2024-09-17T16:37:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0757999999999999e-05 2024-09-17T16:37:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:37:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.015132+0000", 2024-09-17T16:37:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8949999999999997e-06 2024-09-17T16:37:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.015171+0000", 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8884999999999998e-05 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.015247+0000", 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6781999999999998e-05 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 2,11", 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.015311+0000", 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3610999999999998e-05 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.019017+0000", 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0037063080000000002 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.019855+0000", 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00083795500000000004 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.281637+0000", 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26178196599999998 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.281708+0000", 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0224999999999999e-05 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.281771+0000", 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3255000000000003e-05 2024-09-17T16:37:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:37:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:37:50.620 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:37:50.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:50.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:50.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:50 smithi173 ceph-mon[30530]: osdmap e823: 12 total, 12 up, 12 in 2024-09-17T16:37:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:50 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 15847/105375 objects degraded (15.039%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:37:50.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:50 smithi017 ceph-mon[27571]: osdmap e823: 12 total, 12 up, 12 in 2024-09-17T16:37:50.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:50 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 15847/105375 objects degraded (15.039%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-17T16:37:50.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:50.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:50.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:50.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:50.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:50.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:50.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:51.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:51.101 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:51.124 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:51.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:51.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:51.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:51.253 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:51.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:51.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:37:51.355 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:51.383 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:51.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:51 smithi173 ceph-mon[30530]: 8.5 scrub starts 2024-09-17T16:37:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:51 smithi173 ceph-mon[30530]: pgmap v1833: 27 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s; 15845/99051 objects degraded (15.997%); 2.6 MiB/s, 246 objects/s recovering 2024-09-17T16:37:51.484 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:51.513 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:51.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:51.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:51 smithi071 ceph-mon[30002]: 8.5 scrub starts 2024-09-17T16:37:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:51 smithi071 ceph-mon[30002]: pgmap v1833: 27 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s; 15845/99051 objects degraded (15.997%); 2.6 MiB/s, 246 objects/s recovering 2024-09-17T16:37:51.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:51.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:51.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:51.695 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:51.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:51 smithi017 ceph-mon[27571]: 8.5 scrub starts 2024-09-17T16:37:51.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:51 smithi017 ceph-mon[27571]: pgmap v1833: 27 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s; 15845/99051 objects degraded (15.997%); 2.6 MiB/s, 246 objects/s recovering 2024-09-17T16:37:51.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:51.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:51.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:51.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:51.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:51.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:52.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:52.034 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:52.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:52.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:52.162 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:52.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:52.264 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:52.275 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:37:52.275 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:37:52.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.275 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:160194 8.5 8:a035638f:::benchmark_data_smithi173_130566_object65219:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-17T16:37:52.275 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:52.259356+0000", 2024-09-17T16:37:52.275 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.011727972, 2024-09-17T16:37:52.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011765693000000001, 2024-09-17T16:37:52.275 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 160194 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.259356+0000", 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:52.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.259356+0000", 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.259365+0000", 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4549999999999995e-06 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.259372+0000", 2024-09-17T16:37:52.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1999999999999997e-06 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.259389+0000", 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7197000000000002e-05 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.259398+0000", 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4979999999999995e-06 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:52.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.259460+0000", 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1295000000000002e-05 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.259555+0000", 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4832000000000003e-05 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.259652+0000", 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7823000000000002e-05 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.261861+0000", 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022085759999999999 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.265281+0000", 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034201140000000001 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:160200 8.7 e824/811 8:e4dc3b40:::benchmark_data_smithi173_130566_object65225:head v 824'5105, mlcod=824'5103)", 2024-09-17T16:37:52.280 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:52.266709+0000", 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0043753639999999996, 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044862629999999999, 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.266709+0000", 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:52.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.266709+0000", 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.266710+0000", 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9100000000000002e-07 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.266716+0000", 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2779999999999999e-06 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.266733+0000", 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6843000000000001e-05 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.266740+0000", 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4080000000000001e-06 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.266819+0000", 2024-09-17T16:37:52.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9087999999999999e-05 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.266841+0000", 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1559000000000001e-05 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:160204 8.6 8:6722eeec:::benchmark_data_smithi173_130566_object65229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-17T16:37:52.284 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:52.268292+0000", 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0027918639999999998, 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002950252, 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 160204 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.285 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.268292+0000", 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.268292+0000", 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:52.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.268294+0000", 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.438e-06 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.268297+0000", 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3759999999999999e-06 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.268312+0000", 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5213e-05 2024-09-17T16:37:52.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.268320+0000", 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5700000000000004e-06 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.270052+0000", 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017322209999999999 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.270171+0000", 2024-09-17T16:37:52.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118653 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.270272+0000", 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010137 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:160205 8.4 e824/810 8:26c49180:::benchmark_data_smithi173_130566_object65230:head v 824'5096, mlcod=824'5094)", 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:52.269365+0000", 2024-09-17T16:37:52.289 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.001719213, 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019351119999999999, 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.269365+0000", 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.269365+0000", 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.269366+0000", 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2699999999999998e-07 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.269373+0000", 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.294e-06 2024-09-17T16:37:52.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.269404+0000", 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1359999999999998e-05 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.269411+0000", 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8560000000000001e-06 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:37:52.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.269459+0000", 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7571000000000002e-05 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:52.269477+0000", 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8624999999999999e-05 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 4 2024-09-17T16:37:52.293 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:37:52.297 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:52.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:52 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:37:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:52 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:37:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:52 smithi173 ceph-mon[30530]: osdmap e824: 12 total, 12 up, 12 in 2024-09-17T16:37:52.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:52.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:52.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:52.532 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:52.553 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:52.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:52.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:52 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:37:52.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:52 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:37:52.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:52 smithi071 ceph-mon[30002]: osdmap e824: 12 total, 12 up, 12 in 2024-09-17T16:37:52.656 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:52.677 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:52.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:52.686 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:52 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:37:52.686 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:52 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:37:52.686 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:52 smithi017 ceph-mon[27571]: osdmap e824: 12 total, 12 up, 12 in 2024-09-17T16:37:52.779 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:52.803 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:52.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:52.811 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:37:52.812 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:37:52.827 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:37:52.827 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:37:52.851 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:52.851 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:37:52.851 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:37:52.873 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:37:52.874 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:37:52.897 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:37:52.898 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:37:52.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:52.926 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:37:52.926 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:37:52.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:52.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:52.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:52.952 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:37:52.952 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:37:52.975 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:37:52.976 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:37:53.000 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:37:53.001 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:37:53.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:37:53.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:53.032 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:37:53.032 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:37:53.060 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:37:53.060 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:37:53.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:53.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:53.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:53.083 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:37:53.083 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:37:53.172 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:53.194 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:53.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:53.268 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:53.295 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:53.324 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:53.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:53.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:53.426 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:53.453 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:53.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:53.539 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:53 smithi173 ceph-mon[30530]: pgmap v1835: 27 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 1.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 14363/95730 objects degraded (15.004%); 737/95730 objects misplaced (0.770%); 4.3 MiB/s, 369 objects/s recovering 2024-09-17T16:37:53.539 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:53 smithi173 ceph-mon[30530]: osdmap e825: 12 total, 12 up, 12 in 2024-09-17T16:37:53.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3827964989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:53.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:53 smithi071 ceph-mon[30002]: pgmap v1835: 27 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 1.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 14363/95730 objects degraded (15.004%); 737/95730 objects misplaced (0.770%); 4.3 MiB/s, 369 objects/s recovering 2024-09-17T16:37:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:53 smithi071 ceph-mon[30002]: osdmap e825: 12 total, 12 up, 12 in 2024-09-17T16:37:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3827964989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:53.578 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:53.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:53.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:53.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:53.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:53.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:53.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:53 smithi017 ceph-mon[27571]: pgmap v1835: 27 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 19 active+clean; 1.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 725 op/s; 14363/95730 objects degraded (15.004%); 737/95730 objects misplaced (0.770%); 4.3 MiB/s, 369 objects/s recovering 2024-09-17T16:37:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:53 smithi017 ceph-mon[27571]: osdmap e825: 12 total, 12 up, 12 in 2024-09-17T16:37:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3827964989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:53.820 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:53.827 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:37:53.827 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:37:53.844 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:53.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:53.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:53.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:53.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:54.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:54.098 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:54.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:54.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:54.206 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:54.221 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:54.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:54.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:54.304 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.8 to 1 (10000) 2024-09-17T16:37:54.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:54.346 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:54.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:54.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:54.475 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:54.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:54.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:54 smithi173 ceph-mon[30530]: osdmap e826: 12 total, 12 up, 12 in 2024-09-17T16:37:54.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3745512218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:37:54.513 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:54 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:37:54.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:37:54.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:54 smithi071 ceph-mon[30002]: osdmap e826: 12 total, 12 up, 12 in 2024-09-17T16:37:54.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3745512218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:37:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:54 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:37:54.577 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:54.601 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:54.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:54.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:54.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:54.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:54.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:54.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:54 smithi017 ceph-mon[27571]: osdmap e826: 12 total, 12 up, 12 in 2024-09-17T16:37:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3745512218' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:37:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:54 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:37:54.829 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:54.853 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:54.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:54.891 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:37:54.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:54.980 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:54.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:55.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:55.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:55.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:55.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:55.212 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:55.235 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:55.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:55.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:55.350 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-17T16:37:55.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:55.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:55.476 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:37:55.476 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:37:55.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:55.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:55.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:55.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:55 smithi071 ceph-mon[30002]: pgmap v1838: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 13713/92241 objects degraded (14.866%); 737/92241 objects misplaced (0.799%); 5.3 MiB/s, 298 objects/s recovering 2024-09-17T16:37:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:55 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:37:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:55 smithi071 ceph-mon[30002]: osdmap e827: 12 total, 12 up, 12 in 2024-09-17T16:37:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:55 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 13713/92241 objects degraded (14.866%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:37:55.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:55.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:55 smithi017 ceph-mon[27571]: pgmap v1838: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 13713/92241 objects degraded (14.866%); 737/92241 objects misplaced (0.799%); 5.3 MiB/s, 298 objects/s recovering 2024-09-17T16:37:55.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:55 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:37:55.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:55 smithi017 ceph-mon[27571]: osdmap e827: 12 total, 12 up, 12 in 2024-09-17T16:37:55.638 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:55 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 13713/92241 objects degraded (14.866%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:37:55.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:55.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:55.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:55 smithi173 ceph-mon[30530]: pgmap v1838: 27 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 760 op/s; 13713/92241 objects degraded (14.866%); 737/92241 objects misplaced (0.799%); 5.3 MiB/s, 298 objects/s recovering 2024-09-17T16:37:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:55 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:37:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:55 smithi173 ceph-mon[30530]: osdmap e827: 12 total, 12 up, 12 in 2024-09-17T16:37:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:55 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 13713/92241 objects degraded (14.866%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:37:55.743 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:55.777 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:55.782 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:55.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:55.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:55.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:55.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:56.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:56.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:37:56.053 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:56.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:56.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:37:56.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:56.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:56.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:56.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:56.315 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:37:56.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:56.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:56.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3575249171' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:37:56.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:56 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:37:56.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:56.443 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:56.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:56.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:56.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3575249171' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:37:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:56 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:37:56.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:56.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:56.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:56.714 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:56.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:56.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:56.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3575249171' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:37:56.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:56 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:37:56.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:56.846 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:56.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:56.940 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:37:56.940 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:37:56.948 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:56.975 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:56.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:57.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:57.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:57.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:57.209 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:37:57.234 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:57.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:37:57.312 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.9 to 1 (10000) 2024-09-17T16:37:57.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:57.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:57.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:57.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:57 smithi017 ceph-mon[27571]: pgmap v1840: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 618 op/s; 12383/87804 objects degraded (14.103%); 1527/87804 objects misplaced (1.739%); 3.1 MiB/s, 184 objects/s recovering 2024-09-17T16:37:57.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:57 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:37:57.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:57 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:37:57.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:57 smithi017 ceph-mon[27571]: osdmap e828: 12 total, 12 up, 12 in 2024-09-17T16:37:57.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2980553367' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:37:57.462 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:57.486 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:57.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:57 smithi071 ceph-mon[30002]: pgmap v1840: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 618 op/s; 12383/87804 objects degraded (14.103%); 1527/87804 objects misplaced (1.739%); 3.1 MiB/s, 184 objects/s recovering 2024-09-17T16:37:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:57 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:37:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:57 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:37:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:57 smithi071 ceph-mon[30002]: osdmap e828: 12 total, 12 up, 12 in 2024-09-17T16:37:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2980553367' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:37:57.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:37:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:57.611 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:57.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:37:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:57 smithi173 ceph-mon[30530]: pgmap v1840: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 618 op/s; 12383/87804 objects degraded (14.103%); 1527/87804 objects misplaced (1.739%); 3.1 MiB/s, 184 objects/s recovering 2024-09-17T16:37:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:57 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:37:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:57 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:37:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:57 smithi173 ceph-mon[30530]: osdmap e828: 12 total, 12 up, 12 in 2024-09-17T16:37:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2980553367' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:37:57.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:37:57.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:57.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:57.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:57.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:57.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:57.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:57.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:57.937 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:57.958 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:57.981 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:57.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:58.083 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:58.105 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:58.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:58.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:58.230 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:58.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:58.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:58.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:37:58.357 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:58.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:37:58.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:37:58.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:58.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:37:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2980553367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:37:58.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:58 smithi017 ceph-mon[27571]: osdmap e829: 12 total, 12 up, 12 in 2024-09-17T16:37:58.513 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:37:58.513 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:37:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2980553367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:37:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:58 smithi071 ceph-mon[30002]: osdmap e829: 12 total, 12 up, 12 in 2024-09-17T16:37:58.583 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:37:58.605 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:58.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:37:58.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2980553367' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:37:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:58 smithi173 ceph-mon[30530]: osdmap e829: 12 total, 12 up, 12 in 2024-09-17T16:37:58.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:37:58.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:58.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:37:58.830 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:58.854 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:58.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:58.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:58.978 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:58.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:58.982 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:37:58.982 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:37:59.003 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:37:59.003 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:37:59.024 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:37:59.024 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:37:59.045 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:37:59.045 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:37:59.066 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:37:59.066 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:37:59.079 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:37:59.087 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:37:59.088 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:37:59.101 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:59.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:59.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:37:59.110 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:37:59.110 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:37:59.132 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:37:59.132 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:37:59.153 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:37:59.153 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:37:59.171 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:37:59.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:37:59.177 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:37:59.199 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:37:59.199 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:37:59.202 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:59.221 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:37:59.221 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:37:59.237 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:59.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:59.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:37:59.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:37:59.364 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:37:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:59.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:37:59.440 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:37:59.466 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:37:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:59 smithi017 ceph-mon[27571]: pgmap v1843: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 646 op/s; 11900/82875 objects degraded (14.359%); 1237/82875 objects misplaced (1.493%); 6.9 MiB/s, 210 objects/s recovering 2024-09-17T16:37:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:59 smithi017 ceph-mon[27571]: 8.18 scrub starts 2024-09-17T16:37:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:59 smithi017 ceph-mon[27571]: osdmap e830: 12 total, 12 up, 12 in 2024-09-17T16:37:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:37:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1627891322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:59.491 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:59.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:37:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:59 smithi071 ceph-mon[30002]: pgmap v1843: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 646 op/s; 11900/82875 objects degraded (14.359%); 1237/82875 objects misplaced (1.493%); 6.9 MiB/s, 210 objects/s recovering 2024-09-17T16:37:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:59 smithi071 ceph-mon[30002]: 8.18 scrub starts 2024-09-17T16:37:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:59 smithi071 ceph-mon[30002]: osdmap e830: 12 total, 12 up, 12 in 2024-09-17T16:37:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:37:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1627891322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:59.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:37:59.599 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-17T16:37:59.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:59.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:37:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:59 smithi173 ceph-mon[30530]: pgmap v1843: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 20 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 646 op/s; 11900/82875 objects degraded (14.359%); 1237/82875 objects misplaced (1.493%); 6.9 MiB/s, 210 objects/s recovering 2024-09-17T16:37:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:59 smithi173 ceph-mon[30530]: 8.18 scrub starts 2024-09-17T16:37:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:59 smithi173 ceph-mon[30530]: osdmap e830: 12 total, 12 up, 12 in 2024-09-17T16:37:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:37:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1627891322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:37:59.727 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:37:59.765 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:37:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:59.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:37:59.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:37:59.884 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:37:59.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:37:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:37:59.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:37:59.994 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:00.021 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:00.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:00.081 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.17883.0:115204 8.4 e800/797 8:225dd052:::benchmark_data_smithi173_130566_object20229:head v 800'3754, mlcod=752'1253)", 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:55.198694+0000", 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 64.878434858000006, 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18890970600000001, 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.082 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.198694+0000", 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.198694+0000", 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.198695+0000", 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.198700+0000", 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.682e-06 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.386522+0000", 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18782133600000001 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.386527+0000", 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8550000000000001e-06 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.386601+0000", 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4503000000000001e-05 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.386618+0000", 2024-09-17T16:38:00.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7102999999999999e-05 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.387574+0000", 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00095553399999999996 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.387604+0000", 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9984e-05 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.17883.0:133571 8.16 e808/799 8:6cb614d6:::benchmark_data_smithi173_130566_object38596:head v 808'4358, mlcod=808'4354)", 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:17.696732+0000", 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 42.380396976, 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26608319899999999, 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.696732+0000", 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.696732+0000", 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.696734+0000", 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.917e-06 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.696767+0000", 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2948999999999999e-05 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.960933+0000", 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26416631099999999 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.960939+0000", 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3810000000000004e-06 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.960994+0000", 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4166999999999998e-05 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.961015+0000", 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1404e-05 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.962772+0000", 2024-09-17T16:38:00.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017565860000000001 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:17.962815+0000", 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3483999999999999e-05 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:134912 8.19 8:98a79c50:::benchmark_data_smithi173_130566_object39937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:19.226624+0000", 2024-09-17T16:38:00.091 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 40.850505372999997, 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24593282499999999, 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 134912 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.226624+0000", 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.226624+0000", 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.226625+0000", 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.173e-06 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.226629+0000", 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1910000000000002e-06 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.226640+0000", 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1075e-05 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.226647+0000", 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2479999999999999e-06 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.226695+0000", 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7809999999999998e-05 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.226699+0000", 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7859999999999998e-06 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.226999+0000", 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030004100000000002 2024-09-17T16:38:00.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.227088+0000", 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9036000000000006e-05 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,8", 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.227165+0000", 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6668000000000005e-05 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:38:00.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.228880+0000", 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017155110000000001 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.230445+0000", 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001565229 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.472494+0000", 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24204903 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.472526+0000", 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1917e-05 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.472556+0000", 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0110000000000001e-05 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:134912 8.19 e810/800 ondisk, result = 0)", 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:19.229577+0000", 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 40.847552291, 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.242983639, 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.229577+0000", 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.229577+0000", 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.229577+0000", 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.44e-07 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.229582+0000", 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9130000000000002e-06 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.472439+0000", 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.242856877 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.472445+0000", 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8599999999999998e-06 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.472486+0000", 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0661000000000003e-05 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.472491+0000", 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5230000000000003e-06 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:19.472560+0000", 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9160999999999997e-05 2024-09-17T16:38:00.102 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:140508 8.4 8:27a5495c:::benchmark_data_smithi173_130566_object45533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:27.267144+0000", 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 32.809984608999997, 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21021815599999999, 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:38:00.103 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 140508 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.267144+0000", 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.267144+0000", 2024-09-17T16:38:00.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.267145+0000", 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1459999999999999e-06 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.267149+0000", 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8279999999999996e-06 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.267158+0000", 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8070000000000006e-06 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.267162+0000", 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2379999999999999e-06 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.267217+0000", 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4805000000000001e-05 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.267335+0000", 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000117503 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.267444+0000", 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010907499999999999 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.268619+0000", 2024-09-17T16:38:00.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011749110000000001 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.269926+0000", 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001307503 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.477265+0000", 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.207339259 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.477321+0000", 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5893999999999997e-05 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:27.477362+0000", 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1186999999999997e-05 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:151256 8.19 8:981619b0:::benchmark_data_smithi173_130566_object56281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e819)", 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:40.504714+0000", 2024-09-17T16:38:00.109 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 19.572415268, 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19110540200000001, 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 151256 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.504714+0000", 2024-09-17T16:38:00.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.504714+0000", 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.504715+0000", 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.15e-06 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.504719+0000", 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7010000000000004e-06 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.504731+0000", 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1711e-05 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.504738+0000", 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4790000000000001e-06 2024-09-17T16:38:00.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.504796+0000", 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8811999999999999e-05 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.504883+0000", 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6339000000000001e-05 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,8", 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.504956+0000", 2024-09-17T16:38:00.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3052999999999993e-05 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.506808+0000", 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018522949999999999 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.507200+0000", 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00039159900000000002 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.695760+0000", 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18856029399999999 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.695793+0000", 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3030000000000001e-05 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.695819+0000", 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5939e-05 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.116 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:151436 8.19 8:9abfaeab:::benchmark_data_smithi173_130566_object56461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:40.855849+0000", 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 19.221279713000001, 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23466412, 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:38:00.117 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 151436 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.855849+0000", 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.855849+0000", 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.855851+0000", 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4929999999999999e-06 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.855857+0000", 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2879999999999998e-06 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.855872+0000", 2024-09-17T16:38:00.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5071e-05 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.855879+0000", 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5710000000000003e-06 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.855932+0000", 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2896000000000003e-05 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.856011+0000", 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9736999999999999e-05 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,8", 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.856072+0000", 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0377999999999997e-05 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.858032+0000", 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019601029999999999 2024-09-17T16:38:00.121 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.858596+0000", 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00056461 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:41.090442+0000", 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23184555100000001 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:41.090474+0000", 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2533000000000002e-05 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:41.090513+0000", 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8889e-05 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.123 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:151436 8.19 e820/800 ondisk, result = 0)", 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:40.859126+0000", 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 19.218002543000001, 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23139154200000001, 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.859126+0000", 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.859126+0000", 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.859127+0000", 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5400000000000001e-07 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:40.859133+0000", 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5770000000000001e-06 2024-09-17T16:38:00.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:41.090397+0000", 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23126424400000001 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:41.090402+0000", 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7029999999999997e-06 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:41.090433+0000", 2024-09-17T16:38:00.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1238000000000001e-05 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:41.090439+0000", 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0290000000000002e-06 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:41.090518+0000", 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9097000000000005e-05 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:157364 8.4 8:26599a13:::benchmark_data_smithi173_130566_object62389:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e822)", 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:48.384508+0000", 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.692620931, 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26893215100000001, 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 157364 2024-09-17T16:38:00.128 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.384508+0000", 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.384508+0000", 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.384510+0000", 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.798e-06 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.384516+0000", 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2439999999999998e-06 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.384531+0000", 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4639e-05 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.384537+0000", 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3090000000000001e-06 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.384600+0000", 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2972000000000006e-05 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.384722+0000", 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000122538 2024-09-17T16:38:00.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.384839+0000", 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000116395 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.386290+0000", 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001451035 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.389889+0000", 2024-09-17T16:38:00.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0035994960000000002 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.653348+0000", 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26345834200000001 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.653388+0000", 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0543999999999997e-05 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.653440+0000", 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1839000000000003e-05 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:157364 8.4 e822/810 ondisk, result = 0)", 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:48.387147+0000", 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.689981822, 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.266297699, 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.134 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.387147+0000", 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.387147+0000", 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.387148+0000", 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1099999999999995e-07 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.387151+0000", 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.202e-06 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.653276+0000", 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26612553700000002 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.653284+0000", 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0479999999999999e-06 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.653336+0000", 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1895999999999999e-05 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.653344+0000", 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4139999999999997e-06 2024-09-17T16:38:00.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.653445+0000", 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100991 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:157387 8.4 8:259bd7d1:::benchmark_data_smithi173_130566_object62412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e822)", 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:48.418657+0000", 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.658472045, 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28506685900000001, 2024-09-17T16:38:00.138 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 157387 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.418657+0000", 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.139 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.418657+0000", 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.418659+0000", 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6670000000000001e-06 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.418665+0000", 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8970000000000002e-06 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.140 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.418682+0000", 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6421000000000001e-05 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.418689+0000", 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6140000000000003e-06 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.418760+0000", 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0523000000000002e-05 2024-09-17T16:38:00.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.418868+0000", 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000108408 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.418956+0000", 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7328000000000001e-05 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.420340+0000", 2024-09-17T16:38:00.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001384188 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.421798+0000", 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001458144 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.703643+0000", 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28184457000000002 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.143 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.703676+0000", 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3414999999999998e-05 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.703724+0000", 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7683999999999999e-05 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.144 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:157387 8.4 e822/810 ondisk, result = 0)", 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:48.420644+0000", 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.656484497999999, 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28308508599999999, 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.420644+0000", 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.145 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.146 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.146 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.420644+0000", 2024-09-17T16:38:00.146 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.146 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.146 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.146 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.146 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.420645+0000", 2024-09-17T16:38:00.146 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9399999999999993e-07 2024-09-17T16:38:00.146 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.420649+0000", 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9020000000000001e-06 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.703581+0000", 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28293123999999997 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.703587+0000", 2024-09-17T16:38:00.147 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.579e-06 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.703629+0000", 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2209000000000002e-05 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.703637+0000", 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9319999999999996e-06 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.148 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.703729+0000", 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2230000000000003e-05 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:157496 8.4 8:2002af7a:::benchmark_data_smithi173_130566_object62521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e822)", 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:48.711848+0000", 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.365281163000001, 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52764325700000003, 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.149 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 157496 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711848+0000", 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.150 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711848+0000", 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711849+0000", 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1149999999999999e-06 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711853+0000", 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1370000000000004e-06 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711864+0000", 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0913e-05 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711870+0000", 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6130000000000003e-06 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711918+0000", 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8829999999999998e-05 2024-09-17T16:38:00.152 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.712024+0000", 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010582999999999999 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.712119+0000", 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4654000000000005e-05 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.713620+0000", 2024-09-17T16:38:00.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00150146 2024-09-17T16:38:00.154 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.156 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.156 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.156 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714929+0000", 2024-09-17T16:38:00.156 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013090560000000001 2024-09-17T16:38:00.156 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.156 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239387+0000", 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52445777400000004 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239445+0000", 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7960000000000001e-05 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239491+0000", 2024-09-17T16:38:00.157 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5914999999999997e-05 2024-09-17T16:38:00.158 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.158 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.158 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.158 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.158 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.158 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:157496 8.4 e822/810 ondisk, result = 0)", 2024-09-17T16:38:00.158 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:48.714185+0000", 2024-09-17T16:38:00.158 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.362943935000001, 2024-09-17T16:38:00.158 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52531144100000005, 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714185+0000", 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714185+0000", 2024-09-17T16:38:00.159 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714186+0000", 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8699999999999995e-07 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714192+0000", 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0730000000000001e-06 2024-09-17T16:38:00.160 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239260+0000", 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.525067909 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239272+0000", 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2527000000000001e-05 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.161 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239366+0000", 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4143999999999999e-05 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239376+0000", 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7620000000000007e-06 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239496+0000", 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120439 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.162 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:158751 8.19 8:9e60e316:::benchmark_data_smithi173_130566_object63776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e823)", 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:50.479676+0000", 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.597452637, 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22534826699999999, 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:38:00.163 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 158751 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.479676+0000", 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.164 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.479676+0000", 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.479678+0000", 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.734e-06 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.479684+0000", 2024-09-17T16:38:00.165 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1190000000000002e-06 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.479701+0000", 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7311000000000001e-05 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.479710+0000", 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1440000000000003e-06 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.166 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.479781+0000", 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0963999999999993e-05 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.479894+0000", 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000113538 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,8", 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.480005+0000", 2024-09-17T16:38:00.167 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000110839 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.481623+0000", 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016184890000000001 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.483691+0000", 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020674109999999999 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.168 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704964+0000", 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22127358799999999 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704997+0000", 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3062999999999998e-05 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.169 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.705025+0000", 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7067e-05 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:158753 8.19 8:9f2fc863:::benchmark_data_smithi173_130566_object63778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e823)", 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:50.482115+0000", 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.5950143719999996, 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28435587099999998, 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.170 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 158753 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482115+0000", 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.171 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482115+0000", 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482115+0000", 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7000000000000003e-07 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.172 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482120+0000", 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1439999999999996e-06 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482130+0000", 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0539e-05 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482136+0000", 2024-09-17T16:38:00.173 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8880000000000002e-06 2024-09-17T16:38:00.174 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.174 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482181+0000", 2024-09-17T16:38:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5327000000000002e-05 2024-09-17T16:38:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.175 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482283+0000", 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010197200000000001 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,8", 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482386+0000", 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000102304 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.483759+0000", 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013734330000000001 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.176 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.484937+0000", 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011780790000000001 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.766415+0000", 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.281477902 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.766446+0000", 2024-09-17T16:38:00.177 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1287e-05 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.766470+0000", 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4026e-05 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:158751 8.19 e823/800 ondisk, result = 0)", 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:50.482519+0000", 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.5946096379999997, 2024-09-17T16:38:00.178 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.222508083, 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482519+0000", 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482519+0000", 2024-09-17T16:38:00.179 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482520+0000", 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7199999999999999e-07 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.482523+0000", 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3110000000000001e-06 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.180 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704920+0000", 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22239682 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704923+0000", 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4359999999999998e-06 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704958+0000", 2024-09-17T16:38:00.181 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4804000000000001e-05 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704962+0000", 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5090000000000002e-06 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.705027+0000", 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5730999999999998e-05 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.182 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:158753 8.19 e823/800 ondisk, result = 0)", 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:50.484488+0000", 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.5926408989999992, 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28198638199999998, 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.183 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.484488+0000", 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.484488+0000", 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.484489+0000", 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.68e-07 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.184 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.484495+0000", 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7549999999999997e-06 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.766362+0000", 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.281867015 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.766367+0000", 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5279999999999997e-06 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.185 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.766406+0000", 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9051000000000003e-05 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.766411+0000", 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5349999999999998e-06 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.766474+0000", 2024-09-17T16:38:00.186 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3830000000000004e-05 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:158899 8.19 8:9a4620fa:::benchmark_data_smithi173_130566_object63924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e823)", 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:50.702336+0000", 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.3747924810000001, 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.23140433899999999, 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:38:00.187 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 158899 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.702336+0000", 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.702336+0000", 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.188 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.702338+0000", 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2389999999999999e-06 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.702343+0000", 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0039999999999999e-06 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.702355+0000", 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2594e-05 2024-09-17T16:38:00.189 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.702361+0000", 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1299999999999998e-06 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.702416+0000", 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4769999999999999e-05 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.190 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.702500+0000", 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3864999999999997e-05 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,8", 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.702565+0000", 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4719000000000002e-05 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704143+0000", 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015783360000000001 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.191 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704829+0000", 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00068582699999999996 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.933671+0000", 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22884233500000001 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.933703+0000", 2024-09-17T16:38:00.192 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1844999999999997e-05 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.933741+0000", 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7675000000000001e-05 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:158899 8.19 e823/800 ondisk, result = 0)", 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:50.704905+0000", 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.372223473, 2024-09-17T16:38:00.193 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22884399499999999, 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704905+0000", 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704905+0000", 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:38:00.194 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704906+0000", 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5000000000000003e-07 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.704909+0000", 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8899999999999999e-06 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:38:00.195 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.933633+0000", 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.22872442900000001 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.933637+0000", 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.236e-06 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.933665+0000", 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8912999999999999e-05 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.196 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.933669+0000", 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.236e-06 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:50.933749+0000", 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0741000000000005e-05 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:38:00.197 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:38:00.218 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:00.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:00.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:00.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:00.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:00.447 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:00.470 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:00.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:00.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:00 smithi071 ceph-mon[30002]: 8.a starting backfill to osd.9 from (0'0,0'0] MAX to 828'10506 2024-09-17T16:38:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:00 smithi071 ceph-mon[30002]: osdmap e831: 12 total, 12 up, 12 in 2024-09-17T16:38:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:00 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11900/82875 objects degraded (14.359%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:00.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:00.603 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:00.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:00.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:00 smithi017 ceph-mon[27571]: 8.a starting backfill to osd.9 from (0'0,0'0] MAX to 828'10506 2024-09-17T16:38:00.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:00 smithi017 ceph-mon[27571]: osdmap e831: 12 total, 12 up, 12 in 2024-09-17T16:38:00.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:00 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11900/82875 objects degraded (14.359%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:00 smithi173 ceph-mon[30530]: 8.a starting backfill to osd.9 from (0'0,0'0] MAX to 828'10506 2024-09-17T16:38:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:00 smithi173 ceph-mon[30530]: osdmap e831: 12 total, 12 up, 12 in 2024-09-17T16:38:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:00 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11900/82875 objects degraded (14.359%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:00.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:00.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:00.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:00.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:00.854 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:00.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:00.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:00.998 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:01.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:01.100 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:01.136 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:01.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:01.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:01.264 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:01.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:01.366 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:01.391 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:01.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:01.493 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:01.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:01.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:01.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:01 smithi071 ceph-mon[30002]: pgmap v1846: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 11900/78309 objects degraded (15.196%); 2354/78309 objects misplaced (3.006%); 13 MiB/s, 323 objects/s recovering 2024-09-17T16:38:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:01 smithi071 ceph-mon[30002]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:38:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:01 smithi071 ceph-mon[30002]: osdmap e832: 12 total, 12 up, 12 in 2024-09-17T16:38:01.616 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:38:01.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:01.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:01 smithi173 ceph-mon[30530]: pgmap v1846: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 11900/78309 objects degraded (15.196%); 2354/78309 objects misplaced (3.006%); 13 MiB/s, 323 objects/s recovering 2024-09-17T16:38:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:01 smithi173 ceph-mon[30530]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:38:01.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:01 smithi173 ceph-mon[30530]: osdmap e832: 12 total, 12 up, 12 in 2024-09-17T16:38:01.650 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:01 smithi017 ceph-mon[27571]: pgmap v1846: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 791 op/s; 11900/78309 objects degraded (15.196%); 2354/78309 objects misplaced (3.006%); 13 MiB/s, 323 objects/s recovering 2024-09-17T16:38:01.650 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:01 smithi017 ceph-mon[27571]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:38:01.650 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:01 smithi017 ceph-mon[27571]: osdmap e832: 12 total, 12 up, 12 in 2024-09-17T16:38:01.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:01.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:01.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:01.784 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:01.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:01.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:01.933 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:01.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:02.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:02.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:02.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:02.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:02.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:02.188 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:02.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:02.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:02.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:02.320 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:02.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:02.365 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:02.422 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:02.441 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:02.472 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:02.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:02.507 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:02 smithi017 ceph-mon[27571]: 8.9 scrub starts 2024-09-17T16:38:02.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:02.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:02.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:02.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:02 smithi173 ceph-mon[30530]: 8.9 scrub starts 2024-09-17T16:38:02.702 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:02.725 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:02.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:02.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:02 smithi071 ceph-mon[30002]: 8.9 scrub starts 2024-09-17T16:38:02.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:02.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:02.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:02.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:02.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:02.987 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:02.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:02.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:03.089 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:03.114 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:03.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:03.216 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:03.245 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:03.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:03.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:03.341 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:38:03.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:03.368 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:03.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:03.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:03.493 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:03.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:03.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:03.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:03.621 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:03.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:03 smithi173 ceph-mon[30530]: pgmap v1848: 27 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 10754/76722 objects degraded (14.017%); 2655/76722 objects misplaced (3.461%); 14 MiB/s, 217 objects/s recovering 2024-09-17T16:38:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:03 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:03 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2716500002' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:38:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:03 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:38:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:03 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:03 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:38:03.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:03.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:38:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:03 smithi017 ceph-mon[27571]: pgmap v1848: 27 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 10754/76722 objects degraded (14.017%); 2655/76722 objects misplaced (3.461%); 14 MiB/s, 217 objects/s recovering 2024-09-17T16:38:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:03 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:03 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2716500002' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:38:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:03 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:38:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:03 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:03 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:38:03.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:03.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:03.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:03 smithi071 ceph-mon[30002]: pgmap v1848: 27 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 10754/76722 objects degraded (14.017%); 2655/76722 objects misplaced (3.461%); 14 MiB/s, 217 objects/s recovering 2024-09-17T16:38:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:03 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:03 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2716500002' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:38:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:03 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:38:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:03 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:03.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:03 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:38:03.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:03.876 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:03.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:03.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:03.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:04.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:04.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:04.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:04.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:04.147 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:04.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:04.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:04.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:04.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:04.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:04.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:04.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:04.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:04.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:04.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:04.626 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:04.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:04 smithi173 ceph-mon[30530]: osdmap e833: 12 total, 12 up, 12 in 2024-09-17T16:38:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:04.654 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:04.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:04 smithi017 ceph-mon[27571]: osdmap e833: 12 total, 12 up, 12 in 2024-09-17T16:38:04.755 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:04.781 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:04.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:04.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:04 smithi071 ceph-mon[30002]: osdmap e833: 12 total, 12 up, 12 in 2024-09-17T16:38:04.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:04.913 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:04.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:04.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:05.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:05.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:05.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:05.147 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:05.171 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:05.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:05.273 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:05.304 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:05.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:05.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:05.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:05.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:05.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:05.536 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:05.560 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:05.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:05 smithi173 ceph-mon[30530]: pgmap v1850: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 422 op/s; 10323/73908 objects degraded (13.967%); 3224/73908 objects misplaced (4.362%); 14 MiB/s, 297 objects/s recovering 2024-09-17T16:38:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:05 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 10323/73908 objects degraded (13.967%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:05.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:05.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:05.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:05 smithi017 ceph-mon[27571]: pgmap v1850: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 422 op/s; 10323/73908 objects degraded (13.967%); 3224/73908 objects misplaced (4.362%); 14 MiB/s, 297 objects/s recovering 2024-09-17T16:38:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:05 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 10323/73908 objects degraded (13.967%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:05.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:05.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:05 smithi071 ceph-mon[30002]: pgmap v1850: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 422 op/s; 10323/73908 objects degraded (13.967%); 3224/73908 objects misplaced (4.362%); 14 MiB/s, 297 objects/s recovering 2024-09-17T16:38:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:05 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 10323/73908 objects degraded (13.967%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:05.814 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:05.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:05.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:05.946 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:05.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:06.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:06.072 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:06.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:06.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:06.209 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:06.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:06.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:06.343 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:06.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:06.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:06.353 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.10 to 1 (10000) 2024-09-17T16:38:06.445 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:06.470 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:06.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:06.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:06.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:06.597 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:06.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2097346393' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:38:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:06 smithi173 ceph-mon[30530]: from='client.18339 172.21.15.17:0/2097346393' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:38:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2097346393' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:38:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:06 smithi017 ceph-mon[27571]: from='client.18339 172.21.15.17:0/2097346393' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:38:06.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:06.727 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:06.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:06.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:06.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2097346393' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:38:06.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:06 smithi071 ceph-mon[30002]: from='client.18339 172.21.15.17:0/2097346393' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:38:06.830 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:06.853 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:06.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:06.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:06.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:06.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:06.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:06.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:07.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:07.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:07.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:07.213 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:07.237 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:07.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:07.339 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:07.364 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:07.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:07.466 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:07.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:07.501 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:07.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:07.603 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:07.635 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:07.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:07 smithi173 ceph-mon[30530]: pgmap v1851: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 599 op/s; 10323/69036 objects degraded (14.953%); 2412/69036 objects misplaced (3.494%); 17 MiB/s, 371 objects/s recovering 2024-09-17T16:38:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:07 smithi173 ceph-mon[30530]: from='client.18339 172.21.15.17:0/2097346393' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:38:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:07 smithi173 ceph-mon[30530]: osdmap e834: 12 total, 12 up, 12 in 2024-09-17T16:38:07.737 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:07.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:07 smithi017 ceph-mon[27571]: pgmap v1851: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 599 op/s; 10323/69036 objects degraded (14.953%); 2412/69036 objects misplaced (3.494%); 17 MiB/s, 371 objects/s recovering 2024-09-17T16:38:07.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:07 smithi017 ceph-mon[27571]: from='client.18339 172.21.15.17:0/2097346393' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:38:07.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:07 smithi017 ceph-mon[27571]: osdmap e834: 12 total, 12 up, 12 in 2024-09-17T16:38:07.765 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:07.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:07.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:07 smithi071 ceph-mon[30002]: pgmap v1851: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 599 op/s; 10323/69036 objects degraded (14.953%); 2412/69036 objects misplaced (3.494%); 17 MiB/s, 371 objects/s recovering 2024-09-17T16:38:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:07 smithi071 ceph-mon[30002]: from='client.18339 172.21.15.17:0/2097346393' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:38:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:07 smithi071 ceph-mon[30002]: osdmap e834: 12 total, 12 up, 12 in 2024-09-17T16:38:07.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:07.892 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:07.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:07.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:08.016 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:08.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:08.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:08.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:08.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:08.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:08.272 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:08.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:08.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:08.374 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:08.400 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:08.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:08.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:08.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:08.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:08.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:08 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:08 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:08.669 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:08.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:08.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:08 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:08.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:08 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:08.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:08.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:08.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:08 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:08.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:08 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:08.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:08.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:08.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:09.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:09.051 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:09.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:09.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:09.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:09.179 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:09.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:09.281 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:09.310 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:09.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:09.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:09.411 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:09.437 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:09.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:09.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:09.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:09.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:09 smithi173 ceph-mon[30530]: pgmap v1853: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s; 10239/63513 objects degraded (16.121%); 2039/63513 objects misplaced (3.210%); 14 MiB/s, 350 objects/s recovering 2024-09-17T16:38:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:09 smithi173 ceph-mon[30530]: osdmap e835: 12 total, 12 up, 12 in 2024-09-17T16:38:09.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:09 smithi017 ceph-mon[27571]: pgmap v1853: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s; 10239/63513 objects degraded (16.121%); 2039/63513 objects misplaced (3.210%); 14 MiB/s, 350 objects/s recovering 2024-09-17T16:38:09.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:09 smithi017 ceph-mon[27571]: osdmap e835: 12 total, 12 up, 12 in 2024-09-17T16:38:09.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:09.691 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:09.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:09.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:09 smithi071 ceph-mon[30002]: pgmap v1853: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s; 10239/63513 objects degraded (16.121%); 2039/63513 objects misplaced (3.210%); 14 MiB/s, 350 objects/s recovering 2024-09-17T16:38:09.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:09 smithi071 ceph-mon[30002]: osdmap e835: 12 total, 12 up, 12 in 2024-09-17T16:38:09.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:09.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:09.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:09.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:09.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:09.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:10.056 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:10.082 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:10.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:10.160 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:38:10.160 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:38:10.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.160 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:172147 8.9 e835/831 8:936fd7c0:::benchmark_data_smithi173_130566_object77172:head v 835'5839, mlcod=835'5837)", 2024-09-17T16:38:10.160 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:09.649007+0000", 2024-09-17T16:38:10.160 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.001880421, 2024-09-17T16:38:10.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00190613, 2024-09-17T16:38:10.160 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:10.160 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:10.160 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649007+0000", 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649007+0000", 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:10.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649007+0000", 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5700000000000002e-07 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649013+0000", 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.614e-06 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:10.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649039+0000", 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5411000000000001e-05 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649045+0000", 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0549999999999996e-06 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649096+0000", 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1313000000000002e-05 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649118+0000", 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1651e-05 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:172149 8.17 e835/743 8:e8a608b4:::benchmark_data_smithi173_130566_object77174:head v 835'5656, mlcod=835'5653)", 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:09.649637+0000", 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.001249834, 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001328517, 2024-09-17T16:38:10.164 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649637+0000", 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649637+0000", 2024-09-17T16:38:10.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649638+0000", 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6799999999999999e-07 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649644+0000", 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2419999999999997e-06 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649679+0000", 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4745e-05 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649686+0000", 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8310000000000003e-06 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649743+0000", 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6623000000000003e-05 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:10.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:10.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:10.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:09.649766+0000", 2024-09-17T16:38:10.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3659000000000002e-05 2024-09-17T16:38:10.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:10.168 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:10.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:10.168 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:10.168 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:38:10.168 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 2 2024-09-17T16:38:10.168 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:38:10.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:10.212 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:10.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:10.314 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:10.339 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:10.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:10.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:10.467 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:10.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:10.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 10239/63513 objects degraded (16.121%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:10.568 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:10.596 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:10.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:10.632 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 10239/63513 objects degraded (16.121%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 10239/63513 objects degraded (16.121%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:10.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:10.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:10.732 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:10.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:10.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:10.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:10.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:10.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:10.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:10.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:11.099 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:11.129 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:11.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:11.218 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-17T16:38:11.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:11.253 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:11.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:11.354 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:11.378 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:11.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:11.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:11.480 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:11.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:11 smithi017 ceph-mon[27571]: pgmap v1855: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s; 9935/57063 objects degraded (17.411%); 1774/57063 objects misplaced (3.109%); 8.6 MiB/s, 278 objects/s recovering 2024-09-17T16:38:11.512 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:11.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:11.614 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:11.638 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:11.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:11.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:11 smithi173 ceph-mon[30530]: pgmap v1855: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s; 9935/57063 objects degraded (17.411%); 1774/57063 objects misplaced (3.109%); 8.6 MiB/s, 278 objects/s recovering 2024-09-17T16:38:11.740 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:11.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:11.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:11 smithi071 ceph-mon[30002]: pgmap v1855: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s; 9935/57063 objects degraded (17.411%); 1774/57063 objects misplaced (3.109%); 8.6 MiB/s, 278 objects/s recovering 2024-09-17T16:38:11.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:11.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:11.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:12.001 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:12.031 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:12.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:12.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:12.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:12.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:12.168 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:12.173 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:38:12.206 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:12.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:12.286 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:12.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:12.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:12.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:12.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:12.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:12.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:12.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:12.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:12.676 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:12.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:12.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:12.778 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:12.804 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:12.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:12.860 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:38:12.860 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:38:12.868 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:12.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:12.913 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:38:12.914 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:38:12.929 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:12.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:12.951 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:38:12.951 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:38:12.987 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:38:12.987 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:38:13.021 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:38:13.022 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:38:13.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:13.051 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:38:13.051 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:38:13.056 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:13.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:13.078 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:38:13.079 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:38:13.111 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:38:13.111 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:38:13.145 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:38:13.145 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:38:13.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:13.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:38:13.177 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:38:13.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:13.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:13.215 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:38:13.216 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:38:13.250 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:38:13.250 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:38:13.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:13.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:13.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:13.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:13.452 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:13.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:13.554 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:13.580 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:13.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:13.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:13 smithi173 ceph-mon[30530]: pgmap v1856: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 9523/54075 objects degraded (17.611%); 1488/54075 objects misplaced (2.752%); 9.5 MiB/s, 323 objects/s recovering 2024-09-17T16:38:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2099642854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:13.665 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:38:13.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:13.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:13 smithi017 ceph-mon[27571]: pgmap v1856: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 9523/54075 objects degraded (17.611%); 1488/54075 objects misplaced (2.752%); 9.5 MiB/s, 323 objects/s recovering 2024-09-17T16:38:13.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2099642854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:13.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:13.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:13.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:13.811 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:13 smithi071 ceph-mon[30002]: pgmap v1856: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 9523/54075 objects degraded (17.611%); 1488/54075 objects misplaced (2.752%); 9.5 MiB/s, 323 objects/s recovering 2024-09-17T16:38:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2099642854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:13.835 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:13.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:13.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:13.937 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:13.962 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:13.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:14.063 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:14.103 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:14.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:14.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:14.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:14.236 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:14.337 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:14.362 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:14.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:14.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:14.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:14 smithi017 ceph-mon[27571]: osdmap e836: 12 total, 12 up, 12 in 2024-09-17T16:38:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:14.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:14.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:14.595 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:14.632 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:14.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:14.734 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:14.760 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:14.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:14.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:14 smithi071 ceph-mon[30002]: osdmap e836: 12 total, 12 up, 12 in 2024-09-17T16:38:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:14.863 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:14 smithi173 ceph-mon[30530]: osdmap e836: 12 total, 12 up, 12 in 2024-09-17T16:38:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:14.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:14.899 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:14.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:15.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:15.027 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:15.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:15.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:15.152 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:15.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:15.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:15.280 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:15.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:15.346 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:38:15.346 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:38:15.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:15.412 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:15.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:15.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:15.539 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:15.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:15.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:15.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:15.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:15 smithi173 ceph-mon[30530]: pgmap v1858: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 9522/49392 objects degraded (19.278%); 1292/49392 objects misplaced (2.616%); 6.9 MiB/s, 231 objects/s recovering 2024-09-17T16:38:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:15 smithi173 ceph-mon[30530]: osdmap e837: 12 total, 12 up, 12 in 2024-09-17T16:38:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:15 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:15 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:15 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 9522/49392 objects degraded (19.278%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:15.665 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:15.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:15.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:15 smithi017 ceph-mon[27571]: pgmap v1858: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 9522/49392 objects degraded (19.278%); 1292/49392 objects misplaced (2.616%); 6.9 MiB/s, 231 objects/s recovering 2024-09-17T16:38:15.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:15 smithi017 ceph-mon[27571]: osdmap e837: 12 total, 12 up, 12 in 2024-09-17T16:38:15.726 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:15 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:15.727 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:15 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:15.727 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:15 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 9522/49392 objects degraded (19.278%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:15.766 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:15.795 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:15.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:15 smithi071 ceph-mon[30002]: pgmap v1858: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 1.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 9522/49392 objects degraded (19.278%); 1292/49392 objects misplaced (2.616%); 6.9 MiB/s, 231 objects/s recovering 2024-09-17T16:38:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:15 smithi071 ceph-mon[30002]: osdmap e837: 12 total, 12 up, 12 in 2024-09-17T16:38:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:15 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:15 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:15 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 9522/49392 objects degraded (19.278%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:15.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:15.930 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:15.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:16.032 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:16.065 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:16.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:16.126 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:38:16.166 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:16.205 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:16.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:16.307 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:16.334 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:16.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:16.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:16 smithi017 ceph-mon[27571]: osdmap e838: 12 total, 12 up, 12 in 2024-09-17T16:38:16.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:16.461 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:16.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:16.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:16.486 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:16.548 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:16.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:16.604 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:16.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:16.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:16.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:16.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:16.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:16 smithi071 ceph-mon[30002]: osdmap e838: 12 total, 12 up, 12 in 2024-09-17T16:38:16.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:16.856 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:16.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:16.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:16 smithi173 ceph-mon[30530]: osdmap e838: 12 total, 12 up, 12 in 2024-09-17T16:38:16.958 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:16.980 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:16.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:17.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:17.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:17.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:17.196 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:38:17.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:17.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:17.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:17.336 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:17.359 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:17.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:17.420 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.11 to 1 (10000) 2024-09-17T16:38:17.460 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:17.483 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:17.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:17 smithi017 ceph-mon[27571]: pgmap v1861: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 919 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 9185/44091 objects degraded (20.832%); 850/44091 objects misplaced (1.928%); 6.8 MiB/s, 301 objects/s recovering 2024-09-17T16:38:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3726393120' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:38:17.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:17 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:38:17.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:17 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:17.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:17 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:17.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:17 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:17.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:17 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:17.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:17.607 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:17.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:17.709 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:17.731 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:17.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:17 smithi071 ceph-mon[30002]: pgmap v1861: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 919 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 9185/44091 objects degraded (20.832%); 850/44091 objects misplaced (1.928%); 6.8 MiB/s, 301 objects/s recovering 2024-09-17T16:38:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3726393120' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:38:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:17 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:38:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:17 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:17 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:17 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:17 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:17.833 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:17.856 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:17.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:17 smithi173 ceph-mon[30530]: pgmap v1861: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 919 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 720 op/s; 9185/44091 objects degraded (20.832%); 850/44091 objects misplaced (1.928%); 6.8 MiB/s, 301 objects/s recovering 2024-09-17T16:38:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3726393120' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:38:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:17 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:38:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:17 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:17.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:17 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:17.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:17 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:17.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:17 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:17.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:17.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:17.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:17.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:18.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:18.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:18.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:18.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:18.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:18.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:18.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:18.360 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:18.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:18.435 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:38:18.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:18.485 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:18.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:18.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:18.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:18.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:18.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:18.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:18.778 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:18 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:38:18.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:18 smithi017 ceph-mon[27571]: osdmap e839: 12 total, 12 up, 12 in 2024-09-17T16:38:18.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1351228459' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:38:18.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:18 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:38:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:18.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:18 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:38:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:18 smithi071 ceph-mon[30002]: osdmap e839: 12 total, 12 up, 12 in 2024-09-17T16:38:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1351228459' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:38:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:18 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:38:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:18 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:38:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:18 smithi173 ceph-mon[30530]: osdmap e839: 12 total, 12 up, 12 in 2024-09-17T16:38:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1351228459' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:38:18.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:18 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:38:18.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:18.929 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:18.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:19.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:19.054 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:19.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:19.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:19.180 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:19.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:19.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:19.307 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:19.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:19.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:19.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:19.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:19.539 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:19.561 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:19.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:19.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:19.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:19.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:19 smithi017 ceph-mon[27571]: pgmap v1863: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 802 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 8759/38484 objects degraded (22.760%); 850/38484 objects misplaced (2.209%); 2.2 MiB/s, 322 objects/s recovering 2024-09-17T16:38:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:19 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:38:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:19 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:38:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:19 smithi017 ceph-mon[27571]: osdmap e840: 12 total, 12 up, 12 in 2024-09-17T16:38:19.798 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:19 smithi071 ceph-mon[30002]: pgmap v1863: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 802 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 8759/38484 objects degraded (22.760%); 850/38484 objects misplaced (2.209%); 2.2 MiB/s, 322 objects/s recovering 2024-09-17T16:38:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:19 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:38:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:19 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:38:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:19 smithi071 ceph-mon[30002]: osdmap e840: 12 total, 12 up, 12 in 2024-09-17T16:38:19.822 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:19.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:19 smithi173 ceph-mon[30530]: pgmap v1863: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 802 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 767 op/s; 8759/38484 objects degraded (22.760%); 850/38484 objects misplaced (2.209%); 2.2 MiB/s, 322 objects/s recovering 2024-09-17T16:38:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:19 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:38:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:19 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:38:19.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:19 smithi173 ceph-mon[30530]: osdmap e840: 12 total, 12 up, 12 in 2024-09-17T16:38:19.910 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:38:19.910 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:38:19.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:19.954 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:19.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:19.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:20.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:20.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:20.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:20.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:20.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:20.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:20.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:20.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:20.327 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:20.364 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:20.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:20.466 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:20.491 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:20.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:20.593 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:20.618 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:20.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:20.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:20.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:20.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:20 smithi017 ceph-mon[27571]: osdmap e841: 12 total, 12 up, 12 in 2024-09-17T16:38:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:20 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 8759/38484 objects degraded (22.760%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:20.750 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:20.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:20 smithi071 ceph-mon[30002]: osdmap e841: 12 total, 12 up, 12 in 2024-09-17T16:38:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:20 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 8759/38484 objects degraded (22.760%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:20.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:20.874 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:20.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:20 smithi173 ceph-mon[30530]: osdmap e841: 12 total, 12 up, 12 in 2024-09-17T16:38:20.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:20 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 8759/38484 objects degraded (22.760%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:20.975 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:21.000 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:21.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:21.102 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:21.127 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:21.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:21.229 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:21.251 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:21.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:21.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:21.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:21.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:21.477 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:21.502 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:21.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:21.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:21.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:21.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:21.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:21.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:21 smithi173 ceph-mon[30530]: pgmap v1866: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 708 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 8000/33948 objects degraded (23.565%); 687/33948 objects misplaced (2.024%); 2.2 MiB/s, 412 objects/s recovering 2024-09-17T16:38:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:21 smithi173 ceph-mon[30530]: 8.13 scrub starts 2024-09-17T16:38:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:21 smithi173 ceph-mon[30530]: 8.13 scrub ok 2024-09-17T16:38:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:21 smithi173 ceph-mon[30530]: osdmap e842: 12 total, 12 up, 12 in 2024-09-17T16:38:21.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:21.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:21 smithi017 ceph-mon[27571]: pgmap v1866: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 708 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 8000/33948 objects degraded (23.565%); 687/33948 objects misplaced (2.024%); 2.2 MiB/s, 412 objects/s recovering 2024-09-17T16:38:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:21 smithi017 ceph-mon[27571]: 8.13 scrub starts 2024-09-17T16:38:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:21 smithi017 ceph-mon[27571]: 8.13 scrub ok 2024-09-17T16:38:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:21 smithi017 ceph-mon[27571]: osdmap e842: 12 total, 12 up, 12 in 2024-09-17T16:38:21.751 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:21.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:21.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:21 smithi071 ceph-mon[30002]: pgmap v1866: 27 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 708 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 747 op/s; 8000/33948 objects degraded (23.565%); 687/33948 objects misplaced (2.024%); 2.2 MiB/s, 412 objects/s recovering 2024-09-17T16:38:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:21 smithi071 ceph-mon[30002]: 8.13 scrub starts 2024-09-17T16:38:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:21 smithi071 ceph-mon[30002]: 8.13 scrub ok 2024-09-17T16:38:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:21 smithi071 ceph-mon[30002]: osdmap e842: 12 total, 12 up, 12 in 2024-09-17T16:38:21.852 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:21.883 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:21.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:21.985 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:22.014 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:22.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:22.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:22.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:22.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:22.248 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:22.274 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:22.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:22.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:22.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:22.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:22.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:22.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:22.532 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:22.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:22.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:22.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:22.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:22.765 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:22.796 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:22.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:22.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:22.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:22.929 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:22.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:23.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:23.053 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:23.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:23.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:22 smithi071 ceph-mon[30002]: 8.5 scrub starts 2024-09-17T16:38:23.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:22 smithi071 ceph-mon[30002]: 8.5 scrub ok 2024-09-17T16:38:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:22 smithi071 ceph-mon[30002]: osdmap e843: 12 total, 12 up, 12 in 2024-09-17T16:38:23.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:22 smithi071 ceph-mon[30002]: pgmap v1869: 27 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 669 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 6392/32070 objects degraded (19.931%); 687/32070 objects misplaced (2.142%); 752 KiB/s, 540 objects/s recovering 2024-09-17T16:38:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:22 smithi173 ceph-mon[30530]: 8.5 scrub starts 2024-09-17T16:38:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:22 smithi173 ceph-mon[30530]: 8.5 scrub ok 2024-09-17T16:38:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:22 smithi173 ceph-mon[30530]: osdmap e843: 12 total, 12 up, 12 in 2024-09-17T16:38:23.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:22 smithi173 ceph-mon[30530]: pgmap v1869: 27 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 669 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 6392/32070 objects degraded (19.931%); 687/32070 objects misplaced (2.142%); 752 KiB/s, 540 objects/s recovering 2024-09-17T16:38:23.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:23.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:23.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:22 smithi017 ceph-mon[27571]: 8.5 scrub starts 2024-09-17T16:38:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:22 smithi017 ceph-mon[27571]: 8.5 scrub ok 2024-09-17T16:38:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:22 smithi017 ceph-mon[27571]: osdmap e843: 12 total, 12 up, 12 in 2024-09-17T16:38:23.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:22 smithi017 ceph-mon[27571]: pgmap v1869: 27 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 20 active+clean; 669 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 6392/32070 objects degraded (19.931%); 687/32070 objects misplaced (2.142%); 752 KiB/s, 540 objects/s recovering 2024-09-17T16:38:23.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:23.306 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:23.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:23.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:23.429 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:23.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:23.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:23.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:23.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:23.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:23.677 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:23.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:23.779 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:23.803 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:23.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:23.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:23.934 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:23.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:24.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:24.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:23 smithi071 ceph-mon[30002]: osdmap e844: 12 total, 12 up, 12 in 2024-09-17T16:38:24.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:24.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:24.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:23 smithi173 ceph-mon[30530]: osdmap e844: 12 total, 12 up, 12 in 2024-09-17T16:38:24.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:24.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:24.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:24.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:23 smithi017 ceph-mon[27571]: osdmap e844: 12 total, 12 up, 12 in 2024-09-17T16:38:24.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:24.337 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:24.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:24.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:24.439 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:24.464 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:24.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:24.566 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:24.599 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:24.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:24.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:24.701 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:24.730 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:24.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:24.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:24.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:38:24.831 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:24.878 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:24.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:24.981 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:25.014 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:25.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:25.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:24 smithi071 ceph-mon[30002]: osdmap e845: 12 total, 12 up, 12 in 2024-09-17T16:38:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:24 smithi071 ceph-mon[30002]: pgmap v1872: 27 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 609 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5584/29199 objects degraded (19.124%); 1150/29199 objects misplaced (3.938%) 2024-09-17T16:38:25.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:24 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 6392/32070 objects degraded (19.931%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:25.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:24 smithi017 ceph-mon[27571]: osdmap e845: 12 total, 12 up, 12 in 2024-09-17T16:38:25.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:24 smithi017 ceph-mon[27571]: pgmap v1872: 27 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 609 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5584/29199 objects degraded (19.124%); 1150/29199 objects misplaced (3.938%) 2024-09-17T16:38:25.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:24 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 6392/32070 objects degraded (19.931%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:25.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:25.140 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:25.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:25.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:24 smithi173 ceph-mon[30530]: osdmap e845: 12 total, 12 up, 12 in 2024-09-17T16:38:25.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:24 smithi173 ceph-mon[30530]: pgmap v1872: 27 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 609 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5584/29199 objects degraded (19.124%); 1150/29199 objects misplaced (3.938%) 2024-09-17T16:38:25.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:24 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 6392/32070 objects degraded (19.931%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:25.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:25.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:25.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:25.374 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:25.407 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:25.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:25.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:25.509 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:25.533 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:25.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:25.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:25.635 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:25.660 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:25.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:25.743 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T16:38:25.748 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:25.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:25.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:25.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:25.798 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:25.890 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:25.916 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:25.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:25.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:26.018 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:26.051 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:26.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:26.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:26.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:26.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:26.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:26.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:26.312 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:26.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:26.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:26.439 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:26.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:26.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:26.573 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:26.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:26.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:26.698 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:26.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:26.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:26.780 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:38:26.780 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:38:26.799 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:26.814 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:38:26.815 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:38:26.827 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:26.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:26.848 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:38:26.848 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:38:26.885 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:38:26.885 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:38:26.919 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:38:26.919 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:38:26.928 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:26.952 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:26.953 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:38:26.953 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:38:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:26.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:26.973 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:38:26.973 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:38:26.984 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:38:26.984 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:38:27.012 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:38:27.012 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:38:27.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:27.080 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:27.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:27.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:38:27.182 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:27.204 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:27.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:27.242 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:38:27.242 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:38:27.264 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:38:27.264 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:38:27.289 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:38:27.290 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:38:27.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:27.327 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:38:27.328 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:38:27.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:27.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:27.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:27.454 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:27.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:27 smithi017 ceph-mon[27571]: pgmap v1873: 27 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 527 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4725/25260 objects degraded (18.705%); 937/25260 objects misplaced (3.709%); 0 B/s, 45 objects/s recovering 2024-09-17T16:38:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3446162224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:27.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:27 smithi071 ceph-mon[30002]: pgmap v1873: 27 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 527 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4725/25260 objects degraded (18.705%); 937/25260 objects misplaced (3.709%); 0 B/s, 45 objects/s recovering 2024-09-17T16:38:27.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3446162224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:27.579 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:27.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:27 smithi173 ceph-mon[30530]: pgmap v1873: 27 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 19 active+clean; 527 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4725/25260 objects degraded (18.705%); 937/25260 objects misplaced (3.709%); 0 B/s, 45 objects/s recovering 2024-09-17T16:38:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3446162224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:27.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:27.708 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:27.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:27.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:27.840 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:27.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:27.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:27.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:27.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:28.075 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:28.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:28.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:28.207 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:28.238 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:28.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:28.340 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:28.372 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:28.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:28.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:28.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:28.507 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:28.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:28.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:28.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:28.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:28.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:28.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:28.771 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:28.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:28.872 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:28.900 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:28.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:29.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:29.033 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:29.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:29.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:29.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:29.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:29.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:29.298 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:29.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:29.400 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:29.426 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:29.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:29 smithi017 ceph-mon[27571]: pgmap v1874: 27 pgs: 5 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 21 active+clean; 416 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4097/19932 objects degraded (20.555%); 716/19932 objects misplaced (3.592%); 0 B/s, 45 objects/s recovering 2024-09-17T16:38:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:29 smithi017 ceph-mon[27571]: 8.18 scrub starts 2024-09-17T16:38:29.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:29.550 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:29.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:29 smithi071 ceph-mon[30002]: pgmap v1874: 27 pgs: 5 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 21 active+clean; 416 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4097/19932 objects degraded (20.555%); 716/19932 objects misplaced (3.592%); 0 B/s, 45 objects/s recovering 2024-09-17T16:38:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:29 smithi071 ceph-mon[30002]: 8.18 scrub starts 2024-09-17T16:38:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:29 smithi173 ceph-mon[30530]: pgmap v1874: 27 pgs: 5 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 21 active+clean; 416 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4097/19932 objects degraded (20.555%); 716/19932 objects misplaced (3.592%); 0 B/s, 45 objects/s recovering 2024-09-17T16:38:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:29 smithi173 ceph-mon[30530]: 8.18 scrub starts 2024-09-17T16:38:29.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:29.674 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:29.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:29.724 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:38:29.776 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:29.800 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:29.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:29.902 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:29.927 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:29.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:30.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:30.057 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:30.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:30.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:30.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:30.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:30.273 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:30.296 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:30.356 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:30.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:30.458 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:30.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:30 smithi017 ceph-mon[27571]: 8.18 scrub ok 2024-09-17T16:38:30.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:30 smithi017 ceph-mon[27571]: osdmap e846: 12 total, 12 up, 12 in 2024-09-17T16:38:30.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:30 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 4097/19932 objects degraded (20.555%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:30.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:30 smithi017 ceph-mon[27571]: osdmap e847: 12 total, 12 up, 12 in 2024-09-17T16:38:30.482 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:30.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:30.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:30.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:30 smithi071 ceph-mon[30002]: 8.18 scrub ok 2024-09-17T16:38:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:30 smithi071 ceph-mon[30002]: osdmap e846: 12 total, 12 up, 12 in 2024-09-17T16:38:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:30 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 4097/19932 objects degraded (20.555%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:30 smithi071 ceph-mon[30002]: osdmap e847: 12 total, 12 up, 12 in 2024-09-17T16:38:30.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:30.607 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:30.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:30 smithi173 ceph-mon[30530]: 8.18 scrub ok 2024-09-17T16:38:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:30 smithi173 ceph-mon[30530]: osdmap e846: 12 total, 12 up, 12 in 2024-09-17T16:38:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:30 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 4097/19932 objects degraded (20.555%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:30 smithi173 ceph-mon[30530]: osdmap e847: 12 total, 12 up, 12 in 2024-09-17T16:38:30.682 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:30.710 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:30.734 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:30.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:30.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:30.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:30.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:30.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:31.018 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:31.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:31.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:31.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:31.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:31.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:31.287 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:31.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:31.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:31.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:31.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:31.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:31 smithi017 ceph-mon[27571]: 8.12 scrub starts 2024-09-17T16:38:31.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:31 smithi017 ceph-mon[27571]: pgmap v1877: 27 pgs: 1 active+remapped+backfilling, 5 active+recovering+undersized+degraded+remapped, 21 active+clean; 290 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4108/13899 objects degraded (29.556%); 225/13899 objects misplaced (1.619%); 4.5 MiB/s, 340 objects/s recovering 2024-09-17T16:38:31.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:31 smithi017 ceph-mon[27571]: 8.12 scrub ok 2024-09-17T16:38:31.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:31 smithi017 ceph-mon[27571]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T16:38:31.518 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:31.541 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:31.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:31 smithi071 ceph-mon[30002]: 8.12 scrub starts 2024-09-17T16:38:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:31 smithi071 ceph-mon[30002]: pgmap v1877: 27 pgs: 1 active+remapped+backfilling, 5 active+recovering+undersized+degraded+remapped, 21 active+clean; 290 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4108/13899 objects degraded (29.556%); 225/13899 objects misplaced (1.619%); 4.5 MiB/s, 340 objects/s recovering 2024-09-17T16:38:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:31 smithi071 ceph-mon[30002]: 8.12 scrub ok 2024-09-17T16:38:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:31 smithi071 ceph-mon[30002]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T16:38:31.575 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:31.615 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:38:31.615 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:38:31.615 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:38:31.615 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:38:31.615 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:38:31.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:31 smithi173 ceph-mon[30530]: 8.12 scrub starts 2024-09-17T16:38:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:31 smithi173 ceph-mon[30530]: pgmap v1877: 27 pgs: 1 active+remapped+backfilling, 5 active+recovering+undersized+degraded+remapped, 21 active+clean; 290 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4108/13899 objects degraded (29.556%); 225/13899 objects misplaced (1.619%); 4.5 MiB/s, 340 objects/s recovering 2024-09-17T16:38:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:31 smithi173 ceph-mon[30530]: 8.12 scrub ok 2024-09-17T16:38:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:31 smithi173 ceph-mon[30530]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T16:38:31.667 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:31.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:31.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:31.801 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:31.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:31.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:31.947 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:31.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:32.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:32.093 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:32.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:32.194 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:32.212 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T16:38:32.219 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:32.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:32.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:38:32.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:32.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:32.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:32.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:32 smithi017 ceph-mon[27571]: 8.9 scrub starts 2024-09-17T16:38:32.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:32 smithi017 ceph-mon[27571]: 8.9 scrub ok 2024-09-17T16:38:32.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3547383777' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:38:32.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:38:32.444 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:32.484 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:32.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:32 smithi071 ceph-mon[30002]: 8.9 scrub starts 2024-09-17T16:38:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:32 smithi071 ceph-mon[30002]: 8.9 scrub ok 2024-09-17T16:38:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3547383777' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:38:32.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:32.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:32.609 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:32.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:32 smithi173 ceph-mon[30530]: 8.9 scrub starts 2024-09-17T16:38:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:32 smithi173 ceph-mon[30530]: 8.9 scrub ok 2024-09-17T16:38:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3547383777' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:38:32.709 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:32.731 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:32.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:32.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:32.833 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:32.857 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:32.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:32.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:32.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:32.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:32.984 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:33.085 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:33.110 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:33.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:33.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:33.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:33.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:33.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:33.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:33.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:33.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:33.381 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:33.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3547383777' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:38:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:33 smithi017 ceph-mon[27571]: osdmap e848: 12 total, 12 up, 12 in 2024-09-17T16:38:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:38:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:38:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:38:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:38:33.491 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3547383777' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:38:33.491 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:33 smithi071 ceph-mon[30002]: osdmap e848: 12 total, 12 up, 12 in 2024-09-17T16:38:33.491 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:38:33.491 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:38:33.491 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:38:33.491 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:38:33.506 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:33.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:33.608 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:33.633 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:33.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:33.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3547383777' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:38:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:33 smithi173 ceph-mon[30530]: osdmap e848: 12 total, 12 up, 12 in 2024-09-17T16:38:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:38:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:38:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:38:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:38:33.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:33.774 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:33.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:33.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:33.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:33.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:34.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:34.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:34.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:34.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:34.163 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:34.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:34.265 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:34.288 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:34.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:34.391 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:34.414 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:34.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:34.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:34 smithi017 ceph-mon[27571]: osdmap e849: 12 total, 12 up, 12 in 2024-09-17T16:38:34.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:34 smithi017 ceph-mon[27571]: 8.1 scrub starts 2024-09-17T16:38:34.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:34 smithi017 ceph-mon[27571]: 8.1 scrub ok 2024-09-17T16:38:34.516 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:34.540 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:34.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:34.543 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:34 smithi071 ceph-mon[30002]: osdmap e849: 12 total, 12 up, 12 in 2024-09-17T16:38:34.544 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:34 smithi071 ceph-mon[30002]: 8.1 scrub starts 2024-09-17T16:38:34.544 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:34 smithi071 ceph-mon[30002]: 8.1 scrub ok 2024-09-17T16:38:34.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:34 smithi173 ceph-mon[30530]: osdmap e849: 12 total, 12 up, 12 in 2024-09-17T16:38:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:34 smithi173 ceph-mon[30530]: 8.1 scrub starts 2024-09-17T16:38:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:34 smithi173 ceph-mon[30530]: 8.1 scrub ok 2024-09-17T16:38:34.664 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:34.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:34.766 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:34.789 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:34.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:34.891 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:34.916 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:34.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:34.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:34.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T16:38:34.968 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 94972 objects 2024-09-17T16:38:34.968 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :172.339 2024-09-17T16:38:35.017 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:35.039 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:35.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:35.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:35.141 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:35.164 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:35.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:35.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:35.266 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:35.290 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:35.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:35.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:35 smithi071 ceph-mon[30002]: pgmap v1881: 27 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 22 active+clean; 126 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3255/6015 objects degraded (54.115%); 225/6015 objects misplaced (3.741%) 2024-09-17T16:38:35.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:35 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 3255/6015 objects degraded (54.115%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:35.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:35.429 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:35.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:35.484 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:35.485 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:35.485 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:35.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:35 smithi017 ceph-mon[27571]: pgmap v1881: 27 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 22 active+clean; 126 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3255/6015 objects degraded (54.115%); 225/6015 objects misplaced (3.741%) 2024-09-17T16:38:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:35 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 3255/6015 objects degraded (54.115%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:35.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:35.554 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:35.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:35 smithi173 ceph-mon[30530]: pgmap v1881: 27 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 22 active+clean; 126 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3255/6015 objects degraded (54.115%); 225/6015 objects misplaced (3.741%) 2024-09-17T16:38:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:35 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 3255/6015 objects degraded (54.115%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:35.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:35.677 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:35.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:35.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:35.805 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:35.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:35.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:35.931 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:35.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:36.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:36.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:36.055 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:38:36.055 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:38:36.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:38:36.055 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:38:36.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.055 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:95580 7.d 7:b4431bb8:::benchmark_data_smithi173_116396_object95579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e671)", 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:09.919360+0000", 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 446.13219571000002, 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38153392000000003, 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 95580 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919360+0000", 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919360+0000", 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919362+0000", 2024-09-17T16:38:36.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.818e-06 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919391+0000", 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8687000000000002e-05 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919408+0000", 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7034e-05 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919415+0000", 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4680000000000003e-06 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919476+0000", 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0708000000000002e-05 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919603+0000", 2024-09-17T16:38:36.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012697000000000001 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.919734+0000", 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131531 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.922948+0000", 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003214008 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923916+0000", 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096724599999999995 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300815+0000", 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37689942500000001 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300859+0000", 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4095999999999999e-05 2024-09-17T16:38:36.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300894+0000", 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4928999999999997e-05 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:95580 7.d e671/629 ondisk, result = 0)", 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:09.923561+0000", 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 446.12799530799998, 2024-09-17T16:38:36.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37734046100000002, 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923561+0000", 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923561+0000", 2024-09-17T16:38:36.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923562+0000", 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9500000000000001e-07 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.923565+0000", 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.924e-06 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300712+0000", 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37714677099999999 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300721+0000", 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0140000000000004e-06 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300801+0000", 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9324000000000001e-05 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300809+0000", 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7290000000000006e-06 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.300901+0000", 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1804000000000004e-05 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170891 7.b 7:d10181ca:::benchmark_data_smithi173_116396_object75291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.431248+0000", 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 309.620308153, 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.37566875599999999, 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170891 2024-09-17T16:38:36.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431248+0000", 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431248+0000", 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431250+0000", 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.829e-06 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431256+0000", 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1700000000000002e-06 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431270+0000", 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4051e-05 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431278+0000", 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1939999999999998e-06 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431360+0000", 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2365000000000002e-05 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431498+0000", 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013799400000000001 2024-09-17T16:38:36.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.431616+0000", 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117209 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.436464+0000", 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048481389999999996 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.439954+0000", 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034900729999999998 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.806805+0000", 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36685158699999998 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.806875+0000", 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9413999999999996e-05 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.806917+0000", 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1730999999999997e-05 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170904 7.d 7:b74ecf00:::benchmark_data_smithi173_116396_object75304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.444710+0000", 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 309.60684621899998, 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35556337700000001, 2024-09-17T16:38:36.073 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170904 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444710+0000", 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444710+0000", 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444712+0000", 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.874e-06 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444726+0000", 2024-09-17T16:38:36.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4463e-05 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444743+0000", 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7385e-05 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444751+0000", 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8760000000000005e-06 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444808+0000", 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7003999999999998e-05 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.444935+0000", 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012665699999999999 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.445085+0000", 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000150008 2024-09-17T16:38:36.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.454064+0000", 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089791750000000007 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457533+0000", 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034685369999999998 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.463684+0000", 2024-09-17T16:38:36.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061514509999999996 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.800223+0000", 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.336538689 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.800273+0000", 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0257999999999999e-05 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170913 7.d 7:b6585817:::benchmark_data_smithi173_116396_object75313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.457482+0000", 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 309.59407387599998, 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35359675800000001, 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170913 2024-09-17T16:38:36.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457482+0000", 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457482+0000", 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457484+0000", 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4890000000000001e-06 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457490+0000", 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2790000000000004e-06 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457502+0000", 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2573e-05 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457514+0000", 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1386000000000001e-05 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457558+0000", 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4502000000000002e-05 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457681+0000", 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012229 2024-09-17T16:38:36.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:38:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.457796+0000", 2024-09-17T16:38:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114926 2024-09-17T16:38:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:36.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.462659+0000", 2024-09-17T16:38:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048635800000000002 2024-09-17T16:38:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.807157+0000", 2024-09-17T16:38:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34449779000000003 2024-09-17T16:38:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.810897+0000", 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003739769 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.811012+0000", 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000115796 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.811079+0000", 2024-09-17T16:38:36.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6377999999999994e-05 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:170935 7.d 7:b3c7b477:::benchmark_data_smithi173_116396_object75335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:26.480767+0000", 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 309.57078909099999, 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.344245668, 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.088 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 170935 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480767+0000", 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480767+0000", 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480769+0000", 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7770000000000001e-06 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.480776+0000", 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7079999999999997e-06 2024-09-17T16:38:36.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801464+0000", 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32068738499999999 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801474+0000", 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8600000000000005e-06 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801522+0000", 2024-09-17T16:38:36.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7987e-05 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801669+0000", 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014749699999999999 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.801798+0000", 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012916300000000001 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.813516+0000", 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011717589 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.818920+0000", 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054044940000000001 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.824864+0000", 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005943444 2024-09-17T16:38:36.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.824967+0000", 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103049 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:26.825013+0000", 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5714999999999999e-05 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178134 7.d 7:b9b47827:::benchmark_data_smithi173_116396_object82534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e717)", 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.050884+0000", 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 300.00067208299998, 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48526529400000001, 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178134 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.095 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050884+0000", 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050884+0000", 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050885+0000", 2024-09-17T16:38:36.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1299999999999998e-07 2024-09-17T16:38:36.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050889+0000", 2024-09-17T16:38:36.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7529999999999999e-06 2024-09-17T16:38:36.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050899+0000", 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0837e-05 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050905+0000", 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3229999999999997e-06 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.050959+0000", 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4435000000000003e-05 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.051048+0000", 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8609000000000005e-05 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.051121+0000", 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2887999999999997e-05 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.054958+0000", 2024-09-17T16:38:36.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038371149999999999 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055583+0000", 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00062568400000000005 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536053+0000", 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48047000499999998 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536105+0000", 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1311999999999999e-05 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536149+0000", 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4419999999999998e-05 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.101 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:178134 7.d e717/629 ondisk, result = 0)", 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.055085+0000", 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 299.99647093599998, 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48106911299999999, 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055085+0000", 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055085+0000", 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055086+0000", 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.08e-06 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.055091+0000", 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1989999999999997e-06 2024-09-17T16:38:36.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.535995+0000", 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.48090385899999999 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.535999+0000", 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9899999999999999e-06 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536041+0000", 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1597999999999997e-05 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536048+0000", 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3170000000000004e-06 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.536154+0000", 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010607000000000001 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.105 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:178372 7.b 7:d011cddd:::benchmark_data_smithi173_116396_object82772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e718)", 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.909320+0000", 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 299.14223546199997, 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.399272617, 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:38:36.106 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 178372 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909320+0000", 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909320+0000", 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909322+0000", 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6080000000000001e-06 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909329+0000", 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4300000000000003e-06 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909345+0000", 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6188000000000002e-05 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909352+0000", 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7719999999999994e-06 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909421+0000", 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8571000000000005e-05 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909546+0000", 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012531700000000001 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.909678+0000", 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013156999999999999 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.913604+0000", 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039264749999999996 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.915100+0000", 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001495461 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308513+0000", 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39341283700000002 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308546+0000", 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3504999999999999e-05 2024-09-17T16:38:36.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308593+0000", 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6882999999999998e-05 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:178372 7.b e718/629 ondisk, result = 0)", 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 299.13677237600001, 2024-09-17T16:38:36.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.393815833, 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914784+0000", 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9800000000000002e-07 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:36.914790+0000", 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1270000000000002e-06 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308456+0000", 2024-09-17T16:38:36.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39366633899999998 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308463+0000", 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9480000000000002e-06 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308504+0000", 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1247e-05 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308509+0000", 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2499999999999997e-06 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:37.308599+0000", 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0024000000000004e-05 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(7.0 e725,723 7:04a95b88:::benchmark_data_smithi173_116396_object28410:head,690'6998)", 2024-09-17T16:38:36.116 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 278.09194525800001, 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89116632100000004, 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959611+0000", 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4199999999999995e-07 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:57.959615+0000", 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0319999999999997e-06 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850585+0000", 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.89097006499999998 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850600+0000", 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4747e-05 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850689+0000", 2024-09-17T16:38:36.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8546000000000002e-05 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850701+0000", 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1878e-05 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:58.850777+0000", 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6410999999999997e-05 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.120 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(7.0 e725,723 7:05ce1687:::benchmark_data_smithi173_116396_object29751:head,691'7038)", 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:59.220497+0000", 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 276.83105859099999, 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53176433000000001, 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220497+0000", 2024-09-17T16:38:36.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220497+0000", 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220498+0000", 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8599999999999998e-07 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.220501+0000", 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1209999999999998e-06 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752133+0000", 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53163205499999999 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752141+0000", 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4950000000000002e-06 2024-09-17T16:38:36.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752200+0000", 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9954000000000002e-05 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752208+0000", 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9810000000000003e-06 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:59.752262+0000", 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3238000000000001e-05 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.17 740/739 [PushReplyOp(8:ee7fa771:::benchmark_data_smithi173_130566_object904:head)])", 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 249.18757486199999, 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79826256500000004, 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.125 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863981+0000", 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.35e-07 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:26.863983+0000", 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.807e-06 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.661769+0000", 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79778559699999996 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.661776+0000", 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2660000000000004e-06 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.661830+0000", 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3662999999999998e-05 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.662244+0000", 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00041399699999999999 2024-09-17T16:38:36.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:94950 8.15 e769/743 8:aa553242:::benchmark_data_smithi173_130566_object94949:head v 769'3213, mlcod=769'3210)", 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:41.590971+0000", 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 174.46058543500001, 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.47015781000000001, 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.590971+0000", 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.590971+0000", 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.590971+0000", 2024-09-17T16:38:36.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8000000000000004e-07 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.591068+0000", 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6348999999999995e-05 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.059385+0000", 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.46831688199999999 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.059396+0000", 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1151999999999999e-05 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.059468+0000", 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2408000000000002e-05 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.059499+0000", 2024-09-17T16:38:36.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0930999999999999e-05 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.061083+0000", 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001583568 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.061128+0000", 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5639999999999997e-05 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.133 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:94971 8.b e769/728 8:db7e09c1:::benchmark_data_smithi173_130566_object94970:head v 769'5977, mlcod=769'5974)", 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:41.611309+0000", 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 174.440246575, 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.780065801, 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.611309+0000", 2024-09-17T16:38:36.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.611309+0000", 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.611310+0000", 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0899999999999999e-06 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.611418+0000", 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010796799999999999 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.388780+0000", 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.77736194300000006 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.388788+0000", 2024-09-17T16:38:36.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7999999999999999e-06 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.388850+0000", 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2075000000000005e-05 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.388869+0000", 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8453000000000001e-05 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.391341+0000", 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0024723890000000002 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.391375+0000", 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4082999999999999e-05 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.138 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:118344 8.7 8:e64af26a:::benchmark_data_smithi173_130566_object23369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e803)", 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:59.310148+0000", 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 96.741408039000007, 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35985360799999999, 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 118344 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.310148+0000", 2024-09-17T16:38:36.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.310148+0000", 2024-09-17T16:38:36.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.310149+0000", 2024-09-17T16:38:36.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.39e-06 2024-09-17T16:38:36.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.310155+0000", 2024-09-17T16:38:36.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1959999999999999e-06 2024-09-17T16:38:36.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.310166+0000", 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1443999999999999e-05 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.310174+0000", 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-06 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.310326+0000", 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015234999999999999 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.310421+0000", 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4658000000000001e-05 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 10,11", 2024-09-17T16:38:36.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.310558+0000", 2024-09-17T16:38:36.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000137828 2024-09-17T16:38:36.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:36.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.312820+0000", 2024-09-17T16:38:36.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022614229999999998 2024-09-17T16:38:36.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.321203+0000", 2024-09-17T16:38:36.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083830720000000001 2024-09-17T16:38:36.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.668882+0000", 2024-09-17T16:38:36.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34767857299999999 2024-09-17T16:38:36.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.668954+0000", 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2681000000000004e-05 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.670002+0000", 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001047295 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:36.149 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:118344 8.7 e803/792 ondisk, result = 0)", 2024-09-17T16:38:36.149 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:59.313694+0000", 2024-09-17T16:38:36.149 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 96.737861839000004, 2024-09-17T16:38:36.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35631558400000002, 2024-09-17T16:38:36.149 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.149 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:36.150 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.313694+0000", 2024-09-17T16:38:36.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.313694+0000", 2024-09-17T16:38:36.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.313695+0000", 2024-09-17T16:38:36.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4200000000000005e-07 2024-09-17T16:38:36.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.313702+0000", 2024-09-17T16:38:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9870000000000002e-06 2024-09-17T16:38:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.665266+0000", 2024-09-17T16:38:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35156412999999997 2024-09-17T16:38:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.665271+0000", 2024-09-17T16:38:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0540000000000002e-06 2024-09-17T16:38:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.668863+0000", 2024-09-17T16:38:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035916989999999998 2024-09-17T16:38:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.668874+0000", 2024-09-17T16:38:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0898e-05 2024-09-17T16:38:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.670010+0000", 2024-09-17T16:38:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011359740000000001 2024-09-17T16:38:36.155 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:151050 8.b e819/728 8:da77dfd3:::benchmark_data_smithi173_130566_object56075:head v 819'9567, mlcod=819'9564)", 2024-09-17T16:38:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.987724+0000", 2024-09-17T16:38:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.063832384999998, 2024-09-17T16:38:36.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33152717399999998, 2024-09-17T16:38:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.987724+0000", 2024-09-17T16:38:36.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.987724+0000", 2024-09-17T16:38:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.987724+0000", 2024-09-17T16:38:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:38:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.987729+0000", 2024-09-17T16:38:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9969999999999998e-06 2024-09-17T16:38:36.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.313625+0000", 2024-09-17T16:38:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32589519900000002 2024-09-17T16:38:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.313631+0000", 2024-09-17T16:38:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8210000000000004e-06 2024-09-17T16:38:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.314410+0000", 2024-09-17T16:38:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077860799999999997 2024-09-17T16:38:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.314431+0000", 2024-09-17T16:38:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0615000000000002e-05 2024-09-17T16:38:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.319213+0000", 2024-09-17T16:38:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004782868 2024-09-17T16:38:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.319251+0000", 2024-09-17T16:38:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7320999999999997e-05 2024-09-17T16:38:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.163 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:164088 8.1 8:85a81046:::benchmark_data_smithi173_130566_object69113:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e831)", 2024-09-17T16:38:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:59.343854+0000", 2024-09-17T16:38:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.707702142999999, 2024-09-17T16:38:36.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29178101299999998, 2024-09-17T16:38:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:38:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:36.165 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 164088 2024-09-17T16:38:36.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.168 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343854+0000", 2024-09-17T16:38:36.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343854+0000", 2024-09-17T16:38:36.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343854+0000", 2024-09-17T16:38:36.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2100000000000002e-07 2024-09-17T16:38:36.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343857+0000", 2024-09-17T16:38:36.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.819e-06 2024-09-17T16:38:36.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343864+0000", 2024-09-17T16:38:36.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.737e-06 2024-09-17T16:38:36.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343869+0000", 2024-09-17T16:38:36.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3499999999999996e-06 2024-09-17T16:38:36.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.344262+0000", 2024-09-17T16:38:36.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00039309000000000002 2024-09-17T16:38:36.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:38:36.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.344266+0000", 2024-09-17T16:38:36.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.811e-06 2024-09-17T16:38:36.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.602812+0000", 2024-09-17T16:38:36.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25854550199999998 2024-09-17T16:38:36.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.602953+0000", 2024-09-17T16:38:36.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014157899999999999 2024-09-17T16:38:36.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,9", 2024-09-17T16:38:36.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.603088+0000", 2024-09-17T16:38:36.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013441700000000001 2024-09-17T16:38:36.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:36.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.607997+0000", 2024-09-17T16:38:36.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049089659999999999 2024-09-17T16:38:36.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.608063+0000", 2024-09-17T16:38:36.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6913000000000001e-05 2024-09-17T16:38:36.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.634237+0000", 2024-09-17T16:38:36.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.026173528000000001 2024-09-17T16:38:36.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.634410+0000", 2024-09-17T16:38:36.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000172633 2024-09-17T16:38:36.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.635635+0000", 2024-09-17T16:38:36.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012252470000000001 2024-09-17T16:38:36.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.183 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.184 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:164104 8.1 8:849f2a3e:::benchmark_data_smithi173_130566_object69129:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e831)", 2024-09-17T16:38:36.184 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:59.343877+0000", 2024-09-17T16:38:36.184 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.707678934, 2024-09-17T16:38:36.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29261569599999998, 2024-09-17T16:38:36.185 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:36.185 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:36.185 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:36.185 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:36.185 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:38:36.186 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:36.186 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 164104 2024-09-17T16:38:36.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.186 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:36.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:36.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343877+0000", 2024-09-17T16:38:36.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:36.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343877+0000", 2024-09-17T16:38:36.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:36.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:36.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343877+0000", 2024-09-17T16:38:36.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0699999999999998e-07 2024-09-17T16:38:36.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:36.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343880+0000", 2024-09-17T16:38:36.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5059999999999998e-06 2024-09-17T16:38:36.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:36.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343887+0000", 2024-09-17T16:38:36.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5649999999999998e-06 2024-09-17T16:38:36.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.343892+0000", 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4990000000000002e-06 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.344308+0000", 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000416232 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:38:36.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.344315+0000", 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0090000000000001e-06 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.608107+0000", 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.26379139699999998 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.608248+0000", 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014142000000000001 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,9", 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.608396+0000", 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000148276 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.614646+0000", 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0062493790000000002 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.621921+0000", 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072753230000000002 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.635689+0000", 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013768343000000001 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.635782+0000", 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2956999999999998e-05 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.636493+0000", 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00071038299999999998 2024-09-17T16:38:36.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.195 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:36.195 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:36.195 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:38:36.232 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:38:36.249 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:36.271 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:36.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:36.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:36.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:36.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:36.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:36 smithi017 ceph-mon[27571]: 8.3 scrub starts 2024-09-17T16:38:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:36 smithi017 ceph-mon[27571]: 8.3 scrub ok 2024-09-17T16:38:36.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3280766369' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:38:36.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:36.522 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:36.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:36 smithi071 ceph-mon[30002]: 8.3 scrub starts 2024-09-17T16:38:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:36 smithi071 ceph-mon[30002]: 8.3 scrub ok 2024-09-17T16:38:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3280766369' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:38:36.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:36.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:36 smithi173 ceph-mon[30530]: 8.3 scrub starts 2024-09-17T16:38:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:36 smithi173 ceph-mon[30530]: 8.3 scrub ok 2024-09-17T16:38:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3280766369' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:38:36.647 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:36.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:36.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:36.749 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:36.772 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:36.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:36.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:36.896 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:36.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:36.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:37.022 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:37.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:37.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:37.146 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:37.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:37.248 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:37.270 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:37.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:37.372 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:37.396 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:37.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:37 smithi017 ceph-mon[27571]: pgmap v1882: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 23 active+clean; 51 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3255/2412 objects degraded (134.950%) 2024-09-17T16:38:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:37 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3280766369' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:38:37.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:37 smithi017 ceph-mon[27571]: osdmap e850: 12 total, 12 up, 12 in 2024-09-17T16:38:37.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:37.523 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:37.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:37 smithi071 ceph-mon[30002]: pgmap v1882: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 23 active+clean; 51 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3255/2412 objects degraded (134.950%) 2024-09-17T16:38:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:37 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3280766369' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:38:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:37 smithi071 ceph-mon[30002]: osdmap e850: 12 total, 12 up, 12 in 2024-09-17T16:38:37.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:37.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:37 smithi173 ceph-mon[30530]: pgmap v1882: 27 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 23 active+clean; 51 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3255/2412 objects degraded (134.950%) 2024-09-17T16:38:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:37 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3280766369' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:38:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:37 smithi173 ceph-mon[30530]: osdmap e850: 12 total, 12 up, 12 in 2024-09-17T16:38:37.648 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:37.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:37.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:37.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:37.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:37.875 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:37.900 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:37.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:38.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:38.028 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:38.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:38.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:38.154 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:38.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:38.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:38.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:38.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:38.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:38.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:38.406 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:38.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:38.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:38.513 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:38:38.513 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:38:38.531 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:38.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:38.538 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:38:38.538 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:38:38.562 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:38:38.562 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:38:38.582 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:38:38.583 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:38:38.603 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:38:38.604 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:38:38.624 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:38:38.625 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:38:38.632 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:38.647 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:38:38.647 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:38:38.653 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:38.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:38.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:38.668 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:38:38.668 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:38:38.668 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:38 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:38.669 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:38 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:38.689 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:38:38.689 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:38:38.711 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:38:38.711 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:38:38.735 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:38:38.735 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:38:38.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:38.757 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:38:38.758 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:38:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:38.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:38.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:38 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:38 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:38.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:38.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:38 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:38 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:38.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:38.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:39.010 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:39.032 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:39.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:39.134 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:39.157 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:39.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:39.259 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:39.281 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:39.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:39.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:39.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:39.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:38:39.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:39.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:39.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:39.527 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:39.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:39.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:39.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:39.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:39 smithi017 ceph-mon[27571]: pgmap v1884: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 24 active+clean; 11 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2771/492 objects degraded (563.211%) 2024-09-17T16:38:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:39 smithi017 ceph-mon[27571]: osdmap e851: 12 total, 12 up, 12 in 2024-09-17T16:38:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2932117589' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:39 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:39 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:39.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:39.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:39.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:39.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:39 smithi173 ceph-mon[30530]: pgmap v1884: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 24 active+clean; 11 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2771/492 objects degraded (563.211%) 2024-09-17T16:38:39.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:39 smithi173 ceph-mon[30530]: osdmap e851: 12 total, 12 up, 12 in 2024-09-17T16:38:39.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2932117589' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:39.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:39 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:39.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:39 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:39 smithi071 ceph-mon[30002]: pgmap v1884: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 24 active+clean; 11 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2771/492 objects degraded (563.211%) 2024-09-17T16:38:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:39 smithi071 ceph-mon[30002]: osdmap e851: 12 total, 12 up, 12 in 2024-09-17T16:38:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2932117589' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:39 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:39 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:39.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:39.900 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:39.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:39.983 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-09-17T16:38:39.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-09-17T16:38:40.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:40.025 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:40.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:40.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:40.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:40.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:40.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:40.275 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:40.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:40.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:40.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:40.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:40.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:40.527 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:40.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:40.629 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:40.652 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:40.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:40.716 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2062658417' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:38:40.716 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:40 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2771/492 objects degraded (563.211%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2062658417' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:38:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:40 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2771/492 objects degraded (563.211%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:40.753 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:40.776 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:40.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2062658417' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:38:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:40 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2771/492 objects degraded (563.211%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:38:40.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:40.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:40.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:41.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:41.026 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:41.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:41.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:41.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:38:41.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:41.189 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:41.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T16:38:41.292 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:41.316 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:41.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:41.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:41.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:41.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:41.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:41.567 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:41.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:41.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:41.691 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:41.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:41.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:41 smithi017 ceph-mon[27571]: pgmap v1886: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 24 active+clean; 1.9 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 287 op/s; 2389/69 objects degraded (3462.319%); 1.5 MiB/s, 249 objects/s recovering 2024-09-17T16:38:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2062658417' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:38:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:41 smithi017 ceph-mon[27571]: osdmap e852: 12 total, 12 up, 12 in 2024-09-17T16:38:41.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:41 smithi071 ceph-mon[30002]: pgmap v1886: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 24 active+clean; 1.9 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 287 op/s; 2389/69 objects degraded (3462.319%); 1.5 MiB/s, 249 objects/s recovering 2024-09-17T16:38:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2062658417' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:38:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:41 smithi071 ceph-mon[30002]: osdmap e852: 12 total, 12 up, 12 in 2024-09-17T16:38:41.815 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:41.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:41.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:41 smithi173 ceph-mon[30530]: pgmap v1886: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 24 active+clean; 1.9 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 287 op/s; 2389/69 objects degraded (3462.319%); 1.5 MiB/s, 249 objects/s recovering 2024-09-17T16:38:41.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2062658417' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:38:41.848 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:41 smithi173 ceph-mon[30530]: osdmap e852: 12 total, 12 up, 12 in 2024-09-17T16:38:41.910 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T16:38:41.916 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:41.939 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:41.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:42.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:42.064 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:42.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:42.166 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:42.188 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:42.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:42.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:42.290 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:42.314 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:42.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:42.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:42.416 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:42.438 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:42.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:42.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:42.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:42.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:42.618 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:42.664 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:42.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:42.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:42.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:42 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:42 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:42 smithi017 ceph-mon[27571]: osdmap e853: 12 total, 12 up, 12 in 2024-09-17T16:38:42.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:42 smithi017 ceph-mon[27571]: osdmap e854: 12 total, 12 up, 12 in 2024-09-17T16:38:42.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:42.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:42.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:42 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:42 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:42 smithi071 ceph-mon[30002]: osdmap e853: 12 total, 12 up, 12 in 2024-09-17T16:38:42.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:42 smithi071 ceph-mon[30002]: osdmap e854: 12 total, 12 up, 12 in 2024-09-17T16:38:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:42 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:42 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:42 smithi173 ceph-mon[30530]: osdmap e853: 12 total, 12 up, 12 in 2024-09-17T16:38:42.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:42 smithi173 ceph-mon[30530]: osdmap e854: 12 total, 12 up, 12 in 2024-09-17T16:38:42.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:42.936 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:42.936 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:43.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:43.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:43.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:43.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:43.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:43.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:43.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:43.307 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:43.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:43.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:43.434 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:43.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:43.471 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_6' removed 2024-09-17T16:38:43.536 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:43.557 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:43.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:43.659 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:43.682 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:43.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:43 smithi017 ceph-mon[27571]: pgmap v1889: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 24 active+clean; 1.9 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 89 op/s; 1423/69 objects degraded (2062.319%); 0 B/s, 345 objects/s recovering 2024-09-17T16:38:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1455748985' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:38:43.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:43 smithi071 ceph-mon[30002]: pgmap v1889: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 24 active+clean; 1.9 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 89 op/s; 1423/69 objects degraded (2062.319%); 0 B/s, 345 objects/s recovering 2024-09-17T16:38:43.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1455748985' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:38:43.822 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:43.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:43 smithi173 ceph-mon[30530]: pgmap v1889: 27 pgs: 3 active+recovering+undersized+degraded+remapped, 24 active+clean; 1.9 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 89 op/s; 1423/69 objects degraded (2062.319%); 0 B/s, 345 objects/s recovering 2024-09-17T16:38:43.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1455748985' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:38:43.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:43.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:43.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:44.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:44.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:44.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:44.070 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:44.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:44.195 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:44.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:44.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:44.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:44.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:44.421 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:44.445 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:44.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:44.548 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:44.570 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:44.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:44.612 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:38:44.672 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:44.693 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:44.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1455748985' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:38:44.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:44 smithi017 ceph-mon[27571]: osdmap e855: 12 total, 12 up, 12 in 2024-09-17T16:38:44.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1455748985' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:38:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:44 smithi071 ceph-mon[30002]: osdmap e855: 12 total, 12 up, 12 in 2024-09-17T16:38:44.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:44.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1455748985' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:38:44.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:44 smithi173 ceph-mon[30530]: osdmap e855: 12 total, 12 up, 12 in 2024-09-17T16:38:44.920 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:44.944 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:44.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:45.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:45.072 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:45.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:45.095 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:38:45.096 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:38:45.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:45.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:45.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:45.301 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:45.323 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:45.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:45.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:45.425 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:45.447 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:45.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:45.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:45.572 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:45.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:45.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:45.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:45.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:45 smithi017 ceph-mon[27571]: pgmap v1892: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:45 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1423/69 objects degraded (2062.319%), 3 pgs degraded, 2 pgs undersized) 2024-09-17T16:38:45.801 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:45.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:45 smithi071 ceph-mon[30002]: pgmap v1892: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:45.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:45 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1423/69 objects degraded (2062.319%), 3 pgs degraded, 2 pgs undersized) 2024-09-17T16:38:45.824 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:45.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:45 smithi173 ceph-mon[30530]: pgmap v1892: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:45 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1423/69 objects degraded (2062.319%), 3 pgs degraded, 2 pgs undersized) 2024-09-17T16:38:45.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:45.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:45.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:46.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:46.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:46.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:46.174 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-17T16:38:46.175 INFO:tasks.radosbench:Beginning radosbench... 2024-09-17T16:38:46.175 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-09-17T16:38:46.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-09-17T16:38:46.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:46.240 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:46.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:46.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:46.342 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:46.369 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:46.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:46.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:46.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:46.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:46.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:46.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:46.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:46.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:46.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:38:46.760 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:46.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:46.816 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:46.817 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:46.857 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:46.862 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:46.866 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:46.887 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:46.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:46.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:47.015 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:47.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:47.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:47.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:47.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:47.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:47.271 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:47.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:47.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:47.398 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:47.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:47.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:47.525 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:47.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:47.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:47.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:47.653 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:47.755 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:47.780 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:47.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:47.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:47 smithi071 ceph-mon[30002]: pgmap v1893: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:47.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:47.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:47 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:47.863 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:47 smithi017 ceph-mon[27571]: pgmap v1893: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:47.863 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:47.863 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:47 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:47.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:47 smithi173 ceph-mon[30530]: pgmap v1893: 1 pgs: 1 active+clean; 577 KiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:47 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:47.906 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:47.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:48.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:48.031 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:48.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:48.133 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:48.156 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:48.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:48.258 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:48.259 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:38:48.259 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:38:48.283 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:38:48.284 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:38:48.284 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:48.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:48.297 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:48.305 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:38:48.305 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:38:48.331 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:38:48.331 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:38:48.353 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:38:48.353 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:38:48.387 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:48.390 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:38:48.390 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:38:48.411 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:48.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:48.414 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:38:48.415 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:38:48.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:38:48.438 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:38:48.460 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:38:48.460 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:38:48.483 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:38:48.483 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:38:48.494 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T16:38:48.494 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:38:48.507 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:38:48.507 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:38:48.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:48.532 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:38:48.532 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:38:48.538 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:48.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:48.640 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:48.666 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:48.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2927378960' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:38:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3295415748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:38:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2681042378' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:48 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:38:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:48 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:38:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:48 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:48.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:48.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:48.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2927378960' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:38:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3295415748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:38:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2681042378' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:48 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:38:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:48 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:38:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:48 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2927378960' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:38:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3295415748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:38:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2681042378' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:48 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:38:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:48 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:38:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:48 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:38:48.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:48.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:48.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:49.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:49.032 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:38:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:38:49.032 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:38:49.033 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:38:49.033 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:38:49.046 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:49.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:49.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:49.148 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:49.173 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:49.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:49.276 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:49.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:49.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:49.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:49.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:49.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:49.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:49.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:49.590 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:49.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:49.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:49.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:49 smithi071 ceph-mon[30002]: pgmap v1894: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:49 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2927378960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:38:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3295415748' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:38:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:49 smithi071 ceph-mon[30002]: osdmap e856: 12 total, 12 up, 12 in 2024-09-17T16:38:49.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:49.847 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:49.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:49.888 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:49 smithi017 ceph-mon[27571]: pgmap v1894: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:49.888 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:49 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:49.888 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2927378960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:38:49.888 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3295415748' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:38:49.888 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:49 smithi017 ceph-mon[27571]: osdmap e856: 12 total, 12 up, 12 in 2024-09-17T16:38:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:49 smithi173 ceph-mon[30530]: pgmap v1894: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:49 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2927378960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:38:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3295415748' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:38:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:49 smithi173 ceph-mon[30530]: osdmap e856: 12 total, 12 up, 12 in 2024-09-17T16:38:49.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:49.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:49.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:50.077 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:50.103 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:50.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:50.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:50.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:50.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:50.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:50.333 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:50.358 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:50.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:50.460 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:50.487 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:50.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:50.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:50.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:50.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:50.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:50.744 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:50.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:50.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:50 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:50 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:50 smithi071 ceph-mon[30002]: osdmap e857: 12 total, 12 up, 12 in 2024-09-17T16:38:50.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:50.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:50.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:50 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:50 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:50 smithi173 ceph-mon[30530]: osdmap e857: 12 total, 12 up, 12 in 2024-09-17T16:38:50.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:50 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:50 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:50.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:50 smithi017 ceph-mon[27571]: osdmap e857: 12 total, 12 up, 12 in 2024-09-17T16:38:50.998 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:50.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:51.100 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:51.128 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:51.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:51.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:51.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:51.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:51.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:51.385 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:51.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:51.488 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:38:51.514 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:51.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:38:51.617 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:51.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:51.643 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:51.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:51 smithi173 ceph-mon[30530]: pgmap v1897: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:51.745 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:51.773 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:51.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:51 smithi071 ceph-mon[30002]: pgmap v1897: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:51.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:51.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:51.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:51.900 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:51.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:51 smithi017 ceph-mon[27571]: pgmap v1897: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:52.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:52.027 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:52.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:52.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:52.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:52.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:52.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:52.281 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:52.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:52.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:52.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:52.407 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:52.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:52.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:52.532 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:52.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:52.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:52.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:52.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:52.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:52.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:52.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:52.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:52.850 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T16:38:52.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:38:52.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:52.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:52.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:53.017 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:53.043 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:53.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:53.145 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:53.171 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:53.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:53.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:53.291 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:38:53.299 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:53.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:53.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:53.427 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:53.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:53.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:53.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:53.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:53.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:53.679 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:53.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:53.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:53 smithi017 ceph-mon[27571]: pgmap v1898: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:53.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:53.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:53.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:53.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:53 smithi071 ceph-mon[30002]: pgmap v1898: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:53 smithi173 ceph-mon[30530]: pgmap v1898: 1 pgs: 1 active+clean; 577 KiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:53.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:53.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:53.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:54.027 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:54.028 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:54.039 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:54.066 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:54.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:54.104 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:38:54.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:54.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:54.194 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:54.294 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:54.320 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:54.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:54.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:54.448 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:54.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:54.512 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_7' created 2024-09-17T16:38:54.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:54.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:54.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:54.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1915801589' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-17T16:38:54.635 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:38:54.635 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.26665.0:186285 6.f 6:ff66d09f:::benchmark_data_smithi173_102017_object93137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e621)", 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 577.06432609800004, 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53962611299999996, 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.636 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.26665", 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2034484317", 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186285 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536598+0000", 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536599+0000", 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4900000000000005e-07 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536603+0000", 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0380000000000001e-06 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536614+0000", 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1097000000000001e-05 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.536621+0000", 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5560000000000004e-06 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540416+0000", 2024-09-17T16:38:54.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037954429999999999 2024-09-17T16:38:54.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540537+0000", 2024-09-17T16:38:54.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012064499999999999 2024-09-17T16:38:54.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:38:54.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.540685+0000", 2024-09-17T16:38:54.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014845900000000001 2024-09-17T16:38:54.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:54.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.544648+0000", 2024-09-17T16:38:54.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039628129999999999 2024-09-17T16:38:54.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:17.800489+0000", 2024-09-17T16:38:54.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25584077599999999 2024-09-17T16:38:54.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076152+0000", 2024-09-17T16:38:54.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.27566357899999999 2024-09-17T16:38:54.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076194+0000", 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1468999999999999e-05 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:29:18.076224+0000", 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0488999999999999e-05 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:26018 7.8 7:151f2f24:::benchmark_data_smithi173_116396_object26017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e645)", 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 530.56479448000005, 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35239512200000001, 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26018 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.644 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036130+0000", 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036131+0000", 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.029e-06 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036175+0000", 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4579000000000001e-05 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036188+0000", 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.218e-05 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036203+0000", 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5494000000000001e-05 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036337+0000", 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013415599999999999 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036422+0000", 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4427999999999999e-05 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.036500+0000", 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8009000000000005e-05 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037194+0000", 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00069466700000000005 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037673+0000", 2024-09-17T16:38:54.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047842099999999998 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388425+0000", 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35075205700000001 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388482+0000", 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7630999999999998e-05 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388525+0000", 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2471e-05 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:26018 7.8 e645/642 ondisk, result = 0)", 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 530.56352187799996, 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35113110600000003, 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.650 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037402+0000", 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037403+0000", 2024-09-17T16:38:54.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8700000000000001e-07 2024-09-17T16:38:54.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.037406+0000", 2024-09-17T16:38:54.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8710000000000001e-06 2024-09-17T16:38:54.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388344+0000", 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35093866000000001 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388350+0000", 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388413+0000", 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2257999999999999e-05 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388420+0000", 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0450000000000003e-06 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:30:04.388534+0000", 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011397599999999999 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:130635 7.8 7:14a1b993:::benchmark_data_smithi173_116396_object35035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 394.46814038500003, 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.329086555, 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 130635 2024-09-17T16:38:54.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132785+0000", 2024-09-17T16:38:54.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5239999999999999e-06 2024-09-17T16:38:54.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132792+0000", 2024-09-17T16:38:54.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4860000000000002e-06 2024-09-17T16:38:54.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132805+0000", 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2792999999999999e-05 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132812+0000", 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1899999999999998e-06 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.133889+0000", 2024-09-17T16:38:54.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001076881 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.134010+0000", 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120796 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.134124+0000", 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114875 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.136882+0000", 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027574539999999999 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.136893+0000", 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.154e-05 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461760+0000", 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32486675399999998 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461812+0000", 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2231000000000002e-05 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461870+0000", 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8031000000000001e-05 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:191190 7.e 7:7081439d:::benchmark_data_smithi173_116396_object95590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:38:54.661 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:38:54.662 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 300.14783939699998, 2024-09-17T16:38:54.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.428509103, 2024-09-17T16:38:54.662 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.662 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.662 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:54.662 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.662 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:38:54.662 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:38:54.662 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 191190 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453086+0000", 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3510000000000001e-06 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453092+0000", 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6010000000000002e-06 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453103+0000", 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.065e-05 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453110+0000", 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1160000000000001e-06 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453277+0000", 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016785300000000001 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453458+0000", 2024-09-17T16:38:54.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018056699999999999 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453577+0000", 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118678 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.458776+0000", 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051995009999999996 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:54.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.458855+0000", 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8597999999999995e-05 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881478+0000", 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42262325899999997 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881537+0000", 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9354999999999997e-05 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.667 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881594+0000", 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6573999999999997e-05 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:191190 7.e e724/629 ondisk, result = 0)", 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 300.14294634700002, 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42362284100000003, 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.668 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457979+0000", 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457985+0000", 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9460000000000003e-06 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881412+0000", 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42342706200000002 2024-09-17T16:38:54.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881419+0000", 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.525e-06 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881467+0000", 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7667999999999998e-05 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881474+0000", 2024-09-17T16:38:54.671 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9489999999999999e-06 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881601+0000", 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126797 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.16 741/739 [PushReplyOp(8:6b780127:::benchmark_data_smithi173_130566_object3478:head)])", 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:38:54.672 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 266.81025110100001, 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49350836100000001, 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:38:54.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790674+0000", 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4700000000000001e-07 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790677+0000", 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6720000000000002e-06 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283734+0000", 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49305611500000002 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283742+0000", 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7960000000000001e-06 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283814+0000", 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1419000000000002e-05 2024-09-17T16:38:54.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.284182+0000", 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00036781200000000002 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:113372 8.16 8.fa455136 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:51.399293+0000", 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 123.201630954, 2024-09-17T16:38:54.676 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62166801999999999, 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 113372 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399293+0000", 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399293+0000", 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399294+0000", 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0300000000000001e-06 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399299+0000", 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8640000000000004e-06 2024-09-17T16:38:54.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399312+0000", 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2799000000000001e-05 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399325+0000", 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3037000000000001e-05 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399386+0000", 2024-09-17T16:38:54.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0699000000000003e-05 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399394+0000", 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5150000000000006e-06 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.020931+0000", 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62153622500000005 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.020961+0000", 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0851e-05 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:113391 8.16 8.88e8ec76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:51.486004+0000", 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 123.114919845, 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53498688299999997, 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.681 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 113391 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486004+0000", 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486004+0000", 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486006+0000", 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.288e-06 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486011+0000", 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.062e-06 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.683 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486022+0000", 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486027+0000", 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2809999999999999e-06 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486072+0000", 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4428999999999998e-05 2024-09-17T16:38:54.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486080+0000", 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.072e-06 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.020986+0000", 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53490651700000003 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.020991+0000", 2024-09-17T16:38:54.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9699999999999998e-06 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:113410 8.16 8.d6510676 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:51.576292+0000", 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 123.024632631, 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.444716628, 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.686 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 113410 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576292+0000", 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576292+0000", 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576293+0000", 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5060000000000001e-06 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576299+0000", 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3820000000000003e-06 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576313+0000", 2024-09-17T16:38:54.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4595e-05 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576327+0000", 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4236999999999999e-05 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576391+0000", 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3923000000000004e-05 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:38:54.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576405+0000", 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4049000000000001e-05 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.021005+0000", 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44459960999999998 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.021008+0000", 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.326e-06 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.690 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:113412 8.16 8.4f4dd716 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:51.621393+0000", 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 122.979530987, 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39962974299999998, 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:38:54.691 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 113412 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621393+0000", 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621393+0000", 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621395+0000", 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1820000000000001e-06 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621401+0000", 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3579999999999999e-06 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621412+0000", 2024-09-17T16:38:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0729e-05 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621418+0000", 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0419999999999999e-06 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621465+0000", 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7236e-05 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:38:54.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621472+0000", 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3130000000000001e-06 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.021020+0000", 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.399547665 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.021023+0000", 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2179999999999999e-06 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.695 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:151014 8.a e819/728 8:5e424e97:::benchmark_data_smithi173_130566_object56039:head v 819'9699, mlcod=819'9696)", 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.947285+0000", 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 74.653639702999996, 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36445988699999998, 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.947285+0000", 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.947285+0000", 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.947285+0000", 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5899999999999996e-07 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.947292+0000", 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3629999999999999e-06 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293044+0000", 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34575203199999999 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293061+0000", 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7142e-05 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293135+0000", 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4128999999999994e-05 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293149+0000", 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.311699+0000", 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.018550588999999999 2024-09-17T16:38:54.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.311745+0000", 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5015999999999997e-05 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:151027 8.16 8:6fc25e02:::benchmark_data_smithi173_130566_object56052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e819)", 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.957889+0000", 2024-09-17T16:38:54.700 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 74.643035690999994, 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.335442935, 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 151027 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957889+0000", 2024-09-17T16:38:54.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957889+0000", 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957891+0000", 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9300000000000002e-06 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957898+0000", 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6320000000000008e-06 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957914+0000", 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5889e-05 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957923+0000", 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1379999999999996e-06 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957984+0000", 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0322000000000003e-05 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.958074+0000", 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0797000000000005e-05 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.958141+0000", 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000004e-05 2024-09-17T16:38:54.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.960378+0000", 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002237242 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.960408+0000", 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9839999999999999e-05 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293223+0000", 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33281477500000001 2024-09-17T16:38:54.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293281+0000", 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7955000000000003e-05 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293332+0000", 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0995000000000002e-05 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:151027 8.16 e819/799 ondisk, result = 0)", 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.961092+0000", 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 74.639832370999997, 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33224929399999997, 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961092+0000", 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961092+0000", 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961093+0000", 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.99e-07 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961100+0000", 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7400000000000004e-06 2024-09-17T16:38:54.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961136+0000", 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6019999999999997e-05 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961143+0000", 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2469999999999997e-06 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293207+0000", 2024-09-17T16:38:54.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33206466800000001 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293216+0000", 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6759999999999996e-06 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293341+0000", 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000125244 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.710 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:151036 8.14 e819/743 8:2a2a8f98:::benchmark_data_smithi173_130566_object56061:head v 819'5028, mlcod=819'5026)", 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.965815+0000", 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 74.635109487999998, 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33232884699999998, 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.965815+0000", 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.965815+0000", 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.965815+0000", 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3900000000000004e-07 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.965821+0000", 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4e-06 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293041+0000", 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32722048399999998 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293067+0000", 2024-09-17T16:38:54.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5950000000000001e-05 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293125+0000", 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8149999999999997e-05 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293146+0000", 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0139000000000001e-05 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.298094+0000", 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049482839999999998 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.298144+0000", 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9801e-05 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.715 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:151048 8.a e819/728 8:5047d43e:::benchmark_data_smithi173_130566_object56073:head v 819'9700, mlcod=819'9697)", 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.979791+0000", 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 74.621133313000001, 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33199636799999999, 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.979791+0000", 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.979791+0000", 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.979792+0000", 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1820000000000001e-06 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.979800+0000", 2024-09-17T16:38:54.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1680000000000004e-06 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293188+0000", 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.313387315 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293194+0000", 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3169999999999998e-06 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.298071+0000", 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048773879999999999 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.298092+0000", 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0605e-05 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.311752+0000", 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013659538000000001 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.311787+0000", 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5855000000000001e-05 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:184622 8.16 8:6bf99951:::benchmark_data_smithi173_130566_object89647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:27.116186+0000", 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.484738506999999, 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33119328100000001, 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.720 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 184622 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116186+0000", 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.721 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116186+0000", 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116187+0000", 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.333e-06 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116193+0000", 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7100000000000004e-06 2024-09-17T16:38:54.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116208+0000", 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5121e-05 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116218+0000", 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5589999999999994e-06 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116275+0000", 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7401999999999997e-05 2024-09-17T16:38:54.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116386+0000", 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011100200000000001 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116458+0000", 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1557999999999995e-05 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.119255+0000", 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027974969999999999 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.151706+0000", 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.032451458000000002 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447257+0000", 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29555069499999997 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447305+0000", 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7873000000000001e-05 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447379+0000", 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4072999999999994e-05 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.726 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:184622 8.16 e845/799 ondisk, result = 0)", 2024-09-17T16:38:54.727 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:27.118496+0000", 2024-09-17T16:38:54.727 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.482427983000001, 2024-09-17T16:38:54.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32889318000000001, 2024-09-17T16:38:54.727 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.727 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.727 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:54.727 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.118496+0000", 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.118496+0000", 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.118497+0000", 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9299999999999997e-07 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.118501+0000", 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3599999999999998e-06 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447014+0000", 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32851213699999998 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447020+0000", 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2689999999999996e-06 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447245+0000", 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022503200000000001 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447252+0000", 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7469999999999999e-06 2024-09-17T16:38:54.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447390+0000", 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000137842 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:188946 8.16 8:6918a138:::benchmark_data_smithi173_130566_object93971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:32.231935+0000", 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.36898914, 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38706543900000001, 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.731 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 188946 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231935+0000", 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231935+0000", 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231937+0000", 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231944+0000", 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1049999999999997e-06 2024-09-17T16:38:54.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231962+0000", 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7714e-05 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231971+0000", 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9260000000000006e-06 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.232025+0000", 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4568000000000003e-05 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.232166+0000", 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000140822 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.232289+0000", 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012345499999999999 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.236743+0000", 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044531060000000001 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.259185+0000", 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022442655999999998 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618897+0000", 2024-09-17T16:38:54.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35971195 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618955+0000", 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7742999999999999e-05 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.619001+0000", 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5769000000000002e-05 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:188946 8.16 e848/799 ondisk, result = 0)", 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:32.234289+0000", 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.366635214999999, 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38471977499999999, 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.234289+0000", 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.234289+0000", 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.234290+0000", 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1299999999999998e-07 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.234297+0000", 2024-09-17T16:38:54.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5969999999999997e-06 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618656+0000", 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38435904399999998 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618661+0000", 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.733e-06 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618883+0000", 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022163900000000001 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618892+0000", 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.85e-06 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.619009+0000", 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116999 2024-09-17T16:38:54.741 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.742 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.742 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.742 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:38:54.742 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:38:54.742 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:38:54.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:54.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:54.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:54.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1915801589' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-17T16:38:54.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:54.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1915801589' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-17T16:38:54.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:54.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:55.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:55.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:55.025 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:55.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:55.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:55.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:55.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:55.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:55.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:55.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:55.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:55.353 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:55.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:55.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:55.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:55.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:55.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:55.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:55.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:55.653 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:55.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:55 smithi017 ceph-mon[27571]: pgmap v1899: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:55 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:38:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:55 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:38:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1915801589' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-17T16:38:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:55 smithi017 ceph-mon[27571]: osdmap e858: 12 total, 12 up, 12 in 2024-09-17T16:38:55.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:55 smithi017 ceph-mon[27571]: osdmap e859: 12 total, 12 up, 12 in 2024-09-17T16:38:55.756 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:55.778 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:55.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:55 smithi071 ceph-mon[30002]: pgmap v1899: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:55 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:38:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:55 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:38:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1915801589' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-17T16:38:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:55 smithi071 ceph-mon[30002]: osdmap e858: 12 total, 12 up, 12 in 2024-09-17T16:38:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:55 smithi071 ceph-mon[30002]: osdmap e859: 12 total, 12 up, 12 in 2024-09-17T16:38:55.880 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:55 smithi173 ceph-mon[30530]: pgmap v1899: 1 pgs: 1 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:55 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:38:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:55 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:38:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1915801589' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-17T16:38:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:55 smithi173 ceph-mon[30530]: osdmap e858: 12 total, 12 up, 12 in 2024-09-17T16:38:55.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:55 smithi173 ceph-mon[30530]: osdmap e859: 12 total, 12 up, 12 in 2024-09-17T16:38:55.905 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:55.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:56.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:38:56.006 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:38:56.030 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:56.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:56.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:38:56.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:56.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:56.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:56.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:56.288 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:56.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:56.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:56.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:56.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:56.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:56.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:56.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:56.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:56.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:56.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:56.673 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:56.760 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:38:56.767 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:56 smithi017 ceph-mon[27571]: osdmap e860: 12 total, 12 up, 12 in 2024-09-17T16:38:56.767 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3573653157' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:38:56.767 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:56 smithi017 ceph-mon[27571]: pgmap v1903: 17 pgs: 2 active+clean, 6 creating+peering, 9 unknown; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:56.774 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:56.799 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:56.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:56.807 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-09-17T16:38:56.901 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:56.926 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:56.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:57.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:57.050 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:57.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:57.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:56 smithi071 ceph-mon[30002]: osdmap e860: 12 total, 12 up, 12 in 2024-09-17T16:38:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3573653157' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:38:57.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:56 smithi071 ceph-mon[30002]: pgmap v1903: 17 pgs: 2 active+clean, 6 creating+peering, 9 unknown; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:56 smithi173 ceph-mon[30530]: osdmap e860: 12 total, 12 up, 12 in 2024-09-17T16:38:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3573653157' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:38:57.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:56 smithi173 ceph-mon[30530]: pgmap v1903: 17 pgs: 2 active+clean, 6 creating+peering, 9 unknown; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:57.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:57.173 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:57.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:57.275 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:57.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:57.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:57.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:57.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:57.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:57.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:57.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:57.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:57.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:57.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:57.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:57.787 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:57.812 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:57.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:57.897 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:38:57.897 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:38:57.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:57.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:57.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:57.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:58.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:38:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:57 smithi071 ceph-mon[30002]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:38:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:57 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3573653157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:38:58.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:57 smithi071 ceph-mon[30002]: osdmap e861: 12 total, 12 up, 12 in 2024-09-17T16:38:58.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:58.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:38:58.083 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:57 smithi017 ceph-mon[27571]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:38:58.084 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:57 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:58.084 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3573653157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:38:58.084 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:57 smithi017 ceph-mon[27571]: osdmap e861: 12 total, 12 up, 12 in 2024-09-17T16:38:58.112 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:38:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:57 smithi173 ceph-mon[30530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-17T16:38:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:57 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:38:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3573653157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:38:58.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:57 smithi173 ceph-mon[30530]: osdmap e861: 12 total, 12 up, 12 in 2024-09-17T16:38:58.169 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:38:58.195 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:58.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:38:58.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:38:58.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:58.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:38:58.426 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:58.451 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:58.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:58.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:58.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:58.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:58.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:58.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:58.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:58.778 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1904,"stamp":"2024-09-17T16:38:56.758937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":8,"kb":1124941824,"kb_used":7053532,"kb_used_data":268020,"kb_used_omap":85,"kb_used_meta":6784490,"kb_avail":1117888292,"statfs":{"total":1151940427776,"available":1144717611008,"internally_reserved":0,"allocated":274452480,"data_stored":268334360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87602,"internal_metadata":6947318222},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.624575"},"pg_stats":[{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":858,"state":"creating+peering","last_fresh":"2024-09-17T16:38:54.515516+0000","last_change":"2024-09-17T16:38:54.515516+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.515516+0000","last_undegraded":"2024-09-17T16:38:54.515516+0000","last_fullsized":"2024-09-17T16:38:54.515516+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:38:54.506703+0000","last_change":"2024-09-17T16:38:54.506703+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.506703+0000","last_undegraded":"2024-09-17T16:38:54.506703+0000","last_fullsized":"2024-09-17T16:38:54.506703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:38:54.506703+0000","last_change":"2024-09-17T16:38:54.506703+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.506703+0000","last_undegraded":"2024-09-17T16:38:54.506703+0000","last_fullsized":"2024-09-17T16:38:54.506703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-09-17T16:38:54.766230+0000","last_change":"2024-09-17T16:38:54.765585+0000","last_active":"2024-09-17T16:38:54.766230+0000","last_peered":"2024-09-17T16:38:54.766230+0000","last_clean":"2024-09-17T16:38:54.766230+0000","last_became_active":"2024-09-17T16:38:54.765180+0000","last_became_peered":"2024-09-17T16:38:54.765180+0000","last_unstale":"2024-09-17T16:38:54.766230+0000","last_undegraded":"2024-09-17T16:38:54.766230+0000","last_fullsized":"2024-09-17T16:38:54.766230+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:03.662925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":858,"state":"creating+peering","last_fresh":"2024-09-17T16:38:54.513935+0000","last_change":"2024-09-17T16:38:54.513935+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.513935+0000","last_undegraded":"2024-09-17T16:38:54.513935+0000","last_fullsized":"2024-09-17T16:38:54.513935+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":858,"state":"creating+peering","last_fresh":"2024-09-17T16:38:54.515475+0000","last_change":"2024-09-17T16:38:54.515475+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.515475+0000","last_undegraded":"2024-09-17T16:38:54.515475+0000","last_fullsized":"2024-09-17T16:38:54.515475+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:38:54.506703+0000","last_change":"2024-09-17T16:38:54.506703+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.506703+0000","last_undegraded":"2024-09-17T16:38:54.506703+0000","last_fullsized":"2024-09-17T16:38:54.506703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:38:54.506703+0000","last_change":"2024-09-17T16:38:54.506703+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.506703+0000","last_undegraded":"2024-09-17T16:38:54.506703+0000","last_fullsized":"2024-09-17T16:38:54.506703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":858,"state":"creating+peering","last_fresh":"2024-09-17T16:38:54.515408+0000","last_change":"2024-09-17T16:38:54.515408+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.515408+0000","last_undegraded":"2024-09-17T16:38:54.515408+0000","last_fullsized":"2024-09-17T16:38:54.515408+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":858,"state":"creating+peering","last_fresh":"2024-09-17T16:38:54.515506+0000","last_change":"2024-09-17T16:38:54.515506+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.515506+0000","last_undegraded":"2024-09-17T16:38:54.515506+0000","last_fullsized":"2024-09-17T16:38:54.515506+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:38:54.506703+0000","last_change":"2024-09-17T16:38:54.506703+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.506703+0000","last_undegraded":"2024-09-17T16:38:54.506703+0000","last_fullsized":"2024-09-17T16:38:54.506703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":858,"state":"creating+peering","last_fresh":"2024-09-17T16:38:54.515486+0000","last_change":"2024-09-17T16:38:54.515486+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.515486+0000","last_undegraded":"2024-09-17T16:38:54.515486+0000","last_fullsized":"2024-09-17T16:38:54.515486+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:38:54.506703+0000","last_change":"2024-09-17T16:38:54.506703+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.506703+0000","last_undegraded":"2024-09-17T16:38:54.506703+0000","last_fullsized":"2024-09-17T16:38:54.506703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-09-17T16:38:55.759457+0000","last_change":"2024-09-17T16:38:54.765566+0000","last_active":"2024-09-17T16:38:55.759457+0000","last_peered":"2024-09-17T16:38:55.759457+0000","last_clean":"2024-09-17T16:38:55.759457+0000","last_became_active":"2024-09-17T16:38:54.765272+0000","last_became_peered":"2024-09-17T16:38:54.765272+0000","last_unstale":"2024-09-17T16:38:55.759457+0000","last_undegraded":"2024-09-17T16:38:55.759457+0000","last_fullsized":"2024-09-17T16:38:55.759457+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:08.200736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:38:54.506703+0000","last_change":"2024-09-17T16:38:54.506703+0000","last_active":"2024-09-17T16:38:54.506703+0000","last_peered":"2024-09-17T16:38:54.506703+0000","last_clean":"2024-09-17T16:38:54.506703+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:38:54.506703+0000","last_undegraded":"2024-09-17T16:38:54.506703+0000","last_fullsized":"2024-09-17T16:38:54.506703+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":860,"state":"active+clean","last_fresh":"2024-09-17T16:38:55.759434+0000","last_change":"2024-09-17T16:38:54.768093+0000","last_active":"2024-09-17T16:38:55.759434+0000","last_peered":"2024-09-17T16:38:55.759434+0000","last_clean":"2024-09-17T16:38:55.759434+0000","last_became_active":"2024-09-17T16:38:54.767749+0000","last_became_peered":"2024-09-17T16:38:54.767749+0000","last_unstale":"2024-09-17T16:38:55.759434+0000","last_undegraded":"2024-09-17T16:38:55.759434+0000","last_fullsized":"2024-09-17T16:38:55.759434+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:03.514492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":490,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-09-17T16:38:54.756368+0000","last_change":"2024-09-17T16:33:59.364895+0000","last_active":"2024-09-17T16:38:54.756368+0000","last_peered":"2024-09-17T16:38:54.756368+0000","last_clean":"2024-09-17T16:38:54.756368+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:38:54.756368+0000","last_undegraded":"2024-09-17T16:38:54.756368+0000","last_fullsized":"2024-09-17T16:38:54.756368+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:43.251490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":27,"acting":27,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592228,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93152924,"statfs":{"total":95995035648,"available":95388594176,"internally_reserved":0,"allocated":1413120,"data_stored":907334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":604830148},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":224440,"kb_used_data":1380,"kb_used_omap":15,"kb_used_meta":223024,"kb_avail":93520712,"statfs":{"total":95995035648,"available":95765209088,"internally_reserved":0,"allocated":1413120,"data_stored":907354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":228377148},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743787,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758684,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986468,"statfs":{"total":95995035648,"available":95218143232,"internally_reserved":0,"allocated":1449984,"data_stored":931526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":775420352},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005424,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542592,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93202560,"statfs":{"total":95995035648,"available":95439421440,"internally_reserved":0,"allocated":1945600,"data_stored":1436282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":721956,"kb_used_data":1380,"kb_used_omap":41,"kb_used_meta":720342,"kb_avail":93023196,"statfs":{"total":95995035648,"available":95255752704,"internally_reserved":0,"allocated":1413120,"data_stored":907354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":737630515},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546780,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93198372,"statfs":{"total":95995035648,"available":95435132928,"internally_reserved":0,"allocated":1970176,"data_stored":1452338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967642,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110372,"kb_used_data":1380,"kb_used_omap":14,"kb_used_meta":108785,"kb_avail":93634780,"statfs":{"total":95995035648,"available":95882014720,"internally_reserved":0,"allocated":1413120,"data_stored":907354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":111396469},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":637164,"kb_used_data":1900,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93107988,"statfs":{"total":95995035648,"available":95342579712,"internally_reserved":0,"allocated":1945600,"data_stored":1436282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150087,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":881848,"kb_used_data":128740,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92863304,"statfs":{"total":95995035648,"available":95092023296,"internally_reserved":0,"allocated":131829760,"data_stored":131323994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":771160519},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411726,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":624020,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121132,"statfs":{"total":95995035648,"available":95356039168,"internally_reserved":0,"allocated":1437696,"data_stored":923410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575251,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":496248,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":93248904,"statfs":{"total":95995035648,"available":95486877696,"internally_reserved":0,"allocated":1413120,"data_stored":907354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738776,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917200,"kb_used_data":123836,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92827952,"statfs":{"total":95995035648,"available":95055822848,"internally_reserved":0,"allocated":126808064,"data_stored":126293778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:38:58.779 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:38:58.797 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:58.808 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:58.815 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:58.835 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:58.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:58.839 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:38:58.923 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:58 smithi017 ceph-mon[27571]: from='client.18426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:38:58.924 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:58 smithi017 ceph-mon[27571]: pgmap v1905: 17 pgs: 10 active+clean, 6 creating+peering, 1 unknown; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:58.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:38:58.959 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:58.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:38:59.061 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:58 smithi071 ceph-mon[30002]: from='client.18426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:38:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:58 smithi071 ceph-mon[30002]: pgmap v1905: 17 pgs: 10 active+clean, 6 creating+peering, 1 unknown; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:59.088 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:59.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:59.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:58 smithi173 ceph-mon[30530]: from='client.18426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:38:59.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:58 smithi173 ceph-mon[30530]: pgmap v1905: 17 pgs: 10 active+clean, 6 creating+peering, 1 unknown; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:38:59.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:38:59.216 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:59.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:38:59.318 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:59.345 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:59.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:38:59.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:38:59.471 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:59.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:38:59.535 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:38:59.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:38:59.598 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:59.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:38:59.700 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:38:59.700 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:38:59.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:38:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:59.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:38:59.728 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:38:59.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:38:59.853 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:38:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:59.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:38:59.955 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:38:59.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:38:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1817193179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:38:59.981 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:38:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:38:59.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:00.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:38:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1817193179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:00.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:00.108 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:00.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:00.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:38:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1817193179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:00.210 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:00.235 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:00.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:00.324 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:39:00.324 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:39:00.324 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:39:00.324 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:39:00.325 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:39:00.325 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:39:00.325 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:39:00.325 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:39:00.325 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:39:00.326 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:39:00.326 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:39:00.326 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:39:00.326 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:39:00.327 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:39:00.327 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:39:00.327 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:39:00.327 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:39:00.328 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:39:00.328 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:39:00.328 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:39:00.328 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:39:00.328 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:39:00.328 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:39:00.329 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:39:00.337 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:00.359 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:00.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:00.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:00.461 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:00.487 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:00.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:00.589 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:00.611 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:00.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:00.714 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:00.736 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:00.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:00.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:00.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:00.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:00.967 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:00.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:00 smithi017 ceph-mon[27571]: pgmap v1906: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:39:00.993 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:00.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:00.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:01.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:00 smithi071 ceph-mon[30002]: pgmap v1906: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:39:01.095 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:01.121 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:01.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:00 smithi173 ceph-mon[30530]: pgmap v1906: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:39:01.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:01.246 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:01.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:01.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:39:01.348 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:01.373 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:01.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:01.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:01.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:01.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:01.604 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:01.629 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:01.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:01.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:01.754 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:01.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:01.761 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:01.779 INFO:teuthology.orchestra.run.smithi017.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-09-17T16:39:01.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:01.879 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:01.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:01.981 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:01.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2316802744' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:39:02.004 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:02.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:02.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2316802744' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:39:02.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:02.128 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:02.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:02.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2316802744' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-17T16:39:02.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:02.256 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:02.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:02.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:02.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:02.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:02.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:02.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:02.486 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:02.509 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:02.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:02.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:02.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:02.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:02.740 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:02.747 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:39:02.748 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:39:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:02.765 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:02.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:02.867 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:02.894 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:02.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:02.996 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:03.019 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:03.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2316802744' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:39:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:02 smithi071 ceph-mon[30002]: osdmap e862: 12 total, 12 up, 12 in 2024-09-17T16:39:03.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:02 smithi071 ceph-mon[30002]: pgmap v1908: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:39:03.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2316802744' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:39:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:02 smithi173 ceph-mon[30530]: osdmap e862: 12 total, 12 up, 12 in 2024-09-17T16:39:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:02 smithi173 ceph-mon[30530]: pgmap v1908: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:39:03.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:03.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:03.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2316802744' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-17T16:39:03.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:02 smithi017 ceph-mon[27571]: osdmap e862: 12 total, 12 up, 12 in 2024-09-17T16:39:03.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:02 smithi017 ceph-mon[27571]: pgmap v1908: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:39:03.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:03.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:03.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:03.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:03.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:03.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:03.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:03.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:03.534 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:03.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:03.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:03.652 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1908,"stamp":"2024-09-17T16:39:02.136246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":11,"kb":1124941824,"kb_used":7034108,"kb_used_data":249252,"kb_used_omap":85,"kb_used_meta":6784490,"kb_avail":1117907716,"statfs":{"total":1151940427776,"available":1144737501184,"internally_reserved":0,"allocated":255234048,"data_stored":248961058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87602,"internal_metadata":6947318222},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.382577"},"pg_stats":[{"pgid":"9.f","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:56.828547+0000","last_change":"2024-09-17T16:38:54.766586+0000","last_active":"2024-09-17T16:38:56.828547+0000","last_peered":"2024-09-17T16:38:56.828547+0000","last_clean":"2024-09-17T16:38:56.828547+0000","last_became_active":"2024-09-17T16:38:54.765274+0000","last_became_peered":"2024-09-17T16:38:54.765274+0000","last_unstale":"2024-09-17T16:38:56.828547+0000","last_undegraded":"2024-09-17T16:38:56.828547+0000","last_fullsized":"2024-09-17T16:38:56.828547+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:19.855744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:56.862941+0000","last_change":"2024-09-17T16:38:54.765568+0000","last_active":"2024-09-17T16:38:56.862941+0000","last_peered":"2024-09-17T16:38:56.862941+0000","last_clean":"2024-09-17T16:38:56.862941+0000","last_became_active":"2024-09-17T16:38:54.765252+0000","last_became_peered":"2024-09-17T16:38:54.765252+0000","last_unstale":"2024-09-17T16:38:56.862941+0000","last_undegraded":"2024-09-17T16:38:56.862941+0000","last_fullsized":"2024-09-17T16:38:56.862941+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:24.366873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:57.364523+0000","last_change":"2024-09-17T16:38:54.759655+0000","last_active":"2024-09-17T16:38:57.364523+0000","last_peered":"2024-09-17T16:38:57.364523+0000","last_clean":"2024-09-17T16:38:57.364523+0000","last_became_active":"2024-09-17T16:38:54.759320+0000","last_became_peered":"2024-09-17T16:38:54.759320+0000","last_unstale":"2024-09-17T16:38:57.364523+0000","last_undegraded":"2024-09-17T16:38:57.364523+0000","last_fullsized":"2024-09-17T16:38:57.364523+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:01.586787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:57.364388+0000","last_change":"2024-09-17T16:38:54.765585+0000","last_active":"2024-09-17T16:38:57.364388+0000","last_peered":"2024-09-17T16:38:57.364388+0000","last_clean":"2024-09-17T16:38:57.364388+0000","last_became_active":"2024-09-17T16:38:54.765180+0000","last_became_peered":"2024-09-17T16:38:54.765180+0000","last_unstale":"2024-09-17T16:38:57.364388+0000","last_undegraded":"2024-09-17T16:38:57.364388+0000","last_fullsized":"2024-09-17T16:38:57.364388+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:58.754071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:57.657997+0000","last_change":"2024-09-17T16:38:54.766769+0000","last_active":"2024-09-17T16:38:57.657997+0000","last_peered":"2024-09-17T16:38:57.657997+0000","last_clean":"2024-09-17T16:38:57.657997+0000","last_became_active":"2024-09-17T16:38:54.766429+0000","last_became_peered":"2024-09-17T16:38:54.766429+0000","last_unstale":"2024-09-17T16:38:57.657997+0000","last_undegraded":"2024-09-17T16:38:57.657997+0000","last_fullsized":"2024-09-17T16:38:57.657997+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:00.919134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:56.828371+0000","last_change":"2024-09-17T16:38:54.760350+0000","last_active":"2024-09-17T16:38:56.828371+0000","last_peered":"2024-09-17T16:38:56.828371+0000","last_clean":"2024-09-17T16:38:56.828371+0000","last_became_active":"2024-09-17T16:38:54.760031+0000","last_became_peered":"2024-09-17T16:38:54.760031+0000","last_unstale":"2024-09-17T16:38:56.828371+0000","last_undegraded":"2024-09-17T16:38:56.828371+0000","last_fullsized":"2024-09-17T16:38:56.828371+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:04.918729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:56.862918+0000","last_change":"2024-09-17T16:38:54.764119+0000","last_active":"2024-09-17T16:38:56.862918+0000","last_peered":"2024-09-17T16:38:56.862918+0000","last_clean":"2024-09-17T16:38:56.862918+0000","last_became_active":"2024-09-17T16:38:54.763749+0000","last_became_peered":"2024-09-17T16:38:54.763749+0000","last_unstale":"2024-09-17T16:38:56.862918+0000","last_undegraded":"2024-09-17T16:38:56.862918+0000","last_fullsized":"2024-09-17T16:38:56.862918+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:55.447055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:56.862930+0000","last_change":"2024-09-17T16:38:54.765473+0000","last_active":"2024-09-17T16:38:56.862930+0000","last_peered":"2024-09-17T16:38:56.862930+0000","last_clean":"2024-09-17T16:38:56.862930+0000","last_became_active":"2024-09-17T16:38:54.765097+0000","last_became_peered":"2024-09-17T16:38:54.765097+0000","last_unstale":"2024-09-17T16:38:56.862930+0000","last_undegraded":"2024-09-17T16:38:56.862930+0000","last_fullsized":"2024-09-17T16:38:56.862930+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:18.915283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:56.828488+0000","last_change":"2024-09-17T16:38:54.766597+0000","last_active":"2024-09-17T16:38:56.828488+0000","last_peered":"2024-09-17T16:38:56.828488+0000","last_clean":"2024-09-17T16:38:56.828488+0000","last_became_active":"2024-09-17T16:38:54.765661+0000","last_became_peered":"2024-09-17T16:38:54.765661+0000","last_unstale":"2024-09-17T16:38:56.828488+0000","last_undegraded":"2024-09-17T16:38:56.828488+0000","last_fullsized":"2024-09-17T16:38:56.828488+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:17.824425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:56.828536+0000","last_change":"2024-09-17T16:38:54.763911+0000","last_active":"2024-09-17T16:38:56.828536+0000","last_peered":"2024-09-17T16:38:56.828536+0000","last_clean":"2024-09-17T16:38:56.828536+0000","last_became_active":"2024-09-17T16:38:54.763476+0000","last_became_peered":"2024-09-17T16:38:54.763476+0000","last_unstale":"2024-09-17T16:38:56.828536+0000","last_undegraded":"2024-09-17T16:38:56.828536+0000","last_fullsized":"2024-09-17T16:38:56.828536+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:12.320477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:57.363883+0000","last_change":"2024-09-17T16:38:54.763053+0000","last_active":"2024-09-17T16:38:57.363883+0000","last_peered":"2024-09-17T16:38:57.363883+0000","last_clean":"2024-09-17T16:38:57.363883+0000","last_became_active":"2024-09-17T16:38:54.762670+0000","last_became_peered":"2024-09-17T16:38:54.762670+0000","last_unstale":"2024-09-17T16:38:57.363883+0000","last_undegraded":"2024-09-17T16:38:57.363883+0000","last_fullsized":"2024-09-17T16:38:57.363883+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:23.914965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:56.828446+0000","last_change":"2024-09-17T16:38:54.765593+0000","last_active":"2024-09-17T16:38:56.828446+0000","last_peered":"2024-09-17T16:38:56.828446+0000","last_clean":"2024-09-17T16:38:56.828446+0000","last_became_active":"2024-09-17T16:38:54.765189+0000","last_became_peered":"2024-09-17T16:38:54.765189+0000","last_unstale":"2024-09-17T16:38:56.828446+0000","last_undegraded":"2024-09-17T16:38:56.828446+0000","last_fullsized":"2024-09-17T16:38:56.828446+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:14.214467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:57.363905+0000","last_change":"2024-09-17T16:38:54.765533+0000","last_active":"2024-09-17T16:38:57.363905+0000","last_peered":"2024-09-17T16:38:57.363905+0000","last_clean":"2024-09-17T16:38:57.363905+0000","last_became_active":"2024-09-17T16:38:54.765141+0000","last_became_peered":"2024-09-17T16:38:54.765141+0000","last_unstale":"2024-09-17T16:38:57.363905+0000","last_undegraded":"2024-09-17T16:38:57.363905+0000","last_fullsized":"2024-09-17T16:38:57.363905+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:23.105543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:56.769528+0000","last_change":"2024-09-17T16:38:54.765566+0000","last_active":"2024-09-17T16:38:56.769528+0000","last_peered":"2024-09-17T16:38:56.769528+0000","last_clean":"2024-09-17T16:38:56.769528+0000","last_became_active":"2024-09-17T16:38:54.765272+0000","last_became_peered":"2024-09-17T16:38:54.765272+0000","last_unstale":"2024-09-17T16:38:56.769528+0000","last_undegraded":"2024-09-17T16:38:56.769528+0000","last_fullsized":"2024-09-17T16:38:56.769528+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:08.200736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,6],"acting":[9,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:57.364004+0000","last_change":"2024-09-17T16:38:54.757271+0000","last_active":"2024-09-17T16:38:57.364004+0000","last_peered":"2024-09-17T16:38:57.364004+0000","last_clean":"2024-09-17T16:38:57.364004+0000","last_became_active":"2024-09-17T16:38:54.756978+0000","last_became_peered":"2024-09-17T16:38:54.756978+0000","last_unstale":"2024-09-17T16:38:57.364004+0000","last_undegraded":"2024-09-17T16:38:57.364004+0000","last_fullsized":"2024-09-17T16:38:57.364004+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:12.133372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":16,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:56.769538+0000","last_change":"2024-09-17T16:38:54.768093+0000","last_active":"2024-09-17T16:38:56.769538+0000","last_peered":"2024-09-17T16:38:56.769538+0000","last_clean":"2024-09-17T16:38:56.769538+0000","last_became_active":"2024-09-17T16:38:54.767749+0000","last_became_peered":"2024-09-17T16:38:54.767749+0000","last_unstale":"2024-09-17T16:38:56.769538+0000","last_undegraded":"2024-09-17T16:38:56.769538+0000","last_fullsized":"2024-09-17T16:38:56.769538+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:03.514492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":494,"reported_epoch":861,"state":"active+clean","last_fresh":"2024-09-17T16:38:57.364373+0000","last_change":"2024-09-17T16:33:59.364895+0000","last_active":"2024-09-17T16:38:57.364373+0000","last_peered":"2024-09-17T16:38:57.364373+0000","last_clean":"2024-09-17T16:38:57.364373+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:38:57.364373+0000","last_undegraded":"2024-09-17T16:38:57.364373+0000","last_fullsized":"2024-09-17T16:38:57.364373+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:43.251490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285923,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592108,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93153044,"statfs":{"total":95995035648,"available":95388717056,"internally_reserved":0,"allocated":1462272,"data_stored":939295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":604830148},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131165,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":224468,"kb_used_data":1404,"kb_used_omap":15,"kb_used_meta":223024,"kb_avail":93520684,"statfs":{"total":95995035648,"available":95765180416,"internally_reserved":0,"allocated":1437696,"data_stored":923410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":228377148},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743788,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758700,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986452,"statfs":{"total":95995035648,"available":95218126848,"internally_reserved":0,"allocated":1462272,"data_stored":939315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":775420352},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005425,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542644,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93202508,"statfs":{"total":95995035648,"available":95439368192,"internally_reserved":0,"allocated":1994752,"data_stored":1468243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963901,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":721832,"kb_used_data":1428,"kb_used_omap":41,"kb_used_meta":720342,"kb_avail":93023320,"statfs":{"total":95995035648,"available":95255879680,"internally_reserved":0,"allocated":1462272,"data_stored":939315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":737630515},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495999,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546812,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93198340,"statfs":{"total":95995035648,"available":95435100160,"internally_reserved":0,"allocated":1994752,"data_stored":1468243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967643,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110256,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":108785,"kb_avail":93634896,"statfs":{"total":95995035648,"available":95882133504,"internally_reserved":0,"allocated":1462272,"data_stored":939315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":111396469},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986566,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":637044,"kb_used_data":1948,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93108108,"statfs":{"total":95995035648,"available":95342702592,"internally_reserved":0,"allocated":1994752,"data_stored":1468243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150088,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":872284,"kb_used_data":119176,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92872868,"statfs":{"total":95995035648,"available":95101816832,"internally_reserved":0,"allocated":122036224,"data_stored":121517766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":771160519},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411727,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":624040,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121112,"statfs":{"total":95995035648,"available":95356018688,"internally_reserved":0,"allocated":1462272,"data_stored":939315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575252,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":496300,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":93248852,"statfs":{"total":95995035648,"available":95486824448,"internally_reserved":0,"allocated":1462272,"data_stored":939315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738777,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907620,"kb_used_data":114260,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92837532,"statfs":{"total":95995035648,"available":95065632768,"internally_reserved":0,"allocated":117002240,"data_stored":116479283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:39:03.653 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T16:39:03.654 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:39:03.654 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 2, 3, 6, 7, 8, 9, 10, 11, 1, 5, 4] 2024-09-17T16:39:03.654 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd out 3 2024-09-17T16:39:03.662 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:03.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:03.701 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:03.707 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:03.754 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:39:03.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:03.819 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:03.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:03.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:03 smithi017 ceph-mon[27571]: from='client.18444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:39:03.870 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:03 smithi017 ceph-mon[27571]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:39:03.909 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-09-17T16:39:03.912 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-17T16:39:03.912 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-17T16:39:03.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:03.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:03.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:03.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-09-17T16:39:03.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-17T16:39:03.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_150363 2024-09-17T16:39:03.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:39:03.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-09-17T16:39:04.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:03 smithi071 ceph-mon[30002]: from='client.18444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:39:04.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:03 smithi071 ceph-mon[30002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:39:04.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:04.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:03 smithi173 ceph-mon[30530]: from='client.18444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:39:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:03 smithi173 ceph-mon[30530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-17T16:39:04.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:04.212 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:04.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:04.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:04.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:04.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:04.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:04.440 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:04.468 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:04.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:04.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:04.600 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:04.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:04.702 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:04.724 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:04.724 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:04.732 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:04.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:04.757 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:39:04.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:04.858 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:04.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:04.961 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:04.985 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:04.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:04.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 2378 2362 147.62 147.625 0.00150686 0.00672358 2024-09-17T16:39:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:04 smithi017 ceph-mon[27571]: pgmap v1909: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:39:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3540218324' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:04 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:39:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:04 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:39:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3540218324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:04.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:04 smithi017 ceph-mon[27571]: osdmap e863: 12 total, 12 up, 12 in 2024-09-17T16:39:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:04 smithi071 ceph-mon[30002]: pgmap v1909: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:39:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3540218324' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:04 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:39:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:04 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:39:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3540218324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:05.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:04 smithi071 ceph-mon[30002]: osdmap e863: 12 total, 12 up, 12 in 2024-09-17T16:39:05.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:05.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:05.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:05.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:04 smithi173 ceph-mon[30530]: pgmap v1909: 17 pgs: 17 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:39:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3540218324' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:04 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:39:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:04 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:39:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3540218324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:04 smithi173 ceph-mon[30530]: osdmap e863: 12 total, 12 up, 12 in 2024-09-17T16:39:05.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:05.239 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:05.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:05.280 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:05.341 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:05.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:05.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:05.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:05.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:05.491 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:05.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:05.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:05.616 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:05.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:05.718 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:05.743 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:05.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:05.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:05.845 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:05.868 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:05.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:05.900 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:05.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:39:05.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:05.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 3853 3837 119.899 92.1875 0.00169194 0.00828389 2024-09-17T16:39:05.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:05.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:06.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:06.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:06.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:06.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:06.247 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:06.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:06.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:06.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:06.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:06.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:06.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:06.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:06.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:06.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:06.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:06.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:06.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:06.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:06.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:06.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:06.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:06.889 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:06.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:06.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:39:06.898 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:39:06.926 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:39:06.926 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:39:06.952 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:39:06.953 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:39:06.979 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:39:06.979 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:39:06.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 15 5563 5548 115.576 106.938 0.00593556 0.00863212 2024-09-17T16:39:06.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:07.005 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:39:07.006 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:39:07.015 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:07.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:07.030 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:39:07.030 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:39:07.063 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:39:07.065 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:39:07.087 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:39:07.087 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:39:07.112 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:39:07.113 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:39:07.116 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:07.142 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:07.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:07.144 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:39:07.144 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:39:07.168 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:39:07.168 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:39:07.193 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:39:07.194 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:39:07.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:07.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:07.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:07.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:07.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:07.401 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:07.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:07 smithi017 ceph-mon[27571]: pgmap v1911: 17 pgs: 17 active+clean; 52 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s wr, 103 op/s 2024-09-17T16:39:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:07 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:39:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:07 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:39:07.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4249790040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:07.502 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:07.528 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:07.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:07 smithi071 ceph-mon[30002]: pgmap v1911: 17 pgs: 17 active+clean; 52 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s wr, 103 op/s 2024-09-17T16:39:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:07 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:39:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:07 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:39:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4249790040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:07.630 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:07 smithi173 ceph-mon[30530]: pgmap v1911: 17 pgs: 17 active+clean; 52 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s wr, 103 op/s 2024-09-17T16:39:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:07 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:39:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:07 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:39:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4249790040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:07.656 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:07.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:07.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:07.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:07.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:07.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:07.913 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:07.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:07.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 7629 7613 118.945 129.062 0.00267607 0.00839202 2024-09-17T16:39:08.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:08.043 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:08.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:08.145 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:08.171 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:08.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:08.202 INFO:teuthology.orchestra.run.smithi017.stderr:marked out osd.3. 2024-09-17T16:39:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:08 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:39:08.062+0000 7f086a07d640 -1 mon.a@0(leader).osd e864 definitely_dead 0 2024-09-17T16:39:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:08 smithi017 ceph-mon[27571]: osdmap e864: 12 total, 12 up, 12 in 2024-09-17T16:39:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3028504679' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-17T16:39:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:08 smithi017 ceph-mon[27571]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-17T16:39:08.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:08.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:08.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:08.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:39:08.396 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:08.417 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:08.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:08.519 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:08.540 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:08.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:08.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:08 smithi071 ceph-mon[30002]: osdmap e864: 12 total, 12 up, 12 in 2024-09-17T16:39:08.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3028504679' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-17T16:39:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:08 smithi071 ceph-mon[30002]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-17T16:39:08.643 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:08.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:08 smithi173 ceph-mon[30530]: osdmap e864: 12 total, 12 up, 12 in 2024-09-17T16:39:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3028504679' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-17T16:39:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:08 smithi173 ceph-mon[30530]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-17T16:39:08.664 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:08.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:08.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:08.750 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:08.751 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:08.766 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:08.790 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:08.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:08.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:08.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:08.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:08.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 8138 8122 101.516 31.8125 0.00252828 0.00861775 2024-09-17T16:39:09.017 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:09.039 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:09.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:09.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:09.163 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:09.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:09.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:09.289 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:09.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:09.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:09.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:09.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:09.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:09.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:09 smithi017 ceph-mon[27571]: pgmap v1913: 17 pgs: 17 active+clean; 242 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 606 op/s 2024-09-17T16:39:09.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3028504679' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-17T16:39:09.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:09 smithi017 ceph-mon[27571]: osdmap e865: 12 total, 12 up, 11 in 2024-09-17T16:39:09.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:09 smithi071 ceph-mon[30002]: pgmap v1913: 17 pgs: 17 active+clean; 242 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 606 op/s 2024-09-17T16:39:09.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3028504679' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-17T16:39:09.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:09 smithi071 ceph-mon[30002]: osdmap e865: 12 total, 12 up, 11 in 2024-09-17T16:39:09.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:09.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:09.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:09.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:09.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:09 smithi173 ceph-mon[30530]: pgmap v1913: 17 pgs: 17 active+clean; 242 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 606 op/s 2024-09-17T16:39:09.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3028504679' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-17T16:39:09.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:09 smithi173 ceph-mon[30530]: osdmap e865: 12 total, 12 up, 11 in 2024-09-17T16:39:09.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:09.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:09.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:09.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:09.791 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:09.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:09.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:09.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:09.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:09.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:09.983 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:39:09.983 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:39:09.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:39:09.983 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:39:09.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:95587 7.3 7:c3123e23:::benchmark_data_smithi173_116396_object95586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e671)", 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:31:09.928298+0000", 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 480.01121219800001, 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.84987048899999995, 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 95587 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.984 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:09.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:09.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928298+0000", 2024-09-17T16:39:09.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:09.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:09.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928298+0000", 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928300+0000", 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8419999999999999e-06 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928356+0000", 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6459999999999998e-05 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:09.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928372+0000", 2024-09-17T16:39:09.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6030000000000001e-05 2024-09-17T16:39:09.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:09.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928380+0000", 2024-09-17T16:39:09.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3010000000000001e-06 2024-09-17T16:39:09.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928498+0000", 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118095 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928616+0000", 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011820999999999999 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 2,4", 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.928769+0000", 2024-09-17T16:39:09.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015275799999999999 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.932423+0000", 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036541949999999998 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:09.933233+0000", 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00081015899999999997 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.778093+0000", 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.84486028400000002 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.778136+0000", 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2456000000000001e-05 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:31:10.778168+0000", 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2699e-05 2024-09-17T16:39:09.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:09.991 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:09.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:09.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.991 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.18 741/739 [PushReplyOp(8:1f8b3d36:::benchmark_data_smithi173_130566_object2490:head)])", 2024-09-17T16:39:09.991 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:27.809184+0000", 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 282.130326588, 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58530468800000002, 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.809184+0000", 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.809184+0000", 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.809184+0000", 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6599999999999995e-07 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.809189+0000", 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7849999999999999e-06 2024-09-17T16:39:09.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.394061+0000", 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.58487161399999998 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.394069+0000", 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7870000000000002e-06 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:09.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.394126+0000", 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.702e-05 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.394488+0000", 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00036271600000000001 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:118362 8.a e803/728 8:518e453f:::benchmark_data_smithi173_130566_object23387:head v 803'7586, mlcod=803'7583)", 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:59.324853+0000", 2024-09-17T16:39:09.995 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 130.61465685900001, 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33420323000000002, 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.324853+0000", 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.324853+0000", 2024-09-17T16:39:09.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.324856+0000", 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9639999999999999e-06 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.324861+0000", 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1240000000000004e-06 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:09.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.324889+0000", 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7234e-05 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.324895+0000", 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4799999999999998e-06 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.324940+0000", 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4787000000000003e-05 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.324955+0000", 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4749000000000001e-05 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.659014+0000", 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33405911100000002 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.659057+0000", 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2781000000000003e-05 2024-09-17T16:39:09.999 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:118365 8.8 e803/792 8:15f8fc15:::benchmark_data_smithi173_130566_object23390:head v 803'3906, mlcod=750'918)", 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:59.327003+0000", 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 130.61250763800001, 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33686085500000001, 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.327003+0000", 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.327003+0000", 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.327004+0000", 2024-09-17T16:39:10.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0610000000000001e-06 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.327011+0000", 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3799999999999996e-06 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.327048+0000", 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6817999999999999e-05 2024-09-17T16:39:10.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.327055+0000", 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0849999999999999e-06 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.659033+0000", 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.331978043 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.659062+0000", 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9467000000000001e-05 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.663819+0000", 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047562020000000002 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.663863+0000", 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4799000000000003e-05 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.004 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:118368 8.c e803/728 8:30ee9f85:::benchmark_data_smithi173_130566_object23393:head v 803'7117, mlcod=803'7115)", 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:59.330554+0000", 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 130.60895587300001, 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33561433499999999, 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.330554+0000", 2024-09-17T16:39:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.330554+0000", 2024-09-17T16:39:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.330556+0000", 2024-09-17T16:39:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.429e-06 2024-09-17T16:39:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.330564+0000", 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.658989+0000", 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32842557900000002 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.659000+0000", 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1131999999999999e-05 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.659057+0000", 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7083000000000002e-05 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.659079+0000", 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1172999999999999e-05 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.666125+0000", 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070467180000000004 2024-09-17T16:39:10.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:59.666169+0000", 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.333e-05 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:138881 8.13 8:cc7a69da:::benchmark_data_smithi173_130566_object43906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-09-17T16:39:10.010 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:25.399367+0000", 2024-09-17T16:39:10.011 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 104.540143371, 2024-09-17T16:39:10.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22097223399999999, 2024-09-17T16:39:10.011 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.011 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.011 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.011 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:39:10.011 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:39:10.011 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 138881 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.399367+0000", 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.399367+0000", 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.399368+0000", 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.437e-06 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.399374+0000", 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5539999999999997e-06 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.399385+0000", 2024-09-17T16:39:10.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0779e-05 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.399391+0000", 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5400000000000001e-06 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.399453+0000", 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2402000000000003e-05 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.399607+0000", 2024-09-17T16:39:10.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015393199999999999 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.399726+0000", 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011852 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.402338+0000", 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026118859999999999 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.402607+0000", 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026880600000000001 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.620220+0000", 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.217613629 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.620282+0000", 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1933000000000005e-05 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:25.620339+0000", 2024-09-17T16:39:10.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6815999999999999e-05 2024-09-17T16:39:10.017 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.017 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.017 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.017 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:150142 8.0 e818/767 8:07bba7a5:::benchmark_data_smithi173_130566_object55167:head v 818'4950, mlcod=818'4947)", 2024-09-17T16:39:10.018 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:38.746433+0000", 2024-09-17T16:39:10.018 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 91.193077393999999, 2024-09-17T16:39:10.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.16760768600000001, 2024-09-17T16:39:10.019 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.019 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.019 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.019 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:38.746433+0000", 2024-09-17T16:39:10.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:38.746433+0000", 2024-09-17T16:39:10.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:38.746434+0000", 2024-09-17T16:39:10.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.195e-06 2024-09-17T16:39:10.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:38.746441+0000", 2024-09-17T16:39:10.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9800000000000001e-06 2024-09-17T16:39:10.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:38.911002+0000", 2024-09-17T16:39:10.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.16456120799999999 2024-09-17T16:39:10.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:38.911008+0000", 2024-09-17T16:39:10.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8139999999999997e-06 2024-09-17T16:39:10.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:38.911062+0000", 2024-09-17T16:39:10.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4045000000000002e-05 2024-09-17T16:39:10.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:38.911084+0000", 2024-09-17T16:39:10.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2014e-05 2024-09-17T16:39:10.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:38.913981+0000", 2024-09-17T16:39:10.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028966370000000001 2024-09-17T16:39:10.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:38.914040+0000", 2024-09-17T16:39:10.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9793000000000002e-05 2024-09-17T16:39:10.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.028 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.028 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.029 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:151701 8.9 e820/792 8:96192a39:::benchmark_data_smithi173_130566_object56726:head v 820'5177, mlcod=763'2894)", 2024-09-17T16:39:10.029 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:41.280748+0000", 2024-09-17T16:39:10.029 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 88.658762369000002, 2024-09-17T16:39:10.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.230989106, 2024-09-17T16:39:10.030 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.030 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.030 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.030 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:41.280748+0000", 2024-09-17T16:39:10.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:41.280748+0000", 2024-09-17T16:39:10.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:41.280749+0000", 2024-09-17T16:39:10.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4200000000000005e-07 2024-09-17T16:39:10.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:41.280755+0000", 2024-09-17T16:39:10.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6760000000000001e-06 2024-09-17T16:39:10.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:41.508885+0000", 2024-09-17T16:39:10.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22812954999999999 2024-09-17T16:39:10.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:41.508893+0000", 2024-09-17T16:39:10.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2179999999999999e-06 2024-09-17T16:39:10.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:41.508965+0000", 2024-09-17T16:39:10.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1631000000000006e-05 2024-09-17T16:39:10.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:41.508997+0000", 2024-09-17T16:39:10.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1795999999999998e-05 2024-09-17T16:39:10.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:41.511687+0000", 2024-09-17T16:39:10.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026904070000000001 2024-09-17T16:39:10.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:41.511737+0000", 2024-09-17T16:39:10.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9985999999999999e-05 2024-09-17T16:39:10.039 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.039 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.039 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.039 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:152838 8.13 8:ce2e800f:::benchmark_data_smithi173_130566_object57863:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-09-17T16:39:10.040 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:42.922369+0000", 2024-09-17T16:39:10.040 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 87.017141146, 2024-09-17T16:39:10.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.178641841, 2024-09-17T16:39:10.040 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.040 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.040 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.041 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:39:10.041 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:39:10.041 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:10.041 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 152838 2024-09-17T16:39:10.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.042 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.922369+0000", 2024-09-17T16:39:10.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.922369+0000", 2024-09-17T16:39:10.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.922371+0000", 2024-09-17T16:39:10.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.841e-06 2024-09-17T16:39:10.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.922377+0000", 2024-09-17T16:39:10.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6089999999999997e-06 2024-09-17T16:39:10.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.922398+0000", 2024-09-17T16:39:10.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0754000000000002e-05 2024-09-17T16:39:10.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.922407+0000", 2024-09-17T16:39:10.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7700000000000007e-06 2024-09-17T16:39:10.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.922468+0000", 2024-09-17T16:39:10.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1243999999999998e-05 2024-09-17T16:39:10.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.922601+0000", 2024-09-17T16:39:10.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000132955 2024-09-17T16:39:10.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:39:10.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.922734+0000", 2024-09-17T16:39:10.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013286600000000001 2024-09-17T16:39:10.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.925375+0000", 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026409350000000001 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.927770+0000", 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0023946810000000001 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:43.100888+0000", 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.173118787 2024-09-17T16:39:10.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:43.100956+0000", 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7418999999999999e-05 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:43.101011+0000", 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4979999999999999e-05 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.051 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:152838 8.13 e820/742 ondisk, result = 0)", 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:42.926280+0000", 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 87.013230366000002, 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.17473850399999999, 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.926280+0000", 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.926280+0000", 2024-09-17T16:39:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.926280+0000", 2024-09-17T16:39:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.15e-07 2024-09-17T16:39:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:42.926283+0000", 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8600000000000001e-06 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:43.100804+0000", 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.17452113399999999 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:43.100810+0000", 2024-09-17T16:39:10.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4920000000000001e-06 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:43.100870+0000", 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0393000000000003e-05 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:43.100881+0000", 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0429e-05 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:43.101018+0000", 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013778099999999999 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:154655 8.13 8:ccebb91e:::benchmark_data_smithi173_130566_object59680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e821)", 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:45.348138+0000", 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 84.591372483000001, 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23857392999999999, 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.056 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 154655 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.348138+0000", 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.348138+0000", 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.348140+0000", 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9420000000000002e-06 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.348147+0000", 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8280000000000003e-06 2024-09-17T16:39:10.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.348167+0000", 2024-09-17T16:39:10.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9842000000000001e-05 2024-09-17T16:39:10.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.348176+0000", 2024-09-17T16:39:10.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0480000000000004e-06 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.348235+0000", 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8533999999999999e-05 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.348334+0000", 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9205999999999999e-05 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.348454+0000", 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000119961 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.351081+0000", 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026267949999999999 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.351380+0000", 2024-09-17T16:39:10.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029877799999999998 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.586586+0000", 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.235206632 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.586659+0000", 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2495000000000003e-05 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.586712+0000", 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2868999999999998e-05 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:154655 8.13 e821/742 ondisk, result = 0)", 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:45.351335+0000", 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 84.588175480000004, 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23538320600000001, 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:39:10.063 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.351335+0000", 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.351335+0000", 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.351336+0000", 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2900000000000002e-07 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.351341+0000", 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.305e-06 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.585828+0000", 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23448681199999999 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.585834+0000", 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2729999999999999e-06 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.586569+0000", 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000734981 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.586580+0000", 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1525999999999999e-05 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.586718+0000", 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013747999999999999 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.067 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:154710 8.13 8:cffbdfeb:::benchmark_data_smithi173_130566_object59735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e821)", 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:45.429363+0000", 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 84.510146860000006, 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.15843628300000001, 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 154710 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.429363+0000", 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.429363+0000", 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.429365+0000", 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.114e-06 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.429371+0000", 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4160000000000003e-06 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.429385+0000", 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4472e-05 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.429394+0000", 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5820000000000002e-06 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.429463+0000", 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9413999999999996e-05 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.429627+0000", 2024-09-17T16:39:10.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016338 2024-09-17T16:39:10.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:39:10.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.429754+0000", 2024-09-17T16:39:10.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000127391 2024-09-17T16:39:10.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:39:10.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.431890+0000", 2024-09-17T16:39:10.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0021362880000000001 2024-09-17T16:39:10.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.432244+0000", 2024-09-17T16:39:10.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00035323700000000001 2024-09-17T16:39:10.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.587689+0000", 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.15544533899999999 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.587744+0000", 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5334999999999998e-05 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:45.587800+0000", 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5315000000000001e-05 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:172846 8.13 8:cf26dfc6:::benchmark_data_smithi173_130566_object77871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e835)", 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:10.591072+0000", 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 59.348438283, 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21887442800000001, 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:39:10.075 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 172846 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.591072+0000", 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.591072+0000", 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.591075+0000", 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7319999999999998e-06 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.591083+0000", 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1089999999999998e-06 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.591105+0000", 2024-09-17T16:39:10.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2089000000000001e-05 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.591115+0000", 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7999999999999993e-06 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.591179+0000", 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3900999999999997e-05 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.591330+0000", 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000151068 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.591450+0000", 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012060100000000001 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.594049+0000", 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0025990800000000001 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.594647+0000", 2024-09-17T16:39:10.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00059772999999999998 2024-09-17T16:39:10.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.809837+0000", 2024-09-17T16:39:10.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.215190348 2024-09-17T16:39:10.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.809902+0000", 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5121000000000002e-05 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.809946+0000", 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3849e-05 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:172846 8.13 e835/742 ondisk, result = 0)", 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:10.594477+0000", 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 59.345032848000002, 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21547449199999999, 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.594477+0000", 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.594477+0000", 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.594478+0000", 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.049e-06 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.594487+0000", 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7870000000000007e-06 2024-09-17T16:39:10.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.809768+0000", 2024-09-17T16:39:10.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.21528045500000001 2024-09-17T16:39:10.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.809773+0000", 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3229999999999997e-06 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.809821+0000", 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8013999999999998e-05 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.809831+0000", 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7019999999999996e-06 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:10.809952+0000", 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000121162 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:184725 8.8 e845/842 8:143510d3:::benchmark_data_smithi173_130566_object89750:head v 845'5966, mlcod=845'5963)", 2024-09-17T16:39:10.086 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:27.365283+0000", 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.574227131000001, 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22681535799999999, 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.365283+0000", 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.365283+0000", 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.365284+0000", 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.364e-06 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.365292+0000", 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.106e-06 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.589344+0000", 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.22405183000000001 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.589350+0000", 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1210000000000003e-06 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.589420+0000", 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9573000000000006e-05 2024-09-17T16:39:10.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.589445+0000", 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4579999999999998e-05 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.592050+0000", 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026052979999999998 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.592098+0000", 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8486000000000003e-05 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:184737 8.13 8:ce13143a:::benchmark_data_smithi173_130566_object89762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:27.384026+0000", 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.555483899000002, 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.18323952499999999, 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.091 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 184737 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.384026+0000", 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.384026+0000", 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.384028+0000", 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7e-06 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.384033+0000", 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8400000000000002e-06 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.384045+0000", 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2635e-05 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.384051+0000", 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5559999999999998e-06 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.384115+0000", 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4224e-05 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.384256+0000", 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000140555 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.384363+0000", 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010721 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.387676+0000", 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033133210000000001 2024-09-17T16:39:10.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.387696+0000", 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0097000000000001e-05 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.567166+0000", 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.17947008 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.567224+0000", 2024-09-17T16:39:10.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7027999999999997e-05 2024-09-17T16:39:10.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.567266+0000", 2024-09-17T16:39:10.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2279e-05 2024-09-17T16:39:10.097 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.097 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:184737 8.13 e845/742 ondisk, result = 0)", 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:27.387759+0000", 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.551750773999999, 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.17951149499999999, 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.387759+0000", 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.387759+0000", 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.387760+0000", 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7400000000000003e-07 2024-09-17T16:39:10.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.387768+0000", 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8129999999999996e-06 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.567091+0000", 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.17932356699999999 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.567099+0000", 2024-09-17T16:39:10.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.328e-06 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.567150+0000", 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1620999999999999e-05 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.567158+0000", 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5079999999999999e-06 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.567271+0000", 2024-09-17T16:39:10.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000113084 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:186604 8.13 8:cc2aa770:::benchmark_data_smithi173_130566_object91629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e846)", 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:29.549875+0000", 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 40.389634958999999, 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.157155663, 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:39:10.102 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 186604 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.549875+0000", 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.549875+0000", 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.549877+0000", 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.513e-06 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.549881+0000", 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3200000000000001e-06 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.549892+0000", 2024-09-17T16:39:10.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0611999999999999e-05 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.549897+0000", 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1780000000000002e-06 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.549942+0000", 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4796999999999998e-05 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.550026+0000", 2024-09-17T16:39:10.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4473000000000003e-05 2024-09-17T16:39:10.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:39:10.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.550105+0000", 2024-09-17T16:39:10.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9060000000000005e-05 2024-09-17T16:39:10.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:39:10.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.552238+0000", 2024-09-17T16:39:10.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002132414 2024-09-17T16:39:10.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.553561+0000", 2024-09-17T16:39:10.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0013233660000000001 2024-09-17T16:39:10.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.706922+0000", 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.15336113400000001 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.706981+0000", 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8773999999999997e-05 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:29.707031+0000", 2024-09-17T16:39:10.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0022000000000003e-05 2024-09-17T16:39:10.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.109 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:189927 8.13 8:ccb6c39c:::benchmark_data_smithi173_130566_object94952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e849)", 2024-09-17T16:39:10.109 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:34.116432+0000", 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 35.823078158999998, 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44016559399999999, 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 189927 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.116432+0000", 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.116432+0000", 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.116434+0000", 2024-09-17T16:39:10.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9149999999999999e-06 2024-09-17T16:39:10.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:39:10.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.116440+0000", 2024-09-17T16:39:10.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.891e-06 2024-09-17T16:39:10.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:39:10.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.116455+0000", 2024-09-17T16:39:10.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5248999999999999e-05 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.116463+0000", 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2109999999999998e-06 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.116521+0000", 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7839e-05 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:39:10.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.116656+0000", 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013453099999999999 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.116787+0000", 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131598 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.119368+0000", 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0025812510000000001 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.120130+0000", 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076164199999999998 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.556525+0000", 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.43639502899999999 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.556567+0000", 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1449000000000002e-05 2024-09-17T16:39:10.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:39:10.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:39:10.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:39:10.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:34.556598+0000", 2024-09-17T16:39:10.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0988999999999998e-05 2024-09-17T16:39:10.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.116 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:39:10.116 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:39:10.116 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:39:10.117 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:10.118 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 8196 8180 85.2003 3.625 0.00152257 0.0103899 2024-09-17T16:39:10.140 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:10.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:10.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:10.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:10.266 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:10.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:10.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:10.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:10.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:10 smithi017 ceph-mon[27571]: osdmap e866: 12 total, 12 up, 11 in 2024-09-17T16:39:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:10 smithi017 ceph-mon[27571]: osdmap e867: 12 total, 12 up, 11 in 2024-09-17T16:39:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:10 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:39:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:10 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:39:10.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:10.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:10.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:10.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:10 smithi071 ceph-mon[30002]: osdmap e866: 12 total, 12 up, 11 in 2024-09-17T16:39:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:10 smithi071 ceph-mon[30002]: osdmap e867: 12 total, 12 up, 11 in 2024-09-17T16:39:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:10 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:39:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:10 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:39:10.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:10 smithi173 ceph-mon[30530]: osdmap e866: 12 total, 12 up, 11 in 2024-09-17T16:39:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:10 smithi173 ceph-mon[30530]: osdmap e867: 12 total, 12 up, 11 in 2024-09-17T16:39:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:10 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:39:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:10 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:39:10.662 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:10.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:10.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:10.751 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:39:10.764 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:10.794 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:10.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:10.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:10.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:10.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:10.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:10.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 9336 9320 83.2065 71.25 0.0042907 0.0119912 2024-09-17T16:39:11.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:11.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:11.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:11.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:11.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:11.181 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:11.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:11.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:11.309 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:11.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:11.411 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:11.438 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:11.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:11.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:11 smithi017 ceph-mon[27571]: pgmap v1917: 17 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 447 MiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s; 564/21414 objects degraded (2.634%) 2024-09-17T16:39:11.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:11 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 564/21414 objects degraded (2.634%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:39:11.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:11 smithi017 ceph-mon[27571]: osdmap e868: 12 total, 12 up, 11 in 2024-09-17T16:39:11.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1403952584' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:39:11.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:11.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:11 smithi071 ceph-mon[30002]: pgmap v1917: 17 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 447 MiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s; 564/21414 objects degraded (2.634%) 2024-09-17T16:39:11.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:11 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 564/21414 objects degraded (2.634%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:39:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:11 smithi071 ceph-mon[30002]: osdmap e868: 12 total, 12 up, 11 in 2024-09-17T16:39:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1403952584' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:39:11.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:11.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:11.590 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:39:11.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:11 smithi173 ceph-mon[30530]: pgmap v1917: 17 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 447 MiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s; 564/21414 objects degraded (2.634%) 2024-09-17T16:39:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:11 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 564/21414 objects degraded (2.634%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:39:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:11 smithi173 ceph-mon[30530]: osdmap e868: 12 total, 12 up, 11 in 2024-09-17T16:39:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1403952584' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:39:11.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:11.681 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:11.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:11.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:11.766 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:39:11.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:11.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:11.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:11.906 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:39:11.927 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:11.956 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:11.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:11.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 10930 10914 85.2578 99.625 0.00144631 0.0117101 2024-09-17T16:39:12.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:12.083 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:12.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:12.185 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:12.209 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:12.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:12.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:12.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:12.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:12 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:12.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1403952584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:39:12.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:12 smithi071 ceph-mon[30002]: osdmap e869: 12 total, 12 up, 11 in 2024-09-17T16:39:12.338 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:12.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:12.440 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:12.468 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:39:12.469 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:39:12.469 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:12 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:12.469 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1403952584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:39:12.469 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:12 smithi017 ceph-mon[27571]: osdmap e869: 12 total, 12 up, 11 in 2024-09-17T16:39:12.470 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:12.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:12.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:12.474 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:39:12.474 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:39:12.474 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:39:12.474 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:39:12.494 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:39:12.495 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:39:12.520 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:39:12.520 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:39:12.547 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:39:12.547 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:39:12.571 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:12.574 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:39:12.574 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:39:12.598 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:12.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:12.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:12.600 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:39:12.600 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:39:12.626 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:39:12.627 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:39:12.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:12 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1403952584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:39:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:12 smithi173 ceph-mon[30530]: osdmap e869: 12 total, 12 up, 11 in 2024-09-17T16:39:12.651 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:39:12.651 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:39:12.680 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:39:12.681 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:39:12.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:12.706 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:39:12.707 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:39:12.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:12.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:12.731 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:39:12.731 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:39:12.756 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:39:12.756 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:39:12.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:12.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:12.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:12.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:12.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:12.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:12.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 12029 12013 83.416 68.6875 0.0610811 0.0118387 2024-09-17T16:39:13.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:13.098 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:13.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:13.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:13.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:13.212 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:39:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:13.228 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:13.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:13.330 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:13.353 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:13.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:13.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:13.456 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:13.478 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:13.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:13.479 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:39:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:13 smithi017 ceph-mon[27571]: pgmap v1920: 17 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 14 active+clean; 490 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 56 MiB/s wr, 889 op/s; 564/23508 objects degraded (2.399%); 212/23508 objects misplaced (0.902%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:39:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/118216265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:13.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:13 smithi017 ceph-mon[27571]: osdmap e870: 12 total, 12 up, 11 in 2024-09-17T16:39:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:13 smithi071 ceph-mon[30002]: pgmap v1920: 17 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 14 active+clean; 490 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 56 MiB/s wr, 889 op/s; 564/23508 objects degraded (2.399%); 212/23508 objects misplaced (0.902%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/118216265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:13 smithi071 ceph-mon[30002]: osdmap e870: 12 total, 12 up, 11 in 2024-09-17T16:39:13.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:13.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:13.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:13.601 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:13 smithi173 ceph-mon[30530]: pgmap v1920: 17 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 14 active+clean; 490 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 56 MiB/s wr, 889 op/s; 564/23508 objects degraded (2.399%); 212/23508 objects misplaced (0.902%); 5.0 MiB/s, 79 objects/s recovering 2024-09-17T16:39:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/118216265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:13 smithi173 ceph-mon[30530]: osdmap e870: 12 total, 12 up, 11 in 2024-09-17T16:39:13.702 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:13.725 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:13.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:13.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:13.855 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:13.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:13.957 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:13.984 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:13.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:13.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:13.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 12696 12680 79.2428 41.6875 0.00224557 0.0126099 2024-09-17T16:39:14.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:14.089 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:14.089 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:14.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:14.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:14.200 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 8, 9, 11, 7, 10, 5] 2024-09-17T16:39:14.200 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:39:14.200 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-17T16:39:14.201 INFO:tasks.thrashosds.thrasher:Setting osd 11 primary_affinity to 0.000000 2024-09-17T16:39:14.201 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 11 0 2024-09-17T16:39:14.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:14.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:14.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:14.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:14.360 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:14.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:14.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:14.491 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:14.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:14.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:14.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:14.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:14.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:14.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:14.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:14.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:14.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:14.871 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:14.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:14.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:14 smithi017 ceph-mon[27571]: osdmap e871: 12 total, 12 up, 11 in 2024-09-17T16:39:14.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:14 smithi017 ceph-mon[27571]: pgmap v1923: 17 pgs: 3 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 12 active+clean; 620 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 43 MiB/s wr, 694 op/s; 564/29751 objects degraded (1.896%); 343/29751 objects misplaced (1.153%); 18 MiB/s, 291 objects/s recovering 2024-09-17T16:39:14.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:14 smithi017 ceph-mon[27571]: osdmap e872: 12 total, 12 up, 11 in 2024-09-17T16:39:14.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 13779 13763 78.1918 67.6875 0.245758 0.0125104 2024-09-17T16:39:14.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:14.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:14.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:15.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:14 smithi071 ceph-mon[30002]: osdmap e871: 12 total, 12 up, 11 in 2024-09-17T16:39:15.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:14 smithi071 ceph-mon[30002]: pgmap v1923: 17 pgs: 3 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 12 active+clean; 620 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 43 MiB/s wr, 694 op/s; 564/29751 objects degraded (1.896%); 343/29751 objects misplaced (1.153%); 18 MiB/s, 291 objects/s recovering 2024-09-17T16:39:15.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:14 smithi071 ceph-mon[30002]: osdmap e872: 12 total, 12 up, 11 in 2024-09-17T16:39:15.096 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:15.119 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:15.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:15.126 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:39:15.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:14 smithi173 ceph-mon[30530]: osdmap e871: 12 total, 12 up, 11 in 2024-09-17T16:39:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:14 smithi173 ceph-mon[30530]: pgmap v1923: 17 pgs: 3 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 12 active+clean; 620 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 43 MiB/s wr, 694 op/s; 564/29751 objects degraded (1.896%); 343/29751 objects misplaced (1.153%); 18 MiB/s, 291 objects/s recovering 2024-09-17T16:39:15.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:14 smithi173 ceph-mon[30530]: osdmap e872: 12 total, 12 up, 11 in 2024-09-17T16:39:15.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:15.243 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:15.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:15.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:15.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:15.367 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:15.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:15.428 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:15.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:15.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:15.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:15.592 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:15.615 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:15.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:15.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:15.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:15.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:15.768 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.11 primary-affinity to 0 (802) 2024-09-17T16:39:15.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:15.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:39:15.874 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:15.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:15.875 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2025945263' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 0.0} : dispatch 2024-09-17T16:39:15.875 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2025945263' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]': finished 2024-09-17T16:39:15.875 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:15 smithi017 ceph-mon[27571]: osdmap e873: 12 total, 12 up, 11 in 2024-09-17T16:39:15.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:15.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 14129 14113 73.4987 21.875 0.00251793 0.0134908 2024-09-17T16:39:15.999 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:16.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:16.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2025945263' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 0.0} : dispatch 2024-09-17T16:39:16.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2025945263' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]': finished 2024-09-17T16:39:16.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:15 smithi071 ceph-mon[30002]: osdmap e873: 12 total, 12 up, 11 in 2024-09-17T16:39:16.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:16.122 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:16.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:16.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2025945263' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 0.0} : dispatch 2024-09-17T16:39:16.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2025945263' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 0.0}]': finished 2024-09-17T16:39:16.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:15 smithi173 ceph-mon[30530]: osdmap e873: 12 total, 12 up, 11 in 2024-09-17T16:39:16.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:16.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:16.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:16.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:16.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:16.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:16.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:16.517 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:16.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:16.518 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:16.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:16.641 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:16.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:16.744 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:16.766 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:16.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:16.782 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:39:16.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:16.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:16.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:16.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:16.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:39:16.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 15489 15473 74.383 85 0.0137676 0.0134174 2024-09-17T16:39:16.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:16 smithi017 ceph-mon[27571]: pgmap v1926: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 13 active+clean; 771 MiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 343/36966 objects misplaced (0.928%); 30 MiB/s, 473 objects/s recovering 2024-09-17T16:39:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2115894831' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:16.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:17.029 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:17.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:17.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:16 smithi071 ceph-mon[30002]: pgmap v1926: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 13 active+clean; 771 MiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 343/36966 objects misplaced (0.928%); 30 MiB/s, 473 objects/s recovering 2024-09-17T16:39:17.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2115894831' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:17.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:17.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:16 smithi173 ceph-mon[30530]: pgmap v1926: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 13 active+clean; 771 MiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 343/36966 objects misplaced (0.928%); 30 MiB/s, 473 objects/s recovering 2024-09-17T16:39:17.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2115894831' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:17.154 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:17.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:17.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:17.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:17.268 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:39:17.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:17.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:17.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:17.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:17.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:17.523 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:17.548 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:17.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:17.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:17.650 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:17.673 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:17.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:17.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:17.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:17.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:17.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:17.801 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:17.801 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:17.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:17.924 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:17.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:17.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 17326 17310 77.2701 114.812 0.00175151 0.0129321 2024-09-17T16:39:18.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:18.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:18.057 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:18.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:17 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 564/29751 objects degraded (1.896%), 1 pg degraded) 2024-09-17T16:39:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:17 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2115894831' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:18.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:17 smithi071 ceph-mon[30002]: osdmap e874: 12 total, 12 up, 11 in 2024-09-17T16:39:18.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:17 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 564/29751 objects degraded (1.896%), 1 pg degraded) 2024-09-17T16:39:18.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:17 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:18.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2115894831' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:18.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:17 smithi173 ceph-mon[30530]: osdmap e874: 12 total, 12 up, 11 in 2024-09-17T16:39:18.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:18.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:18.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:18.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:17 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 564/29751 objects degraded (1.896%), 1 pg degraded) 2024-09-17T16:39:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:17 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2115894831' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:17 smithi017 ceph-mon[27571]: osdmap e874: 12 total, 12 up, 11 in 2024-09-17T16:39:18.289 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:18.310 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:18.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:18.413 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:18.437 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:18.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:18.539 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:18.573 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:18.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:18.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:39:18.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:18.701 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:18.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:18.803 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:18.827 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:18.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:18.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:18.941 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:39:18.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:18.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:18.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 18942 18926 78.8516 101 0.00147295 0.0126455 2024-09-17T16:39:18.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:18 smithi017 ceph-mon[27571]: pgmap v1928: 17 pgs: 1 peering, 16 active+clean; 922 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 20 MiB/s, 318 objects/s recovering 2024-09-17T16:39:19.052 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:19.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:18 smithi071 ceph-mon[30002]: pgmap v1928: 17 pgs: 1 peering, 16 active+clean; 922 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 20 MiB/s, 318 objects/s recovering 2024-09-17T16:39:19.075 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:19.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:19.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:19.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:18 smithi173 ceph-mon[30530]: pgmap v1928: 17 pgs: 1 peering, 16 active+clean; 922 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 20 MiB/s, 318 objects/s recovering 2024-09-17T16:39:19.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:19.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:19.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:19.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:19.303 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:19.330 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:19.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:19.355 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:19.432 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:19.455 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:19.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:19.482 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:19.482 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:19.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:19.586 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:19.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:19.688 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:19.714 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:19.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:19.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:19.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:19.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:19.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:19.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:19.949 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:39:19.949 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:39:19.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:19.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:19.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:39:19.973 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:39:19.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 20339 20323 79.3799 87.3125 0.023535 0.0125884 2024-09-17T16:39:19.996 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:39:19.997 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:39:20.017 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:39:20.018 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:39:20.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:39:20.042 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:39:20.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:19 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:39:20.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:19 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:39:20.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:19 smithi017 ceph-mon[27571]: osdmap e875: 12 total, 12 up, 11 in 2024-09-17T16:39:20.064 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:39:20.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:19 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:39:20.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:19 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:39:20.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:19 smithi071 ceph-mon[30002]: osdmap e875: 12 total, 12 up, 11 in 2024-09-17T16:39:20.064 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:39:20.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:20.085 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:39:20.085 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:39:20.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:20.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:20.109 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:39:20.109 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:39:20.131 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:39:20.131 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:39:20.187 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:19 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:39:20.187 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:19 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:39:20.188 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:19 smithi173 ceph-mon[30530]: osdmap e875: 12 total, 12 up, 11 in 2024-09-17T16:39:20.188 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:39:20.188 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:39:20.188 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:39:20.189 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:39:20.194 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:39:20.195 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:39:20.198 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:20.223 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:20.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:20.324 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:20.350 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:20.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:20.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:20.482 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:20.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:20.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:39:20.584 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:20.607 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:20.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:20.708 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:20.731 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:20.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:20.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:20.769 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:39:20.796 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1663432744' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:20.796 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:20 smithi071 ceph-mon[30002]: pgmap v1930: 17 pgs: 17 active+clean; 1.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 11 MiB/s, 169 objects/s recovering 2024-09-17T16:39:20.834 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:20.858 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:20.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:20.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:20.986 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:20.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:20.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:20.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 21430 21414 78.7212 68.1875 0.0196108 0.0126938 2024-09-17T16:39:20.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1663432744' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:20.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:20 smithi017 ceph-mon[27571]: pgmap v1930: 17 pgs: 17 active+clean; 1.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 11 MiB/s, 169 objects/s recovering 2024-09-17T16:39:21.039 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:39:21.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:21.114 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:21.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1663432744' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:20 smithi173 ceph-mon[30530]: pgmap v1930: 17 pgs: 17 active+clean; 1.1 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 11 MiB/s, 169 objects/s recovering 2024-09-17T16:39:21.216 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:21.243 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:21.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:21.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:21.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:21.346 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:21.348 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:39:21.348 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:39:21.349 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:39:21.349 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:39:21.349 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:39:21.375 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:21.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:21.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:21.476 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:21.504 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:21.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:21.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:21.632 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:21.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:21.734 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:21.764 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:21.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:21.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:21.896 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:21.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:21.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 23557 23541 81.7326 132.938 0.00948303 0.0122275 2024-09-17T16:39:21.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:22.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:22.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:22.129 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:22.157 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:22.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:22.259 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:22.285 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:22.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:22.387 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:22.408 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:22.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:22.510 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:22.534 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:22.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:22.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:22.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:22.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:22.759 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:22.787 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:22.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:22.889 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:22.915 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:22.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:22.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:22.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:39:22.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 25031 25015 82.2792 92.125 0.00295097 0.0121447 2024-09-17T16:39:23.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:23.044 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:23.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:23.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:23.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:39:23.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:23.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:23.199 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:39:23.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:23.314 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:23.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:23.323 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:39:23.415 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:23.438 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:23.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:23.451 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:23 smithi071 ceph-mon[30002]: pgmap v1931: 17 pgs: 17 active+clean; 1.1 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s; 3.4 MiB/s, 54 objects/s recovering 2024-09-17T16:39:23.451 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1453909066' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:39:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:23 smithi017 ceph-mon[27571]: pgmap v1931: 17 pgs: 17 active+clean; 1.1 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s; 3.4 MiB/s, 54 objects/s recovering 2024-09-17T16:39:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1453909066' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:39:23.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:23.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:23.572 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:23 smithi173 ceph-mon[30530]: pgmap v1931: 17 pgs: 17 active+clean; 1.1 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s; 3.4 MiB/s, 54 objects/s recovering 2024-09-17T16:39:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1453909066' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:39:23.673 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:23.700 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:23.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:23.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:23.830 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:23.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:23.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:23.874 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:39:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:39:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:39:23.874 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:170216 7.11 7:89b9c9a3:::benchmark_data_smithi173_116396_object74616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 358.33132754299999, 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.387146404, 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170216 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.875 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.876 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538980+0000", 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3030000000000001e-06 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538986+0000", 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6579999999999999e-06 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538998+0000", 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2903e-05 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.877 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539004+0000", 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9540000000000001e-06 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539051+0000", 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7067999999999997e-05 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539144+0000", 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2521000000000004e-05 2024-09-17T16:39:23.878 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539219+0000", 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4791999999999998e-05 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541123+0000", 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019039650000000001 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541689+0000", 2024-09-17T16:39:23.879 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00056619600000000004 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925945+0000", 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38425619700000002 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926009+0000", 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3516999999999995e-05 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.880 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926125+0000", 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011633000000000001 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:170216 7.11 e712/703 ondisk, result = 0)", 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 358.32893274000003, 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38475827200000001, 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.881 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.882 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541374+0000", 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541379+0000", 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6210000000000003e-06 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.923614+0000", 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38223536499999999 2024-09-17T16:39:23.883 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.923618+0000", 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7629999999999998e-06 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925930+0000", 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023126259999999999 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.884 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925939+0000", 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0070000000000003e-06 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926132+0000", 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019219300000000001 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:170289 7.11 7:8ff6a0b8:::benchmark_data_smithi173_116396_object74689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:39:23.885 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 358.24599892700002, 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30203754100000002, 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170289 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:39:23.886 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624309+0000", 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.136e-06 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.887 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624319+0000", 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9170000000000001e-06 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624348+0000", 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8368999999999999e-05 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624362+0000", 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4222999999999999e-05 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.888 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624491+0000", 2024-09-17T16:39:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000129262 2024-09-17T16:39:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.889 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624698+0000", 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000206546 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624844+0000", 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014652400000000001 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.626781+0000", 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019372759999999999 2024-09-17T16:39:23.890 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627901+0000", 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011196260000000001 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926178+0000", 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29827702699999997 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926231+0000", 2024-09-17T16:39:23.891 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2892000000000001e-05 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926345+0000", 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000113743 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:170289 7.11 e712/703 ondisk, result = 0)", 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:39:23.892 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 358.24329476499997, 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29933906100000002, 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:39:23.893 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627012+0000", 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627016+0000", 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9809999999999997e-06 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.894 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925949+0000", 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29893330699999998 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925958+0000", 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926164+0000", 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000205942 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.895 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926171+0000", 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9500000000000001e-06 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926350+0000", 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017893299999999999 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDelete(7.19 e717,677 7:9da3d41f:::benchmark_data_smithi173_116396_object4781:head,673'6297)", 2024-09-17T16:39:23.896 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 347.69436858300003, 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37136110999999999, 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.897 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175947+0000", 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3859999999999996e-06 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175959+0000", 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2269999999999999e-05 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.898 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547126+0000", 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37116651099999998 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547136+0000", 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0241000000000001e-05 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547216+0000", 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.051e-05 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547227+0000", 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0356999999999999e-05 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547299+0000", 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1835000000000006e-05 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.900 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:191171 7.11 7:8d50061b:::benchmark_data_smithi173_116396_object95571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 329.43974044999999, 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44239869199999998, 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 191171 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430568+0000", 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.897e-06 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430578+0000", 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0149999999999999e-05 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430616+0000", 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8753e-05 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430622+0000", 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3569999999999997e-06 2024-09-17T16:39:23.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430666+0000", 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3831000000000001e-05 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430740+0000", 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4865999999999998e-05 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430817+0000", 2024-09-17T16:39:23.904 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6490999999999997e-05 2024-09-17T16:39:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433451+0000", 2024-09-17T16:39:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0026335489999999998 2024-09-17T16:39:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433803+0000", 2024-09-17T16:39:23.905 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00035236799999999998 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872847+0000", 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43904401599999998 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872921+0000", 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3867000000000003e-05 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.906 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872964+0000", 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3547000000000002e-05 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:191171 7.11 e724/703 ondisk, result = 0)", 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 329.43671221900001, 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43937563000000002, 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.907 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.908 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433595+0000", 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2900000000000002e-07 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433602+0000", 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9550000000000003e-06 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872622+0000", 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43902024000000001 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872639+0000", 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7516e-05 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872820+0000", 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000180714 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872837+0000", 2024-09-17T16:39:23.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7099999999999999e-05 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872970+0000", 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013227599999999999 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(8.12 743/742 [PushOp(8:4d9389a5:::benchmark_data_smithi173_130566_object4465:head, version: 732'276, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:4d9389a5:::benchmark_data_smithi173_130566_object4465:head@732'276, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:29.740078+0000", 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 294.13022805200001, 2024-09-17T16:39:23.911 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32815092099999998, 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740078+0000", 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.912 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740078+0000", 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740079+0000", 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0900000000000001e-07 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740180+0000", 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010132900000000001 2024-09-17T16:39:23.913 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067836+0000", 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32765588099999998 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067848+0000", 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1754e-05 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.914 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067920+0000", 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2813999999999998e-05 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067929+0000", 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1939999999999998e-06 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.068229+0000", 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030034000000000001 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.915 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:78112 8.12 8:4c421870:::benchmark_data_smithi173_130566_object78111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:26.226347+0000", 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 237.64395909999999, 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.305366213, 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 78112 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.916 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226347+0000", 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226347+0000", 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.917 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226348+0000", 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.435e-06 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226401+0000", 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2386000000000003e-05 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226415+0000", 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4095e-05 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.918 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226421+0000", 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1240000000000002e-06 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226475+0000", 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4376999999999998e-05 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226606+0000", 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013063500000000001 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226759+0000", 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015287100000000001 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228063+0000", 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013040199999999999 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228274+0000", 2024-09-17T16:39:23.920 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000211201 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531553+0000", 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30327892299999998 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531612+0000", 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8836999999999999e-05 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531713+0000", 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000101309 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:78112 8.12 e762/744 ondisk, result = 0)", 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:26.228645+0000", 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 237.64166099799999, 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30307498500000002, 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.922 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228645+0000", 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228645+0000", 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.923 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228646+0000", 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7999999999999995e-07 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228649+0000", 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6370000000000001e-06 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531475+0000", 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.302826076 2024-09-17T16:39:23.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531482+0000", 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7490000000000001e-06 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531537+0000", 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4731000000000002e-05 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531547+0000", 2024-09-17T16:39:23.925 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0200000000000001e-05 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531720+0000", 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000173012 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "rep_scrubmap(8.12 e744 from shard 0)", 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:34.607872+0000", 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 169.26243442000001, 2024-09-17T16:39:23.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26429114300000001, 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.607872+0000", 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.607872+0000", 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.927 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.607872+0000", 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2300000000000001e-07 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.607897+0000", 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4831e-05 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.928 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.871871+0000", 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26397431999999998 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.871878+0000", 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7839999999999998e-06 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.871942+0000", 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3478000000000004e-05 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.929 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.871957+0000", 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5006e-05 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:34.872163+0000", 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00020610100000000001 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.930 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:139590 8.12 8:4ed879cd:::benchmark_data_smithi173_130566_object44615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:26.124420+0000", 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 117.745886552, 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27235878800000002, 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 139590 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.931 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.932 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:23.933 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.124420+0000", 2024-09-17T16:39:23.933 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.933 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.933 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.124420+0000", 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.124421+0000", 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.516e-06 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.124427+0000", 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9569999999999999e-06 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.124443+0000", 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5713e-05 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.124451+0000", 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.875e-06 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.124522+0000", 2024-09-17T16:39:23.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1338e-05 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.124677+0000", 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015493000000000001 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.124827+0000", 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015008400000000001 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.126989+0000", 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021616259999999998 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.127221+0000", 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00023196999999999999 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.395160+0000", 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.267939072 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.395197+0000", 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6940999999999997e-05 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.396778+0000", 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015817660000000001 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:139648 8.12 8:4fc2d70d:::benchmark_data_smithi173_130566_object44673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:26.177531+0000", 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 117.692775314, 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.27021079199999998, 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 139648 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.177531+0000", 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.177531+0000", 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.177532+0000", 2024-09-17T16:39:23.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.279e-06 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.177537+0000", 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.344e-06 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.177554+0000", 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.643e-05 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.941 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.177564+0000", 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7070000000000004e-06 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.177629+0000", 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5328999999999997e-05 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.177810+0000", 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018126700000000001 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.177972+0000", 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016183199999999999 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.180119+0000", 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002147393 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.180135+0000", 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5747e-05 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.446130+0000", 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26599521999999998 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.446171+0000", 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0723999999999998e-05 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:26.447742+0000", 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015705199999999999 2024-09-17T16:39:23.944 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:153469 8.12 8:4c954a71:::benchmark_data_smithi173_130566_object58494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e821)", 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:43.721465+0000", 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 100.148841117, 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38866898999999999, 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.945 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 153469 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721465+0000", 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721465+0000", 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721467+0000", 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.68e-06 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721473+0000", 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6900000000000003e-06 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721493+0000", 2024-09-17T16:39:23.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9811999999999999e-05 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721502+0000", 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1409999999999994e-06 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721562+0000", 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9406e-05 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.948 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721713+0000", 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015121899999999999 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721848+0000", 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013535399999999999 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.724038+0000", 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00218995 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.949 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.724638+0000", 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059941900000000002 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.108614+0000", 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38397621199999998 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.108657+0000", 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3130999999999998e-05 2024-09-17T16:39:23.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.110134+0000", 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014769760000000001 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:154123 8.12 8:4c613ccc:::benchmark_data_smithi173_130566_object59148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e821)", 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:44.579188+0000", 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 99.291117596000007, 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34086830200000001, 2024-09-17T16:39:23.951 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 154123 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579188+0000", 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579188+0000", 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579190+0000", 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.905e-06 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579198+0000", 2024-09-17T16:39:23.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1040000000000001e-06 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579214+0000", 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6089000000000001e-05 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579222+0000", 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8550000000000008e-06 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579290+0000", 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7329000000000005e-05 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579469+0000", 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017971400000000001 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579627+0000", 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015795199999999999 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.581796+0000", 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021689919999999998 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.584094+0000", 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022974509999999998 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918612+0000", 2024-09-17T16:39:23.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33451803200000002 2024-09-17T16:39:23.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918651+0000", 2024-09-17T16:39:23.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9332000000000002e-05 2024-09-17T16:39:23.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.920057+0000", 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001405547 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:154123 8.12 e821/792 ondisk, result = 0)", 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:44.581915+0000", 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 99.288390741000001, 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33814677799999998, 2024-09-17T16:39:23.958 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.581915+0000", 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.581915+0000", 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.581916+0000", 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7400000000000003e-07 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.581920+0000", 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9659999999999998e-06 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918559+0000", 2024-09-17T16:39:23.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.336639297 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918564+0000", 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5950000000000001e-06 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918600+0000", 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6449999999999998e-05 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918607+0000", 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0849999999999999e-06 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.920062+0000", 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001454711 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:164490 8.12 8:4d667ff9:::benchmark_data_smithi173_130566_object69515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e831)", 2024-09-17T16:39:23.962 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:00.135053+0000", 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 83.735252942000002, 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25565550799999998, 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 164490 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.135053+0000", 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.135053+0000", 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.135055+0000", 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2189999999999998e-06 2024-09-17T16:39:23.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.135063+0000", 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4540000000000001e-06 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.135083+0000", 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9814e-05 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.135090+0000", 2024-09-17T16:39:23.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5009999999999998e-06 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.135153+0000", 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2593000000000002e-05 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.135270+0000", 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000117365 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.135422+0000", 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015177999999999999 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.138161+0000", 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0027388310000000002 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.138554+0000", 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00039296900000000001 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.389088+0000", 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25053388599999998 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.389148+0000", 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0191e-05 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:00.390709+0000", 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001560905 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:173387 8.12 8:4852001c:::benchmark_data_smithi173_130566_object78412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e835)", 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:11.175014+0000", 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 72.695291874999995, 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.257018144, 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.969 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 173387 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.175014+0000", 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.175014+0000", 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.175016+0000", 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6109999999999999e-06 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.175023+0000", 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9240000000000001e-06 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.175040+0000", 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6841e-05 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.175048+0000", 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4130000000000006e-06 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.175117+0000", 2024-09-17T16:39:23.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8773e-05 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.175279+0000", 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016208300000000001 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.175435+0000", 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000155811 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.178270+0000", 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028354130000000002 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.178406+0000", 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000135962 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.430873+0000", 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25246731500000003 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.430909+0000", 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5199e-05 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:11.432032+0000", 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001123799 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.975 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:184588 8.12 8:4e86cc59:::benchmark_data_smithi173_130566_object89613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:27.062476+0000", 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 56.807829884999997, 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15509219499999999, 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 184588 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.062476+0000", 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.062476+0000", 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.062478+0000", 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2929999999999999e-06 2024-09-17T16:39:23.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.062483+0000", 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5650000000000001e-06 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.062494+0000", 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1406000000000001e-05 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.062500+0000", 2024-09-17T16:39:23.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7980000000000002e-06 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.062558+0000", 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7908000000000002e-05 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.062686+0000", 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012815400000000001 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.062781+0000", 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4588999999999998e-05 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.065260+0000", 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024789759999999999 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.066231+0000", 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00097070100000000003 2024-09-17T16:39:23.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.217390+0000", 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.151159081 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.217427+0000", 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7589000000000002e-05 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.217568+0000", 2024-09-17T16:39:23.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000141135 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(8.12 e845,792 8:4e161f77:::benchmark_data_smithi173_130566_object64651:head,824'5250)", 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:27.534808+0000", 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 56.335497996999997, 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.14875078999999999, 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:39:23.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.534808+0000", 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.534808+0000", 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.534809+0000", 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9299999999999997e-07 2024-09-17T16:39:23.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.534817+0000", 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7510000000000008e-06 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.683220+0000", 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.148403698 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.683229+0000", 2024-09-17T16:39:23.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2569999999999996e-06 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.683294+0000", 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4160999999999998e-05 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:27.683559+0000", 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026522999999999998 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.985 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:39:23.986 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:39:23.986 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:39:23.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:39:23.987555+0000 min lat: 0.00107045 max lat: 2.12556 avg lat: 0.0120548 2024-09-17T16:39:23.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:39:23.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 26549 26533 82.9086 94.875 0.0343801 0.0120548 2024-09-17T16:39:24.013 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:24.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:24.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:24.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:24.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:24.185 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:24.238 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 8, 9, 11, 7, 10, 5] 2024-09-17T16:39:24.239 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:39:24.239 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-17T16:39:24.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:39:24.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:24.306 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:24 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:24.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1453909066' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:39:24.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:24 smithi017 ceph-mon[27571]: osdmap e876: 12 total, 12 up, 11 in 2024-09-17T16:39:24.307 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:24.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:24.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:24.434 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:24.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:24.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:24.487 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:24 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:24.487 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1453909066' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:39:24.487 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:24 smithi071 ceph-mon[30002]: osdmap e876: 12 total, 12 up, 11 in 2024-09-17T16:39:24.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:24.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:24.557 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:24.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:24 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1453909066' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:39:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:24 smithi173 ceph-mon[30530]: osdmap e876: 12 total, 12 up, 11 in 2024-09-17T16:39:24.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:24.689 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:24.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:24.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:39:24.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:24.810 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:39:24.810 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:39:24.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:24.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:24.828 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:24.839 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:39:24.840 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:39:24.865 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:39:24.865 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:39:24.889 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:39:24.889 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:39:24.915 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:39:24.916 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:39:24.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:24.941 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:39:24.941 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:39:24.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:24.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:24.966 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:39:24.966 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:39:24.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 27814 27798 82.7251 79.0625 0.0134697 0.0120752 2024-09-17T16:39:25.050 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:25.075 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:25.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:25.080 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:25.083 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:39:25.084 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:39:25.109 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:39:25.109 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:39:25.132 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:39:25.133 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:39:25.155 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:39:25.155 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:39:25.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:25.178 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:39:25.178 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:39:25.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:25.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:25.310 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:25.331 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:25.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:25.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:25.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:25.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:25.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:25.458 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:25 smithi173 ceph-mon[30530]: pgmap v1933: 17 pgs: 17 active+clean; 1.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-09-17T16:39:25.458 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:25 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:39:25.458 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:25 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:39:25.458 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:25 smithi173 ceph-mon[30530]: osdmap e877: 12 total, 12 up, 11 in 2024-09-17T16:39:25.458 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2931431377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:25.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:25 smithi017 ceph-mon[27571]: pgmap v1933: 17 pgs: 17 active+clean; 1.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-09-17T16:39:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:25 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:39:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:25 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:39:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:25 smithi017 ceph-mon[27571]: osdmap e877: 12 total, 12 up, 11 in 2024-09-17T16:39:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2931431377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:25.556 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:25.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:25 smithi071 ceph-mon[30002]: pgmap v1933: 17 pgs: 17 active+clean; 1.4 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s; 1.1 MiB/s, 17 objects/s recovering 2024-09-17T16:39:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:25 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:39:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:25 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:39:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:25 smithi071 ceph-mon[30002]: osdmap e877: 12 total, 12 up, 11 in 2024-09-17T16:39:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2931431377' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:25.577 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:39:25.577 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:39:25.581 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:25.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:25.684 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:25.706 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:25.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:25.808 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:25.834 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:25.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:25.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:25.962 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:25.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:25.989 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 28737 28721 81.5869 57.6875 0.0633716 0.0122504 2024-09-17T16:39:26.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:39:26.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:26.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:26.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:26.161 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:26.161 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:26.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:26.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:26.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:26.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:26.339 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:26 smithi017 ceph-mon[27571]: from='client.18519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:39:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:26.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:26.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:26.471 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:26.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:26.553 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:26 smithi173 ceph-mon[30530]: from='client.18519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:39:26.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:26 smithi071 ceph-mon[30002]: from='client.18519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:39:26.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:26.596 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:26.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:26.698 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:26.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:26.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:26.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:26.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:26.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:26.929 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1934,"stamp":"2024-09-17T16:39:24.762305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1635058208,"num_objects":24942,"num_object_clones":0,"num_object_copies":74826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24942,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":25063,"num_write_kb":1598310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2562,"num_bytes_recovered":168698944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25032,"ondisk_log_size":25032,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":10699888,"kb_used_data":4558508,"kb_used_omap":83,"kb_used_meta":6141100,"kb_avail":1020496784,"statfs":{"total":1055945392128,"available":1044988706816,"internally_reserved":0,"allocated":4667912192,"data_stored":4661477482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86009,"internal_metadata":6288486407},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,3,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":668467200,"num_objects":10200,"num_object_clones":0,"num_object_copies":30600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10200,"num_write_kb":652800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.623028"},"pg_stats":[{"pgid":"9.f","version":"876'1687","reported_seq":1734,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T16:39:24.603346+0000","last_change":"2024-09-17T16:38:54.766586+0000","last_active":"2024-09-17T16:39:24.603346+0000","last_peered":"2024-09-17T16:39:24.603346+0000","last_clean":"2024-09-17T16:39:24.603346+0000","last_became_active":"2024-09-17T16:38:54.765274+0000","last_became_peered":"2024-09-17T16:38:54.765274+0000","last_unstale":"2024-09-17T16:39:24.603346+0000","last_undegraded":"2024-09-17T16:39:24.603346+0000","last_fullsized":"2024-09-17T16:39:24.603346+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1687,"log_dups_size":0,"ondisk_log_size":1687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:15.671182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110559232,"num_objects":1687,"num_object_clones":0,"num_object_copies":5061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1687,"num_write_kb":107968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"875'1517","reported_seq":1562,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:22.261281+0000","last_change":"2024-09-17T16:38:54.765568+0000","last_active":"2024-09-17T16:39:22.261281+0000","last_peered":"2024-09-17T16:39:22.261281+0000","last_clean":"2024-09-17T16:39:22.261281+0000","last_became_active":"2024-09-17T16:38:54.765252+0000","last_became_peered":"2024-09-17T16:38:54.765252+0000","last_unstale":"2024-09-17T16:39:22.261281+0000","last_undegraded":"2024-09-17T16:39:22.261281+0000","last_fullsized":"2024-09-17T16:39:22.261281+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:55.899749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99418112,"num_objects":1517,"num_object_clones":0,"num_object_copies":4551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1517,"num_write_kb":97088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"875'1575","reported_seq":1620,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:23.147248+0000","last_change":"2024-09-17T16:38:54.759655+0000","last_active":"2024-09-17T16:39:23.147248+0000","last_peered":"2024-09-17T16:39:23.147248+0000","last_clean":"2024-09-17T16:39:23.147248+0000","last_became_active":"2024-09-17T16:38:54.759320+0000","last_became_peered":"2024-09-17T16:38:54.759320+0000","last_unstale":"2024-09-17T16:39:23.147248+0000","last_undegraded":"2024-09-17T16:39:23.147248+0000","last_fullsized":"2024-09-17T16:39:23.147248+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1575,"log_dups_size":0,"ondisk_log_size":1575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:11.394766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103219200,"num_objects":1575,"num_object_clones":0,"num_object_copies":4725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1575,"num_write_kb":100800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.c","version":"875'1308","reported_seq":2490,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:19.974479+0000","last_change":"2024-09-17T16:39:15.770878+0000","last_active":"2024-09-17T16:39:19.974479+0000","last_peered":"2024-09-17T16:39:19.974479+0000","last_clean":"2024-09-17T16:39:19.974479+0000","last_became_active":"2024-09-17T16:39:15.769803+0000","last_became_peered":"2024-09-17T16:39:15.769803+0000","last_unstale":"2024-09-17T16:39:19.974479+0000","last_undegraded":"2024-09-17T16:39:19.974479+0000","last_fullsized":"2024-09-17T16:39:19.974479+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:58.458126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85721088,"num_objects":1308,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1308,"num_write_kb":83712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":564,"num_bytes_recovered":36962304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"876'1671","reported_seq":1718,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T16:39:24.383871+0000","last_change":"2024-09-17T16:38:54.766769+0000","last_active":"2024-09-17T16:39:24.383871+0000","last_peered":"2024-09-17T16:39:24.383871+0000","last_clean":"2024-09-17T16:39:24.383871+0000","last_became_active":"2024-09-17T16:38:54.766429+0000","last_became_peered":"2024-09-17T16:38:54.766429+0000","last_unstale":"2024-09-17T16:39:24.383871+0000","last_undegraded":"2024-09-17T16:39:24.383871+0000","last_fullsized":"2024-09-17T16:39:24.383871+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:23.558032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109510656,"num_objects":1671,"num_object_clones":0,"num_object_copies":5013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1671,"num_write_kb":106944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.a","version":"876'1713","reported_seq":1759,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T16:39:24.601197+0000","last_change":"2024-09-17T16:38:54.760350+0000","last_active":"2024-09-17T16:39:24.601197+0000","last_peered":"2024-09-17T16:39:24.601197+0000","last_clean":"2024-09-17T16:39:24.601197+0000","last_became_active":"2024-09-17T16:38:54.760031+0000","last_became_peered":"2024-09-17T16:38:54.760031+0000","last_unstale":"2024-09-17T16:39:24.601197+0000","last_undegraded":"2024-09-17T16:39:24.601197+0000","last_fullsized":"2024-09-17T16:39:24.601197+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1713,"log_dups_size":0,"ondisk_log_size":1713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:12.669832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112263168,"num_objects":1713,"num_object_clones":0,"num_object_copies":5139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1713,"num_write_kb":109632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"875'1476","reported_seq":1521,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:22.263149+0000","last_change":"2024-09-17T16:38:54.764119+0000","last_active":"2024-09-17T16:39:22.263149+0000","last_peered":"2024-09-17T16:39:22.263149+0000","last_clean":"2024-09-17T16:39:22.263149+0000","last_became_active":"2024-09-17T16:38:54.763749+0000","last_became_peered":"2024-09-17T16:38:54.763749+0000","last_unstale":"2024-09-17T16:39:22.263149+0000","last_undegraded":"2024-09-17T16:39:22.263149+0000","last_fullsized":"2024-09-17T16:39:22.263149+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:06.640020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96731136,"num_objects":1476,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1476,"num_write_kb":94464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"875'1473","reported_seq":2488,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:22.262909+0000","last_change":"2024-09-17T16:39:13.779432+0000","last_active":"2024-09-17T16:39:22.262909+0000","last_peered":"2024-09-17T16:39:22.262909+0000","last_clean":"2024-09-17T16:39:22.262909+0000","last_became_active":"2024-09-17T16:39:13.777102+0000","last_became_peered":"2024-09-17T16:39:13.777102+0000","last_unstale":"2024-09-17T16:39:22.262909+0000","last_undegraded":"2024-09-17T16:39:22.262909+0000","last_fullsized":"2024-09-17T16:39:22.262909+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:13.911789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96534528,"num_objects":1473,"num_object_clones":0,"num_object_copies":4419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":94272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":31391744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"876'1660","reported_seq":1707,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T16:39:24.596408+0000","last_change":"2024-09-17T16:38:54.766597+0000","last_active":"2024-09-17T16:39:24.596408+0000","last_peered":"2024-09-17T16:39:24.596408+0000","last_clean":"2024-09-17T16:39:24.596408+0000","last_became_active":"2024-09-17T16:38:54.765661+0000","last_became_peered":"2024-09-17T16:38:54.765661+0000","last_unstale":"2024-09-17T16:39:24.596408+0000","last_undegraded":"2024-09-17T16:39:24.596408+0000","last_fullsized":"2024-09-17T16:39:24.596408+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1660,"log_dups_size":0,"ondisk_log_size":1660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:09.402795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108789760,"num_objects":1660,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1660,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"876'1778","reported_seq":1824,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T16:39:24.598243+0000","last_change":"2024-09-17T16:38:54.763911+0000","last_active":"2024-09-17T16:39:24.598243+0000","last_peered":"2024-09-17T16:39:24.598243+0000","last_clean":"2024-09-17T16:39:24.598243+0000","last_became_active":"2024-09-17T16:38:54.763476+0000","last_became_peered":"2024-09-17T16:38:54.763476+0000","last_unstale":"2024-09-17T16:39:24.598243+0000","last_undegraded":"2024-09-17T16:39:24.598243+0000","last_fullsized":"2024-09-17T16:39:24.598243+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1778,"log_dups_size":0,"ondisk_log_size":1778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:00.851346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116523008,"num_objects":1778,"num_object_clones":0,"num_object_copies":5334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1778,"num_write_kb":113792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"875'1558","reported_seq":1603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:22.631925+0000","last_change":"2024-09-17T16:38:54.763053+0000","last_active":"2024-09-17T16:39:22.631925+0000","last_peered":"2024-09-17T16:39:22.631925+0000","last_clean":"2024-09-17T16:39:22.631925+0000","last_became_active":"2024-09-17T16:38:54.762670+0000","last_became_peered":"2024-09-17T16:38:54.762670+0000","last_unstale":"2024-09-17T16:39:22.631925+0000","last_undegraded":"2024-09-17T16:39:22.631925+0000","last_fullsized":"2024-09-17T16:39:22.631925+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1558,"log_dups_size":0,"ondisk_log_size":1558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:10.270523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102105088,"num_objects":1558,"num_object_clones":0,"num_object_copies":4674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1558,"num_write_kb":99712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"876'1707","reported_seq":2729,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T16:39:24.412385+0000","last_change":"2024-09-17T16:39:13.777092+0000","last_active":"2024-09-17T16:39:24.412385+0000","last_peered":"2024-09-17T16:39:24.412385+0000","last_clean":"2024-09-17T16:39:24.412385+0000","last_became_active":"2024-09-17T16:39:13.775396+0000","last_became_peered":"2024-09-17T16:39:13.775396+0000","last_unstale":"2024-09-17T16:39:24.412385+0000","last_undegraded":"2024-09-17T16:39:24.412385+0000","last_fullsized":"2024-09-17T16:39:24.412385+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1707,"log_dups_size":0,"ondisk_log_size":1707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:14.159859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111869952,"num_objects":1707,"num_object_clones":0,"num_object_copies":5121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1707,"num_write_kb":109248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":31522816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"875'1552","reported_seq":2615,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:22.634069+0000","last_change":"2024-09-17T16:39:13.778186+0000","last_active":"2024-09-17T16:39:22.634069+0000","last_peered":"2024-09-17T16:39:22.634069+0000","last_clean":"2024-09-17T16:39:22.634069+0000","last_became_active":"2024-09-17T16:39:13.776232+0000","last_became_peered":"2024-09-17T16:39:13.776232+0000","last_unstale":"2024-09-17T16:39:22.634069+0000","last_undegraded":"2024-09-17T16:39:22.634069+0000","last_fullsized":"2024-09-17T16:39:22.634069+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1552,"log_dups_size":0,"ondisk_log_size":1552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:22.136930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101711872,"num_objects":1552,"num_object_clones":0,"num_object_copies":4656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1552,"num_write_kb":99328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":503,"num_bytes_recovered":32964608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"875'1422","reported_seq":2536,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:21.501642+0000","last_change":"2024-09-17T16:39:15.801706+0000","last_active":"2024-09-17T16:39:21.501642+0000","last_peered":"2024-09-17T16:39:21.501642+0000","last_clean":"2024-09-17T16:39:21.501642+0000","last_became_active":"2024-09-17T16:39:15.784630+0000","last_became_peered":"2024-09-17T16:39:15.784630+0000","last_unstale":"2024-09-17T16:39:21.501642+0000","last_undegraded":"2024-09-17T16:39:21.501642+0000","last_fullsized":"2024-09-17T16:39:21.501642+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1422,"log_dups_size":0,"ondisk_log_size":1422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:22.167302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93192192,"num_objects":1422,"num_object_clones":0,"num_object_copies":4266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1422,"num_write_kb":91008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":531,"num_bytes_recovered":34799616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"875'1478","reported_seq":1523,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:22.628591+0000","last_change":"2024-09-17T16:38:54.757271+0000","last_active":"2024-09-17T16:39:22.628591+0000","last_peered":"2024-09-17T16:39:22.628591+0000","last_clean":"2024-09-17T16:39:22.628591+0000","last_became_active":"2024-09-17T16:38:54.756978+0000","last_became_peered":"2024-09-17T16:38:54.756978+0000","last_unstale":"2024-09-17T16:39:22.628591+0000","last_undegraded":"2024-09-17T16:39:22.628591+0000","last_fullsized":"2024-09-17T16:39:22.628591+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1478,"log_dups_size":0,"ondisk_log_size":1478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:00.524133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96862208,"num_objects":1478,"num_object_clones":0,"num_object_copies":4434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1478,"num_write_kb":94592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"875'1365","reported_seq":1410,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:21.481256+0000","last_change":"2024-09-17T16:38:54.768093+0000","last_active":"2024-09-17T16:39:21.481256+0000","last_peered":"2024-09-17T16:39:21.481256+0000","last_clean":"2024-09-17T16:39:21.481256+0000","last_became_active":"2024-09-17T16:38:54.767749+0000","last_became_peered":"2024-09-17T16:38:54.767749+0000","last_unstale":"2024-09-17T16:39:21.481256+0000","last_undegraded":"2024-09-17T16:39:21.481256+0000","last_fullsized":"2024-09-17T16:39:21.481256+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":1365,"log_dups_size":0,"ondisk_log_size":1365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:08.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89456640,"num_objects":1365,"num_object_clones":0,"num_object_copies":4095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1365,"num_write_kb":87360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":522,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-09-17T16:39:19.761142+0000","last_change":"2024-09-17T16:33:59.364895+0000","last_active":"2024-09-17T16:39:19.761142+0000","last_peered":"2024-09-17T16:39:19.761142+0000","last_clean":"2024-09-17T16:39:19.761142+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:39:19.761142+0000","last_undegraded":"2024-09-17T16:39:19.761142+0000","last_fullsized":"2024-09-17T16:39:19.761142+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:43.251490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1634467840,"num_objects":24940,"num_object_clones":0,"num_object_copies":74820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24940,"num_write_kb":1596160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2558,"num_bytes_recovered":167641088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4824236032,"data_stored":4824236032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24940,"ondisk_log_size":24940,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":5289280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285927,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":994880,"kb_used_data":377840,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92750272,"statfs":{"total":95995035648,"available":94976278528,"internally_reserved":0,"allocated":386908160,"data_stored":386328582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":631830980},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131170,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758864,"kb_used_data":535804,"kb_used_omap":15,"kb_used_meta":223024,"kb_avail":92986288,"statfs":{"total":95995035648,"available":95217958912,"internally_reserved":0,"allocated":548663296,"data_stored":548079302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":228377148},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743792,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1310544,"kb_used_data":517180,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92434608,"statfs":{"total":95995035648,"available":94653038592,"internally_reserved":0,"allocated":529592320,"data_stored":529008326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":812382656},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005430,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":891616,"kb_used_data":328720,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92853536,"statfs":{"total":95995035648,"available":95082020864,"internally_reserved":0,"allocated":336609280,"data_stored":336017248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963906,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1219980,"kb_used_data":503740,"kb_used_omap":41,"kb_used_meta":716182,"kb_avail":92525172,"statfs":{"total":95995035648,"available":94745776128,"internally_reserved":0,"allocated":515829760,"data_stored":515245766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":733370675},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496003,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":703448,"kb_used_data":162756,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93041704,"statfs":{"total":95995035648,"available":95274704896,"internally_reserved":0,"allocated":166662144,"data_stored":166074598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553646534},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967648,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1076700,"kb_used_data":953992,"kb_used_omap":14,"kb_used_meta":122673,"kb_avail":92668452,"statfs":{"total":95995035648,"available":94892494848,"internally_reserved":0,"allocated":976887808,"data_stored":976299328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":125617781},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986570,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":786376,"kb_used_data":159608,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92958776,"statfs":{"total":95995035648,"available":95189786624,"internally_reserved":0,"allocated":163438592,"data_stored":162855482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150093,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411731,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":712844,"kb_used_data":86076,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93032308,"statfs":{"total":95995035648,"available":95265083392,"internally_reserved":0,"allocated":88141824,"data_stored":87557830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575257,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1298892,"kb_used_data":772092,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92446260,"statfs":{"total":95995035648,"available":94664970240,"internally_reserved":0,"allocated":790622208,"data_stored":790038214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":945744,"kb_used_data":160700,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92799408,"statfs":{"total":95995035648,"available":95026593792,"internally_reserved":0,"allocated":164556800,"data_stored":163972806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90767360,"data_stored":90767360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":800391168,"data_stored":800391168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89849856,"data_stored":89849856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":92798976,"data_stored":92798976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":171245568,"data_stored":171245568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":994902016,"data_stored":994902016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":172163072,"data_stored":172163072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":523108352,"data_stored":523108352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":334823424,"data_stored":334823424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":553844736,"data_stored":553844736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":590544896,"data_stored":590544896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":409796608,"data_stored":409796608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:39:26.931 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-17T16:39:26.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-17T16:39:26.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:26.984 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:26.989 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 29865 29849 81.1046 70.5 0.00873315 0.0123199 2024-09-17T16:39:27.006 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:27.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:27.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:27.075 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:39:27.107 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:27.129 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:27.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:27.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:27.251 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:27.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:27.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:27.356 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:27.376 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:27.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:27 smithi173 ceph-mon[30530]: pgmap v1935: 17 pgs: 17 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-17T16:39:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:27 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:39:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:27 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:39:27.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:27 smithi173 ceph-mon[30530]: osdmap e878: 12 total, 12 up, 11 in 2024-09-17T16:39:27.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:27 smithi017 ceph-mon[27571]: pgmap v1935: 17 pgs: 17 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-17T16:39:27.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:27 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:39:27.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:27 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:39:27.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:27 smithi017 ceph-mon[27571]: osdmap e878: 12 total, 12 up, 11 in 2024-09-17T16:39:27.478 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:27.503 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:27.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:27 smithi071 ceph-mon[30002]: pgmap v1935: 17 pgs: 17 active+clean; 1.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-17T16:39:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:27 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:39:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:27 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:39:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:27 smithi071 ceph-mon[30002]: osdmap e878: 12 total, 12 up, 11 in 2024-09-17T16:39:27.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:27.635 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:27.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:27.639 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:27.736 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:27.763 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:27.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:27.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:27.891 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:27.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:27.907 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:27.907 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:27.989 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 31050 31034 80.8108 74.0625 0.0019567 0.0123516 2024-09-17T16:39:27.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:28.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:28.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:28.123 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:28.149 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:28.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:28.183 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:39:28.221 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:39:28.251 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:28.275 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:28.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:28.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:28.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:28.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:28.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:28.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/681938929' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:28.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:28.507 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/681938929' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:28.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:28.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:28.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/681938929' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:28.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:28.660 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:28.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:28.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:39:28.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:28.788 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:28.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:28.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:28.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:28.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:28.989 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 32452 32436 81.0829 87.625 0.0155075 0.0123267 2024-09-17T16:39:29.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:29.052 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:29.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:29.074 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:29.150 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:29.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:29.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:29.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:29.221 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 1 pg_num to 11 2024-09-17T16:39:29.289 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:29.316 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:29.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:29.418 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:29.444 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:29.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:29.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:29 smithi017 ceph-mon[27571]: pgmap v1937: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-17T16:39:29.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:29 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:39:29.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:29 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:39:29.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/681938929' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:29.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:29 smithi017 ceph-mon[27571]: osdmap e879: 12 total, 12 up, 11 in 2024-09-17T16:39:29.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/321641529' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-17T16:39:29.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:29 smithi173 ceph-mon[30530]: pgmap v1937: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-17T16:39:29.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:29 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:39:29.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:29 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:39:29.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/681938929' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:29.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:29 smithi173 ceph-mon[30530]: osdmap e879: 12 total, 12 up, 11 in 2024-09-17T16:39:29.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/321641529' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-17T16:39:29.546 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:29 smithi071 ceph-mon[30002]: pgmap v1937: 17 pgs: 17 active+clean; 1.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-17T16:39:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:29 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:39:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:29 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:39:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/681938929' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:29 smithi071 ceph-mon[30002]: osdmap e879: 12 total, 12 up, 11 in 2024-09-17T16:39:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/321641529' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-17T16:39:29.575 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:29.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:29.661 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:29.661 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:29.676 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:29.701 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:29.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:29.803 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:29.831 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:29.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:29.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:29.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:29.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:29.989 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 34191 34175 82.1443 108.688 0.0134981 0.012167 2024-09-17T16:39:30.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:30.097 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:30.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:30.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:30.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:30.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:30.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:30.361 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:30.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/321641529' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-17T16:39:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:30 smithi173 ceph-mon[30530]: osdmap e880: 12 total, 12 up, 11 in 2024-09-17T16:39:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:30 smithi173 ceph-mon[30530]: osdmap e881: 12 total, 12 up, 11 in 2024-09-17T16:39:30.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-17T16:39:30.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/321641529' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-17T16:39:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:30 smithi017 ceph-mon[27571]: osdmap e880: 12 total, 12 up, 11 in 2024-09-17T16:39:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:30 smithi017 ceph-mon[27571]: osdmap e881: 12 total, 12 up, 11 in 2024-09-17T16:39:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-17T16:39:30.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:30.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:30.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:30.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:39:30.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/321641529' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-17T16:39:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:30 smithi071 ceph-mon[30002]: osdmap e880: 12 total, 12 up, 11 in 2024-09-17T16:39:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:30 smithi071 ceph-mon[30002]: osdmap e881: 12 total, 12 up, 11 in 2024-09-17T16:39:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-17T16:39:30.567 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:39:30.568 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:39:30.595 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:39:30.596 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:39:30.602 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:30.625 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:39:30.625 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:39:30.628 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:30.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:30.649 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:39:30.649 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:39:30.673 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:39:30.673 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:39:30.697 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:39:30.697 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:39:30.722 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:39:30.722 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:39:30.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:30.747 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:39:30.748 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:39:30.756 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:30.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:30.774 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:39:30.774 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:39:30.804 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:39:30.804 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:39:30.829 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:39:30.829 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:39:30.856 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:39:30.857 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:39:30.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:30.885 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:30.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:30.915 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:30.987 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:30.989 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 36043 36027 83.3883 115.75 0.0200419 0.0119867 2024-09-17T16:39:31.016 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:31.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:31.055 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:39:31.118 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:31.144 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:31.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:31.245 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:31.270 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:31.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:31.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:31 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:39:31.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:31 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:39:31.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:31 smithi173 ceph-mon[30530]: pgmap v1941: 17 pgs: 17 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-17T16:39:31.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/300993648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:31.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:31 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-17T16:39:31.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:31 smithi173 ceph-mon[30530]: osdmap e882: 12 total, 12 up, 11 in 2024-09-17T16:39:31.371 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:31.396 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:31.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:31.462 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:39:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:39:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:36440 9.8 9:125dae6d:::benchmark_data_smithi173_150363_object36439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e882)", 2024-09-17T16:39:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:31.449721+0000", 2024-09-17T16:39:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0094553150000000006, 2024-09-17T16:39:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0094897100000000002, 2024-09-17T16:39:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36440 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.449721+0000", 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.449721+0000", 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.449723+0000", 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.705e-06 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.449745+0000", 2024-09-17T16:39:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1508000000000001e-05 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.449755+0000", 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0716e-05 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.449763+0000", 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0749999999999998e-06 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.451551+0000", 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001787369 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.451681+0000", 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000130691 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.451814+0000", 2024-09-17T16:39:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132803 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453298+0000", 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014838570000000001 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:36448 9.8 9:171823b9:::benchmark_data_smithi173_150363_object36447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e882)", 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:31.453698+0000", 2024-09-17T16:39:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0054783360000000003, 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055967109999999999, 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36448 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453698+0000", 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453698+0000", 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453700+0000", 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8059999999999999e-06 2024-09-17T16:39:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453732+0000", 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1479999999999997e-05 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453748+0000", 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6847999999999999e-05 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453757+0000", 2024-09-17T16:39:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1100000000000003e-06 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453837+0000", 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0786999999999997e-05 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.454039+0000", 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000201444 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:39:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.454381+0000", 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034233400000000002 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.459304+0000", 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049224510000000004 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:36449 9.1 9:858be4d2:::benchmark_data_smithi173_150363_object36448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e882)", 2024-09-17T16:39:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:31.453779+0000", 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0053978419999999999, 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055901630000000004, 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36449 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453779+0000", 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453779+0000", 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453779+0000", 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1200000000000003e-07 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453800+0000", 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0565999999999999e-05 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453816+0000", 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6313000000000001e-05 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453825+0000", 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7849999999999997e-06 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.453873+0000", 2024-09-17T16:39:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7889000000000002e-05 2024-09-17T16:39:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.454048+0000", 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017543699999999999 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,10", 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455690+0000", 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001641909 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.459097+0000", 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034070070000000001 2024-09-17T16:39:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.459109+0000", 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2125e-05 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.459143+0000", 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3154999999999997e-05 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.459368+0000", 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022494400000000001 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27874.0:36445 9.3 e882/870 9:ccdb5c45:::benchmark_data_smithi173_150363_object36444:head v 882'2331, mlcod=882'2329)", 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:31.454042+0000", 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0051344279999999999, 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054079460000000003, 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.454042+0000", 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.454042+0000", 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.454043+0000", 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7800000000000001e-07 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.454111+0000", 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7731000000000006e-05 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455476+0000", 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013651679999999999 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455649+0000", 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017339300000000001 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455743+0000", 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4036999999999994e-05 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455766+0000", 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2410000000000001e-05 2024-09-17T16:39:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:36452 9.1 9:8a62d0b2:::benchmark_data_smithi173_150363_object36451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e882)", 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:31.455966+0000", 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0032104960000000002, 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035349159999999999, 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:39:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36452 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455966+0000", 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455966+0000", 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455968+0000", 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.451e-06 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455987+0000", 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9389999999999999e-05 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455998+0000", 2024-09-17T16:39:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0875e-05 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.456006+0000", 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6680000000000001e-06 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457667+0000", 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016616199999999999 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457776+0000", 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010888899999999999 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,10", 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457903+0000", 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000127467 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27874.0:36451 9.3 e882/870 9:c673db57:::benchmark_data_smithi173_150363_object36450:head v 882'2332, mlcod=882'2329)", 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:31.455403+0000", 2024-09-17T16:39:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.003773439, 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041577209999999996, 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455403+0000", 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455403+0000", 2024-09-17T16:39:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455404+0000", 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7199999999999997e-07 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.455456+0000", 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1668000000000001e-05 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.456321+0000", 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000864966 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.456327+0000", 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0399999999999998e-06 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.458529+0000", 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002202046 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.458550+0000", 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1327000000000001e-05 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:36455 9.8 9:1bc55e95:::benchmark_data_smithi173_150363_object36454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e882)", 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:31.457129+0000", 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.002048008, 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024635350000000002, 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36455 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457129+0000", 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457129+0000", 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457130+0000", 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2270000000000001e-06 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457153+0000", 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3031999999999999e-05 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457168+0000", 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5177999999999999e-05 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457176+0000", 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5789999999999998e-06 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457238+0000", 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1931e-05 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457375+0000", 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013722499999999999 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.457494+0000", 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119554 2024-09-17T16:39:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:31.459308+0000", 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001814136 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 7 2024-09-17T16:39:31.496 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:39:31.497 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:31 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:39:31.497 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:31 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:39:31.497 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:31 smithi017 ceph-mon[27571]: pgmap v1941: 17 pgs: 17 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-17T16:39:31.497 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/300993648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:31.497 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:31 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-17T16:39:31.497 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:31 smithi017 ceph-mon[27571]: osdmap e882: 12 total, 12 up, 11 in 2024-09-17T16:39:31.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:31.524 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:31.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:31 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:39:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:31 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:39:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:31 smithi071 ceph-mon[30002]: pgmap v1941: 17 pgs: 17 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-17T16:39:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/300993648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:31 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-17T16:39:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:31 smithi071 ceph-mon[30002]: osdmap e882: 12 total, 12 up, 11 in 2024-09-17T16:39:31.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:31.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:31.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:31.674 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:31.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:31.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:31.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:31.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:31.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:31.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:31.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:31.989 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 37198 37182 82.988 72.1875 0.0020059 0.0120406 2024-09-17T16:39:32.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:32.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:32.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:32.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:32.165 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:32.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:32.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:32.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:32.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:32.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:32.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:39:32.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:32.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:32.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:32.442 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-17T16:39:32.441+0000 7fa0c1d9b640 0 --2- 172.21.15.17:0/2541683039 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7fa0c4107670 0x7fa0c4198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T16:39:32.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:32.563 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:32.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:32.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:32.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:32.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:32.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:32.718 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:32.784 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:39:32.800 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:32.829 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:32.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:32.930 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:32.960 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:32.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:32.990 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 38809 38793 83.5976 100.688 0.013316 0.0119557 2024-09-17T16:39:33.043 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:32 smithi173 ceph-mon[30530]: osdmap e883: 12 total, 12 up, 11 in 2024-09-17T16:39:33.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:32 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:39:33.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:32 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:39:33.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:32 smithi173 ceph-mon[30530]: pgmap v1944: 27 pgs: 10 unknown, 17 active+clean; 2.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 61 MiB/s wr, 974 op/s 2024-09-17T16:39:33.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:32 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-17T16:39:33.044 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2541683039' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:39:33.061 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:33.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:32 smithi071 ceph-mon[30002]: osdmap e883: 12 total, 12 up, 11 in 2024-09-17T16:39:33.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:32 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:39:33.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:32 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:39:33.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:32 smithi071 ceph-mon[30002]: pgmap v1944: 27 pgs: 10 unknown, 17 active+clean; 2.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 61 MiB/s wr, 974 op/s 2024-09-17T16:39:33.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:32 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-17T16:39:33.065 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2541683039' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:39:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:33.085 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:33.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:33.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:32 smithi017 ceph-mon[27571]: osdmap e883: 12 total, 12 up, 11 in 2024-09-17T16:39:33.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:32 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:39:33.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:32 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:39:33.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:32 smithi017 ceph-mon[27571]: pgmap v1944: 27 pgs: 10 unknown, 17 active+clean; 2.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 61 MiB/s wr, 974 op/s 2024-09-17T16:39:33.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:32 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-17T16:39:33.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2541683039' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:39:33.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:33.212 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:33.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:33.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:33.322 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:39:33.322 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:39:33.323 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:39:33.323 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:39:33.323 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:39:33.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:33.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:33.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:33.451 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:33.482 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:33.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:33.583 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:33.604 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:39:33.608 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:33.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:33.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:33.738 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:33.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:33.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:33.840 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:33.864 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:33.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:33.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:33.918 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:33.966 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:33 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:33 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-17T16:39:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2541683039' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:39:33.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:33 smithi017 ceph-mon[27571]: osdmap e884: 12 total, 12 up, 11 in 2024-09-17T16:39:33.989 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 40360 40344 84.042 96.9375 0.00881098 0.011888 2024-09-17T16:39:33.990 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:33.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:34.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:33 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:34.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:33 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-17T16:39:34.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2541683039' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:39:34.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:33 smithi071 ceph-mon[30002]: osdmap e884: 12 total, 12 up, 11 in 2024-09-17T16:39:34.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:34.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:34.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:34.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:34.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:33 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:33 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-17T16:39:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2541683039' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:39:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:33 smithi173 ceph-mon[30530]: osdmap e884: 12 total, 12 up, 11 in 2024-09-17T16:39:34.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:34.240 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:34.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:34.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:34.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:34.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:34.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:34.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:34.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:34.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:34.603 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:34.628 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:34.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:34.730 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:34.757 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:34.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:34.859 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:34.883 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:34.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:34.985 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:34.989 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 41745 41729 84.1231 86.5625 0.0192937 0.0118803 2024-09-17T16:39:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:35.010 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:35.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:35.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:34 smithi071 ceph-mon[30002]: osdmap e885: 12 total, 12 up, 11 in 2024-09-17T16:39:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:34 smithi071 ceph-mon[30002]: 1.9 scrub starts 2024-09-17T16:39:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:34 smithi071 ceph-mon[30002]: 1.7 scrub starts 2024-09-17T16:39:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:34 smithi071 ceph-mon[30002]: pgmap v1947: 27 pgs: 1 peering, 9 unknown, 17 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 1.5 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 23 KiB/s, 0 objects/s recovering 2024-09-17T16:39:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/629982168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:35.112 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:35.138 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:35.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:35.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:34 smithi173 ceph-mon[30530]: osdmap e885: 12 total, 12 up, 11 in 2024-09-17T16:39:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:34 smithi173 ceph-mon[30530]: 1.9 scrub starts 2024-09-17T16:39:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:34 smithi173 ceph-mon[30530]: 1.7 scrub starts 2024-09-17T16:39:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:34 smithi173 ceph-mon[30530]: pgmap v1947: 27 pgs: 1 peering, 9 unknown, 17 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 1.5 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 23 KiB/s, 0 objects/s recovering 2024-09-17T16:39:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/629982168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:34 smithi017 ceph-mon[27571]: osdmap e885: 12 total, 12 up, 11 in 2024-09-17T16:39:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:34 smithi017 ceph-mon[27571]: 1.9 scrub starts 2024-09-17T16:39:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:34 smithi017 ceph-mon[27571]: 1.7 scrub starts 2024-09-17T16:39:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:34 smithi017 ceph-mon[27571]: pgmap v1947: 27 pgs: 1 peering, 9 unknown, 17 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 1.5 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 23 KiB/s, 0 objects/s recovering 2024-09-17T16:39:35.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/629982168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:35.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:35.259 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 8, 9, 11, 7, 10, 5] 2024-09-17T16:39:35.260 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:39:35.260 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:39:35.270 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:35.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:35.313 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:39:35.313 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:39:35.334 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:39:35.334 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:39:35.355 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:39:35.355 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:39:35.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:35.377 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:39:35.377 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:39:35.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:35.396 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:35.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:39:35.401 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:39:35.423 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:39:35.423 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:39:35.446 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:39:35.446 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:39:35.473 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:39:35.474 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:39:35.496 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:39:35.496 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:39:35.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:35.524 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:39:35.525 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:39:35.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:35.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:35.540 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:35.556 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:39:35.556 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:39:35.583 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:39:35.584 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:39:35.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:39:35.634 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:35.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:39:35.661 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:35.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:35.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:35.787 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:35.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:35.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:35 smithi173 ceph-mon[30530]: 1.8 scrub starts 2024-09-17T16:39:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:35 smithi173 ceph-mon[30530]: 1.3 scrub starts 2024-09-17T16:39:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:35 smithi173 ceph-mon[30530]: 1.a scrub starts 2024-09-17T16:39:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:35 smithi173 ceph-mon[30530]: 1.2 scrub starts 2024-09-17T16:39:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:35 smithi173 ceph-mon[30530]: 1.6 scrub starts 2024-09-17T16:39:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:35 smithi173 ceph-mon[30530]: 1.4 scrub starts 2024-09-17T16:39:35.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:35.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:35.988 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:35 smithi017 ceph-mon[27571]: 1.8 scrub starts 2024-09-17T16:39:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:35 smithi017 ceph-mon[27571]: 1.3 scrub starts 2024-09-17T16:39:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:35 smithi017 ceph-mon[27571]: 1.a scrub starts 2024-09-17T16:39:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:35 smithi017 ceph-mon[27571]: 1.2 scrub starts 2024-09-17T16:39:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:35 smithi017 ceph-mon[27571]: 1.6 scrub starts 2024-09-17T16:39:35.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:35 smithi017 ceph-mon[27571]: 1.4 scrub starts 2024-09-17T16:39:35.989 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 43052 43036 84.0467 81.6875 0.019827 0.0118893 2024-09-17T16:39:36.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:36.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:36.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:36.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:36.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:35 smithi071 ceph-mon[30002]: 1.8 scrub starts 2024-09-17T16:39:36.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:35 smithi071 ceph-mon[30002]: 1.3 scrub starts 2024-09-17T16:39:36.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:35 smithi071 ceph-mon[30002]: 1.a scrub starts 2024-09-17T16:39:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:35 smithi071 ceph-mon[30002]: 1.2 scrub starts 2024-09-17T16:39:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:35 smithi071 ceph-mon[30002]: 1.6 scrub starts 2024-09-17T16:39:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:35 smithi071 ceph-mon[30002]: 1.4 scrub starts 2024-09-17T16:39:36.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:36.178 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:36.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:36.280 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:36.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:36.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:36.410 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:36.436 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:36.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:36.539 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:36.564 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:36.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:36.575 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:36.576 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:36.665 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:36.690 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:36.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:36.793 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:36.818 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:36.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:36.920 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:36.945 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:36.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:36.990 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 15 43936 43921 83.1758 55.3125 0.0496788 0.0119972 2024-09-17T16:39:37.046 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:37.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:36 smithi071 ceph-mon[30002]: 1.5 scrub starts 2024-09-17T16:39:37.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:36 smithi071 ceph-mon[30002]: pgmap v1948: 27 pgs: 1 peering, 6 unknown, 20 active+clean; 2.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 89 MiB/s wr, 1.41k op/s 2024-09-17T16:39:37.072 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:37.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:37.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:36 smithi173 ceph-mon[30530]: 1.5 scrub starts 2024-09-17T16:39:37.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:36 smithi173 ceph-mon[30530]: pgmap v1948: 27 pgs: 1 peering, 6 unknown, 20 active+clean; 2.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 89 MiB/s wr, 1.41k op/s 2024-09-17T16:39:37.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:37.198 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:37.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:37.231 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:36 smithi017 ceph-mon[27571]: 1.5 scrub starts 2024-09-17T16:39:37.231 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:36 smithi017 ceph-mon[27571]: pgmap v1948: 27 pgs: 1 peering, 6 unknown, 20 active+clean; 2.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 89 MiB/s wr, 1.41k op/s 2024-09-17T16:39:37.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:39:37.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:37.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:37.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:37.401 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:37.430 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:37.454 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:37.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:37.556 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:37.564 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:37.566 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:39:37.567 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:39:37.581 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:37.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:37.683 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:37.709 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:37.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:37.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:37.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:37.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:37.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:37.937 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:37.963 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:37.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:37.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:37.990 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 45179 45163 83.0124 77.625 0.0123362 0.0120406 2024-09-17T16:39:38.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:37 smithi173 ceph-mon[30530]: 1.1 scrub starts 2024-09-17T16:39:38.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:37 smithi173 ceph-mon[30530]: from='client.18555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:39:38.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:37 smithi071 ceph-mon[30002]: 1.1 scrub starts 2024-09-17T16:39:38.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:37 smithi071 ceph-mon[30002]: from='client.18555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:39:38.065 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:38.089 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:38.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:38.097 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:37 smithi017 ceph-mon[27571]: 1.1 scrub starts 2024-09-17T16:39:38.097 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:37 smithi017 ceph-mon[27571]: from='client.18555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:39:38.191 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:38.218 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:38.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:38.307 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:38.307 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:38.319 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:38.340 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:38.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:38.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:38.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:38.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:38.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:38.561 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":1948,"stamp":"2024-09-17T16:39:36.143066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2576434840,"num_objects":39310,"num_object_clones":0,"num_object_copies":117930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39310,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":39354,"num_write_kb":2516621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2560,"num_bytes_recovered":168458522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39770,"ondisk_log_size":39770,"up":63,"acting":63,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":13671680,"kb_used_data":7488588,"kb_used_omap":83,"kb_used_meta":6182764,"kb_avail":1017524992,"statfs":{"total":1055945392128,"available":1041945591808,"internally_reserved":0,"allocated":7668314112,"data_stored":7661473476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86009,"internal_metadata":6331150343},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":499860600,"num_objects":7631,"num_object_clones":0,"num_object_copies":22893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7631,"num_whiteouts":0,"num_read":-48,"num_read_kb":-40,"num_write":7554,"num_write_kb":487143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-240422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.365099"},"pg_stats":[{"pgid":"1.7","version":"21'92","reported_seq":41,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:34.034859+0000","last_change":"2024-09-17T16:39:35.769113+0000","last_active":"2024-09-17T16:39:34.034859+0000","last_peered":"2024-09-17T16:39:34.034859+0000","last_clean":"2024-09-17T16:39:34.034859+0000","last_became_active":"2024-09-17T16:39:33.798225+0000","last_became_peered":"2024-09-17T16:39:33.798225+0000","last_unstale":"2024-09-17T16:39:34.034859+0000","last_undegraded":"2024-09-17T16:39:34.034859+0000","last_fullsized":"2024-09-17T16:39:34.034859+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:40:04.034934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"885'2553","reported_seq":2617,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:34.596709+0000","last_change":"2024-09-17T16:38:54.766586+0000","last_active":"2024-09-17T16:39:34.596709+0000","last_peered":"2024-09-17T16:39:34.596709+0000","last_clean":"2024-09-17T16:39:34.596709+0000","last_became_active":"2024-09-17T16:38:54.765274+0000","last_became_peered":"2024-09-17T16:38:54.765274+0000","last_unstale":"2024-09-17T16:39:34.596709+0000","last_undegraded":"2024-09-17T16:39:34.596709+0000","last_fullsized":"2024-09-17T16:39:34.596709+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2553,"log_dups_size":0,"ondisk_log_size":2553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:15.671182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167313408,"num_objects":2553,"num_object_clones":0,"num_object_copies":7659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2553,"num_write_kb":163392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:39:30.771193+0000","last_change":"2024-09-17T16:39:30.771193+0000","last_active":"2024-09-17T16:39:30.771193+0000","last_peered":"2024-09-17T16:39:30.771193+0000","last_clean":"2024-09-17T16:39:30.771193+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:39:30.771193+0000","last_undegraded":"2024-09-17T16:39:30.771193+0000","last_fullsized":"2024-09-17T16:39:30.771193+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_clean_scrub_stamp":"2024-09-17T16:39:30.771193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"883'2357","reported_seq":2418,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-09-17T16:39:32.279649+0000","last_change":"2024-09-17T16:38:54.765568+0000","last_active":"2024-09-17T16:39:32.279649+0000","last_peered":"2024-09-17T16:39:32.279649+0000","last_clean":"2024-09-17T16:39:32.279649+0000","last_became_active":"2024-09-17T16:38:54.765252+0000","last_became_peered":"2024-09-17T16:38:54.765252+0000","last_unstale":"2024-09-17T16:39:32.279649+0000","last_undegraded":"2024-09-17T16:39:32.279649+0000","last_fullsized":"2024-09-17T16:39:32.279649+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2357,"log_dups_size":0,"ondisk_log_size":2357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:55.899749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154468352,"num_objects":2357,"num_object_clones":0,"num_object_copies":7071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2357,"num_write_kb":150848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:39:30.771193+0000","last_change":"2024-09-17T16:39:30.771193+0000","last_active":"2024-09-17T16:39:30.771193+0000","last_peered":"2024-09-17T16:39:30.771193+0000","last_clean":"2024-09-17T16:39:30.771193+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:39:30.771193+0000","last_undegraded":"2024-09-17T16:39:30.771193+0000","last_fullsized":"2024-09-17T16:39:30.771193+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_clean_scrub_stamp":"2024-09-17T16:39:30.771193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"884'2389","reported_seq":2451,"reported_epoch":884,"state":"active+clean","last_fresh":"2024-09-17T16:39:33.155369+0000","last_change":"2024-09-17T16:38:54.759655+0000","last_active":"2024-09-17T16:39:33.155369+0000","last_peered":"2024-09-17T16:39:33.155369+0000","last_clean":"2024-09-17T16:39:33.155369+0000","last_became_active":"2024-09-17T16:38:54.759320+0000","last_became_peered":"2024-09-17T16:38:54.759320+0000","last_unstale":"2024-09-17T16:39:33.155369+0000","last_undegraded":"2024-09-17T16:39:33.155369+0000","last_fullsized":"2024-09-17T16:39:33.155369+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2389,"log_dups_size":0,"ondisk_log_size":2389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:11.394766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156565504,"num_objects":2389,"num_object_clones":0,"num_object_copies":7167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2389,"num_write_kb":152896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":20,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:34.345573+0000","last_change":"2024-09-17T16:39:34.345182+0000","last_active":"2024-09-17T16:39:34.345573+0000","last_peered":"2024-09-17T16:39:34.345573+0000","last_clean":"2024-09-17T16:39:34.345573+0000","last_became_active":"2024-09-17T16:39:33.795955+0000","last_became_peered":"2024-09-17T16:39:33.795955+0000","last_unstale":"2024-09-17T16:39:34.345573+0000","last_undegraded":"2024-09-17T16:39:34.345573+0000","last_fullsized":"2024-09-17T16:39:34.345573+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":375765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.c","version":"885'2664","reported_seq":3867,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:35.050249+0000","last_change":"2024-09-17T16:39:15.770878+0000","last_active":"2024-09-17T16:39:35.050249+0000","last_peered":"2024-09-17T16:39:35.050249+0000","last_clean":"2024-09-17T16:39:35.050249+0000","last_became_active":"2024-09-17T16:39:15.769803+0000","last_became_peered":"2024-09-17T16:39:15.769803+0000","last_unstale":"2024-09-17T16:39:35.050249+0000","last_undegraded":"2024-09-17T16:39:35.050249+0000","last_fullsized":"2024-09-17T16:39:35.050249+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2664,"log_dups_size":0,"ondisk_log_size":2664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:39:58.458126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174587904,"num_objects":2664,"num_object_clones":0,"num_object_copies":7992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2664,"num_write_kb":170496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":564,"num_bytes_recovered":36962304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:39:30.771193+0000","last_change":"2024-09-17T16:39:30.771193+0000","last_active":"2024-09-17T16:39:30.771193+0000","last_peered":"2024-09-17T16:39:30.771193+0000","last_clean":"2024-09-17T16:39:30.771193+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:39:30.771193+0000","last_undegraded":"2024-09-17T16:39:30.771193+0000","last_fullsized":"2024-09-17T16:39:30.771193+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_clean_scrub_stamp":"2024-09-17T16:39:30.771193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"885'2562","reported_seq":2627,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:34.321553+0000","last_change":"2024-09-17T16:38:54.766769+0000","last_active":"2024-09-17T16:39:34.321553+0000","last_peered":"2024-09-17T16:39:34.321553+0000","last_clean":"2024-09-17T16:39:34.321553+0000","last_became_active":"2024-09-17T16:38:54.766429+0000","last_became_peered":"2024-09-17T16:38:54.766429+0000","last_unstale":"2024-09-17T16:39:34.321553+0000","last_undegraded":"2024-09-17T16:39:34.321553+0000","last_fullsized":"2024-09-17T16:39:34.321553+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2562,"log_dups_size":0,"ondisk_log_size":2562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:23.558032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167903232,"num_objects":2562,"num_object_clones":0,"num_object_copies":7686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2562,"num_write_kb":163968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:39:30.771193+0000","last_change":"2024-09-17T16:39:30.771193+0000","last_active":"2024-09-17T16:39:30.771193+0000","last_peered":"2024-09-17T16:39:30.771193+0000","last_clean":"2024-09-17T16:39:30.771193+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:39:30.771193+0000","last_undegraded":"2024-09-17T16:39:30.771193+0000","last_fullsized":"2024-09-17T16:39:30.771193+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_clean_scrub_stamp":"2024-09-17T16:39:30.771193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"885'2584","reported_seq":2649,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:34.586387+0000","last_change":"2024-09-17T16:38:54.760350+0000","last_active":"2024-09-17T16:39:34.586387+0000","last_peered":"2024-09-17T16:39:34.586387+0000","last_clean":"2024-09-17T16:39:34.586387+0000","last_became_active":"2024-09-17T16:38:54.760031+0000","last_became_peered":"2024-09-17T16:38:54.760031+0000","last_unstale":"2024-09-17T16:39:34.586387+0000","last_undegraded":"2024-09-17T16:39:34.586387+0000","last_fullsized":"2024-09-17T16:39:34.586387+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2584,"log_dups_size":0,"ondisk_log_size":2584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:12.669832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169345024,"num_objects":2584,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2584,"num_write_kb":165376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:39:30.771193+0000","last_change":"2024-09-17T16:39:30.771193+0000","last_active":"2024-09-17T16:39:30.771193+0000","last_peered":"2024-09-17T16:39:30.771193+0000","last_clean":"2024-09-17T16:39:30.771193+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:39:30.771193+0000","last_undegraded":"2024-09-17T16:39:30.771193+0000","last_fullsized":"2024-09-17T16:39:30.771193+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_clean_scrub_stamp":"2024-09-17T16:39:30.771193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"883'2298","reported_seq":2359,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-09-17T16:39:32.264302+0000","last_change":"2024-09-17T16:38:54.764119+0000","last_active":"2024-09-17T16:39:32.264302+0000","last_peered":"2024-09-17T16:39:32.264302+0000","last_clean":"2024-09-17T16:39:32.264302+0000","last_became_active":"2024-09-17T16:38:54.763749+0000","last_became_peered":"2024-09-17T16:38:54.763749+0000","last_unstale":"2024-09-17T16:39:32.264302+0000","last_undegraded":"2024-09-17T16:39:32.264302+0000","last_fullsized":"2024-09-17T16:39:32.264302+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2298,"log_dups_size":0,"ondisk_log_size":2298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:06.640020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150601728,"num_objects":2298,"num_object_clones":0,"num_object_copies":6894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2298,"num_write_kb":147072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"883'2305","reported_seq":3336,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-09-17T16:39:32.279776+0000","last_change":"2024-09-17T16:39:13.779432+0000","last_active":"2024-09-17T16:39:32.279776+0000","last_peered":"2024-09-17T16:39:32.279776+0000","last_clean":"2024-09-17T16:39:32.279776+0000","last_became_active":"2024-09-17T16:39:13.777102+0000","last_became_peered":"2024-09-17T16:39:13.777102+0000","last_unstale":"2024-09-17T16:39:32.279776+0000","last_undegraded":"2024-09-17T16:39:32.279776+0000","last_fullsized":"2024-09-17T16:39:32.279776+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2305,"log_dups_size":0,"ondisk_log_size":2305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:13.911789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151060480,"num_objects":2305,"num_object_clones":0,"num_object_copies":6915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2305,"num_write_kb":147520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":31391744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"885'2548","reported_seq":2613,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:34.599019+0000","last_change":"2024-09-17T16:38:54.766597+0000","last_active":"2024-09-17T16:39:34.599019+0000","last_peered":"2024-09-17T16:39:34.599019+0000","last_clean":"2024-09-17T16:39:34.599019+0000","last_became_active":"2024-09-17T16:38:54.765661+0000","last_became_peered":"2024-09-17T16:38:54.765661+0000","last_unstale":"2024-09-17T16:39:34.599019+0000","last_undegraded":"2024-09-17T16:39:34.599019+0000","last_fullsized":"2024-09-17T16:39:34.599019+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2548,"log_dups_size":0,"ondisk_log_size":2548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:09.402795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166985728,"num_objects":2548,"num_object_clones":0,"num_object_copies":7644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2548,"num_write_kb":163072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"885'2677","reported_seq":2740,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:34.606036+0000","last_change":"2024-09-17T16:38:54.763911+0000","last_active":"2024-09-17T16:39:34.606036+0000","last_peered":"2024-09-17T16:39:34.606036+0000","last_clean":"2024-09-17T16:39:34.606036+0000","last_became_active":"2024-09-17T16:38:54.763476+0000","last_became_peered":"2024-09-17T16:38:54.763476+0000","last_unstale":"2024-09-17T16:39:34.606036+0000","last_undegraded":"2024-09-17T16:39:34.606036+0000","last_fullsized":"2024-09-17T16:39:34.606036+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2677,"log_dups_size":0,"ondisk_log_size":2677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:00.851346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175439872,"num_objects":2677,"num_object_clones":0,"num_object_copies":8031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2677,"num_write_kb":171328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"883'2455","reported_seq":2515,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-09-17T16:39:32.702653+0000","last_change":"2024-09-17T16:38:54.763053+0000","last_active":"2024-09-17T16:39:32.702653+0000","last_peered":"2024-09-17T16:39:32.702653+0000","last_clean":"2024-09-17T16:39:32.702653+0000","last_became_active":"2024-09-17T16:38:54.762670+0000","last_became_peered":"2024-09-17T16:38:54.762670+0000","last_unstale":"2024-09-17T16:39:32.702653+0000","last_undegraded":"2024-09-17T16:39:32.702653+0000","last_fullsized":"2024-09-17T16:39:32.702653+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2455,"log_dups_size":0,"ondisk_log_size":2455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:10.270523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160890880,"num_objects":2455,"num_object_clones":0,"num_object_copies":7365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2455,"num_write_kb":157120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"885'2642","reported_seq":3682,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:34.589282+0000","last_change":"2024-09-17T16:39:13.777092+0000","last_active":"2024-09-17T16:39:34.589282+0000","last_peered":"2024-09-17T16:39:34.589282+0000","last_clean":"2024-09-17T16:39:34.589282+0000","last_became_active":"2024-09-17T16:39:13.775396+0000","last_became_peered":"2024-09-17T16:39:13.775396+0000","last_unstale":"2024-09-17T16:39:34.589282+0000","last_undegraded":"2024-09-17T16:39:34.589282+0000","last_fullsized":"2024-09-17T16:39:34.589282+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2642,"log_dups_size":0,"ondisk_log_size":2642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:14.159859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":7926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2642,"num_write_kb":169088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":31522816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T16:39:30.771193+0000","last_change":"2024-09-17T16:39:30.771193+0000","last_active":"2024-09-17T16:39:30.771193+0000","last_peered":"2024-09-17T16:39:30.771193+0000","last_clean":"2024-09-17T16:39:30.771193+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T16:39:30.771193+0000","last_undegraded":"2024-09-17T16:39:30.771193+0000","last_fullsized":"2024-09-17T16:39:30.771193+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:39:30.771193+0000","last_clean_scrub_stamp":"2024-09-17T16:39:30.771193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"883'2400","reported_seq":3478,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-09-17T16:39:32.691199+0000","last_change":"2024-09-17T16:39:13.778186+0000","last_active":"2024-09-17T16:39:32.691199+0000","last_peered":"2024-09-17T16:39:32.691199+0000","last_clean":"2024-09-17T16:39:32.691199+0000","last_became_active":"2024-09-17T16:39:13.776232+0000","last_became_peered":"2024-09-17T16:39:13.776232+0000","last_unstale":"2024-09-17T16:39:32.691199+0000","last_undegraded":"2024-09-17T16:39:32.691199+0000","last_fullsized":"2024-09-17T16:39:32.691199+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2400,"log_dups_size":0,"ondisk_log_size":2400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:22.136930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157286400,"num_objects":2400,"num_object_clones":0,"num_object_copies":7200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2400,"num_write_kb":153600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":503,"num_bytes_recovered":32964608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":4,"reported_epoch":884,"state":"peering","last_fresh":"2024-09-17T16:39:32.792226+0000","last_change":"2024-09-17T16:39:31.788506+0000","last_active":"2024-09-17T16:39:31.788506+0000","last_peered":"2024-09-17T16:39:29.770243+0000","last_clean":"2024-09-17T16:39:29.770243+0000","last_became_active":"2024-09-17T16:29:49.761309+0000","last_became_peered":"2024-09-17T16:29:49.761309+0000","last_unstale":"2024-09-17T16:39:32.792226+0000","last_undegraded":"2024-09-17T16:39:32.792226+0000","last_fullsized":"2024-09-17T16:39:32.792226+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":639,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"882'2345","reported_seq":3473,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-09-17T16:39:31.460274+0000","last_change":"2024-09-17T16:39:15.801706+0000","last_active":"2024-09-17T16:39:31.460274+0000","last_peered":"2024-09-17T16:39:31.460274+0000","last_clean":"2024-09-17T16:39:31.460274+0000","last_became_active":"2024-09-17T16:39:15.784630+0000","last_became_peered":"2024-09-17T16:39:15.784630+0000","last_unstale":"2024-09-17T16:39:31.460274+0000","last_undegraded":"2024-09-17T16:39:31.460274+0000","last_fullsized":"2024-09-17T16:39:31.460274+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2345,"log_dups_size":0,"ondisk_log_size":2345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:22.167302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153681920,"num_objects":2345,"num_object_clones":0,"num_object_copies":7035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2345,"num_write_kb":150080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":531,"num_bytes_recovered":34799616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":21,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:33.861224+0000","last_change":"2024-09-17T16:39:34.607980+0000","last_active":"2024-09-17T16:39:33.861224+0000","last_peered":"2024-09-17T16:39:33.861224+0000","last_clean":"2024-09-17T16:39:33.861224+0000","last_became_active":"2024-09-17T16:39:33.796846+0000","last_became_peered":"2024-09-17T16:39:33.796846+0000","last_unstale":"2024-09-17T16:39:33.861224+0000","last_undegraded":"2024-09-17T16:39:33.861224+0000","last_fullsized":"2024-09-17T16:39:33.861224+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:40:03.861291+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":297417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"883'2304","reported_seq":2365,"reported_epoch":883,"state":"active+clean","last_fresh":"2024-09-17T16:39:32.691389+0000","last_change":"2024-09-17T16:38:54.757271+0000","last_active":"2024-09-17T16:39:32.691389+0000","last_peered":"2024-09-17T16:39:32.691389+0000","last_clean":"2024-09-17T16:39:32.691389+0000","last_became_active":"2024-09-17T16:38:54.756978+0000","last_became_peered":"2024-09-17T16:38:54.756978+0000","last_unstale":"2024-09-17T16:39:32.691389+0000","last_undegraded":"2024-09-17T16:39:32.691389+0000","last_fullsized":"2024-09-17T16:39:32.691389+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2304,"log_dups_size":0,"ondisk_log_size":2304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:00.524133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":2304,"num_object_clones":0,"num_object_copies":6912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2304,"num_write_kb":147456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"882'2227","reported_seq":2285,"reported_epoch":882,"state":"active+clean","last_fresh":"2024-09-17T16:39:31.495842+0000","last_change":"2024-09-17T16:38:54.768093+0000","last_active":"2024-09-17T16:39:31.495842+0000","last_peered":"2024-09-17T16:39:31.495842+0000","last_clean":"2024-09-17T16:39:31.495842+0000","last_became_active":"2024-09-17T16:38:54.767749+0000","last_became_peered":"2024-09-17T16:38:54.767749+0000","last_unstale":"2024-09-17T16:39:31.495842+0000","last_undegraded":"2024-09-17T16:39:31.495842+0000","last_fullsized":"2024-09-17T16:39:31.495842+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":2227,"log_dups_size":0,"ondisk_log_size":2227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:08.386652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145948672,"num_objects":2227,"num_object_clones":0,"num_object_copies":6681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2227,"num_write_kb":142528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":560,"reported_epoch":885,"state":"active+clean","last_fresh":"2024-09-17T16:39:33.790340+0000","last_change":"2024-09-17T16:39:32.100237+0000","last_active":"2024-09-17T16:39:33.790340+0000","last_peered":"2024-09-17T16:39:33.790340+0000","last_clean":"2024-09-17T16:39:33.790340+0000","last_became_active":"2024-09-17T16:39:31.793364+0000","last_became_peered":"2024-09-17T16:39:31.793364+0000","last_unstale":"2024-09-17T16:39:33.790340+0000","last_undegraded":"2024-09-17T16:39:33.790340+0000","last_fullsized":"2024-09-17T16:39:33.790340+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:39:32.100164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:39:32.100164+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:38.274225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2576220160,"num_objects":39310,"num_object_clones":0,"num_object_copies":117930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39310,"num_write_kb":2515840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2558,"num_bytes_recovered":167641088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7741046784,"data_stored":7741046784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39310,"ondisk_log_size":39310,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":214680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":44,"num_write_kb":781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":817434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3194880,"data_stored":3173568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":460,"ondisk_log_size":460,"up":15,"acting":15,"num_store_stats":10}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285930,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1413444,"kb_used_data":804724,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92331708,"statfs":{"total":95995035648,"available":94547668992,"internally_reserved":0,"allocated":824037376,"data_stored":823409862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":623311300},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131172,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1138336,"kb_used_data":875024,"kb_used_omap":15,"kb_used_meta":263280,"kb_avail":92606816,"statfs":{"total":95995035648,"available":94829379584,"internally_reserved":0,"allocated":896024576,"data_stored":895410078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":269599292},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743794,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1680812,"kb_used_data":879120,"kb_used_omap":1,"kb_used_meta":801662,"kb_avail":92064340,"statfs":{"total":95995035648,"available":94273884160,"internally_reserved":0,"allocated":900218880,"data_stored":899604382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":820902336},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005432,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053396,"kb_used_data":498812,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92691756,"statfs":{"total":95995035648,"available":94916358144,"internally_reserved":0,"allocated":510783488,"data_stored":510152430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963908,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497732,"kb_used_data":777320,"kb_used_omap":41,"kb_used_meta":720342,"kb_avail":92247420,"statfs":{"total":95995035648,"available":94461358080,"internally_reserved":0,"allocated":795975680,"data_stored":795352596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":737630515},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496006,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865992,"kb_used_data":333620,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92879160,"statfs":{"total":95995035648,"available":95108259840,"internally_reserved":0,"allocated":341626880,"data_stored":340999374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":545126854},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967650,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1587536,"kb_used_data":1450940,"kb_used_omap":14,"kb_used_meta":136561,"kb_avail":92157616,"statfs":{"total":95995035648,"available":94369398784,"internally_reserved":0,"allocated":1485762560,"data_stored":1485131502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":139839093},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986573,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":954188,"kb_used_data":331572,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92790964,"statfs":{"total":95995035648,"available":95017947136,"internally_reserved":0,"allocated":339529728,"data_stored":338902222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150095,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411733,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":775208,"kb_used_data":144272,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92969944,"statfs":{"total":95995035648,"available":95201222656,"internally_reserved":0,"allocated":147734528,"data_stored":147120030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575259,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1716908,"kb_used_data":1194268,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92028244,"statfs":{"total":95995035648,"available":94236921856,"internally_reserved":0,"allocated":1222930432,"data_stored":1222311456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738783,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988128,"kb_used_data":198916,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92757024,"statfs":{"total":95995035648,"available":94983192576,"internally_reserved":0,"allocated":203689984,"data_stored":203079544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":808122821},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151453696,"data_stored":151453696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1248460800,"data_stored":1248460800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":148045824,"data_stored":148045824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":349110272,"data_stored":349110272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1513619456,"data_stored":1513619456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":347144192,"data_stored":347144192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":801898496,"data_stored":801898496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":509607936,"data_stored":509607936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":905183232,"data_stored":905183232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":918552576,"data_stored":918552576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":843251712,"data_stored":843251712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:39:38.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:38.606 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:38.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:38.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:38.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:38.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:38.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:38.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:38.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:38.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:38 smithi173 ceph-mon[30530]: pgmap v1949: 27 pgs: 1 peering, 26 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s; 52 KiB/s, 0 objects/s recovering 2024-09-17T16:39:38.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:38.990 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 46566 46550 83.117 86.6875 0.00297415 0.0120255 2024-09-17T16:39:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:39.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:39.050 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:39:39.075 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:38 smithi017 ceph-mon[27571]: pgmap v1949: 27 pgs: 1 peering, 26 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s; 52 KiB/s, 0 objects/s recovering 2024-09-17T16:39:39.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:39:39.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:39.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:39.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:39.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:39.305 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:39.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:39.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:38 smithi071 ceph-mon[30002]: pgmap v1949: 27 pgs: 1 peering, 26 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s; 52 KiB/s, 0 objects/s recovering 2024-09-17T16:39:39.349 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:39.407 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:39.430 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:39.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:39.448 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:39.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:39.556 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:39.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:39.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:39.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:39.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:39.789 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:39.815 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:39.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:39.824 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:39:39.916 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:39:39.916 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:39:39.916 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:39.936 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:39:39.937 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:39:39.937 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:39.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:39.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:39.957 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:39:39.957 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:39:39.978 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:39:39.978 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:39:39.990 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 47701 47685 82.7786 70.9375 0.150542 0.0120218 2024-09-17T16:39:40.000 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:39:40.000 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:39:40.021 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:39:40.022 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:39:40.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:40.045 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:39:40.046 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:39:40.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4087064618' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:40.053 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4087064618' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:40.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:40.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:40.072 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:39:40.072 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:39:40.095 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:39:40.096 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:39:40.122 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:39:40.122 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:39:40.150 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:39:40.150 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:39:40.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:40.176 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:39:40.176 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:39:40.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:40.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:40.215 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4087064618' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:40.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:40.318 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:40.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:40.319 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:40.334 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:40.334 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:40.420 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:40.445 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:40.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:40.547 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:40.572 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:40.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:40.674 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:40.698 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:40.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:40.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:40.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:40.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:40 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4087064618' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:40 smithi173 ceph-mon[30530]: osdmap e886: 12 total, 12 up, 11 in 2024-09-17T16:39:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2627937217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:40.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:40 smithi173 ceph-mon[30530]: pgmap v1951: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 88 MiB/s wr, 1.40k op/s; 45 KiB/s, 0 objects/s recovering 2024-09-17T16:39:40.933 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:40.962 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:40.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:40.990 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 48706 48690 82.2388 62.8125 0.00515759 0.0121529 2024-09-17T16:39:41.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:41.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:41.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:41.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:41.225 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:41.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:40 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4087064618' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:40 smithi017 ceph-mon[27571]: osdmap e886: 12 total, 12 up, 11 in 2024-09-17T16:39:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2627937217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:40 smithi017 ceph-mon[27571]: pgmap v1951: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 88 MiB/s wr, 1.40k op/s; 45 KiB/s, 0 objects/s recovering 2024-09-17T16:39:41.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:40 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4087064618' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:40 smithi071 ceph-mon[30002]: osdmap e886: 12 total, 12 up, 11 in 2024-09-17T16:39:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2627937217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:40 smithi071 ceph-mon[30002]: pgmap v1951: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 88 MiB/s wr, 1.40k op/s; 45 KiB/s, 0 objects/s recovering 2024-09-17T16:39:41.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:41.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:41.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:41.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:41.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:41.485 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:41.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:41.609 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:41.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:41.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:41.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:41.735 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:41.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:41.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:41.863 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:41.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:41.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:41.989 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:41.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:41.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 49761 49745 81.8096 65.9375 0.00681443 0.0122147 2024-09-17T16:39:42.091 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:42.115 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:42.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:42.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:42.241 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:42.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:42.343 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:42.367 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:42.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:42.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:42.495 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:42.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:42.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:42.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:42.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:42.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:42.757 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:42.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:42.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:42.793 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:39:42.859 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:42.882 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:42.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:42.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:42.984 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:42.990 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 51111 51095 81.8753 84.375 0.0149256 0.012209 2024-09-17T16:39:43.010 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:43.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:43.112 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:43.139 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:43.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:39:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:39:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: osdmap e887: 12 total, 12 up, 11 in 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: pgmap v1953: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s; 29 KiB/s, 0 objects/s recovering 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1174354789,config=2353167246,health=179246142,kv=2091611497}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3755806917,logm=2398157809}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=884460602}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=782121980}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1426879905}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3533906615}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1133144111}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1028765151}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830156865}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017219221}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3163991451}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976657088}) 2024-09-17T16:39:43.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994577394}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228186772}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513011902}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1019089576}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1562040657}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1939701297}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202739429}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429485943}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933017628}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490763907}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413101616}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299953566}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371345230}) 2024-09-17T16:39:43.147 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1244172489}) 2024-09-17T16:39:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412070728}) 2024-09-17T16:39:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695261916}) 2024-09-17T16:39:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3177539318}) 2024-09-17T16:39:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=604246239}) 2024-09-17T16:39:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80318786}) 2024-09-17T16:39:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2617237465}) 2024-09-17T16:39:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=52952923}) 2024-09-17T16:39:43.148 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3001687145}) 2024-09-17T16:39:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=236141891}) 2024-09-17T16:39:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2797269835}) 2024-09-17T16:39:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476669279}) 2024-09-17T16:39:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1667609100}) 2024-09-17T16:39:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4188266929}) 2024-09-17T16:39:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1796752956}) 2024-09-17T16:39:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273151098}) 2024-09-17T16:39:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846632060}) 2024-09-17T16:39:43.149 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1897938182}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=601785389}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455630201}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091204662}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1734277530}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2144843132}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=558558972}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749275044}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2749789449}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1923439397}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=840679038}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604651967}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2547725492}) 2024-09-17T16:39:43.150 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=981494754}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018625546}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3014795720}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=146930893}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4210074076}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277351169}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3433597695}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1721260417}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2135071137}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1818809937}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=711175838}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3329436692}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=67,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9} crc {logm=2301299309,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=1790622788}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1212638832}) 2024-09-17T16:39:43.151 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1668068281}) 2024-09-17T16:39:43.152 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=91,monmap=5,nvmeofgw=3,osd_metadata=1} crc {mon_config_key=1777068896,monmap=2143941632,nvmeofgw=598337686,osd_metadata=3763664975}) 2024-09-17T16:39:43.152 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=11,osd_pg_creating=1,osdmap=88} crc {osd_metadata=4181069896,osd_pg_creating=2235801555,osdmap=1429072611}) 2024-09-17T16:39:43.152 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2304422798}) 2024-09-17T16:39:43.152 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=55} crc {osdmap=3219386627}) 2024-09-17T16:39:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:39:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:39:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: osdmap e887: 12 total, 12 up, 11 in 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: pgmap v1953: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s; 29 KiB/s, 0 objects/s recovering 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1174354789,config=2353167246,health=179246142,kv=2091611497}) 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3755806917,logm=2398157809}) 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=884460602}) 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=782121980}) 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1426879905}) 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3533906615}) 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1133144111}) 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1028765151}) 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830156865}) 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017219221}) 2024-09-17T16:39:43.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3163991451}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976657088}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994577394}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228186772}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513011902}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1019089576}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1562040657}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1939701297}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202739429}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429485943}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933017628}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490763907}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413101616}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299953566}) 2024-09-17T16:39:43.241 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371345230}) 2024-09-17T16:39:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1244172489}) 2024-09-17T16:39:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412070728}) 2024-09-17T16:39:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695261916}) 2024-09-17T16:39:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3177539318}) 2024-09-17T16:39:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=604246239}) 2024-09-17T16:39:43.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80318786}) 2024-09-17T16:39:43.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2617237465}) 2024-09-17T16:39:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=52952923}) 2024-09-17T16:39:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3001687145}) 2024-09-17T16:39:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=236141891}) 2024-09-17T16:39:43.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2797269835}) 2024-09-17T16:39:43.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476669279}) 2024-09-17T16:39:43.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1667609100}) 2024-09-17T16:39:43.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4188266929}) 2024-09-17T16:39:43.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1796752956}) 2024-09-17T16:39:43.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273151098}) 2024-09-17T16:39:43.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846632060}) 2024-09-17T16:39:43.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1897938182}) 2024-09-17T16:39:43.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=601785389}) 2024-09-17T16:39:43.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455630201}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091204662}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1734277530}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2144843132}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=558558972}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749275044}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2749789449}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1923439397}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=840679038}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604651967}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2547725492}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=981494754}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018625546}) 2024-09-17T16:39:43.245 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3014795720}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=146930893}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4210074076}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277351169}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3433597695}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1721260417}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2135071137}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1818809937}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=711175838}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3329436692}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=67,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9} crc {logm=2301299309,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=1790622788}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1212638832}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1668068281}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=91,monmap=5,nvmeofgw=3,osd_metadata=1} crc {mon_config_key=1777068896,monmap=2143941632,nvmeofgw=598337686,osd_metadata=3763664975}) 2024-09-17T16:39:43.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=11,osd_pg_creating=1,osdmap=88} crc {osd_metadata=4181069896,osd_pg_creating=2235801555,osdmap=1429072611}) 2024-09-17T16:39:43.247 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2304422798}) 2024-09-17T16:39:43.247 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=55} crc {osdmap=3219386627}) 2024-09-17T16:39:43.271 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:43.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:39:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:39:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: osdmap e887: 12 total, 12 up, 11 in 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: pgmap v1953: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 2.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s; 29 KiB/s, 0 objects/s recovering 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1174354789,config=2353167246,health=179246142,kv=2091611497}) 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3755806917,logm=2398157809}) 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=884460602}) 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=782121980}) 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1426879905}) 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3533906615}) 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1133144111}) 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1028765151}) 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3830156865}) 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017219221}) 2024-09-17T16:39:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3163991451}) 2024-09-17T16:39:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976657088}) 2024-09-17T16:39:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=994577394}) 2024-09-17T16:39:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2228186772}) 2024-09-17T16:39:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2513011902}) 2024-09-17T16:39:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1019089576}) 2024-09-17T16:39:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1562040657}) 2024-09-17T16:39:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1939701297}) 2024-09-17T16:39:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202739429}) 2024-09-17T16:39:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3429485943}) 2024-09-17T16:39:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933017628}) 2024-09-17T16:39:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3490763907}) 2024-09-17T16:39:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413101616}) 2024-09-17T16:39:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299953566}) 2024-09-17T16:39:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371345230}) 2024-09-17T16:39:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1244172489}) 2024-09-17T16:39:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=412070728}) 2024-09-17T16:39:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695261916}) 2024-09-17T16:39:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3177539318}) 2024-09-17T16:39:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=604246239}) 2024-09-17T16:39:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=80318786}) 2024-09-17T16:39:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2617237465}) 2024-09-17T16:39:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=52952923}) 2024-09-17T16:39:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3001687145}) 2024-09-17T16:39:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=236141891}) 2024-09-17T16:39:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2797269835}) 2024-09-17T16:39:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3476669279}) 2024-09-17T16:39:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1667609100}) 2024-09-17T16:39:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4188266929}) 2024-09-17T16:39:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1796752956}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=273151098}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846632060}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1897938182}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=601785389}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455630201}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091204662}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1734277530}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2144843132}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=558558972}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=749275044}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2749789449}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1923439397}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=840679038}) 2024-09-17T16:39:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604651967}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2547725492}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=981494754}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018625546}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3014795720}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=146930893}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4210074076}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277351169}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3433597695}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1721260417}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2135071137}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1818809937}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=711175838}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3329436692}) 2024-09-17T16:39:43.322 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=67,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9} crc {logm=2301299309,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=1790622788}) 2024-09-17T16:39:43.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1212638832}) 2024-09-17T16:39:43.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1668068281}) 2024-09-17T16:39:43.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=91,monmap=5,nvmeofgw=3,osd_metadata=1} crc {mon_config_key=1777068896,monmap=2143941632,nvmeofgw=598337686,osd_metadata=3763664975}) 2024-09-17T16:39:43.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=11,osd_pg_creating=1,osdmap=88} crc {osd_metadata=4181069896,osd_pg_creating=2235801555,osdmap=1429072611}) 2024-09-17T16:39:43.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2304422798}) 2024-09-17T16:39:43.323 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=55} crc {osdmap=3219386627}) 2024-09-17T16:39:43.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:43.400 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:43.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:43.502 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:43.524 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:43.525 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:43.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:43.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:39:43.563 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 8, 9, 11, 7, 10, 5] 2024-09-17T16:39:43.563 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:39:43.563 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-09-17T16:39:43.563 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T16:39:43.563 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T16:39:43.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-09-17T16:39:43.626 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:43.649 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:43.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:43.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:43.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:43.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:43.830 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:43.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:43.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:43.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:43.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:39:43.989649+0000 min lat: 0.00107045 max lat: 2.12556 avg lat: 0.0122406 2024-09-17T16:39:43.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:39:43.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 52278 52262 81.6517 72.9375 0.0140302 0.0122406 2024-09-17T16:39:44.017 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:44.043 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:44.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:44.145 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:44.170 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:44.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:44.272 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:44.300 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:44.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:44.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:44.420 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:44.438 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:39:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:39:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:39:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:39:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:29928 7.a e646/629 7:54337644:::benchmark_data_smithi173_116396_object29927:head v 646'1856, mlcod=646'1852)", 2024-09-17T16:39:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:39:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 577.14284424200002, 2024-09-17T16:39:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.232420081, 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290201+0000", 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290207+0000", 2024-09-17T16:39:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6579999999999996e-06 2024-09-17T16:39:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.290242+0000", 2024-09-17T16:39:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4696000000000001e-05 2024-09-17T16:39:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521561+0000", 2024-09-17T16:39:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23131880499999999 2024-09-17T16:39:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521567+0000", 2024-09-17T16:39:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.392e-06 2024-09-17T16:39:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521628+0000", 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0473999999999997e-05 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.521650+0000", 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2909e-05 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.522588+0000", 2024-09-17T16:39:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093747100000000005 2024-09-17T16:39:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:30:07.522621+0000", 2024-09-17T16:39:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2675999999999997e-05 2024-09-17T16:39:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 685/677 [PushOp(7:2d0cb49e:::benchmark_data_smithi173_116396_object20958:head, version: 643'1334, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2d0cb49e:::benchmark_data_smithi173_116396_object20958:head@643'1334, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:39:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:39:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 472.406920668, 2024-09-17T16:39:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25935365100000002, 2024-09-17T16:39:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:39:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026124+0000", 2024-09-17T16:39:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026125+0000", 2024-09-17T16:39:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:39:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.026176+0000", 2024-09-17T16:39:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1581e-05 2024-09-17T16:39:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283828+0000", 2024-09-17T16:39:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25765147999999999 2024-09-17T16:39:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283845+0000", 2024-09-17T16:39:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7275e-05 2024-09-17T16:39:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283961+0000", 2024-09-17T16:39:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011592899999999999 2024-09-17T16:39:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.283974+0000", 2024-09-17T16:39:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2724e-05 2024-09-17T16:39:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:52.285478+0000", 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015038720000000001 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 685/677 [PushOp(7:295d1d25:::benchmark_data_smithi173_116396_object23683:head, version: 645'1499, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:295d1d25:::benchmark_data_smithi173_116396_object23683:head@645'1499, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 466.88167336499998, 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24965942299999999, 2024-09-17T16:39:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:39:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551371+0000", 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551375+0000", 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9809999999999997e-06 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.551452+0000", 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7064999999999995e-05 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799411+0000", 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.247958129 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799426+0000", 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5094e-05 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799521+0000", 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5449e-05 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.799534+0000", 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3077e-05 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:31:57.801031+0000", 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014966280000000001 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 687/677 [PushOp(7:2aa25eb8:::benchmark_data_smithi173_116396_object25172:head, version: 645'1590, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2aa25eb8:::benchmark_data_smithi173_116396_object25172:head@645'1590, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 463.28133407299998, 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25723122799999998, 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151711+0000", 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151712+0000", 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3030000000000001e-06 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.151805+0000", 2024-09-17T16:39:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3330000000000003e-05 2024-09-17T16:39:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.406885+0000", 2024-09-17T16:39:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25507977999999998 2024-09-17T16:39:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.406910+0000", 2024-09-17T16:39:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4736000000000002e-05 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.407016+0000", 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106372 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.407029+0000", 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3032e-05 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:01.408942+0000", 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001912675 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(7.14 690/677 [PushOp(7:2d367e55:::benchmark_data_smithi173_116396_object28235:head, version: 646'1783, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2d367e55:::benchmark_data_smithi173_116396_object28235:head@646'1783, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 457.296374426, 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26142421399999999, 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136670+0000", 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136671+0000", 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210000000000001e-06 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.136740+0000", 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8932999999999999e-05 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396250+0000", 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25950979000000002 2024-09-17T16:39:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396266+0000", 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5874000000000001e-05 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396362+0000", 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5543000000000003e-05 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.396374+0000", 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2897999999999999e-05 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:32:07.398095+0000", 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017201549999999999 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:170897 7.5 e712/674 7:a2dbce43:::benchmark_data_smithi173_116396_object75297:head v 712'8535, mlcod=712'8532)", 2024-09-17T16:39:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 377.994302377, 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34638644600000001, 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438742+0000", 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438743+0000", 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0529999999999999e-06 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438750+0000", 2024-09-17T16:39:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7989999999999996e-06 2024-09-17T16:39:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.782943+0000", 2024-09-17T16:39:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34419256799999998 2024-09-17T16:39:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.782967+0000", 2024-09-17T16:39:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3929000000000001e-05 2024-09-17T16:39:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.783013+0000", 2024-09-17T16:39:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6400999999999998e-05 2024-09-17T16:39:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.783027+0000", 2024-09-17T16:39:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4337e-05 2024-09-17T16:39:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785093+0000", 2024-09-17T16:39:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002065271 2024-09-17T16:39:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785129+0000", 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6087999999999997e-05 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:178680 7.13 e719/714 7:cae8152e:::benchmark_data_smithi173_116396_object83080:head v 719'8741, mlcod=719'8738)", 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 366.346987329, 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31485432099999999, 2024-09-17T16:39:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086057+0000", 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086058+0000", 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.85e-07 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.086063+0000", 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0050000000000004e-06 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398605+0000", 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31254172400000002 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398610+0000", 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2789999999999998e-06 2024-09-17T16:39:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398655+0000", 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4715000000000001e-05 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.398670+0000", 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4895999999999999e-05 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.400870+0000", 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022001920000000001 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:38.400912+0000", 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1625000000000002e-05 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27088.0:191187 7.14 e724/717 7:2d119a99:::benchmark_data_smithi173_116396_object95587:head v 724'9235, mlcod=724'9232)", 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:39:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 349.98586111899999, 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48898925799999998, 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:39:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447184+0000", 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7599999999999996e-07 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.447190+0000", 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9440000000000002e-06 2024-09-17T16:39:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933730+0000", 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48653948800000002 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933739+0000", 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9199999999999993e-06 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933837+0000", 2024-09-17T16:39:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8004999999999994e-05 2024-09-17T16:39:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.933856+0000", 2024-09-17T16:39:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.891e-05 2024-09-17T16:39:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.936137+0000", 2024-09-17T16:39:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022811960000000001 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.936173+0000", 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6019000000000002e-05 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.16 740/739 [PushOp(8:68fc3949:::benchmark_data_smithi173_130566_object1854:head, version: 732'115, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:68fc3949:::benchmark_data_smithi173_130566_object1854:head@732'115, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.53380406000002, 2024-09-17T16:39:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.75943818799999996, 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:39:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899241+0000", 2024-09-17T16:39:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5700000000000002e-07 2024-09-17T16:39:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:26.899299+0000", 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7451999999999998e-05 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658299+0000", 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.75900059799999997 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658317+0000", 2024-09-17T16:39:44.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:44.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:44.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:39:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658403+0000", 2024-09-17T16:39:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6600000000000004e-05 2024-09-17T16:39:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658413+0000", 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9340000000000001e-06 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.658679+0000", 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026586099999999999 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "replica_scrub(pg: 8.d,from:0'0,to:772'6583,epoch:782/728,start:8:b03f31a8:::benchmark_data_smithi173_130566_object64387:0,end:8:b08c319a:::benchmark_data_smithi173_130566_object64164:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-17T16:39:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:12.777592+0000", 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 211.65545283200001, 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55434039499999999, 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:12.777592+0000", 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:12.777592+0000", 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:12.777598+0000", 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3629999999999999e-06 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:12.777609+0000", 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0655999999999999e-05 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:13.331787+0000", 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55417758299999997 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:13.331797+0000", 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0525e-05 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:13.331876+0000", 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9132000000000001e-05 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:13.331898+0000", 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1534000000000001e-05 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:13.331932+0000", 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4601999999999998e-05 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:152548 8.3 e820/736 8:c1e3f3aa:::benchmark_data_smithi173_130566_object57573:head v 820'4995, mlcod=820'4992)", 2024-09-17T16:39:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:42.473616+0000", 2024-09-17T16:39:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.95942914299999, 2024-09-17T16:39:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22429043800000001, 2024-09-17T16:39:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.473616+0000", 2024-09-17T16:39:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.473616+0000", 2024-09-17T16:39:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.473617+0000", 2024-09-17T16:39:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.034e-06 2024-09-17T16:39:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.473624+0000", 2024-09-17T16:39:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3560000000000004e-06 2024-09-17T16:39:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.690685+0000", 2024-09-17T16:39:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21706067200000001 2024-09-17T16:39:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.690693+0000", 2024-09-17T16:39:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3909999999999998e-06 2024-09-17T16:39:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.690771+0000", 2024-09-17T16:39:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7776999999999998e-05 2024-09-17T16:39:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.690791+0000", 2024-09-17T16:39:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9731000000000001e-05 2024-09-17T16:39:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.697825+0000", 2024-09-17T16:39:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0070339369999999997 2024-09-17T16:39:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:42.697906+0000", 2024-09-17T16:39:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1539999999999995e-05 2024-09-17T16:39:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(8.6 e848,840 8:61891a4b:::benchmark_data_smithi173_130566_object18031:head,795'3729)", 2024-09-17T16:39:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:32.649553+0000", 2024-09-17T16:39:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 71.783492003999996, 2024-09-17T16:39:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26187870800000002, 2024-09-17T16:39:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.649553+0000", 2024-09-17T16:39:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.649553+0000", 2024-09-17T16:39:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.649553+0000", 2024-09-17T16:39:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1500000000000005e-07 2024-09-17T16:39:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.649557+0000", 2024-09-17T16:39:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9720000000000003e-06 2024-09-17T16:39:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.911315+0000", 2024-09-17T16:39:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26175768399999999 2024-09-17T16:39:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.911323+0000", 2024-09-17T16:39:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6879999999999999e-06 2024-09-17T16:39:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.911372+0000", 2024-09-17T16:39:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9617000000000003e-05 2024-09-17T16:39:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.911378+0000", 2024-09-17T16:39:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3419999999999998e-06 2024-09-17T16:39:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.911431+0000", 2024-09-17T16:39:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3890000000000001e-05 2024-09-17T16:39:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(8.6 e848,840 8:635d89f1:::benchmark_data_smithi173_130566_object18060:head,795'3730)", 2024-09-17T16:39:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:32.916487+0000", 2024-09-17T16:39:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 71.516558141999994, 2024-09-17T16:39:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30438891400000001, 2024-09-17T16:39:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.916487+0000", 2024-09-17T16:39:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.916487+0000", 2024-09-17T16:39:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.916487+0000", 2024-09-17T16:39:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8299999999999996e-07 2024-09-17T16:39:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.916492+0000", 2024-09-17T16:39:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7539999999999997e-06 2024-09-17T16:39:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.220689+0000", 2024-09-17T16:39:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.304197041 2024-09-17T16:39:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.220701+0000", 2024-09-17T16:39:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1854e-05 2024-09-17T16:39:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.220786+0000", 2024-09-17T16:39:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4766999999999997e-05 2024-09-17T16:39:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.220795+0000", 2024-09-17T16:39:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5910000000000002e-06 2024-09-17T16:39:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.220876+0000", 2024-09-17T16:39:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0123999999999994e-05 2024-09-17T16:39:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:189073 8.d e848/728 8:bd72f8b4:::benchmark_data_smithi173_130566_object94098:head v 848'11992, mlcod=848'11989)", 2024-09-17T16:39:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:32.984456+0000", 2024-09-17T16:39:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 71.448588606000001, 2024-09-17T16:39:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33613770399999998, 2024-09-17T16:39:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.984456+0000", 2024-09-17T16:39:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.984456+0000", 2024-09-17T16:39:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.984457+0000", 2024-09-17T16:39:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5600000000000004e-07 2024-09-17T16:39:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:32.984464+0000", 2024-09-17T16:39:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0720000000000002e-06 2024-09-17T16:39:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.315566+0000", 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33110177099999999 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.315572+0000", 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0619999999999997e-06 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.315677+0000", 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104737 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.315697+0000", 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9828999999999999e-05 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.320547+0000", 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048499219999999996 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:33.320594+0000", 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7355000000000003e-05 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(8.6 e849,840 8:67d9555d:::benchmark_data_smithi173_130566_object20892:head,801'3815)", 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:34.101493+0000", 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 70.331551790000006, 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25631964299999999, 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.101493+0000", 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.101493+0000", 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.101494+0000", 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0500000000000002e-07 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.101500+0000", 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3049999999999998e-06 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.357493+0000", 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25599333899999999 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.357507+0000", 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3842999999999999e-05 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.357651+0000", 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014420000000000001 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.357679+0000", 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7373e-05 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.357813+0000", 2024-09-17T16:39:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013377800000000001 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:189927 8.13 e849/742 8:ccb6c39c:::benchmark_data_smithi173_130566_object94952:head v 849'6060, mlcod=849'6057)", 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:34.116914+0000", 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 70.316130637000001, 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.439282177, 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.116914+0000", 2024-09-17T16:39:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.116914+0000", 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.116915+0000", 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.268e-06 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.116925+0000", 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5310000000000007e-06 2024-09-17T16:39:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.553609+0000", 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43668404500000002 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.553614+0000", 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0499999999999999e-06 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.553666+0000", 2024-09-17T16:39:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1514000000000002e-05 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.553690+0000", 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3957000000000001e-05 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.556163+0000", 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002473697 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.556196+0000", 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3114999999999997e-05 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(8.6 e849,840 8:6249963e:::benchmark_data_smithi173_130566_object26709:head,804'3977)", 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:35.314231+0000", 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 69.118813704000004, 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37940180699999998, 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:35.314231+0000", 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:35.314231+0000", 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:35.314232+0000", 2024-09-17T16:39:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7599999999999997e-07 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:35.314236+0000", 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8890000000000004e-06 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:35.693350+0000", 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37911428800000002 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:35.693369+0000", 2024-09-17T16:39:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8919999999999998e-05 2024-09-17T16:39:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:35.693525+0000", 2024-09-17T16:39:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000156336 2024-09-17T16:39:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:35.693540+0000", 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4555999999999999e-05 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:35.693633+0000", 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3241999999999999e-05 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(8.6 e850,840 8:6643c73a:::benchmark_data_smithi173_130566_object39933:head,809'4386)", 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:36.996455+0000", 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 67.436589789999999, 2024-09-17T16:39:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34607146, 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:36.996455+0000", 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:36.996455+0000", 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:36.996456+0000", 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7999999999999995e-07 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:36.996461+0000", 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8899999999999998e-06 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:37.342380+0000", 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34591977899999998 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:37.342388+0000", 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5020000000000003e-06 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:37.342452+0000", 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4478000000000002e-05 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:37.342461+0000", 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9719999999999998e-06 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:37.342526+0000", 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5159000000000004e-05 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27874.0:8120 9.5 e865/858 9:ae62e0e8:::benchmark_data_smithi173_150363_object8119:head v 865'549, mlcod=864'546)", 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:08.215750+0000", 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.217294608000003, 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23218166600000001, 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.215750+0000", 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.215750+0000", 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.215751+0000", 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:39:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.215824+0000", 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3126000000000004e-05 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.447275+0000", 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23145104499999999 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.447280+0000", 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9579999999999998e-06 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.447322+0000", 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2361000000000003e-05 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.447336+0000", 2024-09-17T16:39:44.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3045e-05 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.447891+0000", 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00055580099999999997 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.447932+0000", 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0538999999999999e-05 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27874.0:21050 9.9 e875/858 9:926433bd:::benchmark_data_smithi173_150363_object21049:head v 875'1291, mlcod=875'1287)", 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:20.562882+0000", 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.87016272, 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26229025, 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:20.562882+0000", 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:20.562882+0000", 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:20.562883+0000", 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.02e-07 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:20.562917+0000", 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4039999999999999e-05 2024-09-17T16:39:44.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:20.824336+0000", 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26141935500000002 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:20.824343+0000", 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6470000000000001e-06 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:20.824384+0000", 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1310999999999998e-05 2024-09-17T16:39:44.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:20.824400+0000", 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5216e-05 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:20.825140+0000", 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074071499999999997 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:20.825172+0000", 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2064000000000003e-05 2024-09-17T16:39:44.546 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.547 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.547 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.547 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:44.547 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:44.547 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:39:44.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:44.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:44.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:44.626 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:44.727 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:44.753 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:44.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:44.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:44.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:44.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:44.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:44.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 15 53731 53716 81.8764 90.875 0.00527822 0.0122085 2024-09-17T16:39:45.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:45.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:45.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:45.120 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:45.144 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:45.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:45.200 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:39:45.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:45.269 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:45.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:45.371 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:45.376 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:39:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "" 2024-09-17T16:39:45.376 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:39:45.395 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:45.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:45 smithi017 ceph-mon[27571]: pgmap v1954: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 3.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 76 MiB/s wr, 1.23k op/s; 70 KiB/s, 0 objects/s recovering 2024-09-17T16:39:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:45 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:39:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:45 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:39:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:45 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/295668098' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:39:45.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:45.504 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:39:45.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:45.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:45 smithi071 ceph-mon[30002]: pgmap v1954: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 3.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 76 MiB/s wr, 1.23k op/s; 70 KiB/s, 0 objects/s recovering 2024-09-17T16:39:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:45 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:39:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:45 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:39:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:45 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/295668098' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:39:45.622 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:45 smithi173 ceph-mon[30530]: pgmap v1954: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 3.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 4.4 KiB/s rd, 76 MiB/s wr, 1.23k op/s; 70 KiB/s, 0 objects/s recovering 2024-09-17T16:39:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:45 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:39:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:45 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:39:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:45 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/295668098' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:39:45.646 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:45.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:45.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:45.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:45.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:45.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:45.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:45.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 54852 54836 81.5935 70 0.00328909 0.0122511 2024-09-17T16:39:46.011 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:46.035 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:46.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:46.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:46.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:46.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:46.270 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:46.294 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:46.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:46 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:39:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:46 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:39:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:46 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/295668098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:39:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:46 smithi173 ceph-mon[30530]: osdmap e888: 12 total, 12 up, 11 in 2024-09-17T16:39:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:46 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:39:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:46 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:39:46.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:46.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:46.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:46.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:46.427 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:46 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:39:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:46 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:39:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:46 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/295668098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:39:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:46 smithi017 ceph-mon[27571]: osdmap e888: 12 total, 12 up, 11 in 2024-09-17T16:39:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:46 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:39:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:46 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:39:46.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:46.548 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:46.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:46.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:46 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:39:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:46 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:39:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:46 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/295668098' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:39:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:46 smithi071 ceph-mon[30002]: osdmap e888: 12 total, 12 up, 11 in 2024-09-17T16:39:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:46 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:39:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:46 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:39:46.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:46.653 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:39:46.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:46.678 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:46.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:46.780 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:46.804 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:46.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:46.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:46.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:46.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:46.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 56045 56029 81.4299 74.5625 0.0103024 0.012275 2024-09-17T16:39:47.032 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:39:47.033 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:39:47.039 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:47.060 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:39:47.060 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:39:47.066 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:47.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:47.085 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:39:47.086 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:39:47.112 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:39:47.112 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:39:47.136 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:39:47.137 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:39:47.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:39:47.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:39:47.167 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:47.189 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:39:47.189 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:39:47.198 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:47.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:47.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:47.216 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:39:47.216 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:39:47.239 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:39:47.239 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:39:47.257 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:39:47.261 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:39:47.261 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:39:47.285 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:39:47.285 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:39:47.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:47.307 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:39:47.307 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:39:47.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:47.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:47.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:47.429 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:47.456 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:47.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:47 smithi017 ceph-mon[27571]: pgmap v1956: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s 2024-09-17T16:39:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:47 smithi017 ceph-mon[27571]: osdmap e889: 12 total, 12 up, 11 in 2024-09-17T16:39:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/627410299' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:47.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:47 smithi071 ceph-mon[30002]: pgmap v1956: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s 2024-09-17T16:39:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:47 smithi071 ceph-mon[30002]: osdmap e889: 12 total, 12 up, 11 in 2024-09-17T16:39:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/627410299' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:47.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:47.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:47 smithi173 ceph-mon[30530]: pgmap v1956: 27 pgs: 27 active+clean; 3.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 61 MiB/s wr, 977 op/s 2024-09-17T16:39:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:47 smithi173 ceph-mon[30530]: osdmap e889: 12 total, 12 up, 11 in 2024-09-17T16:39:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/627410299' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:47.690 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:47.715 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:47.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:47.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:47.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:47.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:47.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:47.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:47.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:47.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 57507 57491 81.6557 91.375 0.00321705 0.0122399 2024-09-17T16:39:48.080 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:48.080 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:48.081 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:48.104 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:48.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:48.206 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:48.233 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:48.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:48.310 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:39:48.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:48.362 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:48.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:48.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:48.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:48.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:48.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:48.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:48.611 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:48.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:48.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:48.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:48.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:48.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:48.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:48.880 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:48.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:39:48.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:48.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:48.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 58607 58591 81.3688 68.75 0.00810043 0.0122822 2024-09-17T16:39:49.009 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:49.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:49.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:49.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:49.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:49.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:49.241 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:49.265 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:49.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:49.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:49.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:49.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:49 smithi017 ceph-mon[27571]: pgmap v1958: 27 pgs: 27 active+clean; 3.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-17T16:39:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:39:49.493 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:49.518 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:49.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:49.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:49 smithi071 ceph-mon[30002]: pgmap v1958: 27 pgs: 27 active+clean; 3.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-17T16:39:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:39:49.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:49 smithi173 ceph-mon[30530]: pgmap v1958: 27 pgs: 27 active+clean; 3.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-17T16:39:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:39:49.645 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:49.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:49.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:49.760 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:39:49.784 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:49.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:49.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:49.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:49.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:49.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 15 59887 59872 81.3403 80.0625 0.00422341 0.0122886 2024-09-17T16:39:50.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:50.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:50.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:50.148 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:39:50.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:50.217 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:50.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:50.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:50.224 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:39:50.317 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:50.317 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:50.318 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:50.342 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:50.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:50.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:50.468 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:50.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:50.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:50 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1796154512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:50.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:50 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1796154512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:50.570 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:50.596 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:50.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:50 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1796154512' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:39:50.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:50.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:50.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:50.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:50.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:50.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:50.954 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:50.978 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:50.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:50.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 60820 60804 80.8489 58.25 0.00314791 0.0123428 2024-09-17T16:39:50.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:39:51.063 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 8, 9, 11, 7, 10, 5] 2024-09-17T16:39:51.063 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:39:51.063 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-17T16:39:51.063 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-17T16:39:51.079 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:51.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:51.195 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:51.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:51.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:51.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:51.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:51.393 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:39:51.425 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:51.470 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:51.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:51 smithi017 ceph-mon[27571]: pgmap v1959: 27 pgs: 27 active+clean; 3.5 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-17T16:39:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:51 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:39:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:51 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:39:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1796154512' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:51 smithi017 ceph-mon[27571]: osdmap e890: 12 total, 12 up, 11 in 2024-09-17T16:39:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:39:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:39:51.524 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:51 smithi071 ceph-mon[30002]: pgmap v1959: 27 pgs: 27 active+clean; 3.5 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-17T16:39:51.524 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:51 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:39:51.524 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:51 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:39:51.524 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1796154512' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:51.524 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:51 smithi071 ceph-mon[30002]: osdmap e890: 12 total, 12 up, 11 in 2024-09-17T16:39:51.524 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:39:51.524 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:39:51.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:51.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:51.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:51 smithi173 ceph-mon[30530]: pgmap v1959: 27 pgs: 27 active+clean; 3.5 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-17T16:39:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:51 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:39:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:51 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:39:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1796154512' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:39:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:51 smithi173 ceph-mon[30530]: osdmap e890: 12 total, 12 up, 11 in 2024-09-17T16:39:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:39:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:39:51.708 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:51.733 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:51.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:51.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:51.835 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:51.860 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:51.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:51.916 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:51.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:51.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:51.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:51.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 62217 62201 80.9834 87.3125 0.00812961 0.0123412 2024-09-17T16:39:52.014 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:39:52.015 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:39:52.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:52.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:52.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:52.217 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:52.242 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:52.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:52.344 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:52.373 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:52.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:39:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:39:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:39:52.462 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:52.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:52.478 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:39:52.479 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:39:52.503 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:39:52.504 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:39:52.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:52.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:52.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:39:52.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:39:52.515 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:39:52.524 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:39:52.525 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:39:52.545 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:39:52.545 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:39:52.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:39:52.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:39:52.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:39:52.566 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:39:52.567 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:39:52.587 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:39:52.588 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:39:52.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:52.610 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:39:52.610 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:39:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:52.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:52.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:52.631 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:39:52.631 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:39:52.652 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:39:52.653 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:39:52.674 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:39:52.674 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:39:52.695 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:39:52.695 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:39:52.721 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:39:52.722 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:39:52.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:52.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:52.761 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:52.862 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:52.887 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:52.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:52.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:52.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 63141 63125 80.5091 57.75 0.0142532 0.0124119 2024-09-17T16:39:53.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:53.014 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:53.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:39:53.115 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:53.144 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:53.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:53.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:53.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:53.274 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:53.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:53.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:53.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:53 smithi071 ceph-mon[30002]: pgmap v1961: 27 pgs: 27 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:39:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2250506614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:53.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:53.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:53 smithi173 ceph-mon[30530]: pgmap v1961: 27 pgs: 27 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:39:53.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2250506614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:53.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:53.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:53.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:53.412 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:53 smithi017 ceph-mon[27571]: pgmap v1961: 27 pgs: 27 active+clean; 3.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-17T16:39:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2250506614' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:39:53.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:53.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:53.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:53.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:53.665 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:53.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:53.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:53.790 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:53.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:53.893 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:53.914 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:53.915 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:39:53.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:53.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:53.930 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:39:53.930 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:39:53.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.930 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:63964 9.e 9:711ff3a1:::benchmark_data_smithi173_150363_object63963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e890)", 2024-09-17T16:39:53.930 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:53.780162+0000", 2024-09-17T16:39:53.930 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.14698910100000001, 2024-09-17T16:39:53.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.147020123, 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 63964 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.780162+0000", 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:53.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.780162+0000", 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.780164+0000", 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7540000000000001e-06 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.780184+0000", 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0888999999999999e-05 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.780196+0000", 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1969e-05 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.780204+0000", 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8830000000000006e-06 2024-09-17T16:39:53.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.780259+0000", 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4528999999999999e-05 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.780367+0000", 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108284 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.780466+0000", 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8862000000000003e-05 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.782312+0000", 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001845754 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.782853+0000", 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054167699999999996 2024-09-17T16:39:53.935 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:63982 9.9 9:97016969:::benchmark_data_smithi173_150363_object63981:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e890)", 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:53.801450+0000", 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.12570114700000001, 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12580992299999999, 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:39:53.936 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 63982 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.801450+0000", 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:53.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.801450+0000", 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.801451+0000", 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6029999999999999e-06 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.801473+0000", 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1775000000000001e-05 2024-09-17T16:39:53.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.801486+0000", 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2614e-05 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.801492+0000", 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2500000000000003e-06 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.801543+0000", 2024-09-17T16:39:53.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1304999999999998e-05 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.801644+0000", 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010032200000000001 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.801737+0000", 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3058000000000002e-05 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:39:53.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.803453+0000", 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017166149999999999 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.803738+0000", 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028427200000000002 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.941 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27874.0:63991 9.b e890/858 9:db92012a:::benchmark_data_smithi173_150363_object63990:head v 890'4015, mlcod=890'4012)", 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:53.925976+0000", 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.001174893, 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013485940000000001, 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.925976+0000", 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.925976+0000", 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.925977+0000", 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.127e-06 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.926000+0000", 2024-09-17T16:39:53.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2768999999999998e-05 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.926032+0000", 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1974000000000003e-05 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.926039+0000", 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7100000000000001e-06 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.926098+0000", 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9324000000000003e-05 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:53.926128+0000", 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9967999999999998e-05 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:39:53.945 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 3 2024-09-17T16:39:53.946 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:39:53.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 64040 64024 80.0226 56.1875 0.00402872 0.0124824 2024-09-17T16:39:54.017 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:54.041 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:54.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:54.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:54.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-17T16:39:54.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:54.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:54.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:54.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:54.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:54.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:54.414 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:54.442 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:54.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:54.544 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:54.565 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:54.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:54.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:54.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:54.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:54.709 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:54.712 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:54.797 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:54.822 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:54.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:54.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:54.955 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:54.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:54.991 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 15 65392 65377 80.1115 84.5625 0.0190581 0.012478 2024-09-17T16:39:55.056 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:55.083 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:55.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:55.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:55.209 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:55.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:55.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:55.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:55.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:55.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:55.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:55.467 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:55.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:55.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:55.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:55 smithi017 ceph-mon[27571]: pgmap v1962: 27 pgs: 27 active+clean; 3.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-17T16:39:55.478 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:55 smithi017 ceph-mon[27571]: osdmap e891: 12 total, 12 up, 11 in 2024-09-17T16:39:55.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:55 smithi071 ceph-mon[30002]: pgmap v1962: 27 pgs: 27 active+clean; 3.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-17T16:39:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:55 smithi071 ceph-mon[30002]: osdmap e891: 12 total, 12 up, 11 in 2024-09-17T16:39:55.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:55.596 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:55.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:55.625 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:55 smithi173 ceph-mon[30530]: pgmap v1962: 27 pgs: 27 active+clean; 3.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 75 MiB/s wr, 1.19k op/s 2024-09-17T16:39:55.625 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:55 smithi173 ceph-mon[30530]: osdmap e891: 12 total, 12 up, 11 in 2024-09-17T16:39:55.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:39:55.698 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:55.725 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:55.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:55.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:55.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:55.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:55.958 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:55.981 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:55.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:55.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 66560 66544 79.9734 72.9375 0.0445696 0.012496 2024-09-17T16:39:56.041 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:56.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:56.106 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:56.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:56.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:56.230 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:56.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:56.232 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:39:56.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:56.355 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:56.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:56.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1346767674' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:39:56.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-17T16:39:56.445 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1346767674' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:39:56.456 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:56.480 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:56.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:56.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1346767674' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:39:56.580 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:39:56.580 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:39:56.580 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:39:56.580 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:39:56.580 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:39:56.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:56.603 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:56.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:56.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:56.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:39:56.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:56.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:56.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:56.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:56.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:56.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:56.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:39:56.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 67432 67416 79.4927 54.5 0.0104293 0.0125737 2024-09-17T16:39:56.999 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:57.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:57.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:39:57.102 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:57.128 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:57.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:57.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:57.237 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:57.246 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:57.254 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:57.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:57.357 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:57.383 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:57.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:57.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:57.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:57 smithi017 ceph-mon[27571]: pgmap v1964: 27 pgs: 27 active+clean; 3.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-17T16:39:57.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:57 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:57.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1346767674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:39:57.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:57 smithi017 ceph-mon[27571]: osdmap e892: 12 total, 12 up, 11 in 2024-09-17T16:39:57.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:39:57.510 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:57.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:57.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:39:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:57 smithi071 ceph-mon[30002]: pgmap v1964: 27 pgs: 27 active+clean; 3.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-17T16:39:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:57 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1346767674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:39:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:57 smithi071 ceph-mon[30002]: osdmap e892: 12 total, 12 up, 11 in 2024-09-17T16:39:57.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:39:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:57 smithi173 ceph-mon[30530]: pgmap v1964: 27 pgs: 27 active+clean; 3.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-17T16:39:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:57 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:39:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1346767674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:39:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:57 smithi173 ceph-mon[30530]: osdmap e892: 12 total, 12 up, 11 in 2024-09-17T16:39:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:57.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:57.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:39:57.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:57.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:57.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:57.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:57.910 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:57.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:57.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 68636 68620 79.414 75.25 0.00892332 0.012586 2024-09-17T16:39:58.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:39:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:58.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:58.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:39:58.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:39:58.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:58.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:39:58.294 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:58.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:39:58.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:58.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:58.423 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:58.449 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:58.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:58.551 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:39:58.576 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:58.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:39:58.678 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:39:58.687 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-17T16:39:58.703 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:58.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:39:58.719 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:39:58.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:58.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:58.838 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:58.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:58.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:58.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:58.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 69893 69877 79.3984 78.5625 0.0107802 0.0125892 2024-09-17T16:39:59.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:59.097 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:59.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:59.162 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:39:59.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:59.241 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:59.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:59.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:59.333 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:39:59.333 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:39:59.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:39:59.333 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:39:59.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.333 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104264 8.4 8.cdf0f904 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.333 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958458+0000", 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.370086996, 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76365417400000002, 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104264 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958458+0000", 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958458+0000", 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958460+0000", 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1150000000000001e-06 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958467+0000", 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.776e-06 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958476+0000", 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3300000000000005e-06 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958485+0000", 2024-09-17T16:39:59.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5250000000000005e-06 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958553+0000", 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8027000000000004e-05 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958563+0000", 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210999999999999e-05 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722103+0000", 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76353991499999996 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722112+0000", 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2750000000000001e-06 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.339 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104266 8.4 8.df640d24 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.339 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958502+0000", 2024-09-17T16:39:59.339 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.37004256700001, 2024-09-17T16:39:59.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.763704465, 2024-09-17T16:39:59.339 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.339 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.339 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.339 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.339 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104266 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958502+0000", 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958502+0000", 2024-09-17T16:39:59.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958503+0000", 2024-09-17T16:39:59.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3500000000000002e-07 2024-09-17T16:39:59.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958505+0000", 2024-09-17T16:39:59.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.446e-06 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958510+0000", 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3950000000000003e-06 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958513+0000", 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0790000000000001e-06 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958585+0000", 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2260000000000003e-05 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2109999999999997e-06 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722199+0000", 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360864399999995 2024-09-17T16:39:59.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722207+0000", 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0949999999999996e-06 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104310 8.4 8.3aa61324 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.37000885699999, 2024-09-17T16:39:59.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76370166900000003, 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104310 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:39:59.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1899999999999998e-07 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958538+0000", 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.818e-06 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958545+0000", 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4300000000000003e-06 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958548+0000", 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8279999999999996e-06 2024-09-17T16:39:59.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958613+0000", 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4791999999999999e-05 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958620+0000", 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6980000000000001e-06 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722230+0000", 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76361024600000005 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722238+0000", 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5379999999999997e-06 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104365 8.4 8.98b36004 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.349 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.36998591700001, 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76371162999999997, 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104365 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:39:59.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1399999999999998e-07 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958561+0000", 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.793e-06 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958565+0000", 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4790000000000001e-06 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958568+0000", 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9409999999999999e-06 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958647+0000", 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9058000000000001e-05 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958650+0000", 2024-09-17T16:39:59.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.422e-06 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722263+0000", 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76361347499999999 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722271+0000", 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.148e-06 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104510 8.4 8.71f883e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.36996690699999, 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76372115600000001, 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104510 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.355 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0899999999999997e-07 2024-09-17T16:39:59.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958580+0000", 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3149999999999999e-06 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958583+0000", 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7900000000000001e-06 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958586+0000", 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5639999999999999e-06 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958683+0000", 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6762999999999995e-05 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958689+0000", 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2349999999999996e-06 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722292+0000", 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360352600000003 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722299+0000", 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6540000000000002e-06 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.359 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104538 8.4 8.69da0a4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.36995470400001, 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76373857899999997, 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104538 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:39:59.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.29e-07 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958592+0000", 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4929999999999999e-06 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958595+0000", 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0369999999999999e-06 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958598+0000", 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.932e-06 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958715+0000", 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011665 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958718+0000", 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.591e-06 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:39:59.365 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:39:59.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722322+0000", 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360369100000003 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722329+0000", 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.956e-06 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104542 8.4 8.5948f164 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.36992608200001, 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76374053200000003, 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104542 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:39:59.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8900000000000001e-07 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958621+0000", 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.978e-06 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958625+0000", 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4599999999999999e-06 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958627+0000", 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7240000000000001e-06 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958747+0000", 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012020599999999999 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958753+0000", 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9800000000000003e-06 2024-09-17T16:39:59.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722352+0000", 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.763599005 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722359+0000", 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8900000000000001e-06 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104571 8.4 8.f7b7a3a4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.369902139, 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76374966600000005, 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104571 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9999999999999999e-07 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958644+0000", 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2529999999999999e-06 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958648+0000", 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3620000000000002e-06 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958650+0000", 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.514e-06 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958778+0000", 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012807300000000001 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958781+0000", 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.039e-06 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722385+0000", 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360329100000002 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722392+0000", 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8339999999999999e-06 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104651 8.4 8.2da1dbe4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.369883399, 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76376209500000003, 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.376 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104651 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8999999999999998e-07 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958663+0000", 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2759999999999999e-06 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958667+0000", 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2880000000000003e-06 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958670+0000", 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4650000000000001e-06 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958811+0000", 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014136500000000001 2024-09-17T16:39:59.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958817+0000", 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1160000000000004e-06 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722416+0000", 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76359866899999995 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722424+0000", 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6260000000000003e-06 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104652 8.4 8.77678704 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958673+0000", 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.369871483, 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76378596899999995, 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.381 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104652 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958673+0000", 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958673+0000", 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958674+0000", 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.28e-07 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958675+0000", 2024-09-17T16:39:59.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.141e-06 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958678+0000", 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0010000000000002e-06 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958682+0000", 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2869999999999998e-06 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958843+0000", 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001607 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958846+0000", 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0419999999999998e-06 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722449+0000", 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360327400000005 2024-09-17T16:39:59.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722459+0000", 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0295999999999999e-05 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104673 8.4 8.63bb1ee4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958685+0000", 2024-09-17T16:39:59.386 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.36985958599999, 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76381003599999997, 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104673 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.387 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958685+0000", 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958685+0000", 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958686+0000", 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2600000000000001e-07 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958687+0000", 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2580000000000001e-06 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958690+0000", 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9909999999999998e-06 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958692+0000", 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4360000000000001e-06 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958875+0000", 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000182327 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958881+0000", 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0410000000000003e-06 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722488+0000", 2024-09-17T16:39:59.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360697899999996 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722495+0000", 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7780000000000007e-06 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104674 8.4 8.5e810684 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.369833884, 2024-09-17T16:39:59.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76382820500000004, 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104674 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:39:59.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:39:59.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:39:59.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9299999999999999e-07 2024-09-17T16:39:59.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958715+0000", 2024-09-17T16:39:59.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7869999999999999e-06 2024-09-17T16:39:59.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958719+0000", 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5980000000000001e-06 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958721+0000", 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6589999999999999e-06 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958920+0000", 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000198789 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958924+0000", 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4379999999999999e-06 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722530+0000", 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360632500000003 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722539+0000", 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3160000000000003e-06 2024-09-17T16:39:59.396 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104695 8.4 8.bc0ebd64 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.36981981599999, 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.763851155, 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104695 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4400000000000001e-07 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958727+0000", 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.657e-06 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958730+0000", 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8480000000000001e-06 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958732+0000", 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4200000000000001e-06 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958955+0000", 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022305200000000001 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958962+0000", 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0369999999999997e-06 2024-09-17T16:39:59.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722567+0000", 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360444599999999 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722576+0000", 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104721 8.4 8.83410e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.36980401700001, 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76388141899999995, 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104721 2024-09-17T16:39:59.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.403 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0199999999999998e-07 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958742+0000", 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.26e-06 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958746+0000", 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8319999999999999e-06 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958749+0000", 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7980000000000001e-06 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958987+0000", 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023819600000000001 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958990+0000", 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0110000000000001e-06 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722614+0000", 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76362420200000003 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722622+0000", 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8180000000000004e-06 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104726 8.4 8.bcfb8bc4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958752+0000", 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.369792461, 2024-09-17T16:39:59.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76390264500000005, 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104726 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958752+0000", 2024-09-17T16:39:59.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958752+0000", 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958753+0000", 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.29e-07 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958754+0000", 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3799999999999996e-07 2024-09-17T16:39:59.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958756+0000", 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8329999999999998e-06 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958759+0000", 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7049999999999999e-06 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959018+0000", 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025930899999999999 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959025+0000", 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1990000000000003e-06 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722647+0000", 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76362246600000006 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722655+0000", 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0660000000000004e-06 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104729 8.4 8.1aa6e3c4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.36976983100001, 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76391769399999998, 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.411 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104729 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9499999999999998e-07 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958777+0000", 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.184e-06 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958780+0000", 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7440000000000001e-06 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958783+0000", 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0910000000000001e-06 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959039+0000", 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000255133 2024-09-17T16:39:59.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959042+0000", 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6049999999999998e-06 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722686+0000", 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76364342399999996 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722693+0000", 2024-09-17T16:39:59.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2180000000000002e-06 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:189906 8.9 8:94a958b0:::benchmark_data_smithi173_130566_object94931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e849)", 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:34.082870+0000", 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 85.245674637999997, 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.68727374100000005, 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.416 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 189906 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.082870+0000", 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.082870+0000", 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.082871+0000", 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.64e-07 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.082876+0000", 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8570000000000003e-06 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.082885+0000", 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3409999999999992e-06 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.082898+0000", 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2802e-05 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.082944+0000", 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5765e-05 2024-09-17T16:39:59.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.083039+0000", 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4757000000000001e-05 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.083120+0000", 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1691000000000001e-05 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:39:59.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.085807+0000", 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026868669999999999 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.086000+0000", 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000192783 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.770033+0000", 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.68403293099999996 2024-09-17T16:39:59.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.770094+0000", 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0483999999999999e-05 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.770144+0000", 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0498999999999999e-05 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.17883.0:189906 8.9 e849/831 ondisk, result = 0)", 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:34.086089+0000", 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 85.242455501999999, 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.68406037399999997, 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.086089+0000", 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.086089+0000", 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.086090+0000", 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1999999999999999e-07 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.086094+0000", 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1790000000000001e-06 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.769952+0000", 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.68385790800000001 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.769958+0000", 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4349999999999996e-06 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.770017+0000", 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9688999999999997e-05 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.770026+0000", 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1840000000000005e-06 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.770150+0000", 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123359 2024-09-17T16:39:59.426 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(8.11 e849,797 8:8a388e25:::benchmark_data_smithi173_130566_object53862:head,818'4992)", 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:34.166519+0000", 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 85.162025912000004, 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.68985124099999995, 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.166519+0000", 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.166519+0000", 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.166520+0000", 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6300000000000005e-07 2024-09-17T16:39:59.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.166524+0000", 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1729999999999997e-06 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.856282+0000", 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.68975861000000005 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.856287+0000", 2024-09-17T16:39:59.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8010000000000003e-06 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.856328+0000", 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0738000000000002e-05 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.856333+0000", 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1420000000000001e-06 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.856370+0000", 2024-09-17T16:39:59.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7214e-05 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:8124 9.7 9.3fd06627 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:08.294658+0000", 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 51.033887069000002, 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91228819400000005, 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:39:59.431 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8124 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294658+0000", 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294658+0000", 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294659+0000", 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9999999999999995e-07 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294709+0000", 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0549e-05 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294720+0000", 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0678e-05 2024-09-17T16:39:59.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294725+0000", 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0039999999999999e-06 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294775+0000", 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9611000000000003e-05 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294782+0000", 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.844e-06 2024-09-17T16:39:59.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.206917+0000", 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91213532399999997 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.206946+0000", 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9184e-05 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:39:59.435 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:39:59.436 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:39:59.436 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:39:59 smithi173 ceph-mon[30530]: pgmap v1966: 27 pgs: 27 active+clean; 4.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:39:59.466 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:59.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:39:59.473 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:39:59 smithi017 ceph-mon[27571]: pgmap v1966: 27 pgs: 27 active+clean; 4.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:39:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:39:59 smithi071 ceph-mon[30002]: pgmap v1966: 27 pgs: 27 active+clean; 4.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:39:59.568 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:39:59.593 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:59.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:39:59.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:39:59.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:39:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:59.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:39:59.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:39:59.855 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:39:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:59.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:39:59.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:39:59.984 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:39:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:39:59.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:39:59.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 15 71004 70989 79.2215 69.5 0.00972559 0.0126162 2024-09-17T16:40:00.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:00.112 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:00.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:00.161 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:40:00.162 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:40:00.189 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:40:00.189 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:40:00.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:00.215 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:40:00.216 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:40:00.242 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:40:00.242 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:40:00.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:00.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:00.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:00.269 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:40:00.269 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:40:00.292 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:40:00.293 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:40:00.320 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:40:00.320 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:40:00.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:40:00.345 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:40:00.345 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:40:00.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:00.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:00.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:00.381 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:40:00.381 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:40:00.416 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:40:00.417 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:40:00.417 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2182798889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:00.417 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:00 smithi017 ceph-mon[27571]: osdmap e893: 12 total, 12 up, 11 in 2024-09-17T16:40:00.417 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:00 smithi017 ceph-mon[27571]: overall HEALTH_WARN noscrub flag(s) set 2024-09-17T16:40:00.453 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:40:00.454 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:40:00.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:00.480 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:40:00.480 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:40:00.509 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:00.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:00.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2182798889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:00 smithi071 ceph-mon[30002]: osdmap e893: 12 total, 12 up, 11 in 2024-09-17T16:40:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:00 smithi071 ceph-mon[30002]: overall HEALTH_WARN noscrub flag(s) set 2024-09-17T16:40:00.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:00.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:00.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:00.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2182798889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:00 smithi173 ceph-mon[30530]: osdmap e893: 12 total, 12 up, 11 in 2024-09-17T16:40:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:00 smithi173 ceph-mon[30530]: overall HEALTH_WARN noscrub flag(s) set 2024-09-17T16:40:00.661 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:40:00.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:00.764 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:00.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:00.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:00.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:00.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:00.992 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:00.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 72207 72191 79.1496 75.125 0.00406843 0.0126284 2024-09-17T16:40:01.017 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:01.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:01.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:01.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:01.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:01.235 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:40:01.235 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:40:01.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:01.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:01.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:01.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:01.407 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:01.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:01.470 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:01 smithi017 ceph-mon[27571]: pgmap v1968: 27 pgs: 27 active+clean; 4.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-17T16:40:01.510 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:01.534 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:01.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:01.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:01.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:01 smithi071 ceph-mon[30002]: pgmap v1968: 27 pgs: 27 active+clean; 4.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-17T16:40:01.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:01.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:01 smithi173 ceph-mon[30530]: pgmap v1968: 27 pgs: 27 active+clean; 4.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-17T16:40:01.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:01.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:01.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:01.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:01.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:01.900 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:01.906 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:40:01.924 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:01.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:01.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-09-17T16:40:01.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 15 73412 73397 79.0844 75.375 0.0060609 0.0126371 2024-09-17T16:40:02.007 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:02.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:02.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:02.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:02.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:02.179 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:02.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:02.219 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:40:02.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:02.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:02.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:02.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:02.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:02.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:02.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:02.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:02.557 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:02.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:02.611 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:02.659 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:02.685 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:02.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:02.788 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:02.815 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:02.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:02.918 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:02.942 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:02.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:02.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 74135 74119 78.5087 45.125 0.118555 0.0127299 2024-09-17T16:40:03.044 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:03.071 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:03.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:03.173 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:03.200 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:03.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:03.263 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:40:03.301 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:03.323 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:03.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:03.426 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:03.452 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:03.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:03.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:03.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:03 smithi017 ceph-mon[27571]: pgmap v1969: 27 pgs: 27 active+clean; 4.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 56 MiB/s wr, 894 op/s 2024-09-17T16:40:03.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2527460095' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:03.510 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:40:03.523 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:40:03.523 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:40:03.523 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:03 smithi071 ceph-mon[30002]: pgmap v1969: 27 pgs: 27 active+clean; 4.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 56 MiB/s wr, 894 op/s 2024-09-17T16:40:03.523 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2527460095' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:03.553 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:03.579 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:03.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:03 smithi173 ceph-mon[30530]: pgmap v1969: 27 pgs: 27 active+clean; 4.3 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 56 MiB/s wr, 894 op/s 2024-09-17T16:40:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2527460095' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:03.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:03.709 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:03.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:03.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:03.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:03.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:03.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:03.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:03.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:03.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:40:03.991323+0000 min lat: 0.00107045 max lat: 2.12556 avg lat: 0.0127326 2024-09-17T16:40:03.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:40:03.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 75389 75373 78.5064 78.375 0.0330936 0.0127326 2024-09-17T16:40:04.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:04.104 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:04.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:04.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:04.206 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:04.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:40:04.236 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:04.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:04.337 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:04.361 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:04.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:04.463 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:04.488 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:04.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:04.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:04 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:04.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2527460095' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:04.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:04 smithi017 ceph-mon[27571]: osdmap e894: 12 total, 12 up, 11 in 2024-09-17T16:40:04.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:04 smithi017 ceph-mon[27571]: 1.9 scrub starts 2024-09-17T16:40:04.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:04 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:04.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2527460095' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:04.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:04 smithi071 ceph-mon[30002]: osdmap e894: 12 total, 12 up, 11 in 2024-09-17T16:40:04.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:04 smithi071 ceph-mon[30002]: 1.9 scrub starts 2024-09-17T16:40:04.582 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:40:04.591 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:04.621 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:04.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:04.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:04 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2527460095' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:04 smithi173 ceph-mon[30530]: osdmap e894: 12 total, 12 up, 11 in 2024-09-17T16:40:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:04 smithi173 ceph-mon[30530]: 1.9 scrub starts 2024-09-17T16:40:04.723 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:04.751 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:04.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:04.753 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-09-17T16:40:04.852 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:04.853 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:04.878 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:04.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:04.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:04.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 76653 76637 78.5144 79 0.0167038 0.0127309 2024-09-17T16:40:05.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:05.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:05.112 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:05.130 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:40:05.130 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:40:05.136 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:05.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:05.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:05.262 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:05.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:05.364 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:05.388 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:05.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:05.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:05.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:05.515 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:05.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:05.539 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:05 smithi017 ceph-mon[27571]: 1.7 scrub starts 2024-09-17T16:40:05.539 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:05 smithi017 ceph-mon[27571]: pgmap v1971: 27 pgs: 27 active+clean; 4.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-17T16:40:05.539 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:05 smithi017 ceph-mon[27571]: 1.a scrub starts 2024-09-17T16:40:05.539 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:05 smithi017 ceph-mon[27571]: 1.3 scrub starts 2024-09-17T16:40:05.539 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:05 smithi017 ceph-mon[27571]: 1.6 scrub starts 2024-09-17T16:40:05.540 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:05 smithi017 ceph-mon[27571]: 1.8 scrub starts 2024-09-17T16:40:05.540 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:05 smithi017 ceph-mon[27571]: osdmap e895: 12 total, 12 up, 11 in 2024-09-17T16:40:05.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:05 smithi071 ceph-mon[30002]: 1.7 scrub starts 2024-09-17T16:40:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:05 smithi071 ceph-mon[30002]: pgmap v1971: 27 pgs: 27 active+clean; 4.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-17T16:40:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:05 smithi071 ceph-mon[30002]: 1.a scrub starts 2024-09-17T16:40:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:05 smithi071 ceph-mon[30002]: 1.3 scrub starts 2024-09-17T16:40:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:05 smithi071 ceph-mon[30002]: 1.6 scrub starts 2024-09-17T16:40:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:05 smithi071 ceph-mon[30002]: 1.8 scrub starts 2024-09-17T16:40:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:05 smithi071 ceph-mon[30002]: osdmap e895: 12 total, 12 up, 11 in 2024-09-17T16:40:05.583 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:05.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:05.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:05.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:05 smithi173 ceph-mon[30530]: 1.7 scrub starts 2024-09-17T16:40:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:05 smithi173 ceph-mon[30530]: pgmap v1971: 27 pgs: 27 active+clean; 4.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-17T16:40:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:05 smithi173 ceph-mon[30530]: 1.a scrub starts 2024-09-17T16:40:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:05 smithi173 ceph-mon[30530]: 1.3 scrub starts 2024-09-17T16:40:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:05 smithi173 ceph-mon[30530]: 1.6 scrub starts 2024-09-17T16:40:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:05 smithi173 ceph-mon[30530]: 1.8 scrub starts 2024-09-17T16:40:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:05 smithi173 ceph-mon[30530]: osdmap e895: 12 total, 12 up, 11 in 2024-09-17T16:40:05.745 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:05.769 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:05.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:05.836 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:40:05.871 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:05.896 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:05.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:05.992 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 77704 77688 78.3074 65.6875 0.0458209 0.0127643 2024-09-17T16:40:05.998 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:06.024 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:06.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:06.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:06.143 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:40:06.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:06.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:06.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:06.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:06.282 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:06.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:06.384 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:06.397 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:40:06.397 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:40:06.412 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:06.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:06.431 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:40:06.432 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:40:06.432 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:06 smithi017 ceph-mon[27571]: 1.2 scrub starts 2024-09-17T16:40:06.432 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/921800204' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:06.432 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:06 smithi017 ceph-mon[27571]: 1.5 scrub starts 2024-09-17T16:40:06.463 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:40:06.463 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:40:06.488 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:40:06.489 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:40:06.512 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:40:06.513 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:40:06.514 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:06.536 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:40:06.536 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:40:06.539 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:06.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:06.557 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:40:06.558 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:40:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:06 smithi071 ceph-mon[30002]: 1.2 scrub starts 2024-09-17T16:40:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/921800204' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:06 smithi071 ceph-mon[30002]: 1.5 scrub starts 2024-09-17T16:40:06.584 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:40:06.584 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:40:06.609 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:40:06.609 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:40:06.637 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:40:06.638 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:40:06.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:06 smithi173 ceph-mon[30530]: 1.2 scrub starts 2024-09-17T16:40:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/921800204' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:06 smithi173 ceph-mon[30530]: 1.5 scrub starts 2024-09-17T16:40:06.668 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:40:06.668 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:40:06.670 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:06.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:06.699 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:40:06.700 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:40:06.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:06.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:06.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:06.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:06.925 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:06.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:06.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:06.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 78913 78897 78.2637 75.5625 0.00552584 0.0127725 2024-09-17T16:40:07.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:07.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:07.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:07.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:07.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:07.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:07.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:07.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:07.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:07.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:07.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:07.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:07.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:07 smithi071 ceph-mon[30002]: pgmap v1973: 27 pgs: 27 active+clean; 4.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s 2024-09-17T16:40:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:07 smithi071 ceph-mon[30002]: 1.4 scrub starts 2024-09-17T16:40:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:07 smithi071 ceph-mon[30002]: 1.1 scrub starts 2024-09-17T16:40:07.566 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:07.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:07 smithi173 ceph-mon[30530]: pgmap v1973: 27 pgs: 27 active+clean; 4.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s 2024-09-17T16:40:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:07 smithi173 ceph-mon[30530]: 1.4 scrub starts 2024-09-17T16:40:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:07 smithi173 ceph-mon[30530]: 1.1 scrub starts 2024-09-17T16:40:07.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:07.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:07.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:07.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:07 smithi017 ceph-mon[27571]: pgmap v1973: 27 pgs: 27 active+clean; 4.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s 2024-09-17T16:40:07.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:07 smithi017 ceph-mon[27571]: 1.4 scrub starts 2024-09-17T16:40:07.700 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:07 smithi017 ceph-mon[27571]: 1.1 scrub starts 2024-09-17T16:40:07.798 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:07.822 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:07.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:07.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:07.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:07.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:07.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 79982 79966 78.0847 66.8125 0.00539253 0.0128011 2024-09-17T16:40:08.056 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:08.081 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:08.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:08.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:08.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:08.216 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:08.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:08.318 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:08.344 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:08.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:08.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:08.471 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:08.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:08.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:08.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:08.598 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:08.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:08.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-09-17T16:40:08.700 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:08.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:08.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:08.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:08.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:08.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:08.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:08.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:08.957 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:08.982 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:08.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:08.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 15 81290 81275 78.142 81.8125 0.00506971 0.0127915 2024-09-17T16:40:09.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:09.108 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:09.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:09.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:09.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:09.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:09.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:09.300 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:09 smithi017 ceph-mon[27571]: pgmap v1974: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 4.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:40:09.308 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:09.321 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:40:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:40:09.321 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:40:09.321 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:40:09.340 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:09.368 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:09.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:09.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:09.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:09.498 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:09.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:09.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:09 smithi173 ceph-mon[30530]: pgmap v1974: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 4.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:40:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:09 smithi071 ceph-mon[30002]: pgmap v1974: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 4.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-17T16:40:09.600 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:09.627 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:09.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:09.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:09.753 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:09.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:09.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:09.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:09.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:09.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:09.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 15 82548 82533 78.1492 78.625 0.0214915 0.0127903 2024-09-17T16:40:10.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:10.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:10.127 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:10.138 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:40:10.152 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:10.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:10.186 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:10.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:40:10.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:10.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:10.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:10.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:10.410 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:10.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:10.421 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:10 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:40:10.421 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:10 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:40:10.513 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:10.542 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:10.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:10 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:40:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:10 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:40:10.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:10 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:40:10.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:10 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:40:10.643 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:10.666 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:10.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:10.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:10.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:10.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:10.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:10.921 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:10.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:10.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 83523 83507 77.8913 60.875 0.00327556 0.0128318 2024-09-17T16:40:11.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:11.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:11.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:11.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:11.172 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:11.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:11.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:11.274 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:11.302 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:11.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:11.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:11.404 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:11.432 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:11.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:11.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:11.478 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:11 smithi071 ceph-mon[30002]: pgmap v1975: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 4.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:40:11.478 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:11 smithi071 ceph-mon[30002]: osdmap e896: 12 total, 12 up, 11 in 2024-09-17T16:40:11.479 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1242912713' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:40:11.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:11.540 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:11 smithi173 ceph-mon[30530]: pgmap v1975: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 4.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:40:11.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:11 smithi173 ceph-mon[30530]: osdmap e896: 12 total, 12 up, 11 in 2024-09-17T16:40:11.541 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1242912713' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:40:11.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:11.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:11.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:11.713 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:11.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:11.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:11 smithi017 ceph-mon[27571]: pgmap v1975: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 4.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-17T16:40:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:11 smithi017 ceph-mon[27571]: osdmap e896: 12 total, 12 up, 11 in 2024-09-17T16:40:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1242912713' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:40:11.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:11.845 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:11.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:11.937 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:40:11.946 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:11.970 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:11.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:11.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 15 84398 84383 77.5509 54.75 0.0098279 0.0128895 2024-09-17T16:40:12.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:12.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:12.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:12.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:12.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:12.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:12.307 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-09-17T16:40:12.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:12.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:12.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:12.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:12.472 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:12.498 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:12.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:12.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:12 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1242912713' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:40:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:12 smithi071 ceph-mon[30002]: osdmap e897: 12 total, 12 up, 11 in 2024-09-17T16:40:12.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:12.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:12.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:12 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1242912713' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:40:12.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:12 smithi173 ceph-mon[30530]: osdmap e897: 12 total, 12 up, 11 in 2024-09-17T16:40:12.730 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:12 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1242912713' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:40:12.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:12 smithi017 ceph-mon[27571]: osdmap e897: 12 total, 12 up, 11 in 2024-09-17T16:40:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:12.762 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:12.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:12.863 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:12.891 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:12.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:12.926 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:12.927 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:12.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:12.994 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 85555 85539 77.474 72.25 0.0222157 0.0129023 2024-09-17T16:40:13.018 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:13.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:13.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:13.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:13.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:13.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:13.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:13.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:13.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:13.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:13.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:13.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:13.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:13 smithi017 ceph-mon[27571]: pgmap v1978: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 5.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-17T16:40:13.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:13 smithi017 ceph-mon[27571]: 9.8 scrub starts 2024-09-17T16:40:13.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:13 smithi017 ceph-mon[27571]: 9.4 scrub starts 2024-09-17T16:40:13.520 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:13.549 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:13.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:13.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:13.577 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:40:13.578 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:40:13.603 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:13.603 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:40:13.604 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:40:13.631 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:40:13.631 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:40:13.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:13 smithi173 ceph-mon[30530]: pgmap v1978: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 5.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-17T16:40:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:13 smithi173 ceph-mon[30530]: 9.8 scrub starts 2024-09-17T16:40:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:13 smithi173 ceph-mon[30530]: 9.4 scrub starts 2024-09-17T16:40:13.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:13.654 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:40:13.655 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:40:13.677 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:13.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:13.683 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:40:13.683 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:40:13.711 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:40:13.712 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:40:13.737 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:40:13.737 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:40:13.763 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:40:13.763 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:40:13.779 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:13.789 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:40:13.790 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:40:13.805 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:13.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:13 smithi071 ceph-mon[30002]: pgmap v1978: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 5.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-17T16:40:13.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:13 smithi071 ceph-mon[30002]: 9.8 scrub starts 2024-09-17T16:40:13.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:13 smithi071 ceph-mon[30002]: 9.4 scrub starts 2024-09-17T16:40:13.815 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:40:13.816 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:40:13.838 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:40:13.839 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:40:13.869 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:40:13.869 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:40:13.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:13.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:13.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:13.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 86622 86606 77.3198 66.6875 0.00258001 0.0129276 2024-09-17T16:40:14.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:14.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:14.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:14.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:14.182 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:14.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:14.284 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:14.307 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:14.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:14.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:14.362 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:40:14.363 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:40:14.409 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:14.433 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:14.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:14 smithi017 ceph-mon[27571]: 9.d scrub starts 2024-09-17T16:40:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:14 smithi017 ceph-mon[27571]: 9.4 scrub ok 2024-09-17T16:40:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3213714512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:14.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:14 smithi017 ceph-mon[27571]: 9.8 scrub ok 2024-09-17T16:40:14.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:14.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:14.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:14 smithi173 ceph-mon[30530]: 9.d scrub starts 2024-09-17T16:40:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:14 smithi173 ceph-mon[30530]: 9.4 scrub ok 2024-09-17T16:40:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3213714512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:14 smithi173 ceph-mon[30530]: 9.8 scrub ok 2024-09-17T16:40:14.667 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:14.695 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:14.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:14.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:14.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:14 smithi071 ceph-mon[30002]: 9.d scrub starts 2024-09-17T16:40:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:14 smithi071 ceph-mon[30002]: 9.4 scrub ok 2024-09-17T16:40:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3213714512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:14 smithi071 ceph-mon[30002]: 9.8 scrub ok 2024-09-17T16:40:14.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:14.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:14.859 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:40:14.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:14.961 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:14.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:14.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 87689 87673 77.17 66.6875 0.0146546 0.012953 2024-09-17T16:40:15.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:15.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:15.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:15.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:15.217 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:15.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:15.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:15.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:15.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:15.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:15.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:15.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:15.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:15 smithi017 ceph-mon[27571]: pgmap v1979: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 5.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-17T16:40:15.580 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:15.606 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:15.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:15.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:15 smithi173 ceph-mon[30530]: pgmap v1979: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 5.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-17T16:40:15.708 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:15.739 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:15.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:15 smithi071 ceph-mon[30002]: pgmap v1979: 27 pgs: 2 active+clean+scrubbing, 25 active+clean; 5.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-17T16:40:15.840 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:15.866 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:15.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:15.968 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:15.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 88761 88745 77.0287 67 0.0102004 0.0129767 2024-09-17T16:40:15.997 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:15.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:16.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:16.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:16.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:16.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:16.259 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:16.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:16.305 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:40:16.361 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:16.387 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:16.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:16.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:16 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:40:16.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:16 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:40:16.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:16 smithi017 ceph-mon[27571]: 9.d scrub ok 2024-09-17T16:40:16.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:16 smithi017 ceph-mon[27571]: 9.7 scrub starts 2024-09-17T16:40:16.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:16 smithi017 ceph-mon[27571]: osdmap e898: 12 total, 12 up, 11 in 2024-09-17T16:40:16.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:16 smithi017 ceph-mon[27571]: 9.a scrub starts 2024-09-17T16:40:16.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:16.516 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:16.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:16.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:16.615 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:16.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:16.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:16.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:16.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:16 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:40:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:16 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:40:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:16 smithi173 ceph-mon[30530]: 9.d scrub ok 2024-09-17T16:40:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:16 smithi173 ceph-mon[30530]: 9.7 scrub starts 2024-09-17T16:40:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:16 smithi173 ceph-mon[30530]: osdmap e898: 12 total, 12 up, 11 in 2024-09-17T16:40:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:16 smithi173 ceph-mon[30530]: 9.a scrub starts 2024-09-17T16:40:16.653 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:16.746 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:16.773 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:16.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:16 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:40:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:16 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:40:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:16 smithi071 ceph-mon[30002]: 9.d scrub ok 2024-09-17T16:40:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:16 smithi071 ceph-mon[30002]: 9.7 scrub starts 2024-09-17T16:40:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:16 smithi071 ceph-mon[30002]: osdmap e898: 12 total, 12 up, 11 in 2024-09-17T16:40:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:16 smithi071 ceph-mon[30002]: 9.a scrub starts 2024-09-17T16:40:16.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:16.905 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:16.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:16.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 15 89739 89724 76.8116 61.1875 0.0329599 0.0130127 2024-09-17T16:40:17.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:17.032 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:17.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:17.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:17.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:17.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:17.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:17.297 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:17.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:17.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:17.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:17.431 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:17.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:17.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:17.559 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:17.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:17.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:17 smithi173 ceph-mon[30530]: pgmap v1981: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-17T16:40:17.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:17.684 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:17.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:17.736 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:40:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:17 smithi017 ceph-mon[27571]: pgmap v1981: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-17T16:40:17.786 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:17.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:17 smithi071 ceph-mon[30002]: pgmap v1981: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-17T16:40:17.814 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:17.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:17.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:17.948 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:17.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:17.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 90717 90701 76.5987 61.0625 0.0521133 0.0130478 2024-09-17T16:40:18.050 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:18.075 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:18.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:18.177 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:18.200 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:18.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:18.302 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:18.325 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:18.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:18.427 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:18.448 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:18.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:18.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:18.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:18.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:18.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:18 smithi173 ceph-mon[30530]: 9.3 scrub starts 2024-09-17T16:40:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:18 smithi173 ceph-mon[30530]: 9.a scrub ok 2024-09-17T16:40:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:18 smithi173 ceph-mon[30530]: 9.7 scrub ok 2024-09-17T16:40:18.681 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:18 smithi017 ceph-mon[27571]: 9.3 scrub starts 2024-09-17T16:40:18.681 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:18 smithi017 ceph-mon[27571]: 9.a scrub ok 2024-09-17T16:40:18.681 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:18 smithi017 ceph-mon[27571]: 9.7 scrub ok 2024-09-17T16:40:18.689 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:18.700 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:40:18.700 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:40:18.716 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:18.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:18.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:18 smithi071 ceph-mon[30002]: 9.3 scrub starts 2024-09-17T16:40:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:18 smithi071 ceph-mon[30002]: 9.a scrub ok 2024-09-17T16:40:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:18 smithi071 ceph-mon[30002]: 9.7 scrub ok 2024-09-17T16:40:18.818 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:18.850 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:18.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:18.952 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:18.981 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:18.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:18.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 91716 91700 76.4098 62.4375 0.0150485 0.0130826 2024-09-17T16:40:19.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:19.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:19.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:19.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:19.253 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:19.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:19.337 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:40:19.355 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:19.385 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:19.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:19.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:19.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:19.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:19.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:19.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:19 smithi173 ceph-mon[30530]: pgmap v1982: 27 pgs: 27 active+clean; 5.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-17T16:40:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:19 smithi173 ceph-mon[30530]: 9.3 scrub ok 2024-09-17T16:40:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3900984052' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:19 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:19.646 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:19.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:19 smithi017 ceph-mon[27571]: pgmap v1982: 27 pgs: 27 active+clean; 5.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-17T16:40:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:19 smithi017 ceph-mon[27571]: 9.3 scrub ok 2024-09-17T16:40:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3900984052' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:19 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:19.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:19.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:19.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:19.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:19 smithi071 ceph-mon[30002]: pgmap v1982: 27 pgs: 27 active+clean; 5.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-17T16:40:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:19 smithi071 ceph-mon[30002]: 9.3 scrub ok 2024-09-17T16:40:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3900984052' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:19 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:19.817 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-17T16:40:19.877 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:19.900 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:19.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:19.993 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 92831 92815 76.3213 69.6875 0.0134117 0.0130949 2024-09-17T16:40:20.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:20.024 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:20.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:20.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:20.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:20.151 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:20.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:20.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:20.279 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:20.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:20.381 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:20.405 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:20.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:20 smithi017 ceph-mon[27571]: 9.f scrub starts 2024-09-17T16:40:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:20 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:40:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:20 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:40:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:20 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:20.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:20 smithi017 ceph-mon[27571]: osdmap e899: 12 total, 12 up, 11 in 2024-09-17T16:40:20.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:20.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:20.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:20.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:20.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:20 smithi173 ceph-mon[30530]: 9.f scrub starts 2024-09-17T16:40:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:20 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:40:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:20 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:40:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:20 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:20 smithi173 ceph-mon[30530]: osdmap e899: 12 total, 12 up, 11 in 2024-09-17T16:40:20.664 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:20.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:20.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:20.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:20.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:20 smithi071 ceph-mon[30002]: 9.f scrub starts 2024-09-17T16:40:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:20 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:40:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:20 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:40:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:20 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:20 smithi071 ceph-mon[30002]: osdmap e899: 12 total, 12 up, 11 in 2024-09-17T16:40:20.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:20.922 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:20.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:20.994 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 93849 93833 76.1564 63.625 0.00435921 0.0131249 2024-09-17T16:40:21.025 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:21.048 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:21.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:21.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:21.174 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:21.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:21.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:21.298 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:21.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:21.308 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:21.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:40:21.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:21.425 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:21.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:21.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:21.527 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:21.556 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:21.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:21.624 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:21 smithi017 ceph-mon[27571]: pgmap v1984: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.6 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-17T16:40:21.624 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:21 smithi017 ceph-mon[27571]: 9.f scrub ok 2024-09-17T16:40:21.634 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:21 smithi173 ceph-mon[30530]: pgmap v1984: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.6 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-17T16:40:21.634 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:21 smithi173 ceph-mon[30530]: 9.f scrub ok 2024-09-17T16:40:21.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:21.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:21.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:21.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:21.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:21 smithi071 ceph-mon[30002]: pgmap v1984: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.6 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-17T16:40:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:21 smithi071 ceph-mon[30002]: 9.f scrub ok 2024-09-17T16:40:21.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:21.823 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:21.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:21.928 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:40:21.929 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:40:21.956 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:21.957 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:21.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:21.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:21.963 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:40:21.963 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:40:21.994 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 94775 94759 75.9219 57.875 0.00989763 0.0131667 2024-09-17T16:40:21.995 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:40:21.995 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:40:22.005 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:22.024 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:40:22.024 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:40:22.049 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:40:22.049 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:40:22.059 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:22.076 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:40:22.077 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:40:22.084 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:22.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:22.102 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:40:22.102 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:40:22.128 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:40:22.129 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:40:22.159 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:40:22.159 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:40:22.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:22.187 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:40:22.187 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:40:22.214 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:40:22.215 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:40:22.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:22.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:22.237 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:40:22.238 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:40:22.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:22.347 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:22.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:22.448 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:22.476 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:22.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:22.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:22.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2196786180' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:22.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2196786180' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:22.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:22.604 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:22.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:22.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:40:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2196786180' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:22.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:22.733 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:22.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:22.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:22.835 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:22.863 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:22.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:22.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:22.994 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:22.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:22.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 95908 95892 75.8572 70.8125 0.00696955 0.0131772 2024-09-17T16:40:23.095 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:23.122 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:23.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:23.192 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:40:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:40:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27874.0:96123 9.2 9:43fd92a4:::benchmark_data_smithi173_150363_object96122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e899)", 2024-09-17T16:40:23.192 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:40:23.173508+0000", 2024-09-17T16:40:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.016395574999999999, 2024-09-17T16:40:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016417001000000001, 2024-09-17T16:40:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:40:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:40:23.193 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27874", 2024-09-17T16:40:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:40:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 96123 2024-09-17T16:40:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:23.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.173508+0000", 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.173508+0000", 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:23.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.173510+0000", 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.454e-06 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.173553+0000", 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3374000000000002e-05 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.173567+0000", 2024-09-17T16:40:23.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4015999999999999e-05 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.173575+0000", 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6729999999999992e-06 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.173634+0000", 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8882999999999998e-05 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:23.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.173783+0000", 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000149498 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,11", 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.173932+0000", 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014935799999999999 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.177233+0000", 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0033003500000000001 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:23.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:23.178700+0000", 2024-09-17T16:40:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0014672039999999999 2024-09-17T16:40:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:40:23.201 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 1 2024-09-17T16:40:23.202 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:40:23.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:23.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:23.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:23.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:23.386 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:23.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:23.488 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:23 smithi017 ceph-mon[27571]: pgmap v1985: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.6 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-17T16:40:23.512 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:23.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:23.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:23.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:23.638 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:23.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:23.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:23 smithi173 ceph-mon[30530]: pgmap v1985: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.6 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-17T16:40:23.739 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:23.764 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:23.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:23.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:23 smithi071 ceph-mon[30002]: pgmap v1985: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.6 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-17T16:40:23.866 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:23.892 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:23.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:23.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:23.994 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:23.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-09-17T16:40:23.992981+0000 min lat: 0.00107045 max lat: 2.12556 avg lat: 0.0131926 2024-09-17T16:40:23.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T16:40:23.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 97006 96990 75.7667 68.625 0.00370795 0.0131926 2024-09-17T16:40:24.020 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:24.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:24.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:24.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:24.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:24.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:24.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:24.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:24.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:24.407 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:24.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:24.510 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:24.537 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:24.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:24.638 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:24.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:24 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:40:24.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:24 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:40:24.662 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:24.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:24.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:24 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:40:24.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:24 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:40:24.764 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:24.795 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:24.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:24.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:24 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:40:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:24 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:40:24.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:24.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:40:24.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:24.931 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:24.962 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:40:24.994 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 15 98014 97999 75.6098 63.0625 0.00976 0.0132211 2024-09-17T16:40:25.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:25.065 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:25.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:25.167 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:25.194 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:25.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:25.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:25.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:25.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:25.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:25.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:25.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:25.464 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:25.472 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:25.478 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-17T16:40:25.528 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:25.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:25.588 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:25.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:25.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:25 smithi017 ceph-mon[27571]: pgmap v1986: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-17T16:40:25.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:25 smithi017 ceph-mon[27571]: osdmap e900: 12 total, 12 up, 11 in 2024-09-17T16:40:25.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:25 smithi017 ceph-mon[27571]: osdmap e901: 12 total, 12 up, 11 in 2024-09-17T16:40:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:25 smithi173 ceph-mon[30530]: pgmap v1986: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-17T16:40:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:25 smithi173 ceph-mon[30530]: osdmap e900: 12 total, 12 up, 11 in 2024-09-17T16:40:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:25 smithi173 ceph-mon[30530]: osdmap e901: 12 total, 12 up, 11 in 2024-09-17T16:40:25.690 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:25.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:25.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:25.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:25 smithi071 ceph-mon[30002]: pgmap v1986: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-17T16:40:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:25 smithi071 ceph-mon[30002]: osdmap e900: 12 total, 12 up, 11 in 2024-09-17T16:40:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:25 smithi071 ceph-mon[30002]: osdmap e901: 12 total, 12 up, 11 in 2024-09-17T16:40:25.819 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:25.841 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:25.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:25.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:25.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:25.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:25.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:25.994 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 98996 98980 75.4354 61.3125 0.0438195 0.0132316 2024-09-17T16:40:26.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:26.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:26.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:26.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:26.219 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:26.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:26.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:26.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:26.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:26.453 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:26.481 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:26.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:26.542 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:26.578 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:40:26.578 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:40:26.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:26.590 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:26.607 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:40:26.607 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:40:26.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:26.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:26.633 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:40:26.633 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:40:26.661 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:40:26.662 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:40:26.686 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:40:26.687 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:40:26.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:26.713 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:40:26.714 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:40:26.738 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:26.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:26.740 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:40:26.740 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:40:26.765 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:40:26.765 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:40:26.789 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:40:26.790 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:40:26.813 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:40:26.814 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:40:26.838 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:40:26.838 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:40:26.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:26.868 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:40:26.869 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:40:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:26.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:26.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:26.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:26.994 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 99946 99930 75.2418 59.375 0.0221924 0.0132857 2024-09-17T16:40:27.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:27.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:27.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:27.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:27.131 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:27.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:27.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:27.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:27.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:27.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:27.355 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:40:27.369 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:27.397 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:27.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:27.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:27.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:27.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:27.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:27.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:27 smithi173 ceph-mon[30530]: pgmap v1989: 27 pgs: 27 active+clean; 5.9 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-17T16:40:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4164132794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1078144909' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:40:27.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:27.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:27.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:27 smithi017 ceph-mon[27571]: pgmap v1989: 27 pgs: 27 active+clean; 5.9 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-17T16:40:27.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4164132794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:27.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1078144909' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:40:27.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:27.782 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:27.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:27.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:27.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:27 smithi071 ceph-mon[30002]: pgmap v1989: 27 pgs: 27 active+clean; 5.9 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-17T16:40:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4164132794' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1078144909' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:40:27.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:27.909 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:27.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:27.994 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 15 100895 100880 75.0529 59.375 0.0110149 0.0133192 2024-09-17T16:40:28.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:28.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:28.039 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:28.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:28.166 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:28.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:28.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:28.290 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:28.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:28.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:28.418 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:28.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:28 smithi017 ceph-mon[27571]: 9.b scrub starts 2024-09-17T16:40:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:28 smithi017 ceph-mon[27571]: 9.1 scrub starts 2024-09-17T16:40:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:28 smithi017 ceph-mon[27571]: 9.2 scrub starts 2024-09-17T16:40:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:28 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1078144909' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:40:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:28 smithi017 ceph-mon[27571]: osdmap e902: 12 total, 12 up, 11 in 2024-09-17T16:40:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:28 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:40:28.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:28 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:40:28.520 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:28.549 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:28.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:28.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-17T16:40:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:28 smithi173 ceph-mon[30530]: 9.b scrub starts 2024-09-17T16:40:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:28 smithi173 ceph-mon[30530]: 9.1 scrub starts 2024-09-17T16:40:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:28 smithi173 ceph-mon[30530]: 9.2 scrub starts 2024-09-17T16:40:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:28 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1078144909' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:40:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:28 smithi173 ceph-mon[30530]: osdmap e902: 12 total, 12 up, 11 in 2024-09-17T16:40:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:28 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:40:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:28 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:40:28.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:28.678 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:28.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:28.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:28.780 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:28.801 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:28.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:28.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:28 smithi071 ceph-mon[30002]: 9.b scrub starts 2024-09-17T16:40:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:28 smithi071 ceph-mon[30002]: 9.1 scrub starts 2024-09-17T16:40:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:28 smithi071 ceph-mon[30002]: 9.2 scrub starts 2024-09-17T16:40:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:28 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1078144909' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:40:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:28 smithi071 ceph-mon[30002]: osdmap e902: 12 total, 12 up, 11 in 2024-09-17T16:40:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:28 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:40:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:28 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:40:28.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:28.926 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:28.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:28.994 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 101701 101685 74.7617 50.3125 0.00860141 0.0133651 2024-09-17T16:40:29.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:29.055 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:29.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:29.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:29.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:29.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:29.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:29.261 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:40:29.261 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:40:29.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:29.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:29.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:29.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:29.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:29.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:29.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:29.562 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:29.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:29.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:29 smithi017 ceph-mon[27571]: pgmap v1991: 27 pgs: 27 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-17T16:40:29.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:29 smithi017 ceph-mon[27571]: osdmap e903: 12 total, 12 up, 11 in 2024-09-17T16:40:29.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:29 smithi173 ceph-mon[30530]: pgmap v1991: 27 pgs: 27 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-17T16:40:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:29 smithi173 ceph-mon[30530]: osdmap e903: 12 total, 12 up, 11 in 2024-09-17T16:40:29.664 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:29.693 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:29.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:29.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:29.795 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:29.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:29 smithi071 ceph-mon[30002]: pgmap v1991: 27 pgs: 27 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-17T16:40:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:29 smithi071 ceph-mon[30002]: osdmap e903: 12 total, 12 up, 11 in 2024-09-17T16:40:29.823 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:29.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:29.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:29.953 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:29.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:29.994 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 102666 102650 74.5937 60.3125 0.00600085 0.0134009 2024-09-17T16:40:30.055 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:30.080 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:30.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:30.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:30.213 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:30.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:30.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:30.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:30.339 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:30.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:30.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:30.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:30.465 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:30.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:30.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:30.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:30.592 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:30.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:30.643 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:30.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:40:30.694 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:30.720 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:30.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:30.822 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:30.848 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:30.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:30.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:30.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:30.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:30.994 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 103592 103576 74.4014 57.875 0.010911 0.0134348 2024-09-17T16:40:31.061 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:40:31.085 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:31.113 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:31.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:31.215 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:31.242 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:31.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:31.344 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:31.371 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:31.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:31.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:31.494 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:31.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:31.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:31.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:31.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:31.641 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:31.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:31.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:31 smithi017 ceph-mon[27571]: pgmap v1993: 27 pgs: 27 active+clean; 6.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-17T16:40:31.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:31 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:40:31.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:31 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:40:31.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:31 smithi017 ceph-mon[27571]: osdmap e904: 12 total, 12 up, 11 in 2024-09-17T16:40:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:31 smithi173 ceph-mon[30530]: pgmap v1993: 27 pgs: 27 active+clean; 6.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-17T16:40:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:31 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:40:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:31 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:40:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:31 smithi173 ceph-mon[30530]: osdmap e904: 12 total, 12 up, 11 in 2024-09-17T16:40:31.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:31.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:31.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:31 smithi071 ceph-mon[30002]: pgmap v1993: 27 pgs: 27 active+clean; 6.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-17T16:40:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:31 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:40:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:31 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:40:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:31 smithi071 ceph-mon[30002]: osdmap e904: 12 total, 12 up, 11 in 2024-09-17T16:40:31.878 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:31.902 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:31.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:31.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 104718 104702 74.3556 70.375 0.0073757 0.0134427 2024-09-17T16:40:32.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:32.028 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:32.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:32.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:32.143 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:40:32.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:32.164 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:32.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:32.293 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:32.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:32.395 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:32.421 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:32.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:32.523 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:32.546 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:32.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:32.606 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:40:32.606 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:40:32.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:32.671 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:32.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:32.773 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:32.801 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:32.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:32.863 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:32.902 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:32.929 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:32.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:32.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:32.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 105719 105703 74.2231 62.5625 0.0136081 0.0134676 2024-09-17T16:40:33.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:33.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:33.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:33.164 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:33.190 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:33.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:33.292 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:33.317 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:33.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:33.419 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:33.446 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:33.446 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:33.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:33.481 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:33 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:40:33.481 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:33 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:40:33.481 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:33 smithi017 ceph-mon[27571]: pgmap v1995: 27 pgs: 27 active+clean; 6.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 57 MiB/s wr, 917 op/s 2024-09-17T16:40:33.481 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:33 smithi017 ceph-mon[27571]: osdmap e905: 12 total, 12 up, 11 in 2024-09-17T16:40:33.482 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:40:33.548 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:33.573 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:33.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:33 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:40:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:33 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:40:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:33 smithi173 ceph-mon[30530]: pgmap v1995: 27 pgs: 27 active+clean; 6.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 57 MiB/s wr, 917 op/s 2024-09-17T16:40:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:33 smithi173 ceph-mon[30530]: osdmap e905: 12 total, 12 up, 11 in 2024-09-17T16:40:33.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:33.702 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:33.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:33.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:33.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:33 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:40:33.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:33 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:40:33.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:33 smithi071 ceph-mon[30002]: pgmap v1995: 27 pgs: 27 active+clean; 6.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 57 MiB/s wr, 917 op/s 2024-09-17T16:40:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:33 smithi071 ceph-mon[30002]: osdmap e905: 12 total, 12 up, 11 in 2024-09-17T16:40:33.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:33.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:33.931 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:33.957 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:33.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:33.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:33.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 7 106902 106895 74.2261 74.5 0.0302334 0.013467 2024-09-17T16:40:34.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:34.076 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:34.083 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:34.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:34.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:34.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:34.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:34.207 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:34.308 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:34.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:34.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:34.393 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:40:34.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:34.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:34.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:34.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:34.560 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:34.586 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:34.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:34.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1238017109' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:34.685 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1238017109' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:34.688 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:34.710 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:34.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:34.813 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1238017109' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:34.839 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:34.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:34.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:34.941 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:34.964 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:34.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:34.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.7875 2024-09-17T16:40:34.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 106902 2024-09-17T16:40:34.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-09-17T16:40:34.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-09-17T16:40:34.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 73.5935 2024-09-17T16:40:34.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 21.3398 2024-09-17T16:40:34.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 147.625 2024-09-17T16:40:34.995 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 3.625 2024-09-17T16:40:34.996 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1177 2024-09-17T16:40:34.996 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 341.437 2024-09-17T16:40:34.996 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2362 2024-09-17T16:40:34.996 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 58 2024-09-17T16:40:34.996 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0134903 2024-09-17T16:40:34.996 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.0312082 2024-09-17T16:40:34.996 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 2.12556 2024-09-17T16:40:34.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00107045 2024-09-17T16:40:35.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-09-17T16:40:35.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:35.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:35.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:35.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:35.206 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:35.234 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:35.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:35.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:35.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:35.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:35.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:35 smithi017 ceph-mon[27571]: pgmap v1997: 27 pgs: 27 active+clean; 6.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 53 MiB/s wr, 842 op/s 2024-09-17T16:40:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:35 smithi017 ceph-mon[27571]: 1.a scrub starts 2024-09-17T16:40:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:35 smithi017 ceph-mon[27571]: 1.7 scrub starts 2024-09-17T16:40:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:35 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1238017109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:35 smithi017 ceph-mon[27571]: osdmap e906: 12 total, 12 up, 11 in 2024-09-17T16:40:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:35 smithi017 ceph-mon[27571]: 1.9 scrub starts 2024-09-17T16:40:35.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:35 smithi017 ceph-mon[27571]: osdmap e907: 12 total, 12 up, 11 in 2024-09-17T16:40:35.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:35.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:35.600 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:35.623 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:35.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:35 smithi173 ceph-mon[30530]: pgmap v1997: 27 pgs: 27 active+clean; 6.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 53 MiB/s wr, 842 op/s 2024-09-17T16:40:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:35 smithi173 ceph-mon[30530]: 1.a scrub starts 2024-09-17T16:40:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:35 smithi173 ceph-mon[30530]: 1.7 scrub starts 2024-09-17T16:40:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:35 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1238017109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:35 smithi173 ceph-mon[30530]: osdmap e906: 12 total, 12 up, 11 in 2024-09-17T16:40:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:35 smithi173 ceph-mon[30530]: 1.9 scrub starts 2024-09-17T16:40:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:35 smithi173 ceph-mon[30530]: osdmap e907: 12 total, 12 up, 11 in 2024-09-17T16:40:35.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:35.751 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:35.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:35 smithi071 ceph-mon[30002]: pgmap v1997: 27 pgs: 27 active+clean; 6.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 53 MiB/s wr, 842 op/s 2024-09-17T16:40:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:35 smithi071 ceph-mon[30002]: 1.a scrub starts 2024-09-17T16:40:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:35 smithi071 ceph-mon[30002]: 1.7 scrub starts 2024-09-17T16:40:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:35 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1238017109' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:35 smithi071 ceph-mon[30002]: osdmap e906: 12 total, 12 up, 11 in 2024-09-17T16:40:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:35 smithi071 ceph-mon[30002]: 1.9 scrub starts 2024-09-17T16:40:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:35 smithi071 ceph-mon[30002]: osdmap e907: 12 total, 12 up, 11 in 2024-09-17T16:40:35.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:35.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:35.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:35.913 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:40:35.913 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:40:35.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:36.007 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:36.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:36.110 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:36.134 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:36.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:36.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:36.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:36.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:36.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:36.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:36.399 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:36.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:36.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:36.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:36.525 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:36.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:36.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:36 smithi017 ceph-mon[27571]: 1.8 scrub starts 2024-09-17T16:40:36.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:36 smithi017 ceph-mon[27571]: 1.3 scrub starts 2024-09-17T16:40:36.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:36 smithi017 ceph-mon[27571]: 1.6 scrub starts 2024-09-17T16:40:36.627 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:36.652 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:36.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:40:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:36.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:36.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:36.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:36.781 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:36 smithi071 ceph-mon[30002]: 1.8 scrub starts 2024-09-17T16:40:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:36 smithi071 ceph-mon[30002]: 1.3 scrub starts 2024-09-17T16:40:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:36 smithi071 ceph-mon[30002]: 1.6 scrub starts 2024-09-17T16:40:36.882 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:36 smithi173 ceph-mon[30530]: 1.8 scrub starts 2024-09-17T16:40:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:36 smithi173 ceph-mon[30530]: 1.3 scrub starts 2024-09-17T16:40:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:36 smithi173 ceph-mon[30530]: 1.6 scrub starts 2024-09-17T16:40:36.904 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:36.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:37.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:37.028 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 8, 9, 11, 7, 10, 5] 2024-09-17T16:40:37.028 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:40:37.028 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_7 2024-09-17T16:40:37.028 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:40:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:37.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:37.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:37.128 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:40:37.129 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:40:37.130 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:37.151 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:37.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:37.163 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:40:37.163 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:40:37.170 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:37.192 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:40:37.192 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:40:37.225 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:40:37.225 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:40:37.251 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:40:37.252 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:40:37.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:37.277 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:40:37.277 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:40:37.278 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:37.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:37.301 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:40:37.301 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:40:37.324 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:40:37.324 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:40:37.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:40:37.347 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:40:37.372 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:40:37.373 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:40:37.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:37.398 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:40:37.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:40:37.405 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:37.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:37.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:40:37.424 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:40:37.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:37.538 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:37.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:37.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:37.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:40:37.666 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:37.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:37.667 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:37 smithi017 ceph-mon[27571]: 1.2 scrub starts 2024-09-17T16:40:37.667 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:37 smithi017 ceph-mon[27571]: pgmap v2000: 27 pgs: 27 active+clean; 6.5 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 51 MiB/s wr, 809 op/s 2024-09-17T16:40:37.667 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:37 smithi017 ceph-mon[27571]: 1.5 scrub starts 2024-09-17T16:40:37.667 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2439360940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:37.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:37.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:37.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:37.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:37 smithi071 ceph-mon[30002]: 1.2 scrub starts 2024-09-17T16:40:37.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:37 smithi071 ceph-mon[30002]: pgmap v2000: 27 pgs: 27 active+clean; 6.5 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 51 MiB/s wr, 809 op/s 2024-09-17T16:40:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:37 smithi071 ceph-mon[30002]: 1.5 scrub starts 2024-09-17T16:40:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2439360940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:37 smithi173 ceph-mon[30530]: 1.2 scrub starts 2024-09-17T16:40:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:37 smithi173 ceph-mon[30530]: pgmap v2000: 27 pgs: 27 active+clean; 6.5 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 51 MiB/s wr, 809 op/s 2024-09-17T16:40:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:37 smithi173 ceph-mon[30530]: 1.5 scrub starts 2024-09-17T16:40:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2439360940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:37.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:37.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:37.930 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:37.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:38.031 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:38.058 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:38.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:38.160 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:38.194 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:38.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:38.296 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:38.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:38.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:38.390 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:40:38.390 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:40:38.390 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:40:38.390 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:40:38.390 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:40:38.424 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:38.450 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:38.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:38.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:38.578 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:38.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:38.641 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:38.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:38.706 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:38.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:38 smithi017 ceph-mon[27571]: 1.4 scrub starts 2024-09-17T16:40:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:38 smithi017 ceph-mon[27571]: 1.1 scrub starts 2024-09-17T16:40:38.808 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:38 smithi071 ceph-mon[30002]: 1.4 scrub starts 2024-09-17T16:40:38.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:38 smithi071 ceph-mon[30002]: 1.1 scrub starts 2024-09-17T16:40:38.833 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:38.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:38.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:38 smithi173 ceph-mon[30530]: 1.4 scrub starts 2024-09-17T16:40:38.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:38 smithi173 ceph-mon[30530]: 1.1 scrub starts 2024-09-17T16:40:38.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:38.962 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:38.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:39.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:39.087 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:39.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:39.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:39.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:40:39.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:39.230 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:39.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:39.332 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:39.358 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:39.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:39.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:39.460 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:39.490 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:39.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:39.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:39.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:39.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:39.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:39.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:39 smithi017 ceph-mon[27571]: pgmap v2001: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 6.4 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 53 MiB/s wr, 1.04k op/s 2024-09-17T16:40:39.746 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:39.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:39.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:39.795 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:39 smithi173 ceph-mon[30530]: pgmap v2001: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 6.4 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 53 MiB/s wr, 1.04k op/s 2024-09-17T16:40:39.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:39 smithi071 ceph-mon[30002]: pgmap v2001: 27 pgs: 4 active+clean+scrubbing, 23 active+clean; 6.4 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 53 MiB/s wr, 1.04k op/s 2024-09-17T16:40:39.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:39.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:39.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:39.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:40.012 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:40.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:40.115 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:40.140 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:40.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:40.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:40.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:40:40.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:40.267 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:40.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:40.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:40.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:40.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:40.500 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:40.525 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:40.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:40.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:40.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:40.656 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:40.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:40.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:40.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:40.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:40.888 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:40.912 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:40.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:40.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:40 smithi017 ceph-mon[27571]: pgmap v2002: 27 pgs: 6 active+clean+scrubbing, 21 active+clean; 6.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 131 B/s rd, 34 MiB/s wr, 1.00k op/s 2024-09-17T16:40:40.994 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:41.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:41.040 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:41.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:41.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:40 smithi071 ceph-mon[30002]: pgmap v2002: 27 pgs: 6 active+clean+scrubbing, 21 active+clean; 6.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 131 B/s rd, 34 MiB/s wr, 1.00k op/s 2024-09-17T16:40:41.142 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:41.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:40 smithi173 ceph-mon[30530]: pgmap v2002: 27 pgs: 6 active+clean+scrubbing, 21 active+clean; 6.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 131 B/s rd, 34 MiB/s wr, 1.00k op/s 2024-09-17T16:40:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:41.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:41.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:41.299 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:41.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:41.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:41.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:41.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:41.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:41.557 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:40:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:40:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:40:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:40:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:130635 7.8 7:14a1b993:::benchmark_data_smithi173_116396_object35035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 501.42120581099999, 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.329086555, 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 130635 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132784+0000", 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132785+0000", 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5239999999999999e-06 2024-09-17T16:40:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132792+0000", 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4860000000000002e-06 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132805+0000", 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2792999999999999e-05 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.132812+0000", 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1899999999999998e-06 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.133889+0000", 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001076881 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.134010+0000", 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120796 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,7", 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.134124+0000", 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114875 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.136882+0000", 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027574539999999999 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.136893+0000", 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.154e-05 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461760+0000", 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32486675399999998 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461812+0000", 2024-09-17T16:40:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2231000000000002e-05 2024-09-17T16:40:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:32:20.461870+0000", 2024-09-17T16:40:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8031000000000001e-05 2024-09-17T16:40:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:191190 7.e 7:7081439d:::benchmark_data_smithi173_116396_object95590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 407.10090482300001, 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.428509103, 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 191190 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453085+0000", 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453086+0000", 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3510000000000001e-06 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453092+0000", 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6010000000000002e-06 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453103+0000", 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.065e-05 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453110+0000", 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1160000000000001e-06 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453277+0000", 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016785300000000001 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453458+0000", 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018056699999999999 2024-09-17T16:40:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.453577+0000", 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000118678 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.458776+0000", 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051995009999999996 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.458855+0000", 2024-09-17T16:40:41.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8597999999999995e-05 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881478+0000", 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42262325899999997 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881537+0000", 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9354999999999997e-05 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881594+0000", 2024-09-17T16:40:41.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6573999999999997e-05 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:191190 7.e e724/629 ondisk, result = 0)", 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 407.09601177299999, 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42362284100000003, 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:40:41.571 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457978+0000", 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457979+0000", 2024-09-17T16:40:41.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.457985+0000", 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9460000000000003e-06 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881412+0000", 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.42342706200000002 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881419+0000", 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.525e-06 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881467+0000", 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7667999999999998e-05 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881474+0000", 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9489999999999999e-06 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.881601+0000", 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126797 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.16 741/739 [PushReplyOp(8:6b780127:::benchmark_data_smithi173_130566_object3478:head)])", 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 373.76331652699997, 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49350836100000001, 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.575 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790673+0000", 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790674+0000", 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4700000000000001e-07 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.790677+0000", 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6720000000000002e-06 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283734+0000", 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49305611500000002 2024-09-17T16:40:41.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283742+0000", 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7960000000000001e-06 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.283814+0000", 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1419000000000002e-05 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:28.284182+0000", 2024-09-17T16:40:41.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00036781200000000002 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:113372 8.16 8.fa455136 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:51.399293+0000", 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 230.15469637999999, 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62166801999999999, 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:40:41.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 113372 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399293+0000", 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399293+0000", 2024-09-17T16:40:41.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399294+0000", 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0300000000000001e-06 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399299+0000", 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8640000000000004e-06 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399312+0000", 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2799000000000001e-05 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399325+0000", 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3037000000000001e-05 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399386+0000", 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0699000000000003e-05 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.399394+0000", 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5150000000000006e-06 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.020931+0000", 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.62153622500000005 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.020961+0000", 2024-09-17T16:40:41.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0851e-05 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:113391 8.16 8.88e8ec76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:51.486004+0000", 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 230.067985271, 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53498688299999997, 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:40:41.584 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 113391 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486004+0000", 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486004+0000", 2024-09-17T16:40:41.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486006+0000", 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.288e-06 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486011+0000", 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.062e-06 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486022+0000", 2024-09-17T16:40:41.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486027+0000", 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2809999999999999e-06 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486072+0000", 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4428999999999998e-05 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.486080+0000", 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.072e-06 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.020986+0000", 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.53490651700000003 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.020991+0000", 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9699999999999998e-06 2024-09-17T16:40:41.589 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:113410 8.16 8.d6510676 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:51.576292+0000", 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 229.977698057, 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.444716628, 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:40:41.590 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 113410 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576292+0000", 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576292+0000", 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576293+0000", 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5060000000000001e-06 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576299+0000", 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3820000000000003e-06 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576313+0000", 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4595e-05 2024-09-17T16:40:41.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576327+0000", 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4236999999999999e-05 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576391+0000", 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3923000000000004e-05 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.576405+0000", 2024-09-17T16:40:41.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4049000000000001e-05 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.021005+0000", 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.44459960999999998 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.021008+0000", 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.326e-06 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:113412 8.16 8.4f4dd716 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:51.621393+0000", 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 229.932596413, 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.39962974299999998, 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 113412 2024-09-17T16:40:41.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621393+0000", 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621393+0000", 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621395+0000", 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1820000000000001e-06 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621401+0000", 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3579999999999999e-06 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621412+0000", 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0729e-05 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621418+0000", 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0419999999999999e-06 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621465+0000", 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7236e-05 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:51.621472+0000", 2024-09-17T16:40:41.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3130000000000001e-06 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.021020+0000", 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.399547665 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:52.021023+0000", 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2179999999999999e-06 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:151014 8.a e819/728 8:5e424e97:::benchmark_data_smithi173_130566_object56039:head v 819'9699, mlcod=819'9696)", 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.947285+0000", 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 181.60670512900001, 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.36445988699999998, 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.947285+0000", 2024-09-17T16:40:41.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.947285+0000", 2024-09-17T16:40:41.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.947285+0000", 2024-09-17T16:40:41.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5899999999999996e-07 2024-09-17T16:40:41.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.947292+0000", 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3629999999999999e-06 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293044+0000", 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.34575203199999999 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293061+0000", 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7142e-05 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293135+0000", 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4128999999999994e-05 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293149+0000", 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.311699+0000", 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.018550588999999999 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.311745+0000", 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5015999999999997e-05 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.606 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:151027 8.16 8:6fc25e02:::benchmark_data_smithi173_130566_object56052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e819)", 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.957889+0000", 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 181.59610111699999, 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.335442935, 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 151027 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957889+0000", 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957889+0000", 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957891+0000", 2024-09-17T16:40:41.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9300000000000002e-06 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957898+0000", 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6320000000000008e-06 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957914+0000", 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5889e-05 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957923+0000", 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1379999999999996e-06 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.957984+0000", 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0322000000000003e-05 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.958074+0000", 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0797000000000005e-05 2024-09-17T16:40:41.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.958141+0000", 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000004e-05 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.960378+0000", 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002237242 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.960408+0000", 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9839999999999999e-05 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293223+0000", 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33281477500000001 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293281+0000", 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7955000000000003e-05 2024-09-17T16:40:41.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293332+0000", 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0995000000000002e-05 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:151027 8.16 e819/799 ondisk, result = 0)", 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.961092+0000", 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 181.59289779700001, 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33224929399999997, 2024-09-17T16:40:41.613 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961092+0000", 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961092+0000", 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961093+0000", 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.99e-07 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961100+0000", 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7400000000000004e-06 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961136+0000", 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6019999999999997e-05 2024-09-17T16:40:41.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.961143+0000", 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2469999999999997e-06 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293207+0000", 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33206466800000001 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293216+0000", 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6759999999999996e-06 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293341+0000", 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000125244 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.617 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:151036 8.14 e819/743 8:2a2a8f98:::benchmark_data_smithi173_130566_object56061:head v 819'5028, mlcod=819'5026)", 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.965815+0000", 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 181.58817491400001, 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33232884699999998, 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.965815+0000", 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.965815+0000", 2024-09-17T16:40:41.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:41.619 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:41.620 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:41.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:41.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.965815+0000", 2024-09-17T16:40:41.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3900000000000004e-07 2024-09-17T16:40:41.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.965821+0000", 2024-09-17T16:40:41.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4e-06 2024-09-17T16:40:41.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293041+0000", 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32722048399999998 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293067+0000", 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5950000000000001e-05 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293125+0000", 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8149999999999997e-05 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293146+0000", 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0139000000000001e-05 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.298094+0000", 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049482839999999998 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.298144+0000", 2024-09-17T16:40:41.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9801e-05 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:151048 8.a e819/728 8:5047d43e:::benchmark_data_smithi173_130566_object56073:head v 819'9700, mlcod=819'9697)", 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:39.979791+0000", 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 181.574198739, 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33199636799999999, 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.624 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.979791+0000", 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.979791+0000", 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.979792+0000", 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1820000000000001e-06 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:39.979800+0000", 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1680000000000004e-06 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293188+0000", 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.313387315 2024-09-17T16:40:41.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.293194+0000", 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3169999999999998e-06 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.298071+0000", 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048773879999999999 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.298092+0000", 2024-09-17T16:40:41.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0605e-05 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.311752+0000", 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013659538000000001 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:40.311787+0000", 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5855000000000001e-05 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.628 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:184622 8.16 8:6bf99951:::benchmark_data_smithi173_130566_object89647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e845)", 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:27.116186+0000", 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 134.437803933, 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.33119328100000001, 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:40:41.629 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 184622 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116186+0000", 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116186+0000", 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116187+0000", 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.333e-06 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116193+0000", 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7100000000000004e-06 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116208+0000", 2024-09-17T16:40:41.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5121e-05 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116218+0000", 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5589999999999994e-06 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116275+0000", 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7401999999999997e-05 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116386+0000", 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011100200000000001 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.116458+0000", 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1557999999999995e-05 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.119255+0000", 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027974969999999999 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.151706+0000", 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.032451458000000002 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447257+0000", 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.29555069499999997 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447305+0000", 2024-09-17T16:40:41.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7873000000000001e-05 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447379+0000", 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4072999999999994e-05 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:184622 8.16 e845/799 ondisk, result = 0)", 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:27.118496+0000", 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 134.435493409, 2024-09-17T16:40:41.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32889318000000001, 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.118496+0000", 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.118496+0000", 2024-09-17T16:40:41.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.118497+0000", 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9299999999999997e-07 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.118501+0000", 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3599999999999998e-06 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447014+0000", 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.32851213699999998 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447020+0000", 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2689999999999996e-06 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447245+0000", 2024-09-17T16:40:41.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022503200000000001 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447252+0000", 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7469999999999999e-06 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:27.447390+0000", 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000137842 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.17883.0:188946 8.16 8:6918a138:::benchmark_data_smithi173_130566_object93971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:32.231935+0000", 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 129.32205456599999, 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38706543900000001, 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.17883", 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 188946 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.640 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231935+0000", 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231935+0000", 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231937+0000", 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231944+0000", 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1049999999999997e-06 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231962+0000", 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7714e-05 2024-09-17T16:40:41.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.231971+0000", 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9260000000000006e-06 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.232025+0000", 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4568000000000003e-05 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.232166+0000", 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000140822 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.232289+0000", 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012345499999999999 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.236743+0000", 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044531060000000001 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.259185+0000", 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022442655999999998 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618897+0000", 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.35971195 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618955+0000", 2024-09-17T16:40:41.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7742999999999999e-05 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.619001+0000", 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5769000000000002e-05 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.17883.0:188946 8.16 e848/799 ondisk, result = 0)", 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:32.234289+0000", 2024-09-17T16:40:41.646 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 129.319700641, 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38471977499999999, 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.234289+0000", 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.234289+0000", 2024-09-17T16:40:41.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.234290+0000", 2024-09-17T16:40:41.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1299999999999998e-07 2024-09-17T16:40:41.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.234297+0000", 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5969999999999997e-06 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618656+0000", 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.38435904399999998 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618661+0000", 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.733e-06 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618883+0000", 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022163900000000001 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.618892+0000", 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.85e-06 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:32.619009+0000", 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000116999 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27874.0:20411 9.7 e875/870 9:e477bdee:::benchmark_data_smithi173_150363_object20410:head v 875'1217, mlcod=875'1213)", 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:39:20.067836+0000", 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 81.486153658000006, 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.206569264, 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.651 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:39:20.067836+0000", 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:39:20.067836+0000", 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:39:20.067837+0000", 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3939999999999999e-06 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:39:20.067894+0000", 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6681000000000002e-05 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:39:20.273516+0000", 2024-09-17T16:40:41.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.20562171400000001 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:39:20.273522+0000", 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5799999999999997e-06 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:39:20.273568+0000", 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5909999999999999e-05 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:39:20.273585+0000", 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6257e-05 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:39:20.274377+0000", 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079259199999999995 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:39:20.274405+0000", 2024-09-17T16:40:41.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8136e-05 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27874.0:83249 9.7 e896/870 9:ea416073:::benchmark_data_smithi173_150363_object83248:head v 896'5202, mlcod=896'5197)", 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:40:10.636756+0000", 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.917233787000001, 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.073710364, 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.656 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.636756+0000", 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.636756+0000", 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.636757+0000", 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.15e-06 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.636937+0000", 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017989300000000001 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.708283+0000", 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.071346431000000002 2024-09-17T16:40:41.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.708289+0000", 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5899999999999998e-06 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.708344+0000", 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5408000000000002e-05 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.708358+0000", 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3664e-05 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.710432+0000", 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020737049999999999 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.710466+0000", 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4523000000000001e-05 2024-09-17T16:40:41.660 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27874.0:83298 9.7 e896/870 9:eb566c2a:::benchmark_data_smithi173_150363_object83297:head v 896'5203, mlcod=896'5199)", 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:40:10.702482+0000", 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.851507993999999, 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.077382095999999997, 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:40:41.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.702482+0000", 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.702482+0000", 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.702483+0000", 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3400000000000001e-06 2024-09-17T16:40:41.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.702610+0000", 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000126477 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.777179+0000", 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.074569892999999998 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:40:41.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.777187+0000", 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3180000000000001e-06 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.777238+0000", 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1112000000000001e-05 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.777252+0000", 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4389000000000001e-05 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.779823+0000", 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0025710170000000001 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:10.779864+0000", 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0550000000000003e-05 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:40:41.665 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:40:41.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:41.748 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:41.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:41.804 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:40:41.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/609196417' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:40:41.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:41.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:41.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:41.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:42.007 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:42.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:42.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:42.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/609196417' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:40:42.108 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:42.136 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:42.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:42.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/609196417' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:40:42.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:42.261 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:42.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:42.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:42.388 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:42.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:42.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:42.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:42.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:42.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:42.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:42.640 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:42.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:42.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:42.768 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:42.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:42.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:42.895 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:42.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:42 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/609196417' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:40:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:42 smithi017 ceph-mon[27571]: osdmap e908: 12 total, 12 up, 11 in 2024-09-17T16:40:42.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:42 smithi017 ceph-mon[27571]: pgmap v2004: 27 pgs: 6 active+clean+scrubbing, 21 active+clean; 6.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 131 B/s rd, 24 MiB/s wr, 925 op/s 2024-09-17T16:40:42.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:43.021 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:43.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:43.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:42 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:43.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/609196417' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:40:43.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:42 smithi071 ceph-mon[30002]: osdmap e908: 12 total, 12 up, 11 in 2024-09-17T16:40:43.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:42 smithi071 ceph-mon[30002]: pgmap v2004: 27 pgs: 6 active+clean+scrubbing, 21 active+clean; 6.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 131 B/s rd, 24 MiB/s wr, 925 op/s 2024-09-17T16:40:43.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:43.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:42 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/609196417' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:40:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:42 smithi173 ceph-mon[30530]: osdmap e908: 12 total, 12 up, 11 in 2024-09-17T16:40:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:42 smithi173 ceph-mon[30530]: pgmap v2004: 27 pgs: 6 active+clean+scrubbing, 21 active+clean; 6.3 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 131 B/s rd, 24 MiB/s wr, 925 op/s 2024-09-17T16:40:43.146 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:43.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:43.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:43.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:43.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:43.370 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:43.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:43.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:43.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:43.501 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:43.527 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:43.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:43.589 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:40:43.589 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:40:43.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:43.656 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:43.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:43.758 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:43.784 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:43.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:43.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:43.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:43.910 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:43.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:43.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:43 smithi017 ceph-mon[27571]: from='client.18738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:40:44.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:44.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:44.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:44.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:43 smithi071 ceph-mon[30002]: from='client.18738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:40:44.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:44.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:43 smithi173 ceph-mon[30530]: from='client.18738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:40:44.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:44.162 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:44.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:44.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:44.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:44.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:44.395 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:44.421 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:44.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:44.450 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":2004,"stamp":"2024-09-17T16:40:42.155686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6731214995,"num_objects":102705,"num_object_clones":0,"num_object_copies":308115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102705,"num_whiteouts":0,"num_read":71,"num_read_kb":60,"num_write":111215,"num_write_kb":6843684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2563,"num_bytes_recovered":169035535,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112115,"ondisk_log_size":112115,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":26383600,"kb_used_data":19802168,"kb_used_omap":83,"kb_used_meta":6581164,"kb_avail":1004813072,"statfs":{"total":1055945392128,"available":1028928585728,"internally_reserved":0,"allocated":20277420032,"data_stored":20269414474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86009,"internal_metadata":6739111943},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-79888360,"num_objects":-1218,"num_object_clones":0,"num_object_copies":-3654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7182,"num_write_kb":190785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.764174"},"pg_stats":[{"pgid":"1.7","version":"21'92","reported_seq":91,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:35.004654+0000","last_change":"2024-09-17T16:40:34.968723+0000","last_active":"2024-09-17T16:40:35.004654+0000","last_peered":"2024-09-17T16:40:35.004654+0000","last_clean":"2024-09-17T16:40:35.004654+0000","last_became_active":"2024-09-17T16:39:33.798225+0000","last_became_peered":"2024-09-17T16:39:33.798225+0000","last_unstale":"2024-09-17T16:40:35.004654+0000","last_undegraded":"2024-09-17T16:40:35.004654+0000","last_fullsized":"2024-09-17T16:40:35.004654+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:41:04.218856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"907'6978","reported_seq":7153,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:39.573706+0000","last_change":"2024-09-17T16:40:20.382606+0000","last_active":"2024-09-17T16:40:39.573706+0000","last_peered":"2024-09-17T16:40:39.573706+0000","last_clean":"2024-09-17T16:40:39.573706+0000","last_became_active":"2024-09-17T16:38:54.765274+0000","last_became_peered":"2024-09-17T16:38:54.765274+0000","last_unstale":"2024-09-17T16:40:39.573706+0000","last_undegraded":"2024-09-17T16:40:39.573706+0000","last_fullsized":"2024-09-17T16:40:39.573706+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'5834","last_scrub_stamp":"2024-09-17T16:40:20.382459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:20.382459+0000","objects_scrubbed":5775,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:40.548079+0000","scrub_duration":1522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420085760,"num_objects":6410,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6978,"num_write_kb":428416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":66,"reported_epoch":907,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T16:40:35.628848+0000","last_change":"2024-09-17T16:40:35.628749+0000","last_active":"2024-09-17T16:40:35.628848+0000","last_peered":"2024-09-17T16:40:35.628848+0000","last_clean":"2024-09-17T16:40:35.628848+0000","last_became_active":"2024-09-17T16:39:33.791607+0000","last_became_peered":"2024-09-17T16:39:33.791607+0000","last_unstale":"2024-09-17T16:40:35.628848+0000","last_undegraded":"2024-09-17T16:40:35.628848+0000","last_fullsized":"2024-09-17T16:40:35.628848+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"907'6780","reported_seq":6888,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:37.314995+0000","last_change":"2024-09-17T16:38:54.765568+0000","last_active":"2024-09-17T16:40:37.314995+0000","last_peered":"2024-09-17T16:40:37.314995+0000","last_clean":"2024-09-17T16:40:37.314995+0000","last_became_active":"2024-09-17T16:38:54.765252+0000","last_became_peered":"2024-09-17T16:38:54.765252+0000","last_unstale":"2024-09-17T16:40:37.314995+0000","last_undegraded":"2024-09-17T16:40:37.314995+0000","last_fullsized":"2024-09-17T16:40:37.314995+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":6780,"log_dups_size":0,"ondisk_log_size":6780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:59.663277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424411136,"num_objects":6476,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6780,"num_write_kb":424192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":66,"reported_epoch":907,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T16:40:36.616162+0000","last_change":"2024-09-17T16:40:36.616045+0000","last_active":"2024-09-17T16:40:36.616162+0000","last_peered":"2024-09-17T16:40:36.616162+0000","last_clean":"2024-09-17T16:40:36.616162+0000","last_became_active":"2024-09-17T16:39:33.800521+0000","last_became_peered":"2024-09-17T16:39:33.800521+0000","last_unstale":"2024-09-17T16:40:36.616162+0000","last_undegraded":"2024-09-17T16:40:36.616162+0000","last_fullsized":"2024-09-17T16:40:36.616162+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"907'6721","reported_seq":6891,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:38.181745+0000","last_change":"2024-09-17T16:40:15.312937+0000","last_active":"2024-09-17T16:40:38.181745+0000","last_peered":"2024-09-17T16:40:38.181745+0000","last_clean":"2024-09-17T16:40:38.181745+0000","last_became_active":"2024-09-17T16:38:54.759320+0000","last_became_peered":"2024-09-17T16:38:54.759320+0000","last_unstale":"2024-09-17T16:40:38.181745+0000","last_undegraded":"2024-09-17T16:40:38.181745+0000","last_fullsized":"2024-09-17T16:40:38.181745+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"897'5308","last_scrub_stamp":"2024-09-17T16:40:15.312706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:15.312706+0000","objects_scrubbed":5240,"log_size":6721,"log_dups_size":0,"ondisk_log_size":6721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:17.023547+0000","scrub_duration":1995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414056472,"num_objects":6319,"num_object_clones":0,"num_object_copies":18957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6721,"num_write_kb":417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":73,"reported_epoch":907,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T16:40:37.155569+0000","last_change":"2024-09-17T16:40:37.155461+0000","last_active":"2024-09-17T16:40:37.155569+0000","last_peered":"2024-09-17T16:40:37.155569+0000","last_clean":"2024-09-17T16:40:37.155569+0000","last_became_active":"2024-09-17T16:39:33.795955+0000","last_became_peered":"2024-09-17T16:39:33.795955+0000","last_unstale":"2024-09-17T16:40:37.155569+0000","last_undegraded":"2024-09-17T16:40:37.155569+0000","last_fullsized":"2024-09-17T16:40:37.155569+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":375765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.c","version":"907'7099","reported_seq":8346,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:40.060051+0000","last_change":"2024-09-17T16:39:15.770878+0000","last_active":"2024-09-17T16:40:40.060051+0000","last_peered":"2024-09-17T16:40:40.060051+0000","last_clean":"2024-09-17T16:40:40.060051+0000","last_became_active":"2024-09-17T16:39:15.769803+0000","last_became_peered":"2024-09-17T16:39:15.769803+0000","last_unstale":"2024-09-17T16:40:40.060051+0000","last_undegraded":"2024-09-17T16:40:40.060051+0000","last_fullsized":"2024-09-17T16:40:40.060051+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":7099,"log_dups_size":0,"ondisk_log_size":7099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:04.382851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415301632,"num_objects":6337,"num_object_clones":0,"num_object_copies":19011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7099,"num_write_kb":429952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":564,"num_bytes_recovered":36962304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":66,"reported_epoch":907,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T16:40:35.265463+0000","last_change":"2024-09-17T16:40:35.265323+0000","last_active":"2024-09-17T16:40:35.265463+0000","last_peered":"2024-09-17T16:40:35.265463+0000","last_clean":"2024-09-17T16:40:35.265463+0000","last_became_active":"2024-09-17T16:39:33.824237+0000","last_became_peered":"2024-09-17T16:39:33.824237+0000","last_unstale":"2024-09-17T16:40:35.265463+0000","last_undegraded":"2024-09-17T16:40:35.265463+0000","last_fullsized":"2024-09-17T16:40:35.265463+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"907'6970","reported_seq":7087,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:39.410036+0000","last_change":"2024-09-17T16:40:27.389553+0000","last_active":"2024-09-17T16:40:39.410036+0000","last_peered":"2024-09-17T16:40:39.410036+0000","last_clean":"2024-09-17T16:40:39.410036+0000","last_became_active":"2024-09-17T16:38:54.766429+0000","last_became_peered":"2024-09-17T16:38:54.766429+0000","last_unstale":"2024-09-17T16:40:39.410036+0000","last_undegraded":"2024-09-17T16:40:39.410036+0000","last_fullsized":"2024-09-17T16:40:39.410036+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":599,"log_size":6970,"log_dups_size":0,"ondisk_log_size":6970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:57.384595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415105024,"num_objects":6334,"num_object_clones":0,"num_object_copies":19002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6970,"num_write_kb":425728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":66,"reported_epoch":907,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T16:40:36.139748+0000","last_change":"2024-09-17T16:40:36.139602+0000","last_active":"2024-09-17T16:40:36.139748+0000","last_peered":"2024-09-17T16:40:36.139748+0000","last_clean":"2024-09-17T16:40:36.139748+0000","last_became_active":"2024-09-17T16:39:33.790219+0000","last_became_peered":"2024-09-17T16:39:33.790219+0000","last_unstale":"2024-09-17T16:40:36.139748+0000","last_undegraded":"2024-09-17T16:40:36.139748+0000","last_fullsized":"2024-09-17T16:40:36.139748+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":12,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"907'7121","reported_seq":7294,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:39.642509+0000","last_change":"2024-09-17T16:40:17.005820+0000","last_active":"2024-09-17T16:40:39.642509+0000","last_peered":"2024-09-17T16:40:39.642509+0000","last_clean":"2024-09-17T16:40:39.642509+0000","last_became_active":"2024-09-17T16:38:54.760031+0000","last_became_peered":"2024-09-17T16:38:54.760031+0000","last_unstale":"2024-09-17T16:40:39.642509+0000","last_undegraded":"2024-09-17T16:40:39.642509+0000","last_fullsized":"2024-09-17T16:40:39.642509+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"898'5749","last_scrub_stamp":"2024-09-17T16:40:17.005704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:17.005704+0000","objects_scrubbed":5713,"log_size":7121,"log_dups_size":0,"ondisk_log_size":7121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:34.891870+0000","scrub_duration":1192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425132032,"num_objects":6487,"num_object_clones":0,"num_object_copies":19461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7121,"num_write_kb":435456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":62,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:35.003730+0000","last_change":"2024-09-17T16:40:10.288467+0000","last_active":"2024-09-17T16:40:35.003730+0000","last_peered":"2024-09-17T16:40:35.003730+0000","last_clean":"2024-09-17T16:40:35.003730+0000","last_became_active":"2024-09-17T16:39:33.801025+0000","last_became_peered":"2024-09-17T16:39:33.801025+0000","last_unstale":"2024-09-17T16:40:35.003730+0000","last_undegraded":"2024-09-17T16:40:35.003730+0000","last_fullsized":"2024-09-17T16:40:35.003730+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:40:36.704254+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":12,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":96169,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.9","version":"907'6809","reported_seq":6917,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:37.321223+0000","last_change":"2024-09-17T16:38:54.764119+0000","last_active":"2024-09-17T16:40:37.321223+0000","last_peered":"2024-09-17T16:40:37.321223+0000","last_clean":"2024-09-17T16:40:37.321223+0000","last_became_active":"2024-09-17T16:38:54.763749+0000","last_became_peered":"2024-09-17T16:38:54.763749+0000","last_unstale":"2024-09-17T16:40:37.321223+0000","last_undegraded":"2024-09-17T16:40:37.321223+0000","last_fullsized":"2024-09-17T16:40:37.321223+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":6809,"log_dups_size":0,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:07.751179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426573824,"num_objects":6509,"num_object_clones":0,"num_object_copies":19527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":426176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.7","version":"907'6857","reported_seq":8001,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:37.307012+0000","last_change":"2024-09-17T16:40:17.290553+0000","last_active":"2024-09-17T16:40:37.307012+0000","last_peered":"2024-09-17T16:40:37.307012+0000","last_clean":"2024-09-17T16:40:37.307012+0000","last_became_active":"2024-09-17T16:39:13.777102+0000","last_became_peered":"2024-09-17T16:39:13.777102+0000","last_unstale":"2024-09-17T16:40:37.307012+0000","last_undegraded":"2024-09-17T16:40:37.307012+0000","last_fullsized":"2024-09-17T16:40:37.307012+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":0,"last_scrub":"898'5654","last_scrub_stamp":"2024-09-17T16:40:17.290378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:17.290378+0000","objects_scrubbed":5595,"log_size":6857,"log_dups_size":0,"ondisk_log_size":6857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:28.161431+0000","scrub_duration":1589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429588480,"num_objects":6555,"num_object_clones":0,"num_object_copies":19665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6857,"num_write_kb":429184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":31391744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.6","version":"907'6868","reported_seq":6975,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:39.628786+0000","last_change":"2024-09-17T16:38:54.766597+0000","last_active":"2024-09-17T16:40:39.628786+0000","last_peered":"2024-09-17T16:40:39.628786+0000","last_clean":"2024-09-17T16:40:39.628786+0000","last_became_active":"2024-09-17T16:38:54.765661+0000","last_became_peered":"2024-09-17T16:38:54.765661+0000","last_unstale":"2024-09-17T16:40:39.628786+0000","last_undegraded":"2024-09-17T16:40:39.628786+0000","last_fullsized":"2024-09-17T16:40:39.628786+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":6868,"log_dups_size":0,"ondisk_log_size":6868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:09.748299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412090368,"num_objects":6288,"num_object_clones":0,"num_object_copies":18864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6868,"num_write_kb":420992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"907'7103","reported_seq":7211,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:39.650596+0000","last_change":"2024-09-17T16:38:54.763911+0000","last_active":"2024-09-17T16:40:39.650596+0000","last_peered":"2024-09-17T16:40:39.650596+0000","last_clean":"2024-09-17T16:40:39.650596+0000","last_became_active":"2024-09-17T16:38:54.763476+0000","last_became_peered":"2024-09-17T16:38:54.763476+0000","last_unstale":"2024-09-17T16:40:39.650596+0000","last_undegraded":"2024-09-17T16:40:39.650596+0000","last_fullsized":"2024-09-17T16:40:39.650596+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":7103,"log_dups_size":0,"ondisk_log_size":7103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:01.637495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422903808,"num_objects":6453,"num_object_clones":0,"num_object_copies":19359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7103,"num_write_kb":433792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"907'6965","reported_seq":7137,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:37.723655+0000","last_change":"2024-09-17T16:40:13.527064+0000","last_active":"2024-09-17T16:40:37.723655+0000","last_peered":"2024-09-17T16:40:37.723655+0000","last_clean":"2024-09-17T16:40:37.723655+0000","last_became_active":"2024-09-17T16:38:54.762670+0000","last_became_peered":"2024-09-17T16:38:54.762670+0000","last_unstale":"2024-09-17T16:40:37.723655+0000","last_undegraded":"2024-09-17T16:40:37.723655+0000","last_fullsized":"2024-09-17T16:40:37.723655+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"897'5512","last_scrub_stamp":"2024-09-17T16:40:13.526948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:13.526948+0000","objects_scrubbed":5465,"log_size":6965,"log_dups_size":0,"ondisk_log_size":6965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:39.881204+0000","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434438144,"num_objects":6629,"num_object_clones":0,"num_object_copies":19887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6965,"num_write_kb":435008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"907'6961","reported_seq":8109,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:39.643821+0000","last_change":"2024-09-17T16:40:18.292852+0000","last_active":"2024-09-17T16:40:39.643821+0000","last_peered":"2024-09-17T16:40:39.643821+0000","last_clean":"2024-09-17T16:40:39.643821+0000","last_became_active":"2024-09-17T16:39:13.775396+0000","last_became_peered":"2024-09-17T16:39:13.775396+0000","last_unstale":"2024-09-17T16:40:39.643821+0000","last_undegraded":"2024-09-17T16:40:39.643821+0000","last_fullsized":"2024-09-17T16:40:39.643821+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":0,"last_scrub":"898'5717","last_scrub_stamp":"2024-09-17T16:40:18.292675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:18.292675+0000","objects_scrubbed":5668,"log_size":6961,"log_dups_size":0,"ondisk_log_size":6961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:19.518520+0000","scrub_duration":1468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418840576,"num_objects":6391,"num_object_clones":0,"num_object_copies":19173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":427264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":31522816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":66,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:35.007179+0000","last_change":"2024-09-17T16:40:34.696461+0000","last_active":"2024-09-17T16:40:35.007179+0000","last_peered":"2024-09-17T16:40:35.007179+0000","last_clean":"2024-09-17T16:40:35.007179+0000","last_became_active":"2024-09-17T16:39:33.792174+0000","last_became_peered":"2024-09-17T16:39:33.792174+0000","last_unstale":"2024-09-17T16:40:35.007179+0000","last_undegraded":"2024-09-17T16:40:35.007179+0000","last_fullsized":"2024-09-17T16:40:35.007179+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:41:04.213513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"907'6805","reported_seq":7935,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:37.734890+0000","last_change":"2024-09-17T16:40:27.370125+0000","last_active":"2024-09-17T16:40:37.734890+0000","last_peered":"2024-09-17T16:40:37.734890+0000","last_clean":"2024-09-17T16:40:37.734890+0000","last_became_active":"2024-09-17T16:39:13.776232+0000","last_became_peered":"2024-09-17T16:39:13.776232+0000","last_unstale":"2024-09-17T16:40:37.734890+0000","last_undegraded":"2024-09-17T16:40:37.734890+0000","last_fullsized":"2024-09-17T16:40:37.734890+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":871,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":99,"log_size":6805,"log_dups_size":0,"ondisk_log_size":6805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:57.363894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423428096,"num_objects":6461,"num_object_clones":0,"num_object_copies":19383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6805,"num_write_kb":424512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":503,"num_bytes_recovered":32964608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":67,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:34.996112+0000","last_change":"2024-09-17T16:40:34.996112+0000","last_active":"2024-09-17T16:40:34.996112+0000","last_peered":"2024-09-17T16:40:34.996112+0000","last_clean":"2024-09-17T16:40:34.996112+0000","last_became_active":"2024-09-17T16:39:33.798293+0000","last_became_peered":"2024-09-17T16:39:33.798293+0000","last_unstale":"2024-09-17T16:40:34.996112+0000","last_undegraded":"2024-09-17T16:40:34.996112+0000","last_fullsized":"2024-09-17T16:40:34.996112+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:41:04.522261+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":96168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"907'7249","reported_seq":8435,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:41.524476+0000","last_change":"2024-09-17T16:40:27.378441+0000","last_active":"2024-09-17T16:40:41.524476+0000","last_peered":"2024-09-17T16:40:41.524476+0000","last_clean":"2024-09-17T16:40:41.524476+0000","last_became_active":"2024-09-17T16:39:15.784630+0000","last_became_peered":"2024-09-17T16:39:15.784630+0000","last_unstale":"2024-09-17T16:40:41.524476+0000","last_undegraded":"2024-09-17T16:40:41.524476+0000","last_fullsized":"2024-09-17T16:40:41.524476+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":549,"log_size":7249,"log_dups_size":0,"ondisk_log_size":7249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:57.377129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417660928,"num_objects":6373,"num_object_clones":0,"num_object_copies":19119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7249,"num_write_kb":435904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":531,"num_bytes_recovered":34799616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":70,"reported_epoch":907,"state":"active+clean+scrubbing","last_fresh":"2024-09-17T16:40:34.933860+0000","last_change":"2024-09-17T16:40:34.933663+0000","last_active":"2024-09-17T16:40:34.933860+0000","last_peered":"2024-09-17T16:40:34.933860+0000","last_clean":"2024-09-17T16:40:34.933860+0000","last_became_active":"2024-09-17T16:39:33.796846+0000","last_became_peered":"2024-09-17T16:39:33.796846+0000","last_unstale":"2024-09-17T16:40:34.933860+0000","last_undegraded":"2024-09-17T16:40:34.933860+0000","last_fullsized":"2024-09-17T16:40:34.933860+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:33:59.364821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:33:59.364821+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":297417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"907'6727","reported_seq":6836,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:37.745785+0000","last_change":"2024-09-17T16:38:54.757271+0000","last_active":"2024-09-17T16:40:37.745785+0000","last_peered":"2024-09-17T16:40:37.745785+0000","last_clean":"2024-09-17T16:40:37.745785+0000","last_became_active":"2024-09-17T16:38:54.756978+0000","last_became_peered":"2024-09-17T16:38:54.756978+0000","last_unstale":"2024-09-17T16:40:37.745785+0000","last_undegraded":"2024-09-17T16:40:37.745785+0000","last_fullsized":"2024-09-17T16:40:37.745785+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":0,"log_size":6727,"log_dups_size":0,"ondisk_log_size":6727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:07.228159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419889152,"num_objects":6407,"num_object_clones":0,"num_object_copies":19221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6727,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"907'7090","reported_seq":7261,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:41.526627+0000","last_change":"2024-09-17T16:40:13.810074+0000","last_active":"2024-09-17T16:40:41.526627+0000","last_peered":"2024-09-17T16:40:41.526627+0000","last_clean":"2024-09-17T16:40:41.526627+0000","last_became_active":"2024-09-17T16:38:54.767749+0000","last_became_peered":"2024-09-17T16:38:54.767749+0000","last_unstale":"2024-09-17T16:40:41.526627+0000","last_undegraded":"2024-09-17T16:40:41.526627+0000","last_fullsized":"2024-09-17T16:40:41.526627+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"897'5393","last_scrub_stamp":"2024-09-17T16:40:13.809890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:13.809890+0000","objects_scrubbed":5346,"log_size":7090,"log_dups_size":0,"ondisk_log_size":7090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:41:39.686185+0000","scrub_duration":1637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411172864,"num_objects":6274,"num_object_clones":0,"num_object_copies":18822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":427648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":605,"reported_epoch":907,"state":"active+clean","last_fresh":"2024-09-17T16:40:35.004280+0000","last_change":"2024-09-17T16:39:32.100237+0000","last_active":"2024-09-17T16:40:35.004280+0000","last_peered":"2024-09-17T16:40:35.004280+0000","last_clean":"2024-09-17T16:40:35.004280+0000","last_became_active":"2024-09-17T16:39:31.793364+0000","last_became_peered":"2024-09-17T16:39:31.793364+0000","last_unstale":"2024-09-17T16:40:35.004280+0000","last_undegraded":"2024-09-17T16:40:35.004280+0000","last_fullsized":"2024-09-17T16:40:35.004280+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:39:32.100164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:39:32.100164+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:40:38.274225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6730678296,"num_objects":102703,"num_object_clones":0,"num_object_copies":308109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102703,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111103,"num_write_kb":6841729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2558,"num_bytes_recovered":167641088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20144664576,"data_stored":20144652360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111103,"ondisk_log_size":111103,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":536699,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":112,"num_write_kb":1955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1394447,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285943,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2676876,"kb_used_data":2045948,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":91068276,"statfs":{"total":95995035648,"available":93253914624,"internally_reserved":0,"allocated":2095050752,"data_stored":2094325046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":646052292},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131185,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2809172,"kb_used_data":2519484,"kb_used_omap":15,"kb_used_meta":289648,"kb_avail":90935980,"statfs":{"total":95995035648,"available":93118443520,"internally_reserved":0,"allocated":2579951616,"data_stored":2579225918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":296600124},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743808,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3309644,"kb_used_data":2421884,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":90435508,"statfs":{"total":95995035648,"available":92605960192,"internally_reserved":0,"allocated":2480009216,"data_stored":2479283518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":909048256},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005445,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851732,"kb_used_data":1238852,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":91893420,"statfs":{"total":95995035648,"available":94098862080,"internally_reserved":0,"allocated":1268584448,"data_stored":1267855198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2853264,"kb_used_data":2092608,"kb_used_omap":41,"kb_used_meta":760598,"kb_avail":90891888,"statfs":{"total":95995035648,"available":93073293312,"internally_reserved":0,"allocated":2142830592,"data_stored":2142100822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":778852659},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496019,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1385868,"kb_used_data":813244,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92359284,"statfs":{"total":95995035648,"available":94575906816,"internally_reserved":0,"allocated":832761856,"data_stored":832036158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":586348998},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967663,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3877848,"kb_used_data":3696840,"kb_used_omap":14,"kb_used_meta":180977,"kb_avail":89867304,"statfs":{"total":95995035648,"available":92024119296,"internally_reserved":0,"allocated":3785564160,"data_stored":3784830838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":185321077},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986586,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1455116,"kb_used_data":828348,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92290036,"statfs":{"total":95995035648,"available":94504996864,"internally_reserved":0,"allocated":848228352,"data_stored":847502654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150108,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411747,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1041364,"kb_used_data":406276,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92703788,"statfs":{"total":95995035648,"available":94928678912,"internally_reserved":0,"allocated":416026624,"data_stored":415297374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575272,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3926608,"kb_used_data":3335936,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":89818544,"statfs":{"total":95995035648,"available":91974189056,"internally_reserved":0,"allocated":3415998464,"data_stored":3415268694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738797,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1196108,"kb_used_data":402748,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92549044,"statfs":{"total":95995035648,"available":94770221056,"internally_reserved":0,"allocated":412413952,"data_stored":411688254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":410320896,"data_stored":410320896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3398307840,"data_stored":3398303768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":411631616,"data_stored":411631616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":839581696,"data_stored":839581696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3757051904,"data_stored":3757047832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":827260928,"data_stored":827260928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2120945664,"data_stored":2120941592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1262354432,"data_stored":1262354432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2468282368,"data_stored":2468282368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2564554752,"data_stored":2564554752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2079653888,"data_stored":2079653888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:40:44.452 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-17T16:40:44.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_7 pg_num 26 2024-09-17T16:40:44.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:44.554 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:44.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:44.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:44.656 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:44.682 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:44.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:44.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:40:44.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:40:44.745 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:40:44.745 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:40:44.760 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:40:44.760 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:40:44.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:44.788 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:40:44.788 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:40:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:44.820 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:44.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:44.823 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:40:44.824 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:40:44.850 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:40:44.851 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:40:44.877 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:40:44.877 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:40:44.905 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:40:44.906 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:40:44.921 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:44.933 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:40:44.934 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:40:44.949 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:44.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:44.961 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:40:44.961 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:40:44.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:44 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:40:44.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:44 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:40:44.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:44 smithi017 ceph-mon[27571]: osdmap e909: 12 total, 12 up, 11 in 2024-09-17T16:40:44.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:44 smithi017 ceph-mon[27571]: pgmap v2006: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 6.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 8.1 MiB/s wr, 935 op/s 2024-09-17T16:40:44.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3016846199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:44.997 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:40:44.997 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:40:45.024 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:40:45.025 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:40:45.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:45.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:45.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:45.090 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:40:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:44 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:40:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:44 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:40:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:44 smithi173 ceph-mon[30530]: osdmap e909: 12 total, 12 up, 11 in 2024-09-17T16:40:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:44 smithi173 ceph-mon[30530]: pgmap v2006: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 6.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 8.1 MiB/s wr, 935 op/s 2024-09-17T16:40:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3016846199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:45.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:45.211 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:45.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:45.313 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:44 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:40:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:44 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:40:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:44 smithi071 ceph-mon[30002]: osdmap e909: 12 total, 12 up, 11 in 2024-09-17T16:40:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:44 smithi071 ceph-mon[30002]: pgmap v2006: 27 pgs: 3 active+clean+scrubbing, 24 active+clean; 6.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 8.1 MiB/s wr, 935 op/s 2024-09-17T16:40:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3016846199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:45.341 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:45.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:45.443 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:45.469 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:45.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:45.476 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:45.506 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:40:45.525 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:45.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:45.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:45.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:45.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:45.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:45.726 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:45.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:45.828 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:45.854 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:45.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:45.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:45.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:45.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:46.094 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:46.119 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:46.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:46.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:46.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:46.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:46.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:46.381 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:46.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:46.477 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:46.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:46.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:46.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:46.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:46.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:46.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:46.743 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:46.772 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:46.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:46.874 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:46.898 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:46.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:47.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:47.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:47.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:47.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:47.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:47.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:47.217 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 9 pg_num to 26 2024-09-17T16:40:47.265 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:47.296 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:47.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:47.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:47.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:47.422 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:47.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:47 smithi017 ceph-mon[27571]: pgmap v2007: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 472 KiB/s wr, 944 op/s 2024-09-17T16:40:47.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3981293421' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:40:47.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:47.555 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:47.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:47 smithi071 ceph-mon[30002]: pgmap v2007: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 472 KiB/s wr, 944 op/s 2024-09-17T16:40:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3981293421' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:40:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:47 smithi173 ceph-mon[30530]: pgmap v2007: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 127 B/s rd, 472 KiB/s wr, 944 op/s 2024-09-17T16:40:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3981293421' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-17T16:40:47.657 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:47.680 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:47.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:47.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:47.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:47.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:47.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:47.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:47.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:47.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:47.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:48.046 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:48.071 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:48.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:48.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:48.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:48.202 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:48.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:48.225 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:40:48.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:40:48.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:48.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:48.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:48.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:48.453 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:48.475 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:48.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3981293421' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:40:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:48 smithi017 ceph-mon[27571]: osdmap e910: 12 total, 12 up, 11 in 2024-09-17T16:40:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1284179569' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:48 smithi017 ceph-mon[27571]: 9.2 scrub starts 2024-09-17T16:40:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:48 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:40:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3981293421' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:40:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:48 smithi071 ceph-mon[30002]: osdmap e910: 12 total, 12 up, 11 in 2024-09-17T16:40:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1284179569' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:48 smithi071 ceph-mon[30002]: 9.2 scrub starts 2024-09-17T16:40:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:48 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:40:48.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:48.603 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:48.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3981293421' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-09-17T16:40:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:48 smithi173 ceph-mon[30530]: osdmap e910: 12 total, 12 up, 11 in 2024-09-17T16:40:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1284179569' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:40:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:48 smithi173 ceph-mon[30530]: 9.2 scrub starts 2024-09-17T16:40:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:48 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-17T16:40:48.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:48.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:48.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:48.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:48.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:48.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:48.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:48.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:48.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:48.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:48.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:48.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:49.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:49.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:49.102 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:49.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:49.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:49.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:49.230 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:49.331 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:49.359 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:49.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:49.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:49.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:49 smithi017 ceph-mon[27571]: pgmap v2009: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.8 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-17T16:40:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:49 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:40:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:49 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:40:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1284179569' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:49 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:40:49.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:49 smithi017 ceph-mon[27571]: osdmap e911: 12 total, 12 up, 11 in 2024-09-17T16:40:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:49.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:49 smithi071 ceph-mon[30002]: pgmap v2009: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.8 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-17T16:40:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:49 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:40:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:49 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:40:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1284179569' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:49 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:40:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:49 smithi071 ceph-mon[30002]: osdmap e911: 12 total, 12 up, 11 in 2024-09-17T16:40:49.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:49.617 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:49.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:49 smithi173 ceph-mon[30530]: pgmap v2009: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 5.8 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-17T16:40:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:49 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:40:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:49 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:40:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1284179569' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:40:49.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:49 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-17T16:40:49.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:49 smithi173 ceph-mon[30530]: osdmap e911: 12 total, 12 up, 11 in 2024-09-17T16:40:49.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:49.748 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:49.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:49.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:49.873 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:49.874 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:49.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:50.001 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:50.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:50.103 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:50.116 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:50.133 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:50.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:50.143 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:40:50.143 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:40:50.234 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:50.268 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:50.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:50.369 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:50.399 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:50.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:50.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:50 smithi017 ceph-mon[27571]: 9.c scrub starts 2024-09-17T16:40:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:50 smithi017 ceph-mon[27571]: 9.c scrub ok 2024-09-17T16:40:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:50 smithi017 ceph-mon[27571]: osdmap e912: 12 total, 12 up, 11 in 2024-09-17T16:40:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:50 smithi017 ceph-mon[27571]: 9.4 scrub starts 2024-09-17T16:40:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:50 smithi017 ceph-mon[27571]: 9.7 scrub starts 2024-09-17T16:40:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:50 smithi017 ceph-mon[27571]: osdmap e913: 12 total, 12 up, 11 in 2024-09-17T16:40:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:40:50.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:50.529 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:50.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:50.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:50 smithi071 ceph-mon[30002]: 9.c scrub starts 2024-09-17T16:40:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:50 smithi071 ceph-mon[30002]: 9.c scrub ok 2024-09-17T16:40:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:50 smithi071 ceph-mon[30002]: osdmap e912: 12 total, 12 up, 11 in 2024-09-17T16:40:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:50 smithi071 ceph-mon[30002]: 9.4 scrub starts 2024-09-17T16:40:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:50 smithi071 ceph-mon[30002]: 9.7 scrub starts 2024-09-17T16:40:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:50 smithi071 ceph-mon[30002]: osdmap e913: 12 total, 12 up, 11 in 2024-09-17T16:40:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:40:50.631 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:50 smithi173 ceph-mon[30530]: 9.c scrub starts 2024-09-17T16:40:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:50 smithi173 ceph-mon[30530]: 9.c scrub ok 2024-09-17T16:40:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:50 smithi173 ceph-mon[30530]: osdmap e912: 12 total, 12 up, 11 in 2024-09-17T16:40:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:50 smithi173 ceph-mon[30530]: 9.4 scrub starts 2024-09-17T16:40:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:50 smithi173 ceph-mon[30530]: 9.7 scrub starts 2024-09-17T16:40:50.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:50 smithi173 ceph-mon[30530]: osdmap e913: 12 total, 12 up, 11 in 2024-09-17T16:40:50.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-17T16:40:50.658 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:50.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:50.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:50.790 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:50.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:50.842 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:40:50.843 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:40:50.863 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:40:50.864 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:40:50.884 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:40:50.884 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:40:50.892 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:50.905 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:40:50.905 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:40:50.915 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:50.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:50.927 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:40:50.927 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:40:50.949 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:40:50.949 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:40:50.970 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:40:50.971 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:40:50.992 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:40:50.992 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:40:51.013 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:40:51.014 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:40:51.017 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:51.035 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:40:51.035 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:40:51.039 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:51.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:51.057 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:40:51.057 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:40:51.078 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:40:51.079 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:40:51.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:51.164 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:51.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:51.266 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:51.288 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:51.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:51.390 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:51.412 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:51.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: 9.4 scrub ok 2024-09-17T16:40:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: 9.6 scrub starts 2024-09-17T16:40:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: pgmap v2013: 37 pgs: 7 unknown, 1 active+clean+scrubbing, 29 active+clean; 5.7 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s; 0 B/s, 0 objects/s recovering 2024-09-17T16:40:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: 9.1 scrub starts 2024-09-17T16:40:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: 9.7 scrub ok 2024-09-17T16:40:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: 9.14 scrub starts 2024-09-17T16:40:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: 9.17 scrub starts 2024-09-17T16:40:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: 9.6 scrub ok 2024-09-17T16:40:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:40:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: osdmap e914: 12 total, 12 up, 11 in 2024-09-17T16:40:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2401972198' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:51.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:51 smithi017 ceph-mon[27571]: 9.1 scrub ok 2024-09-17T16:40:51.513 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:51.536 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:51.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: 9.4 scrub ok 2024-09-17T16:40:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: 9.6 scrub starts 2024-09-17T16:40:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: pgmap v2013: 37 pgs: 7 unknown, 1 active+clean+scrubbing, 29 active+clean; 5.7 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s; 0 B/s, 0 objects/s recovering 2024-09-17T16:40:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: 9.1 scrub starts 2024-09-17T16:40:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: 9.7 scrub ok 2024-09-17T16:40:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: 9.14 scrub starts 2024-09-17T16:40:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: 9.17 scrub starts 2024-09-17T16:40:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: 9.6 scrub ok 2024-09-17T16:40:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:40:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: osdmap e914: 12 total, 12 up, 11 in 2024-09-17T16:40:51.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2401972198' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:51.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:51 smithi071 ceph-mon[30002]: 9.1 scrub ok 2024-09-17T16:40:51.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:40:51.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:40:51.638 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: 9.4 scrub ok 2024-09-17T16:40:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: 9.6 scrub starts 2024-09-17T16:40:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: pgmap v2013: 37 pgs: 7 unknown, 1 active+clean+scrubbing, 29 active+clean; 5.7 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.16k op/s; 0 B/s, 0 objects/s recovering 2024-09-17T16:40:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: 9.1 scrub starts 2024-09-17T16:40:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: 9.7 scrub ok 2024-09-17T16:40:51.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: 9.14 scrub starts 2024-09-17T16:40:51.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: 9.17 scrub starts 2024-09-17T16:40:51.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: 9.6 scrub ok 2024-09-17T16:40:51.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-17T16:40:51.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: osdmap e914: 12 total, 12 up, 11 in 2024-09-17T16:40:51.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2401972198' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:40:51.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:51 smithi173 ceph-mon[30530]: 9.1 scrub ok 2024-09-17T16:40:51.685 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:51.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:51.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:51.787 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:51.818 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:51.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:51.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:51.937 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:51.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:51.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:51.944 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:40:52.045 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:52.067 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:52.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:52.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:52.169 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:52.190 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:52.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:52.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:52.292 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:52.314 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:52.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:52.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:52 smithi173 ceph-mon[30530]: 9.5 scrub starts 2024-09-17T16:40:52.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:52 smithi173 ceph-mon[30530]: 9.8 scrub starts 2024-09-17T16:40:52.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:52 smithi173 ceph-mon[30530]: 9.8 scrub ok 2024-09-17T16:40:52.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:52 smithi173 ceph-mon[30530]: 9.2 scrub starts 2024-09-17T16:40:52.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:52 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:40:52.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:52 smithi173 ceph-mon[30530]: 9.9 scrub starts 2024-09-17T16:40:52.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:52 smithi173 ceph-mon[30530]: osdmap e915: 12 total, 12 up, 11 in 2024-09-17T16:40:52.416 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:52.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:52 smithi017 ceph-mon[27571]: 9.5 scrub starts 2024-09-17T16:40:52.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:52 smithi017 ceph-mon[27571]: 9.8 scrub starts 2024-09-17T16:40:52.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:52 smithi017 ceph-mon[27571]: 9.8 scrub ok 2024-09-17T16:40:52.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:52 smithi017 ceph-mon[27571]: 9.2 scrub starts 2024-09-17T16:40:52.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:52 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:40:52.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:52 smithi017 ceph-mon[27571]: 9.9 scrub starts 2024-09-17T16:40:52.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:52 smithi017 ceph-mon[27571]: osdmap e915: 12 total, 12 up, 11 in 2024-09-17T16:40:52.441 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:52.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:52.543 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:52.618 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:52 smithi071 ceph-mon[30002]: 9.5 scrub starts 2024-09-17T16:40:52.618 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:52 smithi071 ceph-mon[30002]: 9.8 scrub starts 2024-09-17T16:40:52.618 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:52 smithi071 ceph-mon[30002]: 9.8 scrub ok 2024-09-17T16:40:52.618 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:52 smithi071 ceph-mon[30002]: 9.2 scrub starts 2024-09-17T16:40:52.618 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:52 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:40:52.619 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:52 smithi071 ceph-mon[30002]: 9.9 scrub starts 2024-09-17T16:40:52.619 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:52 smithi071 ceph-mon[30002]: osdmap e915: 12 total, 12 up, 11 in 2024-09-17T16:40:52.640 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:52.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:52.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:52.742 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:52.763 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:52.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:52.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:52.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:52.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:52.927 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:53.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:53.088 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:53.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:53.189 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:53.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:53.330 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:53.367 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:53.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:53.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:40:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:53 smithi017 ceph-mon[27571]: 9.5 scrub ok 2024-09-17T16:40:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:53 smithi017 ceph-mon[27571]: 9.3 scrub starts 2024-09-17T16:40:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:53 smithi017 ceph-mon[27571]: pgmap v2016: 37 pgs: 8 unknown, 1 active+clean+scrubbing, 28 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail 2024-09-17T16:40:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:53 smithi017 ceph-mon[27571]: 9.2 scrub ok 2024-09-17T16:40:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:53 smithi017 ceph-mon[27571]: 9.9 scrub ok 2024-09-17T16:40:53.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:53 smithi017 ceph-mon[27571]: 9.0 scrub starts 2024-09-17T16:40:53.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:53 smithi017 ceph-mon[27571]: 9.e scrub starts 2024-09-17T16:40:53.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:53 smithi017 ceph-mon[27571]: osdmap e916: 12 total, 12 up, 11 in 2024-09-17T16:40:53.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:53 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:53.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:53 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:53.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:53.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:40:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:53 smithi071 ceph-mon[30002]: 9.5 scrub ok 2024-09-17T16:40:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:53 smithi071 ceph-mon[30002]: 9.3 scrub starts 2024-09-17T16:40:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:53 smithi071 ceph-mon[30002]: pgmap v2016: 37 pgs: 8 unknown, 1 active+clean+scrubbing, 28 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail 2024-09-17T16:40:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:53 smithi071 ceph-mon[30002]: 9.2 scrub ok 2024-09-17T16:40:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:53 smithi071 ceph-mon[30002]: 9.9 scrub ok 2024-09-17T16:40:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:53 smithi071 ceph-mon[30002]: 9.0 scrub starts 2024-09-17T16:40:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:53 smithi071 ceph-mon[30002]: 9.e scrub starts 2024-09-17T16:40:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:53 smithi071 ceph-mon[30002]: osdmap e916: 12 total, 12 up, 11 in 2024-09-17T16:40:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:53 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:53.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:53 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:53.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:53.626 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:53.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:53.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:53.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:53 smithi173 ceph-mon[30530]: 9.5 scrub ok 2024-09-17T16:40:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:53 smithi173 ceph-mon[30530]: 9.3 scrub starts 2024-09-17T16:40:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:53 smithi173 ceph-mon[30530]: pgmap v2016: 37 pgs: 8 unknown, 1 active+clean+scrubbing, 28 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail 2024-09-17T16:40:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:53 smithi173 ceph-mon[30530]: 9.2 scrub ok 2024-09-17T16:40:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:53 smithi173 ceph-mon[30530]: 9.9 scrub ok 2024-09-17T16:40:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:53 smithi173 ceph-mon[30530]: 9.0 scrub starts 2024-09-17T16:40:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:53 smithi173 ceph-mon[30530]: 9.e scrub starts 2024-09-17T16:40:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:53 smithi173 ceph-mon[30530]: osdmap e916: 12 total, 12 up, 11 in 2024-09-17T16:40:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:53 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:53 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:53.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:53.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:53.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:53.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:53.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:53.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:53.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:53.879 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 7, 8, 9, 10, 11, 1, 5, 4] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 8, 9, 11, 7, 10, 5] 2024-09-17T16:40:53.879 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T16:40:53.976 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:54.004 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:54.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:54.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:54.128 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:54.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:54.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:54.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:54.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:54.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:54.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:54.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:54.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:54.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:54.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:54 smithi173 ceph-mon[30530]: 9.0 scrub ok 2024-09-17T16:40:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:54 smithi173 ceph-mon[30530]: 9.3 scrub ok 2024-09-17T16:40:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:54 smithi173 ceph-mon[30530]: 9.e scrub ok 2024-09-17T16:40:54.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:40:54.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:54.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:54 smithi017 ceph-mon[27571]: 9.0 scrub ok 2024-09-17T16:40:54.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:54 smithi017 ceph-mon[27571]: 9.3 scrub ok 2024-09-17T16:40:54.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:54 smithi017 ceph-mon[27571]: 9.e scrub ok 2024-09-17T16:40:54.513 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:54.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:54.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:54 smithi071 ceph-mon[30002]: 9.0 scrub ok 2024-09-17T16:40:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:54 smithi071 ceph-mon[30002]: 9.3 scrub ok 2024-09-17T16:40:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:54 smithi071 ceph-mon[30002]: 9.e scrub ok 2024-09-17T16:40:54.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:54.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:54.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:54.740 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:54.767 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:54.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:54.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:54.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:54.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:54.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:55.014 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:55.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:55.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:55.116 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:55.139 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:55.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:55.184 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:40:55.241 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:55.265 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:55.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:55.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:55.367 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:55.393 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:55.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:55.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:55 smithi173 ceph-mon[30530]: pgmap v2018: 37 pgs: 4 remapped+peering, 4 unknown, 2 active+clean+scrubbing, 27 active+clean; 4.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail 2024-09-17T16:40:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:55 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:40:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:55 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:40:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1191937685' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:40:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:55 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:40:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:55 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:55 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:40:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:55 smithi173 ceph-mon[30530]: osdmap e917: 12 total, 12 up, 11 in 2024-09-17T16:40:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:55 smithi017 ceph-mon[27571]: pgmap v2018: 37 pgs: 4 remapped+peering, 4 unknown, 2 active+clean+scrubbing, 27 active+clean; 4.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail 2024-09-17T16:40:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:55 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:40:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:55 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:40:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1191937685' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:40:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:55 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:40:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:55 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:55 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:40:55.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:55 smithi017 ceph-mon[27571]: osdmap e917: 12 total, 12 up, 11 in 2024-09-17T16:40:55.495 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:55.517 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:55.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:55 smithi071 ceph-mon[30002]: pgmap v2018: 37 pgs: 4 remapped+peering, 4 unknown, 2 active+clean+scrubbing, 27 active+clean; 4.9 GiB data, 23 GiB used, 960 GiB / 983 GiB avail 2024-09-17T16:40:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:55 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:40:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:55 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:40:55.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1191937685' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:40:55.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:55 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:40:55.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:55 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:40:55.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:55 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:40:55.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:55 smithi071 ceph-mon[30002]: osdmap e917: 12 total, 12 up, 11 in 2024-09-17T16:40:55.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:55.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:55.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:55.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:55.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:55.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:55.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:55.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:55.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:55.893 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:55.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:55.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:56.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:56.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:56.019 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:56.119 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:56.142 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:56.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:56.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:56.244 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:56.271 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:56.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:56.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:56.396 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:56.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:56.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:56.521 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:56.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:56.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:56.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:56.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:56.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:56.746 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:56.771 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:56.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:56.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:40:56.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:56.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:40:56.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:57.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:57.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:57.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:57.146 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:57.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:57.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:57.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:57.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:57.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:57.397 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:57.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:57.499 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:57.523 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:57.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:57 smithi071 ceph-mon[30002]: pgmap v2020: 37 pgs: 4 active+recovering+undersized+remapped, 4 remapped+peering, 1 unknown, 2 active+clean+scrubbing, 26 active+clean; 5.4 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 19371/266709 objects misplaced (7.263%) 2024-09-17T16:40:57.625 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:57.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:57 smithi173 ceph-mon[30530]: pgmap v2020: 37 pgs: 4 active+recovering+undersized+remapped, 4 remapped+peering, 1 unknown, 2 active+clean+scrubbing, 26 active+clean; 5.4 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 19371/266709 objects misplaced (7.263%) 2024-09-17T16:40:57.649 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:57.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:57.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:57 smithi017 ceph-mon[27571]: pgmap v2020: 37 pgs: 4 active+recovering+undersized+remapped, 4 remapped+peering, 1 unknown, 2 active+clean+scrubbing, 26 active+clean; 5.4 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 19371/266709 objects misplaced (7.263%) 2024-09-17T16:40:57.751 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:57.774 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:57.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:57.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:57.876 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:57.898 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:57.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:58.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:58.025 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:58.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:58.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:40:58.151 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:58.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:58.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:40:58.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:58.280 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:58.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:58.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:58.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:58.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:58.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:40:58.527 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:58.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:40:58.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:40:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:40:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:58 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:40:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:58 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:40:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:58 smithi071 ceph-mon[30002]: osdmap e918: 12 total, 12 up, 11 in 2024-09-17T16:40:58.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:58.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:40:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:40:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:58 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:40:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:58 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:40:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:58 smithi173 ceph-mon[30530]: osdmap e918: 12 total, 12 up, 11 in 2024-09-17T16:40:58.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:58.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:40:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:40:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:58 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:40:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:58 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:40:58.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:58 smithi017 ceph-mon[27571]: osdmap e918: 12 total, 12 up, 11 in 2024-09-17T16:40:58.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:58.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:58.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:58.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:40:58.875 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:40:58.907 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:58.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:40:59.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:40:59.033 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:59.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:59.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:40:59.134 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:40:59.162 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:59.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:40:59.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:40:59.293 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:40:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:59.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:40:59.395 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:40:59.418 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:59.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:40:59.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:59.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:59.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:59 smithi071 ceph-mon[30002]: pgmap v2022: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 5.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 12888/275400 objects degraded (4.680%); 44872/275400 objects misplaced (16.293%); 4.3 MiB/s, 70 objects/s recovering 2024-09-17T16:40:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:59 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 12888/275400 objects degraded (4.680%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:40:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:59 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:59 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:59 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:40:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:59 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:40:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:40:59 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:59.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:40:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:59 smithi173 ceph-mon[30530]: pgmap v2022: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 5.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 12888/275400 objects degraded (4.680%); 44872/275400 objects misplaced (16.293%); 4.3 MiB/s, 70 objects/s recovering 2024-09-17T16:40:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:59 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 12888/275400 objects degraded (4.680%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:40:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:59 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:59 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:59 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:40:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:59 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:40:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:40:59 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:59.667 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:40:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:59.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:40:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:59 smithi017 ceph-mon[27571]: pgmap v2022: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 5.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 12888/275400 objects degraded (4.680%); 44872/275400 objects misplaced (16.293%); 4.3 MiB/s, 70 objects/s recovering 2024-09-17T16:40:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:59 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 12888/275400 objects degraded (4.680%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:40:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:59 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:59 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:59 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:40:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:59 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:40:59.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:40:59 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:40:59.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:40:59.792 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:59.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:40:59.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:40:59.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:40:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:40:59.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:00.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:00.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:00.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:00.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:00.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:00.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:00.265 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:00.296 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:00.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:00.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:00.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:00.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:00.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:00.560 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:00.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:00.662 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:00.698 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:00.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:00.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:00.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:00.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:00.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:00.927 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:00.966 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:00.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:00.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:01.058 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:41:01.059 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:41:01.059 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-17T16:41:01.059 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-17T16:41:01.059 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:41:01.059 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:00 smithi017 ceph-mon[27571]: osdmap e919: 12 total, 12 up, 11 in 2024-09-17T16:41:01.059 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:00 smithi017 ceph-mon[27571]: pgmap v2024: 37 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 27 active+clean; 5.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 19751/274914 objects degraded (7.184%); 43787/274914 objects misplaced (15.928%); 19 MiB/s, 302 objects/s recovering 2024-09-17T16:41:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:00 smithi071 ceph-mon[30002]: osdmap e919: 12 total, 12 up, 11 in 2024-09-17T16:41:01.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:00 smithi071 ceph-mon[30002]: pgmap v2024: 37 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 27 active+clean; 5.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 19751/274914 objects degraded (7.184%); 43787/274914 objects misplaced (15.928%); 19 MiB/s, 302 objects/s recovering 2024-09-17T16:41:01.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:01.099 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:01.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:01.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:00 smithi173 ceph-mon[30530]: osdmap e919: 12 total, 12 up, 11 in 2024-09-17T16:41:01.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:00 smithi173 ceph-mon[30530]: pgmap v2024: 37 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 27 active+clean; 5.6 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 19751/274914 objects degraded (7.184%); 43787/274914 objects misplaced (15.928%); 19 MiB/s, 302 objects/s recovering 2024-09-17T16:41:01.200 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:01.232 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:01.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:01.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:01.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:01.357 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:01.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:01.460 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:01.495 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:01.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:01.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:01.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:01.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:01.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:01.627 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:01.632 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:01.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:01.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:01.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:01.853 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:01.888 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:01.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:01.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:02.023 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:02.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:02.124 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:02.155 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:02.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:02.257 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:02.269 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:41:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:02.294 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:02.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:02.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:02.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:02.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:02.523 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:02.557 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:02.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:02.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:02.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:02.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:02.787 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:02.821 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:02.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:02.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:02.962 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:02.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:03.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:03.096 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:03.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:03.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:03.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:03.228 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:03.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:03.330 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:03.333 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:41:03.366 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:03.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:03.469 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:03.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:03 smithi017 ceph-mon[27571]: pgmap v2025: 37 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 27 active+clean; 5.6 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 1.2 MiB/s wr, 78 op/s; 19751/274668 objects degraded (7.191%); 42956/274668 objects misplaced (15.639%); 18 MiB/s, 291 objects/s recovering 2024-09-17T16:41:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:03.508 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:03.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:03.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:03 smithi071 ceph-mon[30002]: pgmap v2025: 37 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 27 active+clean; 5.6 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 1.2 MiB/s wr, 78 op/s; 19751/274668 objects degraded (7.191%); 42956/274668 objects misplaced (15.639%); 18 MiB/s, 291 objects/s recovering 2024-09-17T16:41:03.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:03.643 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:03.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:03 smithi173 ceph-mon[30530]: pgmap v2025: 37 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 27 active+clean; 5.6 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 1.2 MiB/s wr, 78 op/s; 19751/274668 objects degraded (7.191%); 42956/274668 objects misplaced (15.639%); 18 MiB/s, 291 objects/s recovering 2024-09-17T16:41:03.747 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:03.782 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:03.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:03.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:03.913 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:03.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:04.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:04.046 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:04.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:04.148 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:04.182 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:04.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:04.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:04.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:04.310 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:04.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:04.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:04.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:04.542 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:04.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:04 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 12660/272304 objects degraded (4.649%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:04.580 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:04.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:04.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:04 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 12660/272304 objects degraded (4.649%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:04.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:04 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 12660/272304 objects degraded (4.649%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:04.682 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:04.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:04.719 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:04.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:04.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:04.851 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:04.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:04.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:04.978 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:04.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:05.079 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:05.112 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:05.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:05.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:05.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:05.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:05.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:05.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:05.373 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:05.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:05.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:05.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:05.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:05 smithi071 ceph-mon[30002]: pgmap v2026: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 5.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 12660/272304 objects degraded (4.649%); 47517/272304 objects misplaced (17.450%); 19 MiB/s, 298 objects/s recovering 2024-09-17T16:41:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:05 smithi071 ceph-mon[30002]: 1.7 scrub starts 2024-09-17T16:41:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:05 smithi071 ceph-mon[30002]: 1.a scrub starts 2024-09-17T16:41:05.603 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:05.636 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:05.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:05.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:05 smithi173 ceph-mon[30530]: pgmap v2026: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 5.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 12660/272304 objects degraded (4.649%); 47517/272304 objects misplaced (17.450%); 19 MiB/s, 298 objects/s recovering 2024-09-17T16:41:05.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:05 smithi173 ceph-mon[30530]: 1.7 scrub starts 2024-09-17T16:41:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:05 smithi173 ceph-mon[30530]: 1.a scrub starts 2024-09-17T16:41:05.737 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:05 smithi017 ceph-mon[27571]: pgmap v2026: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 5.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 12660/272304 objects degraded (4.649%); 47517/272304 objects misplaced (17.450%); 19 MiB/s, 298 objects/s recovering 2024-09-17T16:41:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:05 smithi017 ceph-mon[27571]: 1.7 scrub starts 2024-09-17T16:41:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:05 smithi017 ceph-mon[27571]: 1.a scrub starts 2024-09-17T16:41:05.767 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:05.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:05.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:05.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:05.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:05.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:06.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:06.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:06.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:06.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:06.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:06.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:06.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:06.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:06.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:06.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:06.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:06.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:06.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:06.525 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:06.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:06 smithi071 ceph-mon[30002]: 1.8 scrub starts 2024-09-17T16:41:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:06 smithi071 ceph-mon[30002]: 1.3 scrub starts 2024-09-17T16:41:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:06 smithi071 ceph-mon[30002]: 1.9 scrub starts 2024-09-17T16:41:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2059586469' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:41:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:06 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:41:06.627 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:06 smithi173 ceph-mon[30530]: 1.8 scrub starts 2024-09-17T16:41:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:06 smithi173 ceph-mon[30530]: 1.3 scrub starts 2024-09-17T16:41:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:06 smithi173 ceph-mon[30530]: 1.9 scrub starts 2024-09-17T16:41:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2059586469' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:41:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:06 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:41:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:06.664 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:06.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:06 smithi017 ceph-mon[27571]: 1.8 scrub starts 2024-09-17T16:41:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:06 smithi017 ceph-mon[27571]: 1.3 scrub starts 2024-09-17T16:41:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:06 smithi017 ceph-mon[27571]: 1.9 scrub starts 2024-09-17T16:41:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2059586469' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:41:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:06 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:41:06.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:06.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:06.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:06.800 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T16:41:06.896 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:06.927 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:06.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:07.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:07.063 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:07.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:07.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:07.191 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:07.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:07.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:07.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:07.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:07.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:07.419 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:07.448 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:07.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:07.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:07.560 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:07 smithi173 ceph-mon[30530]: pgmap v2027: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 5.5 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 12660/267945 objects degraded (4.725%); 46320/267945 objects misplaced (17.287%); 19 MiB/s, 308 objects/s recovering 2024-09-17T16:41:07.561 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:07 smithi173 ceph-mon[30530]: 1.2 scrub starts 2024-09-17T16:41:07.561 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:07 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:41:07.561 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:07 smithi173 ceph-mon[30530]: osdmap e920: 12 total, 12 up, 11 in 2024-09-17T16:41:07.561 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:07 smithi173 ceph-mon[30530]: 1.6 scrub starts 2024-09-17T16:41:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:07 smithi071 ceph-mon[30002]: pgmap v2027: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 5.5 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 12660/267945 objects degraded (4.725%); 46320/267945 objects misplaced (17.287%); 19 MiB/s, 308 objects/s recovering 2024-09-17T16:41:07.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:07 smithi071 ceph-mon[30002]: 1.2 scrub starts 2024-09-17T16:41:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:07 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:41:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:07 smithi071 ceph-mon[30002]: osdmap e920: 12 total, 12 up, 11 in 2024-09-17T16:41:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:07 smithi071 ceph-mon[30002]: 1.6 scrub starts 2024-09-17T16:41:07.575 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:07.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:07.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:07.710 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:07.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:07 smithi017 ceph-mon[27571]: pgmap v2027: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 5.5 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 12660/267945 objects degraded (4.725%); 46320/267945 objects misplaced (17.287%); 19 MiB/s, 308 objects/s recovering 2024-09-17T16:41:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:07 smithi017 ceph-mon[27571]: 1.2 scrub starts 2024-09-17T16:41:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:07 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:41:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:07 smithi017 ceph-mon[27571]: osdmap e920: 12 total, 12 up, 11 in 2024-09-17T16:41:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:07 smithi017 ceph-mon[27571]: 1.6 scrub starts 2024-09-17T16:41:07.812 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:07.843 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:07.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:07.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:07.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:07.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:08.071 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:08.105 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:08.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:08.207 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:08.230 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:08.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:08.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:08.365 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:08.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:08.467 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:08.509 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:08.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:08.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:08 smithi071 ceph-mon[30002]: 1.4 scrub starts 2024-09-17T16:41:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:08 smithi071 ceph-mon[30002]: 1.5 scrub starts 2024-09-17T16:41:08.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:08.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:08.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:08.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:08 smithi173 ceph-mon[30530]: 1.4 scrub starts 2024-09-17T16:41:08.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:08 smithi173 ceph-mon[30530]: 1.5 scrub starts 2024-09-17T16:41:08.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:08.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:08 smithi017 ceph-mon[27571]: 1.4 scrub starts 2024-09-17T16:41:08.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:08 smithi017 ceph-mon[27571]: 1.5 scrub starts 2024-09-17T16:41:08.763 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:08.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:08.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:08.894 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:08.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:08.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:09.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:09.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:09.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:09.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:09.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:09.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:09.276 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:09.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:09.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:09.411 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:09.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:09.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:09.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:09.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:09.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:09 smithi173 ceph-mon[30530]: pgmap v2029: 37 pgs: 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 24 active+clean; 5.4 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 12220/263598 objects degraded (4.636%); 44853/263598 objects misplaced (17.016%); 18 MiB/s, 294 objects/s recovering 2024-09-17T16:41:09.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:09 smithi173 ceph-mon[30530]: 1.1 scrub starts 2024-09-17T16:41:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:09 smithi071 ceph-mon[30002]: pgmap v2029: 37 pgs: 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 24 active+clean; 5.4 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 12220/263598 objects degraded (4.636%); 44853/263598 objects misplaced (17.016%); 18 MiB/s, 294 objects/s recovering 2024-09-17T16:41:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:09 smithi071 ceph-mon[30002]: 1.1 scrub starts 2024-09-17T16:41:09.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:09.670 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:09.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:09 smithi017 ceph-mon[27571]: pgmap v2029: 37 pgs: 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 24 active+clean; 5.4 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 12220/263598 objects degraded (4.636%); 44853/263598 objects misplaced (17.016%); 18 MiB/s, 294 objects/s recovering 2024-09-17T16:41:09.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:09 smithi017 ceph-mon[27571]: 1.1 scrub starts 2024-09-17T16:41:09.771 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:09.806 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:09.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:09.908 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:09.951 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:09.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:10.053 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:10.082 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:10.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:10.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:10.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:10.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:10.317 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:10.358 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:10.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:10.459 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:10.487 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:10.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:10.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 12220/263598 objects degraded (4.636%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:10 smithi071 ceph-mon[30002]: osdmap e921: 12 total, 12 up, 11 in 2024-09-17T16:41:10.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:10.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:10.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:10.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 12220/263598 objects degraded (4.636%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:10 smithi173 ceph-mon[30530]: osdmap e921: 12 total, 12 up, 11 in 2024-09-17T16:41:10.715 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:10.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 12220/263598 objects degraded (4.636%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:10 smithi017 ceph-mon[27571]: osdmap e921: 12 total, 12 up, 11 in 2024-09-17T16:41:10.745 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:10.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:10.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:10.875 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:10.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:10.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:11.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:11.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:11.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:11.110 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:11.136 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:11.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:11.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:11.265 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:11.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:11.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:11.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:11 smithi173 ceph-mon[30530]: pgmap v2031: 37 pgs: 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 23 active+clean; 5.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 12220/258333 objects degraded (4.730%); 43660/258333 objects misplaced (16.901%); 22 MiB/s, 352 objects/s recovering 2024-09-17T16:41:11.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:11.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:11.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:11.545 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:11.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:11 smithi071 ceph-mon[30002]: pgmap v2031: 37 pgs: 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 23 active+clean; 5.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 12220/258333 objects degraded (4.730%); 43660/258333 objects misplaced (16.901%); 22 MiB/s, 352 objects/s recovering 2024-09-17T16:41:11.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:11.674 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:11.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:11 smithi017 ceph-mon[27571]: pgmap v2031: 37 pgs: 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 23 active+clean; 5.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 12220/258333 objects degraded (4.730%); 43660/258333 objects misplaced (16.901%); 22 MiB/s, 352 objects/s recovering 2024-09-17T16:41:11.776 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:11.819 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:11.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:11.921 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:11.956 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:11.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:12.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:12.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:12.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:12.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:12.219 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:12.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:12.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:12.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:12.355 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:12.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:12.458 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:12.497 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:12.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:12.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:12.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:12.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:12.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:12.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:12.759 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:12.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:12.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:12.900 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:12.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:13.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:13.034 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:13.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:13.136 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:13.172 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:13.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:13.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:13.311 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:13.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:13.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:13.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:13.442 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:13.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:13.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:13 smithi071 ceph-mon[30002]: pgmap v2032: 37 pgs: 2 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 5.2 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 636 op/s; 12220/257016 objects degraded (4.755%); 43049/257016 objects misplaced (16.750%); 19 MiB/s, 303 objects/s recovering 2024-09-17T16:41:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:13 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:41:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:13 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:41:13.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:13.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:13 smithi173 ceph-mon[30530]: pgmap v2032: 37 pgs: 2 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 5.2 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 636 op/s; 12220/257016 objects degraded (4.755%); 43049/257016 objects misplaced (16.750%); 19 MiB/s, 303 objects/s recovering 2024-09-17T16:41:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:13 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:41:13.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:13 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:41:13.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:13.705 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:13.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:13 smithi017 ceph-mon[27571]: pgmap v2032: 37 pgs: 2 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 5.2 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 636 op/s; 12220/257016 objects degraded (4.755%); 43049/257016 objects misplaced (16.750%); 19 MiB/s, 303 objects/s recovering 2024-09-17T16:41:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:13 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:41:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:13 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:41:13.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:13.840 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:13.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:13.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:13.976 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:13.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:14.077 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:14.110 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:14.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:14.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:14.241 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:14.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:14.343 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:14.370 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:14.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:14.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:14.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:14.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:14.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:14.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:14.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:14.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:41:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:41:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:14 smithi173 ceph-mon[30530]: osdmap e922: 12 total, 12 up, 11 in 2024-09-17T16:41:14.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:14.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:41:14.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:41:14.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:14 smithi017 ceph-mon[27571]: osdmap e922: 12 total, 12 up, 11 in 2024-09-17T16:41:14.760 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:14.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:14.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:41:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:41:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:14 smithi071 ceph-mon[30002]: osdmap e922: 12 total, 12 up, 11 in 2024-09-17T16:41:14.862 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:14.898 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:14.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:15.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:15.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:15.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:15.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:15.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:15.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:15.254 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:15.291 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:15.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:15.377 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:41:15.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:41:15.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:41:15.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:41:15.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:41:15.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:41:15.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:41:15.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:41:15.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:41:15.378 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:41:15.379 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:41:15.379 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:41:15.379 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:41:15.379 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:41:15.379 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:41:15.379 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:41:15.380 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:41:15.380 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:41:15.380 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:41:15.380 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:41:15.380 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:41:15.380 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:41:15.380 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:41:15.381 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:41:15.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:15.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:15.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:15.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:15.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:15.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:15 smithi173 ceph-mon[30530]: pgmap v2034: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 5.1 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 11842/252711 objects degraded (4.686%); 41736/252711 objects misplaced (16.515%); 20 MiB/s, 313 objects/s recovering 2024-09-17T16:41:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1406021030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:41:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:15 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11842/252711 objects degraded (4.686%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:15 smithi173 ceph-mon[30530]: osdmap e923: 12 total, 12 up, 11 in 2024-09-17T16:41:15.653 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:15.690 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:15.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:15 smithi017 ceph-mon[27571]: pgmap v2034: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 5.1 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 11842/252711 objects degraded (4.686%); 41736/252711 objects misplaced (16.515%); 20 MiB/s, 313 objects/s recovering 2024-09-17T16:41:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1406021030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:41:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:15 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11842/252711 objects degraded (4.686%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:15 smithi017 ceph-mon[27571]: osdmap e923: 12 total, 12 up, 11 in 2024-09-17T16:41:15.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:15 smithi071 ceph-mon[30002]: pgmap v2034: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 5.1 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 11842/252711 objects degraded (4.686%); 41736/252711 objects misplaced (16.515%); 20 MiB/s, 313 objects/s recovering 2024-09-17T16:41:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1406021030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:41:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:15 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11842/252711 objects degraded (4.686%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:15 smithi071 ceph-mon[30002]: osdmap e923: 12 total, 12 up, 11 in 2024-09-17T16:41:15.821 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:15.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:15.923 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:15.964 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:15.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:16.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:16.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:16.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:16.194 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:16.222 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:16.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:16.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:16.360 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:16.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:16.461 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:16.490 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:16.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:16.592 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:16.628 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:16.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:16.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:16.754 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:16.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:16.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:16.878 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:16.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:16.980 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:17.009 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:17.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:17.112 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:17.145 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:17.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:17.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:17.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:17.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:17.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:17.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:17.397 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:17.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:17.500 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:17.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:17.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:17.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:17.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:17 smithi173 ceph-mon[30530]: pgmap v2036: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 5.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 11842/247998 objects degraded (4.775%); 40513/247998 objects misplaced (16.336%); 17 MiB/s, 271 objects/s recovering 2024-09-17T16:41:17.662 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:17.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:17 smithi017 ceph-mon[27571]: pgmap v2036: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 5.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 11842/247998 objects degraded (4.775%); 40513/247998 objects misplaced (16.336%); 17 MiB/s, 271 objects/s recovering 2024-09-17T16:41:17.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:17.789 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:17.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:17 smithi071 ceph-mon[30002]: pgmap v2036: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 5.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 11842/247998 objects degraded (4.775%); 40513/247998 objects misplaced (16.336%); 17 MiB/s, 271 objects/s recovering 2024-09-17T16:41:17.890 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:17.922 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:17.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:17.966 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T16:41:18.024 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:18.070 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:18.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:18.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:18.172 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:18.208 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:18.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:18.310 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:18.346 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:18.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:18.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:18.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:18.483 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:18.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:18.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:18.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:18.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:18.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:18.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:18.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:18.843 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:18.874 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:18.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:18.976 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:18.999 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:19.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:19.101 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:19.141 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:19.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:19.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:19.243 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:19.276 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:19.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:19.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:19.378 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:19.407 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:19.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:19.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:19.539 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:19.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:19.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:19.641 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:19.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:19 smithi173 ceph-mon[30530]: pgmap v2037: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 11454/243123 objects degraded (4.711%); 39050/243123 objects misplaced (16.062%); 19 MiB/s, 308 objects/s recovering 2024-09-17T16:41:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:19.677 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:19.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:19 smithi017 ceph-mon[27571]: pgmap v2037: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 11454/243123 objects degraded (4.711%); 39050/243123 objects misplaced (16.062%); 19 MiB/s, 308 objects/s recovering 2024-09-17T16:41:19.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:19.808 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:19.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:19.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:19.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:19 smithi071 ceph-mon[30002]: pgmap v2037: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.9 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 11454/243123 objects degraded (4.711%); 39050/243123 objects misplaced (16.062%); 19 MiB/s, 308 objects/s recovering 2024-09-17T16:41:19.911 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:19.942 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:19.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:20.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:20.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:20.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:20.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:20.203 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:20.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:20.306 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:20.328 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:20.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:20.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:20.430 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:20.466 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:20.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:20.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:20.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:20.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:20.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:20 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11454/243123 objects degraded (4.711%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:20.695 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:20.728 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:20.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:20.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:20 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11454/243123 objects degraded (4.711%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:20 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11454/243123 objects degraded (4.711%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:20.830 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:20.862 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:20.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:20.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:20.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:20.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:20.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:21.092 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:21.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:21.124 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:21.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:21.225 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:21.254 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:21.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:21.356 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:21.384 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:21.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:21.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:21.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:21.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:21.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:21.611 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:21.644 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:21 smithi173 ceph-mon[30530]: pgmap v2038: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 11454/237210 objects degraded (4.829%); 37616/237210 objects misplaced (15.858%); 24 MiB/s, 382 objects/s recovering 2024-09-17T16:41:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:21.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:21.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:21 smithi017 ceph-mon[27571]: pgmap v2038: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 11454/237210 objects degraded (4.829%); 37616/237210 objects misplaced (15.858%); 24 MiB/s, 382 objects/s recovering 2024-09-17T16:41:21.746 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:21.770 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:21.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:21 smithi071 ceph-mon[30002]: pgmap v2038: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 11454/237210 objects degraded (4.829%); 37616/237210 objects misplaced (15.858%); 24 MiB/s, 382 objects/s recovering 2024-09-17T16:41:21.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:21.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:21.896 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:21.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:22.034 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:22.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:22.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:22.170 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:22.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:22.272 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:22.310 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:22.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:22.413 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:22.444 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:22.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:22.546 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:22.575 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:22.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:22.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:22.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:22.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:22.805 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:22.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:22.841 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:22.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:22.941 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:22.941 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:22.947 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:22.970 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:22.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:23.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:23.100 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:23.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:23.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:23.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:23.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:23.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:23.374 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:23.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:23.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:23.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:23.507 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:23.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:23.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:23.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:23.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:23.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:23 smithi173 ceph-mon[30530]: pgmap v2039: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 667 op/s; 11454/235893 objects degraded (4.856%); 36804/235893 objects misplaced (15.602%); 20 MiB/s, 316 objects/s recovering 2024-09-17T16:41:23.719 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:23 smithi071 ceph-mon[30002]: pgmap v2039: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 667 op/s; 11454/235893 objects degraded (4.856%); 36804/235893 objects misplaced (15.602%); 20 MiB/s, 316 objects/s recovering 2024-09-17T16:41:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:23 smithi017 ceph-mon[27571]: pgmap v2039: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 667 op/s; 11454/235893 objects degraded (4.856%); 36804/235893 objects misplaced (15.602%); 20 MiB/s, 316 objects/s recovering 2024-09-17T16:41:23.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:23.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:23.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:23.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:23.895 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:23.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:23.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:24.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:24.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:24.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:24.161 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:24.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:24.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:24.297 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:24.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:24.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:24.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:24.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:24.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:24.529 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:24.555 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:24.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:24.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:24.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:24.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:24.784 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:24.800 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:24.818 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:24.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:24.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:24.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:41:24.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:24.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:25.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:25.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:25.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:25.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:25.235 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:25.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:25.336 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:25.375 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:25.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:25.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:25 smithi017 ceph-mon[27571]: pgmap v2040: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 11026/231480 objects degraded (4.763%); 35510/231480 objects misplaced (15.340%); 21 MiB/s, 338 objects/s recovering 2024-09-17T16:41:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:25 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 11026/231480 objects degraded (4.763%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:25.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:25.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:25.601 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:25.623 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:25.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:25 smithi173 ceph-mon[30530]: pgmap v2040: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 11026/231480 objects degraded (4.763%); 35510/231480 objects misplaced (15.340%); 21 MiB/s, 338 objects/s recovering 2024-09-17T16:41:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:25 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 11026/231480 objects degraded (4.763%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:25.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:25.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:25.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:25.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:25.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:25 smithi071 ceph-mon[30002]: pgmap v2040: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 11026/231480 objects degraded (4.763%); 35510/231480 objects misplaced (15.340%); 21 MiB/s, 338 objects/s recovering 2024-09-17T16:41:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:25 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 11026/231480 objects degraded (4.763%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:25.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:25.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:25.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:25.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:25.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:26.015 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:26.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:26.116 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:26.143 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:26.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:26.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:26.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:26.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:26.345 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:41:26.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:26.406 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:26.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4145094324' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:41:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:26 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:41:26.507 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:26.536 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:26.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:26.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:26.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4145094324' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:41:26.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:26 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:41:26.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:26.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:26.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:26.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:26.804 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:26.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:26.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:26.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4145094324' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:41:26.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:26 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:41:26.907 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:26.936 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:26.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:27.038 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:27.071 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:27.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:27.173 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:27.205 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:27.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:27.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:27.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:27.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:27.437 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:27.466 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:27.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:27.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:27.630 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:27.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:27.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:27 smithi173 ceph-mon[30530]: pgmap v2041: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 11026/226788 objects degraded (4.862%); 34085/226788 objects misplaced (15.029%); 20 MiB/s, 326 objects/s recovering 2024-09-17T16:41:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:27 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:41:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:27 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:41:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:27 smithi173 ceph-mon[30530]: osdmap e924: 12 total, 12 up, 11 in 2024-09-17T16:41:27.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:27.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:27 smithi017 ceph-mon[27571]: pgmap v2041: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 11026/226788 objects degraded (4.862%); 34085/226788 objects misplaced (15.029%); 20 MiB/s, 326 objects/s recovering 2024-09-17T16:41:27.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:27 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:41:27.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:27 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:41:27.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:27 smithi017 ceph-mon[27571]: osdmap e924: 12 total, 12 up, 11 in 2024-09-17T16:41:27.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:27.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:27 smithi071 ceph-mon[30002]: pgmap v2041: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 11026/226788 objects degraded (4.862%); 34085/226788 objects misplaced (15.029%); 20 MiB/s, 326 objects/s recovering 2024-09-17T16:41:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:27 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:41:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:27 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:41:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:27 smithi071 ceph-mon[30002]: osdmap e924: 12 total, 12 up, 11 in 2024-09-17T16:41:27.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:27.895 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:27.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:27.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:28.030 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:28.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:28.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:28.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:28.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:28.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:28.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:28.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:28.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:28.420 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:28.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:28.521 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:28.544 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:28.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:28.646 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:28.681 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:28.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:28.783 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:28.822 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:28.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:28.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:28.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:28.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:29.051 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:29.088 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:29.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:29.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:29.229 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:29.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:29.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:29.364 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:29.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:29.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:29.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:29.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:29.594 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:29.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:29 smithi017 ceph-mon[27571]: pgmap v2043: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 686 op/s; 10650/222528 objects degraded (4.786%); 32546/222528 objects misplaced (14.626%); 22 MiB/s, 359 objects/s recovering 2024-09-17T16:41:29.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:29 smithi173 ceph-mon[30530]: pgmap v2043: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 686 op/s; 10650/222528 objects degraded (4.786%); 32546/222528 objects misplaced (14.626%); 22 MiB/s, 359 objects/s recovering 2024-09-17T16:41:29.647 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:29.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:29.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:29.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:29.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:29.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:29 smithi071 ceph-mon[30002]: pgmap v2043: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 686 op/s; 10650/222528 objects degraded (4.786%); 32546/222528 objects misplaced (14.626%); 22 MiB/s, 359 objects/s recovering 2024-09-17T16:41:29.879 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:29.911 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:29.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:30.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:30.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:30.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:30.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:30.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:30.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:30.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:30.316 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:30.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:30.365 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:41:30.365 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:41:30.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:30.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:30.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:30.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:30.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:30.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:30.573 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:30 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 10650/222528 objects degraded (4.786%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:30 smithi173 ceph-mon[30530]: osdmap e925: 12 total, 12 up, 11 in 2024-09-17T16:41:30.675 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:30.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:30 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 10650/222528 objects degraded (4.786%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:30.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:30 smithi017 ceph-mon[27571]: osdmap e925: 12 total, 12 up, 11 in 2024-09-17T16:41:30.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:30.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:30 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 10650/222528 objects degraded (4.786%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:30.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:30 smithi071 ceph-mon[30002]: osdmap e925: 12 total, 12 up, 11 in 2024-09-17T16:41:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:30.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:30.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:30.937 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:30.963 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:30.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:31.066 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:31.102 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:31.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:31.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:31.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:31.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:31.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:31.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:31.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:31.457 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:31.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:31.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:31.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:31.636 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:31.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:31.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:31 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:41:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:31 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:41:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:31 smithi173 ceph-mon[30530]: pgmap v2045: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 10650/217089 objects degraded (4.906%); 31140/217089 objects misplaced (14.344%); 26 MiB/s, 410 objects/s recovering 2024-09-17T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:31 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:31 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:41:31.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:31 smithi017 ceph-mon[27571]: pgmap v2045: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 10650/217089 objects degraded (4.906%); 31140/217089 objects misplaced (14.344%); 26 MiB/s, 410 objects/s recovering 2024-09-17T16:41:31.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:31.772 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:31.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:31 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:41:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:31 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:41:31.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:31 smithi071 ceph-mon[30002]: pgmap v2045: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 783 op/s; 10650/217089 objects degraded (4.906%); 31140/217089 objects misplaced (14.344%); 26 MiB/s, 410 objects/s recovering 2024-09-17T16:41:31.874 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:31.906 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:31.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:32.009 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:32.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:32.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:32.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:32.133 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:32.159 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:32.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:32.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:32.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:32.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:32.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:32.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:32.352 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T16:41:32.384 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:32.423 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:32.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:32.525 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:32.546 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:32.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1955777223' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:41:32.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:32.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1955777223' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:41:32.682 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:32.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:32.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:32.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:32.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:32.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1955777223' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:41:32.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:32.943 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:32.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:33.045 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:33.084 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:33.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:33.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:33.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:33.222 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:33.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:33.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:33.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:33.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:33.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:33.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:33.485 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:33.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:33.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:33.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:33.630 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:33.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:33 smithi173 ceph-mon[30530]: pgmap v2046: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 10650/215628 objects degraded (4.939%); 30212/215628 objects misplaced (14.011%); 23 MiB/s, 364 objects/s recovering 2024-09-17T16:41:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1955777223' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:41:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:33 smithi173 ceph-mon[30530]: osdmap e926: 12 total, 12 up, 11 in 2024-09-17T16:41:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:33 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:41:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:33 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:41:33.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:33.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:33 smithi017 ceph-mon[27571]: pgmap v2046: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 10650/215628 objects degraded (4.939%); 30212/215628 objects misplaced (14.011%); 23 MiB/s, 364 objects/s recovering 2024-09-17T16:41:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1955777223' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:41:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:33 smithi017 ceph-mon[27571]: osdmap e926: 12 total, 12 up, 11 in 2024-09-17T16:41:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:33 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:41:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:33 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:41:33.759 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:33.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:33 smithi071 ceph-mon[30002]: pgmap v2046: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 10650/215628 objects degraded (4.939%); 30212/215628 objects misplaced (14.011%); 23 MiB/s, 364 objects/s recovering 2024-09-17T16:41:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1955777223' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:41:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:33 smithi071 ceph-mon[30002]: osdmap e926: 12 total, 12 up, 11 in 2024-09-17T16:41:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:33 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:41:33.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:33 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:41:33.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:33.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:33.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:33.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:34.019 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:34.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:34.120 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:34.151 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:34.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:34.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:34.279 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:34.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:34.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:34.381 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:34.420 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:34.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:34.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:34.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:34.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:34 smithi173 ceph-mon[30530]: osdmap e927: 12 total, 12 up, 11 in 2024-09-17T16:41:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:41:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:41:34.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:34.682 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:34.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:34.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:34 smithi017 ceph-mon[27571]: osdmap e927: 12 total, 12 up, 11 in 2024-09-17T16:41:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:41:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:41:34.784 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:34 smithi071 ceph-mon[30002]: osdmap e927: 12 total, 12 up, 11 in 2024-09-17T16:41:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:41:34.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:41:34.816 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:34.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:34.918 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:34.950 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:34.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:35.052 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:35.082 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:35.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:35.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:35.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:35.220 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:35.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:35.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:35.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:35.362 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:35.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:35.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:35.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:35.486 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:35.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:35.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:35.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:35.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:35.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:35 smithi173 ceph-mon[30530]: pgmap v2049: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 626 op/s; 10248/211248 objects degraded (4.851%); 28686/211248 objects misplaced (13.579%); 22 MiB/s, 349 objects/s recovering 2024-09-17T16:41:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:35 smithi173 ceph-mon[30530]: osdmap e928: 12 total, 12 up, 11 in 2024-09-17T16:41:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:35 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 10248/211248 objects degraded (4.851%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:35 smithi173 ceph-mon[30530]: osdmap e929: 12 total, 12 up, 11 in 2024-09-17T16:41:35.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:35.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:35 smithi017 ceph-mon[27571]: pgmap v2049: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 626 op/s; 10248/211248 objects degraded (4.851%); 28686/211248 objects misplaced (13.579%); 22 MiB/s, 349 objects/s recovering 2024-09-17T16:41:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:35 smithi017 ceph-mon[27571]: osdmap e928: 12 total, 12 up, 11 in 2024-09-17T16:41:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:35 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 10248/211248 objects degraded (4.851%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:35 smithi017 ceph-mon[27571]: osdmap e929: 12 total, 12 up, 11 in 2024-09-17T16:41:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:35.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:35 smithi071 ceph-mon[30002]: pgmap v2049: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 4.3 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 626 op/s; 10248/211248 objects degraded (4.851%); 28686/211248 objects misplaced (13.579%); 22 MiB/s, 349 objects/s recovering 2024-09-17T16:41:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:35 smithi071 ceph-mon[30002]: osdmap e928: 12 total, 12 up, 11 in 2024-09-17T16:41:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:35 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 10248/211248 objects degraded (4.851%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:35 smithi071 ceph-mon[30002]: osdmap e929: 12 total, 12 up, 11 in 2024-09-17T16:41:35.841 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:35.866 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:35.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:35.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:36.008 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:36.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:36.110 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:36.139 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:36.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:36.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:36.272 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:36.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:36.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:36.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:36.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:36.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:36.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:36.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:36.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:36.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:36.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:36 smithi173 ceph-mon[30530]: 1.7 scrub starts 2024-09-17T16:41:36.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:36 smithi173 ceph-mon[30530]: 1.a scrub starts 2024-09-17T16:41:36.661 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:36.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:36.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:36 smithi017 ceph-mon[27571]: 1.7 scrub starts 2024-09-17T16:41:36.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:36 smithi017 ceph-mon[27571]: 1.a scrub starts 2024-09-17T16:41:36.762 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:36.795 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:36.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:36 smithi071 ceph-mon[30002]: 1.7 scrub starts 2024-09-17T16:41:36.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:36 smithi071 ceph-mon[30002]: 1.a scrub starts 2024-09-17T16:41:36.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:36.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:36.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:37.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:37.047 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:37.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:37.149 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:37.177 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:37.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:37.279 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:37.307 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:37.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:37.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:37.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:37.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:37.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:37.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:37.573 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:37.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:37 smithi173 ceph-mon[30530]: 1.8 scrub starts 2024-09-17T16:41:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:37 smithi173 ceph-mon[30530]: pgmap v2052: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 4.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 776 op/s; 10248/206304 objects degraded (4.967%); 26969/206304 objects misplaced (13.072%); 29 MiB/s, 471 objects/s recovering 2024-09-17T16:41:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:37 smithi173 ceph-mon[30530]: 1.9 scrub starts 2024-09-17T16:41:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:37 smithi173 ceph-mon[30530]: 1.3 scrub starts 2024-09-17T16:41:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:37 smithi173 ceph-mon[30530]: 1.6 scrub starts 2024-09-17T16:41:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:37 smithi173 ceph-mon[30530]: 1.2 scrub starts 2024-09-17T16:41:37.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:37.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:37.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:37 smithi017 ceph-mon[27571]: 1.8 scrub starts 2024-09-17T16:41:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:37 smithi017 ceph-mon[27571]: pgmap v2052: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 4.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 776 op/s; 10248/206304 objects degraded (4.967%); 26969/206304 objects misplaced (13.072%); 29 MiB/s, 471 objects/s recovering 2024-09-17T16:41:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:37 smithi017 ceph-mon[27571]: 1.9 scrub starts 2024-09-17T16:41:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:37 smithi017 ceph-mon[27571]: 1.3 scrub starts 2024-09-17T16:41:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:37 smithi017 ceph-mon[27571]: 1.6 scrub starts 2024-09-17T16:41:37.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:37 smithi017 ceph-mon[27571]: 1.2 scrub starts 2024-09-17T16:41:37.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:37 smithi071 ceph-mon[30002]: 1.8 scrub starts 2024-09-17T16:41:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:37 smithi071 ceph-mon[30002]: pgmap v2052: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 4.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 776 op/s; 10248/206304 objects degraded (4.967%); 26969/206304 objects misplaced (13.072%); 29 MiB/s, 471 objects/s recovering 2024-09-17T16:41:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:37 smithi071 ceph-mon[30002]: 1.9 scrub starts 2024-09-17T16:41:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:37 smithi071 ceph-mon[30002]: 1.3 scrub starts 2024-09-17T16:41:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:37 smithi071 ceph-mon[30002]: 1.6 scrub starts 2024-09-17T16:41:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:37 smithi071 ceph-mon[30002]: 1.2 scrub starts 2024-09-17T16:41:37.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:37.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:37.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:37.970 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:37.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:38.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:38.106 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:38.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:38.208 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:38.235 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:38.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:38.337 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:38.340 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:41:38.369 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:38.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:38.472 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:38.498 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:38.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:38.600 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:38.631 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:38.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:38.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:38.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:38.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:38.798 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:41:38.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:38.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:38.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:38.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:38.989 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:39.024 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:39.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:39.125 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:39.151 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:39.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:39.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:39.291 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:39.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:39.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:39.423 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:39.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:39 smithi017 ceph-mon[27571]: 1.4 scrub starts 2024-09-17T16:41:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:39 smithi017 ceph-mon[27571]: pgmap v2053: 37 pgs: 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 23 active+clean; 4.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 790 op/s; 9772/201825 objects degraded (4.842%); 25298/201825 objects misplaced (12.535%); 30 MiB/s, 476 objects/s recovering 2024-09-17T16:41:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:39 smithi017 ceph-mon[27571]: 1.5 scrub starts 2024-09-17T16:41:39.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:39.569 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:39.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:39 smithi173 ceph-mon[30530]: 1.4 scrub starts 2024-09-17T16:41:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:39 smithi173 ceph-mon[30530]: pgmap v2053: 37 pgs: 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 23 active+clean; 4.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 790 op/s; 9772/201825 objects degraded (4.842%); 25298/201825 objects misplaced (12.535%); 30 MiB/s, 476 objects/s recovering 2024-09-17T16:41:39.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:39 smithi173 ceph-mon[30530]: 1.5 scrub starts 2024-09-17T16:41:39.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:39.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:39.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:39.703 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T16:41:39.707 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:39.797 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:39 smithi071 ceph-mon[30002]: 1.4 scrub starts 2024-09-17T16:41:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:39 smithi071 ceph-mon[30002]: pgmap v2053: 37 pgs: 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 23 active+clean; 4.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 790 op/s; 9772/201825 objects degraded (4.842%); 25298/201825 objects misplaced (12.535%); 30 MiB/s, 476 objects/s recovering 2024-09-17T16:41:39.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:39 smithi071 ceph-mon[30002]: 1.5 scrub starts 2024-09-17T16:41:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:39.839 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:39.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:39.941 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:39.974 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:39.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:40.077 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:40.108 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:40.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:40.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:40.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:40.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:40.326 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:41:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:41:40.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:40.336 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:40.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:40.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:40.360 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:41:40.360 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:41:40.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:41:40.390 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:41:40.423 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:41:40.423 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:41:40.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:41:40.455 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:41:40.457 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:40.495 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:41:40.496 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:41:40.499 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:40.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:40.528 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:41:40.529 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:41:40.561 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:41:40.561 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:41:40.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:40 smithi017 ceph-mon[27571]: 1.1 scrub starts 2024-09-17T16:41:40.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:40 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 9772/201825 objects degraded (4.842%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:40.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/841073872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:41:40.586 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:41:40.586 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:41:40.601 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:40.613 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:41:40.613 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:41:40.629 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:40.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:40.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:40 smithi173 ceph-mon[30530]: 1.1 scrub starts 2024-09-17T16:41:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:40 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 9772/201825 objects degraded (4.842%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:40.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/841073872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:41:40.646 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:41:40.647 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:41:40.688 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:41:40.688 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:41:40.731 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:40.773 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:40.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:40.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:40 smithi071 ceph-mon[30002]: 1.1 scrub starts 2024-09-17T16:41:40.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:40 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 9772/201825 objects degraded (4.842%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:40.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/841073872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:41:40.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:40.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:40.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:41.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:41.042 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:41.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:41.056 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:41:41.143 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:41.169 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:41.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:41.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:41.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:41.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:41.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:41.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:41.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:41.521 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:41.551 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:41.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:41 smithi173 ceph-mon[30530]: pgmap v2054: 37 pgs: 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 21 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 794 op/s; 9772/195966 objects degraded (4.987%); 23760/195966 objects misplaced (12.125%); 28 MiB/s, 453 objects/s recovering 2024-09-17T16:41:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/939320703' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:41:41.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:41 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:41:41.651 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:41.684 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:41.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:41.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:41 smithi071 ceph-mon[30002]: pgmap v2054: 37 pgs: 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 21 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 794 op/s; 9772/195966 objects degraded (4.987%); 23760/195966 objects misplaced (12.125%); 28 MiB/s, 453 objects/s recovering 2024-09-17T16:41:41.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/939320703' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:41:41.714 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:41 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:41:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:41 smithi017 ceph-mon[27571]: pgmap v2054: 37 pgs: 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 21 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 794 op/s; 9772/195966 objects degraded (4.987%); 23760/195966 objects misplaced (12.125%); 28 MiB/s, 453 objects/s recovering 2024-09-17T16:41:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/939320703' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:41:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:41 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:41:41.786 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:41.816 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:41.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:41.838 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T16:41:41.917 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:41.955 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:41.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:42.056 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:42.085 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:42.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:42.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:42.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:42.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:42.314 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:42.350 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:42.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:42.451 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:42.486 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:42.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:42.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:42.622 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:42.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:42.724 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:42.753 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:42.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:42.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:42.888 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:42.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:42.990 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:43.001 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T16:41:43.004 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:43.020 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:43.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:43.122 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:43.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:42 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:41:43.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:42 smithi017 ceph-mon[27571]: osdmap e930: 12 total, 12 up, 11 in 2024-09-17T16:41:43.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:42 smithi017 ceph-mon[27571]: pgmap v2056: 37 pgs: 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 21 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 9772/194709 objects degraded (5.019%); 22785/194709 objects misplaced (11.702%); 26 MiB/s, 421 objects/s recovering 2024-09-17T16:41:43.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:42 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:41:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:42 smithi173 ceph-mon[30530]: osdmap e930: 12 total, 12 up, 11 in 2024-09-17T16:41:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:42 smithi173 ceph-mon[30530]: pgmap v2056: 37 pgs: 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 21 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 9772/194709 objects degraded (5.019%); 22785/194709 objects misplaced (11.702%); 26 MiB/s, 421 objects/s recovering 2024-09-17T16:41:43.161 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:43.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:43.262 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:43.297 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:43.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:42 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:41:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:42 smithi071 ceph-mon[30002]: osdmap e930: 12 total, 12 up, 11 in 2024-09-17T16:41:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:42 smithi071 ceph-mon[30002]: pgmap v2056: 37 pgs: 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 21 active+clean; 4.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 9772/194709 objects degraded (5.019%); 22785/194709 objects misplaced (11.702%); 26 MiB/s, 421 objects/s recovering 2024-09-17T16:41:43.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:43.420 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:43.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:43.521 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:43.559 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:43.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:43.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:43.673 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:43.675 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:41:43.675 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:41:43.675 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.675 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:152088 9.c 9:3fa001b9:::benchmark_data_smithi173_150363_object45183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e931)", 2024-09-17T16:41:43.675 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:41:43.653573+0000", 2024-09-17T16:41:43.675 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.016405214000000001, 2024-09-17T16:41:43.675 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.016434170000000001, 2024-09-17T16:41:43.675 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152088 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.653573+0000", 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:41:43.676 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.653573+0000", 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.653574+0000", 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.809e-06 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:41:43.677 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.653581+0000", 2024-09-17T16:41:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5529999999999998e-06 2024-09-17T16:41:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:41:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.653601+0000", 2024-09-17T16:41:43.678 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9715999999999999e-05 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.653609+0000", 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1929999999999993e-06 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.653684+0000", 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4704999999999997e-05 2024-09-17T16:41:43.679 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.653851+0000", 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016710100000000001 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.653974+0000", 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012306900000000001 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.680 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.659231+0000", 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0052574429999999997 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.668492+0000", 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0092606779999999996 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.681 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:152092 9.c 9:3ecde3f2:::benchmark_data_smithi173_150363_object45187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e931)", 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:41:43.662599+0000", 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0073790899999999996, 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0074938510000000002, 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152092 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.682 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.662599+0000", 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.662599+0000", 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.662601+0000", 2024-09-17T16:41:43.683 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.911e-06 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.662608+0000", 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8359999999999992e-06 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.662624+0000", 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5478999999999999e-05 2024-09-17T16:41:43.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.662632+0000", 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5010000000000004e-06 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.663154+0000", 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00052150600000000003 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:41:43.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.663478+0000", 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032355200000000001 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.663592+0000", 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000114145 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:152096 9.c 9:3352f42a:::benchmark_data_smithi173_150363_object45191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e931)", 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:41:43.667139+0000", 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0028392220000000002, 2024-09-17T16:41:43.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0030203280000000001, 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 152096 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.667139+0000", 2024-09-17T16:41:43.687 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.667139+0000", 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.667141+0000", 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9570000000000001e-06 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:41:43.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.667148+0000", 2024-09-17T16:41:43.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8129999999999996e-06 2024-09-17T16:41:43.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.692 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:41:43.692 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.667170+0000", 2024-09-17T16:41:43.692 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1718000000000001e-05 2024-09-17T16:41:43.692 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.692 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.692 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:41:43.692 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.667180+0000", 2024-09-17T16:41:43.692 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8409999999999995e-06 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.668529+0000", 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013492339999999999 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.668682+0000", 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015278000000000001 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:41:43.693 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:41:43.694 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:43.668816+0000", 2024-09-17T16:41:43.694 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013396 2024-09-17T16:41:43.694 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:41:43.694 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:41:43.694 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:41:43.694 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:41:43.694 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-09-17T16:41:43.694 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 3 2024-09-17T16:41:43.694 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:41:43.697 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:43.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:43.799 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:43.832 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:43.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:43.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:43.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:43.962 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:43.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:44.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:44.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:43 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:41:44.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:43 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:41:44.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:43 smithi071 ceph-mon[30002]: osdmap e931: 12 total, 12 up, 11 in 2024-09-17T16:41:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:44.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:44.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:44.123 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:43 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:41:44.123 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:43 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:41:44.123 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:43 smithi017 ceph-mon[27571]: osdmap e931: 12 total, 12 up, 11 in 2024-09-17T16:41:44.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:43 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:41:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:43 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:41:44.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:43 smithi173 ceph-mon[30530]: osdmap e931: 12 total, 12 up, 11 in 2024-09-17T16:41:44.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:41:44.188 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:44.231 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:44.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:44.333 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:44.364 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:44.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:44.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:44.466 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:44.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:44.512 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:44.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:44.524 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:44.614 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:44.648 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:44.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:44.750 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:44.783 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:44.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:44.874 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T16:41:44.874 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:41:44.884 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:44.920 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:44.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1530064710' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:41:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:41:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:44 smithi017 ceph-mon[27571]: pgmap v2058: 37 pgs: 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 24 active+clean; 3.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 668 op/s; 9380/190248 objects degraded (4.930%); 21418/190248 objects misplaced (11.258%); 24 MiB/s, 381 objects/s recovering 2024-09-17T16:41:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1566911103' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:41:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:44 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:41:44.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:44 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 9772/194709 objects degraded (5.019%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:45.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:45.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:45.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1530064710' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:41:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:41:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:44 smithi173 ceph-mon[30530]: pgmap v2058: 37 pgs: 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 24 active+clean; 3.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 668 op/s; 9380/190248 objects degraded (4.930%); 21418/190248 objects misplaced (11.258%); 24 MiB/s, 381 objects/s recovering 2024-09-17T16:41:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1566911103' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:41:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:44 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:41:45.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:44 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 9772/194709 objects degraded (5.019%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:45.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:45.197 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:45.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:45.299 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1530064710' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:41:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:41:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:44 smithi071 ceph-mon[30002]: pgmap v2058: 37 pgs: 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 24 active+clean; 3.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 668 op/s; 9380/190248 objects degraded (4.930%); 21418/190248 objects misplaced (11.258%); 24 MiB/s, 381 objects/s recovering 2024-09-17T16:41:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1566911103' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:41:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:44 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:41:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:44 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 9772/194709 objects degraded (5.019%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:41:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:45.337 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:45.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:45.438 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:45.472 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:45.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:45.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:45.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:45.609 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:45.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:45.711 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:45.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:45.746 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:45.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:45.797 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:41:45.798 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:41:45.832 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:41:45.833 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:41:45.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:45.864 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:41:45.865 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:41:45.877 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:45.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:45.888 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:41:45.888 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:41:45.909 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:41:45.910 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:41:45.910 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:45 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:41:45.910 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:45 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:41:45.910 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:45 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:41:45.910 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:45 smithi017 ceph-mon[27571]: osdmap e932: 12 total, 12 up, 11 in 2024-09-17T16:41:45.910 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:45 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3635327030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:41:45.949 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:41:45.950 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:41:45.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:45.983 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:41:45.983 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:41:46.016 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:46.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:46.018 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:41:46.018 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:41:46.047 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:41:46.047 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:41:46.079 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:41:46.079 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:41:46.115 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:41:46.115 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:41:46.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:46.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:46.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:45 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:41:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:45 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:41:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:45 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:41:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:45 smithi173 ceph-mon[30530]: osdmap e932: 12 total, 12 up, 11 in 2024-09-17T16:41:46.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:45 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3635327030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:41:46.146 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:41:46.146 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:41:46.174 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:45 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:41:46.174 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:45 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:41:46.175 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:45 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:41:46.175 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:45 smithi071 ceph-mon[30002]: osdmap e932: 12 total, 12 up, 11 in 2024-09-17T16:41:46.175 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:45 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3635327030' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:41:46.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:46.266 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:46.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:46.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:46.393 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:46.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:46.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:46.520 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:46.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:46.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:46.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:46.659 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:46.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:46.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:46.791 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:46.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:46.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:46.893 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:46.929 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:46.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:47.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:47.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:47.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:47.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:46 smithi173 ceph-mon[30530]: pgmap v2060: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 3.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 571 op/s; 9380/185676 objects degraded (5.052%); 19889/185676 objects misplaced (10.712%); 21 MiB/s, 379 objects/s recovering 2024-09-17T16:41:47.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:47.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:47.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:47.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:46 smithi017 ceph-mon[27571]: pgmap v2060: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 3.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 571 op/s; 9380/185676 objects degraded (5.052%); 19889/185676 objects misplaced (10.712%); 21 MiB/s, 379 objects/s recovering 2024-09-17T16:41:47.282 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:46 smithi071 ceph-mon[30002]: pgmap v2060: 37 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 3.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 571 op/s; 9380/185676 objects degraded (5.052%); 19889/185676 objects misplaced (10.712%); 21 MiB/s, 379 objects/s recovering 2024-09-17T16:41:47.285 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:47.315 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:47.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:47.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:47.417 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:47.450 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:47.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:47.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:47.552 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:47.584 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:47.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:47.676 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-17T16:41:47.687 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:47.751 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:47.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:47.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:47.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:47.892 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:47.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:47.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:48.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:48.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:48.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:48.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:48.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:48.229 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:48.245 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:48.274 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:48.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:48.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:48.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:48.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:48.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:48.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:48.521 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:48.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:48.623 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:48.638 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:41:48.652 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:48.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:48.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:48.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:48.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:48.777 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:48.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:48.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:48.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:48 smithi017 ceph-mon[27571]: osdmap e933: 12 total, 12 up, 11 in 2024-09-17T16:41:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:48 smithi017 ceph-mon[27571]: pgmap v2062: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 27 active+clean; 3.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 4488/181194 objects degraded (2.477%); 21320/181194 objects misplaced (11.766%); 26 MiB/s, 458 objects/s recovering 2024-09-17T16:41:49.010 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:41:49.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:49.094 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:49.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:48 smithi173 ceph-mon[30530]: osdmap e933: 12 total, 12 up, 11 in 2024-09-17T16:41:49.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:48 smithi173 ceph-mon[30530]: pgmap v2062: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 27 active+clean; 3.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 4488/181194 objects degraded (2.477%); 21320/181194 objects misplaced (11.766%); 26 MiB/s, 458 objects/s recovering 2024-09-17T16:41:49.196 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:49.233 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:49.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:49.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:49.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:48 smithi071 ceph-mon[30002]: osdmap e933: 12 total, 12 up, 11 in 2024-09-17T16:41:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:48 smithi071 ceph-mon[30002]: pgmap v2062: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 27 active+clean; 3.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 4488/181194 objects degraded (2.477%); 21320/181194 objects misplaced (11.766%); 26 MiB/s, 458 objects/s recovering 2024-09-17T16:41:49.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:49.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:49.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:49.458 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:49.488 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:49.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:49.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:49.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:49.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:49.619 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:49.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:49.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:49.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:49.816 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.8 to 1 (10000) 2024-09-17T16:41:49.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:49.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:49.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:49.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:50.003 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:50.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:50.106 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:50.135 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:50.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:50.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:50.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:49 smithi173 ceph-mon[30530]: osdmap e934: 12 total, 12 up, 11 in 2024-09-17T16:41:50.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4096475251' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:41:50.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:49 smithi173 ceph-mon[30530]: 9.19 scrub starts 2024-09-17T16:41:50.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:49 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 4488/181194 objects degraded (2.477%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:41:50.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4096475251' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:41:50.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:49 smithi173 ceph-mon[30530]: osdmap e935: 12 total, 12 up, 11 in 2024-09-17T16:41:50.170 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:41:50.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:49 smithi017 ceph-mon[27571]: osdmap e934: 12 total, 12 up, 11 in 2024-09-17T16:41:50.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4096475251' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:41:50.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:49 smithi017 ceph-mon[27571]: 9.19 scrub starts 2024-09-17T16:41:50.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:49 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 4488/181194 objects degraded (2.477%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:41:50.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4096475251' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:41:50.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:49 smithi017 ceph-mon[27571]: osdmap e935: 12 total, 12 up, 11 in 2024-09-17T16:41:50.237 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:50.276 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:50.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:50.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:49 smithi071 ceph-mon[30002]: osdmap e934: 12 total, 12 up, 11 in 2024-09-17T16:41:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4096475251' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:41:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:49 smithi071 ceph-mon[30002]: 9.19 scrub starts 2024-09-17T16:41:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:49 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 4488/181194 objects degraded (2.477%), 1 pg degraded (PG_DEGRADED) 2024-09-17T16:41:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4096475251' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:41:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:49 smithi071 ceph-mon[30002]: osdmap e935: 12 total, 12 up, 11 in 2024-09-17T16:41:50.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:50.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:50.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:50.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:50.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:50.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:50.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:50.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:50.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:50.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:50.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:50.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:50.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:50.887 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:50.909 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:50.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:51.011 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:51.032 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:51.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:51.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:51.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:50 smithi173 ceph-mon[30530]: pgmap v2065: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 27 active+clean; 3.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 828 op/s; 4488/176436 objects degraded (2.544%); 19799/176436 objects misplaced (11.222%); 29 MiB/s, 514 objects/s recovering 2024-09-17T16:41:51.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:50 smithi173 ceph-mon[30530]: 9.19 scrub ok 2024-09-17T16:41:51.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:51.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:51.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:50 smithi017 ceph-mon[27571]: pgmap v2065: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 27 active+clean; 3.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 828 op/s; 4488/176436 objects degraded (2.544%); 19799/176436 objects misplaced (11.222%); 29 MiB/s, 514 objects/s recovering 2024-09-17T16:41:51.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:50 smithi017 ceph-mon[27571]: 9.19 scrub ok 2024-09-17T16:41:51.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:51.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:51.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:50 smithi071 ceph-mon[30002]: pgmap v2065: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+remapped, 27 active+clean; 3.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 828 op/s; 4488/176436 objects degraded (2.544%); 19799/176436 objects misplaced (11.222%); 29 MiB/s, 514 objects/s recovering 2024-09-17T16:41:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:50 smithi071 ceph-mon[30002]: 9.19 scrub ok 2024-09-17T16:41:51.381 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:51.413 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:51.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:51.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:51.545 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:51.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:51.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:51.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:51.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:51.777 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:51.815 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:51.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:51.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:51.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:51.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:52.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:52.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:52.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:52.165 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:52.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:52.189 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:52.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:52.291 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:52.313 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:52.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:52.415 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:52.446 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:52.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:52.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:52.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:52.582 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:52.683 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:52.714 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:52.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:52.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:52.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:52.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:52.944 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:52.979 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:52.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:53.037 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:41:53.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:53.127 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:53.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:53.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:53.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:53.264 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:53.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:53.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:53 smithi071 ceph-mon[30002]: pgmap v2066: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 4488/175017 objects degraded (2.564%); 18885/175017 objects misplaced (10.790%) 2024-09-17T16:41:53.365 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:53.409 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:53.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:53.436 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:41:53.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:53 smithi017 ceph-mon[27571]: pgmap v2066: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 4488/175017 objects degraded (2.564%); 18885/175017 objects misplaced (10.790%) 2024-09-17T16:41:53.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:53.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:53.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:53.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:53.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:53 smithi173 ceph-mon[30530]: pgmap v2066: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 4488/175017 objects degraded (2.564%); 18885/175017 objects misplaced (10.790%) 2024-09-17T16:41:53.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:53.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:53.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:53.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:53.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:53.905 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:53.928 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:53.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:54.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:54.060 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:54.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:54.162 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:54.193 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:54.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:54.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:54.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:54.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:54.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:54.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:54.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:54.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:54.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:54.588 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:54.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:54.690 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:54.714 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:54.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:54.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:54.845 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:54.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:54.947 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:54.970 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:54.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:55.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:55.107 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:55.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:55.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:55.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:55.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:55.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:55.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:55.361 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:55.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:55.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:55.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:55.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:55 smithi017 ceph-mon[27571]: pgmap v2067: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4322/171528 objects degraded (2.520%); 16793/171528 objects misplaced (9.790%) 2024-09-17T16:41:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:55 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 4322/171528 objects degraded (2.520%), 1 pg degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:41:55.489 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:55 smithi071 ceph-mon[30002]: pgmap v2067: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4322/171528 objects degraded (2.520%); 16793/171528 objects misplaced (9.790%) 2024-09-17T16:41:55.490 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:55 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 4322/171528 objects degraded (2.520%), 1 pg degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:41:55.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:55.620 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:55.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:55 smithi173 ceph-mon[30530]: pgmap v2067: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4322/171528 objects degraded (2.520%); 16793/171528 objects misplaced (9.790%) 2024-09-17T16:41:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:55 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 4322/171528 objects degraded (2.520%), 1 pg degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:41:55.722 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:55.753 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:55.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:55.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:55.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:55.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:55.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:56.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:56.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:56.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:56.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:56.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:56.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:56.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:56.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:56.287 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:56.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:56.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:56.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:56.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:56.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:56 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:41:56.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:56 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:41:56.517 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:56.538 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:56.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:56.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:56 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:41:56.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:56 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:41:56.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:56 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:41:56.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:56 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:41:56.670 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:56.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:56.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:56.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:56.798 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:56.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:56.928 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:56.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:57.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:57.053 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:57.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:57.155 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:41:57.186 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:57.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:41:57.220 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:41:57.220 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:41:57.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:41:57.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:57.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:41:57.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:57.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:57.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:57.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:57 smithi017 ceph-mon[27571]: pgmap v2068: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4322/167043 objects degraded (2.587%); 15343/167043 objects misplaced (9.185%); 104 KiB/s, 0 objects/s recovering 2024-09-17T16:41:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:57 smithi017 ceph-mon[27571]: osdmap e936: 12 total, 12 up, 11 in 2024-09-17T16:41:57.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:57 smithi071 ceph-mon[30002]: pgmap v2068: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4322/167043 objects degraded (2.587%); 15343/167043 objects misplaced (9.185%); 104 KiB/s, 0 objects/s recovering 2024-09-17T16:41:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:57 smithi071 ceph-mon[30002]: osdmap e936: 12 total, 12 up, 11 in 2024-09-17T16:41:57.582 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:57.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:57.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:57 smithi173 ceph-mon[30530]: pgmap v2068: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4322/167043 objects degraded (2.587%); 15343/167043 objects misplaced (9.185%); 104 KiB/s, 0 objects/s recovering 2024-09-17T16:41:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:57 smithi173 ceph-mon[30530]: osdmap e936: 12 total, 12 up, 11 in 2024-09-17T16:41:57.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:41:57.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:57.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:41:57.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:57.831 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:57.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:57.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:57.933 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:57.966 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:57.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:58.069 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:41:58.097 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:58.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:41:58.199 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:58.219 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:58.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:58.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:41:58.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:58.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:41:58.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:58.492 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:58.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:58.595 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:41:58.630 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:58.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:41:58.732 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:41:58.762 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:58.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:58.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:41:58.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:58.906 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:58.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:58.930 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:41:59.008 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:41:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:59.056 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:59.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:41:59.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:59.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:59.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:59.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:59.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:59.327 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:59.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:59.377 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:41:59.428 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:41:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:59.467 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:59.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:41:59.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:59 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:41:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:59 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:41:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:59 smithi017 ceph-mon[27571]: pgmap v2070: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 4112/162774 objects degraded (2.526%); 13262/162774 objects misplaced (8.147%); 5.2 MiB/s, 67 objects/s recovering 2024-09-17T16:41:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:59 smithi017 ceph-mon[27571]: osdmap e937: 12 total, 12 up, 11 in 2024-09-17T16:41:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3773358505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:41:59.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:41:59 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:41:59.508 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:59 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:41:59.508 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:59 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:41:59.508 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:59 smithi071 ceph-mon[30002]: pgmap v2070: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 4112/162774 objects degraded (2.526%); 13262/162774 objects misplaced (8.147%); 5.2 MiB/s, 67 objects/s recovering 2024-09-17T16:41:59.508 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:59 smithi071 ceph-mon[30002]: osdmap e937: 12 total, 12 up, 11 in 2024-09-17T16:41:59.508 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3773358505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:41:59.508 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:41:59 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:41:59.550 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:41:59.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:41:59.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:41:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:59.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:41:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:59 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:41:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:59 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:41:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:59 smithi173 ceph-mon[30530]: pgmap v2070: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 99 op/s; 4112/162774 objects degraded (2.526%); 13262/162774 objects misplaced (8.147%); 5.2 MiB/s, 67 objects/s recovering 2024-09-17T16:41:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:59 smithi173 ceph-mon[30530]: osdmap e937: 12 total, 12 up, 11 in 2024-09-17T16:41:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3773358505' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:41:59.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:41:59 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:41:59.699 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:41:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:59.746 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:41:59.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:41:59.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:41:59.876 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:41:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:41:59.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:41:59.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:00.004 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:00.005 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:00.022 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:00.023 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:00.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:00.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:00.145 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:00.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:00.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:00.268 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:00.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:00.370 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:00.398 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:00.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:00.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:00 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:42:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:00 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:42:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:00 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:42:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3773358505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:00 smithi017 ceph-mon[27571]: osdmap e938: 12 total, 12 up, 11 in 2024-09-17T16:42:00.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:00 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 4112/162774 objects degraded (2.526%), 1 pg degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:00.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:00 smithi017 ceph-mon[27571]: osdmap e939: 12 total, 12 up, 11 in 2024-09-17T16:42:00.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:00.533 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:00.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:00.553 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:00 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:42:00.553 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:00 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:42:00.554 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:00 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:42:00.554 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3773358505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:00.554 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:00 smithi071 ceph-mon[30002]: osdmap e938: 12 total, 12 up, 11 in 2024-09-17T16:42:00.554 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:00 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 4112/162774 objects degraded (2.526%), 1 pg degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:00.554 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:00 smithi071 ceph-mon[30002]: osdmap e939: 12 total, 12 up, 11 in 2024-09-17T16:42:00.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:00.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:00 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:42:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:00 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:42:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:00 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:42:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3773358505' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:00 smithi173 ceph-mon[30530]: osdmap e938: 12 total, 12 up, 11 in 2024-09-17T16:42:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:00 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 4112/162774 objects degraded (2.526%), 1 pg degraded, 9 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:00.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:00 smithi173 ceph-mon[30530]: osdmap e939: 12 total, 12 up, 11 in 2024-09-17T16:42:00.663 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:00.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:00.749 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:42:00.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:00.792 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:00.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:00.895 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:00.921 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:00.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:01.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:01.055 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:01.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:01.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:01.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:01.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:01.188 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:42:01.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:01.309 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:01.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:01.411 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:01.441 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:01.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:01.502 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:01 smithi071 ceph-mon[30002]: pgmap v2074: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 777 op/s; 4112/157710 objects degraded (2.607%); 11915/157710 objects misplaced (7.555%); 25 MiB/s, 499 objects/s recovering 2024-09-17T16:42:01.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:01.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:01.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:01.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:01 smithi173 ceph-mon[30530]: pgmap v2074: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 777 op/s; 4112/157710 objects degraded (2.607%); 11915/157710 objects misplaced (7.555%); 25 MiB/s, 499 objects/s recovering 2024-09-17T16:42:01.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:01.717 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:01.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:01.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:01 smithi017 ceph-mon[27571]: pgmap v2074: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 777 op/s; 4112/157710 objects degraded (2.607%); 11915/157710 objects misplaced (7.555%); 25 MiB/s, 499 objects/s recovering 2024-09-17T16:42:01.777 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:01.777 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:01.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:01.850 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:01.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:01.952 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:01.979 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:01.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:02.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:02.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:02.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:02.209 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:02.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:02.238 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:02.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:02.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:02.368 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:02.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:02.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:02.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:42:02.496 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:02.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:02.599 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:02.625 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:02.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:02.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:02.754 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:02.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:02.813 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:42:02.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:02.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:02.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:02.979 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:02.988 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-17T16:42:03.001 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:03.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:03.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:03.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:03.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:03.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:03.227 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:03.249 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:03.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:03.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:03.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:03.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:03.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:03.504 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:03.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:03.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:03 smithi071 ceph-mon[30002]: pgmap v2075: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 614 op/s; 4112/156090 objects degraded (2.634%); 10821/156090 objects misplaced (6.933%); 22 MiB/s, 414 objects/s recovering 2024-09-17T16:42:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:03 smithi071 ceph-mon[30002]: osdmap e940: 12 total, 12 up, 11 in 2024-09-17T16:42:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:42:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:42:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:03 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:42:03.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:03.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:03.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:03.630 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:03.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:03 smithi173 ceph-mon[30530]: pgmap v2075: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 614 op/s; 4112/156090 objects degraded (2.634%); 10821/156090 objects misplaced (6.933%); 22 MiB/s, 414 objects/s recovering 2024-09-17T16:42:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:03 smithi173 ceph-mon[30530]: osdmap e940: 12 total, 12 up, 11 in 2024-09-17T16:42:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:42:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:42:03.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:03 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:42:03.731 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:03 smithi017 ceph-mon[27571]: pgmap v2075: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 614 op/s; 4112/156090 objects degraded (2.634%); 10821/156090 objects misplaced (6.933%); 22 MiB/s, 414 objects/s recovering 2024-09-17T16:42:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:03 smithi017 ceph-mon[27571]: osdmap e940: 12 total, 12 up, 11 in 2024-09-17T16:42:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:42:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:42:03.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:03 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:42:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:03.763 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:03.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:03.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:03.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:03.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:03.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:03.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:04.009 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:04.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:04.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:04.134 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:04.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:04.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:04.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:04.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:04.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:04.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:04.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:04.487 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:04.526 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:04.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:04.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:04 smithi071 ceph-mon[30002]: osdmap e941: 12 total, 12 up, 11 in 2024-09-17T16:42:04.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:04 smithi071 ceph-mon[30002]: 9.18 scrub starts 2024-09-17T16:42:04.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:04 smithi071 ceph-mon[30002]: 9.18 scrub ok 2024-09-17T16:42:04.628 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:04.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:04 smithi173 ceph-mon[30530]: osdmap e941: 12 total, 12 up, 11 in 2024-09-17T16:42:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:04 smithi173 ceph-mon[30530]: 9.18 scrub starts 2024-09-17T16:42:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:04 smithi173 ceph-mon[30530]: 9.18 scrub ok 2024-09-17T16:42:04.649 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:04.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:04 smithi017 ceph-mon[27571]: osdmap e941: 12 total, 12 up, 11 in 2024-09-17T16:42:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:04 smithi017 ceph-mon[27571]: 9.18 scrub starts 2024-09-17T16:42:04.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:04 smithi017 ceph-mon[27571]: 9.18 scrub ok 2024-09-17T16:42:04.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:04.784 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:04.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:04.886 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:04.916 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:04.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:05.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:05.047 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:05.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:05.149 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:05.181 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:05.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:05.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:05.316 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:05.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:05.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:05.444 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:05.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:05.546 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:05 smithi071 ceph-mon[30002]: pgmap v2078: 37 pgs: 1 creating+peering, 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 3.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 691 op/s; 12030/151635 objects misplaced (7.934%); 20 MiB/s, 576 objects/s recovering 2024-09-17T16:42:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:05 smithi071 ceph-mon[30002]: osdmap e942: 12 total, 12 up, 11 in 2024-09-17T16:42:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:05 smithi071 ceph-mon[30002]: 9.14 scrub starts 2024-09-17T16:42:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:05 smithi071 ceph-mon[30002]: 9.9 scrub starts 2024-09-17T16:42:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:05 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:42:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:05 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:42:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:05 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 8 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:05.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:05 smithi071 ceph-mon[30002]: osdmap e943: 12 total, 12 up, 11 in 2024-09-17T16:42:05.570 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:05.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:05 smithi173 ceph-mon[30530]: pgmap v2078: 37 pgs: 1 creating+peering, 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 3.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 691 op/s; 12030/151635 objects misplaced (7.934%); 20 MiB/s, 576 objects/s recovering 2024-09-17T16:42:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:05 smithi173 ceph-mon[30530]: osdmap e942: 12 total, 12 up, 11 in 2024-09-17T16:42:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:05 smithi173 ceph-mon[30530]: 9.14 scrub starts 2024-09-17T16:42:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:05 smithi173 ceph-mon[30530]: 9.9 scrub starts 2024-09-17T16:42:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:05 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:42:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:05 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:42:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:05 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 8 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:05.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:05 smithi173 ceph-mon[30530]: osdmap e943: 12 total, 12 up, 11 in 2024-09-17T16:42:05.672 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:05.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:05.703 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:05.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:05 smithi017 ceph-mon[27571]: pgmap v2078: 37 pgs: 1 creating+peering, 1 active+undersized+remapped, 7 active+recovering+undersized+remapped, 28 active+clean; 3.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 691 op/s; 12030/151635 objects misplaced (7.934%); 20 MiB/s, 576 objects/s recovering 2024-09-17T16:42:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:05 smithi017 ceph-mon[27571]: osdmap e942: 12 total, 12 up, 11 in 2024-09-17T16:42:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:05 smithi017 ceph-mon[27571]: 9.14 scrub starts 2024-09-17T16:42:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:05 smithi017 ceph-mon[27571]: 9.9 scrub starts 2024-09-17T16:42:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:05 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:42:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:05 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:42:05.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:05 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 8 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:05.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:05 smithi017 ceph-mon[27571]: osdmap e943: 12 total, 12 up, 11 in 2024-09-17T16:42:05.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:05.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:05.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:05.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:05.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:05.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:06.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:06.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:06.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:06.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:06.191 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:06.224 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:06.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:06.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:06.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:06.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:06.459 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:06.491 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:06.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:06 smithi071 ceph-mon[30002]: 9.9 scrub ok 2024-09-17T16:42:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:06 smithi071 ceph-mon[30002]: 9.14 scrub ok 2024-09-17T16:42:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:06 smithi071 ceph-mon[30002]: 1.7 scrub starts 2024-09-17T16:42:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:06 smithi071 ceph-mon[30002]: 1.7 scrub ok 2024-09-17T16:42:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:06 smithi071 ceph-mon[30002]: 1.a scrub starts 2024-09-17T16:42:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:06 smithi071 ceph-mon[30002]: 1.a scrub ok 2024-09-17T16:42:06.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:06 smithi071 ceph-mon[30002]: 9.12 scrub starts 2024-09-17T16:42:06.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:06.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:06.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:06.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:06 smithi173 ceph-mon[30530]: 9.9 scrub ok 2024-09-17T16:42:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:06 smithi173 ceph-mon[30530]: 9.14 scrub ok 2024-09-17T16:42:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:06 smithi173 ceph-mon[30530]: 1.7 scrub starts 2024-09-17T16:42:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:06 smithi173 ceph-mon[30530]: 1.7 scrub ok 2024-09-17T16:42:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:06 smithi173 ceph-mon[30530]: 1.a scrub starts 2024-09-17T16:42:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:06 smithi173 ceph-mon[30530]: 1.a scrub ok 2024-09-17T16:42:06.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:06 smithi173 ceph-mon[30530]: 9.12 scrub starts 2024-09-17T16:42:06.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:06 smithi017 ceph-mon[27571]: 9.9 scrub ok 2024-09-17T16:42:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:06 smithi017 ceph-mon[27571]: 9.14 scrub ok 2024-09-17T16:42:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:06 smithi017 ceph-mon[27571]: 1.7 scrub starts 2024-09-17T16:42:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:06 smithi017 ceph-mon[27571]: 1.7 scrub ok 2024-09-17T16:42:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:06 smithi017 ceph-mon[27571]: 1.a scrub starts 2024-09-17T16:42:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:06 smithi017 ceph-mon[27571]: 1.a scrub ok 2024-09-17T16:42:06.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:06 smithi017 ceph-mon[27571]: 9.12 scrub starts 2024-09-17T16:42:06.765 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:06.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:06.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:06.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:06.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:06.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:06.999 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:07.029 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:07.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:07.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:07.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:07.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:07.260 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:07.287 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:07 smithi071 ceph-mon[30002]: 9.12 scrub ok 2024-09-17T16:42:07.296 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:07.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:07.398 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:07.420 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:07.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:07.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:07.552 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:07.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:07 smithi071 ceph-mon[30002]: pgmap v2081: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 2/147870 objects degraded (0.001%); 11470/147870 objects misplaced (7.757%); 14 MiB/s, 579 objects/s recovering 2024-09-17T16:42:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:07 smithi071 ceph-mon[30002]: 1.3 scrub starts 2024-09-17T16:42:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:07 smithi071 ceph-mon[30002]: 1.3 scrub ok 2024-09-17T16:42:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:07 smithi071 ceph-mon[30002]: 1.9 scrub starts 2024-09-17T16:42:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:07 smithi071 ceph-mon[30002]: 1.9 scrub ok 2024-09-17T16:42:07.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:07 smithi071 ceph-mon[30002]: 1.8 scrub starts 2024-09-17T16:42:07.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:07 smithi071 ceph-mon[30002]: 1.8 scrub ok 2024-09-17T16:42:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:07 smithi173 ceph-mon[30530]: 9.12 scrub ok 2024-09-17T16:42:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:07 smithi173 ceph-mon[30530]: pgmap v2081: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 2/147870 objects degraded (0.001%); 11470/147870 objects misplaced (7.757%); 14 MiB/s, 579 objects/s recovering 2024-09-17T16:42:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:07 smithi173 ceph-mon[30530]: 1.3 scrub starts 2024-09-17T16:42:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:07 smithi173 ceph-mon[30530]: 1.3 scrub ok 2024-09-17T16:42:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:07 smithi173 ceph-mon[30530]: 1.9 scrub starts 2024-09-17T16:42:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:07 smithi173 ceph-mon[30530]: 1.9 scrub ok 2024-09-17T16:42:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:07 smithi173 ceph-mon[30530]: 1.8 scrub starts 2024-09-17T16:42:07.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:07 smithi173 ceph-mon[30530]: 1.8 scrub ok 2024-09-17T16:42:07.653 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:07.679 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:07.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:07 smithi017 ceph-mon[27571]: 9.12 scrub ok 2024-09-17T16:42:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:07 smithi017 ceph-mon[27571]: pgmap v2081: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 1 active+undersized+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 682 op/s; 2/147870 objects degraded (0.001%); 11470/147870 objects misplaced (7.757%); 14 MiB/s, 579 objects/s recovering 2024-09-17T16:42:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:07 smithi017 ceph-mon[27571]: 1.3 scrub starts 2024-09-17T16:42:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:07 smithi017 ceph-mon[27571]: 1.3 scrub ok 2024-09-17T16:42:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:07 smithi017 ceph-mon[27571]: 1.9 scrub starts 2024-09-17T16:42:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:07 smithi017 ceph-mon[27571]: 1.9 scrub ok 2024-09-17T16:42:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:07 smithi017 ceph-mon[27571]: 1.8 scrub starts 2024-09-17T16:42:07.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:07 smithi017 ceph-mon[27571]: 1.8 scrub ok 2024-09-17T16:42:07.781 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:07.804 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:07.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:07.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:07.881 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:07.907 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:07.929 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:07.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:07.930 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:08.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:08.058 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:08.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:08.084 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:08.085 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:08.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:08.188 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:08.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:08.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:08.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:08.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:08.418 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:08.455 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:08.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:08 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:42:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:08 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:42:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:08 smithi017 ceph-mon[27571]: osdmap e944: 12 total, 12 up, 11 in 2024-09-17T16:42:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:08 smithi017 ceph-mon[27571]: 1.6 scrub starts 2024-09-17T16:42:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:08 smithi017 ceph-mon[27571]: 1.6 scrub ok 2024-09-17T16:42:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:08 smithi017 ceph-mon[27571]: 1.2 scrub starts 2024-09-17T16:42:08.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:08 smithi017 ceph-mon[27571]: 1.2 scrub ok 2024-09-17T16:42:08.556 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:08.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:08 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:42:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:08 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:42:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:08 smithi071 ceph-mon[30002]: osdmap e944: 12 total, 12 up, 11 in 2024-09-17T16:42:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:08 smithi071 ceph-mon[30002]: 1.6 scrub starts 2024-09-17T16:42:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:08 smithi071 ceph-mon[30002]: 1.6 scrub ok 2024-09-17T16:42:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:08 smithi071 ceph-mon[30002]: 1.2 scrub starts 2024-09-17T16:42:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:08 smithi071 ceph-mon[30002]: 1.2 scrub ok 2024-09-17T16:42:08.587 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:08.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:08 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:42:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:08 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:42:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:08 smithi173 ceph-mon[30530]: osdmap e944: 12 total, 12 up, 11 in 2024-09-17T16:42:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:08 smithi173 ceph-mon[30530]: 1.6 scrub starts 2024-09-17T16:42:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:08 smithi173 ceph-mon[30530]: 1.6 scrub ok 2024-09-17T16:42:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:08 smithi173 ceph-mon[30530]: 1.2 scrub starts 2024-09-17T16:42:08.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:08 smithi173 ceph-mon[30530]: 1.2 scrub ok 2024-09-17T16:42:08.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:08.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:08.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:08.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:08.843 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:08.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:08.945 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:08.969 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:08.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:09.070 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:09.102 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:09.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:09.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:09.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:09.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:09.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:09.352 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:09.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:09.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:09.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:09.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:09.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:09 smithi017 ceph-mon[27571]: pgmap v2083: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 31 active+clean; 2.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 2/143706 objects degraded (0.001%); 9888/143706 objects misplaced (6.881%) 2024-09-17T16:42:09.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:09 smithi017 ceph-mon[27571]: osdmap e945: 12 total, 12 up, 11 in 2024-09-17T16:42:09.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:09 smithi017 ceph-mon[27571]: 9.13 scrub starts 2024-09-17T16:42:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:09 smithi071 ceph-mon[30002]: pgmap v2083: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 31 active+clean; 2.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 2/143706 objects degraded (0.001%); 9888/143706 objects misplaced (6.881%) 2024-09-17T16:42:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:09 smithi071 ceph-mon[30002]: osdmap e945: 12 total, 12 up, 11 in 2024-09-17T16:42:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:09 smithi071 ceph-mon[30002]: 9.13 scrub starts 2024-09-17T16:42:09.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:09.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:09.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:09 smithi173 ceph-mon[30530]: pgmap v2083: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 31 active+clean; 2.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 2/143706 objects degraded (0.001%); 9888/143706 objects misplaced (6.881%) 2024-09-17T16:42:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:09 smithi173 ceph-mon[30530]: osdmap e945: 12 total, 12 up, 11 in 2024-09-17T16:42:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:09 smithi173 ceph-mon[30530]: 9.13 scrub starts 2024-09-17T16:42:09.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:09.750 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:09.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:09.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:42:09.852 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:09.885 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:09.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:09.986 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:10.033 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:10.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:10.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:10.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:10.162 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:10.263 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:10.285 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:10.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:10.387 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:10.422 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:10.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:10.524 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:10.552 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:10.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:10.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:10 smithi071 ceph-mon[30002]: 9.13 scrub ok 2024-09-17T16:42:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:10 smithi071 ceph-mon[30002]: 1.4 scrub starts 2024-09-17T16:42:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:10 smithi071 ceph-mon[30002]: 1.4 scrub ok 2024-09-17T16:42:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:10 smithi071 ceph-mon[30002]: 1.5 scrub starts 2024-09-17T16:42:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:10 smithi071 ceph-mon[30002]: 1.5 scrub ok 2024-09-17T16:42:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:10 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2/143706 objects degraded (0.001%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:10 smithi173 ceph-mon[30530]: 9.13 scrub ok 2024-09-17T16:42:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:10 smithi173 ceph-mon[30530]: 1.4 scrub starts 2024-09-17T16:42:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:10 smithi173 ceph-mon[30530]: 1.4 scrub ok 2024-09-17T16:42:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:10 smithi173 ceph-mon[30530]: 1.5 scrub starts 2024-09-17T16:42:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:10 smithi173 ceph-mon[30530]: 1.5 scrub ok 2024-09-17T16:42:10.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:10 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2/143706 objects degraded (0.001%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:10.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:10.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:10.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:10 smithi017 ceph-mon[27571]: 9.13 scrub ok 2024-09-17T16:42:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:10 smithi017 ceph-mon[27571]: 1.4 scrub starts 2024-09-17T16:42:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:10 smithi017 ceph-mon[27571]: 1.4 scrub ok 2024-09-17T16:42:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:10 smithi017 ceph-mon[27571]: 1.5 scrub starts 2024-09-17T16:42:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:10 smithi017 ceph-mon[27571]: 1.5 scrub ok 2024-09-17T16:42:10.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:10 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2/143706 objects degraded (0.001%), 2 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:10.781 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:10.818 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:10.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:10.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:10.921 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:10.951 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:10.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:11.053 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:11.083 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:11.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:11.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:11.185 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:11.208 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:11.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:11.310 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:11.341 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:11.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:11.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:11.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:11 smithi017 ceph-mon[27571]: pgmap v2085: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 31 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2/138723 objects degraded (0.001%); 9464/138723 objects misplaced (6.822%) 2024-09-17T16:42:11.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:11 smithi017 ceph-mon[27571]: 1.1 scrub starts 2024-09-17T16:42:11.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:11 smithi017 ceph-mon[27571]: 1.1 scrub ok 2024-09-17T16:42:11.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2631580203' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:11.464 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:11 smithi071 ceph-mon[30002]: pgmap v2085: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 31 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2/138723 objects degraded (0.001%); 9464/138723 objects misplaced (6.822%) 2024-09-17T16:42:11.464 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:11 smithi071 ceph-mon[30002]: 1.1 scrub starts 2024-09-17T16:42:11.464 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:11 smithi071 ceph-mon[30002]: 1.1 scrub ok 2024-09-17T16:42:11.464 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2631580203' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:11.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:11.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:11.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:11.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:11.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:11 smithi173 ceph-mon[30530]: pgmap v2085: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 31 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2/138723 objects degraded (0.001%); 9464/138723 objects misplaced (6.822%) 2024-09-17T16:42:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:11 smithi173 ceph-mon[30530]: 1.1 scrub starts 2024-09-17T16:42:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:11 smithi173 ceph-mon[30530]: 1.1 scrub ok 2024-09-17T16:42:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2631580203' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:11.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:42:11.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:11.724 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:11.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:11.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:11.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:11.854 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:11.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:11.896 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:42:11.897 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:42:11.897 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:42:11.897 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:42:11.897 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:42:11.897 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:42:11.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:42:11.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:42:11.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:42:11.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:42:11.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:42:11.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:42:11.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:42:11.898 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:42:11.898 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:42:11.898 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:42:11.899 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:42:11.899 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:42:11.899 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:42:11.899 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:42:11.899 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:42:11.899 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:42:11.899 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:42:11.899 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:42:11.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:11.978 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:11.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:11.999 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:42:12.080 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:12.113 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:12.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:12.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:12.238 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:12.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:12.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:12.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:12.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:12.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:12.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:12.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:12 smithi071 ceph-mon[30002]: 9.a scrub starts 2024-09-17T16:42:12.595 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:12.625 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:12.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:12.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:12 smithi173 ceph-mon[30530]: 9.a scrub starts 2024-09-17T16:42:12.727 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:12 smithi017 ceph-mon[27571]: 9.a scrub starts 2024-09-17T16:42:12.756 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:12.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:12.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:12.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:12.888 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:12.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:12.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:12.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:13.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:13.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:13.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:13.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:13.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:13.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:13.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:13.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:13.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:13.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:13.371 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:13.406 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:13.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:13.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:13.533 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:13.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:13 smithi071 ceph-mon[30002]: 9.a scrub ok 2024-09-17T16:42:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:13 smithi071 ceph-mon[30002]: pgmap v2086: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 32 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 8687/136467 objects misplaced (6.366%) 2024-09-17T16:42:13.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:13.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:13 smithi173 ceph-mon[30530]: 9.a scrub ok 2024-09-17T16:42:13.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:13 smithi173 ceph-mon[30530]: pgmap v2086: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 32 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 8687/136467 objects misplaced (6.366%) 2024-09-17T16:42:13.652 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:42:13.652 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:42:13.652 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:42:13.652 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:42:13.652 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.652 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:170216 7.11 7:89b9c9a3:::benchmark_data_smithi173_116396_object74616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:42:13.652 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 528.10880100899999, 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.387146404, 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170216 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.653 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538979+0000", 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538980+0000", 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3030000000000001e-06 2024-09-17T16:42:13.654 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538986+0000", 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6579999999999999e-06 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.538998+0000", 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2903e-05 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.655 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539004+0000", 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9540000000000001e-06 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539051+0000", 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7067999999999997e-05 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539144+0000", 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2521000000000004e-05 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.539219+0000", 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4791999999999998e-05 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541123+0000", 2024-09-17T16:42:13.657 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019039650000000001 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541689+0000", 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00056619600000000004 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925945+0000", 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38425619700000002 2024-09-17T16:42:13.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926009+0000", 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3516999999999995e-05 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926125+0000", 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011633000000000001 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.659 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:170216 7.11 e712/703 ondisk, result = 0)", 2024-09-17T16:42:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:42:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 528.10640620599997, 2024-09-17T16:42:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38475827200000001, 2024-09-17T16:42:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.660 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:42:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.661 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.662 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541373+0000", 2024-09-17T16:42:13.662 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.662 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541374+0000", 2024-09-17T16:42:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:42:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.541379+0000", 2024-09-17T16:42:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6210000000000003e-06 2024-09-17T16:42:13.663 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.923614+0000", 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38223536499999999 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.923618+0000", 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7629999999999998e-06 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.664 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925930+0000", 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023126259999999999 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925939+0000", 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0070000000000003e-06 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926132+0000", 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019219300000000001 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.665 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:170289 7.11 7:8ff6a0b8:::benchmark_data_smithi173_116396_object74689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 528.02347239300002, 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30203754100000002, 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:13.666 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170289 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624307+0000", 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.667 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624309+0000", 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.136e-06 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624319+0000", 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9170000000000001e-06 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.668 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624348+0000", 2024-09-17T16:42:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8368999999999999e-05 2024-09-17T16:42:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624362+0000", 2024-09-17T16:42:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4222999999999999e-05 2024-09-17T16:42:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.669 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:13.671 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624491+0000", 2024-09-17T16:42:13.671 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000129262 2024-09-17T16:42:13.671 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.671 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624698+0000", 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000206546 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.624844+0000", 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014652400000000001 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.626781+0000", 2024-09-17T16:42:13.672 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019372759999999999 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627901+0000", 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011196260000000001 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926178+0000", 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29827702699999997 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.673 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926231+0000", 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2892000000000001e-05 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926345+0000", 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000113743 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.674 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:170289 7.11 e712/703 ondisk, result = 0)", 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 528.02076823100003, 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29933906100000002, 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627011+0000", 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627012+0000", 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.676 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.627016+0000", 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9809999999999997e-06 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925949+0000", 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29893330699999998 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.925958+0000", 2024-09-17T16:42:13.677 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926164+0000", 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000205942 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926171+0000", 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9500000000000001e-06 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.678 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:25.926350+0000", 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017893299999999999 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDelete(7.19 e717,677 7:9da3d41f:::benchmark_data_smithi173_116396_object4781:head,673'6297)", 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 517.47184204899997, 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37136110999999999, 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.679 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175938+0000", 2024-09-17T16:42:13.680 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175947+0000", 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3859999999999996e-06 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.175959+0000", 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2269999999999999e-05 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547126+0000", 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.37116651099999998 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547136+0000", 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0241000000000001e-05 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547216+0000", 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.051e-05 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.682 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547227+0000", 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0356999999999999e-05 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:36.547299+0000", 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1835000000000006e-05 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.683 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27088.0:191171 7.11 7:8d50061b:::benchmark_data_smithi173_116396_object95571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 499.21721391599999, 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.44239869199999998, 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27088", 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 191171 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430566+0000", 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430568+0000", 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.897e-06 2024-09-17T16:42:13.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430578+0000", 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0149999999999999e-05 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430616+0000", 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8753e-05 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430622+0000", 2024-09-17T16:42:13.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3569999999999997e-06 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430666+0000", 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3831000000000001e-05 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430740+0000", 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4865999999999998e-05 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,9", 2024-09-17T16:42:13.687 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.430817+0000", 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6490999999999997e-05 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433451+0000", 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0026335489999999998 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433803+0000", 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00035236799999999998 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872847+0000", 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43904401599999998 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872921+0000", 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3867000000000003e-05 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872964+0000", 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3547000000000002e-05 2024-09-17T16:42:13.689 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27088.0:191171 7.11 e724/703 ondisk, result = 0)", 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 499.21418568500002, 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43937563000000002, 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:13.690 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433594+0000", 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433595+0000", 2024-09-17T16:42:13.691 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2900000000000002e-07 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.433602+0000", 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9550000000000003e-06 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872622+0000", 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.43902024000000001 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.692 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872639+0000", 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7516e-05 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872820+0000", 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000180714 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872837+0000", 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7099999999999999e-05 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.693 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:33:54.872970+0000", 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013227599999999999 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(8.12 743/742 [PushOp(8:4d9389a5:::benchmark_data_smithi173_130566_object4465:head, version: 732'276, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:4d9389a5:::benchmark_data_smithi173_130566_object4465:head@732'276, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:29.740078+0000", 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 463.90770151800001, 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32815092099999998, 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.694 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740078+0000", 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740078+0000", 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.695 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740079+0000", 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0900000000000001e-07 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.740180+0000", 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010132900000000001 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067836+0000", 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32765588099999998 2024-09-17T16:42:13.696 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067848+0000", 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1754e-05 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067920+0000", 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2813999999999998e-05 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.067929+0000", 2024-09-17T16:42:13.697 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1939999999999998e-06 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.068229+0000", 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030034000000000001 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:78112 8.12 8:4c421870:::benchmark_data_smithi173_130566_object78111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e762)", 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:26.226347+0000", 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 407.42143256600002, 2024-09-17T16:42:13.698 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.305366213, 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 78112 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226347+0000", 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.699 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226347+0000", 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226348+0000", 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.435e-06 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.700 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226401+0000", 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2386000000000003e-05 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226415+0000", 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4095e-05 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226421+0000", 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1240000000000002e-06 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.701 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226475+0000", 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4376999999999998e-05 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226606+0000", 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013063500000000001 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,7", 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.226759+0000", 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015287100000000001 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.702 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228063+0000", 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013040199999999999 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228274+0000", 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000211201 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531553+0000", 2024-09-17T16:42:13.703 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30327892299999998 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531612+0000", 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8836999999999999e-05 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531713+0000", 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000101309 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.704 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:78112 8.12 e762/744 ondisk, result = 0)", 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:26.228645+0000", 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 407.41913446400002, 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30307498500000002, 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.705 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228645+0000", 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228645+0000", 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228646+0000", 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7999999999999995e-07 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.706 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.228649+0000", 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6370000000000001e-06 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531475+0000", 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.302826076 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531482+0000", 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7490000000000001e-06 2024-09-17T16:42:13.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531537+0000", 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4731000000000002e-05 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531547+0000", 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0200000000000001e-05 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.708 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:26.531720+0000", 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000173012 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:153469 8.12 8:4c954a71:::benchmark_data_smithi173_130566_object58494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e821)", 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:43.721465+0000", 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 269.92631458300002, 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38866898999999999, 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.709 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 153469 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721465+0000", 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.710 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721465+0000", 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721467+0000", 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.68e-06 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721473+0000", 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6900000000000003e-06 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721493+0000", 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9811999999999999e-05 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721502+0000", 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1409999999999994e-06 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.712 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721562+0000", 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9406e-05 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721713+0000", 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015121899999999999 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.721848+0000", 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013535399999999999 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.724038+0000", 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00218995 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:43.724638+0000", 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059941900000000002 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.108614+0000", 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38397621199999998 2024-09-17T16:42:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.108657+0000", 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3130999999999998e-05 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.110134+0000", 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014769760000000001 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:154123 8.12 8:4c613ccc:::benchmark_data_smithi173_130566_object59148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e821)", 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:44.579188+0000", 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 269.068591062, 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34086830200000001, 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 154123 2024-09-17T16:42:13.716 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579188+0000", 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579188+0000", 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.717 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579190+0000", 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.905e-06 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579198+0000", 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1040000000000001e-06 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579214+0000", 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6089000000000001e-05 2024-09-17T16:42:13.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579222+0000", 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8550000000000008e-06 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579290+0000", 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7329000000000005e-05 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579469+0000", 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017971400000000001 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,10", 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.579627+0000", 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015795199999999999 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.581796+0000", 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021689919999999998 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.584094+0000", 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022974509999999998 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918612+0000", 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33451803200000002 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918651+0000", 2024-09-17T16:42:13.721 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9332000000000002e-05 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.920057+0000", 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001405547 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:154123 8.12 e821/792 ondisk, result = 0)", 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:44.581915+0000", 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 269.065864207, 2024-09-17T16:42:13.722 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33814677799999998, 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.581915+0000", 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.581915+0000", 2024-09-17T16:42:13.723 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.581916+0000", 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7400000000000003e-07 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.581920+0000", 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9659999999999998e-06 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918559+0000", 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.336639297 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918564+0000", 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5950000000000001e-06 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918600+0000", 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6449999999999998e-05 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.725 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.918607+0000", 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0849999999999999e-06 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:44.920062+0000", 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001454711 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.726 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(9.11 919/915 [PushReplyOp(9:8b4f7364:::benchmark_data_smithi173_150363_object25862:head)])", 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:59.896948+0000", 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 73.750831860999995, 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.406864644, 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:59.896948+0000", 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:59.896948+0000", 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:59.896948+0000", 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8700000000000005e-07 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:59.896953+0000", 2024-09-17T16:42:13.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0019999999999997e-06 2024-09-17T16:42:13.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.729 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.729 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.299632+0000", 2024-09-17T16:42:13.729 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40267869699999997 2024-09-17T16:42:13.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:13.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:13.730 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.730 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.299642+0000", 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8649999999999996e-06 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.299723+0000", 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1200999999999997e-05 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.303812+0000", 2024-09-17T16:42:13.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040891920000000002 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:170673 9.11 9:8dc55899:::benchmark_data_smithi173_150363_object63768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:12.440506+0000", 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.207273372, 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.371210349, 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.732 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170673 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.440506+0000", 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.440506+0000", 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.440508+0000", 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4219999999999999e-06 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.440513+0000", 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3650000000000003e-06 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.440525+0000", 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2012e-05 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.440532+0000", 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3900000000000004e-06 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.440588+0000", 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5902000000000001e-05 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.440720+0000", 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013206000000000001 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.440860+0000", 2024-09-17T16:42:13.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000139954 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.446329+0000", 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0054683140000000002 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.446342+0000", 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3118e-05 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811533+0000", 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.365191243 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811596+0000", 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3085999999999999e-05 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811717+0000", 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012048300000000001 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.738 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27874.0:170673 9.11 e945/915 ondisk, result = 0)", 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:12.443581+0000", 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.204198299, 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.368143045, 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.739 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.443581+0000", 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.443581+0000", 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.443582+0000", 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7899999999999998e-07 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.443586+0000", 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2740000000000001e-06 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808358+0000", 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36477208500000002 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808370+0000", 2024-09-17T16:42:13.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1216000000000001e-05 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811517+0000", 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.003147178 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811526+0000", 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811724+0000", 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019816199999999999 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:170698 9.11 9:88af51f9:::benchmark_data_smithi173_150363_object63793:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:12.468324+0000", 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.179455731, 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.343580618, 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:13.743 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170698 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.468324+0000", 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.468324+0000", 2024-09-17T16:42:13.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.468326+0000", 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9549999999999999e-06 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.468334+0000", 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3180000000000007e-06 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.468353+0000", 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8652e-05 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.468363+0000", 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7690000000000008e-06 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.468424+0000", 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1958000000000005e-05 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.468510+0000", 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5251000000000001e-05 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.468596+0000", 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6293999999999997e-05 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.470824+0000", 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002228346 2024-09-17T16:42:13.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808723+0000", 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33789862700000001 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811770+0000", 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0030473359999999999 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:13.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811803+0000", 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2462000000000002e-05 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811904+0000", 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010165 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27874.0:170698 9.11 e945/915 ondisk, result = 0)", 2024-09-17T16:42:13.749 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:12.471459+0000", 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.176320249, 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33727633699999998, 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.471459+0000", 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.471459+0000", 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.471460+0000", 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.54e-07 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.471464+0000", 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5480000000000002e-06 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.471474+0000", 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0845e-05 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.471479+0000", 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9969999999999998e-06 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808702+0000", 2024-09-17T16:42:13.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33722268399999999 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808716+0000", 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3536000000000001e-05 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808736+0000", 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9973e-05 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27874.0:170698 9.11 e945/915 ondisk, result = 0)", 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:12.471574+0000", 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.1762054100000001, 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34033758800000002, 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.471574+0000", 2024-09-17T16:42:13.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.471574+0000", 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.471575+0000", 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.03e-07 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.471577+0000", 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5050000000000002e-06 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808469+0000", 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33689179600000002 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808474+0000", 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2839999999999997e-06 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811761+0000", 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0032871720000000001 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811767+0000", 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4179999999999996e-06 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811912+0000", 2024-09-17T16:42:13.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014501 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:170701 9.11 9:8b169d3e:::benchmark_data_smithi173_150363_object63796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:12.472250+0000", 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.175529678, 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.34263178799999999, 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:13.758 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 170701 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.472250+0000", 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.472250+0000", 2024-09-17T16:42:13.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.472252+0000", 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7969999999999999e-06 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.472257+0000", 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0100000000000003e-06 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808331+0000", 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.33607442500000001 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808349+0000", 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8349999999999999e-05 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808768+0000", 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00041869299999999998 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.808937+0000", 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016837500000000001 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.809045+0000", 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010843199999999999 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811963+0000", 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029182909999999999 2024-09-17T16:42:13.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.811979+0000", 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5586999999999998e-05 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.814674+0000", 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0026946909999999999 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:13.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.814779+0000", 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000105012 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:12.814882+0000", 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000103125 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:13.764 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:42:13.765 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:13 smithi017 ceph-mon[27571]: 9.a scrub ok 2024-09-17T16:42:13.765 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:13 smithi017 ceph-mon[27571]: pgmap v2086: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 32 active+clean; 2.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 8687/136467 objects misplaced (6.366%) 2024-09-17T16:42:13.830 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:13.856 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:13.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:13.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:13.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:13.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:14.080 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:14.103 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:14.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:14.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:14.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:14.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:14.329 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:14.351 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:14.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:14.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:14.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:14.486 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:14.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:42:14.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:14 smithi173 ceph-mon[30530]: osdmap e946: 12 total, 12 up, 11 in 2024-09-17T16:42:14.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:14.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:14.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:14.716 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:14.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:14 smithi017 ceph-mon[27571]: osdmap e946: 12 total, 12 up, 11 in 2024-09-17T16:42:14.744 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:14.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:14.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:14 smithi071 ceph-mon[30002]: osdmap e946: 12 total, 12 up, 11 in 2024-09-17T16:42:14.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:14.873 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:14.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:14.896 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:14.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:15.004 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:15.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:15.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:15.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:15.139 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:15.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:15.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:15.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:15.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:15.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:15.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:15.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:15.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:15.499 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:15.532 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:15.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:15.581 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:42:15.581 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:42:15.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.581 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:172441 9.b 9:d0ba8ad4:::benchmark_data_smithi173_150363_object65536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e947)", 2024-09-17T16:42:15.581 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:15.568103+0000", 2024-09-17T16:42:15.581 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.010191314999999999, 2024-09-17T16:42:15.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010226139, 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 172441 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:15.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.568103+0000", 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.568103+0000", 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.568105+0000", 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.66e-06 2024-09-17T16:42:15.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.568112+0000", 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.109e-06 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.568125+0000", 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2829e-05 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.568133+0000", 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0940000000000008e-06 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.568207+0000", 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3983e-05 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.568339+0000", 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132008 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,10", 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.568458+0000", 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119318 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.573566+0000", 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0051074079999999999 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:15.573687+0000", 2024-09-17T16:42:15.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121833 2024-09-17T16:42:15.587 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:15.587 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:15.587 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:15.587 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:15.587 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:42:15.587 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-09-17T16:42:15.587 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:42:15.587 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:15 smithi173 ceph-mon[30530]: pgmap v2088: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 32 active+clean; 2.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 2/132030 objects degraded (0.002%); 7016/132030 objects misplaced (5.314%) 2024-09-17T16:42:15.587 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:15 smithi173 ceph-mon[30530]: osdmap e947: 12 total, 12 up, 11 in 2024-09-17T16:42:15.587 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:15 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2/132030 objects degraded (0.002%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:15.634 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:15.666 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:15.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:15 smithi017 ceph-mon[27571]: pgmap v2088: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 32 active+clean; 2.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 2/132030 objects degraded (0.002%); 7016/132030 objects misplaced (5.314%) 2024-09-17T16:42:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:15 smithi017 ceph-mon[27571]: osdmap e947: 12 total, 12 up, 11 in 2024-09-17T16:42:15.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:15 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2/132030 objects degraded (0.002%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:15.767 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:15.796 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:15.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:15.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:15 smithi071 ceph-mon[30002]: pgmap v2088: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 32 active+clean; 2.7 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 2/132030 objects degraded (0.002%); 7016/132030 objects misplaced (5.314%) 2024-09-17T16:42:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:15 smithi071 ceph-mon[30002]: osdmap e947: 12 total, 12 up, 11 in 2024-09-17T16:42:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:15 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2/132030 objects degraded (0.002%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:15.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:15.928 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:15.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:15.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:16.030 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:16.060 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:16.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:16.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:16.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:16.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:16.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:16.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:16.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:16.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:16 smithi173 ceph-mon[30530]: 9.11 scrub starts 2024-09-17T16:42:16.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:16 smithi173 ceph-mon[30530]: 9.11 scrub ok 2024-09-17T16:42:16.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:16.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:16.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:16.557 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:16.591 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:16.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:16.693 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:16.718 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:16.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:16.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:16 smithi017 ceph-mon[27571]: 9.11 scrub starts 2024-09-17T16:42:16.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:16 smithi017 ceph-mon[27571]: 9.11 scrub ok 2024-09-17T16:42:16.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:16 smithi071 ceph-mon[30002]: 9.11 scrub starts 2024-09-17T16:42:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:16 smithi071 ceph-mon[30002]: 9.11 scrub ok 2024-09-17T16:42:16.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:16.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:16.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:16.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:42:16.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:17.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:17.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:17.117 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:17.146 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:17.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:17.248 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:17.278 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:17.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:17.324 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:17.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:17.411 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:17.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:17.513 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:17.551 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:17.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:17.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:17 smithi173 ceph-mon[30530]: 9.f scrub starts 2024-09-17T16:42:17.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:17 smithi173 ceph-mon[30530]: pgmap v2090: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 2.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 2/127761 objects degraded (0.002%); 7016/127761 objects misplaced (5.492%) 2024-09-17T16:42:17.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:17 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:42:17.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:17 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:42:17.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:17 smithi173 ceph-mon[30530]: osdmap e948: 12 total, 12 up, 11 in 2024-09-17T16:42:17.652 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:17.680 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:17.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:17.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:17 smithi017 ceph-mon[27571]: 9.f scrub starts 2024-09-17T16:42:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:17 smithi017 ceph-mon[27571]: pgmap v2090: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 2.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 2/127761 objects degraded (0.002%); 7016/127761 objects misplaced (5.492%) 2024-09-17T16:42:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:17 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:42:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:17 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:42:17.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:17 smithi017 ceph-mon[27571]: osdmap e948: 12 total, 12 up, 11 in 2024-09-17T16:42:17.781 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:17.813 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:17.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:17 smithi071 ceph-mon[30002]: 9.f scrub starts 2024-09-17T16:42:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:17 smithi071 ceph-mon[30002]: pgmap v2090: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 2.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 2/127761 objects degraded (0.002%); 7016/127761 objects misplaced (5.492%) 2024-09-17T16:42:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:17 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:42:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:17 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:42:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:17 smithi071 ceph-mon[30002]: osdmap e948: 12 total, 12 up, 11 in 2024-09-17T16:42:17.905 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:42:17.905 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:42:17.905 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:42:17.905 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:42:17.905 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:42:17.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:17.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:17.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:18.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:18.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:18.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:18.173 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:18.201 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:18.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:18.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:18.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:18.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:18.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:18.428 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:18.460 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:18.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:18.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:18.590 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:18.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:18.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:42:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: 9.3 scrub starts 2024-09-17T16:42:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: 9.f scrub ok 2024-09-17T16:42:18.602 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:42:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:42:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: 9.3 scrub ok 2024-09-17T16:42:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: osdmap e949: 12 total, 12 up, 11 in 2024-09-17T16:42:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:42:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:42:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:42:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:42:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:42:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:42:18.603 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:18 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:42:18.691 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:18.722 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:18.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: 9.3 scrub starts 2024-09-17T16:42:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: 9.f scrub ok 2024-09-17T16:42:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:42:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:42:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: 9.3 scrub ok 2024-09-17T16:42:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: osdmap e949: 12 total, 12 up, 11 in 2024-09-17T16:42:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:42:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:42:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:42:18.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:42:18.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:42:18.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:42:18.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:18 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:42:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: 9.3 scrub starts 2024-09-17T16:42:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: 9.f scrub ok 2024-09-17T16:42:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:42:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:42:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: 9.3 scrub ok 2024-09-17T16:42:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: osdmap e949: 12 total, 12 up, 11 in 2024-09-17T16:42:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:42:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:42:18.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:42:18.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:42:18.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:42:18.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:42:18.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:18 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:42:18.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:18.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:18.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:18.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:18.971 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:18.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:18.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:19.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:19.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:19.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:19.231 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:19.263 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:19.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:19.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:19 smithi173 ceph-mon[30530]: 9.6 scrub starts 2024-09-17T16:42:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:19 smithi173 ceph-mon[30530]: 9.6 scrub ok 2024-09-17T16:42:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:19 smithi173 ceph-mon[30530]: pgmap v2093: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2/123045 objects degraded (0.002%); 4873/123045 objects misplaced (3.960%) 2024-09-17T16:42:19.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:19.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:19.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:19.500 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:19.534 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:19.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:19.597 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:42:19.687 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:42:19.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:42:19.688 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:42:19.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.688 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:178376 7.16 7:6e17509c:::benchmark_data_smithi173_116396_object82776:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e718)", 2024-09-17T16:42:19.689 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:36.932136+0000", 2024-09-17T16:42:19.689 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 522.660753548, 2024-09-17T16:42:19.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.346692789, 2024-09-17T16:42:19.689 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.689 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.690 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.690 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.690 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:42:19.690 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:42:19.690 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 178376 2024-09-17T16:42:19.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.690 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932136+0000", 2024-09-17T16:42:19.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932136+0000", 2024-09-17T16:42:19.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932137+0000", 2024-09-17T16:42:19.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6399999999999998e-07 2024-09-17T16:42:19.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932138+0000", 2024-09-17T16:42:19.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.827e-06 2024-09-17T16:42:19.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932142+0000", 2024-09-17T16:42:19.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0500000000000002e-06 2024-09-17T16:42:19.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932146+0000", 2024-09-17T16:42:19.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4570000000000001e-06 2024-09-17T16:42:19.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932233+0000", 2024-09-17T16:42:19.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6953999999999995e-05 2024-09-17T16:42:19.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932318+0000", 2024-09-17T16:42:19.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4808000000000005e-05 2024-09-17T16:42:19.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,2", 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.932395+0000", 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7099000000000002e-05 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.934437+0000", 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00204199 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:36.934635+0000", 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019845 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:37.278717+0000", 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34408162399999997 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:37.278780+0000", 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3583000000000004e-05 2024-09-17T16:42:19.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:37.278829+0000", 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8582999999999999e-05 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:7992 8.c 8:34878b4c:::benchmark_data_smithi173_130566_object7991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e732)", 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.037799+0000", 2024-09-17T16:42:19.699 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 482.55509099099999, 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.26676612, 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 7992 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037799+0000", 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037799+0000", 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037800+0000", 2024-09-17T16:42:19.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.054e-06 2024-09-17T16:42:19.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037828+0000", 2024-09-17T16:42:19.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8294e-05 2024-09-17T16:42:19.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037846+0000", 2024-09-17T16:42:19.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7649999999999999e-05 2024-09-17T16:42:19.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037854+0000", 2024-09-17T16:42:19.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5029999999999997e-06 2024-09-17T16:42:19.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.037910+0000", 2024-09-17T16:42:19.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5998000000000002e-05 2024-09-17T16:42:19.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.038015+0000", 2024-09-17T16:42:19.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010526000000000001 2024-09-17T16:42:19.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:42:19.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.038108+0000", 2024-09-17T16:42:19.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2848000000000003e-05 2024-09-17T16:42:19.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.038630+0000", 2024-09-17T16:42:19.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00052117799999999996 2024-09-17T16:42:19.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.039017+0000", 2024-09-17T16:42:19.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038750600000000003 2024-09-17T16:42:19.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304468+0000", 2024-09-17T16:42:19.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.265451467 2024-09-17T16:42:19.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304528+0000", 2024-09-17T16:42:19.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9901000000000001e-05 2024-09-17T16:42:19.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304565+0000", 2024-09-17T16:42:19.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6461000000000002e-05 2024-09-17T16:42:19.710 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.710 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.710 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.711 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:7994 8.c 8:379b6ec9:::benchmark_data_smithi173_130566_object7993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e732)", 2024-09-17T16:42:19.711 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.040560+0000", 2024-09-17T16:42:19.711 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 482.55232961199999, 2024-09-17T16:42:19.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.264147548, 2024-09-17T16:42:19.711 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.711 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.711 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.711 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 7994 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040560+0000", 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040560+0000", 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040561+0000", 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3900000000000003e-07 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040585+0000", 2024-09-17T16:42:19.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4145e-05 2024-09-17T16:42:19.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040599+0000", 2024-09-17T16:42:19.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3766e-05 2024-09-17T16:42:19.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040606+0000", 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9990000000000002e-06 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040650+0000", 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4447999999999999e-05 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040735+0000", 2024-09-17T16:42:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4257000000000003e-05 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.040821+0000", 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6341000000000006e-05 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.041276+0000", 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045481100000000001 2024-09-17T16:42:19.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.041625+0000", 2024-09-17T16:42:19.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034894899999999998 2024-09-17T16:42:19.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304619+0000", 2024-09-17T16:42:19.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.262994183 2024-09-17T16:42:19.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304666+0000", 2024-09-17T16:42:19.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7079000000000001e-05 2024-09-17T16:42:19.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304708+0000", 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1631000000000002e-05 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8013 8.c 8:3c762cec:::benchmark_data_smithi173_130566_object8012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e732)", 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.082606+0000", 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 482.51028355599999, 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.222249886, 2024-09-17T16:42:19.719 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8013 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082606+0000", 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082606+0000", 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082607+0000", 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5700000000000001e-07 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082627+0000", 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9565999999999999e-05 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082637+0000", 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0616e-05 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082645+0000", 2024-09-17T16:42:19.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2960000000000001e-06 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082691+0000", 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6681999999999998e-05 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082787+0000", 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6203e-05 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.082873+0000", 2024-09-17T16:42:19.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5687999999999997e-05 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.083367+0000", 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049404099999999997 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.083928+0000", 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00056047200000000001 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304759+0000", 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 14.22083164 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304817+0000", 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7763000000000002e-05 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304856+0000", 2024-09-17T16:42:19.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9062e-05 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8032 8.c 8:3d0a817e:::benchmark_data_smithi173_130566_object8031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e733)", 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.405332+0000", 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 482.18755809700002, 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.899664348, 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.726 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8032 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405332+0000", 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405332+0000", 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:19.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405334+0000", 2024-09-17T16:42:19.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.897e-06 2024-09-17T16:42:19.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405359+0000", 2024-09-17T16:42:19.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5313999999999999e-05 2024-09-17T16:42:19.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405376+0000", 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7538999999999999e-05 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405385+0000", 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1710000000000002e-06 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405454+0000", 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9338999999999995e-05 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405659+0000", 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000204852 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.405792+0000", 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133187 2024-09-17T16:42:19.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.406488+0000", 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069618000000000002 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.406840+0000", 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035165199999999999 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304910+0000", 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.898070153999999 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304960+0000", 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0421999999999999e-05 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.304996+0000", 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5641e-05 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.734 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8037 8.c 8:3f3c220b:::benchmark_data_smithi173_130566_object8036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e733)", 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.409689+0000", 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 482.18320023400003, 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.895437986999999, 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.735 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8037 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409689+0000", 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409689+0000", 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409691+0000", 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.376e-06 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409713+0000", 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2589999999999999e-05 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409727+0000", 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4035e-05 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409736+0000", 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6470000000000004e-06 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409783+0000", 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6811000000000003e-05 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.409909+0000", 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012609599999999999 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.410018+0000", 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108467 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.410535+0000", 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051754199999999996 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.410949+0000", 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00041375800000000002 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305039+0000", 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.894089841 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305091+0000", 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2429000000000002e-05 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305127+0000", 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6395e-05 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.741 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:8040 8.c 8:3ef6c0ff:::benchmark_data_smithi173_130566_object8039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e733)", 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:17.412931+0000", 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 482.179958779, 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.892416228, 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8040 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.742 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412931+0000", 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412931+0000", 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412932+0000", 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5799999999999998e-07 2024-09-17T16:42:19.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412952+0000", 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0636999999999999e-05 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412963+0000", 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0993e-05 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.412969+0000", 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.429e-06 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413016+0000", 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6777000000000003e-05 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413102+0000", 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6059999999999999e-05 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:42:19.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413187+0000", 2024-09-17T16:42:19.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5408999999999995e-05 2024-09-17T16:42:19.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413596+0000", 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040850499999999998 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:17.413999+0000", 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040360899999999998 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305263+0000", 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 13.891264103999999 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305314+0000", 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0522999999999997e-05 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:31.305347+0000", 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3324000000000001e-05 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.748 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.17 741/739 [PushOp(8:ed657130:::benchmark_data_smithi173_130566_object3105:head, version: 732'197, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ed657130:::benchmark_data_smithi173_130566_object3105:head@732'197, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:27.949079+0000", 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 471.64381072999998, 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34690885100000002, 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.949079+0000", 2024-09-17T16:42:19.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.949079+0000", 2024-09-17T16:42:19.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.949086+0000", 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4749999999999996e-06 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.949139+0000", 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2955999999999999e-05 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:28.295598+0000", 2024-09-17T16:42:19.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34645822399999998 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:28.295611+0000", 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3377000000000001e-05 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:28.295697+0000", 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5765000000000003e-05 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.752 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:28.295706+0000", 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5840000000000001e-06 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:28.295988+0000", 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028146999999999997 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:92948 8.a 8:5c06d358:::benchmark_data_smithi173_130566_object92947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:35:39.305492+0000", 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 400.28739802400003, 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42912947499999998, 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 92948 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.754 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305492+0000", 2024-09-17T16:42:19.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305492+0000", 2024-09-17T16:42:19.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305493+0000", 2024-09-17T16:42:19.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4449999999999999e-06 2024-09-17T16:42:19.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305518+0000", 2024-09-17T16:42:19.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.442e-05 2024-09-17T16:42:19.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305530+0000", 2024-09-17T16:42:19.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2873e-05 2024-09-17T16:42:19.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305536+0000", 2024-09-17T16:42:19.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9329999999999998e-06 2024-09-17T16:42:19.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.759 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.759 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305587+0000", 2024-09-17T16:42:19.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0291999999999999e-05 2024-09-17T16:42:19.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305707+0000", 2024-09-17T16:42:19.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119984 2024-09-17T16:42:19.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:42:19.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.305797+0000", 2024-09-17T16:42:19.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0766000000000005e-05 2024-09-17T16:42:19.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.308288+0000", 2024-09-17T16:42:19.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002490361 2024-09-17T16:42:19.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.309496+0000", 2024-09-17T16:42:19.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001208104 2024-09-17T16:42:19.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734535+0000", 2024-09-17T16:42:19.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42503903700000001 2024-09-17T16:42:19.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734580+0000", 2024-09-17T16:42:19.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4672000000000002e-05 2024-09-17T16:42:19.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.765 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.765 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734621+0000", 2024-09-17T16:42:19.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1588000000000002e-05 2024-09-17T16:42:19.766 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.766 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.766 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.767 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.17883.0:92948 8.a e767/728 ondisk, result = 0)", 2024-09-17T16:42:19.767 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:35:39.308966+0000", 2024-09-17T16:42:19.767 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 400.28392396300001, 2024-09-17T16:42:19.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42566467299999999, 2024-09-17T16:42:19.767 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.767 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.768 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:42:19.768 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.308966+0000", 2024-09-17T16:42:19.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.308966+0000", 2024-09-17T16:42:19.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.308967+0000", 2024-09-17T16:42:19.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4600000000000003e-07 2024-09-17T16:42:19.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.772 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.308970+0000", 2024-09-17T16:42:19.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6109999999999998e-06 2024-09-17T16:42:19.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734474+0000", 2024-09-17T16:42:19.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42550323899999998 2024-09-17T16:42:19.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734479+0000", 2024-09-17T16:42:19.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.023e-06 2024-09-17T16:42:19.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734524+0000", 2024-09-17T16:42:19.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5732999999999997e-05 2024-09-17T16:42:19.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734531+0000", 2024-09-17T16:42:19.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1959999999999996e-06 2024-09-17T16:42:19.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:39.734630+0000", 2024-09-17T16:42:19.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9925000000000003e-05 2024-09-17T16:42:19.778 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.778 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.778 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.779 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:94971 8.b 8:db7e09c1:::benchmark_data_smithi173_130566_object94970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e769)", 2024-09-17T16:42:19.779 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:35:41.610934+0000", 2024-09-17T16:42:19.779 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 397.98195532900002, 2024-09-17T16:42:19.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.78101871599999995, 2024-09-17T16:42:19.779 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.780 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.780 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.780 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.780 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.780 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.780 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 94971 2024-09-17T16:42:19.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.781 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.610934+0000", 2024-09-17T16:42:19.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.610934+0000", 2024-09-17T16:42:19.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.783 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.610935+0000", 2024-09-17T16:42:19.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5099999999999999e-07 2024-09-17T16:42:19.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.784 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.784 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.784 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.610982+0000", 2024-09-17T16:42:19.784 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7377999999999999e-05 2024-09-17T16:42:19.784 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.784 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.610992+0000", 2024-09-17T16:42:19.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5109999999999992e-06 2024-09-17T16:42:19.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.611000+0000", 2024-09-17T16:42:19.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8120000000000008e-06 2024-09-17T16:42:19.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.611059+0000", 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9290999999999999e-05 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.611147+0000", 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7466000000000006e-05 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.611227+0000", 2024-09-17T16:42:19.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0873999999999999e-05 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.613680+0000", 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024524719999999998 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:41.614006+0000", 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032607600000000001 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:42.391836+0000", 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77782961699999997 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:42.391912+0000", 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6388000000000001e-05 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:35:42.391953+0000", 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1180000000000002e-05 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:151014 8.a 8:5e424e97:::benchmark_data_smithi173_130566_object56039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e819)", 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:39.945344+0000", 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 279.64754533199999, 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36685722399999998, 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.790 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 151014 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.945344+0000", 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.945344+0000", 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.945346+0000", 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7829999999999999e-06 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.945353+0000", 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9909999999999996e-06 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.945368+0000", 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.485e-05 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.945376+0000", 2024-09-17T16:42:19.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5190000000000003e-06 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.945456+0000", 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0105000000000006e-05 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.945596+0000", 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000140431 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,3", 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.945714+0000", 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117726 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.950578+0000", 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048640630000000001 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:39.951403+0000", 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082543199999999997 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.312097+0000", 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36069357000000002 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.312155+0000", 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7670000000000002e-05 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:40.312202+0000", 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7083999999999998e-05 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:189943 8.a 8:53cfacbc:::benchmark_data_smithi173_130566_object94968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e849)", 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:34.130688+0000", 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 225.46220216, 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38584874499999999, 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.797 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 189943 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.130688+0000", 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.130688+0000", 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.130689+0000", 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0890000000000001e-06 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.130694+0000", 2024-09-17T16:42:19.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3009999999999997e-06 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.130709+0000", 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4548e-05 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.130715+0000", 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3260000000000001e-06 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.130765+0000", 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0086999999999997e-05 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.130853+0000", 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8537999999999998e-05 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.130936+0000", 2024-09-17T16:42:19.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.284e-05 2024-09-17T16:42:19.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135497+0000", 2024-09-17T16:42:19.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045611039999999999 2024-09-17T16:42:19.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135650+0000", 2024-09-17T16:42:19.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015300200000000001 2024-09-17T16:42:19.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.516422+0000", 2024-09-17T16:42:19.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38077177099999998 2024-09-17T16:42:19.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.516488+0000", 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5611000000000006e-05 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.516536+0000", 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8528e-05 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.804 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.17883.0:189943 8.a e849/848 ondisk, result = 0)", 2024-09-17T16:42:19.805 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:34.135121+0000", 2024-09-17T16:42:19.805 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 225.457768566, 2024-09-17T16:42:19.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38142184099999998, 2024-09-17T16:42:19.805 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.805 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.805 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:42:19.805 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.805 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135121+0000", 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135121+0000", 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135122+0000", 2024-09-17T16:42:19.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.06e-07 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135126+0000", 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3549999999999998e-06 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.516337+0000", 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38121122899999998 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.516345+0000", 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5800000000000003e-06 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.516404+0000", 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8695e-05 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.516414+0000", 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.059e-05 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.516543+0000", 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012868600000000001 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.809 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:189945 8.b 8:db7e09c1:::benchmark_data_smithi173_130566_object94970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e849)", 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:34.135836+0000", 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 225.45705399900001, 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.69791321100000003, 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 189945 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.810 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135836+0000", 2024-09-17T16:42:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135836+0000", 2024-09-17T16:42:19.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135837+0000", 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1000000000000001e-06 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135841+0000", 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3150000000000002e-06 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135851+0000", 2024-09-17T16:42:19.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7739999999999999e-06 2024-09-17T16:42:19.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135856+0000", 2024-09-17T16:42:19.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4480000000000002e-06 2024-09-17T16:42:19.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.135920+0000", 2024-09-17T16:42:19.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3132000000000005e-05 2024-09-17T16:42:19.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.136043+0000", 2024-09-17T16:42:19.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123161 2024-09-17T16:42:19.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-09-17T16:42:19.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.136174+0000", 2024-09-17T16:42:19.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013151699999999999 2024-09-17T16:42:19.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.140966+0000", 2024-09-17T16:42:19.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047916879999999997 2024-09-17T16:42:19.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.141087+0000", 2024-09-17T16:42:19.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121066 2024-09-17T16:42:19.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.833639+0000", 2024-09-17T16:42:19.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.69255236600000003 2024-09-17T16:42:19.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.833699+0000", 2024-09-17T16:42:19.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9262999999999998e-05 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.833749+0000", 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0380999999999998e-05 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.17883.0:189945 8.b e849/728 ondisk, result = 0)", 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:34.140356+0000", 2024-09-17T16:42:19.820 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 225.45253363899999, 2024-09-17T16:42:19.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.69339931700000002, 2024-09-17T16:42:19.821 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.821 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.821 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:42:19.821 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.140356+0000", 2024-09-17T16:42:19.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.140356+0000", 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.140357+0000", 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0979999999999999e-06 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.140363+0000", 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3550000000000004e-06 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.833555+0000", 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.69319277700000004 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.833564+0000", 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3650000000000004e-06 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.833621+0000", 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7744999999999997e-05 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.833631+0000", 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.006e-05 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:34.833755+0000", 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123917 2024-09-17T16:42:19.824 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(9.3 867/866 [PushOp(9:cde68664:::benchmark_data_smithi173_150363_object977:head, version: 862'54, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:cde68664:::benchmark_data_smithi173_150363_object977:head@862'54, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:09.924032+0000", 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 189.66885822099999, 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.409034236, 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.924032+0000", 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.924032+0000", 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.924032+0000", 2024-09-17T16:42:19.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5300000000000004e-07 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.924094+0000", 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1455e-05 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.332444+0000", 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.408350247 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.332462+0000", 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8087000000000001e-05 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.332615+0000", 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000153339 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.332630+0000", 2024-09-17T16:42:19.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4635e-05 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.333066+0000", 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043582000000000001 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(1.4 885/884 [PushReplyOp(1:30306672:devicehealth::main.db.0000000000000000:head)])", 2024-09-17T16:42:19.829 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:33.802068+0000", 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 165.79082183700001, 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54286229100000005, 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:33.802068+0000", 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:33.802068+0000", 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:33.802068+0000", 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.73e-07 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:33.802071+0000", 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2899999999999998e-06 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:34.344621+0000", 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54254986900000002 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:34.344631+0000", 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0091e-05 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:34.344699+0000", 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7348999999999995e-05 2024-09-17T16:42:19.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:34.344930+0000", 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023151899999999999 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:122210 9.17 9.d1325e37 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e914)", 2024-09-17T16:42:19.833 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:40:51.134712+0000", 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 88.458177727000006, 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66534664600000004, 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 122210 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:40:51.134712+0000", 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:40:51.134712+0000", 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:40:51.134713+0000", 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5300000000000003e-07 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:40:51.134717+0000", 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9500000000000003e-06 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:40:51.134728+0000", 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0832999999999999e-05 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:40:51.134733+0000", 2024-09-17T16:42:19.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1240000000000004e-06 2024-09-17T16:42:19.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:40:51.134774+0000", 2024-09-17T16:42:19.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0886999999999997e-05 2024-09-17T16:42:19.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:42:19.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:40:51.134780+0000", 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1249999999999998e-06 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:40:51.800048+0000", 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66526829700000001 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:40:51.800059+0000", 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0577000000000001e-05 2024-09-17T16:42:19.838 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:170734 9.b 9:d3cb84a9:::benchmark_data_smithi173_150363_object63829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:12.532676+0000", 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 7.0602140179999999, 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29094349600000002, 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:19.839 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 170734 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.532676+0000", 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.532676+0000", 2024-09-17T16:42:19.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.532677+0000", 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.311e-06 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.532683+0000", 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5029999999999997e-06 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:19.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.532695+0000", 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2286000000000001e-05 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.532701+0000", 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9320000000000001e-06 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.532756+0000", 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4948999999999999e-05 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.532837+0000", 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0889000000000004e-05 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,10", 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.532922+0000", 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5674000000000007e-05 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.536937+0000", 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004015088 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.536946+0000", 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2549999999999997e-06 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.823516+0000", 2024-09-17T16:42:19.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28657081200000001 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.823579+0000", 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2117000000000002e-05 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.823619+0000", 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0679999999999997e-05 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.845 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.846 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.846 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:19.846 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:19.846 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:42:19.847 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:19 smithi071 ceph-mon[30002]: 9.6 scrub starts 2024-09-17T16:42:19.847 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:19 smithi071 ceph-mon[30002]: 9.6 scrub ok 2024-09-17T16:42:19.847 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:19 smithi071 ceph-mon[30002]: pgmap v2093: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2/123045 objects degraded (0.002%); 4873/123045 objects misplaced (3.960%) 2024-09-17T16:42:19.847 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:19 smithi017 ceph-mon[27571]: 9.6 scrub starts 2024-09-17T16:42:19.847 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:19 smithi017 ceph-mon[27571]: 9.6 scrub ok 2024-09-17T16:42:19.847 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:19 smithi017 ceph-mon[27571]: pgmap v2093: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 2/123045 objects degraded (0.002%); 4873/123045 objects misplaced (3.960%) 2024-09-17T16:42:19.873 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:19.874 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:19.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:19.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:19.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:20.100 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:20.122 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:20.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:20.225 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:20.246 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:20.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:20.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:20.348 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:20.371 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:20.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:20.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:20.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:20.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:20.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:42:20.598 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:20.631 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:20.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:20.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:20 smithi173 ceph-mon[30530]: osdmap e950: 12 total, 12 up, 11 in 2024-09-17T16:42:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:20 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2/123045 objects degraded (0.002%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:20.732 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:20.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:20 smithi017 ceph-mon[27571]: osdmap e950: 12 total, 12 up, 11 in 2024-09-17T16:42:20.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:20 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2/123045 objects degraded (0.002%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:20.755 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:20.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:20.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:20 smithi071 ceph-mon[30002]: osdmap e950: 12 total, 12 up, 11 in 2024-09-17T16:42:20.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:20 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2/123045 objects degraded (0.002%), 1 pg degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:20.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:20.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:20.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:20.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:20.915 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:21.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:21.034 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:21.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:21.135 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:21.167 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:21.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:21.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:21.295 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:21.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:21.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:21.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:21.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:21.426 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:21 smithi017 ceph-mon[27571]: pgmap v2095: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 2.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4873/116994 objects misplaced (4.165%) 2024-09-17T16:42:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:21 smithi017 ceph-mon[27571]: osdmap e951: 12 total, 12 up, 11 in 2024-09-17T16:42:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:21 smithi017 ceph-mon[27571]: 9.15 scrub starts 2024-09-17T16:42:21.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:21 smithi017 ceph-mon[27571]: 9.15 scrub ok 2024-09-17T16:42:21.525 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:42:21.526 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:42:21.526 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:42:21.526 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:42:21.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:21.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:21.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:21 smithi173 ceph-mon[30530]: pgmap v2095: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 2.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4873/116994 objects misplaced (4.165%) 2024-09-17T16:42:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:21 smithi173 ceph-mon[30530]: osdmap e951: 12 total, 12 up, 11 in 2024-09-17T16:42:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:21 smithi173 ceph-mon[30530]: 9.15 scrub starts 2024-09-17T16:42:21.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:21 smithi173 ceph-mon[30530]: 9.15 scrub ok 2024-09-17T16:42:21.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:21.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:21.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:21.788 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:21.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:21 smithi071 ceph-mon[30002]: pgmap v2095: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 2.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 4873/116994 objects misplaced (4.165%) 2024-09-17T16:42:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:21 smithi071 ceph-mon[30002]: osdmap e951: 12 total, 12 up, 11 in 2024-09-17T16:42:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:21 smithi071 ceph-mon[30002]: 9.15 scrub starts 2024-09-17T16:42:21.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:21 smithi071 ceph-mon[30002]: 9.15 scrub ok 2024-09-17T16:42:21.815 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:21.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:21.917 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:21.943 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:21.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:22.045 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:22.076 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:22.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:22.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:22.210 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:22.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:22.312 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:22.344 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:22.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:22.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:22.476 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:22.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:22.578 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:22.606 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:22.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:22.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:22.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:22.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:22.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:42:22.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:22.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:22.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:22.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:23.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:23.005 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:23.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:23.051 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:23.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:23.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:23.133 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:23.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:23.144 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:23.218 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:42:23.234 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:23.255 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:23.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:23.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:23.357 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:23.388 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:23.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:23.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:23.391 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.9 to 1 (10000) 2024-09-17T16:42:23.490 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:23.530 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:23.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:23.561 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:23 smithi173 ceph-mon[30530]: pgmap v2097: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 2.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 751 op/s; 2/114582 objects degraded (0.002%); 4720/114582 objects misplaced (4.119%); 8.2 MiB/s, 249 objects/s recovering 2024-09-17T16:42:23.561 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:23 smithi173 ceph-mon[30530]: 9.e scrub starts 2024-09-17T16:42:23.561 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/187420533' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:42:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:23 smithi071 ceph-mon[30002]: pgmap v2097: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 2.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 751 op/s; 2/114582 objects degraded (0.002%); 4720/114582 objects misplaced (4.119%); 8.2 MiB/s, 249 objects/s recovering 2024-09-17T16:42:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:23 smithi071 ceph-mon[30002]: 9.e scrub starts 2024-09-17T16:42:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/187420533' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:42:23.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:23.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:23.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:23.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:23.702 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:42:23.702 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:42:23.702 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:42:23.702 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:42:23.702 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:42:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:23 smithi017 ceph-mon[27571]: pgmap v2097: 37 pgs: 4 active+recovering+undersized+remapped, 33 active+clean; 2.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 751 op/s; 2/114582 objects degraded (0.002%); 4720/114582 objects misplaced (4.119%); 8.2 MiB/s, 249 objects/s recovering 2024-09-17T16:42:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:23 smithi017 ceph-mon[27571]: 9.e scrub starts 2024-09-17T16:42:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/187420533' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:42:23.765 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:23.794 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:23.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:23.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:23.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:23.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:24.028 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:24.068 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:24.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:24.171 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:24.198 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:24.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:24.300 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:24.331 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:24.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:24.403 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:42:24.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:24.469 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:24.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:24.566 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:24 smithi017 ceph-mon[27571]: 9.e scrub ok 2024-09-17T16:42:24.566 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/187420533' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:42:24.566 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:24 smithi017 ceph-mon[27571]: osdmap e952: 12 total, 12 up, 11 in 2024-09-17T16:42:24.566 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:24 smithi017 ceph-mon[27571]: 9.5 scrub starts 2024-09-17T16:42:24.566 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:24 smithi017 ceph-mon[27571]: 9.5 scrub ok 2024-09-17T16:42:24.566 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:24 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1075260504' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:42:24.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:24.599 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:24.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:24.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:24.613 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-17T16:42:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:24 smithi173 ceph-mon[30530]: 9.e scrub ok 2024-09-17T16:42:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/187420533' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:42:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:24 smithi173 ceph-mon[30530]: osdmap e952: 12 total, 12 up, 11 in 2024-09-17T16:42:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:24 smithi173 ceph-mon[30530]: 9.5 scrub starts 2024-09-17T16:42:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:24 smithi173 ceph-mon[30530]: 9.5 scrub ok 2024-09-17T16:42:24.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:24 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1075260504' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:42:24.702 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:24.729 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:24.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:24.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:24 smithi071 ceph-mon[30002]: 9.e scrub ok 2024-09-17T16:42:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/187420533' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:42:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:24 smithi071 ceph-mon[30002]: osdmap e952: 12 total, 12 up, 11 in 2024-09-17T16:42:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:24 smithi071 ceph-mon[30002]: 9.5 scrub starts 2024-09-17T16:42:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:24 smithi071 ceph-mon[30002]: 9.5 scrub ok 2024-09-17T16:42:24.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:24 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1075260504' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:42:24.830 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:24.852 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:24.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:24.953 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:24.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:42:24.979 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:24.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:25.081 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:25.105 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:25.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:25.208 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:25.229 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:25.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:25.243 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:25.331 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:25.353 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:25.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:25.354 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:25.361 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:25.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:25.483 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:25.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:25.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:25.585 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:25.614 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:25.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:25.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:25 smithi173 ceph-mon[30530]: pgmap v2099: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2/109602 objects degraded (0.002%); 3307/109602 objects misplaced (3.017%) 2024-09-17T16:42:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:25 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:42:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:25 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:42:25.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:25 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:25.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1075260504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:42:25.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:25 smithi173 ceph-mon[30530]: osdmap e953: 12 total, 12 up, 11 in 2024-09-17T16:42:25.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:25 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2/109602 objects degraded (0.002%), 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:25.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:25.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:25 smithi017 ceph-mon[27571]: pgmap v2099: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2/109602 objects degraded (0.002%); 3307/109602 objects misplaced (3.017%) 2024-09-17T16:42:25.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:25 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:42:25.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:25 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:42:25.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:25 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:25.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1075260504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:42:25.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:25 smithi017 ceph-mon[27571]: osdmap e953: 12 total, 12 up, 11 in 2024-09-17T16:42:25.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:25 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2/109602 objects degraded (0.002%), 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:25.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:25.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:25.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:25 smithi071 ceph-mon[30002]: pgmap v2099: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 2/109602 objects degraded (0.002%); 3307/109602 objects misplaced (3.017%) 2024-09-17T16:42:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:25 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:42:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:25 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:42:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:25 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1075260504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:42:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:25 smithi071 ceph-mon[30002]: osdmap e953: 12 total, 12 up, 11 in 2024-09-17T16:42:25.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:25 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2/109602 objects degraded (0.002%), 3 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:25.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:25.871 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:42:25.871 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:42:25.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:42:25.871 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:42:25.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.871 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27088.0:170897 7.5 7:a2dbce43:::benchmark_data_smithi173_116396_object75297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e712)", 2024-09-17T16:42:25.871 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:26.438350+0000", 2024-09-17T16:42:25.871 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 539.42912509999996, 2024-09-17T16:42:25.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34698544799999997, 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27088", 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 170897 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438350+0000", 2024-09-17T16:42:25.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438350+0000", 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438351+0000", 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.815e-06 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438358+0000", 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8260000000000004e-06 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438372+0000", 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4131000000000001e-05 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438380+0000", 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4449999999999999e-06 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.874 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438438+0000", 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7833e-05 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438537+0000", 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8987999999999995e-05 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.438658+0000", 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121291 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.441438+0000", 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027800049999999999 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.453632+0000", 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012194439 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785264+0000", 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33163123999999999 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785303+0000", 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8894999999999999e-05 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:26.785335+0000", 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2539999999999997e-05 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(7.0 e724,723 7:00d6d267:::benchmark_data_smithi173_116396_object11771:head,679'6499)", 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:33:54.910007+0000", 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 510.95746778, 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71534004299999998, 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910007+0000", 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910007+0000", 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910009+0000", 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6169999999999999e-06 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:54.910019+0000", 2024-09-17T16:42:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0047000000000001e-05 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.624856+0000", 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71483768199999997 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.624864+0000", 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5909999999999998e-06 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.624943+0000", 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9234999999999996e-05 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:33:55.625347+0000", 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040387099999999998 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.15 740/739 [PushOp(8:a91ad1da:::benchmark_data_smithi173_130566_object2731:head, version: 732'202, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:a91ad1da:::benchmark_data_smithi173_130566_object2731:head@732'202, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:42:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 478.77029068100001, 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448453453, 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097185+0000", 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2799999999999995e-07 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097245+0000", 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9749e-05 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545252+0000", 2024-09-17T16:42:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44800729700000003 2024-09-17T16:42:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545267+0000", 2024-09-17T16:42:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5017999999999999e-05 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545346+0000", 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9491000000000002e-05 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545357+0000", 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0491e-05 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545638+0000", 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028057899999999998 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.13 740/739 [PushOp(8:cce6ff3a:::benchmark_data_smithi173_130566_object3681:head, version: 732'214, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:cce6ff3a:::benchmark_data_smithi173_130566_object3681:head@732'214, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 478.76834819200002, 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448568835, 2024-09-17T16:42:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7899999999999997e-07 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099181+0000", 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3591999999999998e-05 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547347+0000", 2024-09-17T16:42:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448166224 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547363+0000", 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5364000000000001e-05 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547441+0000", 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8873000000000002e-05 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547450+0000", 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5469999999999997e-06 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547695+0000", 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024535600000000001 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:115081 8.0 e800/767 8:02125afa:::benchmark_data_smithi173_130566_object20106:head v 800'3851, mlcod=800'3848)", 2024-09-17T16:42:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:54.801782+0000", 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 331.065692265, 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66971172400000001, 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801782+0000", 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801782+0000", 2024-09-17T16:42:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801783+0000", 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0500000000000002e-07 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801788+0000", 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1939999999999997e-06 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469802+0000", 2024-09-17T16:42:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66801375100000004 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469811+0000", 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.473e-06 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469859+0000", 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7874999999999998e-05 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469877+0000", 2024-09-17T16:42:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8321e-05 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.471459+0000", 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00158165 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.471494+0000", 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5655000000000003e-05 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(8.4 e800,797 8:22e2a6d6:::benchmark_data_smithi173_130566_object14161:head,794'3581)", 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:55.001177+0000", 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 330.86629739900002, 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38830608500000002, 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.001177+0000", 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.001177+0000", 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.001178+0000", 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0500000000000003e-07 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.001183+0000", 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8130000000000003e-06 2024-09-17T16:42:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.387655+0000", 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.386472549 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.387680+0000", 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4977000000000002e-05 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.387871+0000", 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019087999999999999 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.389483+0000", 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001612161 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:116728 8.15 e802/743 8:a96131de:::benchmark_data_smithi173_130566_object21753:head v 802'3888, mlcod=802'3885)", 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:57.515606+0000", 2024-09-17T16:42:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 328.35186894899999, 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.344180555, 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.515606+0000", 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.515606+0000", 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.515607+0000", 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8000000000000004e-07 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.515613+0000", 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0410000000000003e-06 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.857627+0000", 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34201457499999999 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.857635+0000", 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8029999999999997e-06 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.857731+0000", 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6113000000000006e-05 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.857751+0000", 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9983000000000002e-05 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.859742+0000", 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019903719999999998 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:57.859786+0000", 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4787999999999999e-05 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:153532 8.c e821/728 8:37852649:::benchmark_data_smithi173_130566_object58557:head v 821'9257, mlcod=821'9254)", 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:37:43.797542+0000", 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 282.06993276700001, 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.280386733, 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:43.797542+0000", 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:43.797542+0000", 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:43.797550+0000", 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6459999999999993e-06 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:43.797555+0000", 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6200000000000004e-06 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:44.070166+0000", 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27261081199999998 2024-09-17T16:42:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:44.070181+0000", 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5197999999999999e-05 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:44.070235+0000", 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3622999999999998e-05 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:44.070258+0000", 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3439999999999999e-05 2024-09-17T16:42:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:44.077872+0000", 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0076139609999999998 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:37:44.077929+0000", 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6433e-05 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:164504 8.4 e831/810 8:2388bdab:::benchmark_data_smithi173_130566_object69529:head v 831'5246, mlcod=831'5243)", 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:00.177004+0000", 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 265.69047102500002, 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.276534375, 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:00.177004+0000", 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:00.177004+0000", 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:00.177005+0000", 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5199999999999995e-07 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:00.177012+0000", 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4089999999999997e-06 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:00.448489+0000", 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27147656199999998 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:00.448496+0000", 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1770000000000001e-06 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:00.448591+0000", 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4916999999999999e-05 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:00.448610+0000", 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9432999999999998e-05 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:00.453495+0000", 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048850500000000002 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:00.453538+0000", 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2874999999999999e-05 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:169698 8.4 e834/810 8:2314ca98:::benchmark_data_smithi173_130566_object74723:head v 834'5404, mlcod=834'5399)", 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:38:06.544273+0000", 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 259.32320160900002, 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.276294337, 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:06.544273+0000", 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:06.544273+0000", 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:06.544274+0000", 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0249999999999999e-06 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:06.544282+0000", 2024-09-17T16:42:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9589999999999995e-06 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:06.817935+0000", 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27365245999999999 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:06.817941+0000", 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5239999999999997e-06 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:06.817990+0000", 2024-09-17T16:42:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9169999999999998e-05 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:06.818005+0000", 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4283e-05 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:06.820526+0000", 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025219000000000001 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:38:06.820567+0000", 2024-09-17T16:42:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1016000000000002e-05 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:8122 9.1 9.c6cc7e1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:08.217944+0000", 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 197.649530613, 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98969176699999994, 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8122 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217944+0000", 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217944+0000", 2024-09-17T16:42:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217945+0000", 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.319e-06 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217965+0000", 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9749e-05 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217976+0000", 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1021e-05 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217982+0000", 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.412e-06 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.218030+0000", 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8850999999999997e-05 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.218040+0000", 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7289999999999995e-06 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207605+0000", 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98956506499999997 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207636+0000", 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0620999999999997e-05 2024-09-17T16:42:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:8128 9.1 9.32423851 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:08.409343+0000", 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 197.458131412, 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79838889400000002, 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8128 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409343+0000", 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409343+0000", 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409346+0000", 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7300000000000001e-06 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409389+0000", 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3245999999999999e-05 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409431+0000", 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1604999999999999e-05 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409439+0000", 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.106e-06 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409496+0000", 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7380999999999998e-05 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409504+0000", 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4889999999999998e-06 2024-09-17T16:42:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207722+0000", 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.798218223 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207732+0000", 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0114e-05 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:8132 9.1 9.cd430e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-09-17T16:42:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:08.494386+0000", 2024-09-17T16:42:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 197.37308923399999, 2024-09-17T16:42:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71354715499999999, 2024-09-17T16:42:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:42:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8132 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494386+0000", 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494386+0000", 2024-09-17T16:42:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494387+0000", 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.102e-06 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494407+0000", 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0346000000000001e-05 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494418+0000", 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1202e-05 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494424+0000", 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2809999999999997e-06 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494470+0000", 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5935e-05 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494478+0000", 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9540000000000004e-06 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207922+0000", 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71344410400000002 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207933+0000", 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0231000000000001e-05 2024-09-17T16:42:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:8191 9.8 9:1269694a:::benchmark_data_smithi173_150363_object8190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:09.811424+0000", 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 196.05605024900001, 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51999695400000001, 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8191 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811424+0000", 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811424+0000", 2024-09-17T16:42:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811427+0000", 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3290000000000001e-06 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811450+0000", 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3159999999999998e-05 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811461+0000", 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1252e-05 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811467+0000", 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5790000000000003e-06 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811521+0000", 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4491999999999999e-05 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811668+0000", 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014676700000000001 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811813+0000", 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014480100000000001 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812647+0000", 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083428699999999996 2024-09-17T16:42:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812854+0000", 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020655200000000001 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331303+0000", 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51844964000000004 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331373+0000", 2024-09-17T16:42:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9525999999999998e-05 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331421+0000", 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8569000000000002e-05 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27874.0:8191 9.8 e867/858 ondisk, result = 0)", 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:09.812665+0000", 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 196.05480960200001, 2024-09-17T16:42:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51876305700000003, 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812665+0000", 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812665+0000", 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812666+0000", 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0500000000000003e-07 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812670+0000", 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1729999999999997e-06 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331232+0000", 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51856239699999995 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331237+0000", 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9180000000000002e-06 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331289+0000", 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1681000000000002e-05 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331298+0000", 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7180000000000002e-06 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331428+0000", 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013056499999999999 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:124607 9.8 9:163ec663:::benchmark_data_smithi173_150363_object17702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e919)", 2024-09-17T16:42:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:41:03.496934+0000", 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 82.370540422000005, 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25914015600000001, 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124607 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.496934+0000", 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.496934+0000", 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.496936+0000", 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.336e-06 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.496941+0000", 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4970000000000001e-06 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.496953+0000", 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1685000000000001e-05 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.496959+0000", 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0100000000000001e-06 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.497004+0000", 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5142000000000002e-05 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.497090+0000", 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5538999999999995e-05 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.497202+0000", 2024-09-17T16:42:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112354 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.499101+0000", 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018993899999999999 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.499697+0000", 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059550800000000002 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.755997+0000", 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25629974700000002 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.756040+0000", 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3316999999999999e-05 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.756074+0000", 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4631000000000001e-05 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27874.0:124607 9.8 e919/911 ondisk, result = 0)", 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:41:03.499211+0000", 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 82.368263530999997, 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25686837400000001, 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.499211+0000", 2024-09-17T16:42:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.499211+0000", 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.499212+0000", 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7300000000000005e-07 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.499216+0000", 2024-09-17T16:42:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1409999999999998e-06 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.755914+0000", 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25669820599999998 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.755924+0000", 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1549999999999996e-06 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.755984+0000", 2024-09-17T16:42:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0401e-05 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.755992+0000", 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6009999999999997e-06 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:41:03.756080+0000", 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7997000000000005e-05 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:170709 9.1 9:84f0e101:::benchmark_data_smithi173_150363_object63804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:12.481791+0000", 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 13.385684223, 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33834629300000002, 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 170709 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.481791+0000", 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.481791+0000", 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.481792+0000", 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.006e-06 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.481796+0000", 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5800000000000002e-06 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.481804+0000", 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7640000000000005e-06 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.481810+0000", 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0050000000000001e-06 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.481864+0000", 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4616e-05 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.481987+0000", 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012202299999999999 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,10", 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.482088+0000", 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101406 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.484732+0000", 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002643617 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.484931+0000", 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019957500000000001 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.820033+0000", 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33510154599999997 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.820090+0000", 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7640999999999999e-05 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.820137+0000", 2024-09-17T16:42:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6514000000000002e-05 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:170720 9.1 9:81443729:::benchmark_data_smithi173_150363_object63815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:12.496953+0000", 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 13.370522163, 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.326355754, 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 170720 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.496953+0000", 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.496953+0000", 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.496954+0000", 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3629999999999999e-06 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.496960+0000", 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4409999999999998e-06 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.496973+0000", 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2446000000000001e-05 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.496982+0000", 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9420000000000001e-06 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.497035+0000", 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3168000000000003e-05 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.497130+0000", 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4728000000000005e-05 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,10", 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.497204+0000", 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4328999999999999e-05 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.499435+0000", 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002230671 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.499757+0000", 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032231700000000002 2024-09-17T16:42:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.823204+0000", 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.323447437 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.823261+0000", 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6660999999999998e-05 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.823308+0000", 2024-09-17T16:42:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7250999999999999e-05 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:170731 9.8 9:1102d036:::benchmark_data_smithi173_150363_object63826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e945)", 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:12.525164+0000", 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 13.342310838, 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32264344900000003, 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 170731 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.525164+0000", 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.525164+0000", 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.525166+0000", 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.697e-06 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.525172+0000", 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7789999999999998e-06 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.525187+0000", 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5098e-05 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.525195+0000", 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0910000000000004e-06 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.525244+0000", 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9619e-05 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.525336+0000", 2024-09-17T16:42:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1613000000000005e-05 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.525446+0000", 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110605 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.528094+0000", 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026479379999999999 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.528285+0000", 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019111200000000001 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.847704+0000", 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31941861100000002 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.847759+0000", 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5009999999999997e-05 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:42:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:12.847807+0000", 2024-09-17T16:42:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8276000000000003e-05 2024-09-17T16:42:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:25.971 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:42:25.971 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:25.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:26.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:26.100 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:26.101 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:26.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:26.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:26.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:26.330 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:26.360 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:26.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:26.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:26.462 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:26.487 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:26.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:26.589 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:26.616 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:42:26.617 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:42:26.623 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:26.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:26.651 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:42:26.651 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:42:26.678 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:42:26.678 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:42:26.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:26 smithi017 ceph-mon[27571]: osdmap e954: 12 total, 12 up, 11 in 2024-09-17T16:42:26.706 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:42:26.707 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:42:26.725 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:26.733 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:42:26.733 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:42:26.751 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:26.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:26.764 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:42:26.764 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:42:26.792 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:42:26.792 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:42:26.803 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:26 smithi173 ceph-mon[30530]: osdmap e954: 12 total, 12 up, 11 in 2024-09-17T16:42:26.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:26 smithi071 ceph-mon[30002]: osdmap e954: 12 total, 12 up, 11 in 2024-09-17T16:42:26.818 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:42:26.819 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:42:26.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:42:26.850 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:42:26.850 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:42:26.854 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:26.880 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:26.881 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:42:26.881 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:42:26.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:26.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:26.908 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:42:26.908 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:42:26.937 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:42:26.937 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:42:26.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:27.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:27.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:27.116 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:27.144 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:27.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:27.207 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:27.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:27.273 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:27.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:27.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:42:27.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:27.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:27.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:27.422 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.10 to 1 (10000) 2024-09-17T16:42:27.506 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:27.534 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:27.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:27.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:27.638 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:27 smithi173 ceph-mon[30530]: 9.17 scrub starts 2024-09-17T16:42:27.638 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:27 smithi173 ceph-mon[30530]: pgmap v2102: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3307/104094 objects misplaced (3.177%) 2024-09-17T16:42:27.638 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3517853441' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:27.638 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3726212595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:42:27.662 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:27.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:27.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:27.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:27 smithi017 ceph-mon[27571]: 9.17 scrub starts 2024-09-17T16:42:27.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:27 smithi017 ceph-mon[27571]: pgmap v2102: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3307/104094 objects misplaced (3.177%) 2024-09-17T16:42:27.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3517853441' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:27.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3726212595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:42:27.764 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:27.780 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:27.781 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:27.794 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:27.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:27.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:27 smithi071 ceph-mon[30002]: 9.17 scrub starts 2024-09-17T16:42:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:27 smithi071 ceph-mon[30002]: pgmap v2102: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 3307/104094 objects misplaced (3.177%) 2024-09-17T16:42:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3517853441' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:27.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3726212595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:42:27.896 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:27.921 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:27.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:28.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:28.050 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:28.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:28.151 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:28.180 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:28.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:28.281 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:28.305 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:28.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:28.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:28.432 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:28.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:28.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:28.559 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:28.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:28.596 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-17T16:42:28.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:42:28.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:28.664 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:28.699 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:28.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:28 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3726212595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:42:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:28 smithi017 ceph-mon[27571]: osdmap e955: 12 total, 12 up, 11 in 2024-09-17T16:42:28.800 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:28.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:28 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3726212595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:42:28.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:28 smithi071 ceph-mon[30002]: osdmap e955: 12 total, 12 up, 11 in 2024-09-17T16:42:28.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:28.829 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:28.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:28 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3726212595' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:42:28.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:28 smithi173 ceph-mon[30530]: osdmap e955: 12 total, 12 up, 11 in 2024-09-17T16:42:28.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:28.937 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:28.951 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:28.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:29.052 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:29.081 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:29.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:29.183 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:29.207 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:29.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:29.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:29.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:29.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:29.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:29.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:29.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:29.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:29.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:29.499 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:29.499 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:29.575 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:29.600 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:29.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:29 smithi173 ceph-mon[30530]: pgmap v2104: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1619/99120 objects misplaced (1.633%) 2024-09-17T16:42:29.703 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:29.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:42:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:29.730 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:29.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:29.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:29 smithi017 ceph-mon[27571]: pgmap v2104: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1619/99120 objects misplaced (1.633%) 2024-09-17T16:42:29.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:29 smithi071 ceph-mon[30002]: pgmap v2104: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 1619/99120 objects misplaced (1.633%) 2024-09-17T16:42:29.831 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:29.859 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:29.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:29.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:29.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:29.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:30.090 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:30.118 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:30.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:30.220 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:30.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:42:30.246 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:30.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:30.350 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:30.378 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:30.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:30.480 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:30.507 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:30.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:30.530 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:30 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:30.594 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:30.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:30.630 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:30.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:30 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:30.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:30.758 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:30.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:30.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:30 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:30.859 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:30.882 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:30.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:30.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:31.013 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:31.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:31.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:31.142 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:31.142 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:31.145 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:31.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:31.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:31.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:31.274 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:31.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:31.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:31.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:31.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:31 smithi017 ceph-mon[27571]: pgmap v2105: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 1619/92745 objects misplaced (1.746%); 5.9 MiB/s, 132 objects/s recovering 2024-09-17T16:42:31.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:31.530 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:31.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:31.632 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:31.655 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:31.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:31.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:31.753 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:31 smithi173 ceph-mon[30530]: pgmap v2105: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 1619/92745 objects misplaced (1.746%); 5.9 MiB/s, 132 objects/s recovering 2024-09-17T16:42:31.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:31.786 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:31.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:31.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:31 smithi071 ceph-mon[30002]: pgmap v2105: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 936 op/s; 1619/92745 objects misplaced (1.746%); 5.9 MiB/s, 132 objects/s recovering 2024-09-17T16:42:31.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:31.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:31.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:31.966 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:42:32.023 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:32.050 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:32.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:32.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:32.092 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:32.099 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:32.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:32.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:32.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:32.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:32.309 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:32.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:32.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:32.311 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:42:32.411 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:32.441 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:42:32.446 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:32.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:32.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:32.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/421301061' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:42:32.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:32.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:32.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:32.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:32.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:32.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:32.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/421301061' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:42:32.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:32.837 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:32.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:32.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:32.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/421301061' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:42:32.905 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:32.905 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:32.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:32.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:32.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:33.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:33.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:33.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:33.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:33.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:33.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:33.335 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:33.368 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:33.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:33.470 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:33.500 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:33.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:33.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:33.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:33.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:33.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:33.691 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:33 smithi071 ceph-mon[30002]: pgmap v2106: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 817 op/s; 1619/90531 objects misplaced (1.788%); 4.5 MiB/s, 102 objects/s recovering 2024-09-17T16:42:33.691 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:33 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:33.691 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/421301061' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:33.691 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:33 smithi071 ceph-mon[30002]: osdmap e956: 12 total, 12 up, 11 in 2024-09-17T16:42:33.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:33.738 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:42:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:33 smithi017 ceph-mon[27571]: pgmap v2106: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 817 op/s; 1619/90531 objects misplaced (1.788%); 4.5 MiB/s, 102 objects/s recovering 2024-09-17T16:42:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:33 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/421301061' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:33.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:33 smithi017 ceph-mon[27571]: osdmap e956: 12 total, 12 up, 11 in 2024-09-17T16:42:33.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:33.764 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:33.866 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:33 smithi173 ceph-mon[30530]: pgmap v2106: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 1.8 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 817 op/s; 1619/90531 objects misplaced (1.788%); 4.5 MiB/s, 102 objects/s recovering 2024-09-17T16:42:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:33 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/421301061' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:33.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:33 smithi173 ceph-mon[30530]: osdmap e956: 12 total, 12 up, 11 in 2024-09-17T16:42:33.903 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:33.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:34.005 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:34.042 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:34.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:34.105 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:42:34.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:34.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:34.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:34.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:34.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:34.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:34.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:34.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:34.430 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:34.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:34.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:34.559 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:34.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:34.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:34.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:34.701 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:34.701 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:34.789 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:34.822 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:34.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:34.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:34.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:34.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:34.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:34.956 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:35.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:35.089 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:35.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:35.191 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:35.223 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:35.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:35.325 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:35.366 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:35.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:35.374 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:42:35.445 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.11 to 1 (10000) 2024-09-17T16:42:35.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:35.507 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:35.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:35.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:35.646 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:35.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:35 smithi017 ceph-mon[27571]: pgmap v2108: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 1/84417 objects degraded (0.001%); 656/84417 objects misplaced (0.777%); 6.9 MiB/s, 127 objects/s recovering 2024-09-17T16:42:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/809022506' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:42:35.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:35 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/84417 objects degraded (0.001%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:35.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:35.782 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:35.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:35.798 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:42:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:35 smithi071 ceph-mon[30002]: pgmap v2108: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 1/84417 objects degraded (0.001%); 656/84417 objects misplaced (0.777%); 6.9 MiB/s, 127 objects/s recovering 2024-09-17T16:42:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/809022506' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:42:35.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:35 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/84417 objects degraded (0.001%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:35.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:35 smithi173 ceph-mon[30530]: pgmap v2108: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 1/84417 objects degraded (0.001%); 656/84417 objects misplaced (0.777%); 6.9 MiB/s, 127 objects/s recovering 2024-09-17T16:42:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/809022506' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:42:35.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:35 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/84417 objects degraded (0.001%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:35.913 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:35.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:36.015 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:36.047 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:36.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:36.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:36.167 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:42:36.168 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:42:36.177 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:36.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:36.198 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:42:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:42:36.228 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:42:36.228 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:42:36.253 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:42:36.253 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:42:36.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:36.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:42:36.281 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:42:36.306 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:42:36.307 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:42:36.310 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:36.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:36.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:36.338 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:42:36.338 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:42:36.374 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:42:36.374 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:42:36.389 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:94950 8.15 8:aa553242:::benchmark_data_smithi173_130566_object94949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e769)", 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:35:41.590545+0000", 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 414.79434831899999, 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.470882835, 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.390 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 94950 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590545+0000", 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590545+0000", 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590547+0000", 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4330000000000001e-06 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590601+0000", 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4546000000000003e-05 2024-09-17T16:42:36.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590614+0000", 2024-09-17T16:42:36.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2944e-05 2024-09-17T16:42:36.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590622+0000", 2024-09-17T16:42:36.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9109999999999993e-06 2024-09-17T16:42:36.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590681+0000", 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8641999999999999e-05 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590803+0000", 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000121885 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:42:36.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.590902+0000", 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9684000000000003e-05 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.592520+0000", 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001617261 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:41.599662+0000", 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.007142425 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.061357+0000", 2024-09-17T16:42:36.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.461694565 2024-09-17T16:42:36.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:36.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.061398+0000", 2024-09-17T16:42:36.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0868999999999998e-05 2024-09-17T16:42:36.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:35:42.061428+0000", 2024-09-17T16:42:36.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0670000000000003e-05 2024-09-17T16:42:36.397 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.397 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.397 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "rep_scrubmap(8.d e728 from shard 11)", 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:13.537886+0000", 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 382.84700796999999, 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.66040258699999999, 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537886+0000", 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537886+0000", 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537887+0000", 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2410000000000001e-06 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537958+0000", 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0797999999999995e-05 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.197838+0000", 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.65987990699999999 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.197850+0000", 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2785999999999999e-05 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.197989+0000", 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013900800000000001 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.198027+0000", 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7422999999999997e-05 2024-09-17T16:42:36.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.198288+0000", 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026142400000000001 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(8.11 e849,797 8:88a3dd87:::benchmark_data_smithi173_130566_object54925:head,818'5023)", 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:35.655134+0000", 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 240.72976003799999, 2024-09-17T16:42:36.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.587224894, 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:35.655134+0000", 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:35.655134+0000", 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:35.655134+0000", 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5400000000000001e-07 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:35.655137+0000", 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.858e-06 2024-09-17T16:42:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:36.241838+0000", 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58670063699999997 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:36.241848+0000", 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0524e-05 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:36.241945+0000", 2024-09-17T16:42:36.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6963e-05 2024-09-17T16:42:36.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:36.242359+0000", 2024-09-17T16:42:36.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00041335800000000001 2024-09-17T16:42:36.407 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:42:36.408 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:42:36.408 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.408 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.408 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.409 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(8.11 e850,797 8:8a9fdb5c:::benchmark_data_smithi173_130566_object60988:head,821'5209)", 2024-09-17T16:42:36.409 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:37.435017+0000", 2024-09-17T16:42:36.409 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 238.949876534, 2024-09-17T16:42:36.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.89489246099999997, 2024-09-17T16:42:36.410 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.410 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.410 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:36.410 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:37.435017+0000", 2024-09-17T16:42:36.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:37.435017+0000", 2024-09-17T16:42:36.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:37.435018+0000", 2024-09-17T16:42:36.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3e-07 2024-09-17T16:42:36.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:37.435023+0000", 2024-09-17T16:42:36.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1019999999999996e-06 2024-09-17T16:42:36.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:38.328802+0000", 2024-09-17T16:42:36.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.89377951200000005 2024-09-17T16:42:36.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:38.328820+0000", 2024-09-17T16:42:36.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7099999999999999e-05 2024-09-17T16:42:36.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:38.328942+0000", 2024-09-17T16:42:36.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012231199999999999 2024-09-17T16:42:36.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:38.329910+0000", 2024-09-17T16:42:36.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00096770500000000002 2024-09-17T16:42:36.417 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.417 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.417 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.418 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122229 9.d 9:b6455253:::benchmark_data_smithi173_150363_object15324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:42:36.418 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.212600+0000", 2024-09-17T16:42:36.418 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 103.172293736, 2024-09-17T16:42:36.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9547618619999998, 2024-09-17T16:42:36.418 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.419 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.419 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:36.419 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.419 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.419 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.419 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122229 2024-09-17T16:42:36.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.420 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212600+0000", 2024-09-17T16:42:36.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212600+0000", 2024-09-17T16:42:36.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212602+0000", 2024-09-17T16:42:36.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2520000000000002e-06 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212610+0000", 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9200000000000004e-06 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212846+0000", 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000235935 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212856+0000", 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0355e-05 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.155467+0000", 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9426101920000001 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.155546+0000", 2024-09-17T16:42:36.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8991000000000003e-05 2024-09-17T16:42:36.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:42:36.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.155611+0000", 2024-09-17T16:42:36.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5338000000000004e-05 2024-09-17T16:42:36.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163191+0000", 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0075802889999999996 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167191+0000", 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040001250000000002 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167254+0000", 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3126000000000005e-05 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:36.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167314+0000", 2024-09-17T16:42:36.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9871999999999999e-05 2024-09-17T16:42:36.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167362+0000", 2024-09-17T16:42:36.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7466999999999998e-05 2024-09-17T16:42:36.426 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.426 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122253 9.d 9:b93e8df4:::benchmark_data_smithi173_150363_object15348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.259873+0000", 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 103.125021148, 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9117644189999998, 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.427 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122253 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259873+0000", 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259873+0000", 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259874+0000", 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9010000000000001e-06 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259884+0000", 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5370000000000003e-06 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259903+0000", 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8630999999999998e-05 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259914+0000", 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1221e-05 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159148+0000", 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8992343539999998 2024-09-17T16:42:36.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159272+0000", 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012383199999999999 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159390+0000", 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011834500000000001 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167403+0000", 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0080130819999999995 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167441+0000", 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7360000000000001e-05 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:36.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171540+0000", 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040992700000000003 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171594+0000", 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3761000000000002e-05 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171637+0000", 2024-09-17T16:42:36.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3124999999999998e-05 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122272 9.d 9:bbc9765a:::benchmark_data_smithi173_150363_object15367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.299568+0000", 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 103.08532610899999, 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8722430069999998, 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:36.434 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122272 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299568+0000", 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.435 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299568+0000", 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299569+0000", 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6789999999999999e-06 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299576+0000", 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5239999999999997e-06 2024-09-17T16:42:36.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299590+0000", 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3925e-05 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299598+0000", 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6170000000000007e-06 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163236+0000", 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8636377890000002 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163356+0000", 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000119739 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163465+0000", 2024-09-17T16:42:36.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000108793 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167475+0000", 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040103140000000001 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171687+0000", 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0042120179999999997 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171717+0000", 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9887999999999999e-05 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171776+0000", 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9509999999999998e-05 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171811+0000", 2024-09-17T16:42:36.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4211000000000002e-05 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122285 9.d 9:b7f34b49:::benchmark_data_smithi173_150363_object15380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.331869+0000", 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 103.05302455099999, 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8440283700000002, 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.441 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122285 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331869+0000", 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331869+0000", 2024-09-17T16:42:36.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331871+0000", 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5540000000000001e-06 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331877+0000", 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1480000000000003e-06 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331891+0000", 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4338999999999999e-05 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331898+0000", 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2609999999999996e-06 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167505+0000", 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8356064239999998 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.445 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167651+0000", 2024-09-17T16:42:36.445 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014661599999999999 2024-09-17T16:42:36.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:42:36.446 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167776+0000", 2024-09-17T16:42:36.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000124215 2024-09-17T16:42:36.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.446 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171676+0000", 2024-09-17T16:42:36.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0039002640000000001 2024-09-17T16:42:36.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175793+0000", 2024-09-17T16:42:36.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0041174790000000003 2024-09-17T16:42:36.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.448 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:36.448 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175805+0000", 2024-09-17T16:42:36.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1253999999999999e-05 2024-09-17T16:42:36.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:36.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175855+0000", 2024-09-17T16:42:36.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0195000000000003e-05 2024-09-17T16:42:36.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175897+0000", 2024-09-17T16:42:36.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2620999999999997e-05 2024-09-17T16:42:36.450 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.450 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.450 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.450 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27874.0:122289 9.5 e916/911 9:a636923a:::benchmark_data_smithi173_150363_object15384:head v 916'7746, mlcod=913'7744)", 2024-09-17T16:42:36.450 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.359382+0000", 2024-09-17T16:42:36.450 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 103.025511944, 2024-09-17T16:42:36.451 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7997341159999998, 2024-09-17T16:42:36.451 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.451 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.451 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:36.451 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.451 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359382+0000", 2024-09-17T16:42:36.452 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359382+0000", 2024-09-17T16:42:36.452 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359383+0000", 2024-09-17T16:42:36.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0109999999999999e-06 2024-09-17T16:42:36.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359391+0000", 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.366010+0000", 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0066186120000000003 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.366018+0000", 2024-09-17T16:42:36.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9409999999999996e-06 2024-09-17T16:42:36.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.455 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.152980+0000", 2024-09-17T16:42:36.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7869612779999997 2024-09-17T16:42:36.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.455 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.153008+0000", 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8198e-05 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159064+0000", 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0060564929999999996 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159116+0000", 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1403999999999998e-05 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.456 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.457 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.457 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122305 9.d 9:bbff1c34:::benchmark_data_smithi173_150363_object15400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:42:36.457 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.420875+0000", 2024-09-17T16:42:36.457 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 102.964018849, 2024-09-17T16:42:36.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7590298520000003, 2024-09-17T16:42:36.457 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.458 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.458 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:36.458 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.458 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.458 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.458 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122305 2024-09-17T16:42:36.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.458 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420875+0000", 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420875+0000", 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420876+0000", 2024-09-17T16:42:36.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.449e-06 2024-09-17T16:42:36.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420882+0000", 2024-09-17T16:42:36.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6110000000000001e-06 2024-09-17T16:42:36.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420893+0000", 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1562999999999999e-05 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420900+0000", 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7179999999999999e-06 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171839+0000", 2024-09-17T16:42:36.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7509386400000002 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171976+0000", 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013730900000000001 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.172054+0000", 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8250999999999993e-05 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175930+0000", 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038752579999999999 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179802+0000", 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.003872084 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179813+0000", 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0951999999999999e-05 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179859+0000", 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6289999999999999e-05 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179905+0000", 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5726999999999998e-05 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.464 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122326 9.d 9:bd61e234:::benchmark_data_smithi173_150363_object15421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:42:36.465 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.786589+0000", 2024-09-17T16:42:36.465 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 102.59830495999999, 2024-09-17T16:42:36.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4114004009999999, 2024-09-17T16:42:36.465 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.465 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.465 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:36.465 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.465 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.465 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.465 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122326 2024-09-17T16:42:36.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.466 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.466 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786589+0000", 2024-09-17T16:42:36.466 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.466 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:36.467 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:42:36.468 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:42:36.468 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786589+0000", 2024-09-17T16:42:36.468 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.469 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.469 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.469 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.469 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786591+0000", 2024-09-17T16:42:36.469 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7940000000000001e-06 2024-09-17T16:42:36.469 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.469 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.469 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786598+0000", 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5830000000000001e-06 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786616+0000", 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8210999999999999e-05 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786626+0000", 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7380000000000006e-06 2024-09-17T16:42:36.470 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175957+0000", 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3893305729999996 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.176075+0000", 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000118678 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:42:36.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.176185+0000", 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011011800000000001 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179935+0000", 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.003749877 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.192607+0000", 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.01267193 2024-09-17T16:42:36.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.197883+0000", 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0052760949999999997 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.197938+0000", 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4676000000000003e-05 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.473 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.197989+0000", 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1128000000000002e-05 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27874.0:122329 9.5 e916/911 9:a441955e:::benchmark_data_smithi173_150363_object15424:head v 916'7747, mlcod=913'7744)", 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.796651+0000", 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 102.58824276599999, 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3665418100000002, 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:36.474 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.475 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.475 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.475 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796651+0000", 2024-09-17T16:42:36.475 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.475 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.475 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.475 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.475 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796651+0000", 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796652+0000", 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2999999999999999e-07 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796659+0000", 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9929999999999998e-06 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.476 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.801428+0000", 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0047693509999999998 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.801435+0000", 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3810000000000001e-06 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.154575+0000", 2024-09-17T16:42:36.477 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3531394639999998 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.154586+0000", 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1387e-05 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163139+0000", 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0085529979999999992 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.478 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.479 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163193+0000", 2024-09-17T16:42:36.479 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3406000000000003e-05 2024-09-17T16:42:36.479 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.479 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.479 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.479 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.479 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122341 9.d 9:b2726d38:::benchmark_data_smithi173_150363_object15436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:42:36.479 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.879538+0000", 2024-09-17T16:42:36.479 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 102.505355642, 2024-09-17T16:42:36.480 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3567290740000004, 2024-09-17T16:42:36.480 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.480 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.480 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:36.480 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.480 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.480 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.480 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122341 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879538+0000", 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879538+0000", 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879540+0000", 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.95e-06 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879548+0000", 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6899999999999992e-06 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.482 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.483 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879563+0000", 2024-09-17T16:42:36.483 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5814000000000001e-05 2024-09-17T16:42:36.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.483 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.483 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.483 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879574+0000", 2024-09-17T16:42:36.483 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0594999999999999e-05 2024-09-17T16:42:36.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.484 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.484 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.484 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179959+0000", 2024-09-17T16:42:36.484 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3003848480000002 2024-09-17T16:42:36.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.484 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.484 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:36.485 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.180062+0000", 2024-09-17T16:42:36.485 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000103226 2024-09-17T16:42:36.485 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.485 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.485 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:42:36.485 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.180161+0000", 2024-09-17T16:42:36.486 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8348000000000001e-05 2024-09-17T16:42:36.486 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.486 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.487 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.487 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.192567+0000", 2024-09-17T16:42:36.487 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.012406188 2024-09-17T16:42:36.487 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.487 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.487 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:36.488 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.192660+0000", 2024-09-17T16:42:36.488 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3721999999999995e-05 2024-09-17T16:42:36.488 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.488 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.488 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:36.488 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.236167+0000", 2024-09-17T16:42:36.489 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.043506396000000003 2024-09-17T16:42:36.489 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.489 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.489 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:36.489 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.236222+0000", 2024-09-17T16:42:36.490 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5356000000000003e-05 2024-09-17T16:42:36.490 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.490 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.490 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.490 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.236267+0000", 2024-09-17T16:42:36.490 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4941000000000002e-05 2024-09-17T16:42:36.491 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.491 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.491 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.491 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.492 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.492 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122171 9.15 9:aa774fbb:::benchmark_data_smithi173_150363_object15266:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:42:36.492 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.335919+0000", 2024-09-17T16:42:36.492 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 98.048975085999999, 2024-09-17T16:42:36.492 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8620733170000001, 2024-09-17T16:42:36.493 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.493 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.493 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:36.493 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.493 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.493 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.494 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122171 2024-09-17T16:42:36.494 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.494 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.494 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.494 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.495 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335919+0000", 2024-09-17T16:42:36.495 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.495 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.495 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.495 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.495 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335919+0000", 2024-09-17T16:42:36.496 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.496 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.496 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.496 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.496 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335920+0000", 2024-09-17T16:42:36.497 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8449999999999999e-06 2024-09-17T16:42:36.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.497 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.497 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.497 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335926+0000", 2024-09-17T16:42:36.497 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5969999999999999e-06 2024-09-17T16:42:36.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.497 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.497 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.497 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335937+0000", 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1409e-05 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335944+0000", 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8539999999999999e-06 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.193009+0000", 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8570646320000002 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.498 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.197992+0000", 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0049829799999999997 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122218 9.15 9:ac90a645:::benchmark_data_smithi173_150363_object15313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.335968+0000", 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 98.048925295000004, 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8804516200000001, 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.499 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122218 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335968+0000", 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.500 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335968+0000", 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335969+0000", 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9000000000000002e-07 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335971+0000", 2024-09-17T16:42:36.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8789999999999999e-06 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335978+0000", 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8019999999999992e-06 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335984+0000", 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2680000000000002e-06 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.198340+0000", 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8623560939999999 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.216420+0000", 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.018080187000000001 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.503 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122234 9.15 9:ac0fc00a:::benchmark_data_smithi173_150363_object15329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.335994+0000", 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 98.048899707999993, 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.90027729, 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122234 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.504 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335994+0000", 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335994+0000", 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335994+0000", 2024-09-17T16:42:36.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4999999999999998e-07 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335997+0000", 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.441e-06 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336004+0000", 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2810000000000003e-06 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.507 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336008+0000", 2024-09-17T16:42:36.507 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4020000000000002e-06 2024-09-17T16:42:36.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.507 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.507 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.216401+0000", 2024-09-17T16:42:36.507 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.880393312 2024-09-17T16:42:36.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.236271+0000", 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.019870404000000001 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122249 9.15 9:ac85c97a:::benchmark_data_smithi173_150363_object15344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.336012+0000", 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 98.048881244, 2024-09-17T16:42:36.508 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.923836176, 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122249 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336012+0000", 2024-09-17T16:42:36.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336012+0000", 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336013+0000", 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6899999999999999e-07 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336014+0000", 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4810000000000001e-06 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336017+0000", 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9809999999999999e-06 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336021+0000", 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3589999999999999e-06 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.259055+0000", 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.923034589 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.259849+0000", 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000793497 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122256 9.15 9:a9803287:::benchmark_data_smithi173_150363_object15351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.336030+0000", 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 98.048863621999999, 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.924521964, 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.513 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122256 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336030+0000", 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336030+0000", 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336030+0000", 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6100000000000002e-07 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336032+0000", 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5880000000000001e-06 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336037+0000", 2024-09-17T16:42:36.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9690000000000002e-06 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336040+0000", 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.259881+0000", 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9238411449999999 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.260552+0000", 2024-09-17T16:42:36.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00067100099999999995 2024-09-17T16:42:36.517 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.517 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.517 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.517 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122261 9.15 9:aa7ff69f:::benchmark_data_smithi173_150363_object15356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:42:36.517 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.336044+0000", 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 98.048850162999997, 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9253561939999999, 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122261 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336044+0000", 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336044+0000", 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336044+0000", 2024-09-17T16:42:36.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2499999999999999e-07 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336045+0000", 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3060000000000001e-06 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336048+0000", 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9239999999999999e-06 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336051+0000", 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8540000000000001e-06 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.260584+0000", 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9245329839999998 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.261400+0000", 2024-09-17T16:42:36.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00081590099999999995 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122284 9.15 9:aff350b8:::benchmark_data_smithi173_150363_object15379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.336057+0000", 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 98.048837028999998, 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9260394750000001, 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:36.522 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122284 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336057+0000", 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336057+0000", 2024-09-17T16:42:36.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:36.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:36.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336057+0000", 2024-09-17T16:42:36.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7599999999999998e-07 2024-09-17T16:42:36.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336058+0000", 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.519e-06 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336062+0000", 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0759999999999999e-06 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336064+0000", 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.666e-06 2024-09-17T16:42:36.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.261433+0000", 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9253688499999999 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.262096+0000", 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00066308800000000005 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.526 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.527 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:36.527 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:36.527 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:42:36.527 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:42:36.527 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:42:36.528 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:42:36.528 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:42:36.556 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:36.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:36.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/809022506' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:42:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:36 smithi071 ceph-mon[30002]: osdmap e957: 12 total, 12 up, 11 in 2024-09-17T16:42:36.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1974044905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:36.578 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/809022506' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:42:36.578 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:36 smithi017 ceph-mon[27571]: osdmap e957: 12 total, 12 up, 11 in 2024-09-17T16:42:36.578 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1974044905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:36.619 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T16:42:36.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:36.690 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:36.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:36.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:36.820 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:36.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/809022506' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:42:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:36 smithi173 ceph-mon[30530]: osdmap e957: 12 total, 12 up, 11 in 2024-09-17T16:42:36.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1974044905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:36.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:36.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:36.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:37.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:37.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:37.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:37.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:37.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:37.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:37.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:37.338 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:37.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:37.440 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:37.444 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:37.465 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:37.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:37.568 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:37.574 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:37 smithi017 ceph-mon[27571]: pgmap v2110: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 916 op/s; 1/77112 objects degraded (0.001%); 656/77112 objects misplaced (0.851%); 2.5 MiB/s, 57 objects/s recovering 2024-09-17T16:42:37.595 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:37.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:37.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:42:37.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:37.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:37.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:37.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:37 smithi071 ceph-mon[30002]: pgmap v2110: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 916 op/s; 1/77112 objects degraded (0.001%); 656/77112 objects misplaced (0.851%); 2.5 MiB/s, 57 objects/s recovering 2024-09-17T16:42:37.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:37.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:37.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:37.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:37.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:37 smithi173 ceph-mon[30530]: pgmap v2110: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 916 op/s; 1/77112 objects degraded (0.001%); 656/77112 objects misplaced (0.851%); 2.5 MiB/s, 57 objects/s recovering 2024-09-17T16:42:37.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:37.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:37.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:38.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:38.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:38.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:38.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:38.156 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:38.201 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:38.230 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:38.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:38.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:38.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:38.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:38.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:38.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:38.456 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:38.461 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T16:42:38.493 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:38.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:38.595 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:38.625 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:38.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:38.731 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:38.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2511919675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:42:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:38.761 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:38.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:38.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2511919675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:42:38.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:38.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:38.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:38.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2511919675' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:42:38.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:39.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:39.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:39.114 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:39.142 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:39.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:39.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:39.245 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:39.267 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:39.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:39.336 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:42:39.369 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:39.393 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:39.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:39.442 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T16:42:39.473 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:42:39.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:39.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:39.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:39.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:39 smithi071 ceph-mon[30002]: pgmap v2111: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 952 op/s; 1/69891 objects degraded (0.001%); 4.2 MiB/s, 133 objects/s recovering 2024-09-17T16:42:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2511919675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:42:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:39 smithi071 ceph-mon[30002]: osdmap e958: 12 total, 12 up, 11 in 2024-09-17T16:42:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2907588635' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:42:39.624 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:39.650 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:39.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:39.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:39 smithi017 ceph-mon[27571]: pgmap v2111: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 952 op/s; 1/69891 objects degraded (0.001%); 4.2 MiB/s, 133 objects/s recovering 2024-09-17T16:42:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2511919675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:42:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:39 smithi017 ceph-mon[27571]: osdmap e958: 12 total, 12 up, 11 in 2024-09-17T16:42:39.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2907588635' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:42:39.750 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:39.782 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:39.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:39.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:39.796 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:39 smithi173 ceph-mon[30530]: pgmap v2111: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.4 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 952 op/s; 1/69891 objects degraded (0.001%); 4.2 MiB/s, 133 objects/s recovering 2024-09-17T16:42:39.796 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2511919675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:42:39.797 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:39 smithi173 ceph-mon[30530]: osdmap e958: 12 total, 12 up, 11 in 2024-09-17T16:42:39.797 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2907588635' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:42:39.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:42:39.884 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:39.905 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:39.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:40.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:40.029 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:40.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:40.100 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:40.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:40.153 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:40.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:40.255 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:40.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:40.279 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:40.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:40.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:40.403 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:40.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:40.505 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:40.533 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:40.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:40.595 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:40 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:40.595 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2907588635' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:42:40.595 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:40 smithi173 ceph-mon[30530]: osdmap e959: 12 total, 12 up, 11 in 2024-09-17T16:42:40.595 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:40 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1/69891 objects degraded (0.001%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:40.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:40.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:40.666 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:40 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2907588635' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:42:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:40 smithi017 ceph-mon[27571]: osdmap e959: 12 total, 12 up, 11 in 2024-09-17T16:42:40.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:40 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1/69891 objects degraded (0.001%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:40.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:40.788 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:40.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:40 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2907588635' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:42:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:40 smithi071 ceph-mon[30002]: osdmap e959: 12 total, 12 up, 11 in 2024-09-17T16:42:40.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:40 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1/69891 objects degraded (0.001%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-17T16:42:40.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:40.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:40.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:41.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:41.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:41.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:41.137 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:41.166 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:41.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:41.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:41.296 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:41.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:41.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:41.417 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:42:41.417 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:42:41.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:193568 9.9 9:95fde147:::benchmark_data_smithi173_150363_object86663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:40.584532+0000", 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.14582105100000001, 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14585645999999999, 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193568 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:42:41.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.584532+0000", 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.584532+0000", 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:42:41.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.584533+0000", 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4330000000000001e-06 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.584539+0000", 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3990000000000003e-06 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.584550+0000", 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1066999999999999e-05 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.584557+0000", 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8850000000000001e-06 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.584648+0000", 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1855999999999996e-05 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.584829+0000", 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018023599999999999 2024-09-17T16:42:41.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.585056+0000", 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022746499999999999 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.589699+0000", 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046431290000000002 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.589712+0000", 2024-09-17T16:42:41.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2493e-05 2024-09-17T16:42:41.423 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:41.423 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:42:41.423 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:41.423 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:42:41.423 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-09-17T16:42:41.423 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-09-17T16:42:41.423 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:42:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:41.424 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:41.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:41.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:41.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:41.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:41.662 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:41.688 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:41.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:41 smithi017 ceph-mon[27571]: pgmap v2114: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 1/64653 objects degraded (0.002%); 2.3 MiB/s, 100 objects/s recovering 2024-09-17T16:42:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:41 smithi017 ceph-mon[27571]: osdmap e960: 12 total, 12 up, 11 in 2024-09-17T16:42:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:41 smithi017 ceph-mon[27571]: 9.8 scrub starts 2024-09-17T16:42:41.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:41 smithi017 ceph-mon[27571]: 9.8 scrub ok 2024-09-17T16:42:41.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:41.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:41 smithi173 ceph-mon[30530]: pgmap v2114: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 1/64653 objects degraded (0.002%); 2.3 MiB/s, 100 objects/s recovering 2024-09-17T16:42:41.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:41 smithi173 ceph-mon[30530]: osdmap e960: 12 total, 12 up, 11 in 2024-09-17T16:42:41.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:41 smithi173 ceph-mon[30530]: 9.8 scrub starts 2024-09-17T16:42:41.808 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:41 smithi173 ceph-mon[30530]: 9.8 scrub ok 2024-09-17T16:42:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:41 smithi071 ceph-mon[30002]: pgmap v2114: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 1/64653 objects degraded (0.002%); 2.3 MiB/s, 100 objects/s recovering 2024-09-17T16:42:41.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:41 smithi071 ceph-mon[30002]: osdmap e960: 12 total, 12 up, 11 in 2024-09-17T16:42:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:41 smithi071 ceph-mon[30002]: 9.8 scrub starts 2024-09-17T16:42:41.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:41 smithi071 ceph-mon[30002]: 9.8 scrub ok 2024-09-17T16:42:41.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:41.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:41.921 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:41.956 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:41.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:41.972 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:41.991 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:42.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:42.090 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:42.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:42.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:42.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:42.227 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:42.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:42.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:42.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:42:42.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:42.354 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:42.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:42.456 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:42.491 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:42.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:42.580 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:42.593 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:42.625 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:42.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:42.643 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:42 smithi017 ceph-mon[27571]: 9.16 scrub starts 2024-09-17T16:42:42.644 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:42 smithi017 ceph-mon[27571]: 9.16 scrub ok 2024-09-17T16:42:42.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:42.734 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:42 smithi173 ceph-mon[30530]: 9.16 scrub starts 2024-09-17T16:42:42.734 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:42 smithi173 ceph-mon[30530]: 9.16 scrub ok 2024-09-17T16:42:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:42.758 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:42.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:42.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:42 smithi071 ceph-mon[30002]: 9.16 scrub starts 2024-09-17T16:42:42.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:42 smithi071 ceph-mon[30002]: 9.16 scrub ok 2024-09-17T16:42:42.818 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:42:42.818 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:42:42.852 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:42:42.852 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:42:42.859 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:42.877 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:42:42.877 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:42:42.889 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:42.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:42.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:42.904 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:42:42.905 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:42:42.934 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:42:42.934 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:42:42.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:42:42.974 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:42:42.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:43.001 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:42:43.001 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:42:43.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:43.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:43.040 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:42:43.040 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:42:43.073 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:42:43.074 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:42:43.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:42:43.106 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:42:43.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:43.133 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:42:43.133 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:42:43.133 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:43.134 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:43.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:43.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:43.159 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:42:43.160 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:42:43.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:43.285 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:43.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:43.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:43.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:43.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:43.490 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T16:42:43.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:43.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:43.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:43.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:43.667 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:43.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:43.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:43.737 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:43 smithi173 ceph-mon[30530]: pgmap v2116: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail 2024-09-17T16:42:43.737 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3493485778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:43.737 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2777918485' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:42:43.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:43 smithi017 ceph-mon[27571]: pgmap v2116: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail 2024-09-17T16:42:43.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3493485778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:43.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2777918485' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:42:43.742 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:43 smithi071 ceph-mon[30002]: pgmap v2116: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail 2024-09-17T16:42:43.743 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3493485778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:43.743 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2777918485' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:42:43.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:43.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:43.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:43.794 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:43.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:43.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:43.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:43.972 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:42:43.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:42:44.021 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:44.072 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:44.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:44.174 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:44.197 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:44.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:44.225 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:42:44.298 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:44.322 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:44.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:44.424 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:44.444 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:44.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:44.547 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:44.570 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:44.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:44.634 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2777918485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:42:44.635 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:44 smithi173 ceph-mon[30530]: osdmap e961: 12 total, 12 up, 11 in 2024-09-17T16:42:44.635 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:44 smithi173 ceph-mon[30530]: 9.d scrub starts 2024-09-17T16:42:44.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:44.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:44.711 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:44.729 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2777918485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:42:44.730 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:44 smithi017 ceph-mon[27571]: osdmap e961: 12 total, 12 up, 11 in 2024-09-17T16:42:44.730 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:44 smithi017 ceph-mon[27571]: 9.d scrub starts 2024-09-17T16:42:44.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T16:42:44.775 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:44.775 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:44.812 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:44.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2777918485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:42:44.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:44 smithi071 ceph-mon[30002]: osdmap e961: 12 total, 12 up, 11 in 2024-09-17T16:42:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:44 smithi071 ceph-mon[30002]: 9.d scrub starts 2024-09-17T16:42:44.847 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:44.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:44.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:44.893 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:44.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:44.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:44.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:45.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:45.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:45.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:45.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:45.246 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:45.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:45.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:45.377 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:45.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:45.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:45.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:45.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:45.511 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:45.513 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:42:45.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:45.631 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:45.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:45.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:45.636 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:45 smithi017 ceph-mon[27571]: pgmap v2118: 37 pgs: 1 active+undersized+remapped, 36 active+clean; 1.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail 2024-09-17T16:42:45.637 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:45 smithi017 ceph-mon[27571]: osdmap e962: 12 total, 12 up, 11 in 2024-09-17T16:42:45.637 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:45 smithi017 ceph-mon[27571]: 9.d scrub ok 2024-09-17T16:42:45.637 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:45 smithi017 ceph-mon[27571]: 9.4 scrub starts 2024-09-17T16:42:45.637 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:45 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-17T16:42:45.733 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:45.772 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:45.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:45.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:45 smithi071 ceph-mon[30002]: pgmap v2118: 37 pgs: 1 active+undersized+remapped, 36 active+clean; 1.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail 2024-09-17T16:42:45.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:45 smithi071 ceph-mon[30002]: osdmap e962: 12 total, 12 up, 11 in 2024-09-17T16:42:45.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:45 smithi071 ceph-mon[30002]: 9.d scrub ok 2024-09-17T16:42:45.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:45 smithi071 ceph-mon[30002]: 9.4 scrub starts 2024-09-17T16:42:45.816 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:45 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-17T16:42:45.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:45.894 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:42:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:45 smithi173 ceph-mon[30530]: pgmap v2118: 37 pgs: 1 active+undersized+remapped, 36 active+clean; 1.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail 2024-09-17T16:42:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:45 smithi173 ceph-mon[30530]: osdmap e962: 12 total, 12 up, 11 in 2024-09-17T16:42:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:45 smithi173 ceph-mon[30530]: 9.d scrub ok 2024-09-17T16:42:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:45 smithi173 ceph-mon[30530]: 9.4 scrub starts 2024-09-17T16:42:45.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:45 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-17T16:42:45.902 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:45.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:46.004 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:46.010 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:42:46.035 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:46.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:46.137 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:46.167 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:46.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:46.269 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:46.297 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:46.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:46.395 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:46.399 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:46.431 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:46.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:46.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:46.531 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T16:42:46.532 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:42:46.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:46.570 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:46.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:46.631 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:46 smithi071 ceph-mon[30002]: 9.4 scrub ok 2024-09-17T16:42:46.631 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:46 smithi071 ceph-mon[30002]: 9.10 scrub starts 2024-09-17T16:42:46.631 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:46 smithi071 ceph-mon[30002]: 9.10 scrub ok 2024-09-17T16:42:46.631 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/414901664' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:42:46.631 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3660985154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:42:46.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:46.698 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:46.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:46.789 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:46.789 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:46.800 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:46.834 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:46.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:46 smithi173 ceph-mon[30530]: 9.4 scrub ok 2024-09-17T16:42:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:46 smithi173 ceph-mon[30530]: 9.10 scrub starts 2024-09-17T16:42:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:46 smithi173 ceph-mon[30530]: 9.10 scrub ok 2024-09-17T16:42:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/414901664' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:42:46.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3660985154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:42:46.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:46.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:46.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:46.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:46 smithi017 ceph-mon[27571]: 9.4 scrub ok 2024-09-17T16:42:46.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:46 smithi017 ceph-mon[27571]: 9.10 scrub starts 2024-09-17T16:42:46.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:46 smithi017 ceph-mon[27571]: 9.10 scrub ok 2024-09-17T16:42:46.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/414901664' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:42:46.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3660985154' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:42:47.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:47.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:47.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:47.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:47.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:47.221 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:47.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:47.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:47.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:47.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:47.433 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:42:47.433 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:42:47.450 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:47.462 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:42:47.462 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:42:47.481 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:47.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:47.492 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:42:47.493 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:42:47.520 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:42:47.520 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:42:47.548 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:42:47.549 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:42:47.549 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:47 smithi017 ceph-mon[27571]: pgmap v2120: 37 pgs: 37 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail 2024-09-17T16:42:47.549 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:47 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T16:42:47.549 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:47 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:42:47.549 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/414901664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:47.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3660985154' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:42:47.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:47 smithi017 ceph-mon[27571]: osdmap e963: 12 total, 12 up, 11 in 2024-09-17T16:42:47.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:47 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:42:47.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:47 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:42:47.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/704131521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:47.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:42:47.581 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:42:47.581 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:42:47.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:47.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:47 smithi071 ceph-mon[30002]: pgmap v2120: 37 pgs: 37 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail 2024-09-17T16:42:47.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:47 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T16:42:47.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:47 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:42:47.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/414901664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:47.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3660985154' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:42:47.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:47 smithi071 ceph-mon[30002]: osdmap e963: 12 total, 12 up, 11 in 2024-09-17T16:42:47.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:47 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:42:47.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:47 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:42:47.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/704131521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:47.609 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:42:47.609 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:42:47.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:47.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:47.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:47.636 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:42:47.636 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:42:47.664 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:42:47.664 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:42:47.693 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:42:47.693 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:42:47.712 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:47.726 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:42:47.726 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:42:47.738 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:47.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:47.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:47.752 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:42:47.752 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:42:47.840 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:47.879 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:47.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:47 smithi173 ceph-mon[30530]: pgmap v2120: 37 pgs: 37 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail 2024-09-17T16:42:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:47 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-17T16:42:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:47 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:42:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/414901664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3660985154' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:42:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:47 smithi173 ceph-mon[30530]: osdmap e963: 12 total, 12 up, 11 in 2024-09-17T16:42:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:47 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:42:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:47 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:42:47.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/704131521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:47.952 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:42:47.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:48.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:48.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:48.108 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:48.132 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:48.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:48.234 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:48.262 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:48.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:48.364 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:48.393 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:48.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:48.495 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:48.520 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:48.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:48.552 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:42:48.552 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:42:48.552 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:42:48.552 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:42:48.553 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:42:48.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:48.659 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:48.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:48.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:48 smithi017 ceph-mon[27571]: 9.0 scrub starts 2024-09-17T16:42:48.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:48 smithi017 ceph-mon[27571]: 9.0 scrub ok 2024-09-17T16:42:48.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:48.785 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:48.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:48.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:48 smithi071 ceph-mon[30002]: 9.0 scrub starts 2024-09-17T16:42:48.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:48 smithi071 ceph-mon[30002]: 9.0 scrub ok 2024-09-17T16:42:48.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:48 smithi173 ceph-mon[30530]: 9.0 scrub starts 2024-09-17T16:42:48.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:48 smithi173 ceph-mon[30530]: 9.0 scrub ok 2024-09-17T16:42:48.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:48.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:49.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:49.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:49.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:49.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:49.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:49.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:49.227 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T16:42:49.227 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:42:49.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:49.309 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:49.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:49.391 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:42:49.411 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:49.437 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:49.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:49.539 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:49.568 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:49.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:49.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:49.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:49.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:49.741 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:42:49.780 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:49.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:49.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:49 smithi071 ceph-mon[30002]: pgmap v2122: 37 pgs: 37 active+clean; 990 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 0 B/s, 58 objects/s recovering 2024-09-17T16:42:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:49 smithi071 ceph-mon[30002]: 9.1 scrub starts 2024-09-17T16:42:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:49 smithi071 ceph-mon[30002]: 9.c scrub starts 2024-09-17T16:42:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:49 smithi071 ceph-mon[30002]: 9.c scrub ok 2024-09-17T16:42:49.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:49 smithi071 ceph-mon[30002]: 9.1 scrub ok 2024-09-17T16:42:49.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:49.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:49 smithi173 ceph-mon[30530]: pgmap v2122: 37 pgs: 37 active+clean; 990 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 0 B/s, 58 objects/s recovering 2024-09-17T16:42:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:49 smithi173 ceph-mon[30530]: 9.1 scrub starts 2024-09-17T16:42:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:49 smithi173 ceph-mon[30530]: 9.c scrub starts 2024-09-17T16:42:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:49 smithi173 ceph-mon[30530]: 9.c scrub ok 2024-09-17T16:42:49.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:49 smithi173 ceph-mon[30530]: 9.1 scrub ok 2024-09-17T16:42:49.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:49 smithi017 ceph-mon[27571]: pgmap v2122: 37 pgs: 37 active+clean; 990 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 848 op/s; 0 B/s, 58 objects/s recovering 2024-09-17T16:42:49.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:49 smithi017 ceph-mon[27571]: 9.1 scrub starts 2024-09-17T16:42:49.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:49 smithi017 ceph-mon[27571]: 9.c scrub starts 2024-09-17T16:42:49.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:49 smithi017 ceph-mon[27571]: 9.c scrub ok 2024-09-17T16:42:49.907 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:49 smithi017 ceph-mon[27571]: 9.1 scrub ok 2024-09-17T16:42:49.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:49.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:49.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:50.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:50.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:50.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:50.169 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:42:50.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:50.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:50.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:50.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:50.373 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:157496 8.4 8:2002af7a:::benchmark_data_smithi173_130566_object62521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e822)", 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:48.711848+0000", 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 301.61342350799998, 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52764325700000003, 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.374 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 157496 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711848+0000", 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711848+0000", 2024-09-17T16:42:50.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711849+0000", 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1149999999999999e-06 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711853+0000", 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1370000000000004e-06 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711864+0000", 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0913e-05 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711870+0000", 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6130000000000003e-06 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711918+0000", 2024-09-17T16:42:50.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8829999999999998e-05 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.712024+0000", 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010582999999999999 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.712119+0000", 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4654000000000005e-05 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.713620+0000", 2024-09-17T16:42:50.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00150146 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714929+0000", 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013090560000000001 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239387+0000", 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52445777400000004 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239445+0000", 2024-09-17T16:42:50.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7960000000000001e-05 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239491+0000", 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5914999999999997e-05 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:157496 8.4 e822/810 ondisk, result = 0)", 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:48.714185+0000", 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 301.61108627999999, 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52531144100000005, 2024-09-17T16:42:50.380 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714185+0000", 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714185+0000", 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714186+0000", 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8699999999999995e-07 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714192+0000", 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0730000000000001e-06 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239260+0000", 2024-09-17T16:42:50.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.525067909 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239272+0000", 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2527000000000001e-05 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239366+0000", 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4143999999999999e-05 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239376+0000", 2024-09-17T16:42:50.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7620000000000007e-06 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239496+0000", 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120439 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:169614 8.4 8:2531a7fa:::benchmark_data_smithi173_130566_object74639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:06.439805+0000", 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 283.88546618100003, 2024-09-17T16:42:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30158808100000001, 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 169614 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.439805+0000", 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.439805+0000", 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.439807+0000", 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3250000000000002e-06 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.439815+0000", 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6960000000000005e-06 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.439830+0000", 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.525e-05 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.439838+0000", 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.306e-06 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.439889+0000", 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1700999999999998e-05 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.439978+0000", 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8276000000000007e-05 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.440049+0000", 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1278000000000004e-05 2024-09-17T16:42:50.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.442108+0000", 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002058612 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.442177+0000", 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9445999999999999e-05 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.741331+0000", 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.299154225 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.741363+0000", 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1974000000000003e-05 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.741393+0000", 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9992e-05 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:169669 8.4 8:23db35bb:::benchmark_data_smithi173_130566_object74694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:06.512762+0000", 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 283.81250878100002, 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30802659500000001, 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 169669 2024-09-17T16:42:50.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.512762+0000", 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.512762+0000", 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.512764+0000", 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3e-06 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.512771+0000", 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.985e-06 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.512782+0000", 2024-09-17T16:42:50.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1501000000000001e-05 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.512788+0000", 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0229999999999998e-06 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.512835+0000", 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6372999999999998e-05 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.512940+0000", 2024-09-17T16:42:50.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000104954 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.513041+0000", 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000101238 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.514589+0000", 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015477749999999999 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.546380+0000", 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.031791400999999997 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.820718+0000", 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.274338152 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.820760+0000", 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2049999999999999e-05 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.820789+0000", 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8843000000000002e-05 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:169698 8.4 8:2314ca98:::benchmark_data_smithi173_130566_object74723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e834)", 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:06.543955+0000", 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 283.78131602600001, 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30621367799999999, 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.397 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 169698 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.543955+0000", 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.543955+0000", 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.543957+0000", 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1090000000000001e-06 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.543965+0000", 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9610000000000005e-06 2024-09-17T16:42:50.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.543985+0000", 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9408000000000001e-05 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.543994+0000", 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1330000000000005e-06 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.544059+0000", 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5609000000000002e-05 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.544149+0000", 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9598000000000005e-05 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.544220+0000", 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1041e-05 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.545799+0000", 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015787010000000001 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.740785+0000", 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19498642899999999 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.850108+0000", 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.109322451 2024-09-17T16:42:50.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.850141+0000", 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2894e-05 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:06.850169+0000", 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8343999999999998e-05 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8192 9.f 9:f572e93a:::benchmark_data_smithi173_150363_object8191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.811494+0000", 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 220.51377759100001, 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52189015699999997, 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8192 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.404 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811494+0000", 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811494+0000", 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811494+0000", 2024-09-17T16:42:50.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5799999999999998e-07 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811560+0000", 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5425999999999993e-05 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811569+0000", 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8629999999999997e-06 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811574+0000", 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0300000000000001e-06 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811621+0000", 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7695999999999999e-05 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811739+0000", 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011728899999999999 2024-09-17T16:42:50.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811879+0000", 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014007000000000001 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.812390+0000", 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00051132900000000004 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.812862+0000", 2024-09-17T16:42:50.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00047182799999999998 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333287+0000", 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52042468099999994 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333343+0000", 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6687999999999997e-05 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333384+0000", 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0499e-05 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27874.0:8192 9.f e867/858 ondisk, result = 0)", 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.813396+0000", 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 220.51187526499999, 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51999520200000005, 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.410 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813396+0000", 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813396+0000", 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813397+0000", 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.92e-07 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813401+0000", 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7280000000000003e-06 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333226+0000", 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51982427099999995 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333233+0000", 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4560000000000003e-06 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333275+0000", 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1662000000000002e-05 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333280+0000", 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4650000000000002e-06 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333391+0000", 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011092800000000001 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "rep_scrubmap(9.3 e911 from shard 8)", 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:52.369653+0000", 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 117.955618079, 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70106850600000004, 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.414 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.369653+0000", 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.369653+0000", 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.369654+0000", 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8199999999999999e-07 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.369691+0000", 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7397999999999997e-05 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.821987+0000", 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45229588599999998 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.821996+0000", 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6400000000000003e-06 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.070508+0000", 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24851257400000001 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.070523+0000", 2024-09-17T16:42:50.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4718999999999999e-05 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.070722+0000", 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000198607 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122289 9.5 9:a636923a:::benchmark_data_smithi173_150363_object15384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.358953+0000", 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 116.966318598, 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8005457949999997, 2024-09-17T16:42:50.418 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122289 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358953+0000", 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358953+0000", 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358955+0000", 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.956e-06 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358962+0000", 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1550000000000001e-06 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358975+0000", 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3658999999999999e-05 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358985+0000", 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1749999999999994e-06 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359048+0000", 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3594000000000001e-05 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359164+0000", 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011561300000000001 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359275+0000", 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000111024 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.361539+0000", 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022637820000000002 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.364742+0000", 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0032029329999999998 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159395+0000", 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7946534700000001 2024-09-17T16:42:50.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159454+0000", 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8721999999999998e-05 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159498+0000", 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4712000000000002e-05 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122329 9.5 9:a441955e:::benchmark_data_smithi173_150363_object15424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.796199+0000", 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 116.529071949, 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3680691940000003, 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122329 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796199+0000", 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796199+0000", 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796201+0000", 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5340000000000001e-06 2024-09-17T16:42:50.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796211+0000", 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.008e-05 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796224+0000", 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2993000000000001e-05 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796232+0000", 2024-09-17T16:42:50.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0809999999999994e-06 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796296+0000", 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3929000000000004e-05 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796476+0000", 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000180132 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:42:50.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796577+0000", 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010087299999999999 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.798689+0000", 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021117190000000002 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.798875+0000", 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000186737 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.164183+0000", 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3653075699999997 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.164232+0000", 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9259999999999999e-05 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.164268+0000", 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6285999999999998e-05 2024-09-17T16:42:50.430 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(9.12 917/915 [PushOp(9:4fdad5be:::benchmark_data_smithi173_150363_object19229:head, version: 874'1187, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:4fdad5be:::benchmark_data_smithi173_150363_object19229:head@874'1187, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:55.455908+0000", 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 114.869362834, 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.780314744, 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:55.455908+0000", 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:55.455908+0000", 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:55.455910+0000", 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.347e-06 2024-09-17T16:42:50.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:55.455946+0000", 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6569000000000002e-05 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.234382+0000", 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.77843612299999998 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.234395+0000", 2024-09-17T16:42:50.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2415000000000001e-05 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.234486+0000", 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0741000000000004e-05 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.234499+0000", 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3269e-05 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.236223+0000", 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017242799999999999 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(9.19 917/915 [PushReplyOp(9:9d6f52ec:::benchmark_data_smithi173_150363_object21907:head)])", 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:56.373991+0000", 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 113.951279787, 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71017067599999995, 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:50.435 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.373991+0000", 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.373991+0000", 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.373992+0000", 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8400000000000005e-07 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.373996+0000", 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4490000000000001e-06 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.081536+0000", 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70753952099999995 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.081543+0000", 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7220000000000002e-06 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.081606+0000", 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3528999999999994e-05 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.084162+0000", 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0025559710000000002 2024-09-17T16:42:50.438 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(9.18 918/915 [PushReplyOp(9:1c4020f2:::benchmark_data_smithi173_150363_object23328:head)])", 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:57.664778+0000", 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 112.660493309, 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39019172400000002, 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.664778+0000", 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.664778+0000", 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.664779+0000", 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7599999999999996e-07 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.664784+0000", 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1100000000000002e-06 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.052773+0000", 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.387988995 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.052779+0000", 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5089999999999999e-06 2024-09-17T16:42:50.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.052844+0000", 2024-09-17T16:42:50.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4342000000000002e-05 2024-09-17T16:42:50.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.054970+0000", 2024-09-17T16:42:50.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021258919999999999 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:193634 9.5 9:a586ba40:::benchmark_data_smithi173_150363_object86729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:40.654219+0000", 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.6710522569999995, 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.402057626, 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 193634 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654219+0000", 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654219+0000", 2024-09-17T16:42:50.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654220+0000", 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1710000000000001e-06 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654224+0000", 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.25e-06 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.445 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654235+0000", 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0648e-05 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654241+0000", 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1959999999999996e-06 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654291+0000", 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0024000000000001e-05 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654369+0000", 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8002000000000003e-05 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654430+0000", 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0375999999999999e-05 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.657036+0000", 2024-09-17T16:42:50.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0026063409999999999 2024-09-17T16:42:50.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.448 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.448 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.705730+0000", 2024-09-17T16:42:50.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.048693624999999997 2024-09-17T16:42:50.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.448 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.056224+0000", 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.35049413699999998 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.056254+0000", 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.021e-05 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.056277+0000", 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2646e-05 2024-09-17T16:42:50.449 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:193635 9.3 9:c6336636:::benchmark_data_smithi173_150363_object86730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:40.656217+0000", 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.6690546869999991, 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39941113, 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.450 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 193635 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.656217+0000", 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.656217+0000", 2024-09-17T16:42:50.451 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.656218+0000", 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9900000000000009e-07 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.656222+0000", 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4610000000000001e-06 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.656234+0000", 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2033e-05 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.656241+0000", 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.357e-06 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.656278+0000", 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7051999999999997e-05 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.656351+0000", 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2739999999999998e-05 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.656412+0000", 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1172000000000003e-05 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.658852+0000", 2024-09-17T16:42:50.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024399840000000001 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.659814+0000", 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00096131600000000004 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.055447+0000", 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.39563356 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.055578+0000", 2024-09-17T16:42:50.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000130331 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.055628+0000", 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0124999999999999e-05 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:193640 9.3 9:c662e500:::benchmark_data_smithi173_150363_object86735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:40.690162+0000", 2024-09-17T16:42:50.456 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.6351091049999997, 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36556348900000002, 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 193640 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.690162+0000", 2024-09-17T16:42:50.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.690162+0000", 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.690163+0000", 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2559999999999999e-06 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.690168+0000", 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2270000000000004e-06 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.690178+0000", 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0282000000000001e-05 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.690185+0000", 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1489999999999997e-06 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.690242+0000", 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6644000000000002e-05 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.690360+0000", 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000118461 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 8,9", 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.690465+0000", 2024-09-17T16:42:50.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000104972 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.693000+0000", 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002534892 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.694254+0000", 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001253971 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.055664+0000", 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.361409865 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.055700+0000", 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.578e-05 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.055726+0000", 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.599e-05 2024-09-17T16:42:50.462 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:193642 9.6 9:65aef124:::benchmark_data_smithi173_150363_object86737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:40.693276+0000", 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.6319949929999993, 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36663233000000001, 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:50.463 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 193642 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.693276+0000", 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.693276+0000", 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.693278+0000", 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3629999999999999e-06 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.693283+0000", 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1279999999999999e-06 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.693295+0000", 2024-09-17T16:42:50.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2428e-05 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.693303+0000", 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5800000000000003e-06 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.693359+0000", 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6490000000000003e-05 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.467 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.467 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.693454+0000", 2024-09-17T16:42:50.467 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4927999999999996e-05 2024-09-17T16:42:50.467 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.467 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.467 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,10", 2024-09-17T16:42:50.467 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.693542+0000", 2024-09-17T16:42:50.467 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7978000000000004e-05 2024-09-17T16:42:50.467 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.467 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.696291+0000", 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0027492049999999998 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.696773+0000", 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00048184699999999998 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.059823+0000", 2024-09-17T16:42:50.468 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36304952600000001 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.059864+0000", 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1285999999999997e-05 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.059909+0000", 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4570999999999997e-05 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.469 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27874.0:193642 9.6 e960/911 ondisk, result = 0)", 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:40.698086+0000", 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.6271853919999995, 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36182967300000002, 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.698086+0000", 2024-09-17T16:42:50.470 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.698086+0000", 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.698087+0000", 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9999999999999997e-07 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.698090+0000", 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3550000000000001e-06 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.059736+0000", 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36164618599999998 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.059740+0000", 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8240000000000001e-06 2024-09-17T16:42:50.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.473 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.473 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.059799+0000", 2024-09-17T16:42:50.473 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9277999999999997e-05 2024-09-17T16:42:50.473 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.473 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.474 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.059816+0000", 2024-09-17T16:42:50.474 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6688e-05 2024-09-17T16:42:50.474 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.474 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.474 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.059916+0000", 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9642000000000006e-05 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:193647 9.a 9:5582abdf:::benchmark_data_smithi173_150363_object86742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:40.701444+0000", 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.6238275669999993, 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36468911300000001, 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.475 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 193647 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.701444+0000", 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.476 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.701444+0000", 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.701445+0000", 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.302e-06 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.701449+0000", 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3109999999999999e-06 2024-09-17T16:42:50.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.701461+0000", 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1511999999999999e-05 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.701466+0000", 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2000000000000002e-06 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.701509+0000", 2024-09-17T16:42:50.478 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2617000000000002e-05 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.701591+0000", 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2008999999999994e-05 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.701661+0000", 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0796000000000005e-05 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:42:50.479 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.705314+0000", 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036529399999999999 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.705769+0000", 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00045502500000000001 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.066046+0000", 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36027662199999999 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.480 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.066096+0000", 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9623999999999998e-05 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.066133+0000", 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7154999999999999e-05 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.481 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27874.0:193647 9.a e960/858 ondisk, result = 0)", 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:40.705649+0000", 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.6196217500000003, 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.36049705900000001, 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.705649+0000", 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:42:50.482 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.705649+0000", 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.705650+0000", 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.705654+0000", 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6569999999999999e-06 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.483 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.060773+0000", 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.35511864799999998 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.060778+0000", 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5720000000000002e-06 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.066034+0000", 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0052555759999999997 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.484 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.066041+0000", 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9770000000000003e-06 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.066147+0000", 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010592 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:42:50.485 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:42:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:50.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:50.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:50.489 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:50.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:50.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:50.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:42:50.629 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:42:50.710 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:50.733 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:50.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:50.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:50.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:50 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:42:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:50 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:42:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:50 smithi071 ceph-mon[30002]: osdmap e964: 12 total, 12 up, 11 in 2024-09-17T16:42:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [5, 2]} : dispatch 2024-09-17T16:42:50.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:50 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [5, 2]} : dispatch 2024-09-17T16:42:50.834 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:50.857 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:50.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:50 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:42:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:50 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:42:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:50 smithi173 ceph-mon[30530]: osdmap e964: 12 total, 12 up, 11 in 2024-09-17T16:42:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [5, 2]} : dispatch 2024-09-17T16:42:50.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:50 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [5, 2]} : dispatch 2024-09-17T16:42:50.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:50.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:50 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:42:50.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:50 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:42:50.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:50 smithi017 ceph-mon[27571]: osdmap e964: 12 total, 12 up, 11 in 2024-09-17T16:42:50.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [5, 2]} : dispatch 2024-09-17T16:42:50.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:50 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [5, 2]} : dispatch 2024-09-17T16:42:50.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:50.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:50.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:51.088 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:51.112 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:51.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:51.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:51.239 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:51.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:51.342 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:51.367 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:51.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:51.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:51.494 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:51.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:51.550 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:42:51.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:51.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:51.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:51.721 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:51 smithi017 ceph-mon[27571]: pgmap v2124: 37 pgs: 37 active+clean; 851 MiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:42:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:51 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [5, 2]}]': finished 2024-09-17T16:42:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:51 smithi017 ceph-mon[27571]: osdmap e965: 12 total, 12 up, 11 in 2024-09-17T16:42:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:51 smithi017 ceph-mon[27571]: from='client.28304 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:42:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2186296789' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:42:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:51 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2186296789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:42:51.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:51 smithi017 ceph-mon[27571]: osdmap e966: 12 total, 12 up, 11 in 2024-09-17T16:42:51.744 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:51.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:51.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:42:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:51 smithi071 ceph-mon[30002]: pgmap v2124: 37 pgs: 37 active+clean; 851 MiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:42:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:51 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [5, 2]}]': finished 2024-09-17T16:42:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:51 smithi071 ceph-mon[30002]: osdmap e965: 12 total, 12 up, 11 in 2024-09-17T16:42:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:51 smithi071 ceph-mon[30002]: from='client.28304 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:42:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2186296789' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:42:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:51 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2186296789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:42:51.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:51 smithi071 ceph-mon[30002]: osdmap e966: 12 total, 12 up, 11 in 2024-09-17T16:42:51.846 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:51.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:51.867 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:51.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:51 smithi173 ceph-mon[30530]: pgmap v2124: 37 pgs: 37 active+clean; 851 MiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:42:51.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:51 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.3", "id": [5, 2]}]': finished 2024-09-17T16:42:51.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:51 smithi173 ceph-mon[30530]: osdmap e965: 12 total, 12 up, 11 in 2024-09-17T16:42:51.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:51 smithi173 ceph-mon[30530]: from='client.28304 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:42:51.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2186296789' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:42:51.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:51 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:51.896 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2186296789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:42:51.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:51 smithi173 ceph-mon[30530]: osdmap e966: 12 total, 12 up, 11 in 2024-09-17T16:42:51.968 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:42:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:51.992 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:51.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:42:52.093 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:52.114 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:52.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:52.127 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":2125,"stamp":"2024-09-17T16:42:50.547748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":891847224,"num_objects":13603,"num_object_clones":0,"num_object_copies":40809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113434,"num_write_kb":2775553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4186,"num_bytes_recovered":242778112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261534,"ondisk_log_size":261534,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":10930588,"kb_used_data":3514628,"kb_used_omap":83,"kb_used_meta":7415532,"kb_avail":1020266084,"statfs":{"total":1055945392128,"available":1044752470016,"internally_reserved":0,"allocated":3598979072,"data_stored":3595815962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86009,"internal_metadata":7593504775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-394461184,"num_objects":-6019,"num_object_clones":0,"num_object_copies":-18057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-20,"num_write_kb":-210144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3195,"num_bytes_recovered":-179699712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.053934"},"pg_stats":[{"pgid":"9.19","version":"962'9896","reported_seq":16875,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-09-17T16:42:45.809652+0000","last_change":"2024-09-17T16:41:50.281835+0000","last_active":"2024-09-17T16:42:45.809652+0000","last_peered":"2024-09-17T16:42:45.809652+0000","last_clean":"2024-09-17T16:42:45.809652+0000","last_became_active":"2024-09-17T16:41:48.873276+0000","last_became_peered":"2024-09-17T16:41:48.873276+0000","last_unstale":"2024-09-17T16:42:45.809652+0000","last_undegraded":"2024-09-17T16:42:45.809652+0000","last_fullsized":"2024-09-17T16:42:45.809652+0000","mapping_epoch":933,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"935'8588","last_scrub_stamp":"2024-09-17T16:41:50.281678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:41:50.281678+0000","objects_scrubbed":1837,"log_size":9896,"log_dups_size":0,"ondisk_log_size":9896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:51.713140+0000","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":512,"num_object_clones":0,"num_object_copies":1536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1325,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,11],"acting":[6,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"963'9967","reported_seq":20256,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:47.349123+0000","last_change":"2024-09-17T16:42:03.825102+0000","last_active":"2024-09-17T16:42:47.349123+0000","last_peered":"2024-09-17T16:42:47.349123+0000","last_clean":"2024-09-17T16:42:47.349123+0000","last_became_active":"2024-09-17T16:42:03.276347+0000","last_became_peered":"2024-09-17T16:42:03.276347+0000","last_unstale":"2024-09-17T16:42:47.349123+0000","last_undegraded":"2024-09-17T16:42:47.349123+0000","last_fullsized":"2024-09-17T16:42:47.349123+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":5,"last_scrub":"941'8902","last_scrub_stamp":"2024-09-17T16:42:03.825048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:03.825048+0000","objects_scrubbed":1516,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:04.792108+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29163520,"num_objects":445,"num_object_clones":0,"num_object_copies":1335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1071,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,5],"acting":[10,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.17","version":"963'10107","reported_seq":24924,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.459470+0000","last_change":"2024-09-17T16:42:26.135276+0000","last_active":"2024-09-17T16:42:49.459470+0000","last_peered":"2024-09-17T16:42:49.459470+0000","last_clean":"2024-09-17T16:42:49.459470+0000","last_became_active":"2024-09-17T16:42:25.414817+0000","last_became_peered":"2024-09-17T16:42:25.414817+0000","last_unstale":"2024-09-17T16:42:49.459470+0000","last_undegraded":"2024-09-17T16:42:49.459470+0000","last_fullsized":"2024-09-17T16:42:49.459470+0000","mapping_epoch":953,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":954,"parent":"0.0","parent_split_bits":5,"last_scrub":"898'5654","last_scrub_stamp":"2024-09-17T16:40:17.290378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:17.290378+0000","objects_scrubbed":0,"log_size":10107,"log_dups_size":0,"ondisk_log_size":10107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:42:56.128180+0000","scrub_duration":1589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23363584,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6350,"num_write_kb":214592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3622,"num_bytes_recovered":205815808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.16","version":"963'9762","reported_seq":23041,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.557339+0000","last_change":"2024-09-17T16:42:41.552983+0000","last_active":"2024-09-17T16:42:46.557339+0000","last_peered":"2024-09-17T16:42:46.557339+0000","last_clean":"2024-09-17T16:42:46.557339+0000","last_became_active":"2024-09-17T16:42:40.485958+0000","last_became_peered":"2024-09-17T16:42:40.485958+0000","last_unstale":"2024-09-17T16:42:46.557339+0000","last_undegraded":"2024-09-17T16:42:46.557339+0000","last_fullsized":"2024-09-17T16:42:46.557339+0000","mapping_epoch":959,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":960,"parent":"0.0","parent_split_bits":5,"last_scrub":"960'9616","last_scrub_stamp":"2024-09-17T16:42:41.552720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:41.552720+0000","objects_scrubbed":628,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:47.290002+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31391744,"num_objects":479,"num_object_clones":0,"num_object_copies":1437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":149,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,2],"acting":[9,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.15","version":"963'10199","reported_seq":20748,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:47.824145+0000","last_change":"2024-09-17T16:42:20.761094+0000","last_active":"2024-09-17T16:42:47.824145+0000","last_peered":"2024-09-17T16:42:47.824145+0000","last_clean":"2024-09-17T16:42:47.824145+0000","last_became_active":"2024-09-17T16:42:20.392020+0000","last_became_peered":"2024-09-17T16:42:20.392020+0000","last_unstale":"2024-09-17T16:42:47.824145+0000","last_undegraded":"2024-09-17T16:42:47.824145+0000","last_fullsized":"2024-09-17T16:42:47.824145+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"951'9496","last_scrub_stamp":"2024-09-17T16:42:20.760988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:20.760988+0000","objects_scrubbed":1136,"log_size":10199,"log_dups_size":0,"ondisk_log_size":10199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:22.400389+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28377088,"num_objects":433,"num_object_clones":0,"num_object_copies":1299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":703,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,4],"acting":[7,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.14","version":"963'10227","reported_seq":20783,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:47.850342+0000","last_change":"2024-09-17T16:42:05.329669+0000","last_active":"2024-09-17T16:42:47.850342+0000","last_peered":"2024-09-17T16:42:47.850342+0000","last_clean":"2024-09-17T16:42:47.850342+0000","last_became_active":"2024-09-17T16:42:04.264667+0000","last_became_peered":"2024-09-17T16:42:04.264667+0000","last_unstale":"2024-09-17T16:42:47.850342+0000","last_undegraded":"2024-09-17T16:42:47.850342+0000","last_fullsized":"2024-09-17T16:42:47.850342+0000","mapping_epoch":941,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":942,"parent":"0.0","parent_split_bits":5,"last_scrub":"943'9130","last_scrub_stamp":"2024-09-17T16:42:05.329618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.329618+0000","objects_scrubbed":1529,"log_size":10227,"log_dups_size":0,"ondisk_log_size":10227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:29.348392+0000","scrub_duration":975,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27459584,"num_objects":419,"num_object_clones":0,"num_object_copies":1257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"962'9911","reported_seq":21032,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-09-17T16:42:45.741138+0000","last_change":"2024-09-17T16:42:08.667081+0000","last_active":"2024-09-17T16:42:45.741138+0000","last_peered":"2024-09-17T16:42:45.741138+0000","last_clean":"2024-09-17T16:42:45.741138+0000","last_became_active":"2024-09-17T16:42:08.310308+0000","last_became_peered":"2024-09-17T16:42:08.310308+0000","last_unstale":"2024-09-17T16:42:45.741138+0000","last_undegraded":"2024-09-17T16:42:45.741138+0000","last_fullsized":"2024-09-17T16:42:45.741138+0000","mapping_epoch":944,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":945,"parent":"0.0","parent_split_bits":5,"last_scrub":"945'9032","last_scrub_stamp":"2024-09-17T16:42:08.666860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:08.666860+0000","objects_scrubbed":1361,"log_size":9911,"log_dups_size":0,"ondisk_log_size":9911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:11.740990+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":477,"num_object_clones":0,"num_object_copies":1431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":884,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"964'9948","reported_seq":21056,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.992187+0000","last_change":"2024-09-17T16:42:06.167819+0000","last_active":"2024-09-17T16:42:49.992187+0000","last_peered":"2024-09-17T16:42:49.992187+0000","last_clean":"2024-09-17T16:42:49.992187+0000","last_became_active":"2024-09-17T16:42:04.823052+0000","last_became_peered":"2024-09-17T16:42:04.823052+0000","last_unstale":"2024-09-17T16:42:49.992187+0000","last_undegraded":"2024-09-17T16:42:49.992187+0000","last_fullsized":"2024-09-17T16:42:49.992187+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"943'8844","last_scrub_stamp":"2024-09-17T16:42:06.167470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.167470+0000","objects_scrubbed":1473,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:33.369474+0000","scrub_duration":627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24051712,"num_objects":367,"num_object_clones":0,"num_object_copies":1101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1106,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":728,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.547858+0000","last_change":"2024-09-17T16:42:46.565973+0000","last_active":"2024-09-17T16:42:49.547858+0000","last_peered":"2024-09-17T16:42:49.547858+0000","last_clean":"2024-09-17T16:42:49.547858+0000","last_became_active":"2024-09-17T16:39:31.793364+0000","last_became_peered":"2024-09-17T16:39:31.793364+0000","last_unstale":"2024-09-17T16:42:49.547858+0000","last_undegraded":"2024-09-17T16:42:49.547858+0000","last_fullsized":"2024-09-17T16:42:49.547858+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:46.565926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:46.565926+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:51.587508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.8","version":"962'9956","reported_seq":10301,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.554041+0000","last_change":"2024-09-17T16:42:41.075939+0000","last_active":"2024-09-17T16:42:46.554041+0000","last_peered":"2024-09-17T16:42:46.554041+0000","last_clean":"2024-09-17T16:42:46.554041+0000","last_became_active":"2024-09-17T16:40:49.304891+0000","last_became_peered":"2024-09-17T16:40:49.304891+0000","last_unstale":"2024-09-17T16:42:46.554041+0000","last_undegraded":"2024-09-17T16:42:46.554041+0000","last_fullsized":"2024-09-17T16:42:46.554041+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'9806","last_scrub_stamp":"2024-09-17T16:42:41.075793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:41.075793+0000","objects_scrubbed":657,"log_size":9956,"log_dups_size":0,"ondisk_log_size":9956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:03.945834+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32899072,"num_objects":502,"num_object_clones":0,"num_object_copies":1506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2395,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"963'9882","reported_seq":10161,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:47.942941+0000","last_change":"2024-09-17T16:42:47.942879+0000","last_active":"2024-09-17T16:42:47.942941+0000","last_peered":"2024-09-17T16:42:47.942941+0000","last_clean":"2024-09-17T16:42:47.942941+0000","last_became_active":"2024-09-17T16:40:49.310214+0000","last_became_peered":"2024-09-17T16:40:49.310214+0000","last_unstale":"2024-09-17T16:42:47.942941+0000","last_undegraded":"2024-09-17T16:42:47.942941+0000","last_fullsized":"2024-09-17T16:42:47.942941+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"963'9881","last_scrub_stamp":"2024-09-17T16:42:47.942656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:47.942656+0000","objects_scrubbed":448,"log_size":9882,"log_dups_size":0,"ondisk_log_size":9882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:49.940242+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":446,"num_object_clones":0,"num_object_copies":1338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2451,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":199,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.551812+0000","last_change":"2024-09-17T16:42:06.575613+0000","last_active":"2024-09-17T16:42:49.551812+0000","last_peered":"2024-09-17T16:42:49.551812+0000","last_clean":"2024-09-17T16:42:49.551812+0000","last_became_active":"2024-09-17T16:39:33.796846+0000","last_became_peered":"2024-09-17T16:39:33.796846+0000","last_unstale":"2024-09-17T16:42:49.551812+0000","last_undegraded":"2024-09-17T16:42:49.551812+0000","last_fullsized":"2024-09-17T16:42:49.551812+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:06.575413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.575413+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:15.721581+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"963'10085","reported_seq":11432,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.554118+0000","last_change":"2024-09-17T16:40:50.882931+0000","last_active":"2024-09-17T16:42:46.554118+0000","last_peered":"2024-09-17T16:42:46.554118+0000","last_clean":"2024-09-17T16:42:46.554118+0000","last_became_active":"2024-09-17T16:40:49.262776+0000","last_became_peered":"2024-09-17T16:40:49.262776+0000","last_unstale":"2024-09-17T16:42:46.554118+0000","last_undegraded":"2024-09-17T16:42:46.554118+0000","last_fullsized":"2024-09-17T16:42:46.554118+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"913'7714","last_scrub_stamp":"2024-09-17T16:40:50.882796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:50.882796+0000","objects_scrubbed":2882,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33292288,"num_objects":508,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2374,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":194,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.534693+0000","last_change":"2024-09-17T16:42:06.468724+0000","last_active":"2024-09-17T16:42:46.534693+0000","last_peered":"2024-09-17T16:42:46.534693+0000","last_clean":"2024-09-17T16:42:46.534693+0000","last_became_active":"2024-09-17T16:39:33.798293+0000","last_became_peered":"2024-09-17T16:39:33.798293+0000","last_unstale":"2024-09-17T16:42:46.534693+0000","last_undegraded":"2024-09-17T16:42:46.534693+0000","last_fullsized":"2024-09-17T16:42:46.534693+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:06.468655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.468655+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:07.673839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"963'9988","reported_seq":11298,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:47.891687+0000","last_change":"2024-09-17T16:40:52.448115+0000","last_active":"2024-09-17T16:42:47.891687+0000","last_peered":"2024-09-17T16:42:47.891687+0000","last_clean":"2024-09-17T16:42:47.891687+0000","last_became_active":"2024-09-17T16:40:49.295864+0000","last_became_peered":"2024-09-17T16:40:49.295864+0000","last_unstale":"2024-09-17T16:42:47.891687+0000","last_undegraded":"2024-09-17T16:42:47.891687+0000","last_fullsized":"2024-09-17T16:42:47.891687+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'7525","last_scrub_stamp":"2024-09-17T16:40:52.448045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:52.448045+0000","objects_scrubbed":2910,"log_size":9988,"log_dups_size":0,"ondisk_log_size":9988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29294592,"num_objects":447,"num_object_clones":0,"num_object_copies":1341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2463,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":193,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.554029+0000","last_change":"2024-09-17T16:42:05.432898+0000","last_active":"2024-09-17T16:42:46.554029+0000","last_peered":"2024-09-17T16:42:46.554029+0000","last_clean":"2024-09-17T16:42:46.554029+0000","last_became_active":"2024-09-17T16:39:33.792174+0000","last_became_peered":"2024-09-17T16:39:33.792174+0000","last_unstale":"2024-09-17T16:42:46.554029+0000","last_undegraded":"2024-09-17T16:42:46.554029+0000","last_fullsized":"2024-09-17T16:42:46.554029+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:05.432822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.432822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:26.275153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"964'10126","reported_seq":11458,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.982280+0000","last_change":"2024-09-17T16:42:17.121447+0000","last_active":"2024-09-17T16:42:49.982280+0000","last_peered":"2024-09-17T16:42:49.982280+0000","last_clean":"2024-09-17T16:42:49.982280+0000","last_became_active":"2024-09-17T16:40:49.234476+0000","last_became_peered":"2024-09-17T16:40:49.234476+0000","last_unstale":"2024-09-17T16:42:49.982280+0000","last_undegraded":"2024-09-17T16:42:49.982280+0000","last_fullsized":"2024-09-17T16:42:49.982280+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"948'9289","last_scrub_stamp":"2024-09-17T16:42:17.121366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:17.121366+0000","objects_scrubbed":1221,"log_size":10126,"log_dups_size":0,"ondisk_log_size":10126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:23.254022+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":377,"num_object_clones":0,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2532,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"963'10165","reported_seq":10514,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:47.852312+0000","last_change":"2024-09-17T16:42:45.248860+0000","last_active":"2024-09-17T16:42:47.852312+0000","last_peered":"2024-09-17T16:42:47.852312+0000","last_clean":"2024-09-17T16:42:47.852312+0000","last_became_active":"2024-09-17T16:40:49.258147+0000","last_became_peered":"2024-09-17T16:40:49.258147+0000","last_unstale":"2024-09-17T16:42:47.852312+0000","last_undegraded":"2024-09-17T16:42:47.852312+0000","last_fullsized":"2024-09-17T16:42:47.852312+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'10076","last_scrub_stamp":"2024-09-17T16:42:45.248801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:45.248801+0000","objects_scrubbed":562,"log_size":10165,"log_dups_size":0,"ondisk_log_size":10165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:54.968792+0000","scrub_duration":491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949952,"num_objects":457,"num_object_clones":0,"num_object_copies":1371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2475,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"964'10226","reported_seq":10515,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.906784+0000","last_change":"2024-09-17T16:42:23.843834+0000","last_active":"2024-09-17T16:42:49.906784+0000","last_peered":"2024-09-17T16:42:49.906784+0000","last_clean":"2024-09-17T16:42:49.906784+0000","last_became_active":"2024-09-17T16:40:49.253571+0000","last_became_peered":"2024-09-17T16:40:49.253571+0000","last_unstale":"2024-09-17T16:42:49.906784+0000","last_undegraded":"2024-09-17T16:42:49.906784+0000","last_fullsized":"2024-09-17T16:42:49.906784+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"952'9519","last_scrub_stamp":"2024-09-17T16:42:23.843759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:23.843759+0000","objects_scrubbed":1106,"log_size":10226,"log_dups_size":0,"ondisk_log_size":10226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:51.709170+0000","scrub_duration":342,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624576,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2481,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"964'9940","reported_seq":10229,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.936475+0000","last_change":"2024-09-17T16:42:18.007084+0000","last_active":"2024-09-17T16:42:49.936475+0000","last_peered":"2024-09-17T16:42:49.936475+0000","last_clean":"2024-09-17T16:42:49.936475+0000","last_became_active":"2024-09-17T16:40:49.238682+0000","last_became_peered":"2024-09-17T16:40:49.238682+0000","last_unstale":"2024-09-17T16:42:49.936475+0000","last_undegraded":"2024-09-17T16:42:49.936475+0000","last_fullsized":"2024-09-17T16:42:49.936475+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"949'9106","last_scrub_stamp":"2024-09-17T16:42:18.006511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:18.006511+0000","objects_scrubbed":1196,"log_size":9940,"log_dups_size":0,"ondisk_log_size":9940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:22.388884+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":355,"num_object_clones":0,"num_object_copies":1065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"963'9992","reported_seq":11296,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:47.267546+0000","last_change":"2024-09-17T16:40:50.415860+0000","last_active":"2024-09-17T16:42:47.267546+0000","last_peered":"2024-09-17T16:42:47.267546+0000","last_clean":"2024-09-17T16:42:47.267546+0000","last_became_active":"2024-09-17T16:40:49.242408+0000","last_became_peered":"2024-09-17T16:40:49.242408+0000","last_unstale":"2024-09-17T16:42:47.267546+0000","last_undegraded":"2024-09-17T16:42:47.267546+0000","last_fullsized":"2024-09-17T16:42:47.267546+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"913'7556","last_scrub_stamp":"2024-09-17T16:40:50.415791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:50.415791+0000","objects_scrubbed":2906,"log_size":9992,"log_dups_size":0,"ondisk_log_size":9992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:28.533538+0000","scrub_duration":823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949952,"num_objects":457,"num_object_clones":0,"num_object_copies":1371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2449,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.9","version":"963'9922","reported_seq":10216,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:47.370323+0000","last_change":"2024-09-17T16:42:05.243447+0000","last_active":"2024-09-17T16:42:47.370323+0000","last_peered":"2024-09-17T16:42:47.370323+0000","last_clean":"2024-09-17T16:42:47.370323+0000","last_became_active":"2024-09-17T16:40:49.286737+0000","last_became_peered":"2024-09-17T16:40:49.286737+0000","last_unstale":"2024-09-17T16:42:47.370323+0000","last_undegraded":"2024-09-17T16:42:47.370323+0000","last_fullsized":"2024-09-17T16:42:47.370323+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"943'8883","last_scrub_stamp":"2024-09-17T16:42:05.243243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.243243+0000","objects_scrubbed":1534,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:19.321430+0000","scrub_duration":865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31784960,"num_objects":485,"num_object_clones":0,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2385,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":192,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.545672+0000","last_change":"2024-09-17T16:42:10.506218+0000","last_active":"2024-09-17T16:42:46.545672+0000","last_peered":"2024-09-17T16:42:46.545672+0000","last_clean":"2024-09-17T16:42:46.545672+0000","last_became_active":"2024-09-17T16:39:33.801025+0000","last_became_peered":"2024-09-17T16:39:33.801025+0000","last_unstale":"2024-09-17T16:42:46.545672+0000","last_undegraded":"2024-09-17T16:42:46.545672+0000","last_fullsized":"2024-09-17T16:42:46.545672+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:10.506170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:10.506170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:34.538274+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"964'12830","reported_seq":13157,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.958650+0000","last_change":"2024-09-17T16:42:11.454096+0000","last_active":"2024-09-17T16:42:49.958650+0000","last_peered":"2024-09-17T16:42:49.958650+0000","last_clean":"2024-09-17T16:42:49.958650+0000","last_became_active":"2024-09-17T16:38:54.760031+0000","last_became_peered":"2024-09-17T16:38:54.760031+0000","last_unstale":"2024-09-17T16:42:49.958650+0000","last_undegraded":"2024-09-17T16:42:49.958650+0000","last_fullsized":"2024-09-17T16:42:49.958650+0000","mapping_epoch":858,"log_start":"885'2800","ondisk_log_start":"885'2800","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"945'10806","last_scrub_stamp":"2024-09-17T16:42:11.453836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:11.453836+0000","objects_scrubbed":2821,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:24.472631+0000","scrub_duration":892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50987008,"num_objects":778,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12830,"num_write_kb":435456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":191,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-09-17T16:42:44.496815+0000","last_change":"2024-09-17T16:42:07.852349+0000","last_active":"2024-09-17T16:42:44.496815+0000","last_peered":"2024-09-17T16:42:44.496815+0000","last_clean":"2024-09-17T16:42:44.496815+0000","last_became_active":"2024-09-17T16:39:33.790219+0000","last_became_peered":"2024-09-17T16:39:33.790219+0000","last_unstale":"2024-09-17T16:42:44.496815+0000","last_undegraded":"2024-09-17T16:42:44.496815+0000","last_fullsized":"2024-09-17T16:42:44.496815+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:07.852278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:07.852278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:37.529237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"963'12539","reported_seq":12773,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.435962+0000","last_change":"2024-09-17T16:40:27.389553+0000","last_active":"2024-09-17T16:42:49.435962+0000","last_peered":"2024-09-17T16:42:49.435962+0000","last_clean":"2024-09-17T16:42:49.435962+0000","last_became_active":"2024-09-17T16:38:54.766429+0000","last_became_peered":"2024-09-17T16:38:54.766429+0000","last_unstale":"2024-09-17T16:42:49.435962+0000","last_undegraded":"2024-09-17T16:42:49.435962+0000","last_fullsized":"2024-09-17T16:42:49.435962+0000","mapping_epoch":858,"log_start":"884'2500","ondisk_log_start":"884'2500","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":599,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:25.093620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":765,"num_object_clones":0,"num_object_copies":2295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12539,"num_write_kb":425728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":193,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.548618+0000","last_change":"2024-09-17T16:42:06.400060+0000","last_active":"2024-09-17T16:42:46.548618+0000","last_peered":"2024-09-17T16:42:46.548618+0000","last_clean":"2024-09-17T16:42:46.548618+0000","last_became_active":"2024-09-17T16:39:33.824237+0000","last_became_peered":"2024-09-17T16:39:33.824237+0000","last_unstale":"2024-09-17T16:42:46.548618+0000","last_undegraded":"2024-09-17T16:42:46.548618+0000","last_fullsized":"2024-09-17T16:42:46.548618+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:06.399992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.399992+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:11.241466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"964'12745","reported_seq":14193,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:50.119457+0000","last_change":"2024-09-17T16:42:48.736927+0000","last_active":"2024-09-17T16:42:50.119457+0000","last_peered":"2024-09-17T16:42:50.119457+0000","last_clean":"2024-09-17T16:42:50.119457+0000","last_became_active":"2024-09-17T16:39:15.769803+0000","last_became_peered":"2024-09-17T16:39:15.769803+0000","last_unstale":"2024-09-17T16:42:50.119457+0000","last_undegraded":"2024-09-17T16:42:50.119457+0000","last_fullsized":"2024-09-17T16:42:50.119457+0000","mapping_epoch":872,"log_start":"885'2700","ondisk_log_start":"885'2700","created":858,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"963'12653","last_scrub_stamp":"2024-09-17T16:42:48.736834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:48.736834+0000","objects_scrubbed":789,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:49.530135+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45285376,"num_objects":691,"num_object_clones":0,"num_object_copies":2073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12745,"num_write_kb":429952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":564,"num_bytes_recovered":36962304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":200,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.530798+0000","last_change":"2024-09-17T16:42:09.023547+0000","last_active":"2024-09-17T16:42:46.530798+0000","last_peered":"2024-09-17T16:42:46.530798+0000","last_clean":"2024-09-17T16:42:46.530798+0000","last_became_active":"2024-09-17T16:39:33.795955+0000","last_became_peered":"2024-09-17T16:39:33.795955+0000","last_unstale":"2024-09-17T16:42:46.530798+0000","last_undegraded":"2024-09-17T16:42:46.530798+0000","last_fullsized":"2024-09-17T16:42:46.530798+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:09.023469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:09.023469+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:29.153143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"963'12136","reported_seq":12438,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:47.866019+0000","last_change":"2024-09-17T16:42:44.523674+0000","last_active":"2024-09-17T16:42:47.866019+0000","last_peered":"2024-09-17T16:42:47.866019+0000","last_clean":"2024-09-17T16:42:47.866019+0000","last_became_active":"2024-09-17T16:38:54.759320+0000","last_became_peered":"2024-09-17T16:38:54.759320+0000","last_unstale":"2024-09-17T16:42:47.866019+0000","last_undegraded":"2024-09-17T16:42:47.866019+0000","last_fullsized":"2024-09-17T16:42:47.866019+0000","mapping_epoch":858,"log_start":"881'2100","ondisk_log_start":"881'2100","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'11903","last_scrub_stamp":"2024-09-17T16:42:44.523358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:44.523358+0000","objects_scrubbed":1145,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:13.799770+0000","scrub_duration":784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59179032,"num_objects":904,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12136,"num_write_kb":417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":193,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.545742+0000","last_change":"2024-09-17T16:42:09.395309+0000","last_active":"2024-09-17T16:42:46.545742+0000","last_peered":"2024-09-17T16:42:46.545742+0000","last_clean":"2024-09-17T16:42:46.545742+0000","last_became_active":"2024-09-17T16:39:33.800521+0000","last_became_peered":"2024-09-17T16:39:33.800521+0000","last_unstale":"2024-09-17T16:42:46.545742+0000","last_undegraded":"2024-09-17T16:42:46.545742+0000","last_fullsized":"2024-09-17T16:42:46.545742+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:09.395264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:09.395264+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:27.608341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"963'12394","reported_seq":12714,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:47.379641+0000","last_change":"2024-09-17T16:42:23.154240+0000","last_active":"2024-09-17T16:42:47.379641+0000","last_peered":"2024-09-17T16:42:47.379641+0000","last_clean":"2024-09-17T16:42:47.379641+0000","last_became_active":"2024-09-17T16:38:54.765252+0000","last_became_peered":"2024-09-17T16:38:54.765252+0000","last_unstale":"2024-09-17T16:42:47.379641+0000","last_undegraded":"2024-09-17T16:42:47.379641+0000","last_fullsized":"2024-09-17T16:42:47.379641+0000","mapping_epoch":858,"log_start":"882'2300","ondisk_log_start":"882'2300","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'11094","last_scrub_stamp":"2024-09-17T16:42:23.154153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:23.154153+0000","objects_scrubbed":2180,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:48.315354+0000","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56492032,"num_objects":862,"num_object_clones":0,"num_object_copies":2586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12394,"num_write_kb":424192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":193,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.545811+0000","last_change":"2024-09-17T16:42:07.377489+0000","last_active":"2024-09-17T16:42:46.545811+0000","last_peered":"2024-09-17T16:42:46.545811+0000","last_clean":"2024-09-17T16:42:46.545811+0000","last_became_active":"2024-09-17T16:39:33.791607+0000","last_became_peered":"2024-09-17T16:39:33.791607+0000","last_unstale":"2024-09-17T16:42:46.545811+0000","last_undegraded":"2024-09-17T16:42:46.545811+0000","last_fullsized":"2024-09-17T16:42:46.545811+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:07.377443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:07.377443+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:19.127776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"964'12642","reported_seq":12969,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.998775+0000","last_change":"2024-09-17T16:42:16.649461+0000","last_active":"2024-09-17T16:42:49.998775+0000","last_peered":"2024-09-17T16:42:49.998775+0000","last_clean":"2024-09-17T16:42:49.998775+0000","last_became_active":"2024-09-17T16:38:54.765274+0000","last_became_peered":"2024-09-17T16:38:54.765274+0000","last_unstale":"2024-09-17T16:42:49.998775+0000","last_undegraded":"2024-09-17T16:42:49.998775+0000","last_fullsized":"2024-09-17T16:42:49.998775+0000","mapping_epoch":858,"log_start":"885'2600","ondisk_log_start":"885'2600","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"948'10844","last_scrub_stamp":"2024-09-17T16:42:16.649272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:16.649272+0000","objects_scrubbed":2574,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:32.523327+0000","scrub_duration":1084,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48889856,"num_objects":746,"num_object_clones":0,"num_object_copies":2238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12642,"num_write_kb":428416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":216,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-09-17T16:42:44.528053+0000","last_change":"2024-09-17T16:42:05.413918+0000","last_active":"2024-09-17T16:42:44.528053+0000","last_peered":"2024-09-17T16:42:44.528053+0000","last_clean":"2024-09-17T16:42:44.528053+0000","last_became_active":"2024-09-17T16:39:33.798225+0000","last_became_peered":"2024-09-17T16:39:33.798225+0000","last_unstale":"2024-09-17T16:42:44.528053+0000","last_undegraded":"2024-09-17T16:42:44.528053+0000","last_fullsized":"2024-09-17T16:42:44.528053+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:05.413835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.413835+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:17.071323+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"964'9792","reported_seq":22829,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:50.080696+0000","last_change":"2024-09-17T16:42:45.699969+0000","last_active":"2024-09-17T16:42:50.080696+0000","last_peered":"2024-09-17T16:42:50.080696+0000","last_clean":"2024-09-17T16:42:50.080696+0000","last_became_active":"2024-09-17T16:42:44.497555+0000","last_became_peered":"2024-09-17T16:42:44.497555+0000","last_unstale":"2024-09-17T16:42:50.080696+0000","last_undegraded":"2024-09-17T16:42:50.080696+0000","last_fullsized":"2024-09-17T16:42:50.080696+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":5,"last_scrub":"962'9662","last_scrub_stamp":"2024-09-17T16:42:45.699853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:45.699853+0000","objects_scrubbed":528,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:02.569635+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25952256,"num_objects":396,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"962'10145","reported_seq":21819,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-09-17T16:42:45.816587+0000","last_change":"2024-09-17T16:42:15.606868+0000","last_active":"2024-09-17T16:42:45.816587+0000","last_peered":"2024-09-17T16:42:45.816587+0000","last_clean":"2024-09-17T16:42:45.816587+0000","last_became_active":"2024-09-17T16:42:14.417074+0000","last_became_peered":"2024-09-17T16:42:14.417074+0000","last_unstale":"2024-09-17T16:42:45.816587+0000","last_undegraded":"2024-09-17T16:42:45.816587+0000","last_fullsized":"2024-09-17T16:42:45.816587+0000","mapping_epoch":946,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":947,"parent":"0.0","parent_split_bits":5,"last_scrub":"947'9362","last_scrub_stamp":"2024-09-17T16:42:15.606714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:15.606714+0000","objects_scrubbed":1334,"log_size":10145,"log_dups_size":0,"ondisk_log_size":10145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:41.420955+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35717120,"num_objects":545,"num_object_clones":0,"num_object_copies":1635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":789,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":891256856,"num_objects":13601,"num_object_clones":0,"num_object_copies":40803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113434,"num_write_kb":2775553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4186,"num_bytes_recovered":242778112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3504484352,"data_stored":3504472136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260522,"ondisk_log_size":260522,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285969,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1435960,"kb_used_data":714792,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92309192,"statfs":{"total":95995035648,"available":94524612608,"internally_reserved":0,"allocated":731947008,"data_stored":731660710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":738458052},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131211,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":668416,"kb_used_data":306536,"kb_used_omap":15,"kb_used_meta":361840,"kb_avail":93076736,"statfs":{"total":95995035648,"available":95310577664,"internally_reserved":0,"allocated":313892864,"data_stored":313606566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":370524732},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743833,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240240,"kb_used_data":280284,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92504912,"statfs":{"total":95995035648,"available":94725029888,"internally_reserved":0,"allocated":287010816,"data_stored":286728996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":982972864},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005472,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828160,"kb_used_data":156976,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92916992,"statfs":{"total":95995035648,"available":95146999808,"internally_reserved":0,"allocated":160743424,"data_stored":160453574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963946,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1364416,"kb_used_data":487084,"kb_used_omap":41,"kb_used_meta":877206,"kb_avail":92380736,"statfs":{"total":95995035648,"available":94597873664,"internally_reserved":0,"allocated":498774016,"data_stored":498483646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":898259251},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496044,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":990452,"kb_used_data":345628,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92754700,"statfs":{"total":95995035648,"available":94980812800,"internally_reserved":0,"allocated":353923072,"data_stored":353641252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":660273606},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682516,"kb_used_data":425152,"kb_used_omap":14,"kb_used_meta":257329,"kb_avail":93062636,"statfs":{"total":95995035648,"available":95296139264,"internally_reserved":0,"allocated":435355648,"data_stored":435057290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":263505525},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986612,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827336,"kb_used_data":124212,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92917816,"statfs":{"total":95995035648,"available":95147843584,"internally_reserved":0,"allocated":127193088,"data_stored":126902354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150134,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411772,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":709108,"kb_used_data":64292,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93036044,"statfs":{"total":95995035648,"available":95268909056,"internally_reserved":0,"allocated":65835008,"data_stored":65549636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575298,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1167168,"kb_used_data":454316,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":92577984,"statfs":{"total":95995035648,"available":94799855616,"internally_reserved":0,"allocated":465219584,"data_stored":464929214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":729938373},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1016816,"kb_used_data":155356,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92728336,"statfs":{"total":95995035648,"available":94953816064,"internally_reserved":0,"allocated":159084544,"data_stored":158802724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":882047429},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":158269440,"data_stored":158269440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":460460032,"data_stored":460455960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61997056,"data_stored":61997056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":120061952,"data_stored":120061952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":418648064,"data_stored":418643992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":345112576,"data_stored":345112576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":494473216,"data_stored":494469144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":152109056,"data_stored":152109056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":284950528,"data_stored":284950528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":297009152,"data_stored":297009152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":706674688,"data_stored":706674688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:42:52.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:42:52.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:52.238 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:52.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:52.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:52.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:52.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:52.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:52.463 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:52.484 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:52.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:52.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:52.608 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:52.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:52.693 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-17T16:42:52.711 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:52.732 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:52.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:52.774 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:52.834 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:52.855 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:52.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:52.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:52.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:52.986 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:52.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:53.087 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:53.120 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:53.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:53.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:53.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:53.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:53.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:53.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:53.349 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:53.377 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:42:53.378 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:42:53.379 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:53.379 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:53.380 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:53.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:53.413 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:42:53.413 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:42:53.448 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:42:53.449 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:42:53.475 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:42:53.475 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:42:53.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:53.500 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:42:53.500 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:42:53.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:53.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:53.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:53.525 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:42:53.525 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:42:53.550 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:42:53.550 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:42:53.577 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:53 smithi017 ceph-mon[27571]: pgmap v2127: 37 pgs: 37 active+clean; 796 MiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:42:53.578 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:53 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:42:53.578 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:53 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:42:53.578 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:53 smithi017 ceph-mon[27571]: osdmap e967: 12 total, 12 up, 11 in 2024-09-17T16:42:53.578 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1440849501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:53.608 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:53.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:53.635 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:53.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:53.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:53.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:53.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:53 smithi071 ceph-mon[30002]: pgmap v2127: 37 pgs: 37 active+clean; 796 MiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:42:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:53 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:42:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:53 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:42:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:53 smithi071 ceph-mon[30002]: osdmap e967: 12 total, 12 up, 11 in 2024-09-17T16:42:53.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1440849501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:53.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:53.890 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:53.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:53.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:53 smithi173 ceph-mon[30530]: pgmap v2127: 37 pgs: 37 active+clean; 796 MiB data, 10 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:42:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:53 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:42:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:53 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:42:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:53 smithi173 ceph-mon[30530]: osdmap e967: 12 total, 12 up, 11 in 2024-09-17T16:42:53.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1440849501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:42:53.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:54.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:54.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:54.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:54.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:54.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:54.231 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:42:54.232 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:42:54.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:54.277 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:54.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:54.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:54.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:42:54.386 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:42:54.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:54.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:54.425 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:42:54.425 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:42:54.450 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:42:54.451 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:42:54.476 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:42:54.477 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:42:54.507 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:42:54.507 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:42:54.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:54.521 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:54.521 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:54.537 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:54.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:54.639 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:54.671 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:54.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:54.773 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:54.798 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:42:54.803 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:54.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:54.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:54.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:54.936 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:54.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:55.038 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:55.064 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:55.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:55.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:55.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:55.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:55.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:55.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:55.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:55.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:55.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:55.437 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:55.463 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:55.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:55.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:55.568 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-17T16:42:55.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:55.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:55.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:55.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:55.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:55.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:55 smithi071 ceph-mon[30002]: pgmap v2129: 37 pgs: 37 active+clean; 715 MiB data, 9.9 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:42:55.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:55 smithi071 ceph-mon[30002]: from='client.28321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:42:55.827 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:55.856 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:55.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:55.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:55 smithi017 ceph-mon[27571]: pgmap v2129: 37 pgs: 37 active+clean; 715 MiB data, 9.9 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:42:55.892 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:55 smithi017 ceph-mon[27571]: from='client.28321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:42:55.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:55 smithi173 ceph-mon[30530]: pgmap v2129: 37 pgs: 37 active+clean; 715 MiB data, 9.9 GiB used, 973 GiB / 983 GiB avail 2024-09-17T16:42:55.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:55 smithi173 ceph-mon[30530]: from='client.28321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:42:55.957 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:55.981 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:55.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:56.083 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:56.108 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:56.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:56.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:56.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:56.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:56.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:56.246 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":2129,"stamp":"2024-09-17T16:42:54.186939+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":749830712,"num_objects":11436,"num_object_clones":0,"num_object_copies":34308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11436,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115601,"num_write_kb":2775553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4186,"num_bytes_recovered":242778112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":263201,"ondisk_log_size":263201,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":10423820,"kb_used_data":3028788,"kb_used_omap":83,"kb_used_meta":7394732,"kb_avail":1020772852,"statfs":{"total":1055945392128,"available":1045271400448,"internally_reserved":0,"allocated":3101478912,"data_stored":3098227200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":86009,"internal_metadata":7572205575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,10,2,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-256114688,"num_objects":-3908,"num_object_clones":0,"num_object_copies":-11724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2131,"num_write_kb":-210144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3195,"num_bytes_recovered":-179699712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.640108"},"pg_stats":[{"pgid":"9.19","version":"964'10029","reported_seq":17012,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:50.386890+0000","last_change":"2024-09-17T16:41:50.281835+0000","last_active":"2024-09-17T16:42:50.386890+0000","last_peered":"2024-09-17T16:42:50.386890+0000","last_clean":"2024-09-17T16:42:50.386890+0000","last_became_active":"2024-09-17T16:41:48.873276+0000","last_became_peered":"2024-09-17T16:41:48.873276+0000","last_unstale":"2024-09-17T16:42:50.386890+0000","last_undegraded":"2024-09-17T16:42:50.386890+0000","last_fullsized":"2024-09-17T16:42:50.386890+0000","mapping_epoch":933,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"935'8588","last_scrub_stamp":"2024-09-17T16:41:50.281678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:41:50.281678+0000","objects_scrubbed":1837,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:51.713140+0000","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24838144,"num_objects":379,"num_object_clones":0,"num_object_copies":1137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,11],"acting":[6,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"966'10085","reported_seq":20380,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.268606+0000","last_change":"2024-09-17T16:42:03.825102+0000","last_active":"2024-09-17T16:42:52.268606+0000","last_peered":"2024-09-17T16:42:52.268606+0000","last_clean":"2024-09-17T16:42:52.268606+0000","last_became_active":"2024-09-17T16:42:03.276347+0000","last_became_peered":"2024-09-17T16:42:03.276347+0000","last_unstale":"2024-09-17T16:42:52.268606+0000","last_undegraded":"2024-09-17T16:42:52.268606+0000","last_fullsized":"2024-09-17T16:42:52.268606+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":941,"parent":"0.0","parent_split_bits":5,"last_scrub":"941'8902","last_scrub_stamp":"2024-09-17T16:42:03.825048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:03.825048+0000","objects_scrubbed":1516,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:04.792108+0000","scrub_duration":421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21430272,"num_objects":327,"num_object_clones":0,"num_object_copies":981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1189,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,5],"acting":[10,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.17","version":"963'10107","reported_seq":24924,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.459470+0000","last_change":"2024-09-17T16:42:26.135276+0000","last_active":"2024-09-17T16:42:49.459470+0000","last_peered":"2024-09-17T16:42:49.459470+0000","last_clean":"2024-09-17T16:42:49.459470+0000","last_became_active":"2024-09-17T16:42:25.414817+0000","last_became_peered":"2024-09-17T16:42:25.414817+0000","last_unstale":"2024-09-17T16:42:49.459470+0000","last_undegraded":"2024-09-17T16:42:49.459470+0000","last_fullsized":"2024-09-17T16:42:49.459470+0000","mapping_epoch":953,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":954,"parent":"0.0","parent_split_bits":5,"last_scrub":"898'5654","last_scrub_stamp":"2024-09-17T16:40:17.290378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:17.290378+0000","objects_scrubbed":0,"log_size":10107,"log_dups_size":0,"ondisk_log_size":10107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:42:56.128180+0000","scrub_duration":1589,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23363584,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6350,"num_write_kb":214592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3622,"num_bytes_recovered":205815808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.16","version":"966'9918","reported_seq":23204,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:51.587459+0000","last_change":"2024-09-17T16:42:41.552983+0000","last_active":"2024-09-17T16:42:51.587459+0000","last_peered":"2024-09-17T16:42:51.587459+0000","last_clean":"2024-09-17T16:42:51.587459+0000","last_became_active":"2024-09-17T16:42:40.485958+0000","last_became_peered":"2024-09-17T16:42:40.485958+0000","last_unstale":"2024-09-17T16:42:51.587459+0000","last_undegraded":"2024-09-17T16:42:51.587459+0000","last_fullsized":"2024-09-17T16:42:51.587459+0000","mapping_epoch":959,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":960,"parent":"0.0","parent_split_bits":5,"last_scrub":"960'9616","last_scrub_stamp":"2024-09-17T16:42:41.552720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:41.552720+0000","objects_scrubbed":628,"log_size":9918,"log_dups_size":0,"ondisk_log_size":9918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:47.290002+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21168128,"num_objects":323,"num_object_clones":0,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":305,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,2],"acting":[9,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.15","version":"966'10294","reported_seq":20851,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.553256+0000","last_change":"2024-09-17T16:42:20.761094+0000","last_active":"2024-09-17T16:42:52.553256+0000","last_peered":"2024-09-17T16:42:52.553256+0000","last_clean":"2024-09-17T16:42:52.553256+0000","last_became_active":"2024-09-17T16:42:20.392020+0000","last_became_peered":"2024-09-17T16:42:20.392020+0000","last_unstale":"2024-09-17T16:42:52.553256+0000","last_undegraded":"2024-09-17T16:42:52.553256+0000","last_fullsized":"2024-09-17T16:42:52.553256+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"951'9496","last_scrub_stamp":"2024-09-17T16:42:20.760988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:20.760988+0000","objects_scrubbed":1136,"log_size":10294,"log_dups_size":0,"ondisk_log_size":10294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:22.400389+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22151168,"num_objects":338,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":798,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,4],"acting":[7,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.14","version":"966'10338","reported_seq":20903,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.554015+0000","last_change":"2024-09-17T16:42:05.329669+0000","last_active":"2024-09-17T16:42:52.554015+0000","last_peered":"2024-09-17T16:42:52.554015+0000","last_clean":"2024-09-17T16:42:52.554015+0000","last_became_active":"2024-09-17T16:42:04.264667+0000","last_became_peered":"2024-09-17T16:42:04.264667+0000","last_unstale":"2024-09-17T16:42:52.554015+0000","last_undegraded":"2024-09-17T16:42:52.554015+0000","last_fullsized":"2024-09-17T16:42:52.554015+0000","mapping_epoch":941,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":942,"parent":"0.0","parent_split_bits":5,"last_scrub":"943'9130","last_scrub_stamp":"2024-09-17T16:42:05.329618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.329618+0000","objects_scrubbed":1529,"log_size":10338,"log_dups_size":0,"ondisk_log_size":10338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:29.348392+0000","scrub_duration":975,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20185088,"num_objects":308,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1221,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"964'10053","reported_seq":21177,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:50.407729+0000","last_change":"2024-09-17T16:42:08.667081+0000","last_active":"2024-09-17T16:42:50.407729+0000","last_peered":"2024-09-17T16:42:50.407729+0000","last_clean":"2024-09-17T16:42:50.407729+0000","last_became_active":"2024-09-17T16:42:08.310308+0000","last_became_peered":"2024-09-17T16:42:08.310308+0000","last_unstale":"2024-09-17T16:42:50.407729+0000","last_undegraded":"2024-09-17T16:42:50.407729+0000","last_fullsized":"2024-09-17T16:42:50.407729+0000","mapping_epoch":944,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":945,"parent":"0.0","parent_split_bits":5,"last_scrub":"945'9032","last_scrub_stamp":"2024-09-17T16:42:08.666860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:08.666860+0000","objects_scrubbed":1361,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:11.740990+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21954560,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1026,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"964'9948","reported_seq":21056,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.992187+0000","last_change":"2024-09-17T16:42:06.167819+0000","last_active":"2024-09-17T16:42:49.992187+0000","last_peered":"2024-09-17T16:42:49.992187+0000","last_clean":"2024-09-17T16:42:49.992187+0000","last_became_active":"2024-09-17T16:42:04.823052+0000","last_became_peered":"2024-09-17T16:42:04.823052+0000","last_unstale":"2024-09-17T16:42:49.992187+0000","last_undegraded":"2024-09-17T16:42:49.992187+0000","last_fullsized":"2024-09-17T16:42:49.992187+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"943'8844","last_scrub_stamp":"2024-09-17T16:42:06.167470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.167470+0000","objects_scrubbed":1473,"log_size":9948,"log_dups_size":0,"ondisk_log_size":9948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:33.369474+0000","scrub_duration":627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24051712,"num_objects":367,"num_object_clones":0,"num_object_copies":1101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1106,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":728,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.547858+0000","last_change":"2024-09-17T16:42:46.565973+0000","last_active":"2024-09-17T16:42:49.547858+0000","last_peered":"2024-09-17T16:42:49.547858+0000","last_clean":"2024-09-17T16:42:49.547858+0000","last_became_active":"2024-09-17T16:39:31.793364+0000","last_became_peered":"2024-09-17T16:39:31.793364+0000","last_unstale":"2024-09-17T16:42:49.547858+0000","last_undegraded":"2024-09-17T16:42:49.547858+0000","last_fullsized":"2024-09-17T16:42:49.547858+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:46.565926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:46.565926+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:51.587508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.8","version":"965'10096","reported_seq":10447,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:51.555740+0000","last_change":"2024-09-17T16:42:41.075939+0000","last_active":"2024-09-17T16:42:51.555740+0000","last_peered":"2024-09-17T16:42:51.555740+0000","last_clean":"2024-09-17T16:42:51.555740+0000","last_became_active":"2024-09-17T16:40:49.304891+0000","last_became_peered":"2024-09-17T16:40:49.304891+0000","last_unstale":"2024-09-17T16:42:51.555740+0000","last_undegraded":"2024-09-17T16:42:51.555740+0000","last_fullsized":"2024-09-17T16:42:51.555740+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'9806","last_scrub_stamp":"2024-09-17T16:42:41.075793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:41.075793+0000","objects_scrubbed":657,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:03.945834+0000","scrub_duration":662,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23724032,"num_objects":362,"num_object_clones":0,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2535,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.0","version":"966'9978","reported_seq":10266,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.553966+0000","last_change":"2024-09-17T16:42:47.942879+0000","last_active":"2024-09-17T16:42:52.553966+0000","last_peered":"2024-09-17T16:42:52.553966+0000","last_clean":"2024-09-17T16:42:52.553966+0000","last_became_active":"2024-09-17T16:40:49.310214+0000","last_became_peered":"2024-09-17T16:40:49.310214+0000","last_unstale":"2024-09-17T16:42:52.553966+0000","last_undegraded":"2024-09-17T16:42:52.553966+0000","last_fullsized":"2024-09-17T16:42:52.553966+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"963'9881","last_scrub_stamp":"2024-09-17T16:42:47.942656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:47.942656+0000","objects_scrubbed":448,"log_size":9978,"log_dups_size":0,"ondisk_log_size":9978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:49.940242+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22937600,"num_objects":350,"num_object_clones":0,"num_object_copies":1050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2547,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":199,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.551812+0000","last_change":"2024-09-17T16:42:06.575613+0000","last_active":"2024-09-17T16:42:49.551812+0000","last_peered":"2024-09-17T16:42:49.551812+0000","last_clean":"2024-09-17T16:42:49.551812+0000","last_became_active":"2024-09-17T16:39:33.796846+0000","last_became_peered":"2024-09-17T16:39:33.796846+0000","last_unstale":"2024-09-17T16:42:49.551812+0000","last_undegraded":"2024-09-17T16:42:49.551812+0000","last_fullsized":"2024-09-17T16:42:49.551812+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:06.575413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.575413+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:15.721581+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"966'10217","reported_seq":11582,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:51.564295+0000","last_change":"2024-09-17T16:42:48.765638+0000","last_active":"2024-09-17T16:42:51.564295+0000","last_peered":"2024-09-17T16:42:51.564295+0000","last_clean":"2024-09-17T16:42:51.564295+0000","last_became_active":"2024-09-17T16:40:49.262776+0000","last_became_peered":"2024-09-17T16:40:49.262776+0000","last_unstale":"2024-09-17T16:42:51.564295+0000","last_undegraded":"2024-09-17T16:42:51.564295+0000","last_fullsized":"2024-09-17T16:42:51.564295+0000","mapping_epoch":872,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"963'10142","last_scrub_stamp":"2024-09-17T16:42:48.765564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:48.765564+0000","objects_scrubbed":456,"log_size":10217,"log_dups_size":0,"ondisk_log_size":10217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:07.553216+0000","scrub_duration":357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2506,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,10],"acting":[9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":202,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.553295+0000","last_change":"2024-09-17T16:42:06.468724+0000","last_active":"2024-09-17T16:42:52.553295+0000","last_peered":"2024-09-17T16:42:52.553295+0000","last_clean":"2024-09-17T16:42:52.553295+0000","last_became_active":"2024-09-17T16:39:33.798293+0000","last_became_peered":"2024-09-17T16:39:33.798293+0000","last_unstale":"2024-09-17T16:42:52.553295+0000","last_undegraded":"2024-09-17T16:42:52.553295+0000","last_fullsized":"2024-09-17T16:42:52.553295+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:06.468655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.468655+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:07.673839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"967'10092","reported_seq":11410,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.948737+0000","last_change":"2024-09-17T16:40:52.448115+0000","last_active":"2024-09-17T16:42:52.948737+0000","last_peered":"2024-09-17T16:42:52.948737+0000","last_clean":"2024-09-17T16:42:52.948737+0000","last_became_active":"2024-09-17T16:40:49.295864+0000","last_became_peered":"2024-09-17T16:40:49.295864+0000","last_unstale":"2024-09-17T16:42:52.948737+0000","last_undegraded":"2024-09-17T16:42:52.948737+0000","last_fullsized":"2024-09-17T16:42:52.948737+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"914'7525","last_scrub_stamp":"2024-09-17T16:40:52.448045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:52.448045+0000","objects_scrubbed":2910,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:52.685419+0000","scrub_duration":942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22478848,"num_objects":343,"num_object_clones":0,"num_object_copies":1029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,11],"acting":[1,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":199,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:51.555824+0000","last_change":"2024-09-17T16:42:05.432898+0000","last_active":"2024-09-17T16:42:51.555824+0000","last_peered":"2024-09-17T16:42:51.555824+0000","last_clean":"2024-09-17T16:42:51.555824+0000","last_became_active":"2024-09-17T16:39:33.792174+0000","last_became_peered":"2024-09-17T16:39:33.792174+0000","last_unstale":"2024-09-17T16:42:51.555824+0000","last_undegraded":"2024-09-17T16:42:51.555824+0000","last_fullsized":"2024-09-17T16:42:51.555824+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:05.432822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.432822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:26.275153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"964'10126","reported_seq":11458,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.982280+0000","last_change":"2024-09-17T16:42:17.121447+0000","last_active":"2024-09-17T16:42:49.982280+0000","last_peered":"2024-09-17T16:42:49.982280+0000","last_clean":"2024-09-17T16:42:49.982280+0000","last_became_active":"2024-09-17T16:40:49.234476+0000","last_became_peered":"2024-09-17T16:40:49.234476+0000","last_unstale":"2024-09-17T16:42:49.982280+0000","last_undegraded":"2024-09-17T16:42:49.982280+0000","last_fullsized":"2024-09-17T16:42:49.982280+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"948'9289","last_scrub_stamp":"2024-09-17T16:42:17.121366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:17.121366+0000","objects_scrubbed":1221,"log_size":10126,"log_dups_size":0,"ondisk_log_size":10126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:23.254022+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":377,"num_object_clones":0,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2532,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,8],"acting":[5,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"967'10283","reported_seq":10641,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.961513+0000","last_change":"2024-09-17T16:42:45.248860+0000","last_active":"2024-09-17T16:42:52.961513+0000","last_peered":"2024-09-17T16:42:52.961513+0000","last_clean":"2024-09-17T16:42:52.961513+0000","last_became_active":"2024-09-17T16:40:49.258147+0000","last_became_peered":"2024-09-17T16:40:49.258147+0000","last_unstale":"2024-09-17T16:42:52.961513+0000","last_undegraded":"2024-09-17T16:42:52.961513+0000","last_fullsized":"2024-09-17T16:42:52.961513+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'10076","last_scrub_stamp":"2024-09-17T16:42:45.248801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:45.248801+0000","objects_scrubbed":562,"log_size":10283,"log_dups_size":0,"ondisk_log_size":10283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:54.968792+0000","scrub_duration":491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2593,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"964'10226","reported_seq":10515,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.906784+0000","last_change":"2024-09-17T16:42:23.843834+0000","last_active":"2024-09-17T16:42:49.906784+0000","last_peered":"2024-09-17T16:42:49.906784+0000","last_clean":"2024-09-17T16:42:49.906784+0000","last_became_active":"2024-09-17T16:40:49.253571+0000","last_became_peered":"2024-09-17T16:40:49.253571+0000","last_unstale":"2024-09-17T16:42:49.906784+0000","last_undegraded":"2024-09-17T16:42:49.906784+0000","last_fullsized":"2024-09-17T16:42:49.906784+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"952'9519","last_scrub_stamp":"2024-09-17T16:42:23.843759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:23.843759+0000","objects_scrubbed":1106,"log_size":10226,"log_dups_size":0,"ondisk_log_size":10226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:51.709170+0000","scrub_duration":342,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624576,"num_objects":391,"num_object_clones":0,"num_object_copies":1173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2481,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,7],"acting":[5,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"964'9940","reported_seq":10229,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.936475+0000","last_change":"2024-09-17T16:42:18.007084+0000","last_active":"2024-09-17T16:42:49.936475+0000","last_peered":"2024-09-17T16:42:49.936475+0000","last_clean":"2024-09-17T16:42:49.936475+0000","last_became_active":"2024-09-17T16:40:49.238682+0000","last_became_peered":"2024-09-17T16:40:49.238682+0000","last_unstale":"2024-09-17T16:42:49.936475+0000","last_undegraded":"2024-09-17T16:42:49.936475+0000","last_fullsized":"2024-09-17T16:42:49.936475+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"949'9106","last_scrub_stamp":"2024-09-17T16:42:18.006511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:18.006511+0000","objects_scrubbed":1196,"log_size":9940,"log_dups_size":0,"ondisk_log_size":9940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:22.388884+0000","scrub_duration":418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23265280,"num_objects":355,"num_object_clones":0,"num_object_copies":1065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"966'10110","reported_seq":11420,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:51.730580+0000","last_change":"2024-09-17T16:40:50.415860+0000","last_active":"2024-09-17T16:42:51.730580+0000","last_peered":"2024-09-17T16:42:51.730580+0000","last_clean":"2024-09-17T16:42:51.730580+0000","last_became_active":"2024-09-17T16:40:49.242408+0000","last_became_peered":"2024-09-17T16:40:49.242408+0000","last_unstale":"2024-09-17T16:42:51.730580+0000","last_undegraded":"2024-09-17T16:42:51.730580+0000","last_fullsized":"2024-09-17T16:42:51.730580+0000","mapping_epoch":870,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"913'7556","last_scrub_stamp":"2024-09-17T16:40:50.415791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:40:50.415791+0000","objects_scrubbed":2906,"log_size":10110,"log_dups_size":0,"ondisk_log_size":10110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:28.533538+0000","scrub_duration":823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.9","version":"966'10053","reported_seq":10353,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.284138+0000","last_change":"2024-09-17T16:42:05.243447+0000","last_active":"2024-09-17T16:42:52.284138+0000","last_peered":"2024-09-17T16:42:52.284138+0000","last_clean":"2024-09-17T16:42:52.284138+0000","last_became_active":"2024-09-17T16:40:49.286737+0000","last_became_peered":"2024-09-17T16:40:49.286737+0000","last_unstale":"2024-09-17T16:42:52.284138+0000","last_undegraded":"2024-09-17T16:42:52.284138+0000","last_fullsized":"2024-09-17T16:42:52.284138+0000","mapping_epoch":858,"log_start":"0'0","ondisk_log_start":"0'0","created":858,"last_epoch_clean":912,"parent":"0.0","parent_split_bits":0,"last_scrub":"943'8883","last_scrub_stamp":"2024-09-17T16:42:05.243243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.243243+0000","objects_scrubbed":1534,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:19.321430+0000","scrub_duration":865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23199744,"num_objects":354,"num_object_clones":0,"num_object_copies":1062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":198,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:51.554486+0000","last_change":"2024-09-17T16:42:10.506218+0000","last_active":"2024-09-17T16:42:51.554486+0000","last_peered":"2024-09-17T16:42:51.554486+0000","last_clean":"2024-09-17T16:42:51.554486+0000","last_became_active":"2024-09-17T16:39:33.801025+0000","last_became_peered":"2024-09-17T16:39:33.801025+0000","last_unstale":"2024-09-17T16:42:51.554486+0000","last_undegraded":"2024-09-17T16:42:51.554486+0000","last_fullsized":"2024-09-17T16:42:51.554486+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:10.506170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:10.506170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:34.538274+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"964'12830","reported_seq":13157,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.958650+0000","last_change":"2024-09-17T16:42:11.454096+0000","last_active":"2024-09-17T16:42:49.958650+0000","last_peered":"2024-09-17T16:42:49.958650+0000","last_clean":"2024-09-17T16:42:49.958650+0000","last_became_active":"2024-09-17T16:38:54.760031+0000","last_became_peered":"2024-09-17T16:38:54.760031+0000","last_unstale":"2024-09-17T16:42:49.958650+0000","last_undegraded":"2024-09-17T16:42:49.958650+0000","last_fullsized":"2024-09-17T16:42:49.958650+0000","mapping_epoch":858,"log_start":"885'2800","ondisk_log_start":"885'2800","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"945'10806","last_scrub_stamp":"2024-09-17T16:42:11.453836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:11.453836+0000","objects_scrubbed":2821,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:24.472631+0000","scrub_duration":892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50987008,"num_objects":778,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12830,"num_write_kb":435456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":197,"reported_epoch":965,"state":"active+clean","last_fresh":"2024-09-17T16:42:50.571851+0000","last_change":"2024-09-17T16:42:07.852349+0000","last_active":"2024-09-17T16:42:50.571851+0000","last_peered":"2024-09-17T16:42:50.571851+0000","last_clean":"2024-09-17T16:42:50.571851+0000","last_became_active":"2024-09-17T16:39:33.790219+0000","last_became_peered":"2024-09-17T16:39:33.790219+0000","last_unstale":"2024-09-17T16:42:50.571851+0000","last_undegraded":"2024-09-17T16:42:50.571851+0000","last_fullsized":"2024-09-17T16:42:50.571851+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:07.852278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:07.852278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:37.529237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"963'12539","reported_seq":12773,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.435962+0000","last_change":"2024-09-17T16:40:27.389553+0000","last_active":"2024-09-17T16:42:49.435962+0000","last_peered":"2024-09-17T16:42:49.435962+0000","last_clean":"2024-09-17T16:42:49.435962+0000","last_became_active":"2024-09-17T16:38:54.766429+0000","last_became_peered":"2024-09-17T16:38:54.766429+0000","last_unstale":"2024-09-17T16:42:49.435962+0000","last_undegraded":"2024-09-17T16:42:49.435962+0000","last_fullsized":"2024-09-17T16:42:49.435962+0000","mapping_epoch":858,"log_start":"884'2500","ondisk_log_start":"884'2500","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:38:54.506703+0000","objects_scrubbed":599,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:25.093620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":765,"num_object_clones":0,"num_object_copies":2295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12539,"num_write_kb":425728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":201,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.553920+0000","last_change":"2024-09-17T16:42:06.400060+0000","last_active":"2024-09-17T16:42:52.553920+0000","last_peered":"2024-09-17T16:42:52.553920+0000","last_clean":"2024-09-17T16:42:52.553920+0000","last_became_active":"2024-09-17T16:39:33.824237+0000","last_became_peered":"2024-09-17T16:39:33.824237+0000","last_unstale":"2024-09-17T16:42:52.553920+0000","last_undegraded":"2024-09-17T16:42:52.553920+0000","last_fullsized":"2024-09-17T16:42:52.553920+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:06.399992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.399992+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:11.241466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.c","version":"964'12745","reported_seq":14193,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:50.119457+0000","last_change":"2024-09-17T16:42:48.736927+0000","last_active":"2024-09-17T16:42:50.119457+0000","last_peered":"2024-09-17T16:42:50.119457+0000","last_clean":"2024-09-17T16:42:50.119457+0000","last_became_active":"2024-09-17T16:39:15.769803+0000","last_became_peered":"2024-09-17T16:39:15.769803+0000","last_unstale":"2024-09-17T16:42:50.119457+0000","last_undegraded":"2024-09-17T16:42:50.119457+0000","last_fullsized":"2024-09-17T16:42:50.119457+0000","mapping_epoch":872,"log_start":"885'2700","ondisk_log_start":"885'2700","created":858,"last_epoch_clean":873,"parent":"0.0","parent_split_bits":0,"last_scrub":"963'12653","last_scrub_stamp":"2024-09-17T16:42:48.736834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:48.736834+0000","objects_scrubbed":789,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:49.530135+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45285376,"num_objects":691,"num_object_clones":0,"num_object_copies":2073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12745,"num_write_kb":429952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":564,"num_bytes_recovered":36962304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,0],"acting":[4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":200,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-17T16:42:46.530798+0000","last_change":"2024-09-17T16:42:09.023547+0000","last_active":"2024-09-17T16:42:46.530798+0000","last_peered":"2024-09-17T16:42:46.530798+0000","last_clean":"2024-09-17T16:42:46.530798+0000","last_became_active":"2024-09-17T16:39:33.795955+0000","last_became_peered":"2024-09-17T16:39:33.795955+0000","last_unstale":"2024-09-17T16:42:46.530798+0000","last_undegraded":"2024-09-17T16:42:46.530798+0000","last_fullsized":"2024-09-17T16:42:46.530798+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:09.023469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:09.023469+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:29.153143+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"966'12319","reported_seq":12631,"reported_epoch":967,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.553411+0000","last_change":"2024-09-17T16:42:44.523674+0000","last_active":"2024-09-17T16:42:52.553411+0000","last_peered":"2024-09-17T16:42:52.553411+0000","last_clean":"2024-09-17T16:42:52.553411+0000","last_became_active":"2024-09-17T16:38:54.759320+0000","last_became_peered":"2024-09-17T16:38:54.759320+0000","last_unstale":"2024-09-17T16:42:52.553411+0000","last_undegraded":"2024-09-17T16:42:52.553411+0000","last_fullsized":"2024-09-17T16:42:52.553411+0000","mapping_epoch":858,"log_start":"883'2300","ondisk_log_start":"883'2300","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'11903","last_scrub_stamp":"2024-09-17T16:42:44.523358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:44.523358+0000","objects_scrubbed":1145,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:13.799770+0000","scrub_duration":784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47185944,"num_objects":721,"num_object_clones":0,"num_object_copies":2163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12319,"num_write_kb":417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":199,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:51.554588+0000","last_change":"2024-09-17T16:42:09.395309+0000","last_active":"2024-09-17T16:42:51.554588+0000","last_peered":"2024-09-17T16:42:51.554588+0000","last_clean":"2024-09-17T16:42:51.554588+0000","last_became_active":"2024-09-17T16:39:33.800521+0000","last_became_peered":"2024-09-17T16:39:33.800521+0000","last_unstale":"2024-09-17T16:42:51.554588+0000","last_undegraded":"2024-09-17T16:42:51.554588+0000","last_fullsized":"2024-09-17T16:42:51.554588+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:09.395264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:09.395264+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:27.608341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"966'12620","reported_seq":12946,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:52.265225+0000","last_change":"2024-09-17T16:42:23.154240+0000","last_active":"2024-09-17T16:42:52.265225+0000","last_peered":"2024-09-17T16:42:52.265225+0000","last_clean":"2024-09-17T16:42:52.265225+0000","last_became_active":"2024-09-17T16:38:54.765252+0000","last_became_peered":"2024-09-17T16:38:54.765252+0000","last_unstale":"2024-09-17T16:42:52.265225+0000","last_undegraded":"2024-09-17T16:42:52.265225+0000","last_fullsized":"2024-09-17T16:42:52.265225+0000","mapping_epoch":858,"log_start":"885'2600","ondisk_log_start":"885'2600","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"951'11094","last_scrub_stamp":"2024-09-17T16:42:23.154153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:23.154153+0000","objects_scrubbed":2180,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:48.315354+0000","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12620,"num_write_kb":424192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":199,"reported_epoch":966,"state":"active+clean","last_fresh":"2024-09-17T16:42:51.555019+0000","last_change":"2024-09-17T16:42:07.377489+0000","last_active":"2024-09-17T16:42:51.555019+0000","last_peered":"2024-09-17T16:42:51.555019+0000","last_clean":"2024-09-17T16:42:51.555019+0000","last_became_active":"2024-09-17T16:39:33.791607+0000","last_became_peered":"2024-09-17T16:39:33.791607+0000","last_unstale":"2024-09-17T16:42:51.555019+0000","last_undegraded":"2024-09-17T16:42:51.555019+0000","last_fullsized":"2024-09-17T16:42:51.555019+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:07.377443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:07.377443+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:19.127776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"964'12642","reported_seq":12969,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.998775+0000","last_change":"2024-09-17T16:42:16.649461+0000","last_active":"2024-09-17T16:42:49.998775+0000","last_peered":"2024-09-17T16:42:49.998775+0000","last_clean":"2024-09-17T16:42:49.998775+0000","last_became_active":"2024-09-17T16:38:54.765274+0000","last_became_peered":"2024-09-17T16:38:54.765274+0000","last_unstale":"2024-09-17T16:42:49.998775+0000","last_undegraded":"2024-09-17T16:42:49.998775+0000","last_fullsized":"2024-09-17T16:42:49.998775+0000","mapping_epoch":858,"log_start":"885'2600","ondisk_log_start":"885'2600","created":858,"last_epoch_clean":859,"parent":"0.0","parent_split_bits":0,"last_scrub":"948'10844","last_scrub_stamp":"2024-09-17T16:42:16.649272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:16.649272+0000","objects_scrubbed":2574,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:32.523327+0000","scrub_duration":1084,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48889856,"num_objects":746,"num_object_clones":0,"num_object_copies":2238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12642,"num_write_kb":428416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":220,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:49.579799+0000","last_change":"2024-09-17T16:42:05.413918+0000","last_active":"2024-09-17T16:42:49.579799+0000","last_peered":"2024-09-17T16:42:49.579799+0000","last_clean":"2024-09-17T16:42:49.579799+0000","last_became_active":"2024-09-17T16:39:33.798225+0000","last_became_peered":"2024-09-17T16:39:33.798225+0000","last_unstale":"2024-09-17T16:42:49.579799+0000","last_undegraded":"2024-09-17T16:42:49.579799+0000","last_fullsized":"2024-09-17T16:42:49.579799+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:05.413835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.413835+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:17.071323+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"964'9792","reported_seq":22829,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:50.080696+0000","last_change":"2024-09-17T16:42:45.699969+0000","last_active":"2024-09-17T16:42:50.080696+0000","last_peered":"2024-09-17T16:42:50.080696+0000","last_clean":"2024-09-17T16:42:50.080696+0000","last_became_active":"2024-09-17T16:42:44.497555+0000","last_became_peered":"2024-09-17T16:42:44.497555+0000","last_unstale":"2024-09-17T16:42:50.080696+0000","last_undegraded":"2024-09-17T16:42:50.080696+0000","last_fullsized":"2024-09-17T16:42:50.080696+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":5,"last_scrub":"962'9662","last_scrub_stamp":"2024-09-17T16:42:45.699853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:45.699853+0000","objects_scrubbed":528,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:02.569635+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25952256,"num_objects":396,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"964'10309","reported_seq":21988,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-17T16:42:50.428055+0000","last_change":"2024-09-17T16:42:15.606868+0000","last_active":"2024-09-17T16:42:50.428055+0000","last_peered":"2024-09-17T16:42:50.428055+0000","last_clean":"2024-09-17T16:42:50.428055+0000","last_became_active":"2024-09-17T16:42:14.417074+0000","last_became_peered":"2024-09-17T16:42:14.417074+0000","last_unstale":"2024-09-17T16:42:50.428055+0000","last_undegraded":"2024-09-17T16:42:50.428055+0000","last_fullsized":"2024-09-17T16:42:50.428055+0000","mapping_epoch":946,"log_start":"0'0","ondisk_log_start":"0'0","created":911,"last_epoch_clean":947,"parent":"0.0","parent_split_bits":5,"last_scrub":"947'9362","last_scrub_stamp":"2024-09-17T16:42:15.606714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:38:54.506703+0000","last_clean_scrub_stamp":"2024-09-17T16:42:15.606714+0000","objects_scrubbed":1334,"log_size":10309,"log_dups_size":0,"ondisk_log_size":10309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:43:41.420955+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24969216,"num_objects":381,"num_object_clones":0,"num_object_copies":1143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":953,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":749240344,"num_objects":11434,"num_object_clones":0,"num_object_copies":34302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115601,"num_write_kb":2775553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4186,"num_bytes_recovered":242778112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3027054592,"data_stored":3027042376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":262189,"ondisk_log_size":262189,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":11}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285969,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1435960,"kb_used_data":714792,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92309192,"statfs":{"total":95995035648,"available":94524612608,"internally_reserved":0,"allocated":731947008,"data_stored":731660710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":738458052},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131212,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":565396,"kb_used_data":216000,"kb_used_omap":15,"kb_used_meta":349360,"kb_avail":93179756,"statfs":{"total":95995035648,"available":95416070144,"internally_reserved":0,"allocated":221184000,"data_stored":220888838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":357745212},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743834,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1153428,"kb_used_data":201792,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92591724,"statfs":{"total":95995035648,"available":94813925376,"internally_reserved":0,"allocated":206635008,"data_stored":206339846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":974453184},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005472,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828160,"kb_used_data":156976,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92916992,"statfs":{"total":95995035648,"available":95146999808,"internally_reserved":0,"allocated":160743424,"data_stored":160453574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963947,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1263088,"kb_used_data":398300,"kb_used_omap":41,"kb_used_meta":864726,"kb_avail":92482064,"statfs":{"total":95995035648,"available":94701633536,"internally_reserved":0,"allocated":407859200,"data_stored":407551217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":885479731},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496045,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":935112,"kb_used_data":286132,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92810040,"statfs":{"total":95995035648,"available":95037480960,"internally_reserved":0,"allocated":292999168,"data_stored":292708434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":664533446},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682516,"kb_used_data":425152,"kb_used_omap":14,"kb_used_meta":257329,"kb_avail":93062636,"statfs":{"total":95995035648,"available":95296139264,"internally_reserved":0,"allocated":435355648,"data_stored":435057290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":263505525},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986612,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":827336,"kb_used_data":124212,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92917816,"statfs":{"total":95995035648,"available":95147843584,"internally_reserved":0,"allocated":127193088,"data_stored":126902354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669150135,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411773,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694048,"kb_used_data":45064,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93051104,"statfs":{"total":95995035648,"available":95284330496,"internally_reserved":0,"allocated":46145536,"data_stored":45846822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575299,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1063540,"kb_used_data":346524,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":92681612,"statfs":{"total":95995035648,"available":94905970688,"internally_reserved":0,"allocated":354840576,"data_stored":354532593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738823,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":975236,"kb_used_data":113844,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92769916,"statfs":{"total":95995035648,"available":94996393984,"internally_reserved":0,"allocated":116576256,"data_stored":116285522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":882047429},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":111607808,"data_stored":111607808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":353374208,"data_stored":353370136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":43384832,"data_stored":43384832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4718592,"data_stored":4718592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":120061952,"data_stored":120061952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":418648064,"data_stored":418643992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":289406976,"data_stored":289406976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":406720512,"data_stored":406716440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":152109056,"data_stored":152109056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":203030528,"data_stored":203030528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":217317376,"data_stored":217317376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":706674688,"data_stored":706674688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:42:56.248 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T16:42:56.248 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:42:56.249 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-17T16:42:56.249 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd in 3 2024-09-17T16:42:56.334 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:56.360 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:56.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:56.462 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:56.491 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:56.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:56.593 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:56.618 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:56.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:56.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:56.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:56.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:56.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:56.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:56.871 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:56.886 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:56.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:56.888 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:56.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:42:57.013 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:57.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:42:57.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:57.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:57.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:57.242 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:42:57.272 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:57.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:57.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:42:57.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:57.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:57.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:57.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:57.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:57.527 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:57.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:42:57.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:57.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:42:57.721 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:57 smithi017 ceph-mon[27571]: pgmap v2130: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 603 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 285/28908 objects misplaced (0.986%); 864 KiB/s, 13 objects/s recovering 2024-09-17T16:42:57.753 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:42:57.780 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:57.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:42:57.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:57 smithi071 ceph-mon[30002]: pgmap v2130: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 603 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 285/28908 objects misplaced (0.986%); 864 KiB/s, 13 objects/s recovering 2024-09-17T16:42:57.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:57.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:57 smithi173 ceph-mon[30530]: pgmap v2130: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 603 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 285/28908 objects misplaced (0.986%); 864 KiB/s, 13 objects/s recovering 2024-09-17T16:42:57.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:57.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:57.938 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:42:58.009 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:58.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:58.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:58.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:58.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:42:58.163 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:58.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:42:58.265 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:58.292 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:58.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:58.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:58 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:42:58.244+0000 7f086a07d640 -1 mon.a@0(leader).osd e967 definitely_dead 0 2024-09-17T16:42:58.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:58.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:58.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:58.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:58.562 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:58.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:58.564 INFO:teuthology.orchestra.run.smithi017.stderr:marked in osd.3. 2024-09-17T16:42:58.566 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:42:58.652 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:42:58.652 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:42:58.663 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:58.689 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:58.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:58.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:58.791 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:42:58.813 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:42:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:58.814 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:58.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:42:58.877 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:58 smithi017 ceph-mon[27571]: 9.17 scrub starts 2024-09-17T16:42:58.877 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/209594381' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:42:58.877 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1048110512' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-17T16:42:58.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:58 smithi173 ceph-mon[30530]: 9.17 scrub starts 2024-09-17T16:42:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/209594381' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:42:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1048110512' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-17T16:42:58.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:58.941 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:58.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:58.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:59.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:58 smithi071 ceph-mon[30002]: 9.17 scrub starts 2024-09-17T16:42:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/209594381' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:42:59.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1048110512' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-17T16:42:59.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:59.066 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:42:59.168 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:59.190 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:59.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:59.292 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:42:59.313 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:42:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:59.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:42:59.362 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-17T16:42:59.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:42:59.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:59.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:59.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:42:59.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:59.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:59.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:59.662 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:42:59.687 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:42:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:59.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:42:59.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:42:59.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:59.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:42:59.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:59 smithi071 ceph-mon[30002]: pgmap v2131: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 474 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 787 op/s; 285/22749 objects misplaced (1.253%); 679 KiB/s, 10 objects/s recovering 2024-09-17T16:42:59.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:59 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/209594381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1048110512' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-17T16:42:59.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:42:59 smithi071 ceph-mon[30002]: osdmap e968: 12 total, 12 up, 12 in 2024-09-17T16:42:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:59 smithi173 ceph-mon[30530]: pgmap v2131: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 474 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 787 op/s; 285/22749 objects misplaced (1.253%); 679 KiB/s, 10 objects/s recovering 2024-09-17T16:42:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:59 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/209594381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1048110512' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-17T16:42:59.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:42:59 smithi173 ceph-mon[30530]: osdmap e968: 12 total, 12 up, 12 in 2024-09-17T16:42:59.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:42:59.920 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:59 smithi017 ceph-mon[27571]: pgmap v2131: 37 pgs: 1 active+recovering+remapped, 36 active+clean; 474 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 787 op/s; 285/22749 objects misplaced (1.253%); 679 KiB/s, 10 objects/s recovering 2024-09-17T16:42:59.920 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:59 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:42:59.920 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/209594381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:42:59.920 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1048110512' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-17T16:42:59.920 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:42:59 smithi017 ceph-mon[27571]: osdmap e968: 12 total, 12 up, 12 in 2024-09-17T16:42:59.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:42:59.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:42:59.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:00.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:00.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:00.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:00.159 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:00.184 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:00.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:00.278 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:00.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:00.310 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:00.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:00.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:00.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:00.434 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:00.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:00.536 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:00.563 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:00.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:00.664 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:00.687 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:00.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:00.756 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-17T16:43:00.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:00.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:00.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:00.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:00.830 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:00 smithi017 ceph-mon[27571]: osdmap e969: 12 total, 12 up, 12 in 2024-09-17T16:43:00.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:00 smithi173 ceph-mon[30530]: osdmap e969: 12 total, 12 up, 12 in 2024-09-17T16:43:00.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:00.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:00.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:00.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:01.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:01.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:00 smithi071 ceph-mon[30002]: osdmap e969: 12 total, 12 up, 12 in 2024-09-17T16:43:01.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:01.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:01.169 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:01.202 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:01.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:01.304 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:01.337 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:01.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:01.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:01.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:01.465 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:01.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:01.599 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:01.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:01.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:01.731 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:01.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:01.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:01.872 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:01.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:01.894 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:01 smithi173 ceph-mon[30530]: pgmap v2134: 37 pgs: 3 remapped+peering, 34 active+clean; 376 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-17T16:43:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:01 smithi173 ceph-mon[30530]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 967'13118 2024-09-17T16:43:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:01 smithi173 ceph-mon[30530]: osdmap e970: 12 total, 12 up, 12 in 2024-09-17T16:43:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:01 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:43:01.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:01 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:43:01.895 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:01 smithi017 ceph-mon[27571]: pgmap v2134: 37 pgs: 3 remapped+peering, 34 active+clean; 376 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-17T16:43:01.895 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:01 smithi017 ceph-mon[27571]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 967'13118 2024-09-17T16:43:01.895 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:01 smithi017 ceph-mon[27571]: osdmap e970: 12 total, 12 up, 12 in 2024-09-17T16:43:01.895 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:01 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:43:01.896 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:01 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:43:01.948 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:43:01.948 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:43:01.975 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:01.982 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:43:01.982 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:43:01.998 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:01.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:02.022 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:43:02.023 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:43:02.056 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:43:02.056 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:43:02.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:01 smithi071 ceph-mon[30002]: pgmap v2134: 37 pgs: 3 remapped+peering, 34 active+clean; 376 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-17T16:43:02.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:01 smithi071 ceph-mon[30002]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 967'13118 2024-09-17T16:43:02.063 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:01 smithi071 ceph-mon[30002]: osdmap e970: 12 total, 12 up, 12 in 2024-09-17T16:43:02.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:01 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:43:02.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:01 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:43:02.087 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:43:02.087 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:43:02.088 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:02.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:02.122 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:43:02.123 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:43:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:02.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:02.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:02.152 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:43:02.152 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:43:02.175 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:43:02.175 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:43:02.199 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:43:02.199 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:43:02.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:02.226 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:43:02.227 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:43:02.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:02.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:02.256 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:43:02.257 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:43:02.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:43:02.295 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:43:02.295 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:43:02.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:02.392 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:02.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:02.493 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:02.523 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:02.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:02.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:02.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:02.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:02.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:02.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:02.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:02.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:02.865 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:43:02.865 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27088.0:191141 7.6 7:602fc0c3:::benchmark_data_smithi173_116396_object95541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e724)", 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.395970+0000", 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 548.46569200099998, 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.51113990499999995, 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27088", 2024-09-17T16:43:02.866 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/4033045501", 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 191141 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.395970+0000", 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.395970+0000", 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.395971+0000", 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9599999999999998e-07 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.395975+0000", 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0790000000000002e-06 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.395984+0000", 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9609999999999994e-06 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.395991+0000", 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.029e-06 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.396053+0000", 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2483999999999993e-05 2024-09-17T16:43:02.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.396158+0000", 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104453 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3,7", 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.396261+0000", 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010355499999999999 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.399949+0000", 2024-09-17T16:43:02.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0036877939999999999 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.403371+0000", 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0034221770000000002 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.907041+0000", 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.50366957999999995 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.907079+0000", 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8107e-05 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.907110+0000", 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0889999999999997e-05 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.872 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop_reply(client.27088.0:191141 7.6 e724/674 ondisk, result = 0)", 2024-09-17T16:43:02.873 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:33:54.403796+0000", 2024-09-17T16:43:02.873 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 548.45786591800004, 2024-09-17T16:43:02.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.50331859599999995, 2024-09-17T16:43:02.873 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.873 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.873 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.873 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.403796+0000", 2024-09-17T16:43:02.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.403796+0000", 2024-09-17T16:43:02.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.403797+0000", 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6400000000000001e-07 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.403801+0000", 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.313e-06 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.906975+0000", 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.50317347499999998 2024-09-17T16:43:02.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.906982+0000", 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.075e-06 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.907029+0000", 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7361000000000003e-05 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.907035+0000", 2024-09-17T16:43:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2480000000000001e-06 2024-09-17T16:43:02.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:33:54.907115+0000", 2024-09-17T16:43:02.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9359999999999999e-05 2024-09-17T16:43:02.877 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.877 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.877 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.878 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.19 740/739 [PushReplyOp(8:9c90d566:::benchmark_data_smithi173_130566_object2878:head)])", 2024-09-17T16:43:02.878 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:34:27.234131+0000", 2024-09-17T16:43:02.878 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 515.62753136000003, 2024-09-17T16:43:02.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49980637100000003, 2024-09-17T16:43:02.878 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.878 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.878 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:43:02.878 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.234131+0000", 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.234131+0000", 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.234131+0000", 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2799999999999996e-07 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.234135+0000", 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7120000000000002e-06 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.731984+0000", 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49784911599999998 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.731991+0000", 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4200000000000001e-06 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.733529+0000", 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015378010000000001 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:34:27.733937+0000", 2024-09-17T16:43:02.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040779400000000001 2024-09-17T16:43:02.882 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.882 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(8.8 769/767 [PushOp(8:11db60ea:::benchmark_data_smithi173_130566_object39395:head, version: 752'1440, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:11db60ea:::benchmark_data_smithi173_130566_object39395:head@752'1440, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:41.589891+0000", 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 441.271770925, 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.66289996799999995, 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.589891+0000", 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.589891+0000", 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.589892+0000", 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3500000000000005e-07 2024-09-17T16:43:02.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:41.589920+0000", 2024-09-17T16:43:02.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8104999999999999e-05 2024-09-17T16:43:02.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.251356+0000", 2024-09-17T16:43:02.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.66143590900000004 2024-09-17T16:43:02.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.251408+0000", 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2209000000000001e-05 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.251553+0000", 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000144838 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.251565+0000", 2024-09-17T16:43:02.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2187999999999999e-05 2024-09-17T16:43:02.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:42.252791+0000", 2024-09-17T16:43:02.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001225884 2024-09-17T16:43:02.887 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.887 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.887 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.17883.0:104211 8.17 e772/743 8:ed1d3a15:::benchmark_data_smithi173_130566_object9236:head v 772'3544, mlcod=772'3541)", 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:35:51.157650+0000", 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 431.704012052, 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.501241029, 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.157650+0000", 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.157650+0000", 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.157651+0000", 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.189e-06 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.157659+0000", 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4159999999999998e-06 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.657123+0000", 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.49946442899999999 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.657128+0000", 2024-09-17T16:43:02.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1159999999999998e-06 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.657214+0000", 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5395999999999994e-05 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.657237+0000", 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3680999999999999e-05 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.891 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.658861+0000", 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001623659 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:35:51.658891+0000", 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0142999999999999e-05 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.892 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDScrubReserve(8.8,GRANT,e:787,from:9,reservation_nonce:0)", 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:32.153626+0000", 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 390.70803642499999, 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79404361499999998, 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:32.153626+0000", 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:32.153626+0000", 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:32.153626+0000", 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3900000000000005e-07 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:32.153629+0000", 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6419999999999999e-06 2024-09-17T16:43:02.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:32.947358+0000", 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.79372936599999999 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:32.947372+0000", 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3906999999999999e-05 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:32.947438+0000", 2024-09-17T16:43:02.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5702000000000002e-05 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:32.947454+0000", 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6133e-05 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:32.947669+0000", 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000215326 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.896 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "rep_scrubmap(8.8 e775 from shard 9)", 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:33.255991+0000", 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 389.60567156899998, 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.70614632099999997, 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:33.255991+0000", 2024-09-17T16:43:02.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:33.255991+0000", 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:33.255992+0000", 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.147e-06 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:33.256016+0000", 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4131999999999999e-05 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:33.961950+0000", 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.70593458200000003 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:33.961958+0000", 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9489999999999996e-06 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:33.962037+0000", 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8449999999999996e-05 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:33.962048+0000", 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1e-05 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:33.962137+0000", 2024-09-17T16:43:02.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9061000000000007e-05 2024-09-17T16:43:02.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.901 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.901 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.901 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "replica_scrub(pg: 8.12,from:0'0,to:772'3529,epoch:788/744,start:8:4a214603:::benchmark_data_smithi173_130566_object68364:0,end:8:4a6b1b79:::benchmark_data_smithi173_130566_object27326:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-17T16:43:02.901 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:34.898559+0000", 2024-09-17T16:43:02.901 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 387.963102795, 2024-09-17T16:43:02.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.87656035099999996, 2024-09-17T16:43:02.902 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.902 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.902 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.902 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.902 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:34.898559+0000", 2024-09-17T16:43:02.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:34.898559+0000", 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:34.898560+0000", 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3300000000000002e-07 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:34.898564+0000", 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1910000000000002e-06 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:35.774968+0000", 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.87640386100000001 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:35.774978+0000", 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0322999999999999e-05 2024-09-17T16:43:02.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:35.775067+0000", 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8775000000000003e-05 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:35.775081+0000", 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3862e-05 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:35.775120+0000", 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8705999999999998e-05 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDScrubReserve(8.17,RELEASE,e:789,from:7,reservation_nonce:0)", 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:36:36.930483+0000", 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 385.93117869500003, 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.90503131800000003, 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.906 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:36.930483+0000", 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:36.930483+0000", 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:36.930484+0000", 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5199999999999998e-07 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:36.930486+0000", 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1380000000000001e-06 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:37.835363+0000", 2024-09-17T16:43:02.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.90487698000000005 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:37.835373+0000", 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7489999999999993e-06 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:37.835475+0000", 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000102632 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:37.835497+0000", 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1874000000000001e-05 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:36:37.835515+0000", 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7693000000000002e-05 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPushReply(8.8 830/792 [PushReplyOp(8:10b13bcc:::benchmark_data_smithi173_130566_object88053:head)])", 2024-09-17T16:43:02.910 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:37:58.561923+0000", 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 304.29973867299998, 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.507729086, 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:58.561923+0000", 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:58.561923+0000", 2024-09-17T16:43:02.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:58.561925+0000", 2024-09-17T16:43:02.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3960000000000001e-06 2024-09-17T16:43:02.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:58.561930+0000", 2024-09-17T16:43:02.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7600000000000002e-06 2024-09-17T16:43:02.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.068398+0000", 2024-09-17T16:43:02.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.50646889900000003 2024-09-17T16:43:02.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.068408+0000", 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2969999999999992e-06 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.068484+0000", 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6341999999999995e-05 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:37:59.069652+0000", 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011683920000000001 2024-09-17T16:43:02.914 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDeleteReply(8.2 e850,792 8:4005af97:::benchmark_data_smithi173_130566_object36175:head,807'4249)", 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:38:36.941441+0000", 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 265.92022154799997, 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.825007292, 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:36.941441+0000", 2024-09-17T16:43:02.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:36.941441+0000", 2024-09-17T16:43:02.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:36.941442+0000", 2024-09-17T16:43:02.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5799999999999999e-06 2024-09-17T16:43:02.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:36.941452+0000", 2024-09-17T16:43:02.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5140000000000007e-06 2024-09-17T16:43:02.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:37.765700+0000", 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.82424872800000004 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:37.765711+0000", 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0154e-05 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:37.765762+0000", 2024-09-17T16:43:02.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1924e-05 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:38:37.766448+0000", 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00068539200000000005 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(9.17 918/915 [PushOp(9:e85bc9a6:::benchmark_data_smithi173_150363_object20928:head, version: 875'1253, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:e85bc9a6:::benchmark_data_smithi173_150363_object20928:head@875'1253, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:40:57.726291+0000", 2024-09-17T16:43:02.919 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 125.135371101, 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.571449031, 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:57.726291+0000", 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:57.726291+0000", 2024-09-17T16:43:02.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:57.726292+0000", 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0299999999999998e-07 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:57.726326+0000", 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3723000000000002e-05 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:58.295763+0000", 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56943744399999996 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:58.295782+0000", 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8870000000000001e-05 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:58.295894+0000", 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112328 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:58.295910+0000", 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.571e-05 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:40:58.297740+0000", 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018301529999999999 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(9.17 946/915 [PushOp(9:ea63715a:::benchmark_data_smithi173_150363_object99267:head, version: 901'6256, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:ea63715a:::benchmark_data_smithi173_150363_object99267:head@901'6256, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:43:02.923 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:42:13.664442+0000", 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.197220053000002, 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.259087872, 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.664442+0000", 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.664442+0000", 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.664443+0000", 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7499999999999999e-07 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.664474+0000", 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0620000000000002e-05 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.921504+0000", 2024-09-17T16:43:02.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.257030641 2024-09-17T16:43:02.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.921517+0000", 2024-09-17T16:43:02.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2381999999999999e-05 2024-09-17T16:43:02.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:02.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.921595+0000", 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8275999999999994e-05 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.921604+0000", 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1730000000000001e-06 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.923530+0000", 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019259050000000001 2024-09-17T16:43:02.928 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(9.10 946/915 [PushOp(9:0ed2bb03:::benchmark_data_smithi173_150363_object71945:head, version: 893'4383, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:0ed2bb03:::benchmark_data_smithi173_150363_object71945:head@893'4383, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:42:13.906251+0000", 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 48.955411503000001, 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.50911889499999996, 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.906251+0000", 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.906251+0000", 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.906252+0000", 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8169999999999999e-06 2024-09-17T16:43:02.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:13.906293+0000", 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0531000000000002e-05 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:14.412532+0000", 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.50623894800000002 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:14.412551+0000", 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9126e-05 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:14.412674+0000", 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000122922 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:14.412689+0000", 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5373999999999999e-05 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:14.415369+0000", 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026801770000000002 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27874.0:174791 9.12 e949/942 9:4ce52c5f:::benchmark_data_smithi173_150363_object67886:head v 949'9134, mlcod=949'9131)", 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:42:18.198497+0000", 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 44.663165171000003, 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.14284300699999999, 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.933 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:18.198497+0000", 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:18.198497+0000", 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:18.198498+0000", 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.026e-06 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:18.198506+0000", 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0220000000000004e-06 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:18.339449+0000", 2024-09-17T16:43:02.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.140942664 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:18.339454+0000", 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9409999999999998e-06 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:18.339508+0000", 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4273e-05 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:18.339537+0000", 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.951e-05 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:18.341282+0000", 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001744879 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:18.341340+0000", 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7692000000000003e-05 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.937 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(9.17 e950,915 9:ebfdf575:::benchmark_data_smithi173_150363_object4137:head,907'6967)", 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:42:20.032859+0000", 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.828803499999999, 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25055217000000002, 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:20.032859+0000", 2024-09-17T16:43:02.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:20.032859+0000", 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:20.032859+0000", 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:20.032864+0000", 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7690000000000004e-06 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:20.283245+0000", 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.25038057499999999 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:20.283254+0000", 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9910000000000008e-06 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:20.283333+0000", 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9342e-05 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:20.283341+0000", 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7190000000000001e-06 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:20.283411+0000", 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0049999999999995e-05 2024-09-17T16:43:02.941 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(9.10 951/915 [PushOp(9:0e7af462:::benchmark_data_smithi173_150363_object85378:head, version: 897'5196, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:0e7af462:::benchmark_data_smithi173_150363_object85378:head@897'5196, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:42:22.995737+0000", 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 39.865925226999998, 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.20506871099999999, 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:22.995737+0000", 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:22.995737+0000", 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:22.995738+0000", 2024-09-17T16:43:02.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.155e-06 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:22.995798+0000", 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0254000000000003e-05 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.198643+0000", 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.20284453299999999 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.198664+0000", 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1095999999999999e-05 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.198767+0000", 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103211 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.198783+0000", 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5569999999999998e-05 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.200806+0000", 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020228920000000001 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGRecoveryDelete(9.17 e951,915 9:eae987e6:::benchmark_data_smithi173_150363_object11700:head,909'7404)", 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:42:23.120854+0000", 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 39.740807635000003, 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.14347962, 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.946 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.120854+0000", 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.120854+0000", 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.120855+0000", 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8100000000000001e-07 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.120868+0000", 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2974e-05 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.264136+0000", 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.14326718799999999 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.264144+0000", 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6000000000000007e-06 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.264236+0000", 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1825999999999998e-05 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.264244+0000", 2024-09-17T16:43:02.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3480000000000004e-06 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:23.264334+0000", 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9703000000000005e-05 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27874.0:192218 9.c e957/872 9:3af6c2de:::benchmark_data_smithi173_150363_object85313:head v 957'12110, mlcod=957'12106)", 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:42:37.949021+0000", 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 24.912641312000002, 2024-09-17T16:43:02.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.23372583599999999, 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:37.949021+0000", 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:37.949021+0000", 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:37.949022+0000", 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1800000000000005e-07 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:37.949028+0000", 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1770000000000003e-06 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:38.177945+0000", 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.228917236 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:38.177955+0000", 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0348e-05 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:38.178024+0000", 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8201999999999995e-05 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:38.178038+0000", 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4914e-05 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:38.182659+0000", 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0046205600000000001 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:38.182747+0000", 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7581000000000001e-05 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.954 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27874.0:193343 9.12 e959/942 9:4c5a7ff8:::benchmark_data_smithi173_150363_object86438:head v 959'9694, mlcod=959'9690)", 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:42:39.677426+0000", 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.184235610999998, 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24830959899999999, 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:39.677426+0000", 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:39.677426+0000", 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:39.677428+0000", 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6199999999999999e-06 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:39.677438+0000", 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8640000000000008e-06 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:39.923965+0000", 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.24652733299999999 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:39.923970+0000", 2024-09-17T16:43:02.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3930000000000001e-06 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:39.924009+0000", 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9273999999999997e-05 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:39.924023+0000", 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3823e-05 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:39.925704+0000", 2024-09-17T16:43:02.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016811860000000001 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:42:39.925736+0000", 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2106000000000001e-05 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:02.959 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:43:02.985 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:02.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:02 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:43:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:02 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:43:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:02 smithi017 ceph-mon[27571]: osdmap e971: 12 total, 12 up, 12 in 2024-09-17T16:43:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2376645455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:02.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:02 smithi017 ceph-mon[27571]: pgmap v2137: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 32 active+clean; 353 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 6810/16941 objects degraded (40.198%); 4.1 MiB/s, 64 objects/s recovering 2024-09-17T16:43:03.087 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:03.120 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:03.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:02 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:43:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:02 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:43:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:02 smithi173 ceph-mon[30530]: osdmap e971: 12 total, 12 up, 12 in 2024-09-17T16:43:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2376645455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:03.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:02 smithi173 ceph-mon[30530]: pgmap v2137: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 32 active+clean; 353 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 6810/16941 objects degraded (40.198%); 4.1 MiB/s, 64 objects/s recovering 2024-09-17T16:43:03.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:03.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:03.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:02 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:43:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:02 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:43:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:02 smithi071 ceph-mon[30002]: osdmap e971: 12 total, 12 up, 12 in 2024-09-17T16:43:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2376645455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:03.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:02 smithi071 ceph-mon[30002]: pgmap v2137: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 32 active+clean; 353 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 6810/16941 objects degraded (40.198%); 4.1 MiB/s, 64 objects/s recovering 2024-09-17T16:43:03.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:03.373 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:03.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:03.476 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:03.504 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:03.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:03.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:03.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:03.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:03.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:03.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:03.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:03.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:03.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:03.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:03.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:03.952 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:03.989 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:04.020 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:04.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:04.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:04.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:04.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:04.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:03 smithi173 ceph-mon[30530]: Health check failed: Degraded data redundancy: 6810/16941 objects degraded (40.198%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:04.163 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:03 smithi017 ceph-mon[27571]: Health check failed: Degraded data redundancy: 6810/16941 objects degraded (40.198%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:04.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:04.273 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:04.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:04.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:03 smithi071 ceph-mon[30002]: Health check failed: Degraded data redundancy: 6810/16941 objects degraded (40.198%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:04.363 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 7, 8, 9, 10, 11, 1, 5, 4, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 8, 9, 11, 7, 10, 5] 2024-09-17T16:43:04.363 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T16:43:04.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:04.437 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:04.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:04.468 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-17T16:43:04.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:04.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:04.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:04.671 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:04.708 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:04.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:04.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:04.850 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:04.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:04.952 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:04.980 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:04.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:04.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:05.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:04 smithi017 ceph-mon[27571]: pgmap v2138: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 30 active+clean; 299 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 6815/14349 objects degraded (47.495%); 96/14349 objects misplaced (0.669%); 5.4 MiB/s, 88 objects/s recovering 2024-09-17T16:43:05.068 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:04 smithi017 ceph-mon[27571]: osdmap e972: 12 total, 12 up, 12 in 2024-09-17T16:43:05.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:05.105 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:05.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:05.130 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:04 smithi173 ceph-mon[30530]: pgmap v2138: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 30 active+clean; 299 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 6815/14349 objects degraded (47.495%); 96/14349 objects misplaced (0.669%); 5.4 MiB/s, 88 objects/s recovering 2024-09-17T16:43:05.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:04 smithi173 ceph-mon[30530]: osdmap e972: 12 total, 12 up, 12 in 2024-09-17T16:43:05.158 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:05.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:05.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:05.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:05.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:05.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:04 smithi071 ceph-mon[30002]: pgmap v2138: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 remapped+peering, 30 active+clean; 299 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 6815/14349 objects degraded (47.495%); 96/14349 objects misplaced (0.669%); 5.4 MiB/s, 88 objects/s recovering 2024-09-17T16:43:05.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:04 smithi071 ceph-mon[30002]: osdmap e972: 12 total, 12 up, 12 in 2024-09-17T16:43:05.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:05.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:05.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:05.461 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:05.486 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:05.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:05.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:05.594 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:43:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:05.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:05.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:05.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:05.748 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:05.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:05.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:05.851 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:05.882 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:43:05.892 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:05.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1118656102' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:43:05.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:05 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:43:05.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:06.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:06.022 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:06.123 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:06.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1118656102' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:43:06.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:05 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:43:06.151 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:06.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:06.244 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:43:06.244 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-17T16:43:06.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.27874.0:212496 9.4 9:272f45c5:::benchmark_data_smithi173_150363_object105591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e973)", 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:43:06.229740+0000", 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.010589321, 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010624829000000001, 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for sub ops", 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.27874", 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 212496 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.245 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.229740+0000", 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.229740+0000", 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:06.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.229741+0000", 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6470000000000001e-06 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.229747+0000", 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8730000000000004e-06 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.229758+0000", 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0200000000000001e-05 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.229765+0000", 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989999999999999e-06 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.229850+0000", 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5118999999999996e-05 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.229978+0000", 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000128133 2024-09-17T16:43:06.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 4,8", 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.230511+0000", 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00053262299999999997 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.234012+0000", 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00350155 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.234061+0000", 2024-09-17T16:43:06.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8958000000000001e-05 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.27874.0:212502 9.d e973/858 9:b20590f6:::benchmark_data_smithi173_150363_object105597:head v 973'12957, mlcod=973'12954)", 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-17T16:43:06.235247+0000", 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.0050816999999999998, 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052017779999999998, 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-17T16:43:06.250 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.235247+0000", 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.235247+0000", 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.235248+0000", 2024-09-17T16:43:06.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.235254+0000", 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0090000000000004e-06 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.235294+0000", 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9582000000000002e-05 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-17T16:43:06.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.235313+0000", 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9361e-05 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.235397+0000", 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3384e-05 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-17T16:43:06.235427+0000", 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0290999999999998e-05 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-17T16:43:06.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:06.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-17T16:43:06.254 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-17T16:43:06.254 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 2 2024-09-17T16:43:06.254 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:43:06.254 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:06.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:06.281 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:06.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:06.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1118656102' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:43:06.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:05 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:43:06.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:06.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:06.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:06.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:06.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:06.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:06.639 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:06.670 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:06.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:06.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:06.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:06.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:06.880 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T16:43:06.901 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:06.902 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:06 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:06.902 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:06 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:43:06.902 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:06 smithi017 ceph-mon[27571]: osdmap e973: 12 total, 12 up, 12 in 2024-09-17T16:43:06.902 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:06 smithi017 ceph-mon[27571]: pgmap v2141: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 192 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 10276/9198 objects degraded (111.720%); 381/9198 objects misplaced (4.142%); 2.7 MiB/s, 81 objects/s recovering 2024-09-17T16:43:06.903 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3926678287' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:43:06.927 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:06.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:07.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:07.058 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:07.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:06 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:06 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:43:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:06 smithi173 ceph-mon[30530]: osdmap e973: 12 total, 12 up, 12 in 2024-09-17T16:43:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:06 smithi173 ceph-mon[30530]: pgmap v2141: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 192 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 10276/9198 objects degraded (111.720%); 381/9198 objects misplaced (4.142%); 2.7 MiB/s, 81 objects/s recovering 2024-09-17T16:43:07.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3926678287' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:43:07.159 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:07.188 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:07.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:07.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:07.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:06 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:06 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:43:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:06 smithi071 ceph-mon[30002]: osdmap e973: 12 total, 12 up, 12 in 2024-09-17T16:43:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:06 smithi071 ceph-mon[30002]: pgmap v2141: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 192 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 10276/9198 objects degraded (111.720%); 381/9198 objects misplaced (4.142%); 2.7 MiB/s, 81 objects/s recovering 2024-09-17T16:43:07.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3926678287' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:43:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:07.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:07.416 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:07.442 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:07.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:07.544 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:07.567 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:07.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:07.669 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:07.698 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:07.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:07.801 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:07.833 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:07.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:07.936 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:07.964 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:07.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:08.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:08.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:08.088 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:08.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:07 smithi173 ceph-mon[30530]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:43:08.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3926678287' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:43:08.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:07 smithi173 ceph-mon[30530]: osdmap e974: 12 total, 12 up, 12 in 2024-09-17T16:43:08.189 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:08.215 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:08.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:07 smithi017 ceph-mon[27571]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:43:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3926678287' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:43:08.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:07 smithi017 ceph-mon[27571]: osdmap e974: 12 total, 12 up, 12 in 2024-09-17T16:43:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:07 smithi071 ceph-mon[30002]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-17T16:43:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3926678287' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:43:08.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:07 smithi071 ceph-mon[30002]: osdmap e974: 12 total, 12 up, 12 in 2024-09-17T16:43:08.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:08.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:08.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:08.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:08.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:08.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:08.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:08.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:08.586 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:08.687 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:08.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:08.710 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:08.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:08.811 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:08.835 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:08.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:08.936 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:08.960 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:08.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:09.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:09.087 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:09.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:09.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:08 smithi173 ceph-mon[30530]: pgmap v2143: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 78 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 10273/3714 objects degraded (276.602%); 285/3714 objects misplaced (7.674%); 3.5 MiB/s, 189 objects/s recovering 2024-09-17T16:43:09.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:09.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:08 smithi017 ceph-mon[27571]: pgmap v2143: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 78 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 10273/3714 objects degraded (276.602%); 285/3714 objects misplaced (7.674%); 3.5 MiB/s, 189 objects/s recovering 2024-09-17T16:43:09.209 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:09.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:09.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:09.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:08 smithi071 ceph-mon[30002]: pgmap v2143: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 78 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s; 10273/3714 objects degraded (276.602%); 285/3714 objects misplaced (7.674%); 3.5 MiB/s, 189 objects/s recovering 2024-09-17T16:43:09.333 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:09.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:09.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:09.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:09.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:09.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:09.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:09.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:09.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:09.684 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:09.707 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:09.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:09.809 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:09.833 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:09.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:09.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:09.958 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:09.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:10.059 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:10.080 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:10.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:10.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:09 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 10273/3714 objects degraded (276.602%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:10.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:09 smithi173 ceph-mon[30530]: osdmap e975: 12 total, 12 up, 12 in 2024-09-17T16:43:10.182 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:10.203 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:10.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:10.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:09 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 10273/3714 objects degraded (276.602%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:10.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:09 smithi017 ceph-mon[27571]: osdmap e975: 12 total, 12 up, 12 in 2024-09-17T16:43:10.305 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:10.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:09 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 10273/3714 objects degraded (276.602%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:10.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:09 smithi071 ceph-mon[30002]: osdmap e975: 12 total, 12 up, 12 in 2024-09-17T16:43:10.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:10.330 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:10.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:10.432 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:10.454 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:10.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:10.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:10.577 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:10.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:10.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:10.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:10.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:10.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:10.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:10.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:10.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:10.925 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:10.951 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:10.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:11.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:11.075 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:11.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:11.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:10 smithi173 ceph-mon[30530]: pgmap v2145: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 16 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s; 10315/753 objects degraded (1369.854%); 151/753 objects misplaced (20.053%); 1.2 MiB/s, 226 objects/s recovering 2024-09-17T16:43:11.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:11.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:11.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:11.215 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:10 smithi071 ceph-mon[30002]: pgmap v2145: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 16 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s; 10315/753 objects degraded (1369.854%); 151/753 objects misplaced (20.053%); 1.2 MiB/s, 226 objects/s recovering 2024-09-17T16:43:11.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:10 smithi017 ceph-mon[27571]: pgmap v2145: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 16 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 804 op/s; 10315/753 objects degraded (1369.854%); 151/753 objects misplaced (20.053%); 1.2 MiB/s, 226 objects/s recovering 2024-09-17T16:43:11.300 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:11.321 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:11.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:11.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-17T16:43:11.392 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:11.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:11.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:11.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:11.547 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:11.568 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:11.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:11.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:11.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:11.693 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:11.794 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:11.817 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:11.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:11.915 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T16:43:11.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:11.941 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:11.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:11.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:11 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:43:11.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:11 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:43:12.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:12.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:12.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:12.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:11 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:43:12.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:11 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:43:12.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:12.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:12.191 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:12.292 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:12.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:11 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:43:12.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:11 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:43:12.316 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:12.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:12.417 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:12.438 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:12.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:12.510 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:43:12.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:12.562 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:12.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:12.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:12.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:12.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:12.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:12.702 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:12.730 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:43:12.730 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:43:12.765 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:43:12.765 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:43:12.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:12.789 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:43:12.790 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:43:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:12.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:12.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:12.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:43:12.816 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:43:12.839 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:43:12.839 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:43:12.872 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:43:12.873 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:43:12.901 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:43:12.901 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:43:12.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:12.926 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:43:12.927 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:43:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:12.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:12.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:12.952 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:43:12.953 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:43:12.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:12 smithi017 ceph-mon[27571]: osdmap e976: 12 total, 12 up, 12 in 2024-09-17T16:43:12.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:12 smithi017 ceph-mon[27571]: pgmap v2147: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 4.3 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 502 op/s; 10315/189 objects degraded (5457.672%); 151/189 objects misplaced (79.894%); 1024 KiB/s, 165 objects/s recovering 2024-09-17T16:43:12.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1776083934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:12.977 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:43:12.977 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:43:13.003 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:43:13.003 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:43:13.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:13.038 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:43:13.038 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:43:13.057 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:13.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:12 smithi173 ceph-mon[30530]: osdmap e976: 12 total, 12 up, 12 in 2024-09-17T16:43:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:12 smithi173 ceph-mon[30530]: pgmap v2147: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 4.3 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 502 op/s; 10315/189 objects degraded (5457.672%); 151/189 objects misplaced (79.894%); 1024 KiB/s, 165 objects/s recovering 2024-09-17T16:43:13.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1776083934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:13.161 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:13.194 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:13.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:13.292 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:13.292 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:13.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:13.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:12 smithi071 ceph-mon[30002]: osdmap e976: 12 total, 12 up, 12 in 2024-09-17T16:43:13.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:12 smithi071 ceph-mon[30002]: pgmap v2147: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 4.3 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 502 op/s; 10315/189 objects degraded (5457.672%); 151/189 objects misplaced (79.894%); 1024 KiB/s, 165 objects/s recovering 2024-09-17T16:43:13.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1776083934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:13.320 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:13.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:13.422 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:13.443 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:13.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:13.545 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:13.570 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:13.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:13.672 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:13.698 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:13.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:13.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:13.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:13.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:13.923 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:13.947 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:13.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:13.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:13.979 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:13 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:43:13.979 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:13 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:43:14.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:14.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:14.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:14.144 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:13 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:43:14.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:13 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:43:14.172 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:14.194 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:14.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:14.295 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:14.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:13 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:43:14.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:13 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:43:14.317 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:14.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:14.419 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:14.446 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:14.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:14.548 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:14.575 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:14.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:14.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:14.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:14.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:14.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:14.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:14.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:14.877 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-17T16:43:14.888 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:14.913 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:14 smithi017 ceph-mon[27571]: osdmap e977: 12 total, 12 up, 12 in 2024-09-17T16:43:14.913 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:14 smithi017 ceph-mon[27571]: pgmap v2149: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 208 op/s; 10315/9 objects degraded (114611.111%); 151/9 objects misplaced (1677.778%); 0 B/s, 260 objects/s recovering 2024-09-17T16:43:14.913 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:14 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 10315/189 objects degraded (5457.672%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:14.913 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:14 smithi017 ceph-mon[27571]: osdmap e978: 12 total, 12 up, 12 in 2024-09-17T16:43:14.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:14.936 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:14.946 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:14.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:15.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:15.072 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:15.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:15.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:15.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:15.199 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:15.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:15.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:15.301 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:14 smithi071 ceph-mon[30002]: osdmap e977: 12 total, 12 up, 12 in 2024-09-17T16:43:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:14 smithi071 ceph-mon[30002]: pgmap v2149: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 208 op/s; 10315/9 objects degraded (114611.111%); 151/9 objects misplaced (1677.778%); 0 B/s, 260 objects/s recovering 2024-09-17T16:43:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:14 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 10315/189 objects degraded (5457.672%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:15.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:14 smithi071 ceph-mon[30002]: osdmap e978: 12 total, 12 up, 12 in 2024-09-17T16:43:15.324 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:15.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:14 smithi173 ceph-mon[30530]: osdmap e977: 12 total, 12 up, 12 in 2024-09-17T16:43:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:14 smithi173 ceph-mon[30530]: pgmap v2149: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 208 op/s; 10315/9 objects degraded (114611.111%); 151/9 objects misplaced (1677.778%); 0 B/s, 260 objects/s recovering 2024-09-17T16:43:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:14 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 10315/189 objects degraded (5457.672%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:15.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:14 smithi173 ceph-mon[30530]: osdmap e978: 12 total, 12 up, 12 in 2024-09-17T16:43:15.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:15.448 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:15.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:15.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:15.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:15.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:15.673 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:15.700 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:15.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:15.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:15.823 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:15.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:15.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:15.946 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:15.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:16.024 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:43:16.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:16.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:16.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:16.171 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:16.195 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:16.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:16.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:16.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:16.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:16.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:16.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:16.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:16.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:16.544 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:16.569 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:16.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:16.615 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:16.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:16.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:16.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:16.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:16.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:16.815 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:16.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:16.884 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:16.911 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:43:16.911 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T16:43:16.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:16.945 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:16.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:16 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:43:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:16 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:43:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:16 smithi017 ceph-mon[27571]: osdmap e979: 12 total, 12 up, 12 in 2024-09-17T16:43:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/100141039' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:43:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:16 smithi017 ceph-mon[27571]: pgmap v2152: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 13 op/s; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 448 objects/s recovering 2024-09-17T16:43:16.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1369381878' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:43:17.047 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:17.072 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:17.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:17.150 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:17.150 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:17.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:17.198 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:17.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:17.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:17.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:17.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:16 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:43:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:16 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:43:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:16 smithi071 ceph-mon[30002]: osdmap e979: 12 total, 12 up, 12 in 2024-09-17T16:43:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/100141039' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:43:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:16 smithi071 ceph-mon[30002]: pgmap v2152: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 13 op/s; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 448 objects/s recovering 2024-09-17T16:43:17.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1369381878' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:43:17.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:17.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:16 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:43:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:16 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:43:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:16 smithi173 ceph-mon[30530]: osdmap e979: 12 total, 12 up, 12 in 2024-09-17T16:43:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/100141039' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:43:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:16 smithi173 ceph-mon[30530]: pgmap v2152: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 13 op/s; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 448 objects/s recovering 2024-09-17T16:43:17.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1369381878' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:43:17.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:17.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:17.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:17.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:17.548 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:17.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:17.575 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:17.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:17.677 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:17.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:17.699 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:17.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:17.800 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:17.822 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:17.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:17.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:17.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:17.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:18.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:18.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:18.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:18.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:18.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:18.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:17 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:43:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/100141039' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:43:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1369381878' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:43:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:17 smithi017 ceph-mon[27571]: osdmap e980: 12 total, 12 up, 12 in 2024-09-17T16:43:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:17 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:43:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:17 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:43:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:17 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:43:18.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:17 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:43:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/100141039' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:43:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1369381878' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:43:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:17 smithi071 ceph-mon[30002]: osdmap e980: 12 total, 12 up, 12 in 2024-09-17T16:43:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:17 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:43:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:17 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:43:18.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:17 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:43:18.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:18.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:18.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:17 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:43:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/100141039' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:43:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1369381878' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:43:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:17 smithi173 ceph-mon[30530]: osdmap e980: 12 total, 12 up, 12 in 2024-09-17T16:43:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:17 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:43:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:17 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:43:18.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:17 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:43:18.419 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:18.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:18.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:18.483 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:43:18.483 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:43:18.508 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:43:18.508 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:43:18.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:43:18.530 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:43:18.542 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:18.551 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:43:18.551 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:43:18.563 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:18.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:18.572 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:43:18.572 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:43:18.665 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:18.688 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:18.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:18.789 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:18.803 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:43:18.804 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:43:18.811 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:18.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:18.913 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:18.936 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:18.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:18.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:19.032 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:43:19.032 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:43:19.037 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:19.059 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:43:19.059 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:43:19.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:18 smithi017 ceph-mon[27571]: osdmap e981: 12 total, 12 up, 12 in 2024-09-17T16:43:19.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:18 smithi017 ceph-mon[27571]: pgmap v2155: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 489 objects/s recovering 2024-09-17T16:43:19.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:43:19.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:18 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:43:19.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:18 smithi017 ceph-mon[27571]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:43:19.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2374026156' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:19.061 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:19.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:19.083 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:43:19.083 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:43:19.109 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:43:19.109 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:43:19.134 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:43:19.135 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:43:19.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:19.163 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:43:19.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:43:19.184 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:19.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:19.196 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T16:43:19.286 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:19.307 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:19.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:19.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:18 smithi071 ceph-mon[30002]: osdmap e981: 12 total, 12 up, 12 in 2024-09-17T16:43:19.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:18 smithi071 ceph-mon[30002]: pgmap v2155: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 489 objects/s recovering 2024-09-17T16:43:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:43:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:18 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:43:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:18 smithi071 ceph-mon[30002]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:43:19.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2374026156' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:18 smithi173 ceph-mon[30530]: osdmap e981: 12 total, 12 up, 12 in 2024-09-17T16:43:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:18 smithi173 ceph-mon[30530]: pgmap v2155: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 489 objects/s recovering 2024-09-17T16:43:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:43:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:18 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:43:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:18 smithi173 ceph-mon[30530]: from='mgr.14152 ' entity='mgr.y' 2024-09-17T16:43:19.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2374026156' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:19.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:19.431 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:19.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:19.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:19.533 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:19.556 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:19.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:19.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:19.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:19.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:19.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:19.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:19.806 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:19.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:19.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:19.935 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:19.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:19.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:20.037 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:20.064 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:20.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:20.166 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:20.190 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:20.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:20.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:19 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 10317/9 objects degraded (114633.333%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:20.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:19 smithi017 ceph-mon[27571]: osdmap e982: 12 total, 12 up, 12 in 2024-09-17T16:43:20.293 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:20.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:19 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 10317/9 objects degraded (114633.333%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:20.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:19 smithi071 ceph-mon[30002]: osdmap e982: 12 total, 12 up, 12 in 2024-09-17T16:43:20.314 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:20.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:19 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 10317/9 objects degraded (114633.333%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:20.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:19 smithi173 ceph-mon[30530]: osdmap e982: 12 total, 12 up, 12 in 2024-09-17T16:43:20.415 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:20.437 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:20.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:20.539 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:20.564 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:20.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:20.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:20.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:20.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:20.693 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-17T16:43:20.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:20.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:20.816 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:20.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:20.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:20.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:20.943 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:21.044 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:21.094 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:21.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:21.160 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:21.195 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:21.222 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:21.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:21.240 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:20 smithi017 ceph-mon[27571]: pgmap v2157: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 410 objects/s recovering 2024-09-17T16:43:21.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:20 smithi173 ceph-mon[30530]: pgmap v2157: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 410 objects/s recovering 2024-09-17T16:43:21.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:20 smithi071 ceph-mon[30002]: pgmap v2157: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 410 objects/s recovering 2024-09-17T16:43:21.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:21.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:21.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:21.438 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:21.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:21.475 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:21.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:21.577 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:21.600 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:21.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:21.700 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:21.722 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:21.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:21.825 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:21.845 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:21.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:21.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:21.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:21.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:22.031 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:43:22.072 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:22.093 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:22.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:22.146 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:43:22.146 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:43:22.147 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:43:22.147 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:43:22.190 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T16:43:22.195 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:22.222 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:22.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:22.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:22 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2594184755' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:43:22.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:22.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:22.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:22.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:22.476 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:22.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:22.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:22 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2594184755' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:43:22.577 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:22.599 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:22.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:22.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:22 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2594184755' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:43:22.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:22.725 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:22.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:22.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:22.850 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:22.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:22.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:22.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:22.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:22.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:22.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:23.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:23.101 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:23.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:23.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:43:23.203 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:23.230 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:23.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:23.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:23.356 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:23.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:23.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:23.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:23.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:23 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2594184755' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:43:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:23 smithi017 ceph-mon[27571]: osdmap e983: 12 total, 12 up, 12 in 2024-09-17T16:43:23.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:23 smithi017 ceph-mon[27571]: pgmap v2159: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 333 objects/s recovering 2024-09-17T16:43:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:23 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2594184755' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:43:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:23 smithi071 ceph-mon[30002]: osdmap e983: 12 total, 12 up, 12 in 2024-09-17T16:43:23.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:23 smithi071 ceph-mon[30002]: pgmap v2159: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 333 objects/s recovering 2024-09-17T16:43:23.581 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:23.603 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:23.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:23 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2594184755' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:43:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:23 smithi173 ceph-mon[30530]: osdmap e983: 12 total, 12 up, 12 in 2024-09-17T16:43:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:23 smithi173 ceph-mon[30530]: pgmap v2159: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 333 objects/s recovering 2024-09-17T16:43:23.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:23.726 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:23.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:23.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:23.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:23.850 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:23.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:23.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:23.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:23.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:23.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:24.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:24.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:24.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:24.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:24.226 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:24.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:24.328 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:24.350 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:24.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:24.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:24.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:24.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:24.579 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:24.603 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:24.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:24.705 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:24.727 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:24.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:24.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:24.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:24.856 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:24.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:24.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:24.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:24.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:25.080 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:25.102 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:25.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:25.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:25.204 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:25.226 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:25.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:25.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:25.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:25.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:25.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:25.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:25.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:25.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:25 smithi017 ceph-mon[27571]: pgmap v2160: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 397 objects/s recovering 2024-09-17T16:43:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:25 smithi017 ceph-mon[27571]: osdmap e984: 12 total, 12 up, 12 in 2024-09-17T16:43:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:25 smithi071 ceph-mon[30002]: pgmap v2160: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 397 objects/s recovering 2024-09-17T16:43:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:25 smithi071 ceph-mon[30002]: osdmap e984: 12 total, 12 up, 12 in 2024-09-17T16:43:25.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:25.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:25.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:25 smithi173 ceph-mon[30530]: pgmap v2160: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10317/9 objects degraded (114633.333%); 151/9 objects misplaced (1677.778%); 0 B/s, 397 objects/s recovering 2024-09-17T16:43:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:25 smithi173 ceph-mon[30530]: osdmap e984: 12 total, 12 up, 12 in 2024-09-17T16:43:25.700 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:25.722 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:25.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:25.825 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:25.847 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:25.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:25.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:25.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:25.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:25.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:25.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:26.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:26.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:26.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:26.095 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:26.197 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:26.220 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:26.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:26.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:26.344 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:26.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:26.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:26.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:26.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:26.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:26 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:43:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:26 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:43:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:26 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:43:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:26 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:43:26.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:26 smithi017 ceph-mon[27571]: osdmap e985: 12 total, 12 up, 12 in 2024-09-17T16:43:26.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:26 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:43:26.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:26 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:43:26.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:26 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:43:26.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:26 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:43:26.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:26 smithi071 ceph-mon[30002]: osdmap e985: 12 total, 12 up, 12 in 2024-09-17T16:43:26.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:26.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:26.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:26 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:43:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:26 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:43:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:26 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:43:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:26 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:43:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:26 smithi173 ceph-mon[30530]: osdmap e985: 12 total, 12 up, 12 in 2024-09-17T16:43:26.692 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:26.715 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:26.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:26.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:26.837 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:26.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:26.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:26.939 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:26.965 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:26.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:27.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-17T16:43:27.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:27.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:27.090 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:27.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:27.213 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:27.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:27 smithi017 ceph-mon[27571]: pgmap v2163: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10315/9 objects degraded (114611.111%); 151/9 objects misplaced (1677.778%); 0 B/s, 351 objects/s recovering 2024-09-17T16:43:27.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:27 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 10315/9 objects degraded (114611.111%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:27.315 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:27.337 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:27.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:27.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:27.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:27.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:27.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:27 smithi071 ceph-mon[30002]: pgmap v2163: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10315/9 objects degraded (114611.111%); 151/9 objects misplaced (1677.778%); 0 B/s, 351 objects/s recovering 2024-09-17T16:43:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:27 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 10315/9 objects degraded (114611.111%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:27.564 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:27.588 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:27.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:27 smithi173 ceph-mon[30530]: pgmap v2163: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10315/9 objects degraded (114611.111%); 151/9 objects misplaced (1677.778%); 0 B/s, 351 objects/s recovering 2024-09-17T16:43:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:27 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 10315/9 objects degraded (114611.111%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:27.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:27.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:27.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:27.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:27.839 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:27.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:27.941 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:27.963 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:27.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:28.066 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:28.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:28.089 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:28.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:28.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:28.212 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:28.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:28.269 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:28.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T16:43:28.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:28.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:28.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:28.399 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:28 smithi017 ceph-mon[27571]: 9.17 scrub starts 2024-09-17T16:43:28.399 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:28 smithi017 ceph-mon[27571]: 9.17 scrub ok 2024-09-17T16:43:28.399 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:28 smithi017 ceph-mon[27571]: osdmap e986: 12 total, 12 up, 12 in 2024-09-17T16:43:28.400 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:28 smithi017 ceph-mon[27571]: 9.18 scrub starts 2024-09-17T16:43:28.400 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:28 smithi017 ceph-mon[27571]: 9.18 scrub ok 2024-09-17T16:43:28.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:28.467 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:28.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:28.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:28 smithi071 ceph-mon[30002]: 9.17 scrub starts 2024-09-17T16:43:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:28 smithi071 ceph-mon[30002]: 9.17 scrub ok 2024-09-17T16:43:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:28 smithi071 ceph-mon[30002]: osdmap e986: 12 total, 12 up, 12 in 2024-09-17T16:43:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:28 smithi071 ceph-mon[30002]: 9.18 scrub starts 2024-09-17T16:43:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:28 smithi071 ceph-mon[30002]: 9.18 scrub ok 2024-09-17T16:43:28.569 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:28.596 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:28.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:28 smithi173 ceph-mon[30530]: 9.17 scrub starts 2024-09-17T16:43:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:28 smithi173 ceph-mon[30530]: 9.17 scrub ok 2024-09-17T16:43:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:28 smithi173 ceph-mon[30530]: osdmap e986: 12 total, 12 up, 12 in 2024-09-17T16:43:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:28 smithi173 ceph-mon[30530]: 9.18 scrub starts 2024-09-17T16:43:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:28 smithi173 ceph-mon[30530]: 9.18 scrub ok 2024-09-17T16:43:28.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:28.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:28.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:28.825 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:28.847 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:28.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:28.926 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:28.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:28.970 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:28.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:28.974 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:29.072 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:29.096 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:29.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:29.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:29.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:29.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:29.320 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:43:29.320 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:43:29.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:29 smithi017 ceph-mon[27571]: pgmap v2165: 37 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10313/9 objects degraded (114588.889%); 151/9 objects misplaced (1677.778%); 0 B/s, 635 objects/s recovering 2024-09-17T16:43:29.321 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:29 smithi017 ceph-mon[27571]: osdmap e987: 12 total, 12 up, 12 in 2024-09-17T16:43:29.326 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:29.343 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:43:29.344 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:43:29.348 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:29.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:29.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:29.366 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:43:29.366 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:43:29.396 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:43:29.396 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:43:29.420 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:43:29.421 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:43:29.445 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:43:29.445 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:43:29.450 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:29.472 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:43:29.473 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:43:29.477 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:29.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:29.498 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:43:29.498 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:43:29.524 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:29.528 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:43:29.528 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:43:29.563 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:43:29.563 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:43:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:29 smithi071 ceph-mon[30002]: pgmap v2165: 37 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10313/9 objects degraded (114588.889%); 151/9 objects misplaced (1677.778%); 0 B/s, 635 objects/s recovering 2024-09-17T16:43:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:29 smithi071 ceph-mon[30002]: osdmap e987: 12 total, 12 up, 12 in 2024-09-17T16:43:29.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:29.587 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:43:29.587 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:43:29.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:29.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:29.610 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:43:29.611 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:43:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:29 smithi173 ceph-mon[30530]: pgmap v2165: 37 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10313/9 objects degraded (114588.889%); 151/9 objects misplaced (1677.778%); 0 B/s, 635 objects/s recovering 2024-09-17T16:43:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:29 smithi173 ceph-mon[30530]: osdmap e987: 12 total, 12 up, 12 in 2024-09-17T16:43:29.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:29.734 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:29.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:29.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:29.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:29.860 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:29.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:29.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:29.984 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:29.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:30.086 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:30.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:30.111 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:30.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:30.184 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:30.184 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:30.212 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:30.236 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:30.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:30.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:30.360 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:30.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:30.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:30 smithi017 ceph-mon[27571]: osdmap e988: 12 total, 12 up, 12 in 2024-09-17T16:43:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:43:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:43:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:43:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:43:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:43:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:43:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:43:30.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1137845881' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:30.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:30.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:30 smithi071 ceph-mon[30002]: osdmap e988: 12 total, 12 up, 12 in 2024-09-17T16:43:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:43:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:43:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:43:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:43:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:43:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:43:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:43:30.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1137845881' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:30.591 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:30.616 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:30.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:30 smithi173 ceph-mon[30530]: osdmap e988: 12 total, 12 up, 12 in 2024-09-17T16:43:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:43:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:43:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:43:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:43:30.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:43:30.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:43:30.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:43:30.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1137845881' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:30.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:30.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:30.742 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:30.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:30.844 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:30.870 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:30.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:30.878 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-17T16:43:30.972 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:30.998 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:30.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:31.100 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:31.126 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:31.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:31.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:31.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:31.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:31.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:31.379 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:31.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:31.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:31 smithi017 ceph-mon[27571]: 9.b scrub starts 2024-09-17T16:43:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:31 smithi017 ceph-mon[27571]: 9.b scrub ok 2024-09-17T16:43:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:31 smithi017 ceph-mon[27571]: pgmap v2168: 37 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7144/9 objects degraded (79377.778%); 151/9 objects misplaced (1677.778%) 2024-09-17T16:43:31.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:31 smithi017 ceph-mon[27571]: osdmap e989: 12 total, 12 up, 12 in 2024-09-17T16:43:31.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:31 smithi017 ceph-mon[27571]: 9.7 scrub starts 2024-09-17T16:43:31.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:31 smithi017 ceph-mon[27571]: 9.7 scrub ok 2024-09-17T16:43:31.505 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:31.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:31.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:31 smithi071 ceph-mon[30002]: 9.b scrub starts 2024-09-17T16:43:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:31 smithi071 ceph-mon[30002]: 9.b scrub ok 2024-09-17T16:43:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:31 smithi071 ceph-mon[30002]: pgmap v2168: 37 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7144/9 objects degraded (79377.778%); 151/9 objects misplaced (1677.778%) 2024-09-17T16:43:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:31 smithi071 ceph-mon[30002]: osdmap e989: 12 total, 12 up, 12 in 2024-09-17T16:43:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:31 smithi071 ceph-mon[30002]: 9.7 scrub starts 2024-09-17T16:43:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:31 smithi071 ceph-mon[30002]: 9.7 scrub ok 2024-09-17T16:43:31.607 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:31.630 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:31.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:31 smithi173 ceph-mon[30530]: 9.b scrub starts 2024-09-17T16:43:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:31 smithi173 ceph-mon[30530]: 9.b scrub ok 2024-09-17T16:43:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:31 smithi173 ceph-mon[30530]: pgmap v2168: 37 pgs: 1 peering, 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7144/9 objects degraded (79377.778%); 151/9 objects misplaced (1677.778%) 2024-09-17T16:43:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:31 smithi173 ceph-mon[30530]: osdmap e989: 12 total, 12 up, 12 in 2024-09-17T16:43:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:31 smithi173 ceph-mon[30530]: 9.7 scrub starts 2024-09-17T16:43:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:31 smithi173 ceph-mon[30530]: 9.7 scrub ok 2024-09-17T16:43:31.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:31.753 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:31.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:31.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:31.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:31.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:31.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:32.009 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:32.010 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:32.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:32.133 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:32.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:32.235 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:32.259 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:32.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:32.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:32 smithi017 ceph-mon[27571]: 1.4 scrub starts 2024-09-17T16:43:32.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:32 smithi017 ceph-mon[27571]: 1.4 scrub ok 2024-09-17T16:43:32.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:32 smithi017 ceph-mon[27571]: 1.6 scrub starts 2024-09-17T16:43:32.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:32 smithi017 ceph-mon[27571]: 1.6 scrub ok 2024-09-17T16:43:32.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:32.383 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:32.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:32.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:32.507 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:32.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:32.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:32 smithi071 ceph-mon[30002]: 1.4 scrub starts 2024-09-17T16:43:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:32 smithi071 ceph-mon[30002]: 1.4 scrub ok 2024-09-17T16:43:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:32 smithi071 ceph-mon[30002]: 1.6 scrub starts 2024-09-17T16:43:32.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:32 smithi071 ceph-mon[30002]: 1.6 scrub ok 2024-09-17T16:43:32.609 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:32.632 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:32.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:32.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:32 smithi173 ceph-mon[30530]: 1.4 scrub starts 2024-09-17T16:43:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:32 smithi173 ceph-mon[30530]: 1.4 scrub ok 2024-09-17T16:43:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:32 smithi173 ceph-mon[30530]: 1.6 scrub starts 2024-09-17T16:43:32.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:32 smithi173 ceph-mon[30530]: 1.6 scrub ok 2024-09-17T16:43:32.735 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:32.758 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:32.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:32.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:32.883 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:32.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:32.985 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:32.992 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:33.006 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:33.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:33.108 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:33.130 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:33.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:33.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:33.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:33.240 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:43:33.242 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T16:43:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:33.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:33.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:33.358 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:33.384 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:33.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:33.486 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:33.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:33 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:43:33.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:33 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:43:33.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:33 smithi017 ceph-mon[27571]: pgmap v2170: 37 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3873/9 objects degraded (43033.333%); 151/9 objects misplaced (1677.778%) 2024-09-17T16:43:33.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:33 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 3873/9 objects degraded (43033.333%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:33.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:33 smithi017 ceph-mon[27571]: osdmap e990: 12 total, 12 up, 12 in 2024-09-17T16:43:33.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1122216836' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:43:33.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:33 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:43:33.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1128001044' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:43:33.519 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:33.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:33 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:43:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:33 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:43:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:33 smithi071 ceph-mon[30002]: pgmap v2170: 37 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3873/9 objects degraded (43033.333%); 151/9 objects misplaced (1677.778%) 2024-09-17T16:43:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:33 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 3873/9 objects degraded (43033.333%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:33 smithi071 ceph-mon[30002]: osdmap e990: 12 total, 12 up, 12 in 2024-09-17T16:43:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1122216836' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:43:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:33 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:43:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1128001044' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:43:33.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:33.643 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:33.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:33 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:43:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:33 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:43:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:33 smithi173 ceph-mon[30530]: pgmap v2170: 37 pgs: 1 peering, 4 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 30 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3873/9 objects degraded (43033.333%); 151/9 objects misplaced (1677.778%) 2024-09-17T16:43:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:33 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 3873/9 objects degraded (43033.333%), 5 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:33 smithi173 ceph-mon[30530]: osdmap e990: 12 total, 12 up, 12 in 2024-09-17T16:43:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1122216836' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:43:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:33 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:43:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1128001044' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:43:33.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:43:33.745 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:33.768 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:33.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:33.871 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:33.895 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:33.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:33.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:34.022 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:34.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:34.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:34.146 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:34.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:34.249 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:34.273 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:34.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:34.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:34.398 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:34.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:34.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:34.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:34.523 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:34.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:34 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:34 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:43:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1128001044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:43:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:34 smithi071 ceph-mon[30002]: osdmap e991: 12 total, 12 up, 12 in 2024-09-17T16:43:34.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:34 smithi071 ceph-mon[30002]: osdmap e992: 12 total, 12 up, 12 in 2024-09-17T16:43:34.625 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:34 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:34 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:43:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1128001044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:43:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:34 smithi173 ceph-mon[30530]: osdmap e991: 12 total, 12 up, 12 in 2024-09-17T16:43:34.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:34 smithi173 ceph-mon[30530]: osdmap e992: 12 total, 12 up, 12 in 2024-09-17T16:43:34.647 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:34.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:34 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:34 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:43:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1128001044' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:43:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:34 smithi017 ceph-mon[27571]: osdmap e991: 12 total, 12 up, 12 in 2024-09-17T16:43:34.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:34 smithi017 ceph-mon[27571]: osdmap e992: 12 total, 12 up, 12 in 2024-09-17T16:43:34.749 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:34.771 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:34.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:34.873 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:34.896 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:34.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:34.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:34.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:35.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:35.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:35.072 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:35.072 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:35.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:35.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:35.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:35.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:35.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:35.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:35.370 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:35.395 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:35.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:35.441 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-17T16:43:35.497 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:35.518 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:35.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:35.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:35 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:43:35.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:35 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:43:35.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:35 smithi017 ceph-mon[27571]: pgmap v2173: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 32 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3870/9 objects degraded (43000.000%); 151/9 objects misplaced (1677.778%) 2024-09-17T16:43:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:35 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:43:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:35 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:43:35.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:35 smithi071 ceph-mon[30002]: pgmap v2173: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 32 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3870/9 objects degraded (43000.000%); 151/9 objects misplaced (1677.778%) 2024-09-17T16:43:35.620 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:35.643 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:35.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:35 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:43:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:35 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:43:35.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:35 smithi173 ceph-mon[30530]: pgmap v2173: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 32 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3870/9 objects degraded (43000.000%); 151/9 objects misplaced (1677.778%) 2024-09-17T16:43:35.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:35.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:35.768 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:35.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:35.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:35.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:35.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-17T16:43:35.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:36.015 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:36.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:36.035 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:36.077 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:36.117 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:36.140 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:36.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:36.241 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:36.263 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:36.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:36.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:36.300 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:43:36.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:36.389 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:36.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:36.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:36.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:36.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:36.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:36.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:36.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:36.741 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:36.762 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:36.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:36.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:36.832 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:43:36.832 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:43:36.855 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:43:36.855 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:43:36.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:36.876 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:43:36.876 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:43:36.886 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:36.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:36.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:36.897 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:43:36.897 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:43:36.919 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:43:36.919 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:43:36.940 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:43:36.940 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:43:36.962 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:43:36.963 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:43:36.985 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:43:36.985 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:43:36.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:37.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:37.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:37.067 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:37.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:37.137 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:37.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:37.209 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:43:37.209 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:43:37.231 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:43:37.231 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:43:37.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:37.252 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:43:37.253 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:43:37.260 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:37.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:37.274 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:43:37.274 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:43:37.362 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:37.384 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:37.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:37.486 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:37.509 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:37.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:37.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:37.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:37 smithi071 ceph-mon[30002]: pgmap v2175: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 32 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3870/9 objects degraded (43000.000%); 151/9 objects misplaced (1677.778%); 0 B/s, 355 objects/s recovering 2024-09-17T16:43:37.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3949273347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:37.611 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:37.635 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:37.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:37 smithi173 ceph-mon[30530]: pgmap v2175: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 32 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3870/9 objects degraded (43000.000%); 151/9 objects misplaced (1677.778%); 0 B/s, 355 objects/s recovering 2024-09-17T16:43:37.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3949273347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:37.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:37 smithi017 ceph-mon[27571]: pgmap v2175: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 32 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3870/9 objects degraded (43000.000%); 151/9 objects misplaced (1677.778%); 0 B/s, 355 objects/s recovering 2024-09-17T16:43:37.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3949273347' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:37.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:37.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:37.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:37.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:37.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:37.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:37.986 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:38.009 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:38.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:38.111 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:38.138 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:38.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:38.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:38.257 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.8 to 1 (10000) 2024-09-17T16:43:38.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:38.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:38.364 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:38.390 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:38.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:38.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3047111613' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:43:38.429 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-17T16:43:38.429 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-17T16:43:38.430 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-17T16:43:38.430 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-17T16:43:38.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:38.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:38.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:38.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3047111613' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:43:38.615 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:38.638 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:38.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:38.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3047111613' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:43:38.740 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:38.770 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:38.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:38.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:38.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:38.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:38.996 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:39.019 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:39.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:39.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:39.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:39.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:39.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:39.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-17T16:43:39.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:39.272 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:39.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:39.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:39.398 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:39.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:39 smithi017 ceph-mon[27571]: pgmap v2176: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 370 objects/s recovering 2024-09-17T16:43:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:39 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 556/9 objects degraded (6177.778%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3047111613' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:43:39.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:39 smithi017 ceph-mon[27571]: osdmap e993: 12 total, 12 up, 12 in 2024-09-17T16:43:39.499 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:39.525 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:39.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:39.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:39 smithi173 ceph-mon[30530]: pgmap v2176: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 370 objects/s recovering 2024-09-17T16:43:39.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:39 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 556/9 objects degraded (6177.778%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:39.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3047111613' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:43:39.545 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:39 smithi173 ceph-mon[30530]: osdmap e993: 12 total, 12 up, 12 in 2024-09-17T16:43:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:39 smithi071 ceph-mon[30002]: pgmap v2176: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 370 objects/s recovering 2024-09-17T16:43:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:39 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 556/9 objects degraded (6177.778%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3047111613' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:43:39.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:39 smithi071 ceph-mon[30002]: osdmap e993: 12 total, 12 up, 12 in 2024-09-17T16:43:39.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:39.653 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:39.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:39.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:39.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:39.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:39.880 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:39.903 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:39.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:40.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:40.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:40.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:40.131 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:40.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:43:40.154 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:40.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:40.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:40.278 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:40.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:40.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-17T16:43:40.379 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:40.402 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:40.432 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:40.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:40.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:40.436 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:40.534 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:40.557 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:40.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:40.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:40.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:40.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:40.783 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:40.806 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:40.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:40.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:40.930 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:40.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:40.987 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:41.032 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:41.056 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:41.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:41.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:40 smithi173 ceph-mon[30530]: osdmap e994: 12 total, 12 up, 12 in 2024-09-17T16:43:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:40 smithi173 ceph-mon[30530]: pgmap v2179: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 418 objects/s recovering 2024-09-17T16:43:41.158 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:41.169 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:40 smithi017 ceph-mon[27571]: osdmap e994: 12 total, 12 up, 12 in 2024-09-17T16:43:41.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:40 smithi017 ceph-mon[27571]: pgmap v2179: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 418 objects/s recovering 2024-09-17T16:43:41.179 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:41.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:41.281 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:41.305 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:41.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:41.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:40 smithi071 ceph-mon[30002]: osdmap e994: 12 total, 12 up, 12 in 2024-09-17T16:43:41.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:40 smithi071 ceph-mon[30002]: pgmap v2179: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 418 objects/s recovering 2024-09-17T16:43:41.407 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:41.430 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:41.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:41.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:41.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:41.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:41.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:41.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:41.580 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:43:41.581 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:43:41.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:43:41.591 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:43:41.613 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:43:41.613 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:43:41.634 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:43:41.634 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:43:41.657 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:43:41.657 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:43:41.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:41.678 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:43:41.679 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:43:41.681 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:41.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:41.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:41.700 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:43:41.700 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:43:41.721 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:43:41.722 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:43:41.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:43:41.744 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:43:41.766 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:43:41.766 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:43:41.783 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:41.789 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:43:41.789 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:43:41.806 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:41.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:41.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:43:41.817 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:43:41.862 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:43:41.907 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:41.931 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:41.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:41.992 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/11021963' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:43:41.993 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3192239209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:42.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:42.055 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:42.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:42.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/11021963' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:43:42.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3192239209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:42.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:42.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:42.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:42.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:42.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:42.304 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:42.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:42.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:42.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/11021963' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:43:42.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3192239209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:42.353 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:42.353 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:42.406 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:42.428 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:42.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:42.530 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:42.553 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:42.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:42.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:42.655 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:42.676 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:42.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:42.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:42.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:42.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:42.902 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:42.924 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:42.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:43.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:43.050 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:43.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:43.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:42 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/11021963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:43:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:42 smithi173 ceph-mon[30530]: osdmap e995: 12 total, 12 up, 12 in 2024-09-17T16:43:43.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:42 smithi173 ceph-mon[30530]: pgmap v2181: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 406 objects/s recovering 2024-09-17T16:43:43.151 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:43.173 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:43.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:42 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/11021963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:43:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:42 smithi017 ceph-mon[27571]: osdmap e995: 12 total, 12 up, 12 in 2024-09-17T16:43:43.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:42 smithi017 ceph-mon[27571]: pgmap v2181: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 406 objects/s recovering 2024-09-17T16:43:43.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:43.296 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:43.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:43.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:42 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:43.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/11021963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:43:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:42 smithi071 ceph-mon[30002]: osdmap e995: 12 total, 12 up, 12 in 2024-09-17T16:43:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:42 smithi071 ceph-mon[30002]: pgmap v2181: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 406 objects/s recovering 2024-09-17T16:43:43.359 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-17T16:43:43.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:43.426 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:43.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:43.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:43.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:43.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:43.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:43.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:43.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:43.784 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:43.807 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:43.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:43.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:43.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:43.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:43.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:44.034 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:44.058 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:44.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:44.160 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:44.181 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:44.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:44.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:44.306 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:44.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:44.389 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:44.407 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:44.431 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:44.434 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:44.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:44.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:44.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:44.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:44.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:44.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:44.686 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:44.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:44.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:44.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:44.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:44.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:44.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:44.936 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:44.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:45.038 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:45.060 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:45.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:45.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:45.183 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:45.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:45.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:45.240 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:45.240 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:45.285 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:45.310 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:45.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:45.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:45.435 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:45.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:45.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:45.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:45.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:45.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:45 smithi071 ceph-mon[30002]: pgmap v2182: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 239 objects/s recovering 2024-09-17T16:43:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:45 smithi071 ceph-mon[30002]: osdmap e996: 12 total, 12 up, 12 in 2024-09-17T16:43:45.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:45 smithi173 ceph-mon[30530]: pgmap v2182: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 239 objects/s recovering 2024-09-17T16:43:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:45 smithi173 ceph-mon[30530]: osdmap e996: 12 total, 12 up, 12 in 2024-09-17T16:43:45.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:45.681 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:45.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:45 smithi017 ceph-mon[27571]: pgmap v2182: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 239 objects/s recovering 2024-09-17T16:43:45.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:45 smithi017 ceph-mon[27571]: osdmap e996: 12 total, 12 up, 12 in 2024-09-17T16:43:45.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:45.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:45.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:45.908 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:45.929 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:45.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:45.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:46.031 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:46.052 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:46.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:46.154 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:46.176 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:46.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:46.196 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:43:46.259 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.9 to 1 (10000) 2024-09-17T16:43:46.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:46.301 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:46.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:46.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:46.419 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:43:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:46.428 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:46.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:46.501 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4157081902' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:43:46.521 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-17T16:43:46.529 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:46.544 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4157081902' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:43:46.567 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:46.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:46.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4157081902' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:43:46.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:46.940 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:46.940 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:43:46.941 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:46.962 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:46.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:47.064 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:47.086 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:47.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:47.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:43:47.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:47.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:47.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:43:47.300 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-17T16:43:47.316 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:47.340 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:47.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:47.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:47.464 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:47.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:47.495 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:47.495 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:47.495 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:47 smithi071 ceph-mon[30002]: pgmap v2184: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 407 objects/s recovering 2024-09-17T16:43:47.495 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4157081902' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:43:47.495 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:47 smithi071 ceph-mon[30002]: osdmap e997: 12 total, 12 up, 12 in 2024-09-17T16:43:47.517 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:47 smithi017 ceph-mon[27571]: pgmap v2184: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 407 objects/s recovering 2024-09-17T16:43:47.517 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4157081902' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:43:47.517 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:47 smithi017 ceph-mon[27571]: osdmap e997: 12 total, 12 up, 12 in 2024-09-17T16:43:47.566 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:47.587 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:47.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:47 smithi173 ceph-mon[30530]: pgmap v2184: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 407 objects/s recovering 2024-09-17T16:43:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4157081902' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:43:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:47 smithi173 ceph-mon[30530]: osdmap e997: 12 total, 12 up, 12 in 2024-09-17T16:43:47.689 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:47.712 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:47.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:47.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:47.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:47.836 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:47.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:47.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:47.938 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:47.960 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:47.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:47.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:48.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:48.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:48.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:48.189 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:48.212 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:48.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:48.263 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:43:48.295 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-17T16:43:48.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:48.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:48.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:48.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:48.463 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:48.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:48.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1151215673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:48.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3224564884' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:43:48.547 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:43:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1151215673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:48.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3224564884' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:43:48.565 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:48.587 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:48.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:48.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1151215673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3224564884' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:43:48.688 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:48.712 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:48.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:48.813 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:48.836 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:48.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:48.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:48.951 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:43:48.951 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:43:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:48.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:48.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:48.974 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:43:48.975 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:43:48.997 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:43:48.997 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:43:49.026 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:43:49.027 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:43:49.051 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:43:49.052 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:43:49.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:49.069 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:43:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:43:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:43:49.070 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:43:49.070 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:43:49.073 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:43:49.073 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:43:49.086 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:49.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:49.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:49.095 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:43:49.095 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:43:49.119 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:43:49.119 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:43:49.119 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:49.142 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:43:49.142 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:43:49.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:43:49.165 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:43:49.187 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:43:49.187 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:43:49.188 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:49.209 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:43:49.209 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:43:49.209 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:49.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:49.311 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:49.335 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:49.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:49.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:49 smithi017 ceph-mon[27571]: pgmap v2186: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 257 objects/s recovering 2024-09-17T16:43:49.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:49 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:49.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3224564884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:43:49.377 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:49 smithi017 ceph-mon[27571]: osdmap e998: 12 total, 12 up, 12 in 2024-09-17T16:43:49.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:49.457 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:49.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:49.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:49 smithi071 ceph-mon[30002]: pgmap v2186: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 257 objects/s recovering 2024-09-17T16:43:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:49 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3224564884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:43:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:49 smithi071 ceph-mon[30002]: osdmap e998: 12 total, 12 up, 12 in 2024-09-17T16:43:49.579 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:49.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:49 smithi173 ceph-mon[30530]: pgmap v2186: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 257 objects/s recovering 2024-09-17T16:43:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:49 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:43:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3224564884' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:43:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:49 smithi173 ceph-mon[30530]: osdmap e998: 12 total, 12 up, 12 in 2024-09-17T16:43:49.681 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:49.703 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:49.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:49.717 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-17T16:43:49.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:49.827 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:49.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:49.928 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:49.951 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:49.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:50.015 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:43:50.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:50.077 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:50.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:50.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:50.200 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:50.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:50.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:50.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:50.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:50.394 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:50 smithi071 ceph-mon[30002]: 9.e scrub starts 2024-09-17T16:43:50.394 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:50 smithi071 ceph-mon[30002]: 9.e scrub ok 2024-09-17T16:43:50.428 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:50.452 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:50.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:50 smithi017 ceph-mon[27571]: 9.e scrub starts 2024-09-17T16:43:50.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:50 smithi017 ceph-mon[27571]: 9.e scrub ok 2024-09-17T16:43:50.495 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:43:50.495 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:43:50.495 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:43:50.495 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:43:50.495 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.495 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(8.19 741/739 [PushOp(8:9b4c75cc:::benchmark_data_smithi173_130566_object6584:head, version: 732'423, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:9b4c75cc:::benchmark_data_smithi173_130566_object6584:head@732'423, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:43:50.495 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:28.096915+0000", 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 562.39534421899998, 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.64022687700000003, 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.096915+0000", 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.496 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.096915+0000", 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.096915+0000", 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6400000000000002e-07 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.097032+0000", 2024-09-17T16:43:50.497 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000116988 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736654+0000", 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.63962142799999999 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736671+0000", 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7062999999999999e-05 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.498 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736829+0000", 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000158808 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736842+0000", 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3021e-05 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.737141+0000", 2024-09-17T16:43:50.499 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00029900499999999997 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(8.19 741/739 [PushOp(8:98915622:::benchmark_data_smithi173_130566_object6588:head, version: 732'424, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:98915622:::benchmark_data_smithi173_130566_object6588:head@732'424, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:28.098526+0000", 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 562.39373288299998, 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.63899293999999995, 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.500 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.098526+0000", 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.098526+0000", 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.501 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.098526+0000", 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5300000000000004e-07 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.098638+0000", 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000111109 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.502 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736712+0000", 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.63807422999999996 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.736726+0000", 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4058e-05 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.737217+0000", 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00049099300000000001 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.503 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.737229+0000", 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2534999999999999e-05 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.737519+0000", 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00028946199999999999 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.504 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(8.14 742/739 [PushOp(8:29340722:::benchmark_data_smithi173_130566_object8035:head, version: 733'489, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:29340722:::benchmark_data_smithi173_130566_object8035:head@733'489, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:28.785447+0000", 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 561.70681226199997, 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.873767925, 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.785447+0000", 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.785447+0000", 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.785447+0000", 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:28.785506+0000", 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8687999999999998e-05 2024-09-17T16:43:50.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.507 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.507 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.658772+0000", 2024-09-17T16:43:50.507 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87326642799999998 2024-09-17T16:43:50.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.507 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.507 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.658800+0000", 2024-09-17T16:43:50.508 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7483000000000001e-05 2024-09-17T16:43:50.508 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.508 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.508 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.508 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.658964+0000", 2024-09-17T16:43:50.508 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016395400000000001 2024-09-17T16:43:50.508 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.508 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.508 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.658998+0000", 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4059e-05 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.659214+0000", 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00021660399999999999 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.509 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(8.10 743/742 [PushReplyOp(8:0bc28d4c:::benchmark_data_smithi173_130566_object2319:head)])", 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:34:29.824885+0000", 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 560.66737367999997, 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86261194500000005, 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.824885+0000", 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.824885+0000", 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.824886+0000", 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.13e-07 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:29.824889+0000", 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6689999999999999e-06 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.684407+0000", 2024-09-17T16:43:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.85951759500000002 2024-09-17T16:43:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.684415+0000", 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1089999999999998e-06 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.686916+0000", 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002500979 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:34:30.687497+0000", 2024-09-17T16:43:50.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058107999999999999 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113329 8.19 8.cd7b9579 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.083476+0000", 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 419.408783035, 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93999810299999997, 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:43:50.514 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113329 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083476+0000", 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083476+0000", 2024-09-17T16:43:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083476+0000", 2024-09-17T16:43:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.75e-07 2024-09-17T16:43:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083483+0000", 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3199999999999996e-06 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083493+0000", 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9580000000000002e-06 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083498+0000", 2024-09-17T16:43:50.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9629999999999997e-06 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083558+0000", 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9917999999999998e-05 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.083570+0000", 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1942e-05 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023439+0000", 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93986991600000003 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023474+0000", 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4411e-05 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113344 8.19 8.40c48eb9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.120379+0000", 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 419.37187970000002, 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.90320710199999998, 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113344 2024-09-17T16:43:50.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120379+0000", 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120379+0000", 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120380+0000", 2024-09-17T16:43:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.54e-07 2024-09-17T16:43:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120387+0000", 2024-09-17T16:43:50.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0949999999999998e-06 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120396+0000", 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1980000000000007e-06 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120402+0000", 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2719999999999997e-06 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120449+0000", 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7304e-05 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.120456+0000", 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0090000000000001e-06 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023577+0000", 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.903121226 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023586+0000", 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1439999999999992e-06 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113366 8.19 8.da4bff99 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.226795+0000", 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 419.26546337100001, 2024-09-17T16:43:50.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79682678600000001, 2024-09-17T16:43:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:43:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:43:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:43:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113366 2024-09-17T16:43:50.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226795+0000", 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226795+0000", 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226799+0000", 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.163e-06 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226814+0000", 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5837000000000001e-05 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226854+0000", 2024-09-17T16:43:50.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9303e-05 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.226877+0000", 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3516e-05 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.227001+0000", 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012416099999999999 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.227011+0000", 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6439999999999995e-06 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023615+0000", 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79660426600000001 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023622+0000", 2024-09-17T16:43:50.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8959999999999997e-06 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113400 8.19 8.6f899219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.512591+0000", 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 418.979667662, 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51106722999999998, 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:43:50.531 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113400 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512591+0000", 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512591+0000", 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512593+0000", 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0669999999999999e-06 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512604+0000", 2024-09-17T16:43:50.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0526000000000001e-05 2024-09-17T16:43:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512632+0000", 2024-09-17T16:43:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8595999999999999e-05 2024-09-17T16:43:50.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512653+0000", 2024-09-17T16:43:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0299999999999999e-05 2024-09-17T16:43:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512799+0000", 2024-09-17T16:43:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014620099999999999 2024-09-17T16:43:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512817+0000", 2024-09-17T16:43:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7683e-05 2024-09-17T16:43:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023651+0000", 2024-09-17T16:43:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51083464700000003 2024-09-17T16:43:50.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023658+0000", 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2099999999999996e-06 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:113401 8.19 8.6a0bcc79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e796)", 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:51.512908+0000", 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 418.97935073500003, 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51079329600000001, 2024-09-17T16:43:50.538 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 113401 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512908+0000", 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512908+0000", 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512909+0000", 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8400000000000004e-07 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512912+0000", 2024-09-17T16:43:50.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0309999999999999e-06 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512919+0000", 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0360000000000001e-06 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512925+0000", 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0680000000000002e-06 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512969+0000", 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4558999999999998e-05 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:51.512977+0000", 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1849999999999998e-06 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023693+0000", 2024-09-17T16:43:50.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51071658499999995 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:52.023701+0000", 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2479999999999996e-06 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:115081 8.0 8:02125afa:::benchmark_data_smithi173_130566_object20106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:54.801418+0000", 2024-09-17T16:43:50.543 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 415.69084050399999, 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.67051243999999999, 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 115081 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801418+0000", 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801418+0000", 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801420+0000", 2024-09-17T16:43:50.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3710000000000001e-06 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801424+0000", 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.724e-06 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801437+0000", 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2724999999999999e-05 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801445+0000", 2024-09-17T16:43:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4789999999999999e-06 2024-09-17T16:43:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801495+0000", 2024-09-17T16:43:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0609000000000003e-05 2024-09-17T16:43:50.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801598+0000", 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000103088 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 3,9", 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.801699+0000", 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100719 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.804026+0000", 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023265709999999999 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:54.805302+0000", 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001276358 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.471864+0000", 2024-09-17T16:43:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.66656182399999997 2024-09-17T16:43:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.471901+0000", 2024-09-17T16:43:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7225999999999999e-05 2024-09-17T16:43:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:55.471931+0000", 2024-09-17T16:43:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9745999999999999e-05 2024-09-17T16:43:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8109 9.c 9:37996cec:::benchmark_data_smithi173_150363_object8108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:43:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.764335+0000", 2024-09-17T16:43:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 280.72792358700002, 2024-09-17T16:43:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56464964100000004, 2024-09-17T16:43:50.552 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8109 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764335+0000", 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764335+0000", 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764340+0000", 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2000000000000002e-06 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764448+0000", 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010775700000000001 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764473+0000", 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5218000000000001e-05 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764544+0000", 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0397999999999999e-05 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764961+0000", 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000417685 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764973+0000", 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1939999999999999e-05 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.326276+0000", 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56130267199999995 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.326404+0000", 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012778799999999999 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.326517+0000", 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000113355 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.328647+0000", 2024-09-17T16:43:50.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021297579999999998 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.328667+0000", 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9918000000000001e-05 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.328707+0000", 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9960999999999999e-05 2024-09-17T16:43:50.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.328751+0000", 2024-09-17T16:43:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4001000000000001e-05 2024-09-17T16:43:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:50.560 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:50 smithi173 ceph-mon[30530]: 9.e scrub starts 2024-09-17T16:43:50.560 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:50 smithi173 ceph-mon[30530]: 9.e scrub ok 2024-09-17T16:43:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.328985+0000", 2024-09-17T16:43:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00023399 2024-09-17T16:43:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.561 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8119 9.c 9:3297cfd2:::benchmark_data_smithi173_150363_object8118:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.764602+0000", 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 280.72765650100001, 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56583742800000003, 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.562 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8119 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764602+0000", 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764602+0000", 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764603+0000", 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3500000000000005e-07 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764657+0000", 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4261000000000001e-05 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764671+0000", 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3378e-05 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764679+0000", 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8720000000000002e-06 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.765027+0000", 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00034875999999999999 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.765038+0000", 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0784999999999999e-05 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.327436+0000", 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56239745399999996 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.327525+0000", 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9095e-05 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.327618+0000", 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3419999999999997e-05 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.329074+0000", 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014561559999999999 2024-09-17T16:43:50.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.329131+0000", 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6277999999999999e-05 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.330065+0000", 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093476400000000004 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.330099+0000", 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3371000000000003e-05 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.330440+0000", 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00034099899999999998 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8125 9.c 9:327a8821:::benchmark_data_smithi173_150363_object8124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:43:50.569 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.764694+0000", 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 280.72756460800002, 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56626069999999995, 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8125 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764694+0000", 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764694+0000", 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764695+0000", 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.75e-07 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764732+0000", 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7082999999999997e-05 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764739+0000", 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0820000000000001e-06 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764756+0000", 2024-09-17T16:43:50.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7424000000000001e-05 2024-09-17T16:43:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.765075+0000", 2024-09-17T16:43:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031863900000000001 2024-09-17T16:43:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.765080+0000", 2024-09-17T16:43:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0599999999999998e-06 2024-09-17T16:43:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.328101+0000", 2024-09-17T16:43:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56302109700000003 2024-09-17T16:43:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.328178+0000", 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6557000000000006e-05 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.328270+0000", 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2468999999999998e-05 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.329964+0000", 2024-09-17T16:43:50.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016932469999999999 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.330018+0000", 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4234999999999998e-05 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.330579+0000", 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00056089900000000001 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.330640+0000", 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1185000000000005e-05 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.330955+0000", 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031504800000000001 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.577 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8140 9.c 9:3a535b45:::benchmark_data_smithi173_150363_object8139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:43:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.764769+0000", 2024-09-17T16:43:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 280.72749014700003, 2024-09-17T16:43:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56676879599999996, 2024-09-17T16:43:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:43:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:50.578 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8140 2024-09-17T16:43:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764769+0000", 2024-09-17T16:43:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764769+0000", 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764769+0000", 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3099999999999997e-07 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764803+0000", 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3535999999999999e-05 2024-09-17T16:43:50.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764813+0000", 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0179e-05 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.764820+0000", 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5899999999999996e-06 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.765127+0000", 2024-09-17T16:43:50.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030690600000000002 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.765136+0000", 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7329999999999998e-06 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.329182+0000", 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56404590700000001 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.329327+0000", 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000144695 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.329489+0000", 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016254900000000001 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.330509+0000", 2024-09-17T16:43:50.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010194550000000001 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.331032+0000", 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00052363300000000002 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.331093+0000", 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0809e-05 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.331208+0000", 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000114338 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.331537+0000", 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000329835 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.585 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8161 9.c 9:30b7f7c4:::benchmark_data_smithi173_150363_object8160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.798487+0000", 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 280.69377168900002, 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53934285599999998, 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8161 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.798487+0000", 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.798487+0000", 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.798490+0000", 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5179999999999999e-06 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.798577+0000", 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7280999999999993e-05 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.798599+0000", 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1815999999999999e-05 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.798627+0000", 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8581e-05 2024-09-17T16:43:50.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.798694+0000", 2024-09-17T16:43:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7056000000000003e-05 2024-09-17T16:43:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.798705+0000", 2024-09-17T16:43:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1104000000000001e-05 2024-09-17T16:43:50.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.331594+0000", 2024-09-17T16:43:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53288858400000005 2024-09-17T16:43:50.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.331784+0000", 2024-09-17T16:43:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019019300000000001 2024-09-17T16:43:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:43:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.331940+0000", 2024-09-17T16:43:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015581699999999999 2024-09-17T16:43:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333865+0000", 2024-09-17T16:43:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001924697 2024-09-17T16:43:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333881+0000", 2024-09-17T16:43:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6633000000000002e-05 2024-09-17T16:43:50.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.337426+0000", 2024-09-17T16:43:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0035448820000000001 2024-09-17T16:43:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.337496+0000", 2024-09-17T16:43:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0168999999999999e-05 2024-09-17T16:43:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.337830+0000", 2024-09-17T16:43:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00033352499999999999 2024-09-17T16:43:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8164 9.c 9:33a94f2a:::benchmark_data_smithi173_150363_object8163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:43:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.799845+0000", 2024-09-17T16:43:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 280.69241418299998, 2024-09-17T16:43:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53843401300000004, 2024-09-17T16:43:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.597 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8164 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.799845+0000", 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.799845+0000", 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.799846+0000", 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2550000000000001e-06 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.799904+0000", 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8424999999999997e-05 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.799918+0000", 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3454e-05 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.799926+0000", 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7729999999999999e-06 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.799966+0000", 2024-09-17T16:43:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0308000000000001e-05 2024-09-17T16:43:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.799971+0000", 2024-09-17T16:43:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5350000000000004e-06 2024-09-17T16:43:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.332469+0000", 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53249764099999997 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.332597+0000", 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000127501 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.332729+0000", 2024-09-17T16:43:50.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013275900000000001 2024-09-17T16:43:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333924+0000", 2024-09-17T16:43:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001194528 2024-09-17T16:43:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.334510+0000", 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058583500000000004 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.337951+0000", 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0034414799999999998 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.338009+0000", 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7559999999999998e-05 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.338279+0000", 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026995900000000001 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8183 9.c 9:30ced98c:::benchmark_data_smithi173_150363_object8182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.807394+0000", 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 280.684864957, 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53134801399999998, 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.605 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8183 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.807394+0000", 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.807394+0000", 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.807396+0000", 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.137e-06 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.807467+0000", 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0959999999999998e-05 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.807489+0000", 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2075000000000001e-05 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.807501+0000", 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.148e-05 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.807571+0000", 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0592000000000004e-05 2024-09-17T16:43:50.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.807582+0000", 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0441999999999999e-05 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333205+0000", 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52562342799999995 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333307+0000", 2024-09-17T16:43:50.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010220799999999999 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333410+0000", 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000103113 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.335225+0000", 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018151879999999999 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.335345+0000", 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000119061 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.338384+0000", 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0030398259999999998 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.338447+0000", 2024-09-17T16:43:50.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2657999999999995e-05 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.338742+0000", 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00029484599999999998 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8187 9.c 9:38d30baf:::benchmark_data_smithi173_150363_object8186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.809329+0000", 2024-09-17T16:43:50.612 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 280.68293018600002, 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.529937824, 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8187 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.809329+0000", 2024-09-17T16:43:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.809329+0000", 2024-09-17T16:43:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.809330+0000", 2024-09-17T16:43:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4649999999999999e-06 2024-09-17T16:43:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.809472+0000", 2024-09-17T16:43:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000142383 2024-09-17T16:43:50.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.809490+0000", 2024-09-17T16:43:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7464000000000001e-05 2024-09-17T16:43:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.809499+0000", 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0240000000000003e-06 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.809562+0000", 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3237000000000005e-05 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.809570+0000", 2024-09-17T16:43:50.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.038e-06 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333981+0000", 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52441062299999996 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.334076+0000", 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5613999999999996e-05 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:43:50.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.334173+0000", 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6429999999999997e-05 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.335326+0000", 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00115274 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.336923+0000", 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001597497 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.338906+0000", 2024-09-17T16:43:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019828990000000002 2024-09-17T16:43:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.338959+0000", 2024-09-17T16:43:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2614000000000001e-05 2024-09-17T16:43:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.621 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.339266+0000", 2024-09-17T16:43:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030779599999999999 2024-09-17T16:43:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8194 9.c 9:3514db32:::benchmark_data_smithi173_150363_object8193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:43:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.813800+0000", 2024-09-17T16:43:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 280.678459299, 2024-09-17T16:43:50.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52591709900000005, 2024-09-17T16:43:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:43:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8194 2024-09-17T16:43:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813800+0000", 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813800+0000", 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813801+0000", 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.762e-06 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813847+0000", 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5804999999999999e-05 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813864+0000", 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6952e-05 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813871+0000", 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.097e-06 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813934+0000", 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3322000000000001e-05 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for peered", 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813946+0000", 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1080000000000001e-05 2024-09-17T16:43:50.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.334544+0000", 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.520598326 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.334639+0000", 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4840000000000007e-05 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.334742+0000", 2024-09-17T16:43:50.627 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010334 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.336933+0000", 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002190577 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.337892+0000", 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000959227 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.628 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.339357+0000", 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014648179999999999 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.339421+0000", 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3848000000000003e-05 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.339717+0000", 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00029610500000000001 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.629 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:193633 9.c 9:3a9cd4bd:::benchmark_data_smithi173_150363_object86728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:42:40.653954+0000", 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 69.838304651000001, 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41202159199999999, 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:43:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 193633 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.653954+0000", 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.653954+0000", 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:43:50.632 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.653956+0000", 2024-09-17T16:43:50.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4610000000000001e-06 2024-09-17T16:43:50.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.632 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:43:50.632 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.653962+0000", 2024-09-17T16:43:50.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3729999999999998e-06 2024-09-17T16:43:50.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.632 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:43:50.633 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.653977+0000", 2024-09-17T16:43:50.633 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4800000000000001e-05 2024-09-17T16:43:50.633 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.633 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.633 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:43:50.633 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.653984+0000", 2024-09-17T16:43:50.633 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2490000000000004e-06 2024-09-17T16:43:50.633 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.634 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:43:50.634 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654043+0000", 2024-09-17T16:43:50.634 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9422999999999997e-05 2024-09-17T16:43:50.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.634 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:43:50.634 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654169+0000", 2024-09-17T16:43:50.634 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000125191 2024-09-17T16:43:50.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 0,9", 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.654289+0000", 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012061 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.662051+0000", 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0077621959999999999 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:43:50.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:40.662115+0000", 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3464000000000001e-05 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.065816+0000", 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40370149500000002 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.065926+0000", 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000109478 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:43:50.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:43:50.637 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:42:41.065976+0000", 2024-09-17T16:43:50.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9852000000000003e-05 2024-09-17T16:43:50.637 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.637 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.637 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.637 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:43:50.637 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:43:50.637 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:43:50.658 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:50.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:50.760 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:50.784 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:50.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:50.885 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:50.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:50.907 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:50.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:51.008 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:51.030 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:51.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:51.132 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:51.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:43:51.154 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:51.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:51.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:51.257 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:51.274 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.10 to 1 (10000) 2024-09-17T16:43:51.279 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:51.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:51.382 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:51 smithi173 ceph-mon[30530]: pgmap v2188: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 374 objects/s recovering 2024-09-17T16:43:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:51 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3790848244' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:43:51.403 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:51.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:51.452 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:51 smithi071 ceph-mon[30002]: pgmap v2188: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 374 objects/s recovering 2024-09-17T16:43:51.452 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:51 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3790848244' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:43:51.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-17T16:43:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:51 smithi017 ceph-mon[27571]: pgmap v2188: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 374 objects/s recovering 2024-09-17T16:43:51.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:51 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3790848244' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:43:51.505 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:51.548 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:51.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:51.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:51.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:51.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:51.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:51.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:51.744 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:43:51.774 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:51.799 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:51.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:51.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:51.923 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:51.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:51.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-17T16:43:52.025 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:52.047 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:52.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:52.067 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:52.149 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:52.171 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:52.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:52.205 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:43:52.205 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:43:52.205 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:43:52.205 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:43:52.241 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:43:52.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:52.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:52.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:52.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:52.422 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:52.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3790848244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:43:52.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:52 smithi017 ceph-mon[27571]: osdmap e999: 12 total, 12 up, 12 in 2024-09-17T16:43:52.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:52.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:52.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:52.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3790848244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:43:52.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:52 smithi071 ceph-mon[30002]: osdmap e999: 12 total, 12 up, 12 in 2024-09-17T16:43:52.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3790848244' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:43:52.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:52 smithi173 ceph-mon[30530]: osdmap e999: 12 total, 12 up, 12 in 2024-09-17T16:43:52.650 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:52.665 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:52.679 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:52.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:52.781 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:52.806 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:52.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:52.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:52.922 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-17T16:43:52.932 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:52.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:52.941 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:53.034 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:53.057 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:53.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:53.159 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:53.181 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:53.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:53.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:53.193 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:43:53.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:53.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:53.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:53.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:53.319 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:43:53.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:53 smithi017 ceph-mon[27571]: pgmap v2190: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 377 objects/s recovering 2024-09-17T16:43:53.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:53 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:43:53.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:53 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:43:53.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:53 smithi017 ceph-mon[27571]: 9.0 scrub starts 2024-09-17T16:43:53.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:53 smithi017 ceph-mon[27571]: 9.0 scrub ok 2024-09-17T16:43:53.320 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/454783260' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:43:53.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:53.430 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:53.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:53.495 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:43:53.495 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:43:53.518 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:43:53.519 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:43:53.532 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:53.541 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:43:53.541 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:43:53.554 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:53.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:53.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:53.562 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:43:53.562 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:43:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:53 smithi071 ceph-mon[30002]: pgmap v2190: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 377 objects/s recovering 2024-09-17T16:43:53.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:53 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:43:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:53 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:43:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:53 smithi071 ceph-mon[30002]: 9.0 scrub starts 2024-09-17T16:43:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:53 smithi071 ceph-mon[30002]: 9.0 scrub ok 2024-09-17T16:43:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/454783260' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:43:53.583 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:43:53.584 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:43:53.605 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:43:53.606 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:43:53.627 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:43:53.627 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:43:53.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:53 smithi173 ceph-mon[30530]: pgmap v2190: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 377 objects/s recovering 2024-09-17T16:43:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:53 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:43:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:53 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:43:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:53 smithi173 ceph-mon[30530]: 9.0 scrub starts 2024-09-17T16:43:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:53 smithi173 ceph-mon[30530]: 9.0 scrub ok 2024-09-17T16:43:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/454783260' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:43:53.656 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:53.679 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:53.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:53.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:53.702 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:53.702 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:53.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:53.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:53.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:53.851 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:43:53.851 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:43:53.872 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:43:53.872 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:43:53.896 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:43:53.896 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:43:53.906 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:53.919 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:43:53.919 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:43:53.929 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:53.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:53.948 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:43:53.949 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:43:54.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:54.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:54.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:54.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:54.155 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:43:54.178 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:54.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:43:54.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:54.303 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:54.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:54.336 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.11 to 1 (10000) 2024-09-17T16:43:54.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:54 smithi071 ceph-mon[30002]: 1.9 scrub starts 2024-09-17T16:43:54.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:54 smithi071 ceph-mon[30002]: 1.9 scrub ok 2024-09-17T16:43:54.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:54 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:43:54.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/454783260' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:43:54.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:54 smithi071 ceph-mon[30002]: osdmap e1000: 12 total, 12 up, 12 in 2024-09-17T16:43:54.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3821171605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:54.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3808313635' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:43:54.352 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:54 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:43:54.379 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-17T16:43:54.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:54.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:54.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:54.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:54.551 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:54.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:54.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:54.639 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:43:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:54 smithi173 ceph-mon[30530]: 1.9 scrub starts 2024-09-17T16:43:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:54 smithi173 ceph-mon[30530]: 1.9 scrub ok 2024-09-17T16:43:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:54 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:43:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/454783260' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:43:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:54 smithi173 ceph-mon[30530]: osdmap e1000: 12 total, 12 up, 12 in 2024-09-17T16:43:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3821171605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3808313635' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:43:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:54 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:43:54.653 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:54.675 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:54.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:54 smithi017 ceph-mon[27571]: 1.9 scrub starts 2024-09-17T16:43:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:54 smithi017 ceph-mon[27571]: 1.9 scrub ok 2024-09-17T16:43:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:54 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:43:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/454783260' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:43:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:54 smithi017 ceph-mon[27571]: osdmap e1000: 12 total, 12 up, 12 in 2024-09-17T16:43:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3821171605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:43:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3808313635' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:43:54.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:54 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:43:54.777 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:54.800 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:54.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:54.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:54.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:54.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:55.025 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:55.048 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:55.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:55.146 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:55.146 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:55.150 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:55.172 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:55.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:55.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:55.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:55.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:55.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:55.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:55 smithi071 ceph-mon[30002]: pgmap v2192: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 296 objects/s recovering 2024-09-17T16:43:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:55 smithi071 ceph-mon[30002]: 9.15 scrub starts 2024-09-17T16:43:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:55 smithi071 ceph-mon[30002]: 9.15 scrub ok 2024-09-17T16:43:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:55 smithi071 ceph-mon[30002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:43:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:55 smithi071 ceph-mon[30002]: osdmap e1001: 12 total, 12 up, 12 in 2024-09-17T16:43:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:55 smithi071 ceph-mon[30002]: osdmap e1002: 12 total, 12 up, 12 in 2024-09-17T16:43:55.399 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:55.422 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:55.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:55.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:55 smithi017 ceph-mon[27571]: pgmap v2192: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 296 objects/s recovering 2024-09-17T16:43:55.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:55 smithi017 ceph-mon[27571]: 9.15 scrub starts 2024-09-17T16:43:55.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:55 smithi017 ceph-mon[27571]: 9.15 scrub ok 2024-09-17T16:43:55.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:55 smithi017 ceph-mon[27571]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:43:55.491 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:55 smithi017 ceph-mon[27571]: osdmap e1001: 12 total, 12 up, 12 in 2024-09-17T16:43:55.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:55 smithi017 ceph-mon[27571]: osdmap e1002: 12 total, 12 up, 12 in 2024-09-17T16:43:55.524 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:55.547 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:55.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:55 smithi173 ceph-mon[30530]: pgmap v2192: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 33 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556/9 objects degraded (6177.778%); 0 B/s, 296 objects/s recovering 2024-09-17T16:43:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:55 smithi173 ceph-mon[30530]: 9.15 scrub starts 2024-09-17T16:43:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:55 smithi173 ceph-mon[30530]: 9.15 scrub ok 2024-09-17T16:43:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:55 smithi173 ceph-mon[30530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:43:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:55 smithi173 ceph-mon[30530]: osdmap e1001: 12 total, 12 up, 12 in 2024-09-17T16:43:55.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:55 smithi173 ceph-mon[30530]: osdmap e1002: 12 total, 12 up, 12 in 2024-09-17T16:43:55.648 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:55.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:55.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-17T16:43:55.773 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:55.796 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:55.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:55.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:55.920 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:55.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:55.999 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:43:56.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:56.044 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:56.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:56.146 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:56.167 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:56.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:56.176 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T16:43:56.269 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:56.292 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:56.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:56.377 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:56 smithi173 ceph-mon[30530]: 9.4 scrub starts 2024-09-17T16:43:56.377 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:56 smithi173 ceph-mon[30530]: 9.4 scrub ok 2024-09-17T16:43:56.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:56.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:56.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:56.445 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:56.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:56 smithi017 ceph-mon[27571]: 9.4 scrub starts 2024-09-17T16:43:56.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:56 smithi017 ceph-mon[27571]: 9.4 scrub ok 2024-09-17T16:43:56.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:56.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:56.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:56.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:56 smithi071 ceph-mon[30002]: 9.4 scrub starts 2024-09-17T16:43:56.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:56 smithi071 ceph-mon[30002]: 9.4 scrub ok 2024-09-17T16:43:56.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:56.659 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:56.659 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:56.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:56.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:56.769 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:56.792 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:56.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:56.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-17T16:43:56.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:56.916 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:56.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:57.018 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:43:57.041 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:57.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:43:57.106 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:57.143 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:57.169 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:57.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:57.271 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:57.290 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T16:43:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:57.297 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:57.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:57.399 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:57.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-17T16:43:57.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:57.454 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:57.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:57 smithi017 ceph-mon[27571]: pgmap v2195: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383/9 objects degraded (4255.556%); 0 B/s, 319 objects/s recovering 2024-09-17T16:43:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:57 smithi017 ceph-mon[27571]: Health check update: Degraded data redundancy: 383/9 objects degraded (4255.556%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:57.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4287753735' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:43:57.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:43:57.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:57 smithi071 ceph-mon[30002]: pgmap v2195: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383/9 objects degraded (4255.556%); 0 B/s, 319 objects/s recovering 2024-09-17T16:43:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:57 smithi071 ceph-mon[30002]: Health check update: Degraded data redundancy: 383/9 objects degraded (4255.556%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:57.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4287753735' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:43:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:57.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:57.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:43:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:57 smithi173 ceph-mon[30530]: pgmap v2195: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383/9 objects degraded (4255.556%); 0 B/s, 319 objects/s recovering 2024-09-17T16:43:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:57 smithi173 ceph-mon[30530]: Health check update: Degraded data redundancy: 383/9 objects degraded (4255.556%), 2 pgs degraded (PG_DEGRADED) 2024-09-17T16:43:57.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4287753735' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:43:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:43:57.679 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:57.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:43:57.701 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:57.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:57.806 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:57.828 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:57.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:57.928 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:57.951 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:57.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:58.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:58.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:58.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:58.075 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:58.136 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T16:43:58.153 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:58.153 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:58.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:58.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:58.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:58.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:58.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:58.325 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:58.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4287753735' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:43:58.359 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:58 smithi017 ceph-mon[27571]: osdmap e1003: 12 total, 12 up, 12 in 2024-09-17T16:43:58.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:58.438 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:58.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:58.448 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:58.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:43:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4287753735' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:43:58.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:58 smithi071 ceph-mon[30002]: osdmap e1003: 12 total, 12 up, 12 in 2024-09-17T16:43:58.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:58.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:43:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4287753735' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:43:58.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:58 smithi173 ceph-mon[30530]: osdmap e1003: 12 total, 12 up, 12 in 2024-09-17T16:43:58.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:58.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:58.694 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:58.795 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:43:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:58.822 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:58.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:43:58.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-17T16:43:58.923 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:58.947 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:58.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:58.969 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:43:59.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:43:59.072 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:59.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:59.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:43:59.087 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:43:59.174 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:59.197 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:59.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:59.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:43:59.300 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:43:59.317 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T16:43:59.317 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-17T16:43:59.326 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:59.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:43:59.428 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:43:59.450 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:43:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:59.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:43:59.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:59 smithi173 ceph-mon[30530]: pgmap v2197: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383/9 objects degraded (4255.556%); 0 B/s, 332 objects/s recovering 2024-09-17T16:43:59.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2981232744' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:43:59.480 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:43:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/195039077' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:43:59.513 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:59 smithi017 ceph-mon[27571]: pgmap v2197: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383/9 objects degraded (4255.556%); 0 B/s, 332 objects/s recovering 2024-09-17T16:43:59.513 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2981232744' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:43:59.513 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:43:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/195039077' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:43:59.517 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:43:59.518 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:43:59.523 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:43:59.523 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:43:59.545 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:43:59.545 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:43:59.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:59.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:59 smithi071 ceph-mon[30002]: pgmap v2197: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383/9 objects degraded (4255.556%); 0 B/s, 332 objects/s recovering 2024-09-17T16:43:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2981232744' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-17T16:43:59.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:43:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/195039077' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:43:59.566 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:43:59.567 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:43:59.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:59.577 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:59.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:59.589 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:43:59.589 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:43:59.616 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:43:59.617 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:43:59.641 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:43:59.641 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:43:59.664 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:43:59.664 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:43:59.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:43:59.687 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:43:59.687 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:43:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:59.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:43:59.702 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:43:59.709 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:43:59.710 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:43:59.732 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:43:59.732 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:43:59.754 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:43:59.755 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:43:59.777 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:43:59.778 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:43:59.803 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:43:59.821 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 106902 objects 2024-09-17T16:43:59.822 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :204.818 2024-09-17T16:43:59.829 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:59.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:43:59.931 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:43:59.956 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:43:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:43:59.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:44:00.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:44:00.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:00.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:44:00.182 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:44:00.206 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:00.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:00.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:44:00.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-17T16:44:00.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:44:00.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:00.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:44:00.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:44:00.456 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:00.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:00.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:44:00.486 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:44:00.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:44:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:00 smithi071 ceph-mon[30002]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:44:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2981232744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:44:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/195039077' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:44:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:00 smithi071 ceph-mon[30002]: osdmap e1004: 12 total, 12 up, 12 in 2024-09-17T16:44:00.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2032425564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:44:00.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:00.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:00.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:44:00.607 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:00 smithi173 ceph-mon[30530]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:44:00.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2981232744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:44:00.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/195039077' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:44:00.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:00 smithi173 ceph-mon[30530]: osdmap e1004: 12 total, 12 up, 12 in 2024-09-17T16:44:00.608 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2032425564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:44:00.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:44:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:00.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:00.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:44:00.714 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:00 smithi017 ceph-mon[27571]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:44:00.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2981232744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-17T16:44:00.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/195039077' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:44:00.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:00 smithi017 ceph-mon[27571]: osdmap e1004: 12 total, 12 up, 12 in 2024-09-17T16:44:00.715 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2032425564' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:44:00.807 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:44:00.829 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:00.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:44:00.931 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:44:00.953 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:00.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:44:00.972 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:44:00.972 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:44:00.972 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:44:00.972 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:44:01.055 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:44:01.076 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:01.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:44:01.180 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:44:01.201 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:01.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:44:01.303 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:44:01.331 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:01.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:44:01.433 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:44:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:01.458 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:01.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:44:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:01 smithi017 ceph-mon[27571]: pgmap v2199: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383/9 objects degraded (4255.556%); 0 B/s, 276 objects/s recovering 2024-09-17T16:44:01.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:01 smithi017 ceph-mon[27571]: osdmap e1005: 12 total, 12 up, 12 in 2024-09-17T16:44:01.559 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T16:44:01.559 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:44:01.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:01 smithi173 ceph-mon[30530]: pgmap v2199: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383/9 objects degraded (4255.556%); 0 B/s, 276 objects/s recovering 2024-09-17T16:44:01.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:01 smithi173 ceph-mon[30530]: osdmap e1005: 12 total, 12 up, 12 in 2024-09-17T16:44:01.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-17T16:44:01.583 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:01.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:44:01.686 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:44:01.712 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:01.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:44:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:01 smithi071 ceph-mon[30002]: pgmap v2199: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 34 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 383/9 objects degraded (4255.556%); 0 B/s, 276 objects/s recovering 2024-09-17T16:44:01.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:01 smithi071 ceph-mon[30002]: osdmap e1005: 12 total, 12 up, 12 in 2024-09-17T16:44:01.814 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:44:01.821 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:44:01.839 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:01.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:44:01.941 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:44:01.964 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:01.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:44:02.050 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:02.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:44:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:02.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:02.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:44:02.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:44:02.216 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:02.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:02.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:44:02.318 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:44:02.343 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:02.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:02.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:44:02.356 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:44:02.356 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:44:02.356 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:44:02.356 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:44:02.356 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:44:02.445 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:44:02.467 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:02.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:02.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:44:02.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:44:02.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:02.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:44:02.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:02 smithi017 ceph-mon[27571]: osdmap e1006: 12 total, 12 up, 12 in 2024-09-17T16:44:02.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:02 smithi173 ceph-mon[30530]: osdmap e1006: 12 total, 12 up, 12 in 2024-09-17T16:44:02.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:44:02.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:02.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:44:02.741 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-17T16:44:02.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:44:02.817 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:02 smithi071 ceph-mon[30002]: osdmap e1006: 12 total, 12 up, 12 in 2024-09-17T16:44:02.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:02.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:44:02.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9 2024-09-17T16:44:02.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:02.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.9' 2024-09-17T16:44:02.995 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:03.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-17T16:44:03.064 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:44:03.074 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-09-17T16:44:03.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-09-17T16:44:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:03.087 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:03.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:44:03.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:44:03.214 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:03.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:44:03.280 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:44:03.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:44:03.329 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T16:44:03.352 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:03.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:44:03.387 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:03 smithi017 ceph-mon[27571]: pgmap v2202: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 194 objects/s recovering 2024-09-17T16:44:03.387 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:03 smithi017 ceph-mon[27571]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 383/9 objects degraded (4255.556%), 2 pgs degraded) 2024-09-17T16:44:03.387 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:03 smithi017 ceph-mon[27571]: osdmap e1007: 12 total, 12 up, 12 in 2024-09-17T16:44:03.387 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/180255949' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:44:03.454 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:44:03.477 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:03.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:03.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:44:03.579 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:44:03.592 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:03 smithi173 ceph-mon[30530]: pgmap v2202: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 194 objects/s recovering 2024-09-17T16:44:03.592 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:03 smithi173 ceph-mon[30530]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 383/9 objects degraded (4255.556%), 2 pgs degraded) 2024-09-17T16:44:03.592 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:03 smithi173 ceph-mon[30530]: osdmap e1007: 12 total, 12 up, 12 in 2024-09-17T16:44:03.592 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/180255949' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:44:03.605 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:03.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:44:03.707 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:44:03.729 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:03.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:44:03.780 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:44:03.780 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:44:03.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:44:03.780 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:44:03.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.780 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.15 740/739 [PushOp(8:a91ad1da:::benchmark_data_smithi173_130566_object2731:head, version: 732'202, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:a91ad1da:::benchmark_data_smithi173_130566_object2731:head@732'202, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 576.68020137600001, 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448453453, 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097184+0000", 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097185+0000", 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2799999999999995e-07 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.097245+0000", 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9749e-05 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545252+0000", 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44800729700000003 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545267+0000", 2024-09-17T16:44:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5017999999999999e-05 2024-09-17T16:44:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545346+0000", 2024-09-17T16:44:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9491000000000002e-05 2024-09-17T16:44:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545357+0000", 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0491e-05 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.545638+0000", 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028057899999999998 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(8.13 740/739 [PushOp(8:cce6ff3a:::benchmark_data_smithi173_130566_object3681:head, version: 732'214, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:cce6ff3a:::benchmark_data_smithi173_130566_object3681:head@732'214, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:44:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 576.67825888699997, 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448568835, 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099127+0000", 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7899999999999997e-07 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.099181+0000", 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3591999999999998e-05 2024-09-17T16:44:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547347+0000", 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.448166224 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547363+0000", 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5364000000000001e-05 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547441+0000", 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8873000000000002e-05 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547450+0000", 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5469999999999997e-06 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:34:27.547695+0000", 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024535600000000001 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.17883.0:115081 8.0 e800/767 8:02125afa:::benchmark_data_smithi173_130566_object20106:head v 800'3851, mlcod=800'3848)", 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:54.801782+0000", 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 428.97560296, 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66971172400000001, 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801782+0000", 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801782+0000", 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801783+0000", 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0500000000000002e-07 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:54.801788+0000", 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1939999999999997e-06 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469802+0000", 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66801375100000004 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469811+0000", 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.473e-06 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469859+0000", 2024-09-17T16:44:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7874999999999998e-05 2024-09-17T16:44:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.469877+0000", 2024-09-17T16:44:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8321e-05 2024-09-17T16:44:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:44:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.471459+0000", 2024-09-17T16:44:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00158165 2024-09-17T16:44:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:55.471494+0000", 2024-09-17T16:44:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5655000000000003e-05 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:8122 9.1 9.c6cc7e1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:08.217944+0000", 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 295.55944130799998, 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98969176699999994, 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8122 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217944+0000", 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217944+0000", 2024-09-17T16:44:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217945+0000", 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.319e-06 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217965+0000", 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9749e-05 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217976+0000", 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1021e-05 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.217982+0000", 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.412e-06 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.218030+0000", 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8850999999999997e-05 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.218040+0000", 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7289999999999995e-06 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207605+0000", 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98956506499999997 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207636+0000", 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0620999999999997e-05 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:8128 9.1 9.32423851 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:08.409343+0000", 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 295.36804210700001, 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79838889400000002, 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8128 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409343+0000", 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409343+0000", 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409346+0000", 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7300000000000001e-06 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409389+0000", 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3245999999999999e-05 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409431+0000", 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1604999999999999e-05 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409439+0000", 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.106e-06 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409496+0000", 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7380999999999998e-05 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.409504+0000", 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4889999999999998e-06 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207722+0000", 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.798218223 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207732+0000", 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0114e-05 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:8132 9.1 9.cd430e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:08.494386+0000", 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 295.28299992900003, 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71354715499999999, 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8132 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494386+0000", 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494386+0000", 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494387+0000", 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.102e-06 2024-09-17T16:44:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494407+0000", 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0346000000000001e-05 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494418+0000", 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1202e-05 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494424+0000", 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2809999999999997e-06 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494470+0000", 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5935e-05 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.494478+0000", 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9540000000000004e-06 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207922+0000", 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71344410400000002 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.207933+0000", 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0231000000000001e-05 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:8191 9.8 9:1269694a:::benchmark_data_smithi173_150363_object8190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:09.811424+0000", 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 293.96596094400002, 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51999695400000001, 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8191 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811424+0000", 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811424+0000", 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811427+0000", 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3290000000000001e-06 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811450+0000", 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3159999999999998e-05 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811461+0000", 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1252e-05 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811467+0000", 2024-09-17T16:44:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5790000000000003e-06 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811521+0000", 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4491999999999999e-05 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811668+0000", 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014676700000000001 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.811813+0000", 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014480100000000001 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812647+0000", 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083428699999999996 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812854+0000", 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020655200000000001 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331303+0000", 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51844964000000004 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331373+0000", 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9525999999999998e-05 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331421+0000", 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8569000000000002e-05 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27874.0:8191 9.8 e867/858 ondisk, result = 0)", 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:09.812665+0000", 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 293.96472029699999, 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51876305700000003, 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812665+0000", 2024-09-17T16:44:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812665+0000", 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812666+0000", 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0500000000000003e-07 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.812670+0000", 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1729999999999997e-06 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331232+0000", 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51856239699999995 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331237+0000", 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9180000000000002e-06 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331289+0000", 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1681000000000002e-05 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331298+0000", 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7180000000000002e-06 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:10.331428+0000", 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013056499999999999 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:193627 9.8 9:109bd254:::benchmark_data_smithi173_150363_object86722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:40.645978+0000", 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 83.131407550000006, 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41238860500000002, 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193627 2024-09-17T16:44:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.645978+0000", 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.645978+0000", 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.645980+0000", 2024-09-17T16:44:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7570000000000001e-06 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.645987+0000", 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0040000000000002e-06 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.646001+0000", 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4114000000000001e-05 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.646009+0000", 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.79e-06 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.646076+0000", 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7353000000000004e-05 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.646211+0000", 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013471300000000001 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.646333+0000", 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122488 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.649567+0000", 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032343340000000002 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.649627+0000", 2024-09-17T16:44:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9431000000000001e-05 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:41.058253+0000", 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40862622500000001 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:41.058308+0000", 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5232000000000002e-05 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:41.058366+0000", 2024-09-17T16:44:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8164000000000001e-05 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27874.0:193633 9.c e960/872 9:3a9cd4bd:::benchmark_data_smithi173_150363_object86728:head v 960'12190, mlcod=960'12188)", 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:40.654380+0000", 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 83.123005058000004, 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41040242399999999, 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.654380+0000", 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.654380+0000", 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.654382+0000", 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.68e-06 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.654390+0000", 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8790000000000003e-06 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.685205+0000", 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.030814658000000002 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.685215+0000", 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0540000000000001e-05 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.685342+0000", 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012727300000000001 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.685365+0000", 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2503000000000001e-05 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:41.064741+0000", 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37937595499999999 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:41.064783+0000", 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1936e-05 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:193637 9.16 9:6e5f241d:::benchmark_data_smithi173_150363_object86732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:40.657990+0000", 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 83.119395053999995, 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40149604999999999, 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193637 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.657990+0000", 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.657990+0000", 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.657992+0000", 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7400000000000001e-06 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.657998+0000", 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2999999999999998e-06 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.685532+0000", 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027533841 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.685543+0000", 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0767999999999999e-05 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.685605+0000", 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2144000000000007e-05 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.685707+0000", 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010187 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,11", 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.685810+0000", 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102685 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:40.690424+0000", 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046147640000000004 2024-09-17T16:44:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:41.055004+0000", 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36457908100000003 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:41.059355+0000", 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043513459999999999 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:41.059447+0000", 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2181000000000004e-05 2024-09-17T16:44:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:41.059486+0000", 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9329999999999998e-05 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:202378 9.1 9:85421b6a:::benchmark_data_smithi173_150363_object95473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e964)", 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:50.502997+0000", 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.274388157000004, 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42475576500000001, 2024-09-17T16:44:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 202378 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.502997+0000", 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.502997+0000", 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.502998+0000", 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9999999999999996e-07 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.503002+0000", 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2930000000000002e-06 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.503015+0000", 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2232000000000001e-05 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.503021+0000", 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.72e-06 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.503065+0000", 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3516000000000001e-05 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.503154+0000", 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8599999999999999e-05 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,10", 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.503563+0000", 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040913100000000002 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.505906+0000", 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023431649999999999 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.530776+0000", 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024870361000000001 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.927621+0000", 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39684499400000001 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.927704+0000", 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3233999999999996e-05 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.927753+0000", 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8619000000000003e-05 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:202399 9.1 9:84a7de00:::benchmark_data_smithi173_150363_object95494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e964)", 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:50.527384+0000", 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.250001001000001, 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40658674700000003, 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 202399 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.527384+0000", 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.527384+0000", 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.527386+0000", 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.361e-06 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.527391+0000", 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7840000000000003e-06 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.527402+0000", 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0986e-05 2024-09-17T16:44:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.527408+0000", 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9129999999999997e-06 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.527463+0000", 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4416999999999998e-05 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.527565+0000", 2024-09-17T16:44:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101681 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,10", 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.527990+0000", 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042568800000000002 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.530188+0000", 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021972850000000002 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.557659+0000", 2024-09-17T16:44:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027471808 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.933863+0000", 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37620327799999997 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.933928+0000", 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4998000000000003e-05 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.933971+0000", 2024-09-17T16:44:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3547999999999997e-05 2024-09-17T16:44:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:202413 9.1 9:83938581:::benchmark_data_smithi173_150363_object95508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-09-17T16:44:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:50.554261+0000", 2024-09-17T16:44:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.223124231, 2024-09-17T16:44:03.848 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:44:03.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:03 smithi071 ceph-mon[30002]: pgmap v2202: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 194 objects/s recovering 2024-09-17T16:44:03.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:03 smithi071 ceph-mon[30002]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 383/9 objects degraded (4255.556%), 2 pgs degraded) 2024-09-17T16:44:03.850 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:03 smithi071 ceph-mon[30002]: osdmap e1007: 12 total, 12 up, 12 in 2024-09-17T16:44:03.850 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/180255949' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:44:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40409942399999998, 2024-09-17T16:44:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 202413 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.554261+0000", 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.554261+0000", 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.554263+0000", 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.592e-06 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.554268+0000", 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7090000000000001e-06 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.554278+0000", 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0798999999999999e-05 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.554284+0000", 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3909999999999997e-06 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.554326+0000", 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2469999999999998e-05 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.554403+0000", 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7099000000000002e-05 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,10", 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.554826+0000", 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042277800000000001 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.557078+0000", 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022523510000000001 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.934065+0000", 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37698659000000001 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.958281+0000", 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024216062 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.958331+0000", 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9524999999999998e-05 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:50.958361+0000", 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0057999999999999e-05 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:202822 9.1 9:874630f1:::benchmark_data_smithi173_150363_object95917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:51.574029+0000", 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 72.203356458000002, 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52138344299999995, 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 202822 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.574029+0000", 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.574029+0000", 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.574030+0000", 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4100000000000001e-06 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.574036+0000", 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9580000000000004e-06 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.574048+0000", 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1702000000000001e-05 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.574056+0000", 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6669999999999996e-06 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.574106+0000", 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0090000000000003e-05 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.574197+0000", 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0704000000000004e-05 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,10", 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.574611+0000", 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00041444399999999998 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.576875+0000", 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022642579999999999 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.577350+0000", 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047518899999999999 2024-09-17T16:44:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:52.095339+0000", 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51798867299999996 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:52.095372+0000", 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3238999999999998e-05 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:52.095412+0000", 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0108999999999998e-05 2024-09-17T16:44:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27874.0:202822 9.1 e966/911 ondisk, result = 0)", 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:51.577460+0000", 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 72.199925914000005, 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51796006699999997, 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.577460+0000", 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.577460+0000", 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.577460+0000", 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4900000000000005e-07 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.577464+0000", 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.326e-06 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:52.095273+0000", 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51780932700000004 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:52.095284+0000", 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1046999999999999e-05 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:52.095331+0000", 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6934999999999997e-05 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:52.095336+0000", 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7369999999999997e-06 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:52.095420+0000", 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3945999999999999e-05 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:202822 9.1 9:874630f1:::benchmark_data_smithi173_150363_object95917:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e966)", 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:42:51.630112+0000", 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 72.147273893000005, 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.46527030000000003, 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for ondisk", 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 202822 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.630112+0000", 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.630112+0000", 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.630112+0000", 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.02e-07 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.630117+0000", 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.352e-06 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.630125+0000", 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7970000000000001e-06 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.630130+0000", 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1590000000000001e-06 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.630175+0000", 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5547999999999998e-05 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for ondisk", 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:51.630221+0000", 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6133e-05 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:42:52.095382+0000", 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.465160409 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(9.1 e992,969 9:80a9c6aa:::benchmark_data_smithi173_150363_object19968:head,920'7849)", 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:43:37.079538+0000", 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.697847838000001, 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.56123029400000002, 2024-09-17T16:44:03.870 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:37.079538+0000", 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:37.079538+0000", 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:37.079539+0000", 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.593e-06 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:37.079549+0000", 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9000000000000001e-06 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:37.639988+0000", 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.56043859699999998 2024-09-17T16:44:03.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:37.639999+0000", 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1233000000000001e-05 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:37.640114+0000", 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001154 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:37.640768+0000", 2024-09-17T16:44:03.873 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065357100000000003 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(9.1 e994,969 9:84a3724f:::benchmark_data_smithi173_150363_object34209:head,924'8302)", 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:43:40.747546+0000", 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.029839774999999, 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45171141599999998, 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.874 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:40.747546+0000", 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:40.747546+0000", 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:40.747546+0000", 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2099999999999996e-07 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:40.747551+0000", 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5029999999999999e-06 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:41.198518+0000", 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45096703300000002 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:41.198526+0000", 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1510000000000004e-06 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:41.198624+0000", 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8210999999999999e-05 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:41.199257+0000", 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00063279700000000005 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.877 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(9.1 e1004,969 9:83ac7066:::benchmark_data_smithi173_150363_object105067:head,972'10526)", 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:43:59.806462+0000", 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.9709231589999998, 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053421000000000002, 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:59.806462+0000", 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:59.806462+0000", 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:59.806468+0000", 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1600000000000003e-06 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:59.806479+0000", 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0536e-05 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:59.806540+0000", 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1428999999999997e-05 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:59.806555+0000", 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4289e-05 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:59.806720+0000", 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016488700000000001 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:03.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:03.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:59.806996+0000", 2024-09-17T16:44:03.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00027690899999999999 2024-09-17T16:44:03.881 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.881 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.881 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.881 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:03.881 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:03.881 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:44:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:03.882 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:03.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:44:03.983 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:44:04.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:44:04.009 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:04.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:04.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:44:04.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:44:04.133 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:04.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:04.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:44:04.236 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:44:04.261 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:04.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:44:04.346 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-17T16:44:04.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:44:04.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-17T16:44:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:04.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:04.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:44:04.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:44:04.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:04.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:44:04.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:44:04.636 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:04.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:44:04.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/180255949' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:44:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:04 smithi173 ceph-mon[30530]: osdmap e1008: 12 total, 12 up, 12 in 2024-09-17T16:44:04.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1541161668' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:44:04.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-17T16:44:04.728 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/180255949' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:44:04.729 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:04 smithi017 ceph-mon[27571]: osdmap e1008: 12 total, 12 up, 12 in 2024-09-17T16:44:04.729 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1541161668' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:44:04.737 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:44:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:04.760 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:04.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:44:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/180255949' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:44:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:04 smithi071 ceph-mon[30002]: osdmap e1008: 12 total, 12 up, 12 in 2024-09-17T16:44:04.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1541161668' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:44:04.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:44:04.887 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:04.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:04.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:44:04.923 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:44:04.989 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:44:05.014 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:05.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:05.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:44:05.052 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:05.116 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:44:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:05.139 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:05.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:44:05.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:44:05.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:05.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:44:05.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:44:05.384 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:44:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:44:05.384 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:44:05.384 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:44:05.386 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:05.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:44:05.488 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:44:05.510 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:05.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:44:05.612 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:44:05.633 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:05 smithi017 ceph-mon[27571]: pgmap v2205: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 0 B/s, 238 objects/s recovering 2024-09-17T16:44:05.633 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:05 smithi017 ceph-mon[27571]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:44:05.633 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1541161668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:44:05.633 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:05 smithi017 ceph-mon[27571]: osdmap e1009: 12 total, 12 up, 12 in 2024-09-17T16:44:05.633 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:05 smithi017 ceph-mon[27571]: osdmap e1010: 12 total, 12 up, 12 in 2024-09-17T16:44:05.634 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:44:05.635 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:44:05.639 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:05.640 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:05.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:44:05.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:05 smithi173 ceph-mon[30530]: pgmap v2205: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 0 B/s, 238 objects/s recovering 2024-09-17T16:44:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:05 smithi173 ceph-mon[30530]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:44:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1541161668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:44:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:05 smithi173 ceph-mon[30530]: osdmap e1009: 12 total, 12 up, 12 in 2024-09-17T16:44:05.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:05 smithi173 ceph-mon[30530]: osdmap e1010: 12 total, 12 up, 12 in 2024-09-17T16:44:05.658 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:44:05.658 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:44:05.680 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:44:05.680 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:44:05.700 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:44:05.700 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:44:05.722 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:44:05.722 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:44:05.741 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:44:05.744 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:44:05.744 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:44:05.765 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:05.765 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:44:05.766 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:44:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:05.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:44:05.787 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:44:05.787 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:44:05.809 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:44:05.809 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:44:05.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:05 smithi071 ceph-mon[30002]: pgmap v2205: 37 pgs: 1 active+remapped+backfill_toofull, 36 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 0 B/s, 238 objects/s recovering 2024-09-17T16:44:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:05 smithi071 ceph-mon[30002]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:44:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1541161668' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-17T16:44:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:05 smithi071 ceph-mon[30002]: osdmap e1009: 12 total, 12 up, 12 in 2024-09-17T16:44:05.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:05 smithi071 ceph-mon[30002]: osdmap e1010: 12 total, 12 up, 12 in 2024-09-17T16:44:05.830 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:44:05.830 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:44:05.851 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:44:05.851 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:44:05.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:44:05.872 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:44:05.873 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:44:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:05.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:05.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:44:05.994 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5 2024-09-17T16:44:06.020 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:06.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.5' 2024-09-17T16:44:06.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-17T16:44:06.122 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:44:06.144 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:06.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:44:06.247 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:44:06.269 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:06.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:44:06.348 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:44:06.370 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:44:06.393 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:06.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:44:06.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:44:06.521 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:06.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:44:06.611 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1316022404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:44:06.623 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:44:06.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1316022404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:44:06.648 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:06.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:44:06.750 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:44:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:06.773 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:06.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:44:06.813 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1316022404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:44:06.825 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:44:06.825 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:44:06.874 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:44:06.897 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:06.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:06.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:44:06.999 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6 2024-09-17T16:44:07.024 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:07.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.6' 2024-09-17T16:44:07.038 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:07.081 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:44:07.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:44:07.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:07.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:07.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:44:07.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:44:07.273 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:07.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:44:07.345 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T16:44:07.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4 2024-09-17T16:44:07.399 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:07.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:07.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.4' 2024-09-17T16:44:07.430 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:07 smithi173 ceph-mon[30530]: pgmap v2208: 37 pgs: 37 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-17T16:44:07.430 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:07 smithi173 ceph-mon[30530]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T16:44:07.430 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/966694609' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:44:07.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:07 smithi017 ceph-mon[27571]: pgmap v2208: 37 pgs: 37 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-17T16:44:07.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:07 smithi017 ceph-mon[27571]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T16:44:07.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/966694609' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:44:07.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-17T16:44:07.501 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:44:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:07.529 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:07.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:44:07.630 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:44:07.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:07.655 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:07.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:44:07.701 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:44:07.756 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:44:07.777 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:07.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:44:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:07 smithi071 ceph-mon[30002]: pgmap v2208: 37 pgs: 37 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-09-17T16:44:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:07 smithi071 ceph-mon[30002]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-17T16:44:07.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/966694609' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:44:07.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:44:07.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:07.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:44:08.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:44:08.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:08.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:44:08.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:44:08.142 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-09-17T16:44:08.142 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:44:08.150 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:08.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:44:08.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8 2024-09-17T16:44:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:08.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:08.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.8' 2024-09-17T16:44:08.354 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'unique_pool_7' removed 2024-09-17T16:44:08.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11 2024-09-17T16:44:08.381 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T16:44:08.381 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:44:08.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/966694609' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:44:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:08 smithi173 ceph-mon[30530]: osdmap e1011: 12 total, 12 up, 12 in 2024-09-17T16:44:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2300874852' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:44:08.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:08.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.11' 2024-09-17T16:44:08.499 DEBUG:teuthology.orchestra.run.smithi071:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7 2024-09-17T16:44:08.522 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: docker: command not found 2024-09-17T16:44:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:08.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi071 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.7' 2024-09-17T16:44:08.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0 2024-09-17T16:44:08.648 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:08.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.0' 2024-09-17T16:44:08.694 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:44:08.716 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/966694609' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:44:08.716 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:08 smithi017 ceph-mon[27571]: osdmap e1011: 12 total, 12 up, 12 in 2024-09-17T16:44:08.716 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2300874852' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:44:08.752 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1 2024-09-17T16:44:08.775 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:08.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.1' 2024-09-17T16:44:08.814 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-17T16:44:08.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/966694609' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:44:08.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:08 smithi071 ceph-mon[30002]: osdmap e1011: 12 total, 12 up, 12 in 2024-09-17T16:44:08.815 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2300874852' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-17T16:44:08.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10 2024-09-17T16:44:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-09-17T16:44:08.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.10' 2024-09-17T16:44:09.001 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3 2024-09-17T16:44:09.023 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:09.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:09.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.3' 2024-09-17T16:44:09.058 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:44:09.125 DEBUG:teuthology.orchestra.run.smithi017:> sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2 2024-09-17T16:44:09.149 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: docker: command not found 2024-09-17T16:44:09.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:44:09.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi017 with status 1: 'sudo docker kill -s 1 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd.2' 2024-09-17T16:44:09.162 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-17T16:44:09.171 INFO:tasks.thrashosds:joining thrashosds 2024-09-17T16:44:09.209 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:09.248 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:09.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:09 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:09.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:09 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:09.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:09 smithi071 ceph-mon[30002]: pgmap v2210: 37 pgs: 37 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:09.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2300874852' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:44:09.433 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:09 smithi071 ceph-mon[30002]: osdmap e1012: 12 total, 12 up, 12 in 2024-09-17T16:44:09.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:09 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:09.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:09 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:09.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:09 smithi017 ceph-mon[27571]: pgmap v2210: 37 pgs: 37 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:09.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2300874852' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:44:09.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:09 smithi017 ceph-mon[27571]: osdmap e1012: 12 total, 12 up, 12 in 2024-09-17T16:44:09.536 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-09-17T16:44:09.536 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:44:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:09 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:09 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:09 smithi173 ceph-mon[30530]: pgmap v2210: 37 pgs: 37 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2300874852' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-17T16:44:09.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:09 smithi173 ceph-mon[30530]: osdmap e1012: 12 total, 12 up, 12 in 2024-09-17T16:44:09.821 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:10.168 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-17T16:44:10.360 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:44:10.412 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:44:10.425 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4128390839' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:44:10.508 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-17T16:44:10.508 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:44:10.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-17T16:44:10.529 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:44:10.532 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4128390839' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:44:10.549 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-17T16:44:10.550 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:44:10.570 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-17T16:44:10.570 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:44:10.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-17T16:44:10.591 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:44:10.612 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-17T16:44:10.612 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:44:10.612 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4128390839' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:44:10.633 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6: {} 2024-09-17T16:44:10.633 INFO:teuthology.orchestra.run.smithi017.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:44:10.654 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7: {} 2024-09-17T16:44:10.654 INFO:teuthology.orchestra.run.smithi017.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:44:10.676 INFO:teuthology.orchestra.run.smithi017.stdout:osd.8: {} 2024-09-17T16:44:10.676 INFO:teuthology.orchestra.run.smithi017.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:44:10.697 INFO:teuthology.orchestra.run.smithi017.stdout:osd.9: {} 2024-09-17T16:44:10.697 INFO:teuthology.orchestra.run.smithi017.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:44:10.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.10: {} 2024-09-17T16:44:10.728 INFO:teuthology.orchestra.run.smithi017.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:44:10.764 INFO:teuthology.orchestra.run.smithi017.stdout:osd.11: {} 2024-09-17T16:44:10.765 INFO:teuthology.orchestra.run.smithi017.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:44:10.908 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:44:10.908 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:44:10.908 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:44:10.908 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:44:10.908 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:44:10.925 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:44:10.926 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:44:11.562 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-17T16:44:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:11 smithi173 ceph-mon[30530]: pgmap v2212: 11 pgs: 11 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:11 smithi173 ceph-mon[30530]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:44:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4128390839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:44:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:11 smithi173 ceph-mon[30530]: osdmap e1013: 12 total, 12 up, 12 in 2024-09-17T16:44:11.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/883952823' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:44:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:11 smithi017 ceph-mon[27571]: pgmap v2212: 11 pgs: 11 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:11 smithi017 ceph-mon[27571]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:44:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4128390839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:44:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:11 smithi017 ceph-mon[27571]: osdmap e1013: 12 total, 12 up, 12 in 2024-09-17T16:44:11.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/883952823' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:44:11.806 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:44:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:11 smithi071 ceph-mon[30002]: pgmap v2212: 11 pgs: 11 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:11 smithi071 ceph-mon[30002]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-17T16:44:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4128390839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:44:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:11 smithi071 ceph-mon[30002]: osdmap e1013: 12 total, 12 up, 12 in 2024-09-17T16:44:11.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/883952823' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-17T16:44:12.270 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:44:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:44:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:44:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:44:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.270 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "rep_scrubmap(8.d e728 from shard 11)", 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:36:13.537886+0000", 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 478.72832159400002, 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.66040258699999999, 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537886+0000", 2024-09-17T16:44:12.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537886+0000", 2024-09-17T16:44:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537887+0000", 2024-09-17T16:44:12.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2410000000000001e-06 2024-09-17T16:44:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:13.537958+0000", 2024-09-17T16:44:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0797999999999995e-05 2024-09-17T16:44:12.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.197838+0000", 2024-09-17T16:44:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.65987990699999999 2024-09-17T16:44:12.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.197850+0000", 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2785999999999999e-05 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.197989+0000", 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013900800000000001 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.198027+0000", 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7422999999999997e-05 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:36:14.198288+0000", 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026142400000000001 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(8.11 e849,797 8:88a3dd87:::benchmark_data_smithi173_130566_object54925:head,818'5023)", 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:35.655134+0000", 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 336.61107366200002, 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.587224894, 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:35.655134+0000", 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:35.655134+0000", 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:35.655134+0000", 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5400000000000001e-07 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:35.655137+0000", 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.858e-06 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:36.241838+0000", 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58670063699999997 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:36.241848+0000", 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0524e-05 2024-09-17T16:44:12.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:36.241945+0000", 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6963e-05 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:36.242359+0000", 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00041335800000000001 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(8.11 e850,797 8:8a9fdb5c:::benchmark_data_smithi173_130566_object60988:head,821'5209)", 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:38:37.435017+0000", 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 334.83119015800003, 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.89489246099999997, 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:37.435017+0000", 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:37.435017+0000", 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:37.435018+0000", 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3e-07 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:37.435023+0000", 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1019999999999996e-06 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:38.328802+0000", 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.89377951200000005 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:38.328820+0000", 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7099999999999999e-05 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:38.328942+0000", 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012231199999999999 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:38:38.329910+0000", 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00096770500000000002 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122229 9.d 9:b6455253:::benchmark_data_smithi173_150363_object15324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:44:12.284 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.212600+0000", 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 199.05360736, 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9547618619999998, 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122229 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212600+0000", 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212600+0000", 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212602+0000", 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2520000000000002e-06 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212610+0000", 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9200000000000004e-06 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212846+0000", 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000235935 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.212856+0000", 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0355e-05 2024-09-17T16:44:12.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.155467+0000", 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9426101920000001 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.155546+0000", 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8991000000000003e-05 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.155611+0000", 2024-09-17T16:44:12.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5338000000000004e-05 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163191+0000", 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0075802889999999996 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167191+0000", 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040001250000000002 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167254+0000", 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3126000000000005e-05 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167314+0000", 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9871999999999999e-05 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167362+0000", 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7466999999999998e-05 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.290 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122253 9.d 9:b93e8df4:::benchmark_data_smithi173_150363_object15348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:44:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.259873+0000", 2024-09-17T16:44:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 199.006334772, 2024-09-17T16:44:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9117644189999998, 2024-09-17T16:44:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:12.291 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122253 2024-09-17T16:44:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259873+0000", 2024-09-17T16:44:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259873+0000", 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259874+0000", 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9010000000000001e-06 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259884+0000", 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5370000000000003e-06 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259903+0000", 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8630999999999998e-05 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.259914+0000", 2024-09-17T16:44:12.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1221e-05 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159148+0000", 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8992343539999998 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159272+0000", 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012383199999999999 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:44:12.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159390+0000", 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011834500000000001 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167403+0000", 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0080130819999999995 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167441+0000", 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7360000000000001e-05 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171540+0000", 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040992700000000003 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171594+0000", 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3761000000000002e-05 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171637+0000", 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3124999999999998e-05 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.297 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122272 9.d 9:bbc9765a:::benchmark_data_smithi173_150363_object15367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.299568+0000", 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 198.96663973299999, 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8722430069999998, 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.298 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122272 2024-09-17T16:44:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299568+0000", 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299568+0000", 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299569+0000", 2024-09-17T16:44:12.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6789999999999999e-06 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299576+0000", 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5239999999999997e-06 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299590+0000", 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3925e-05 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.299598+0000", 2024-09-17T16:44:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6170000000000007e-06 2024-09-17T16:44:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163236+0000", 2024-09-17T16:44:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8636377890000002 2024-09-17T16:44:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163356+0000", 2024-09-17T16:44:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000119739 2024-09-17T16:44:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163465+0000", 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000108793 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167475+0000", 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040103140000000001 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171687+0000", 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0042120179999999997 2024-09-17T16:44:12.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171717+0000", 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9887999999999999e-05 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171776+0000", 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9509999999999998e-05 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171811+0000", 2024-09-17T16:44:12.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4211000000000002e-05 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122285 9.d 9:b7f34b49:::benchmark_data_smithi173_150363_object15380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.331869+0000", 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 198.93433817499999, 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8440283700000002, 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:12.306 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122285 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331869+0000", 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331869+0000", 2024-09-17T16:44:12.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331871+0000", 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5540000000000001e-06 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331877+0000", 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1480000000000003e-06 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331891+0000", 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4338999999999999e-05 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.331898+0000", 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2609999999999996e-06 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167505+0000", 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8356064239999998 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167651+0000", 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014661599999999999 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.167776+0000", 2024-09-17T16:44:12.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000124215 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171676+0000", 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0039002640000000001 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175793+0000", 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0041174790000000003 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175805+0000", 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1253999999999999e-05 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175855+0000", 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0195000000000003e-05 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175897+0000", 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2620999999999997e-05 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.312 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27874.0:122289 9.5 e916/911 9:a636923a:::benchmark_data_smithi173_150363_object15384:head v 916'7746, mlcod=913'7744)", 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.359382+0000", 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 198.90682556799999, 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7997341159999998, 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.313 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359382+0000", 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359382+0000", 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359383+0000", 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0109999999999999e-06 2024-09-17T16:44:12.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359391+0000", 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.366010+0000", 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0066186120000000003 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.366018+0000", 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9409999999999996e-06 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.152980+0000", 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7869612779999997 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.153008+0000", 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8198e-05 2024-09-17T16:44:12.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159064+0000", 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0060564929999999996 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159116+0000", 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1403999999999998e-05 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122305 9.d 9:bbff1c34:::benchmark_data_smithi173_150363_object15400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.420875+0000", 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 198.84533247300001, 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7590298520000003, 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122305 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.318 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420875+0000", 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420875+0000", 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420876+0000", 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.449e-06 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420882+0000", 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6110000000000001e-06 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420893+0000", 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1562999999999999e-05 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.420900+0000", 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7179999999999999e-06 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171839+0000", 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7509386400000002 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.171976+0000", 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013730900000000001 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.172054+0000", 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8250999999999993e-05 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175930+0000", 2024-09-17T16:44:12.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0038752579999999999 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179802+0000", 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.003872084 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179813+0000", 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0951999999999999e-05 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179859+0000", 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6289999999999999e-05 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179905+0000", 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5726999999999998e-05 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122326 9.d 9:bd61e234:::benchmark_data_smithi173_150363_object15421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:44:12.324 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.786589+0000", 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 198.47961858400001, 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4114004009999999, 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122326 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786589+0000", 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786589+0000", 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786591+0000", 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7940000000000001e-06 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786598+0000", 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5830000000000001e-06 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786616+0000", 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8210999999999999e-05 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.786626+0000", 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7380000000000006e-06 2024-09-17T16:44:12.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.175957+0000", 2024-09-17T16:44:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3893305729999996 2024-09-17T16:44:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.176075+0000", 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000118678 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.176185+0000", 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011011800000000001 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179935+0000", 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.003749877 2024-09-17T16:44:12.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.192607+0000", 2024-09-17T16:44:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.01267193 2024-09-17T16:44:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.197883+0000", 2024-09-17T16:44:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0052760949999999997 2024-09-17T16:44:12.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.197938+0000", 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4676000000000003e-05 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.197989+0000", 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1128000000000002e-05 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.331 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.27874.0:122329 9.5 e916/911 9:a441955e:::benchmark_data_smithi173_150363_object15424:head v 916'7747, mlcod=913'7744)", 2024-09-17T16:44:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.796651+0000", 2024-09-17T16:44:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 198.46955639000001, 2024-09-17T16:44:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3665418100000002, 2024-09-17T16:44:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:12.332 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796651+0000", 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796651+0000", 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796652+0000", 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2999999999999999e-07 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796659+0000", 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9929999999999998e-06 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.801428+0000", 2024-09-17T16:44:12.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0047693509999999998 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.801435+0000", 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3810000000000001e-06 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.154575+0000", 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3531394639999998 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.154586+0000", 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1387e-05 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163139+0000", 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0085529979999999992 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.163193+0000", 2024-09-17T16:44:12.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3406000000000003e-05 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122341 9.d 9:b2726d38:::benchmark_data_smithi173_150363_object15436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.879538+0000", 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 198.38666926600001, 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3567290740000004, 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:12.337 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122341 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879538+0000", 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879538+0000", 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879540+0000", 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.95e-06 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879548+0000", 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6899999999999992e-06 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879563+0000", 2024-09-17T16:44:12.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5814000000000001e-05 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.879574+0000", 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0594999999999999e-05 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.179959+0000", 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3003848480000002 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.180062+0000", 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000103226 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,5", 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.180161+0000", 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8348000000000001e-05 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.192567+0000", 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.012406188 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.192660+0000", 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3721999999999995e-05 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.236167+0000", 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.043506396000000003 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.236222+0000", 2024-09-17T16:44:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5356000000000003e-05 2024-09-17T16:44:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.236267+0000", 2024-09-17T16:44:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4941000000000002e-05 2024-09-17T16:44:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.343 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122171 9.15 9:aa774fbb:::benchmark_data_smithi173_150363_object15266:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.335919+0000", 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.93028871000001, 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8620733170000001, 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.344 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122171 2024-09-17T16:44:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335919+0000", 2024-09-17T16:44:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335919+0000", 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335920+0000", 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8449999999999999e-06 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335926+0000", 2024-09-17T16:44:12.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5969999999999999e-06 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335937+0000", 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1409e-05 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335944+0000", 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8539999999999999e-06 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.193009+0000", 2024-09-17T16:44:12.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8570646320000002 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.197992+0000", 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0049829799999999997 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122218 9.15 9:ac90a645:::benchmark_data_smithi173_150363_object15313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.335968+0000", 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.930238919, 2024-09-17T16:44:12.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8804516200000001, 2024-09-17T16:44:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.349 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.350 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122218 2024-09-17T16:44:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335968+0000", 2024-09-17T16:44:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335968+0000", 2024-09-17T16:44:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335969+0000", 2024-09-17T16:44:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9000000000000002e-07 2024-09-17T16:44:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335971+0000", 2024-09-17T16:44:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8789999999999999e-06 2024-09-17T16:44:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335978+0000", 2024-09-17T16:44:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8019999999999992e-06 2024-09-17T16:44:12.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335984+0000", 2024-09-17T16:44:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2680000000000002e-06 2024-09-17T16:44:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.198340+0000", 2024-09-17T16:44:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8623560939999999 2024-09-17T16:44:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.216420+0000", 2024-09-17T16:44:12.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.018080187000000001 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122234 9.15 9:ac0fc00a:::benchmark_data_smithi173_150363_object15329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.335994+0000", 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.93021333199999, 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.90027729, 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:12.357 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122234 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335994+0000", 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335994+0000", 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335994+0000", 2024-09-17T16:44:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4999999999999998e-07 2024-09-17T16:44:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.335997+0000", 2024-09-17T16:44:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.441e-06 2024-09-17T16:44:12.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336004+0000", 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2810000000000003e-06 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336008+0000", 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4020000000000002e-06 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.216401+0000", 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.880393312 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.236271+0000", 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.019870404000000001 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122249 9.15 9:ac85c97a:::benchmark_data_smithi173_150363_object15344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:44:12.361 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.336012+0000", 2024-09-17T16:44:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.930194868, 2024-09-17T16:44:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.923836176, 2024-09-17T16:44:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.362 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122249 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336012+0000", 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336012+0000", 2024-09-17T16:44:12.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336013+0000", 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6899999999999999e-07 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336014+0000", 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4810000000000001e-06 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336017+0000", 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9809999999999999e-06 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336021+0000", 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3589999999999999e-06 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.259055+0000", 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.923034589 2024-09-17T16:44:12.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.259849+0000", 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000793497 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122256 9.15 9:a9803287:::benchmark_data_smithi173_150363_object15351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.336030+0000", 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.930177246, 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.924521964, 2024-09-17T16:44:12.366 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122256 2024-09-17T16:44:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336030+0000", 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336030+0000", 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336030+0000", 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6100000000000002e-07 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336032+0000", 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5880000000000001e-06 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336037+0000", 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9690000000000002e-06 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336040+0000", 2024-09-17T16:44:12.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.259881+0000", 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9238411449999999 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.260552+0000", 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00067100099999999995 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122261 9.15 9:aa7ff69f:::benchmark_data_smithi173_150363_object15356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.336044+0000", 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.930163787, 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9253561939999999, 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122261 2024-09-17T16:44:12.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336044+0000", 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336044+0000", 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336044+0000", 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2499999999999999e-07 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336045+0000", 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3060000000000001e-06 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336048+0000", 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9239999999999999e-06 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336051+0000", 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8540000000000001e-06 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.260584+0000", 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9245329839999998 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.261400+0000", 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00081590099999999995 2024-09-17T16:44:12.374 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122284 9.15 9:aff350b8:::benchmark_data_smithi173_150363_object15379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:58.336057+0000", 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 193.930150653, 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9260394750000001, 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.375 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122284 2024-09-17T16:44:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336057+0000", 2024-09-17T16:44:12.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336057+0000", 2024-09-17T16:44:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336057+0000", 2024-09-17T16:44:12.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7599999999999998e-07 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336058+0000", 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.519e-06 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336062+0000", 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0759999999999999e-06 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:58.336064+0000", 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.666e-06 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.261433+0000", 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9253688499999999 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.262096+0000", 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00066308800000000005 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.379 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:213757 9.15 9:a9f3dcd9:::benchmark_data_smithi173_150363_object106852:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e974)", 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:07.891701+0000", 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 64.374506655000005, 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50200806799999997, 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:12.380 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 213757 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.891701+0000", 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.891701+0000", 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:12.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.891702+0000", 2024-09-17T16:44:12.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2810000000000001e-06 2024-09-17T16:44:12.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:12.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.891707+0000", 2024-09-17T16:44:12.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8849999999999998e-06 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.891720+0000", 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2839e-05 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.891726+0000", 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7360000000000004e-06 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:12.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.893857+0000", 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021308009999999999 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.893968+0000", 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011035 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 4,8", 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.894394+0000", 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00042630700000000001 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:12.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.897714+0000", 2024-09-17T16:44:12.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0033200920000000002 2024-09-17T16:44:12.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.354970+0000", 2024-09-17T16:44:12.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45725571399999998 2024-09-17T16:44:12.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:12.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.393624+0000", 2024-09-17T16:44:12.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.038654045999999997 2024-09-17T16:44:12.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:12.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.393668+0000", 2024-09-17T16:44:12.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.464e-05 2024-09-17T16:44:12.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:12.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:12.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:12.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.393709+0000", 2024-09-17T16:44:12.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0377000000000003e-05 2024-09-17T16:44:12.387 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.388 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.388 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.388 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:12.388 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:12.388 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:44:12.400 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":2213,"stamp":"2024-09-17T16:44:10.360834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8363052,"kb_used_data":49692,"kb_used_omap":85,"kb_used_meta":8313002,"kb_avail":1116578772,"statfs":{"total":1151940427776,"available":1143376662528,"internally_reserved":0,"allocated":50884608,"data_stored":44992706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87602,"internal_metadata":8512514510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-120328,"num_write_kb":-2560961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-21164,"num_bytes_recovered":-81068032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.504521"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":833,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-09-17T16:44:09.636347+0000","last_change":"2024-09-17T16:43:52.363146+0000","last_active":"2024-09-17T16:44:09.636347+0000","last_peered":"2024-09-17T16:44:09.636347+0000","last_clean":"2024-09-17T16:44:09.636347+0000","last_became_active":"2024-09-17T16:39:31.793364+0000","last_became_peered":"2024-09-17T16:39:31.793364+0000","last_unstale":"2024-09-17T16:44:09.636347+0000","last_undegraded":"2024-09-17T16:44:09.636347+0000","last_fullsized":"2024-09-17T16:44:09.636347+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:43:52.363103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:43:52.363103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:01.970925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":307,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-09-17T16:44:09.636148+0000","last_change":"2024-09-17T16:42:59.577739+0000","last_active":"2024-09-17T16:44:09.636148+0000","last_peered":"2024-09-17T16:44:09.636148+0000","last_clean":"2024-09-17T16:44:09.636148+0000","last_became_active":"2024-09-17T16:42:59.573697+0000","last_became_peered":"2024-09-17T16:42:59.573697+0000","last_unstale":"2024-09-17T16:44:09.636148+0000","last_undegraded":"2024-09-17T16:44:09.636148+0000","last_fullsized":"2024-09-17T16:44:09.636148+0000","mapping_epoch":968,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":969,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:06.575413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.575413+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:06.547783+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":299,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-09-17T16:44:07.345384+0000","last_change":"2024-09-17T16:43:53.192247+0000","last_active":"2024-09-17T16:44:07.345384+0000","last_peered":"2024-09-17T16:44:07.345384+0000","last_clean":"2024-09-17T16:44:07.345384+0000","last_became_active":"2024-09-17T16:39:33.798293+0000","last_became_peered":"2024-09-17T16:39:33.798293+0000","last_unstale":"2024-09-17T16:44:07.345384+0000","last_undegraded":"2024-09-17T16:44:07.345384+0000","last_fullsized":"2024-09-17T16:44:07.345384+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:43:53.192197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:43:53.192197+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:18.763452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":287,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-09-17T16:44:05.096241+0000","last_change":"2024-09-17T16:42:05.432898+0000","last_active":"2024-09-17T16:44:05.096241+0000","last_peered":"2024-09-17T16:44:05.096241+0000","last_clean":"2024-09-17T16:44:05.096241+0000","last_became_active":"2024-09-17T16:39:33.792174+0000","last_became_peered":"2024-09-17T16:39:33.792174+0000","last_unstale":"2024-09-17T16:44:05.096241+0000","last_undegraded":"2024-09-17T16:44:05.096241+0000","last_fullsized":"2024-09-17T16:44:05.096241+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:05.432822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.432822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:01.564804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":286,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-09-17T16:44:05.652859+0000","last_change":"2024-09-17T16:42:10.506218+0000","last_active":"2024-09-17T16:44:05.652859+0000","last_peered":"2024-09-17T16:44:05.652859+0000","last_clean":"2024-09-17T16:44:05.652859+0000","last_became_active":"2024-09-17T16:39:33.801025+0000","last_became_peered":"2024-09-17T16:39:33.801025+0000","last_unstale":"2024-09-17T16:44:05.652859+0000","last_undegraded":"2024-09-17T16:44:05.652859+0000","last_fullsized":"2024-09-17T16:44:05.652859+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:10.506170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:10.506170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:41.773388+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":287,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-09-17T16:44:05.653127+0000","last_change":"2024-09-17T16:42:07.852349+0000","last_active":"2024-09-17T16:44:05.653127+0000","last_peered":"2024-09-17T16:44:05.653127+0000","last_clean":"2024-09-17T16:44:05.653127+0000","last_became_active":"2024-09-17T16:39:33.790219+0000","last_became_peered":"2024-09-17T16:39:33.790219+0000","last_unstale":"2024-09-17T16:44:05.653127+0000","last_undegraded":"2024-09-17T16:44:05.653127+0000","last_fullsized":"2024-09-17T16:44:05.653127+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:07.852278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:07.852278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:41.791063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":290,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-09-17T16:44:07.516894+0000","last_change":"2024-09-17T16:42:06.400060+0000","last_active":"2024-09-17T16:44:07.516894+0000","last_peered":"2024-09-17T16:44:07.516894+0000","last_clean":"2024-09-17T16:44:07.516894+0000","last_became_active":"2024-09-17T16:39:33.824237+0000","last_became_peered":"2024-09-17T16:39:33.824237+0000","last_unstale":"2024-09-17T16:44:07.516894+0000","last_undegraded":"2024-09-17T16:44:07.516894+0000","last_fullsized":"2024-09-17T16:44:07.516894+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:06.399992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.399992+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:39.419505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":307,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-09-17T16:44:08.357972+0000","last_change":"2024-09-17T16:43:30.878832+0000","last_active":"2024-09-17T16:44:08.357972+0000","last_peered":"2024-09-17T16:44:08.357972+0000","last_clean":"2024-09-17T16:44:08.357972+0000","last_became_active":"2024-09-17T16:39:33.795955+0000","last_became_peered":"2024-09-17T16:39:33.795955+0000","last_unstale":"2024-09-17T16:44:08.357972+0000","last_undegraded":"2024-09-17T16:44:08.357972+0000","last_fullsized":"2024-09-17T16:44:08.357972+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:43:30.878762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:43:30.878762+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:41.770045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":287,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-09-17T16:44:05.652928+0000","last_change":"2024-09-17T16:42:09.395309+0000","last_active":"2024-09-17T16:44:05.652928+0000","last_peered":"2024-09-17T16:44:05.652928+0000","last_clean":"2024-09-17T16:44:05.652928+0000","last_became_active":"2024-09-17T16:39:33.800521+0000","last_became_peered":"2024-09-17T16:39:33.800521+0000","last_unstale":"2024-09-17T16:44:05.652928+0000","last_undegraded":"2024-09-17T16:44:05.652928+0000","last_fullsized":"2024-09-17T16:44:05.652928+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:09.395264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:09.395264+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:36.728753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":295,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-09-17T16:44:05.653013+0000","last_change":"2024-09-17T16:43:31.659048+0000","last_active":"2024-09-17T16:44:05.653013+0000","last_peered":"2024-09-17T16:44:05.653013+0000","last_clean":"2024-09-17T16:44:05.653013+0000","last_became_active":"2024-09-17T16:39:33.791607+0000","last_became_peered":"2024-09-17T16:39:33.791607+0000","last_unstale":"2024-09-17T16:44:05.653013+0000","last_undegraded":"2024-09-17T16:44:05.653013+0000","last_fullsized":"2024-09-17T16:44:05.653013+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:43:31.658992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:43:31.658992+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:49.267558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":320,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-09-17T16:44:05.653456+0000","last_change":"2024-09-17T16:42:59.574526+0000","last_active":"2024-09-17T16:44:05.653456+0000","last_peered":"2024-09-17T16:44:05.653456+0000","last_clean":"2024-09-17T16:44:05.653456+0000","last_became_active":"2024-09-17T16:42:59.573860+0000","last_became_peered":"2024-09-17T16:42:59.573860+0000","last_unstale":"2024-09-17T16:44:05.653456+0000","last_undegraded":"2024-09-17T16:44:05.653456+0000","last_fullsized":"2024-09-17T16:44:05.653456+0000","mapping_epoch":968,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":969,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:05.413835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.413835+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:04.246763+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285985,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":753212,"kb_used_data":13992,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92991940,"statfs":{"total":95995035648,"available":95223746560,"internally_reserved":0,"allocated":14327808,"data_stored":13835075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":756939204},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131227,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":363184,"kb_used_data":1308,"kb_used_omap":15,"kb_used_meta":361840,"kb_avail":93381968,"statfs":{"total":95995035648,"available":95623135232,"internally_reserved":0,"allocated":1339392,"data_stored":851108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":370524732},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743849,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965432,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":964094,"kb_avail":92779720,"statfs":{"total":95995035648,"available":95006433280,"internally_reserved":0,"allocated":1339392,"data_stored":851108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":987232704},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005488,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":664528,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93080624,"statfs":{"total":95995035648,"available":95314558976,"internally_reserved":0,"allocated":1695744,"data_stored":1194504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963962,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866416,"kb_used_data":1628,"kb_used_omap":41,"kb_used_meta":864726,"kb_avail":92878736,"statfs":{"total":95995035648,"available":95107825664,"internally_reserved":0,"allocated":1667072,"data_stored":1178788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":885479731},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496060,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":678628,"kb_used_data":21328,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":93066524,"statfs":{"total":95995035648,"available":95300120576,"internally_reserved":0,"allocated":21839872,"data_stored":21356045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":673053126},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967705,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":286664,"kb_used_data":1520,"kb_used_omap":14,"kb_used_meta":285105,"kb_avail":93458488,"statfs":{"total":95995035648,"available":95701491712,"internally_reserved":0,"allocated":1556480,"data_stored":1060195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":291948149},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986629,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696124,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93049028,"statfs":{"total":95995035648,"available":95282204672,"internally_reserved":0,"allocated":1351680,"data_stored":858947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150150,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826804,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92918348,"statfs":{"total":95995035648,"available":95148388352,"internally_reserved":0,"allocated":1531904,"data_stored":1044525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":845085127},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411788,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":664384,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93080768,"statfs":{"total":95995035648,"available":95314706432,"internally_reserved":0,"allocated":1544192,"data_stored":1052356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575314,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726652,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93018500,"statfs":{"total":95995035648,"available":95250944000,"internally_reserved":0,"allocated":1351680,"data_stored":858947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738839,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871024,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92874128,"statfs":{"total":95995035648,"available":95103107072,"internally_reserved":0,"allocated":1339392,"data_stored":851108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":890567109},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:44:12.402 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:44:12.536 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:12 smithi071 ceph-mon[30002]: from='client.19149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:44:12.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:12 smithi173 ceph-mon[30530]: from='client.19149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:44:12.738 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:12 smithi017 ceph-mon[27571]: from='client.19149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:44:13.130 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:13.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-17T16:44:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:13 smithi071 ceph-mon[30002]: pgmap v2214: 11 pgs: 11 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:13.585 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:44:13.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:13 smithi173 ceph-mon[30530]: pgmap v2214: 11 pgs: 11 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:13.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:13 smithi017 ceph-mon[27571]: pgmap v2214: 11 pgs: 11 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:44:13.799 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:44:14.116 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:44:14.116 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:44:14.117 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-09-17T16:44:14.117 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:44:14.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:14 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:14.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:14 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:14.626 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:14 smithi017 ceph-mon[27571]: osdmap e1014: 12 total, 12 up, 12 in 2024-09-17T16:44:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:14 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:14 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:14.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:14 smithi173 ceph-mon[30530]: osdmap e1014: 12 total, 12 up, 12 in 2024-09-17T16:44:14.654 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":2215,"stamp":"2024-09-17T16:44:13.363071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8339032,"kb_used_data":42320,"kb_used_omap":85,"kb_used_meta":8296362,"kb_avail":1116602792,"statfs":{"total":1151940427776,"available":1143401259008,"internally_reserved":0,"allocated":43335680,"data_stored":37306559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87602,"internal_metadata":8495475150},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":74,"apply_latency_ms":74,"commit_latency_ns":74000000,"apply_latency_ns":74000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-120328,"num_write_kb":-2560961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-21164,"num_bytes_recovered":-81068032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.019278"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":833,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-09-17T16:44:09.636347+0000","last_change":"2024-09-17T16:43:52.363146+0000","last_active":"2024-09-17T16:44:09.636347+0000","last_peered":"2024-09-17T16:44:09.636347+0000","last_clean":"2024-09-17T16:44:09.636347+0000","last_became_active":"2024-09-17T16:39:31.793364+0000","last_became_peered":"2024-09-17T16:39:31.793364+0000","last_unstale":"2024-09-17T16:44:09.636347+0000","last_undegraded":"2024-09-17T16:44:09.636347+0000","last_fullsized":"2024-09-17T16:44:09.636347+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:43:52.363103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:43:52.363103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:01.970925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":307,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-09-17T16:44:09.636148+0000","last_change":"2024-09-17T16:42:59.577739+0000","last_active":"2024-09-17T16:44:09.636148+0000","last_peered":"2024-09-17T16:44:09.636148+0000","last_clean":"2024-09-17T16:44:09.636148+0000","last_became_active":"2024-09-17T16:42:59.573697+0000","last_became_peered":"2024-09-17T16:42:59.573697+0000","last_unstale":"2024-09-17T16:44:09.636148+0000","last_undegraded":"2024-09-17T16:44:09.636148+0000","last_fullsized":"2024-09-17T16:44:09.636148+0000","mapping_epoch":968,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":969,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:06.575413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.575413+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:06.547783+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":303,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-09-17T16:44:10.953679+0000","last_change":"2024-09-17T16:43:53.192247+0000","last_active":"2024-09-17T16:44:10.953679+0000","last_peered":"2024-09-17T16:44:10.953679+0000","last_clean":"2024-09-17T16:44:10.953679+0000","last_became_active":"2024-09-17T16:39:33.798293+0000","last_became_peered":"2024-09-17T16:39:33.798293+0000","last_unstale":"2024-09-17T16:44:10.953679+0000","last_undegraded":"2024-09-17T16:44:10.953679+0000","last_fullsized":"2024-09-17T16:44:10.953679+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:43:53.192197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:43:53.192197+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:18.763452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":293,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-09-17T16:44:10.953567+0000","last_change":"2024-09-17T16:42:05.432898+0000","last_active":"2024-09-17T16:44:10.953567+0000","last_peered":"2024-09-17T16:44:10.953567+0000","last_clean":"2024-09-17T16:44:10.953567+0000","last_became_active":"2024-09-17T16:39:33.792174+0000","last_became_peered":"2024-09-17T16:39:33.792174+0000","last_unstale":"2024-09-17T16:44:10.953567+0000","last_undegraded":"2024-09-17T16:44:10.953567+0000","last_fullsized":"2024-09-17T16:44:10.953567+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:05.432822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.432822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:01.564804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":292,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-09-17T16:44:10.953528+0000","last_change":"2024-09-17T16:42:10.506218+0000","last_active":"2024-09-17T16:44:10.953528+0000","last_peered":"2024-09-17T16:44:10.953528+0000","last_clean":"2024-09-17T16:44:10.953528+0000","last_became_active":"2024-09-17T16:39:33.801025+0000","last_became_peered":"2024-09-17T16:39:33.801025+0000","last_unstale":"2024-09-17T16:44:10.953528+0000","last_undegraded":"2024-09-17T16:44:10.953528+0000","last_fullsized":"2024-09-17T16:44:10.953528+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:10.506170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:10.506170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:41.773388+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":293,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-09-17T16:44:10.953624+0000","last_change":"2024-09-17T16:42:07.852349+0000","last_active":"2024-09-17T16:44:10.953624+0000","last_peered":"2024-09-17T16:44:10.953624+0000","last_clean":"2024-09-17T16:44:10.953624+0000","last_became_active":"2024-09-17T16:39:33.790219+0000","last_became_peered":"2024-09-17T16:39:33.790219+0000","last_unstale":"2024-09-17T16:44:10.953624+0000","last_undegraded":"2024-09-17T16:44:10.953624+0000","last_fullsized":"2024-09-17T16:44:10.953624+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:07.852278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:07.852278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:41.791063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":294,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-09-17T16:44:10.953633+0000","last_change":"2024-09-17T16:42:06.400060+0000","last_active":"2024-09-17T16:44:10.953633+0000","last_peered":"2024-09-17T16:44:10.953633+0000","last_clean":"2024-09-17T16:44:10.953633+0000","last_became_active":"2024-09-17T16:39:33.824237+0000","last_became_peered":"2024-09-17T16:39:33.824237+0000","last_unstale":"2024-09-17T16:44:10.953633+0000","last_undegraded":"2024-09-17T16:44:10.953633+0000","last_fullsized":"2024-09-17T16:44:10.953633+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:06.399992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:06.399992+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:39.419505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":307,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-09-17T16:44:08.357972+0000","last_change":"2024-09-17T16:43:30.878832+0000","last_active":"2024-09-17T16:44:08.357972+0000","last_peered":"2024-09-17T16:44:08.357972+0000","last_clean":"2024-09-17T16:44:08.357972+0000","last_became_active":"2024-09-17T16:39:33.795955+0000","last_became_peered":"2024-09-17T16:39:33.795955+0000","last_unstale":"2024-09-17T16:44:08.357972+0000","last_undegraded":"2024-09-17T16:44:08.357972+0000","last_fullsized":"2024-09-17T16:44:08.357972+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:43:30.878762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:43:30.878762+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:41.770045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":293,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-09-17T16:44:10.953528+0000","last_change":"2024-09-17T16:42:09.395309+0000","last_active":"2024-09-17T16:44:10.953528+0000","last_peered":"2024-09-17T16:44:10.953528+0000","last_clean":"2024-09-17T16:44:10.953528+0000","last_became_active":"2024-09-17T16:39:33.800521+0000","last_became_peered":"2024-09-17T16:39:33.800521+0000","last_unstale":"2024-09-17T16:44:10.953528+0000","last_undegraded":"2024-09-17T16:44:10.953528+0000","last_fullsized":"2024-09-17T16:44:10.953528+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:09.395264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:09.395264+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:36.728753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":301,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-09-17T16:44:10.953567+0000","last_change":"2024-09-17T16:43:31.659048+0000","last_active":"2024-09-17T16:44:10.953567+0000","last_peered":"2024-09-17T16:44:10.953567+0000","last_clean":"2024-09-17T16:44:10.953567+0000","last_became_active":"2024-09-17T16:39:33.791607+0000","last_became_peered":"2024-09-17T16:39:33.791607+0000","last_unstale":"2024-09-17T16:44:10.953567+0000","last_undegraded":"2024-09-17T16:44:10.953567+0000","last_fullsized":"2024-09-17T16:44:10.953567+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:43:31.658992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:43:31.658992+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:44:49.267558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":326,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-09-17T16:44:10.502040+0000","last_change":"2024-09-17T16:42:59.574526+0000","last_active":"2024-09-17T16:44:10.502040+0000","last_peered":"2024-09-17T16:44:10.502040+0000","last_clean":"2024-09-17T16:44:10.502040+0000","last_became_active":"2024-09-17T16:42:59.573860+0000","last_became_peered":"2024-09-17T16:42:59.573860+0000","last_unstale":"2024-09-17T16:44:10.502040+0000","last_undegraded":"2024-09-17T16:44:10.502040+0000","last_fullsized":"2024-09-17T16:44:10.502040+0000","mapping_epoch":968,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":969,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:42:05.413835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:42:05.413835+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:04.246763+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482285985,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":753212,"kb_used_data":13992,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92991940,"statfs":{"total":95995035648,"available":95223746560,"internally_reserved":0,"allocated":14327808,"data_stored":13835075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":756939204},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131228,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":363228,"kb_used_data":1348,"kb_used_omap":15,"kb_used_meta":361840,"kb_avail":93381924,"statfs":{"total":95995035648,"available":95623090176,"internally_reserved":0,"allocated":1380352,"data_stored":874261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":370524732},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743850,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965468,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":964094,"kb_avail":92779684,"statfs":{"total":95995035648,"available":95006396416,"internally_reserved":0,"allocated":1380352,"data_stored":874261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":987232704},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005488,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":664528,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93080624,"statfs":{"total":95995035648,"available":95314558976,"internally_reserved":0,"allocated":1695744,"data_stored":1194504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963963,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866456,"kb_used_data":1668,"kb_used_omap":41,"kb_used_meta":864726,"kb_avail":92878696,"statfs":{"total":95995035648,"available":95107784704,"internally_reserved":0,"allocated":1708032,"data_stored":1201941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":885479731},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496061,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654352,"kb_used_data":13688,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93090800,"statfs":{"total":95995035648,"available":95324979200,"internally_reserved":0,"allocated":14016512,"data_stored":13515272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":656013766},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967705,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":286664,"kb_used_data":1520,"kb_used_omap":14,"kb_used_meta":285105,"kb_avail":93458488,"statfs":{"total":95995035648,"available":95701491712,"internally_reserved":0,"allocated":1556480,"data_stored":1060195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":291948149},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986629,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696124,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93049028,"statfs":{"total":95995035648,"available":95282204672,"internally_reserved":0,"allocated":1351680,"data_stored":858947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826836,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92918316,"statfs":{"total":95995035648,"available":95148355584,"internally_reserved":0,"allocated":1572864,"data_stored":1068072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":845085127},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411789,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":664420,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93080732,"statfs":{"total":95995035648,"available":95314669568,"internally_reserved":0,"allocated":1585152,"data_stored":1075509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575315,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726680,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93018472,"statfs":{"total":95995035648,"available":95250915328,"internally_reserved":0,"allocated":1380352,"data_stored":874261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738840,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871064,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92874088,"statfs":{"total":95995035648,"available":95103066112,"internally_reserved":0,"allocated":1380352,"data_stored":874261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":890567109},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:44:14.655 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T16:44:14.656 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:44:14.656 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-09-17T16:44:14.656 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-17T16:44:14.656 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-17T16:44:14.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-09-17T16:44:14.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-17T16:44:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:14 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:14 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:14.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:14 smithi071 ceph-mon[30002]: osdmap e1014: 12 total, 12 up, 12 in 2024-09-17T16:44:14.915 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:44:15.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-17T16:44:15.367 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:15.632 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:15 smithi017 ceph-mon[27571]: from='client.19155 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:44:15.632 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:15 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:15.632 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:15 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:15.632 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:15 smithi017 ceph-mon[27571]: pgmap v2216: 11 pgs: 11 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:15.632 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:15 smithi017 ceph-mon[27571]: osdmap e1015: 12 total, 12 up, 12 in 2024-09-17T16:44:15.632 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:15 smithi017 ceph-mon[27571]: osdmap e1016: 12 total, 12 up, 12 in 2024-09-17T16:44:15.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:15 smithi173 ceph-mon[30530]: from='client.19155 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:44:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:15 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:15 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:15 smithi173 ceph-mon[30530]: pgmap v2216: 11 pgs: 11 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:15 smithi173 ceph-mon[30530]: osdmap e1015: 12 total, 12 up, 12 in 2024-09-17T16:44:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:15 smithi173 ceph-mon[30530]: osdmap e1016: 12 total, 12 up, 12 in 2024-09-17T16:44:15.658 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:44:15.658 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-17T16:44:15.658 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:44:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:15 smithi071 ceph-mon[30002]: from='client.19155 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:44:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:15 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:15 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:15 smithi071 ceph-mon[30002]: pgmap v2216: 11 pgs: 11 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:15 smithi071 ceph-mon[30002]: osdmap e1015: 12 total, 12 up, 12 in 2024-09-17T16:44:15.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:15 smithi071 ceph-mon[30002]: osdmap e1016: 12 total, 12 up, 12 in 2024-09-17T16:44:16.382 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-17T16:44:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/800520684' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:44:16.741 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/800520684' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:44:16.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/800520684' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-17T16:44:17.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-17T16:44:17.412 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:44:17.474 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:17.488 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:17 smithi017 ceph-mon[27571]: pgmap v2219: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/800520684' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:44:17.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:17 smithi017 ceph-mon[27571]: osdmap e1017: 12 total, 12 up, 12 in 2024-09-17T16:44:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:17 smithi173 ceph-mon[30530]: pgmap v2219: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:17.665 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/800520684' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:44:17.665 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:17 smithi173 ceph-mon[30530]: osdmap e1017: 12 total, 12 up, 12 in 2024-09-17T16:44:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:17 smithi071 ceph-mon[30002]: pgmap v2219: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/800520684' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-17T16:44:17.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:17 smithi071 ceph-mon[30002]: osdmap e1017: 12 total, 12 up, 12 in 2024-09-17T16:44:18.395 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-17T16:44:18.635 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:18 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:18.635 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:18 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:18.635 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3148323032' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:44:18.655 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:44:18.655 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-09-17T16:44:18.655 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-09-17T16:44:18.655 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-09-17T16:44:18.655 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:44:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:18 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:18 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:18.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3148323032' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:44:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:18 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:18 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:18.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3148323032' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-17T16:44:19.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-17T16:44:19.586 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.c/config 2024-09-17T16:44:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:19 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:19 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:19 smithi173 ceph-mon[30530]: pgmap v2221: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:19 smithi173 ceph-mon[30530]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:44:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:19 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:44:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:19 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3148323032' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:44:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:19 smithi173 ceph-mon[30530]: osdmap e1018: 12 total, 12 up, 12 in 2024-09-17T16:44:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:19 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:19 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:19.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:19 smithi017 ceph-mon[27571]: pgmap v2221: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:19 smithi017 ceph-mon[27571]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:44:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:19 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:44:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:19 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3148323032' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:44:19.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:19 smithi017 ceph-mon[27571]: osdmap e1018: 12 total, 12 up, 12 in 2024-09-17T16:44:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:19 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:19 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:19 smithi071 ceph-mon[30002]: pgmap v2221: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:19 smithi071 ceph-mon[30002]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-17T16:44:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:19 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:44:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:19 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3148323032' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-17T16:44:19.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:19 smithi071 ceph-mon[30002]: osdmap e1018: 12 total, 12 up, 12 in 2024-09-17T16:44:20.092 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-09-17T16:44:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-09-17T16:44:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-09-17T16:44:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-09-17T16:44:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104264 8.4 8.cdf0f904 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958458+0000", 2024-09-17T16:44:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13061887100002, 2024-09-17T16:44:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76365417400000002, 2024-09-17T16:44:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104264 2024-09-17T16:44:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958458+0000", 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958458+0000", 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958460+0000", 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1150000000000001e-06 2024-09-17T16:44:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958467+0000", 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.776e-06 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958476+0000", 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3300000000000005e-06 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958485+0000", 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5250000000000005e-06 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958553+0000", 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8027000000000004e-05 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958563+0000", 2024-09-17T16:44:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210999999999999e-05 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722103+0000", 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76353991499999996 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722112+0000", 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2750000000000001e-06 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104266 8.4 8.df640d24 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958502+0000", 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13057444200001, 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.763704465, 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104266 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958502+0000", 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958502+0000", 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958503+0000", 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3500000000000002e-07 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958505+0000", 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.446e-06 2024-09-17T16:44:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958510+0000", 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3950000000000003e-06 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958513+0000", 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0790000000000001e-06 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958585+0000", 2024-09-17T16:44:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2260000000000003e-05 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2109999999999997e-06 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722199+0000", 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360864399999995 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722207+0000", 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0949999999999996e-06 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104310 8.4 8.3aa61324 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13054073199999, 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76370166900000003, 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104310 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:44:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958536+0000", 2024-09-17T16:44:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1899999999999998e-07 2024-09-17T16:44:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958538+0000", 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.818e-06 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958545+0000", 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4300000000000003e-06 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958548+0000", 2024-09-17T16:44:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8279999999999996e-06 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958613+0000", 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4791999999999999e-05 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958620+0000", 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6980000000000001e-06 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722230+0000", 2024-09-17T16:44:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76361024600000005 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722238+0000", 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5379999999999997e-06 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104365 8.4 8.98b36004 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13051779199998, 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76371162999999997, 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104365 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958559+0000", 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1399999999999998e-07 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958561+0000", 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.793e-06 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958565+0000", 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4790000000000001e-06 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958568+0000", 2024-09-17T16:44:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9409999999999999e-06 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958647+0000", 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9058000000000001e-05 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958650+0000", 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.422e-06 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722263+0000", 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76361347499999999 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722271+0000", 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.148e-06 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104510 8.4 8.71f883e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13049878200002, 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76372115600000001, 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104510 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958578+0000", 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0899999999999997e-07 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958580+0000", 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3149999999999999e-06 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958583+0000", 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7900000000000001e-06 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958586+0000", 2024-09-17T16:44:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5639999999999999e-06 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958683+0000", 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6762999999999995e-05 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958689+0000", 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2349999999999996e-06 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722292+0000", 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360352600000003 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722299+0000", 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6540000000000002e-06 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104538 8.4 8.69da0a4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13048657899998, 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76373857899999997, 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104538 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958590+0000", 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.29e-07 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958592+0000", 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4929999999999999e-06 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958595+0000", 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0369999999999999e-06 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958598+0000", 2024-09-17T16:44:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.932e-06 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958715+0000", 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011665 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958718+0000", 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.591e-06 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722322+0000", 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360369100000003 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722329+0000", 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.956e-06 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104542 8.4 8.5948f164 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13045795699998, 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76374053200000003, 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104542 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958619+0000", 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8900000000000001e-07 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958621+0000", 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.978e-06 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958625+0000", 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4599999999999999e-06 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958627+0000", 2024-09-17T16:44:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7240000000000001e-06 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958747+0000", 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012020599999999999 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958753+0000", 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9800000000000003e-06 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722352+0000", 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.763599005 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722359+0000", 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8900000000000001e-06 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104571 8.4 8.f7b7a3a4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.130434014, 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76374966600000005, 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104571 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958643+0000", 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9999999999999999e-07 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958644+0000", 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2529999999999999e-06 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958648+0000", 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3620000000000002e-06 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958650+0000", 2024-09-17T16:44:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.514e-06 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958778+0000", 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012807300000000001 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958781+0000", 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.039e-06 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722385+0000", 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360329100000002 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722392+0000", 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8339999999999999e-06 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104651 8.4 8.2da1dbe4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:44:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13041527399997, 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76376209500000003, 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104651 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:44:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958662+0000", 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8999999999999998e-07 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958663+0000", 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2759999999999999e-06 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958667+0000", 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2880000000000003e-06 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958670+0000", 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4650000000000001e-06 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958811+0000", 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014136500000000001 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958817+0000", 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1160000000000004e-06 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722416+0000", 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76359866899999995 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722424+0000", 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6260000000000003e-06 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104652 8.4 8.77678704 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958673+0000", 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13040335800002, 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76378596899999995, 2024-09-17T16:44:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104652 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958673+0000", 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958673+0000", 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958674+0000", 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.28e-07 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958675+0000", 2024-09-17T16:44:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.141e-06 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958678+0000", 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0010000000000002e-06 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958682+0000", 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2869999999999998e-06 2024-09-17T16:44:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958843+0000", 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001607 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958846+0000", 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0419999999999998e-06 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722449+0000", 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360327400000005 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722459+0000", 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0295999999999999e-05 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104673 8.4 8.63bb1ee4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958685+0000", 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13039146099999, 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76381003599999997, 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104673 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958685+0000", 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958685+0000", 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958686+0000", 2024-09-17T16:44:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2600000000000001e-07 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958687+0000", 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2580000000000001e-06 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958690+0000", 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9909999999999998e-06 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958692+0000", 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4360000000000001e-06 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958875+0000", 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000182327 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958881+0000", 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0410000000000003e-06 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722488+0000", 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360697899999996 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722495+0000", 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7780000000000007e-06 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104674 8.4 8.5e810684 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13036575899997, 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76382820500000004, 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104674 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958711+0000", 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9299999999999999e-07 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958715+0000", 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7869999999999999e-06 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958719+0000", 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5980000000000001e-06 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958721+0000", 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6589999999999999e-06 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958920+0000", 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000198789 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958924+0000", 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4379999999999999e-06 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722530+0000", 2024-09-17T16:44:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360632500000003 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722539+0000", 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3160000000000003e-06 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104695 8.4 8.bc0ebd64 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:44:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13035169099999, 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.763851155, 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104695 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958725+0000", 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4400000000000001e-07 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958727+0000", 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.657e-06 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958730+0000", 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8480000000000001e-06 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958732+0000", 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4200000000000001e-06 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958955+0000", 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022305200000000001 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958962+0000", 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0369999999999997e-06 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722567+0000", 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76360444599999999 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722576+0000", 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104721 8.4 8.83410e4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13033589200001, 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76388141899999995, 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104721 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958741+0000", 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0199999999999998e-07 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958742+0000", 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.26e-06 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958746+0000", 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8319999999999999e-06 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958749+0000", 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7980000000000001e-06 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958987+0000", 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023819600000000001 2024-09-17T16:44:20.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958990+0000", 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0110000000000001e-06 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722614+0000", 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76362420200000003 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722622+0000", 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8180000000000004e-06 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104726 8.4 8.bcfb8bc4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958752+0000", 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.130324336, 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76390264500000005, 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.165 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104726 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958752+0000", 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958752+0000", 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958753+0000", 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.29e-07 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958754+0000", 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3799999999999996e-07 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958756+0000", 2024-09-17T16:44:20.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8329999999999998e-06 2024-09-17T16:44:20.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958759+0000", 2024-09-17T16:44:20.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7049999999999999e-06 2024-09-17T16:44:20.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959018+0000", 2024-09-17T16:44:20.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025930899999999999 2024-09-17T16:44:20.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959025+0000", 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1990000000000003e-06 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722647+0000", 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76362246600000006 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722655+0000", 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0660000000000004e-06 2024-09-17T16:44:20.170 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.17883.0:104729 8.4 8.1aa6e3c4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e791)", 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 461.13030170600001, 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76391769399999998, 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.17883", 2024-09-17T16:44:20.171 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 104729 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958775+0000", 2024-09-17T16:44:20.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9499999999999998e-07 2024-09-17T16:44:20.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958777+0000", 2024-09-17T16:44:20.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.184e-06 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958780+0000", 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7440000000000001e-06 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.958783+0000", 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0910000000000001e-06 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959039+0000", 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000255133 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:38.959042+0000", 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6049999999999998e-06 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722686+0000", 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76364342399999996 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:36:39.722693+0000", 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2180000000000002e-06 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27874.0:8124 9.7 9.3fd06627 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e865)", 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:39:08.294658+0000", 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 311.79441894399997, 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91228819400000005, 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.176 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27874", 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 8124 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294658+0000", 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294658+0000", 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294659+0000", 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9999999999999995e-07 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294709+0000", 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0549e-05 2024-09-17T16:44:20.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294720+0000", 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0678e-05 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294725+0000", 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0039999999999999e-06 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294775+0000", 2024-09-17T16:44:20.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9611000000000003e-05 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:08.294782+0000", 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.844e-06 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.206917+0000", 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91213532399999997 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:39:09.206946+0000", 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9184e-05 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(9.7 e982,969 9:e431e063:::benchmark_data_smithi173_150363_object78586:head,955'9551)", 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:43:20.112269+0000", 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 59.976807508999997, 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94037968999999999, 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.181 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:20.112269+0000", 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:20.112269+0000", 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:20.112271+0000", 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.764e-06 2024-09-17T16:44:20.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:20.112283+0000", 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.219e-05 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:21.052013+0000", 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93972984900000001 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:21.052037+0000", 2024-09-17T16:44:20.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3808999999999999e-05 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:21.052213+0000", 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017588700000000001 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:21.052649+0000", 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000436191 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.184 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(9.7 e982,969 9:e739e5df:::benchmark_data_smithi173_150363_object79842:head,956'9605)", 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:43:21.260968+0000", 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.828108528999998, 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95308890800000001, 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:21.260968+0000", 2024-09-17T16:44:20.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:21.260968+0000", 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:21.260969+0000", 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6499999999999999e-07 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:21.260972+0000", 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.501e-06 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:22.213449+0000", 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95247672100000003 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:22.213474+0000", 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4663000000000001e-05 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:22.213605+0000", 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013116700000000001 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:22.214057+0000", 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045229100000000001 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(9.7 e983,969 9:e1563ce2:::benchmark_data_smithi173_150363_object90688:head,962'9941)", 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-09-17T16:43:24.628870+0000", 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 55.460206784999997, 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70834767600000004, 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:24.628870+0000", 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:24.628870+0000", 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:24.628871+0000", 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9400000000000004e-07 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-09-17T16:44:20.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:24.628881+0000", 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7559999999999994e-06 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:25.336780+0000", 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70789961999999995 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:25.336787+0000", 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1740000000000003e-06 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:25.336870+0000", 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2958000000000001e-05 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-09-17T16:43:25.337218+0000", 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034727400000000001 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-09-17T16:44:20.192 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-09-17T16:44:20.906 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-17T16:44:21.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:20 smithi173 ceph-mon[30530]: pgmap v2223: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:21.152 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:44:21.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:20 smithi017 ceph-mon[27571]: pgmap v2223: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:21.273 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:20 smithi071 ceph-mon[30002]: pgmap v2223: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:21.588 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:44:21.588 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:44:21.588 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-09-17T16:44:21.588 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:44:22.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-17T16:44:22.598 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:44:23.050 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:44:23.051 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-09-17T16:44:23.051 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-09-17T16:44:23.051 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-09-17T16:44:23.051 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:44:23.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:23 smithi071 ceph-mon[30002]: pgmap v2224: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:23.352 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-17T16:44:23.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-17T16:44:23.622 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:23 smithi173 ceph-mon[30530]: pgmap v2224: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:23.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:23 smithi017 ceph-mon[27571]: pgmap v2224: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:23.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-17T16:44:24.081 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.b/config 2024-09-17T16:44:24.592 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-09-17T16:44:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-09-17T16:44:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-09-17T16:44:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-09-17T16:44:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.17883.0:157496 8.4 8:2002af7a:::benchmark_data_smithi173_130566_object62521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e822)", 2024-09-17T16:44:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:48.711848+0000", 2024-09-17T16:44:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 395.87643135000002, 2024-09-17T16:44:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52764325700000003, 2024-09-17T16:44:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.593 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.17883", 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/2740350064", 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 157496 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711848+0000", 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711848+0000", 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711849+0000", 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1149999999999999e-06 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711853+0000", 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1370000000000004e-06 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711864+0000", 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0913e-05 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711870+0000", 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6130000000000003e-06 2024-09-17T16:44:24.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.711918+0000", 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8829999999999998e-05 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.712024+0000", 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010582999999999999 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 9,10", 2024-09-17T16:44:24.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.712119+0000", 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4654000000000005e-05 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.713620+0000", 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00150146 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714929+0000", 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013090560000000001 2024-09-17T16:44:24.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239387+0000", 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52445777400000004 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239445+0000", 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7960000000000001e-05 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239491+0000", 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5914999999999997e-05 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.17883.0:157496 8.4 e822/810 ondisk, result = 0)", 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:37:48.714185+0000", 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 395.87409412199997, 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52531144100000005, 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.600 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714185+0000", 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714185+0000", 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714186+0000", 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8699999999999995e-07 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:48.714192+0000", 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0730000000000001e-06 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239260+0000", 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.525067909 2024-09-17T16:44:24.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239272+0000", 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2527000000000001e-05 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239366+0000", 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4143999999999999e-05 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239376+0000", 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7620000000000007e-06 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:37:49.239496+0000", 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120439 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:8192 9.f 9:f572e93a:::benchmark_data_smithi173_150363_object8191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e867)", 2024-09-17T16:44:24.604 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.811494+0000", 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 314.77678543299999, 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52189015699999997, 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 8192 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811494+0000", 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811494+0000", 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811494+0000", 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5799999999999998e-07 2024-09-17T16:44:24.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811560+0000", 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5425999999999993e-05 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811569+0000", 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8629999999999997e-06 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811574+0000", 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0300000000000001e-06 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811621+0000", 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7695999999999999e-05 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811739+0000", 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011728899999999999 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,9", 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.811879+0000", 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014007000000000001 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.812390+0000", 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00051132900000000004 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.812862+0000", 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00047182799999999998 2024-09-17T16:44:24.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333287+0000", 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52042468099999994 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333343+0000", 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6687999999999997e-05 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333384+0000", 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0499e-05 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop_reply(client.27874.0:8192 9.f e867/858 ondisk, result = 0)", 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:39:09.813396+0000", 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 314.77488310699999, 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51999520200000005, 2024-09-17T16:44:24.611 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.612 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.612 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:44:24.612 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813396+0000", 2024-09-17T16:44:24.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813396+0000", 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813397+0000", 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.92e-07 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:09.813401+0000", 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7280000000000003e-06 2024-09-17T16:44:24.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333226+0000", 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51982427099999995 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333233+0000", 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4560000000000003e-06 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333275+0000", 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1662000000000002e-05 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333280+0000", 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4650000000000002e-06 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:39:10.333391+0000", 2024-09-17T16:44:24.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011092800000000001 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "rep_scrubmap(9.3 e911 from shard 8)", 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:52.369653+0000", 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 212.21862592100001, 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70106850600000004, 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.616 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.369653+0000", 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.369653+0000", 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.369654+0000", 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8199999999999999e-07 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.369691+0000", 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7397999999999997e-05 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.821987+0000", 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45229588599999998 2024-09-17T16:44:24.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:52.821996+0000", 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6400000000000003e-06 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.070508+0000", 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24851257400000001 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:24.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.070523+0000", 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4718999999999999e-05 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.070722+0000", 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000198607 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.620 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122289 9.5 9:a636923a:::benchmark_data_smithi173_150363_object15384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.358953+0000", 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 211.22932643999999, 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8005457949999997, 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122289 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.621 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358953+0000", 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358953+0000", 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358955+0000", 2024-09-17T16:44:24.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.956e-06 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358962+0000", 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1550000000000001e-06 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358975+0000", 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3658999999999999e-05 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.623 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.358985+0000", 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1749999999999994e-06 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359048+0000", 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3594000000000001e-05 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359164+0000", 2024-09-17T16:44:24.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011561300000000001 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.359275+0000", 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000111024 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.361539+0000", 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022637820000000002 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.364742+0000", 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0032029329999999998 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159395+0000", 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7946534700000001 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159454+0000", 2024-09-17T16:44:24.627 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8721999999999998e-05 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.159498+0000", 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4712000000000002e-05 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:122329 9.5 9:a441955e:::benchmark_data_smithi173_150363_object15424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e916)", 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:53.796199+0000", 2024-09-17T16:44:24.628 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 210.79207979099999, 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3680691940000003, 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 122329 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.629 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796199+0000", 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796199+0000", 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796201+0000", 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5340000000000001e-06 2024-09-17T16:44:24.630 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796211+0000", 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.008e-05 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796224+0000", 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2993000000000001e-05 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796232+0000", 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0809999999999994e-06 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796296+0000", 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3929000000000004e-05 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796476+0000", 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000180132 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.632 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 7,11", 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.796577+0000", 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010087299999999999 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.798689+0000", 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021117190000000002 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:53.798875+0000", 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000186737 2024-09-17T16:44:24.633 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.164183+0000", 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3653075699999997 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.164232+0000", 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9259999999999999e-05 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:41:00.164268+0000", 2024-09-17T16:44:24.634 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6285999999999998e-05 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPush(9.12 917/915 [PushOp(9:4fdad5be:::benchmark_data_smithi173_150363_object19229:head, version: 874'1187, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:4fdad5be:::benchmark_data_smithi173_150363_object19229:head@874'1187, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0={}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:55.455908+0000", 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 209.13237067599999, 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.780314744, 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:55.455908+0000", 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:55.455908+0000", 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:55.455910+0000", 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.347e-06 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:55.455946+0000", 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6569000000000002e-05 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.234382+0000", 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.77843612299999998 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.637 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.234395+0000", 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2415000000000001e-05 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.234486+0000", 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0741000000000004e-05 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.234499+0000", 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3269e-05 2024-09-17T16:44:24.638 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.236223+0000", 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017242799999999999 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(9.19 917/915 [PushReplyOp(9:9d6f52ec:::benchmark_data_smithi173_150363_object21907:head)])", 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:40:56.373991+0000", 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 208.21428762900001, 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71017067599999995, 2024-09-17T16:44:24.639 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.373991+0000", 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.373991+0000", 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.640 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.373992+0000", 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8400000000000005e-07 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:56.373996+0000", 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4490000000000001e-06 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.081536+0000", 2024-09-17T16:44:24.641 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70753952099999995 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.081543+0000", 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7220000000000002e-06 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.081606+0000", 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3528999999999994e-05 2024-09-17T16:44:24.642 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:40:57.084162+0000", 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0025559710000000002 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(9.3 e974,969 9:c36db137:::benchmark_data_smithi173_150363_object11528:head,909'7404)", 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:07.918773+0000", 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 76.669505896999993, 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.47562009399999999, 2024-09-17T16:44:24.643 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.918773+0000", 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.918773+0000", 2024-09-17T16:44:24.644 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.918774+0000", 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9999999999999998e-07 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.918777+0000", 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5269999999999999e-06 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.645 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.393426+0000", 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.47464831000000002 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.393438+0000", 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2489e-05 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.393523+0000", 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5111999999999994e-05 2024-09-17T16:44:24.646 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.394393+0000", 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00087015599999999997 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.27874.0:213802 9.a 9:5ae34212:::benchmark_data_smithi173_150363_object106897:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e974)", 2024-09-17T16:44:24.647 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:07.949098+0000", 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 76.639181000999997, 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.73586259099999995, 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.27874", 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.173:0/469178814", 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 213802 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.648 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.949098+0000", 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.949098+0000", 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.949099+0000", 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0399999999999996e-07 2024-09-17T16:44:24.649 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.949101+0000", 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.666e-06 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.949116+0000", 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5000999999999999e-05 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.650 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.949123+0000", 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6880000000000002e-06 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.953148+0000", 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0040246070000000004 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.953256+0000", 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000108402 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.651 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for subops from 1,7", 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.953369+0000", 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000112662 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op_commit", 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.958197+0000", 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0048282350000000002 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:24.652 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:07.958619+0000", 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00042208300000000002 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.684894+0000", 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.72627503000000004 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.684932+0000", 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7592000000000002e-05 2024-09-17T16:44:24.653 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:08.684961+0000", 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9020999999999999e-05 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(9.3 e975,969 9:c67ae0fe:::benchmark_data_smithi173_150363_object19684:head,920'7749)", 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:11.294979+0000", 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 73.293300006999999, 2024-09-17T16:44:24.654 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55988219500000003, 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:11.294979+0000", 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.655 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:11.294979+0000", 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:11.294980+0000", 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8100000000000001e-07 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:11.294987+0000", 2024-09-17T16:44:24.656 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2189999999999998e-06 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:11.854215+0000", 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.55922801200000005 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:11.854241+0000", 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5392e-05 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:11.854312+0000", 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1853000000000005e-05 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:11.854861+0000", 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054883799999999995 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(9.3 e982,969 9:c069c856:::benchmark_data_smithi173_150363_object58163:head,942'9048)", 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:21.523996+0000", 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 63.064282972999997, 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87685215500000002, 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:21.523996+0000", 2024-09-17T16:44:24.659 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:21.523996+0000", 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:21.523997+0000", 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0100000000000005e-07 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.660 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:21.524001+0000", 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1150000000000004e-06 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:22.400148+0000", 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.87614699699999998 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:22.400164+0000", 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6473e-05 2024-09-17T16:44:24.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:22.400274+0000", 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010932799999999999 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:22.400848+0000", 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057464099999999998 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.662 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.663 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.663 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.663 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(9.3 e984,969 9:c18161c9:::benchmark_data_smithi173_150363_object70824:head,951'9416)", 2024-09-17T16:44:24.663 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:25.069461+0000", 2024-09-17T16:44:24.663 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.518818226999997, 2024-09-17T16:44:24.663 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52654153800000003, 2024-09-17T16:44:24.663 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.663 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.663 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.069461+0000", 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.069461+0000", 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.664 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.069469+0000", 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4999999999999999e-06 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.069478+0000", 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2400000000000007e-06 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.595378+0000", 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52590027100000003 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.665 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.595395+0000", 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.702e-05 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.595546+0000", 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015135800000000001 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.596002+0000", 2024-09-17T16:44:24.666 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00045614899999999999 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(9.3 e984,969 9:c177cdfd:::benchmark_data_smithi173_150363_object70857:head,951'9417)", 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:25.073249+0000", 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.515029701000003, 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52745197099999996, 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.667 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.073249+0000", 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.073249+0000", 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.073256+0000", 2024-09-17T16:44:24.668 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9659999999999999e-06 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.073266+0000", 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5929999999999995e-06 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.600349+0000", 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.52708264599999999 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.669 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.600354+0000", 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1580000000000004e-06 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.600429+0000", 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4791999999999998e-05 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:25.600701+0000", 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000272816 2024-09-17T16:44:24.670 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(9.3 e985,969 9:c69b8fd7:::benchmark_data_smithi173_150363_object77304:head,955'9622)", 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:26.141862+0000", 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 58.446416769000002, 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.63100178699999998, 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.671 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.141862+0000", 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.141862+0000", 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.672 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.141863+0000", 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.5099999999999999e-07 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.141867+0000", 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8940000000000003e-06 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.772234+0000", 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.63036744 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.673 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.772252+0000", 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7329e-05 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.772375+0000", 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012371499999999999 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.772864+0000", 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00048875800000000005 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.674 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(9.3 e985,969 9:c6d34c1a:::benchmark_data_smithi173_150363_object77315:head,955'9623)", 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:26.145717+0000", 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 58.442561810000001, 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.63186866799999997, 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.675 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.145717+0000", 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.145717+0000", 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.145718+0000", 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5099999999999999e-07 2024-09-17T16:44:24.676 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.145723+0000", 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6530000000000002e-06 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.777293+0000", 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.63157010400000002 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.677 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.777298+0000", 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9069999999999998e-06 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.777341+0000", 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3126e-05 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:26.777586+0000", 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024512699999999999 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.678 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(9.19 e993,969 9:9b83b8b7:::benchmark_data_smithi173_150363_object32925:head,923'8075)", 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:38.428837+0000", 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 46.159441805999997, 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51133975300000001, 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.679 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:38.428837+0000", 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:38.428837+0000", 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:38.428838+0000", 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8299999999999996e-07 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.680 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:38.428843+0000", 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2569999999999998e-06 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:38.939491+0000", 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51064790299999996 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:38.939500+0000", 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7919999999999998e-06 2024-09-17T16:44:24.681 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:38.939564+0000", 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4375000000000006e-05 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:38.940177+0000", 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00061264299999999995 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.682 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(9.19 e998,969 9:996e88f5:::benchmark_data_smithi173_150363_object69376:head,950'9216)", 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:50.040535+0000", 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 34.547743834999999, 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56779404899999997, 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:50.040535+0000", 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.683 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:50.040535+0000", 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:50.040536+0000", 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7199999999999999e-07 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.684 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:50.040539+0000", 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.196e-06 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:50.607818+0000", 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.56727872599999996 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:50.607824+0000", 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3509999999999998e-06 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.685 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:50.607877+0000", 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3041999999999998e-05 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:50.608329+0000", 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00045226199999999999 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.686 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGRecoveryDeleteReply(9.19 e999,969 9:9b03a90c:::benchmark_data_smithi173_150363_object74081:head,954'9364)", 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-09-17T16:43:51.683830+0000", 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 32.904449479, 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.67604968099999996, 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:51.683830+0000", 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.687 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:51.683830+0000", 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:51.683830+0000", 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2900000000000002e-07 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-09-17T16:44:24.688 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:51.683835+0000", 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.057e-06 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:52.359252+0000", 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.67541662999999996 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:52.359267+0000", 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4749999999999999e-05 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.689 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:52.359378+0000", 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000111181 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-09-17T16:43:52.359879+0000", 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00050123400000000003 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-09-17T16:44:24.690 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-09-17T16:44:25.269 INFO:teuthology.orchestra.run.smithi017.stderr:set pool 1 pgp_num to 11 2024-09-17T16:44:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:25 smithi017 ceph-mon[27571]: pgmap v2225: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:25.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:25 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1473441632' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-17T16:44:25.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:25 smithi071 ceph-mon[30002]: pgmap v2225: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:25 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1473441632' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-17T16:44:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:25 smithi173 ceph-mon[30530]: pgmap v2225: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:25 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1473441632' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-17T16:44:26.001 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-17T16:44:26.263 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:26.539 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:26 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1473441632' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-17T16:44:26.539 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:26 smithi017 ceph-mon[27571]: osdmap e1019: 12 total, 12 up, 12 in 2024-09-17T16:44:26.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:26 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1473441632' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-17T16:44:26.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:26 smithi071 ceph-mon[30002]: osdmap e1019: 12 total, 12 up, 12 in 2024-09-17T16:44:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:26 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1473441632' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-17T16:44:26.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:26 smithi173 ceph-mon[30530]: osdmap e1019: 12 total, 12 up, 12 in 2024-09-17T16:44:26.778 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a: {} 2024-09-17T16:44:26.779 INFO:teuthology.orchestra.run.smithi017.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:44:26.794 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c: {} 2024-09-17T16:44:26.794 INFO:teuthology.orchestra.run.smithi017.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:44:26.808 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b: {} 2024-09-17T16:44:26.808 INFO:teuthology.orchestra.run.smithi017.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:44:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:27 smithi017 ceph-mon[27571]: pgmap v2227: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:27.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:27 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/736621828' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:44:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:27 smithi071 ceph-mon[30002]: pgmap v2227: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:27 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/736621828' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:44:27.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-17T16:44:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:27 smithi173 ceph-mon[30530]: pgmap v2227: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:27 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/736621828' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-17T16:44:27.893 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:28 smithi071 ceph-mon[30002]: 1.8 scrub starts 2024-09-17T16:44:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:28 smithi071 ceph-mon[30002]: 1.8 scrub ok 2024-09-17T16:44:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:28 smithi071 ceph-mon[30002]: 1.3 scrub starts 2024-09-17T16:44:28.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:28 smithi071 ceph-mon[30002]: 1.3 scrub ok 2024-09-17T16:44:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:28 smithi173 ceph-mon[30530]: 1.8 scrub starts 2024-09-17T16:44:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:28 smithi173 ceph-mon[30530]: 1.8 scrub ok 2024-09-17T16:44:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:28 smithi173 ceph-mon[30530]: 1.3 scrub starts 2024-09-17T16:44:28.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:28 smithi173 ceph-mon[30530]: 1.3 scrub ok 2024-09-17T16:44:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:28 smithi017 ceph-mon[27571]: 1.8 scrub starts 2024-09-17T16:44:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:28 smithi017 ceph-mon[27571]: 1.8 scrub ok 2024-09-17T16:44:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:28 smithi017 ceph-mon[27571]: 1.3 scrub starts 2024-09-17T16:44:28.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:28 smithi017 ceph-mon[27571]: 1.3 scrub ok 2024-09-17T16:44:29.283 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T16:44:29.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:29 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:29.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:29 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:29.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:29 smithi017 ceph-mon[27571]: 1.5 scrub starts 2024-09-17T16:44:29.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:29 smithi017 ceph-mon[27571]: 1.5 scrub ok 2024-09-17T16:44:29.557 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:29 smithi017 ceph-mon[27571]: pgmap v2228: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:29.557 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:29 smithi017 ceph-mon[27571]: osdmap e1020: 12 total, 12 up, 12 in 2024-09-17T16:44:29.557 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:29 smithi017 ceph-mon[27571]: 1.a scrub starts 2024-09-17T16:44:29.557 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:29 smithi017 ceph-mon[27571]: 1.a scrub ok 2024-09-17T16:44:29.557 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:29 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1273192999' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:44:29.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:29 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:29 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:29 smithi071 ceph-mon[30002]: 1.5 scrub starts 2024-09-17T16:44:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:29 smithi071 ceph-mon[30002]: 1.5 scrub ok 2024-09-17T16:44:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:29 smithi071 ceph-mon[30002]: pgmap v2228: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:29 smithi071 ceph-mon[30002]: osdmap e1020: 12 total, 12 up, 12 in 2024-09-17T16:44:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:29 smithi071 ceph-mon[30002]: 1.a scrub starts 2024-09-17T16:44:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:29 smithi071 ceph-mon[30002]: 1.a scrub ok 2024-09-17T16:44:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:29 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1273192999' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:44:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:29 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:29 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:29 smithi173 ceph-mon[30530]: 1.5 scrub starts 2024-09-17T16:44:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:29 smithi173 ceph-mon[30530]: 1.5 scrub ok 2024-09-17T16:44:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:29 smithi173 ceph-mon[30530]: pgmap v2228: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:29 smithi173 ceph-mon[30530]: osdmap e1020: 12 total, 12 up, 12 in 2024-09-17T16:44:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:29 smithi173 ceph-mon[30530]: 1.a scrub starts 2024-09-17T16:44:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:29 smithi173 ceph-mon[30530]: 1.a scrub ok 2024-09-17T16:44:29.646 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:29 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1273192999' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:44:30.079 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-17T16:44:30.358 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:30.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:30 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1273192999' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:44:30.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:30 smithi017 ceph-mon[27571]: osdmap e1021: 12 total, 12 up, 12 in 2024-09-17T16:44:30.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:44:30.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:30 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:30.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:30 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:30.480 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:30 smithi017 ceph-mon[27571]: osdmap e1022: 12 total, 12 up, 12 in 2024-09-17T16:44:30.481 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:44:30.481 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:44:30.481 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:44:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:30 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1273192999' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:44:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:30 smithi071 ceph-mon[30002]: osdmap e1021: 12 total, 12 up, 12 in 2024-09-17T16:44:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:44:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:30 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:30 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:30 smithi071 ceph-mon[30002]: osdmap e1022: 12 total, 12 up, 12 in 2024-09-17T16:44:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:44:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:44:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:44:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:30 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1273192999' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:44:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:30 smithi173 ceph-mon[30530]: osdmap e1021: 12 total, 12 up, 12 in 2024-09-17T16:44:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:44:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:30 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:30 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:30 smithi173 ceph-mon[30530]: osdmap e1022: 12 total, 12 up, 12 in 2024-09-17T16:44:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:44:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:44:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:44:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:31 smithi071 ceph-mon[30002]: pgmap v2232: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:31 smithi071 ceph-mon[30002]: 1.7 scrub starts 2024-09-17T16:44:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:31 smithi071 ceph-mon[30002]: 1.7 scrub ok 2024-09-17T16:44:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:31 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/731210804' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:44:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:31 smithi173 ceph-mon[30530]: pgmap v2232: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:31 smithi173 ceph-mon[30530]: 1.7 scrub starts 2024-09-17T16:44:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:31 smithi173 ceph-mon[30530]: 1.7 scrub ok 2024-09-17T16:44:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:31 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/731210804' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:44:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:31 smithi017 ceph-mon[27571]: pgmap v2232: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:31 smithi017 ceph-mon[27571]: 1.7 scrub starts 2024-09-17T16:44:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:31 smithi017 ceph-mon[27571]: 1.7 scrub ok 2024-09-17T16:44:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:31 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/731210804' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-17T16:44:31.874 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-17T16:44:32.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-17T16:44:32.932 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:32 smithi017 ceph-mon[27571]: 1.1 scrub starts 2024-09-17T16:44:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:32 smithi017 ceph-mon[27571]: 1.1 scrub ok 2024-09-17T16:44:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:32 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/731210804' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:44:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:32 smithi017 ceph-mon[27571]: osdmap e1023: 12 total, 12 up, 12 in 2024-09-17T16:44:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:32 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:32 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:32.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:32 smithi017 ceph-mon[27571]: pgmap v2234: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:32 smithi173 ceph-mon[30530]: 1.1 scrub starts 2024-09-17T16:44:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:32 smithi173 ceph-mon[30530]: 1.1 scrub ok 2024-09-17T16:44:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:32 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/731210804' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:44:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:32 smithi173 ceph-mon[30530]: osdmap e1023: 12 total, 12 up, 12 in 2024-09-17T16:44:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:32 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:32 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:33.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:32 smithi173 ceph-mon[30530]: pgmap v2234: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:32 smithi071 ceph-mon[30002]: 1.1 scrub starts 2024-09-17T16:44:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:32 smithi071 ceph-mon[30002]: 1.1 scrub ok 2024-09-17T16:44:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:32 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/731210804' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-17T16:44:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:32 smithi071 ceph-mon[30002]: osdmap e1023: 12 total, 12 up, 12 in 2024-09-17T16:44:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:32 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:32 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:33.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:32 smithi071 ceph-mon[30002]: pgmap v2234: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:33.881 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T16:44:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:33 smithi173 ceph-mon[30530]: osdmap e1024: 12 total, 12 up, 12 in 2024-09-17T16:44:34.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:33 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3258552602' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:44:34.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:33 smithi017 ceph-mon[27571]: osdmap e1024: 12 total, 12 up, 12 in 2024-09-17T16:44:34.237 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:33 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3258552602' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:44:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:33 smithi071 ceph-mon[30002]: osdmap e1024: 12 total, 12 up, 12 in 2024-09-17T16:44:34.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:33 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3258552602' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:44:34.553 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-17T16:44:34.809 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:34 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3258552602' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:44:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:34 smithi173 ceph-mon[30530]: osdmap e1025: 12 total, 12 up, 12 in 2024-09-17T16:44:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:34 smithi173 ceph-mon[30530]: pgmap v2237: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:34 smithi173 ceph-mon[30530]: osdmap e1026: 12 total, 12 up, 12 in 2024-09-17T16:44:35.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:34 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3258552602' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:44:35.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:34 smithi017 ceph-mon[27571]: osdmap e1025: 12 total, 12 up, 12 in 2024-09-17T16:44:35.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:34 smithi017 ceph-mon[27571]: pgmap v2237: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:35.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:34 smithi017 ceph-mon[27571]: osdmap e1026: 12 total, 12 up, 12 in 2024-09-17T16:44:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:34 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3258552602' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:44:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:34 smithi071 ceph-mon[30002]: osdmap e1025: 12 total, 12 up, 12 in 2024-09-17T16:44:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:34 smithi071 ceph-mon[30002]: pgmap v2237: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:35.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:34 smithi071 ceph-mon[30002]: osdmap e1026: 12 total, 12 up, 12 in 2024-09-17T16:44:35.897 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-17T16:44:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:35 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:35 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:35 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1241349679' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:44:36.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:35 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:36.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:35 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:36.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:35 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1241349679' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:44:36.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:35 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:36.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:35 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:36.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:35 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1241349679' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-17T16:44:36.797 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-17T16:44:37.058 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:37.180 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1241349679' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:44:37.180 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:36 smithi017 ceph-mon[27571]: osdmap e1027: 12 total, 12 up, 12 in 2024-09-17T16:44:37.180 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:36 smithi017 ceph-mon[27571]: pgmap v2240: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1241349679' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:44:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:36 smithi071 ceph-mon[30002]: osdmap e1027: 12 total, 12 up, 12 in 2024-09-17T16:44:37.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:36 smithi071 ceph-mon[30002]: pgmap v2240: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1241349679' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-17T16:44:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:36 smithi173 ceph-mon[30530]: osdmap e1027: 12 total, 12 up, 12 in 2024-09-17T16:44:37.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:36 smithi173 ceph-mon[30530]: pgmap v2240: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:37.909 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T16:44:38.234 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/989146504' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:44:38.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/989146504' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:44:38.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/989146504' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:44:38.666 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-17T16:44:38.922 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:39.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:38 smithi017 ceph-mon[27571]: 1.2 scrub starts 2024-09-17T16:44:39.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:38 smithi017 ceph-mon[27571]: 1.2 scrub ok 2024-09-17T16:44:39.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/989146504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:44:39.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:38 smithi017 ceph-mon[27571]: osdmap e1028: 12 total, 12 up, 12 in 2024-09-17T16:44:39.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:38 smithi017 ceph-mon[27571]: pgmap v2242: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:39.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:38 smithi173 ceph-mon[30530]: 1.2 scrub starts 2024-09-17T16:44:39.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:38 smithi173 ceph-mon[30530]: 1.2 scrub ok 2024-09-17T16:44:39.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/989146504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:44:39.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:38 smithi173 ceph-mon[30530]: osdmap e1028: 12 total, 12 up, 12 in 2024-09-17T16:44:39.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:38 smithi173 ceph-mon[30530]: pgmap v2242: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:38 smithi071 ceph-mon[30002]: 1.2 scrub starts 2024-09-17T16:44:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:38 smithi071 ceph-mon[30002]: 1.2 scrub ok 2024-09-17T16:44:39.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/989146504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:44:39.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:38 smithi071 ceph-mon[30002]: osdmap e1028: 12 total, 12 up, 12 in 2024-09-17T16:44:39.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:38 smithi071 ceph-mon[30002]: pgmap v2242: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:39.932 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-17T16:44:40.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2806909033' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:44:40.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2806909033' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:44:40.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2806909033' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-17T16:44:40.747 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-17T16:44:41.000 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:41.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2806909033' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:44:41.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:40 smithi017 ceph-mon[27571]: osdmap e1029: 12 total, 12 up, 12 in 2024-09-17T16:44:41.224 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:40 smithi017 ceph-mon[27571]: pgmap v2244: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2806909033' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:44:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:40 smithi071 ceph-mon[30002]: osdmap e1029: 12 total, 12 up, 12 in 2024-09-17T16:44:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:40 smithi071 ceph-mon[30002]: pgmap v2244: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:41.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2806909033' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-17T16:44:41.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:40 smithi173 ceph-mon[30530]: osdmap e1029: 12 total, 12 up, 12 in 2024-09-17T16:44:41.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:40 smithi173 ceph-mon[30530]: pgmap v2244: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:41.945 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T16:44:42.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:41 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1226857337' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:44:42.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:41 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1226857337' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:44:42.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:41 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1226857337' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:44:42.696 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-17T16:44:42.950 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:43.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1226857337' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:44:43.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: osdmap e1030: 12 total, 12 up, 12 in 2024-09-17T16:44:43.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: pgmap v2246: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:43.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=2160301444,config=2353167246,health=3897446751,kv=2591757224,logm=1072436096}) 2024-09-17T16:44:43.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1551205841}) 2024-09-17T16:44:43.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=217064387}) 2024-09-17T16:44:43.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3173702551}) 2024-09-17T16:44:43.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1883357022}) 2024-09-17T16:44:43.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054411601}) 2024-09-17T16:44:43.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933444917}) 2024-09-17T16:44:43.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3805424563}) 2024-09-17T16:44:43.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3273012725}) 2024-09-17T16:44:43.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508872391}) 2024-09-17T16:44:43.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263597814}) 2024-09-17T16:44:43.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069994879}) 2024-09-17T16:44:43.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870994419}) 2024-09-17T16:44:43.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899488714}) 2024-09-17T16:44:43.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850967252}) 2024-09-17T16:44:43.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466115059}) 2024-09-17T16:44:43.209 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128063089}) 2024-09-17T16:44:43.209 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416515822}) 2024-09-17T16:44:43.209 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612570454}) 2024-09-17T16:44:43.209 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1477800255}) 2024-09-17T16:44:43.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=343802155}) 2024-09-17T16:44:43.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656743918}) 2024-09-17T16:44:43.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512547663}) 2024-09-17T16:44:43.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716113655}) 2024-09-17T16:44:43.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218247557}) 2024-09-17T16:44:43.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1903902501}) 2024-09-17T16:44:43.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3309274036}) 2024-09-17T16:44:43.210 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189839154}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4101254976}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237405855}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1262088066}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1542592686}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091124578}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665280088}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2697881465}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806033984}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1931047507}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=312154787}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=116463153}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3400083532}) 2024-09-17T16:44:43.211 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606352551}) 2024-09-17T16:44:43.212 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1252699069}) 2024-09-17T16:44:43.212 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3911981418}) 2024-09-17T16:44:43.212 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=723423421}) 2024-09-17T16:44:43.212 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3654643081}) 2024-09-17T16:44:43.212 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=96716821}) 2024-09-17T16:44:43.212 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711743825}) 2024-09-17T16:44:43.212 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3495750289}) 2024-09-17T16:44:43.212 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=435342520}) 2024-09-17T16:44:43.212 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3250354111}) 2024-09-17T16:44:43.212 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2004148776}) 2024-09-17T16:44:43.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=810600387}) 2024-09-17T16:44:43.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1881762765}) 2024-09-17T16:44:43.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=275530813}) 2024-09-17T16:44:43.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773980664}) 2024-09-17T16:44:43.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238877813}) 2024-09-17T16:44:43.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744997321}) 2024-09-17T16:44:43.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3772773723}) 2024-09-17T16:44:43.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3496834618}) 2024-09-17T16:44:43.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082329055}) 2024-09-17T16:44:43.213 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=522714626}) 2024-09-17T16:44:43.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561043273}) 2024-09-17T16:44:43.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272704203}) 2024-09-17T16:44:43.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891476169}) 2024-09-17T16:44:43.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753628656}) 2024-09-17T16:44:43.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237382731}) 2024-09-17T16:44:43.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1786387659}) 2024-09-17T16:44:43.214 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189499977}) 2024-09-17T16:44:43.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730811525}) 2024-09-17T16:44:43.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1058696497}) 2024-09-17T16:44:43.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256395385}) 2024-09-17T16:44:43.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2503428675}) 2024-09-17T16:44:43.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4043630228}) 2024-09-17T16:44:43.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095636221}) 2024-09-17T16:44:43.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762004158}) 2024-09-17T16:44:43.215 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169814360}) 2024-09-17T16:44:43.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=570541596}) 2024-09-17T16:44:43.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {logm=8,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=59} crc {logm=835072058,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=136678609,mon_config_key=365660596}) 2024-09-17T16:44:43.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1805569682}) 2024-09-17T16:44:43.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3252778003}) 2024-09-17T16:44:43.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=47} crc {mon_config_key=214115934,monmap=2143941632,nvmeofgw=598337686,osd_metadata=923789942,osd_pg_creating=2272419594,osdmap=4098181295}) 2024-09-17T16:44:43.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2066620641}) 2024-09-17T16:44:43.217 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:42 smithi017 ceph-mon[27571]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=26} crc {osdmap=3238826362}) 2024-09-17T16:44:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1226857337' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:44:43.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: osdmap e1030: 12 total, 12 up, 12 in 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: pgmap v2246: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=2160301444,config=2353167246,health=3897446751,kv=2591757224,logm=1072436096}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1551205841}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=217064387}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3173702551}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1883357022}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054411601}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933444917}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3805424563}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3273012725}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508872391}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263597814}) 2024-09-17T16:44:43.315 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069994879}) 2024-09-17T16:44:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870994419}) 2024-09-17T16:44:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899488714}) 2024-09-17T16:44:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850967252}) 2024-09-17T16:44:43.316 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466115059}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128063089}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416515822}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612570454}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1477800255}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=343802155}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656743918}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512547663}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716113655}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218247557}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1903902501}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3309274036}) 2024-09-17T16:44:43.317 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189839154}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4101254976}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237405855}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1262088066}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1542592686}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091124578}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665280088}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2697881465}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806033984}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1931047507}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=312154787}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=116463153}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3400083532}) 2024-09-17T16:44:43.318 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606352551}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1252699069}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3911981418}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=723423421}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3654643081}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=96716821}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711743825}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3495750289}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=435342520}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3250354111}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2004148776}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=810600387}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1881762765}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=275530813}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773980664}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238877813}) 2024-09-17T16:44:43.319 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744997321}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3772773723}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3496834618}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082329055}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=522714626}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561043273}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272704203}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891476169}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753628656}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237382731}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1786387659}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189499977}) 2024-09-17T16:44:43.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730811525}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1058696497}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256395385}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2503428675}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4043630228}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095636221}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762004158}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169814360}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=570541596}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {logm=8,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=59} crc {logm=835072058,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=136678609,mon_config_key=365660596}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1805569682}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3252778003}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=47} crc {mon_config_key=214115934,monmap=2143941632,nvmeofgw=598337686,osd_metadata=923789942,osd_pg_creating=2272419594,osdmap=4098181295}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2066620641}) 2024-09-17T16:44:43.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:42 smithi071 ceph-mon[30002]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=26} crc {osdmap=3238826362}) 2024-09-17T16:44:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1226857337' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:44:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: osdmap e1030: 12 total, 12 up, 12 in 2024-09-17T16:44:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: pgmap v2246: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=56,logm=4} crc {auth=2160301444,config=2353167246,health=3897446751,kv=2591757224,logm=1072436096}) 2024-09-17T16:44:43.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1551205841}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=217064387}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3173702551}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1883357022}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2054411601}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1933444917}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3805424563}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3273012725}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2508872391}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1263597814}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069994879}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=870994419}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2899488714}) 2024-09-17T16:44:43.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850967252}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2466115059}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2128063089}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1416515822}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=612570454}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1477800255}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=343802155}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656743918}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512547663}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=716113655}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218247557}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1903902501}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3309274036}) 2024-09-17T16:44:43.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2189839154}) 2024-09-17T16:44:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4101254976}) 2024-09-17T16:44:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237405855}) 2024-09-17T16:44:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1262088066}) 2024-09-17T16:44:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1542592686}) 2024-09-17T16:44:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091124578}) 2024-09-17T16:44:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665280088}) 2024-09-17T16:44:43.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2697881465}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806033984}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1931047507}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=312154787}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=116463153}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3400083532}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606352551}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1252699069}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3911981418}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=723423421}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3654643081}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=96716821}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711743825}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3495750289}) 2024-09-17T16:44:43.399 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=435342520}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3250354111}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2004148776}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=810600387}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1881762765}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=275530813}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773980664}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238877813}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744997321}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3772773723}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3496834618}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082329055}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=522714626}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2561043273}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272704203}) 2024-09-17T16:44:43.400 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891476169}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753628656}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3237382731}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1786387659}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189499977}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730811525}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1058696497}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256395385}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2503428675}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4043630228}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095636221}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762004158}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3169814360}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=570541596}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {logm=8,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9,mon_config_key=59} crc {logm=835072058,mdsmap=376187277,mgr=4131498508,mgr_command_descs=1828700858,mgr_metadata=1024005251,mgrstat=136678609,mon_config_key=365660596}) 2024-09-17T16:44:43.401 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1805569682}) 2024-09-17T16:44:43.402 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3252778003}) 2024-09-17T16:44:43.402 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,nvmeofgw=3,osd_metadata=12,osd_pg_creating=1,osdmap=47} crc {mon_config_key=214115934,monmap=2143941632,nvmeofgw=598337686,osd_metadata=923789942,osd_pg_creating=2272419594,osdmap=4098181295}) 2024-09-17T16:44:43.402 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2066620641}) 2024-09-17T16:44:43.402 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:42 smithi173 ceph-mon[30530]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=26} crc {osdmap=3238826362}) 2024-09-17T16:44:43.956 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-17T16:44:44.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1258259157' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:44:44.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1258259157' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:44:44.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1258259157' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-17T16:44:44.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-17T16:44:44.933 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:45.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1258259157' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:44:45.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:44 smithi017 ceph-mon[27571]: osdmap e1031: 12 total, 12 up, 12 in 2024-09-17T16:44:45.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:44 smithi017 ceph-mon[27571]: pgmap v2248: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:45.226 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:44 smithi017 ceph-mon[27571]: osdmap e1032: 12 total, 12 up, 12 in 2024-09-17T16:44:45.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1258259157' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:44:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:44 smithi071 ceph-mon[30002]: osdmap e1031: 12 total, 12 up, 12 in 2024-09-17T16:44:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:44 smithi071 ceph-mon[30002]: pgmap v2248: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:45.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:44 smithi071 ceph-mon[30002]: osdmap e1032: 12 total, 12 up, 12 in 2024-09-17T16:44:45.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1258259157' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-17T16:44:45.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:44 smithi173 ceph-mon[30530]: osdmap e1031: 12 total, 12 up, 12 in 2024-09-17T16:44:45.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:44 smithi173 ceph-mon[30530]: pgmap v2248: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:45.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:44 smithi173 ceph-mon[30530]: osdmap e1032: 12 total, 12 up, 12 in 2024-09-17T16:44:45.963 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T16:44:46.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:45 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1644470747' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:44:46.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:45 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1644470747' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:44:46.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:45 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1644470747' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:44:46.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-17T16:44:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:47.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1644470747' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:44:47.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:46 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:47.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:46 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:47.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:46 smithi017 ceph-mon[27571]: osdmap e1033: 12 total, 12 up, 12 in 2024-09-17T16:44:47.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:46 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:47.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:46 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:47.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:46 smithi017 ceph-mon[27571]: pgmap v2251: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:47.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1644470747' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:44:47.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:46 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:47.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:46 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:47.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:46 smithi071 ceph-mon[30002]: osdmap e1033: 12 total, 12 up, 12 in 2024-09-17T16:44:47.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:46 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:47.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:46 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:47.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:46 smithi071 ceph-mon[30002]: pgmap v2251: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1644470747' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:44:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:46 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:46 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:46 smithi173 ceph-mon[30530]: osdmap e1033: 12 total, 12 up, 12 in 2024-09-17T16:44:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:46 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:46 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:47.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:46 smithi173 ceph-mon[30530]: pgmap v2251: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:47.985 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-17T16:44:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:47 smithi017 ceph-mon[27571]: osdmap e1034: 12 total, 12 up, 12 in 2024-09-17T16:44:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:47 smithi017 ceph-mon[27571]: 1.6 scrub starts 2024-09-17T16:44:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:47 smithi017 ceph-mon[27571]: 1.6 scrub ok 2024-09-17T16:44:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:47 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:44:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:47 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:44:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:47 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/258024384' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:44:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:47 smithi071 ceph-mon[30002]: osdmap e1034: 12 total, 12 up, 12 in 2024-09-17T16:44:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:47 smithi071 ceph-mon[30002]: 1.6 scrub starts 2024-09-17T16:44:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:47 smithi071 ceph-mon[30002]: 1.6 scrub ok 2024-09-17T16:44:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:47 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:44:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:47 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:44:48.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:47 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/258024384' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:44:48.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:47 smithi173 ceph-mon[30530]: osdmap e1034: 12 total, 12 up, 12 in 2024-09-17T16:44:48.491 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:47 smithi173 ceph-mon[30530]: 1.6 scrub starts 2024-09-17T16:44:48.491 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:47 smithi173 ceph-mon[30530]: 1.6 scrub ok 2024-09-17T16:44:48.491 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:47 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:44:48.491 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:47 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:44:48.491 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:47 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/258024384' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-17T16:44:48.795 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-17T16:44:49.060 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:49.181 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/258024384' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:44:49.182 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:48 smithi017 ceph-mon[27571]: osdmap e1035: 12 total, 12 up, 12 in 2024-09-17T16:44:49.182 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:48 smithi017 ceph-mon[27571]: 1.4 scrub starts 2024-09-17T16:44:49.182 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:48 smithi017 ceph-mon[27571]: 1.4 scrub ok 2024-09-17T16:44:49.182 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:48 smithi017 ceph-mon[27571]: pgmap v2254: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/258024384' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:44:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:48 smithi071 ceph-mon[30002]: osdmap e1035: 12 total, 12 up, 12 in 2024-09-17T16:44:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:48 smithi071 ceph-mon[30002]: 1.4 scrub starts 2024-09-17T16:44:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:48 smithi071 ceph-mon[30002]: 1.4 scrub ok 2024-09-17T16:44:49.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:48 smithi071 ceph-mon[30002]: pgmap v2254: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/258024384' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-17T16:44:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:48 smithi173 ceph-mon[30530]: osdmap e1035: 12 total, 12 up, 12 in 2024-09-17T16:44:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:48 smithi173 ceph-mon[30530]: 1.4 scrub starts 2024-09-17T16:44:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:48 smithi173 ceph-mon[30530]: 1.4 scrub ok 2024-09-17T16:44:49.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:48 smithi173 ceph-mon[30530]: pgmap v2254: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:49.869 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T16:44:50.181 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2304604685' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:44:50.181 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2304604685' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:44:50.181 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:49 smithi017 ceph-mon[27571]: osdmap e1036: 12 total, 12 up, 12 in 2024-09-17T16:44:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2304604685' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:44:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2304604685' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:44:50.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:49 smithi071 ceph-mon[30002]: osdmap e1036: 12 total, 12 up, 12 in 2024-09-17T16:44:50.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2304604685' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:44:50.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2304604685' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:44:50.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:49 smithi173 ceph-mon[30530]: osdmap e1036: 12 total, 12 up, 12 in 2024-09-17T16:44:50.736 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-17T16:44:50.996 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:51.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:50 smithi017 ceph-mon[27571]: pgmap v2256: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:51.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:50 smithi071 ceph-mon[30002]: pgmap v2256: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:51.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:50 smithi173 ceph-mon[30530]: pgmap v2256: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:52.006 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-17T16:44:52.298 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:52 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3708723301' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:44:52.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:52 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3708723301' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:44:52.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:52 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3708723301' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-17T16:44:52.743 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 8 1 2024-09-17T16:44:53.006 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:53.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:53 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3708723301' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:44:53.276 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:53 smithi017 ceph-mon[27571]: osdmap e1037: 12 total, 12 up, 12 in 2024-09-17T16:44:53.276 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:53 smithi017 ceph-mon[27571]: pgmap v2258: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:53.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:53 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3708723301' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:44:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:53 smithi071 ceph-mon[30002]: osdmap e1037: 12 total, 12 up, 12 in 2024-09-17T16:44:53.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:53 smithi071 ceph-mon[30002]: pgmap v2258: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:53.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:53 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3708723301' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-17T16:44:53.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:53 smithi173 ceph-mon[30530]: osdmap e1037: 12 total, 12 up, 12 in 2024-09-17T16:44:53.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:53 smithi173 ceph-mon[30530]: pgmap v2258: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:54.019 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.8 to 1 (10000) 2024-09-17T16:44:54.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1660329077' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:44:54.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1660329077' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:44:54.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1660329077' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:44:54.743 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 8 1 2024-09-17T16:44:55.009 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:55.280 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:55 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1660329077' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:44:55.280 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:55 smithi017 ceph-mon[27571]: osdmap e1038: 12 total, 12 up, 12 in 2024-09-17T16:44:55.280 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:55 smithi017 ceph-mon[27571]: pgmap v2260: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:55 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1660329077' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:44:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:55 smithi071 ceph-mon[30002]: osdmap e1038: 12 total, 12 up, 12 in 2024-09-17T16:44:55.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:55 smithi071 ceph-mon[30002]: pgmap v2260: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:55 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1660329077' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:44:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:55 smithi173 ceph-mon[30530]: osdmap e1038: 12 total, 12 up, 12 in 2024-09-17T16:44:55.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:55 smithi173 ceph-mon[30530]: pgmap v2260: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:56.033 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-17T16:44:56.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:56 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1768768127' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:44:56.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:56 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1768768127' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:44:56.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:56 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1768768127' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-17T16:44:56.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 9 1 2024-09-17T16:44:57.238 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:57.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:57 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1768768127' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:44:57.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:57 smithi017 ceph-mon[27571]: osdmap e1039: 12 total, 12 up, 12 in 2024-09-17T16:44:57.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:57 smithi017 ceph-mon[27571]: pgmap v2262: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:57 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1768768127' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:44:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:57 smithi071 ceph-mon[30002]: osdmap e1039: 12 total, 12 up, 12 in 2024-09-17T16:44:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:57 smithi071 ceph-mon[30002]: pgmap v2262: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:57 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1768768127' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-17T16:44:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:57 smithi173 ceph-mon[30530]: osdmap e1039: 12 total, 12 up, 12 in 2024-09-17T16:44:57.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:57 smithi173 ceph-mon[30530]: pgmap v2262: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:58.043 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.9 to 1 (10000) 2024-09-17T16:44:58.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:58 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4163738430' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:44:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:58 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4163738430' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:44:58.419 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:58 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4163738430' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:44:58.839 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 9 1 2024-09-17T16:44:59.101 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:44:59.221 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:59 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4163738430' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:44:59.221 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:59 smithi017 ceph-mon[27571]: osdmap e1040: 12 total, 12 up, 12 in 2024-09-17T16:44:59.221 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:44:59 smithi017 ceph-mon[27571]: pgmap v2264: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:59 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4163738430' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:44:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:59 smithi071 ceph-mon[30002]: osdmap e1040: 12 total, 12 up, 12 in 2024-09-17T16:44:59.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:44:59 smithi071 ceph-mon[30002]: pgmap v2264: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:44:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:59 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4163738430' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:44:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:59 smithi173 ceph-mon[30530]: osdmap e1040: 12 total, 12 up, 12 in 2024-09-17T16:44:59.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:44:59 smithi173 ceph-mon[30530]: pgmap v2264: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:00.056 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-17T16:45:00.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:00 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2361108336' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:45:00.328 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:00 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2361108336' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:45:00.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:00 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2361108336' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-09-17T16:45:00.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 11 1 2024-09-17T16:45:01.041 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:01 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2361108336' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:45:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:01 smithi071 ceph-mon[30002]: osdmap e1041: 12 total, 12 up, 12 in 2024-09-17T16:45:01.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:01 smithi071 ceph-mon[30002]: pgmap v2266: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:01.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:01 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2361108336' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:45:01.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:01 smithi017 ceph-mon[27571]: osdmap e1041: 12 total, 12 up, 12 in 2024-09-17T16:45:01.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:01 smithi017 ceph-mon[27571]: pgmap v2266: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:01 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2361108336' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-17T16:45:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:01 smithi173 ceph-mon[30530]: osdmap e1041: 12 total, 12 up, 12 in 2024-09-17T16:45:01.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:01 smithi173 ceph-mon[30530]: pgmap v2266: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:02.063 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.11 to 1 (10000) 2024-09-17T16:45:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:02 smithi173 ceph-mon[30530]: 1.0 scrub starts 2024-09-17T16:45:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:02 smithi173 ceph-mon[30530]: 1.0 scrub ok 2024-09-17T16:45:02.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:02 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2748299593' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:45:02.426 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:02 smithi017 ceph-mon[27571]: 1.0 scrub starts 2024-09-17T16:45:02.426 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:02 smithi017 ceph-mon[27571]: 1.0 scrub ok 2024-09-17T16:45:02.426 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:02 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2748299593' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:45:02.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:02 smithi071 ceph-mon[30002]: 1.0 scrub starts 2024-09-17T16:45:02.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:02 smithi071 ceph-mon[30002]: 1.0 scrub ok 2024-09-17T16:45:02.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:02 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2748299593' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:45:02.805 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 11 1 2024-09-17T16:45:03.070 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:03.339 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:03 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2748299593' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:45:03.339 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:03 smithi017 ceph-mon[27571]: osdmap e1042: 12 total, 12 up, 12 in 2024-09-17T16:45:03.339 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:03 smithi017 ceph-mon[27571]: pgmap v2268: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:03.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:03 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2748299593' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:45:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:03 smithi173 ceph-mon[30530]: osdmap e1042: 12 total, 12 up, 12 in 2024-09-17T16:45:03.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:03 smithi173 ceph-mon[30530]: pgmap v2268: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:03 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2748299593' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:45:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:03 smithi071 ceph-mon[30002]: osdmap e1042: 12 total, 12 up, 12 in 2024-09-17T16:45:03.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:03 smithi071 ceph-mon[30002]: pgmap v2268: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:04.078 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-17T16:45:04.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:04 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/769381595' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:45:04.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:04 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/769381595' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:45:04.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:04 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/769381595' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-17T16:45:04.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-17T16:45:05.162 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:05 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/769381595' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:45:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:05 smithi017 ceph-mon[27571]: osdmap e1043: 12 total, 12 up, 12 in 2024-09-17T16:45:05.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:05 smithi017 ceph-mon[27571]: pgmap v2270: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:05 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/769381595' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:45:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:05 smithi173 ceph-mon[30530]: osdmap e1043: 12 total, 12 up, 12 in 2024-09-17T16:45:05.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:05 smithi173 ceph-mon[30530]: pgmap v2270: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:05 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/769381595' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-17T16:45:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:05 smithi071 ceph-mon[30002]: osdmap e1043: 12 total, 12 up, 12 in 2024-09-17T16:45:05.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:05 smithi071 ceph-mon[30002]: pgmap v2270: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:06.084 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T16:45:06.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:06 smithi173 ceph-mon[30530]: osdmap e1044: 12 total, 12 up, 12 in 2024-09-17T16:45:06.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:06 smithi173 ceph-mon[30530]: 1.4 scrub starts 2024-09-17T16:45:06.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:06 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3717064974' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:45:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:06 smithi017 ceph-mon[27571]: osdmap e1044: 12 total, 12 up, 12 in 2024-09-17T16:45:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:06 smithi017 ceph-mon[27571]: 1.4 scrub starts 2024-09-17T16:45:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:06 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3717064974' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:45:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:06 smithi071 ceph-mon[30002]: osdmap e1044: 12 total, 12 up, 12 in 2024-09-17T16:45:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:06 smithi071 ceph-mon[30002]: 1.4 scrub starts 2024-09-17T16:45:06.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:06 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3717064974' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:45:06.803 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-17T16:45:07.062 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:07 smithi173 ceph-mon[30530]: 1.4 scrub ok 2024-09-17T16:45:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:07 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/3717064974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:45:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:07 smithi173 ceph-mon[30530]: osdmap e1045: 12 total, 12 up, 12 in 2024-09-17T16:45:07.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:07 smithi173 ceph-mon[30530]: pgmap v2273: 11 pgs: 1 peering, 10 active+clean; 309 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 47 KiB/s wr, 4 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:45:07.399 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:07 smithi017 ceph-mon[27571]: 1.4 scrub ok 2024-09-17T16:45:07.400 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:07 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/3717064974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:45:07.400 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:07 smithi017 ceph-mon[27571]: osdmap e1045: 12 total, 12 up, 12 in 2024-09-17T16:45:07.400 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:07 smithi017 ceph-mon[27571]: pgmap v2273: 11 pgs: 1 peering, 10 active+clean; 309 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 47 KiB/s wr, 4 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:45:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:07 smithi071 ceph-mon[30002]: 1.4 scrub ok 2024-09-17T16:45:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:07 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/3717064974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:45:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:07 smithi071 ceph-mon[30002]: osdmap e1045: 12 total, 12 up, 12 in 2024-09-17T16:45:07.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:07 smithi071 ceph-mon[30002]: pgmap v2273: 11 pgs: 1 peering, 10 active+clean; 309 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 47 KiB/s wr, 4 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:45:08.095 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-17T16:45:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:08 smithi173 ceph-mon[30530]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:45:08.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:08 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1208454150' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:45:08.432 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:08 smithi017 ceph-mon[27571]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:45:08.432 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:08 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1208454150' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:45:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:08 smithi071 ceph-mon[30002]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-17T16:45:08.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:08 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1208454150' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-17T16:45:08.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 10 1 2024-09-17T16:45:09.164 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:09.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:09 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1208454150' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:45:09.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:09 smithi017 ceph-mon[27571]: osdmap e1046: 12 total, 12 up, 12 in 2024-09-17T16:45:09.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:09 smithi017 ceph-mon[27571]: pgmap v2275: 11 pgs: 1 peering, 10 active+clean; 309 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 47 KiB/s wr, 4 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:45:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:09 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1208454150' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:45:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:09 smithi173 ceph-mon[30530]: osdmap e1046: 12 total, 12 up, 12 in 2024-09-17T16:45:09.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:09 smithi173 ceph-mon[30530]: pgmap v2275: 11 pgs: 1 peering, 10 active+clean; 309 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 47 KiB/s wr, 4 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:45:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:09 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1208454150' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-17T16:45:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:09 smithi071 ceph-mon[30002]: osdmap e1046: 12 total, 12 up, 12 in 2024-09-17T16:45:09.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:09 smithi071 ceph-mon[30002]: pgmap v2275: 11 pgs: 1 peering, 10 active+clean; 309 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 47 KiB/s wr, 4 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-17T16:45:10.108 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.10 to 1 (10000) 2024-09-17T16:45:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:10 smithi173 ceph-mon[30530]: osdmap e1047: 12 total, 12 up, 12 in 2024-09-17T16:45:10.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:10 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2354512972' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:45:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:10 smithi017 ceph-mon[27571]: osdmap e1047: 12 total, 12 up, 12 in 2024-09-17T16:45:10.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:10 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2354512972' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:45:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:10 smithi071 ceph-mon[30002]: osdmap e1047: 12 total, 12 up, 12 in 2024-09-17T16:45:10.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:10 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2354512972' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:45:10.934 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 10 1 2024-09-17T16:45:11.199 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:11.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:11 smithi017 ceph-mon[27571]: 1.2 scrub starts 2024-09-17T16:45:11.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:11 smithi017 ceph-mon[27571]: 1.2 scrub ok 2024-09-17T16:45:11.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:11 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2354512972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:45:11.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:11 smithi017 ceph-mon[27571]: osdmap e1048: 12 total, 12 up, 12 in 2024-09-17T16:45:11.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:11 smithi017 ceph-mon[27571]: pgmap v2278: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:11.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:11 smithi173 ceph-mon[30530]: 1.2 scrub starts 2024-09-17T16:45:11.394 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:11 smithi173 ceph-mon[30530]: 1.2 scrub ok 2024-09-17T16:45:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:11 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2354512972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:45:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:11 smithi173 ceph-mon[30530]: osdmap e1048: 12 total, 12 up, 12 in 2024-09-17T16:45:11.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:11 smithi173 ceph-mon[30530]: pgmap v2278: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:11 smithi071 ceph-mon[30002]: 1.2 scrub starts 2024-09-17T16:45:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:11 smithi071 ceph-mon[30002]: 1.2 scrub ok 2024-09-17T16:45:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:11 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2354512972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:45:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:11 smithi071 ceph-mon[30002]: osdmap e1048: 12 total, 12 up, 12 in 2024-09-17T16:45:11.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:11 smithi071 ceph-mon[30002]: pgmap v2278: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:12.122 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-17T16:45:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:12 smithi173 ceph-mon[30530]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-17T16:45:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:12 smithi173 ceph-mon[30530]: Cluster is now healthy 2024-09-17T16:45:12.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:12 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2368297946' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:45:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:12 smithi017 ceph-mon[27571]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-17T16:45:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:12 smithi017 ceph-mon[27571]: Cluster is now healthy 2024-09-17T16:45:12.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:12 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2368297946' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:45:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:12 smithi071 ceph-mon[30002]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-17T16:45:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:12 smithi071 ceph-mon[30002]: Cluster is now healthy 2024-09-17T16:45:12.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:12 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2368297946' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-09-17T16:45:12.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-17T16:45:13.189 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:13.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:13 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2368297946' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:45:13.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:13 smithi017 ceph-mon[27571]: osdmap e1049: 12 total, 12 up, 12 in 2024-09-17T16:45:13.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:13 smithi017 ceph-mon[27571]: pgmap v2280: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:13 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2368297946' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:45:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:13 smithi173 ceph-mon[30530]: osdmap e1049: 12 total, 12 up, 12 in 2024-09-17T16:45:13.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:13 smithi173 ceph-mon[30530]: pgmap v2280: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:13.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:13 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2368297946' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-17T16:45:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:13 smithi071 ceph-mon[30002]: osdmap e1049: 12 total, 12 up, 12 in 2024-09-17T16:45:13.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:13 smithi071 ceph-mon[30002]: pgmap v2280: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:14.144 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T16:45:14.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:14 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2671909546' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:45:14.425 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:14 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2671909546' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:45:14.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:14 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2671909546' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:45:14.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-17T16:45:15.120 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:15.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:15 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2671909546' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:45:15.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:15 smithi017 ceph-mon[27571]: osdmap e1050: 12 total, 12 up, 12 in 2024-09-17T16:45:15.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:15 smithi017 ceph-mon[27571]: pgmap v2282: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:15.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:15 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2671909546' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:45:15.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:15 smithi071 ceph-mon[30002]: osdmap e1050: 12 total, 12 up, 12 in 2024-09-17T16:45:15.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:15 smithi071 ceph-mon[30002]: pgmap v2282: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:15 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2671909546' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:45:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:15 smithi173 ceph-mon[30530]: osdmap e1050: 12 total, 12 up, 12 in 2024-09-17T16:45:15.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:15 smithi173 ceph-mon[30530]: pgmap v2282: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:16.163 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-17T16:45:16.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:16 smithi017 ceph-mon[27571]: 1.9 scrub starts 2024-09-17T16:45:16.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:16 smithi017 ceph-mon[27571]: 1.9 scrub ok 2024-09-17T16:45:16.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:16 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:45:16.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:16 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:45:16.490 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:16 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1076034511' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:45:16.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:16 smithi071 ceph-mon[30002]: 1.9 scrub starts 2024-09-17T16:45:16.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:16 smithi071 ceph-mon[30002]: 1.9 scrub ok 2024-09-17T16:45:16.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:16 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:45:16.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:16 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:45:16.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:16 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1076034511' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:45:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:16 smithi173 ceph-mon[30530]: 1.9 scrub starts 2024-09-17T16:45:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:16 smithi173 ceph-mon[30530]: 1.9 scrub ok 2024-09-17T16:45:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:16 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:45:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:16 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:45:16.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:16 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1076034511' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-17T16:45:16.987 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-17T16:45:16.987 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-17T16:45:16.987 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-17T16:45:16.987 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-17T16:45:16.987 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-17T16:45:16.987 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T16:45:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:17 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1076034511' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:45:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:17 smithi017 ceph-mon[27571]: osdmap e1051: 12 total, 12 up, 12 in 2024-09-17T16:45:17.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:17 smithi017 ceph-mon[27571]: pgmap v2284: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:17.252 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:17 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1076034511' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:45:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:17 smithi071 ceph-mon[30002]: osdmap e1051: 12 total, 12 up, 12 in 2024-09-17T16:45:17.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:17 smithi071 ceph-mon[30002]: pgmap v2284: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:17 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1076034511' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-17T16:45:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:17 smithi173 ceph-mon[30530]: osdmap e1051: 12 total, 12 up, 12 in 2024-09-17T16:45:17.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:17 smithi173 ceph-mon[30530]: pgmap v2284: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:17.924 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:45:17.924 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":1051,"fsid":"19341aec-750e-11ef-bceb-c7b262605968","created":"2024-09-17T16:02:46.978716+0000","modified":"2024-09-17T16:45:16.154401+0000","last_up_change":"2024-09-17T16:36:51.015941+0000","last_in_change":"2024-09-17T16:42:58.555013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":29,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:06:06.088375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1019","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"882","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.2699999809265137,"score_stable":3.2699999809265137,"optimal_score":1,"raw_score_acting":3.2699999809265137,"raw_score_stable":3.2699999809265137,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"139c41b7-e38b-42d2-ae0b-70b94041dbfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":884,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6803","nonce":3081446730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6805","nonce":3081446730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6809","nonce":3081446730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6807","nonce":3081446730}]},"public_addr":"172.21.15.17:6803/3081446730","cluster_addr":"172.21.15.17:6805/3081446730","heartbeat_back_addr":"172.21.15.17:6809/3081446730","heartbeat_front_addr":"172.21.15.17:6807/3081446730","state":["exists","up"]},{"osd":1,"uuid":"ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":988,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6811","nonce":603840826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6813","nonce":603840826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6817","nonce":603840826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6815","nonce":603840826}]},"public_addr":"172.21.15.17:6811/603840826","cluster_addr":"172.21.15.17:6813/603840826","heartbeat_back_addr":"172.21.15.17:6817/603840826","heartbeat_front_addr":"172.21.15.17:6815/603840826","state":["exists","up"]},{"osd":2,"uuid":"efb6c3f1-6157-4581-a39d-bda978aea6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":991,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6819","nonce":2706983780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6821","nonce":2706983780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6825","nonce":2706983780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6823","nonce":2706983780}]},"public_addr":"172.21.15.17:6819/2706983780","cluster_addr":"172.21.15.17:6821/2706983780","heartbeat_back_addr":"172.21.15.17:6825/2706983780","heartbeat_front_addr":"172.21.15.17:6823/2706983780","state":["exists","up"]},{"osd":3,"uuid":"37609e4c-d0cb-464f-a3c6-2d6e530149e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":1004,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6827","nonce":3512825275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6829","nonce":3512825275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6833","nonce":3512825275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6831","nonce":3512825275}]},"public_addr":"172.21.15.17:6827/3512825275","cluster_addr":"172.21.15.17:6829/3512825275","heartbeat_back_addr":"172.21.15.17:6833/3512825275","heartbeat_front_addr":"172.21.15.17:6831/3512825275","state":["exists","up"]},{"osd":4,"uuid":"bd9e473c-bf74-4621-92d0-7031d7db10e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1006,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6801","nonce":146827020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6803","nonce":146827020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6807","nonce":146827020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6805","nonce":146827020}]},"public_addr":"172.21.15.71:6801/146827020","cluster_addr":"172.21.15.71:6803/146827020","heartbeat_back_addr":"172.21.15.71:6807/146827020","heartbeat_front_addr":"172.21.15.71:6805/146827020","state":["exists","up"]},{"osd":5,"uuid":"f08464f8-610f-4775-87d0-803b569eb211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":772,"up_from":796,"up_thru":1003,"down_at":773,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":501265341},{"type":"v1","addr":"172.21.15.71:6809","nonce":501265341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":501265341},{"type":"v1","addr":"172.21.15.71:6811","nonce":501265341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":501265341},{"type":"v1","addr":"172.21.15.71:6815","nonce":501265341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":501265341},{"type":"v1","addr":"172.21.15.71:6813","nonce":501265341}]},"public_addr":"172.21.15.71:6809/501265341","cluster_addr":"172.21.15.71:6811/501265341","heartbeat_back_addr":"172.21.15.71:6815/501265341","heartbeat_front_addr":"172.21.15.71:6813/501265341","state":["exists","up"]},{"osd":6,"uuid":"d05170c6-0258-43da-b81b-cc9cc64b5d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":1000,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6817","nonce":2384780628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6819","nonce":2384780628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6823","nonce":2384780628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6821","nonce":2384780628}]},"public_addr":"172.21.15.71:6817/2384780628","cluster_addr":"172.21.15.71:6819/2384780628","heartbeat_back_addr":"172.21.15.71:6823/2384780628","heartbeat_front_addr":"172.21.15.71:6821/2384780628","state":["exists","up"]},{"osd":7,"uuid":"0d34db33-5d4d-4761-887d-cf1a63dbf3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":54,"last_clean_end":132,"up_from":154,"up_thru":1046,"down_at":133,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3827414222},{"type":"v1","addr":"172.21.15.71:6825","nonce":3827414222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3827414222},{"type":"v1","addr":"172.21.15.71:6827","nonce":3827414222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3827414222},{"type":"v1","addr":"172.21.15.71:6831","nonce":3827414222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3827414222},{"type":"v1","addr":"172.21.15.71:6829","nonce":3827414222}]},"public_addr":"172.21.15.71:6825/3827414222","cluster_addr":"172.21.15.71:6827/3827414222","heartbeat_back_addr":"172.21.15.71:6831/3827414222","heartbeat_front_addr":"172.21.15.71:6829/3827414222","state":["exists","up"]},{"osd":8,"uuid":"f419836d-0416-4789-80ad-a45ae2ccf035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":953,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6801","nonce":527945865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6803","nonce":527945865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6807","nonce":527945865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6805","nonce":527945865}]},"public_addr":"172.21.15.173:6801/527945865","cluster_addr":"172.21.15.173:6803/527945865","heartbeat_back_addr":"172.21.15.173:6807/527945865","heartbeat_front_addr":"172.21.15.173:6805/527945865","state":["exists","up"]},{"osd":9,"uuid":"9dcaa1d8-e921-427c-8a2c-a368658a5bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":1005,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6809","nonce":3589162352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6811","nonce":3589162352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6815","nonce":3589162352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6813","nonce":3589162352}]},"public_addr":"172.21.15.173:6809/3589162352","cluster_addr":"172.21.15.173:6811/3589162352","heartbeat_back_addr":"172.21.15.173:6815/3589162352","heartbeat_front_addr":"172.21.15.173:6813/3589162352","state":["exists","up"]},{"osd":10,"uuid":"ee08b73f-f02b-4795-87bc-ab559b7faf61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":75,"last_clean_end":740,"up_from":791,"up_thru":986,"down_at":741,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2988338661},{"type":"v1","addr":"172.21.15.173:6817","nonce":2988338661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2988338661},{"type":"v1","addr":"172.21.15.173:6819","nonce":2988338661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2988338661},{"type":"v1","addr":"172.21.15.173:6823","nonce":2988338661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2988338661},{"type":"v1","addr":"172.21.15.173:6821","nonce":2988338661}]},"public_addr":"172.21.15.173:6817/2988338661","cluster_addr":"172.21.15.173:6819/2988338661","heartbeat_back_addr":"172.21.15.173:6823/2988338661","heartbeat_front_addr":"172.21.15.173:6821/2988338661","state":["exists","up"]},{"osd":11,"uuid":"ce178d21-377e-40ac-90fa-46009901114d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":1043,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6825","nonce":1731820377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6827","nonce":1731820377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6831","nonce":1731820377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6829","nonce":1731820377}]},"public_addr":"172.21.15.173:6825/1731820377","cluster_addr":"172.21.15.173:6827/1731820377","heartbeat_back_addr":"172.21.15.173:6831/1731820377","heartbeat_front_addr":"172.21.15.173:6829/1731820377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:13.071265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:31.915720+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:28.092360+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:35.000796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:47.184562+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:46.103418+0000","dead_epoch":772},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:18.203280+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:17.645883+0000","dead_epoch":132},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:29.315752+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:45:15.732576+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:45.962110+0000","dead_epoch":740},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:13.800409+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[4,6,8]}],"pg_upmap_items":[{"pgid":"1.0","mappings":[{"from":6,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/3205448177":"2024-09-18T16:03:49.423969+0000","172.21.15.17:6801/250768364":"2024-09-18T16:03:49.423969+0000","172.21.15.17:6800/250768364":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/394168293":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/4225461221":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/3672343901":"2024-09-18T16:03:21.713930+0000","172.21.15.17:0/419827434":"2024-09-18T16:03:21.713930+0000","172.21.15.17:6801/365103448":"2024-09-18T16:03:21.713930+0000","172.21.15.17:6800/365103448":"2024-09-18T16:03:21.713930+0000","172.21.15.17:0/2688765754":"2024-09-18T16:03:21.713930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:45:18.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:18 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/818364910' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:45:18.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:18 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/818364910' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:45:18.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:18 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/818364910' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:45:18.690 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-17T16:45:18.690 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T16:45:18.951 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:19.382 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:19 smithi017 ceph-mon[27571]: pgmap v2285: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:19.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:19 smithi071 ceph-mon[30002]: pgmap v2285: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:19.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:19 smithi173 ceph-mon[30530]: pgmap v2285: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:19.665 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:45:19.666 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":1051,"fsid":"19341aec-750e-11ef-bceb-c7b262605968","created":"2024-09-17T16:02:46.978716+0000","modified":"2024-09-17T16:45:16.154401+0000","last_up_change":"2024-09-17T16:36:51.015941+0000","last_in_change":"2024-09-17T16:42:58.555013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":29,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:06:06.088375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1019","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"882","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.2699999809265137,"score_stable":3.2699999809265137,"optimal_score":1,"raw_score_acting":3.2699999809265137,"raw_score_stable":3.2699999809265137,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"139c41b7-e38b-42d2-ae0b-70b94041dbfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":884,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6803","nonce":3081446730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6805","nonce":3081446730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6809","nonce":3081446730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":3081446730},{"type":"v1","addr":"172.21.15.17:6807","nonce":3081446730}]},"public_addr":"172.21.15.17:6803/3081446730","cluster_addr":"172.21.15.17:6805/3081446730","heartbeat_back_addr":"172.21.15.17:6809/3081446730","heartbeat_front_addr":"172.21.15.17:6807/3081446730","state":["exists","up"]},{"osd":1,"uuid":"ffa42caf-fd6a-4ff4-bf4d-4b6bca735fe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":988,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6811","nonce":603840826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6813","nonce":603840826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6817","nonce":603840826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":603840826},{"type":"v1","addr":"172.21.15.17:6815","nonce":603840826}]},"public_addr":"172.21.15.17:6811/603840826","cluster_addr":"172.21.15.17:6813/603840826","heartbeat_back_addr":"172.21.15.17:6817/603840826","heartbeat_front_addr":"172.21.15.17:6815/603840826","state":["exists","up"]},{"osd":2,"uuid":"efb6c3f1-6157-4581-a39d-bda978aea6b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":991,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6819","nonce":2706983780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6821","nonce":2706983780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6825","nonce":2706983780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":2706983780},{"type":"v1","addr":"172.21.15.17:6823","nonce":2706983780}]},"public_addr":"172.21.15.17:6819/2706983780","cluster_addr":"172.21.15.17:6821/2706983780","heartbeat_back_addr":"172.21.15.17:6825/2706983780","heartbeat_front_addr":"172.21.15.17:6823/2706983780","state":["exists","up"]},{"osd":3,"uuid":"37609e4c-d0cb-464f-a3c6-2d6e530149e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":1004,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6827","nonce":3512825275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6829","nonce":3512825275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6833","nonce":3512825275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":3512825275},{"type":"v1","addr":"172.21.15.17:6831","nonce":3512825275}]},"public_addr":"172.21.15.17:6827/3512825275","cluster_addr":"172.21.15.17:6829/3512825275","heartbeat_back_addr":"172.21.15.17:6833/3512825275","heartbeat_front_addr":"172.21.15.17:6831/3512825275","state":["exists","up"]},{"osd":4,"uuid":"bd9e473c-bf74-4621-92d0-7031d7db10e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1006,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6801","nonce":146827020}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6803","nonce":146827020}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6807","nonce":146827020}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":146827020},{"type":"v1","addr":"172.21.15.71:6805","nonce":146827020}]},"public_addr":"172.21.15.71:6801/146827020","cluster_addr":"172.21.15.71:6803/146827020","heartbeat_back_addr":"172.21.15.71:6807/146827020","heartbeat_front_addr":"172.21.15.71:6805/146827020","state":["exists","up"]},{"osd":5,"uuid":"f08464f8-610f-4775-87d0-803b569eb211","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":772,"up_from":796,"up_thru":1003,"down_at":773,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":501265341},{"type":"v1","addr":"172.21.15.71:6809","nonce":501265341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":501265341},{"type":"v1","addr":"172.21.15.71:6811","nonce":501265341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":501265341},{"type":"v1","addr":"172.21.15.71:6815","nonce":501265341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":501265341},{"type":"v1","addr":"172.21.15.71:6813","nonce":501265341}]},"public_addr":"172.21.15.71:6809/501265341","cluster_addr":"172.21.15.71:6811/501265341","heartbeat_back_addr":"172.21.15.71:6815/501265341","heartbeat_front_addr":"172.21.15.71:6813/501265341","state":["exists","up"]},{"osd":6,"uuid":"d05170c6-0258-43da-b81b-cc9cc64b5d31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":1000,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6817","nonce":2384780628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6819","nonce":2384780628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6823","nonce":2384780628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2384780628},{"type":"v1","addr":"172.21.15.71:6821","nonce":2384780628}]},"public_addr":"172.21.15.71:6817/2384780628","cluster_addr":"172.21.15.71:6819/2384780628","heartbeat_back_addr":"172.21.15.71:6823/2384780628","heartbeat_front_addr":"172.21.15.71:6821/2384780628","state":["exists","up"]},{"osd":7,"uuid":"0d34db33-5d4d-4761-887d-cf1a63dbf3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":54,"last_clean_end":132,"up_from":154,"up_thru":1046,"down_at":133,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3827414222},{"type":"v1","addr":"172.21.15.71:6825","nonce":3827414222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3827414222},{"type":"v1","addr":"172.21.15.71:6827","nonce":3827414222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3827414222},{"type":"v1","addr":"172.21.15.71:6831","nonce":3827414222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3827414222},{"type":"v1","addr":"172.21.15.71:6829","nonce":3827414222}]},"public_addr":"172.21.15.71:6825/3827414222","cluster_addr":"172.21.15.71:6827/3827414222","heartbeat_back_addr":"172.21.15.71:6831/3827414222","heartbeat_front_addr":"172.21.15.71:6829/3827414222","state":["exists","up"]},{"osd":8,"uuid":"f419836d-0416-4789-80ad-a45ae2ccf035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":953,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6801","nonce":527945865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6803","nonce":527945865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6807","nonce":527945865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":527945865},{"type":"v1","addr":"172.21.15.173:6805","nonce":527945865}]},"public_addr":"172.21.15.173:6801/527945865","cluster_addr":"172.21.15.173:6803/527945865","heartbeat_back_addr":"172.21.15.173:6807/527945865","heartbeat_front_addr":"172.21.15.173:6805/527945865","state":["exists","up"]},{"osd":9,"uuid":"9dcaa1d8-e921-427c-8a2c-a368658a5bfe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":1005,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6809","nonce":3589162352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6811","nonce":3589162352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6815","nonce":3589162352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3589162352},{"type":"v1","addr":"172.21.15.173:6813","nonce":3589162352}]},"public_addr":"172.21.15.173:6809/3589162352","cluster_addr":"172.21.15.173:6811/3589162352","heartbeat_back_addr":"172.21.15.173:6815/3589162352","heartbeat_front_addr":"172.21.15.173:6813/3589162352","state":["exists","up"]},{"osd":10,"uuid":"ee08b73f-f02b-4795-87bc-ab559b7faf61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":75,"last_clean_end":740,"up_from":791,"up_thru":986,"down_at":741,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2988338661},{"type":"v1","addr":"172.21.15.173:6817","nonce":2988338661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2988338661},{"type":"v1","addr":"172.21.15.173:6819","nonce":2988338661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2988338661},{"type":"v1","addr":"172.21.15.173:6823","nonce":2988338661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2988338661},{"type":"v1","addr":"172.21.15.173:6821","nonce":2988338661}]},"public_addr":"172.21.15.173:6817/2988338661","cluster_addr":"172.21.15.173:6819/2988338661","heartbeat_back_addr":"172.21.15.173:6823/2988338661","heartbeat_front_addr":"172.21.15.173:6821/2988338661","state":["exists","up"]},{"osd":11,"uuid":"ce178d21-377e-40ac-90fa-46009901114d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":83,"up_thru":1043,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6825","nonce":1731820377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6827","nonce":1731820377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6831","nonce":1731820377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":1731820377},{"type":"v1","addr":"172.21.15.173:6829","nonce":1731820377}]},"public_addr":"172.21.15.173:6825/1731820377","cluster_addr":"172.21.15.173:6827/1731820377","heartbeat_back_addr":"172.21.15.173:6831/1731820377","heartbeat_front_addr":"172.21.15.173:6829/1731820377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:13.071265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:31.915720+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:28.092360+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:35.000796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:47.184562+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:46.103418+0000","dead_epoch":772},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:18.203280+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:17.645883+0000","dead_epoch":132},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:29.315752+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:45:15.732576+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:45.962110+0000","dead_epoch":740},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:44:13.800409+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[4,6,8]}],"pg_upmap_items":[{"pgid":"1.0","mappings":[{"from":6,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/3205448177":"2024-09-18T16:03:49.423969+0000","172.21.15.17:6801/250768364":"2024-09-18T16:03:49.423969+0000","172.21.15.17:6800/250768364":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/394168293":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/4225461221":"2024-09-18T16:03:49.423969+0000","172.21.15.17:0/3672343901":"2024-09-18T16:03:21.713930+0000","172.21.15.17:0/419827434":"2024-09-18T16:03:21.713930+0000","172.21.15.17:6801/365103448":"2024-09-18T16:03:21.713930+0000","172.21.15.17:6800/365103448":"2024-09-18T16:03:21.713930+0000","172.21.15.17:0/2688765754":"2024-09-18T16:03:21.713930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:45:20.437 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:20 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1988693268' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:45:20.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-17T16:45:20.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-17T16:45:20.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-17T16:45:20.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-17T16:45:20.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-17T16:45:20.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-17T16:45:20.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-17T16:45:20.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-17T16:45:20.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-17T16:45:20.478 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-17T16:45:20.478 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-17T16:45:20.478 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-17T16:45:20.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:20 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1988693268' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:45:20.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:20 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1988693268' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-17T16:45:21.277 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:21.278 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:21.278 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:21.286 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:21.286 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:21.339 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:21.339 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:21.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:21 smithi173 ceph-mon[30530]: pgmap v2286: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:21.472 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:21 smithi017 ceph-mon[27571]: pgmap v2286: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:21.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:21 smithi071 ceph-mon[30002]: pgmap v2286: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:23.151 INFO:teuthology.orchestra.run.smithi017.stdout:296352743866 2024-09-17T16:45:23.151 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-17T16:45:23.173 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:23.174 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:23.174 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:23.174 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:23.175 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:23.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:23 smithi017 ceph-mon[27571]: pgmap v2287: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:23.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:23 smithi071 ceph-mon[30002]: pgmap v2287: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:23.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:23 smithi173 ceph-mon[30530]: pgmap v2287: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:25.524 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:25 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:45:25.524 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:25 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:45:25.524 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:25 smithi017 ceph-mon[27571]: pgmap v2288: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:25.524 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:25 smithi017 ceph-mon[27571]: osdmap e1052: 12 total, 12 up, 12 in 2024-09-17T16:45:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:25 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:45:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:25 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:45:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:25 smithi071 ceph-mon[30002]: pgmap v2288: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:25.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:25 smithi071 ceph-mon[30002]: osdmap e1052: 12 total, 12 up, 12 in 2024-09-17T16:45:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:25 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:45:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:25 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:45:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:25 smithi173 ceph-mon[30530]: pgmap v2288: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:25.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:25 smithi173 ceph-mon[30530]: osdmap e1052: 12 total, 12 up, 12 in 2024-09-17T16:45:26.409 INFO:teuthology.orchestra.run.smithi017.stdout:3418793967720 2024-09-17T16:45:26.410 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-17T16:45:27.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:27 smithi071 ceph-mon[30002]: pgmap v2290: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:27.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:27 smithi173 ceph-mon[30530]: pgmap v2290: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:27.653 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:27 smithi017 ceph-mon[27571]: pgmap v2290: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:27.689 INFO:teuthology.orchestra.run.smithi017.stdout:3397319131243 2024-09-17T16:45:27.690 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-17T16:45:28.808 INFO:teuthology.orchestra.run.smithi017.stdout:661424963979 2024-09-17T16:45:28.808 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-17T16:45:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:29 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:45:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:29 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:45:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:29 smithi017 ceph-mon[27571]: pgmap v2291: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:29 smithi017 ceph-mon[27571]: osdmap e1053: 12 total, 12 up, 12 in 2024-09-17T16:45:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:29 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:45:29.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:29 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:45:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:29 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:45:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:29 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:45:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:29 smithi071 ceph-mon[30002]: pgmap v2291: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:29 smithi071 ceph-mon[30002]: osdmap e1053: 12 total, 12 up, 12 in 2024-09-17T16:45:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:29 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:45:29.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:29 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:45:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:29 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:45:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:29 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:45:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:29 smithi173 ceph-mon[30530]: pgmap v2291: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:29 smithi173 ceph-mon[30530]: osdmap e1053: 12 total, 12 up, 12 in 2024-09-17T16:45:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:29 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:45:29.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:29 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:45:30.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:30 smithi017 ceph-mon[27571]: osdmap e1054: 12 total, 12 up, 12 in 2024-09-17T16:45:30.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:30 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:45:30.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:30 smithi071 ceph-mon[30002]: osdmap e1054: 12 total, 12 up, 12 in 2024-09-17T16:45:30.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:30 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:45:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:30 smithi173 ceph-mon[30530]: osdmap e1054: 12 total, 12 up, 12 in 2024-09-17T16:45:30.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:30 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:45:31.019 INFO:teuthology.orchestra.run.smithi017.stdout:133143986645 2024-09-17T16:45:31.019 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-17T16:45:31.474 INFO:teuthology.orchestra.run.smithi017.stdout:111669150166 2024-09-17T16:45:31.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-17T16:45:31.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:31 smithi071 ceph-mon[30002]: pgmap v2294: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:31.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:31 smithi173 ceph-mon[30530]: pgmap v2294: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:31.739 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:31 smithi017 ceph-mon[27571]: pgmap v2294: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:32.128 INFO:teuthology.orchestra.run.smithi017.stdout:356482286001 2024-09-17T16:45:32.128 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-17T16:45:33.172 INFO:teuthology.orchestra.run.smithi017.stdout:34359738856 2024-09-17T16:45:33.173 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-17T16:45:33.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:33 smithi017 ceph-mon[27571]: pgmap v2295: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:33.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:33 smithi071 ceph-mon[30002]: pgmap v2295: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:33.618 INFO:teuthology.orchestra.run.smithi017.stdout:197568496077 2024-09-17T16:45:33.618 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-17T16:45:33.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:33 smithi173 ceph-mon[30530]: pgmap v2295: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:34.060 INFO:teuthology.orchestra.run.smithi017.stdout:261993005504 2024-09-17T16:45:34.061 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-17T16:45:34.612 INFO:teuthology.orchestra.run.smithi017.stdout:55834575331 2024-09-17T16:45:34.613 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-17T16:45:34.726 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:34.727 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:34.727 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:34.729 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:34.729 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:34.730 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:34.733 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:34.738 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:34.872 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:34.872 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:34 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:34.872 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:34 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:45:34.872 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:34 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:45:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:34 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:34 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:45:35.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:34 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:45:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:34 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:34 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:45:35.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:34 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:45:35.189 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:35.203 INFO:teuthology.orchestra.run.smithi017.stdout:77309411805 2024-09-17T16:45:35.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-17T16:45:35.241 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:35.761 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:35 smithi017 ceph-mon[27571]: pgmap v2296: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:35.762 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:35 smithi017 ceph-mon[27571]: osdmap e1055: 12 total, 12 up, 12 in 2024-09-17T16:45:35.762 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:35 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:45:35.762 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:35 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:45:35.762 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:35 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:35 smithi071 ceph-mon[30002]: pgmap v2296: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:35 smithi071 ceph-mon[30002]: osdmap e1055: 12 total, 12 up, 12 in 2024-09-17T16:45:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:35 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:45:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:35 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:45:36.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:35 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:35 smithi173 ceph-mon[30530]: pgmap v2296: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:35 smithi173 ceph-mon[30530]: osdmap e1055: 12 total, 12 up, 12 in 2024-09-17T16:45:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:35 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:45:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:35 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:45:36.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:35 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:36.512 INFO:teuthology.orchestra.run.smithi017.stdout:34359738857 2024-09-17T16:45:36.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:36 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4238980691' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-17T16:45:37.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:36 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4238980691' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-17T16:45:37.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:36 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4238980691' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-17T16:45:37.886 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:37 smithi017 ceph-mon[27571]: pgmap v2298: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:37.886 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1837642365' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-17T16:45:37.886 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:37 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2531481689' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-17T16:45:37.919 INFO:teuthology.orchestra.run.smithi017.stdout:261993005506 2024-09-17T16:45:38.005 INFO:teuthology.orchestra.run.smithi017.stdout:661424963980 2024-09-17T16:45:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:37 smithi071 ceph-mon[30002]: pgmap v2298: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1837642365' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-17T16:45:38.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:37 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2531481689' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-17T16:45:38.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:37 smithi173 ceph-mon[30530]: pgmap v2298: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:38.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1837642365' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-17T16:45:38.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:37 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2531481689' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-17T16:45:38.710 INFO:tasks.cephadm.ceph_manager.ceph:need seq 661424963979 got 661424963980 for osd.7 2024-09-17T16:45:38.711 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:38.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:38 smithi017 ceph-mon[27571]: 1.1 scrub starts 2024-09-17T16:45:38.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:38 smithi017 ceph-mon[27571]: 1.1 scrub ok 2024-09-17T16:45:38.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1867981866' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-17T16:45:38.928 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:38 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4063038644' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-17T16:45:39.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:38 smithi173 ceph-mon[30530]: 1.1 scrub starts 2024-09-17T16:45:39.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:38 smithi173 ceph-mon[30530]: 1.1 scrub ok 2024-09-17T16:45:39.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1867981866' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-17T16:45:39.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:38 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4063038644' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-17T16:45:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:38 smithi071 ceph-mon[30002]: 1.1 scrub starts 2024-09-17T16:45:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:38 smithi071 ceph-mon[30002]: 1.1 scrub ok 2024-09-17T16:45:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1867981866' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-17T16:45:39.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:38 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4063038644' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-17T16:45:39.276 INFO:teuthology.orchestra.run.smithi017.stdout:3397319131246 2024-09-17T16:45:39.632 INFO:teuthology.orchestra.run.smithi017.stdout:296352743868 2024-09-17T16:45:39.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:39 smithi017 ceph-mon[27571]: pgmap v2299: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:39.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:39 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/579664897' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-17T16:45:40.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:39 smithi071 ceph-mon[30002]: pgmap v2299: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:40.064 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:39 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/579664897' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-17T16:45:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:39 smithi173 ceph-mon[30530]: pgmap v2299: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:40.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:39 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/579664897' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-17T16:45:40.831 INFO:teuthology.orchestra.run.smithi017.stdout:111669150168 2024-09-17T16:45:40.882 INFO:teuthology.orchestra.run.smithi017.stdout:3418793967723 2024-09-17T16:45:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:40 smithi173 ceph-mon[30530]: 1.3 scrub starts 2024-09-17T16:45:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:40 smithi173 ceph-mon[30530]: 1.3 scrub ok 2024-09-17T16:45:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:40 smithi173 ceph-mon[30530]: 1.8 scrub starts 2024-09-17T16:45:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:40 smithi173 ceph-mon[30530]: 1.8 scrub ok 2024-09-17T16:45:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:40 smithi173 ceph-mon[30530]: pgmap v2300: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:41.145 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:40 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/521150574' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-17T16:45:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:40 smithi017 ceph-mon[27571]: 1.3 scrub starts 2024-09-17T16:45:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:40 smithi017 ceph-mon[27571]: 1.3 scrub ok 2024-09-17T16:45:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:40 smithi017 ceph-mon[27571]: 1.8 scrub starts 2024-09-17T16:45:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:40 smithi017 ceph-mon[27571]: 1.8 scrub ok 2024-09-17T16:45:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:40 smithi017 ceph-mon[27571]: pgmap v2300: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:41.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:40 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/521150574' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-17T16:45:41.254 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743866 got 296352743868 for osd.9 2024-09-17T16:45:41.254 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:41.313 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:40 smithi071 ceph-mon[30002]: 1.3 scrub starts 2024-09-17T16:45:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:40 smithi071 ceph-mon[30002]: 1.3 scrub ok 2024-09-17T16:45:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:40 smithi071 ceph-mon[30002]: 1.8 scrub starts 2024-09-17T16:45:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:40 smithi071 ceph-mon[30002]: 1.8 scrub ok 2024-09-17T16:45:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:40 smithi071 ceph-mon[30002]: pgmap v2300: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:41.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:40 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/521150574' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-17T16:45:42.034 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738856 got 34359738857 for osd.0 2024-09-17T16:45:42.034 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:42.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:42 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:45:42.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:42 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:45:42.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:42 smithi017 ceph-mon[27571]: osdmap e1056: 12 total, 12 up, 12 in 2024-09-17T16:45:42.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:42 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:45:42.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:42 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:45:42.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:42 smithi173 ceph-mon[30530]: osdmap e1056: 12 total, 12 up, 12 in 2024-09-17T16:45:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:42 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:45:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:42 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:45:42.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:42 smithi071 ceph-mon[30002]: osdmap e1056: 12 total, 12 up, 12 in 2024-09-17T16:45:42.580 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3397319131243 got 3397319131246 for osd.10 2024-09-17T16:45:42.580 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:42.909 INFO:teuthology.orchestra.run.smithi017.stdout:356482286004 2024-09-17T16:45:43.098 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005504 got 261993005506 for osd.8 2024-09-17T16:45:43.099 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:43.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:43 smithi017 ceph-mon[27571]: pgmap v2302: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:43.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:43 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:45:43.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:43 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:45:43.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:43 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/4002569630' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-17T16:45:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:43 smithi071 ceph-mon[30002]: pgmap v2302: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:43 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:45:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:43 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:45:43.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:43 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/4002569630' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-17T16:45:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:43 smithi173 ceph-mon[30530]: pgmap v2302: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:43 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:45:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:43 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:45:43.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:43 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/4002569630' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-17T16:45:43.701 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150166 got 111669150168 for osd.3 2024-09-17T16:45:43.702 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:44.087 INFO:teuthology.orchestra.run.smithi017.stdout:197568496080 2024-09-17T16:45:44.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:44 smithi017 ceph-mon[27571]: osdmap e1057: 12 total, 12 up, 12 in 2024-09-17T16:45:44.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:44 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/1618569355' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-17T16:45:44.339 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3418793967720 got 3418793967723 for osd.5 2024-09-17T16:45:44.339 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:44.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:44 smithi071 ceph-mon[30002]: osdmap e1057: 12 total, 12 up, 12 in 2024-09-17T16:45:44.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:44 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/1618569355' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-17T16:45:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:44 smithi173 ceph-mon[30530]: osdmap e1057: 12 total, 12 up, 12 in 2024-09-17T16:45:44.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:44 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/1618569355' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-17T16:45:45.187 INFO:teuthology.orchestra.run.smithi017.stdout:133143986648 2024-09-17T16:45:45.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:45 smithi017 ceph-mon[27571]: pgmap v2304: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:45.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:45 smithi071 ceph-mon[30002]: pgmap v2304: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:45.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:45 smithi173 ceph-mon[30530]: pgmap v2304: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:45.785 INFO:tasks.cephadm.ceph_manager.ceph:need seq 356482286001 got 356482286004 for osd.11 2024-09-17T16:45:45.785 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:46.238 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568496077 got 197568496080 for osd.6 2024-09-17T16:45:46.238 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:46 smithi017 ceph-mon[27571]: 1.a scrub starts 2024-09-17T16:45:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:46 smithi017 ceph-mon[27571]: 1.a scrub ok 2024-09-17T16:45:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:46 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/944180976' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-17T16:45:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:46 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:45:46.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:46 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:45:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:46 smithi071 ceph-mon[30002]: 1.a scrub starts 2024-09-17T16:45:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:46 smithi071 ceph-mon[30002]: 1.a scrub ok 2024-09-17T16:45:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:46 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/944180976' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-17T16:45:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:46 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:45:46.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:46 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:45:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:46 smithi173 ceph-mon[30530]: 1.a scrub starts 2024-09-17T16:45:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:46 smithi173 ceph-mon[30530]: 1.a scrub ok 2024-09-17T16:45:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:46 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/944180976' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-17T16:45:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:46 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:45:46.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:46 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:45:46.866 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986645 got 133143986648 for osd.4 2024-09-17T16:45:46.866 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:46.869 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:46.870 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:47.415 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:47 smithi017 ceph-mon[27571]: osdmap e1058: 12 total, 12 up, 12 in 2024-09-17T16:45:47.415 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:47 smithi017 ceph-mon[27571]: pgmap v2306: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:47.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:47 smithi071 ceph-mon[30002]: osdmap e1058: 12 total, 12 up, 12 in 2024-09-17T16:45:47.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:47 smithi071 ceph-mon[30002]: pgmap v2306: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:47.622 INFO:teuthology.orchestra.run.smithi017.stdout:77309411808 2024-09-17T16:45:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:47 smithi173 ceph-mon[30530]: osdmap e1058: 12 total, 12 up, 12 in 2024-09-17T16:45:47.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:47 smithi173 ceph-mon[30530]: pgmap v2306: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:48.239 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:48 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/917514417' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-17T16:45:48.249 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-17T16:45:48.246+0000 7f9ce77fe640 0 --2- 172.21.15.17:0/203068172 >> [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] conn(0x7f9ce81021a0 0x7f9ce819b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T16:45:48.447 INFO:teuthology.orchestra.run.smithi017.stdout:55834575334 2024-09-17T16:45:48.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:48 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/917514417' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-17T16:45:48.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:48 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/917514417' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-17T16:45:48.660 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411805 got 77309411808 for osd.2 2024-09-17T16:45:48.660 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:49.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:49 smithi017 ceph-mon[27571]: 1.5 scrub starts 2024-09-17T16:45:49.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:49 smithi017 ceph-mon[27571]: 1.5 scrub ok 2024-09-17T16:45:49.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:49 smithi017 ceph-mon[27571]: pgmap v2307: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:49.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:49 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/203068172' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-17T16:45:49.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:49 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:45:49.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:49 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:45:49.383 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575331 got 55834575334 for osd.1 2024-09-17T16:45:49.383 DEBUG:teuthology.parallel:result is None 2024-09-17T16:45:49.383 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T16:45:49.383 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:45:49.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:49 smithi071 ceph-mon[30002]: 1.5 scrub starts 2024-09-17T16:45:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:49 smithi071 ceph-mon[30002]: 1.5 scrub ok 2024-09-17T16:45:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:49 smithi071 ceph-mon[30002]: pgmap v2307: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:49 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/203068172' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-17T16:45:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:49 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:45:49.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:49 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:45:49.642 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:49.644 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:49 smithi173 ceph-mon[30530]: 1.5 scrub starts 2024-09-17T16:45:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:49 smithi173 ceph-mon[30530]: 1.5 scrub ok 2024-09-17T16:45:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:49 smithi173 ceph-mon[30530]: pgmap v2307: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:49 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/203068172' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-17T16:45:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:49 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:45:49.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:49 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:45:50.316 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:50 smithi017 ceph-mon[27571]: osdmap e1059: 12 total, 12 up, 12 in 2024-09-17T16:45:50.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:45:50.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:45:50.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:45:50.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:45:50.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:45:50.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:45:50.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:50 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:50.318 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:45:50.318 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:45:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:50 smithi071 ceph-mon[30002]: osdmap e1059: 12 total, 12 up, 12 in 2024-09-17T16:45:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:45:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:45:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:45:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:45:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:45:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:45:50.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:50 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:50 smithi173 ceph-mon[30530]: osdmap e1059: 12 total, 12 up, 12 in 2024-09-17T16:45:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-17T16:45:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-17T16:45:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-17T16:45:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:45:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:45:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:45:50.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:50 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:51.083 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":2309,"stamp":"2024-09-17T16:45:50.224603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8290580,"kb_used_data":10512,"kb_used_omap":85,"kb_used_meta":8279722,"kb_avail":1116651244,"statfs":{"total":1151940427776,"available":1143450873856,"internally_reserved":0,"allocated":10764288,"data_stored":6962942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87602,"internal_metadata":8478435790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.078817"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":936,"reported_epoch":1059,"state":"active+clean","last_fresh":"2024-09-17T16:45:49.171186+0000","last_change":"2024-09-17T16:45:01.386198+0000","last_active":"2024-09-17T16:45:49.171186+0000","last_peered":"2024-09-17T16:45:49.171186+0000","last_clean":"2024-09-17T16:45:49.171186+0000","last_became_active":"2024-09-17T16:39:31.793364+0000","last_became_peered":"2024-09-17T16:39:31.793364+0000","last_unstale":"2024-09-17T16:45:49.171186+0000","last_undegraded":"2024-09-17T16:45:49.171186+0000","last_fullsized":"2024-09-17T16:45:49.171186+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:01.386133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:01.386133+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:08.038561+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":416,"reported_epoch":1059,"state":"active+clean","last_fresh":"2024-09-17T16:45:49.773797+0000","last_change":"2024-09-17T16:45:40.106455+0000","last_active":"2024-09-17T16:45:49.773797+0000","last_peered":"2024-09-17T16:45:49.773797+0000","last_clean":"2024-09-17T16:45:49.773797+0000","last_became_active":"2024-09-17T16:42:59.573697+0000","last_became_peered":"2024-09-17T16:42:59.573697+0000","last_unstale":"2024-09-17T16:45:49.773797+0000","last_undegraded":"2024-09-17T16:45:49.773797+0000","last_fullsized":"2024-09-17T16:45:49.773797+0000","mapping_epoch":968,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":969,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:40.106391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:40.106391+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:52.901807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":402,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.159138+0000","last_change":"2024-09-17T16:45:15.257030+0000","last_active":"2024-09-17T16:45:46.159138+0000","last_peered":"2024-09-17T16:45:46.159138+0000","last_clean":"2024-09-17T16:45:46.159138+0000","last_became_active":"2024-09-17T16:39:33.798293+0000","last_became_peered":"2024-09-17T16:39:33.798293+0000","last_unstale":"2024-09-17T16:45:46.159138+0000","last_undegraded":"2024-09-17T16:45:46.159138+0000","last_fullsized":"2024-09-17T16:45:46.159138+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:15.256945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:15.256945+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:20.705675+0000","scrub_duration":816,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":396,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-17T16:45:45.029557+0000","last_change":"2024-09-17T16:45:45.029557+0000","last_active":"2024-09-17T16:45:45.029557+0000","last_peered":"2024-09-17T16:45:45.029557+0000","last_clean":"2024-09-17T16:45:45.029557+0000","last_became_active":"2024-09-17T16:39:33.792174+0000","last_became_peered":"2024-09-17T16:39:33.792174+0000","last_unstale":"2024-09-17T16:45:45.029557+0000","last_undegraded":"2024-09-17T16:45:45.029557+0000","last_fullsized":"2024-09-17T16:45:45.029557+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:45.029480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:45.029480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:56.416558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":396,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.159313+0000","last_change":"2024-09-17T16:45:36.967224+0000","last_active":"2024-09-17T16:45:46.159313+0000","last_peered":"2024-09-17T16:45:46.159313+0000","last_clean":"2024-09-17T16:45:46.159313+0000","last_became_active":"2024-09-17T16:39:33.801025+0000","last_became_peered":"2024-09-17T16:39:33.801025+0000","last_unstale":"2024-09-17T16:45:46.159313+0000","last_undegraded":"2024-09-17T16:45:46.159313+0000","last_fullsized":"2024-09-17T16:45:46.159313+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:36.967139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:36.967139+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:39.826194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":44,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.159121+0000","last_change":"2024-09-17T16:45:09.531437+0000","last_active":"2024-09-17T16:45:46.159121+0000","last_peered":"2024-09-17T16:45:46.159121+0000","last_clean":"2024-09-17T16:45:46.159121+0000","last_became_active":"2024-09-17T16:45:09.116800+0000","last_became_peered":"2024-09-17T16:45:09.116800+0000","last_unstale":"2024-09-17T16:45:46.159121+0000","last_undegraded":"2024-09-17T16:45:46.159121+0000","last_fullsized":"2024-09-17T16:45:46.159121+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:09.531372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:09.531372+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:13.384803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":400,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.862789+0000","last_change":"2024-09-17T16:45:39.471528+0000","last_active":"2024-09-17T16:45:46.862789+0000","last_peered":"2024-09-17T16:45:46.862789+0000","last_clean":"2024-09-17T16:45:46.862789+0000","last_became_active":"2024-09-17T16:39:33.824237+0000","last_became_peered":"2024-09-17T16:39:33.824237+0000","last_unstale":"2024-09-17T16:45:46.862789+0000","last_undegraded":"2024-09-17T16:45:46.862789+0000","last_fullsized":"2024-09-17T16:45:46.862789+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:39.471476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:39.471476+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:44.801964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":55,"reported_epoch":1059,"state":"active+clean","last_fresh":"2024-09-17T16:45:49.773447+0000","last_change":"2024-09-17T16:45:05.833039+0000","last_active":"2024-09-17T16:45:49.773447+0000","last_peered":"2024-09-17T16:45:49.773447+0000","last_clean":"2024-09-17T16:45:49.773447+0000","last_became_active":"2024-09-17T16:45:05.085233+0000","last_became_peered":"2024-09-17T16:45:05.085233+0000","last_unstale":"2024-09-17T16:45:49.773447+0000","last_undegraded":"2024-09-17T16:45:49.773447+0000","last_fullsized":"2024-09-17T16:45:49.773447+0000","mapping_epoch":1043,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":1044,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:05.832960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:05.832960+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:06.109411+0000","scrub_duration":604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":389,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.159329+0000","last_change":"2024-09-17T16:44:28.102580+0000","last_active":"2024-09-17T16:45:46.159329+0000","last_peered":"2024-09-17T16:45:46.159329+0000","last_clean":"2024-09-17T16:45:46.159329+0000","last_became_active":"2024-09-17T16:39:33.800521+0000","last_became_peered":"2024-09-17T16:39:33.800521+0000","last_unstale":"2024-09-17T16:45:46.159329+0000","last_undegraded":"2024-09-17T16:45:46.159329+0000","last_fullsized":"2024-09-17T16:45:46.159329+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:44:28.102516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:44:28.102516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:47.178083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":397,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.159329+0000","last_change":"2024-09-17T16:44:47.011248+0000","last_active":"2024-09-17T16:45:46.159329+0000","last_peered":"2024-09-17T16:45:46.159329+0000","last_clean":"2024-09-17T16:45:46.159329+0000","last_became_active":"2024-09-17T16:39:33.791607+0000","last_became_peered":"2024-09-17T16:39:33.791607+0000","last_unstale":"2024-09-17T16:45:46.159329+0000","last_undegraded":"2024-09-17T16:45:46.159329+0000","last_fullsized":"2024-09-17T16:45:46.159329+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:44:47.011188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:44:47.011188+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:04.895257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":422,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-17T16:45:43.217214+0000","last_change":"2024-09-17T16:44:30.296885+0000","last_active":"2024-09-17T16:45:43.217214+0000","last_peered":"2024-09-17T16:45:43.217214+0000","last_clean":"2024-09-17T16:45:43.217214+0000","last_became_active":"2024-09-17T16:42:59.573860+0000","last_became_peered":"2024-09-17T16:42:59.573860+0000","last_unstale":"2024-09-17T16:45:43.217214+0000","last_undegraded":"2024-09-17T16:45:43.217214+0000","last_fullsized":"2024-09-17T16:45:43.217214+0000","mapping_epoch":968,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":969,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:44:30.296833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:44:30.296833+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:00.190211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482286006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740292,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93004860,"statfs":{"total":95995035648,"available":95236976640,"internally_reserved":0,"allocated":1093632,"data_stored":776941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":756939204},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131248,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":362624,"kb_used_data":748,"kb_used_omap":15,"kb_used_meta":361840,"kb_avail":93382528,"statfs":{"total":95995035648,"available":95623708672,"internally_reserved":0,"allocated":765952,"data_stored":449261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":370524732},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948212,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92796940,"statfs":{"total":95995035648,"available":95024066560,"internally_reserved":0,"allocated":753664,"data_stored":441774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":970193344},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005509,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663936,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081216,"statfs":{"total":95995035648,"available":95315165184,"internally_reserved":0,"allocated":1093632,"data_stored":776941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963983,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865860,"kb_used_data":1068,"kb_used_omap":41,"kb_used_meta":864726,"kb_avail":92879292,"statfs":{"total":95995035648,"available":95108395008,"internally_reserved":0,"allocated":1093632,"data_stored":776941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":885479731},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496081,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641400,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93103752,"statfs":{"total":95995035648,"available":95338242048,"internally_reserved":0,"allocated":753664,"data_stored":441774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":656013766},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":286100,"kb_used_data":960,"kb_used_omap":14,"kb_used_meta":285105,"kb_avail":93459052,"statfs":{"total":95995035648,"available":95702069248,"internally_reserved":0,"allocated":983040,"data_stored":657996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":291948149},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986650,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":695564,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93049588,"statfs":{"total":95995035648,"available":95282778112,"internally_reserved":0,"allocated":778240,"data_stored":456748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150171,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826240,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92918912,"statfs":{"total":95995035648,"available":95148965888,"internally_reserved":0,"allocated":958464,"data_stored":643022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":845085127},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663816,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081336,"statfs":{"total":95995035648,"available":95315288064,"internally_reserved":0,"allocated":970752,"data_stored":650509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726080,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93019072,"statfs":{"total":95995035648,"available":95251529728,"internally_reserved":0,"allocated":765952,"data_stored":449261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870456,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92874696,"statfs":{"total":95995035648,"available":95103688704,"internally_reserved":0,"allocated":753664,"data_stored":441774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":890567109},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:45:51.084 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T16:45:51.341 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:51.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:51 smithi017 ceph-mon[27571]: pgmap v2309: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:51.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:51 smithi017 ceph-mon[27571]: from='client.19491 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:45:51.563 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:51 smithi071 ceph-mon[30002]: pgmap v2309: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:51.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:51 smithi071 ceph-mon[30002]: from='client.19491 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:45:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:51 smithi173 ceph-mon[30530]: pgmap v2309: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:51.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:51 smithi173 ceph-mon[30530]: from='client.19491 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:45:52.016 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-17T16:45:52.017 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-17T16:45:52.922 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":2309,"stamp":"2024-09-17T16:45:50.224603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8290580,"kb_used_data":10512,"kb_used_omap":85,"kb_used_meta":8279722,"kb_avail":1116651244,"statfs":{"total":1151940427776,"available":1143450873856,"internally_reserved":0,"allocated":10764288,"data_stored":6962942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87602,"internal_metadata":8478435790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.078817"},"pg_stats":[{"pgid":"1.0","version":"21'92","reported_seq":936,"reported_epoch":1059,"state":"active+clean","last_fresh":"2024-09-17T16:45:49.171186+0000","last_change":"2024-09-17T16:45:01.386198+0000","last_active":"2024-09-17T16:45:49.171186+0000","last_peered":"2024-09-17T16:45:49.171186+0000","last_clean":"2024-09-17T16:45:49.171186+0000","last_became_active":"2024-09-17T16:39:31.793364+0000","last_became_peered":"2024-09-17T16:39:31.793364+0000","last_unstale":"2024-09-17T16:45:49.171186+0000","last_undegraded":"2024-09-17T16:45:49.171186+0000","last_fullsized":"2024-09-17T16:45:49.171186+0000","mapping_epoch":637,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:01.386133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:01.386133+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:08.038561+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8],"acting":[4,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"21'92","reported_seq":416,"reported_epoch":1059,"state":"active+clean","last_fresh":"2024-09-17T16:45:49.773797+0000","last_change":"2024-09-17T16:45:40.106455+0000","last_active":"2024-09-17T16:45:49.773797+0000","last_peered":"2024-09-17T16:45:49.773797+0000","last_clean":"2024-09-17T16:45:49.773797+0000","last_became_active":"2024-09-17T16:42:59.573697+0000","last_became_peered":"2024-09-17T16:42:59.573697+0000","last_unstale":"2024-09-17T16:45:49.773797+0000","last_undegraded":"2024-09-17T16:45:49.773797+0000","last_fullsized":"2024-09-17T16:45:49.773797+0000","mapping_epoch":968,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":969,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:40.106391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:40.106391+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:52.901807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"21'92","reported_seq":402,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.159138+0000","last_change":"2024-09-17T16:45:15.257030+0000","last_active":"2024-09-17T16:45:46.159138+0000","last_peered":"2024-09-17T16:45:46.159138+0000","last_clean":"2024-09-17T16:45:46.159138+0000","last_became_active":"2024-09-17T16:39:33.798293+0000","last_became_peered":"2024-09-17T16:39:33.798293+0000","last_unstale":"2024-09-17T16:45:46.159138+0000","last_undegraded":"2024-09-17T16:45:46.159138+0000","last_fullsized":"2024-09-17T16:45:46.159138+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:15.256945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:15.256945+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:20.705675+0000","scrub_duration":816,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"21'92","reported_seq":396,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-17T16:45:45.029557+0000","last_change":"2024-09-17T16:45:45.029557+0000","last_active":"2024-09-17T16:45:45.029557+0000","last_peered":"2024-09-17T16:45:45.029557+0000","last_clean":"2024-09-17T16:45:45.029557+0000","last_became_active":"2024-09-17T16:39:33.792174+0000","last_became_peered":"2024-09-17T16:39:33.792174+0000","last_unstale":"2024-09-17T16:45:45.029557+0000","last_undegraded":"2024-09-17T16:45:45.029557+0000","last_fullsized":"2024-09-17T16:45:45.029557+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:45.029480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:45.029480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:56.416558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1","version":"21'92","reported_seq":396,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.159313+0000","last_change":"2024-09-17T16:45:36.967224+0000","last_active":"2024-09-17T16:45:46.159313+0000","last_peered":"2024-09-17T16:45:46.159313+0000","last_clean":"2024-09-17T16:45:46.159313+0000","last_became_active":"2024-09-17T16:39:33.801025+0000","last_became_peered":"2024-09-17T16:39:33.801025+0000","last_unstale":"2024-09-17T16:45:46.159313+0000","last_undegraded":"2024-09-17T16:45:46.159313+0000","last_fullsized":"2024-09-17T16:45:46.159313+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:36.967139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:36.967139+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:39.826194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"21'92","reported_seq":44,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.159121+0000","last_change":"2024-09-17T16:45:09.531437+0000","last_active":"2024-09-17T16:45:46.159121+0000","last_peered":"2024-09-17T16:45:46.159121+0000","last_clean":"2024-09-17T16:45:46.159121+0000","last_became_active":"2024-09-17T16:45:09.116800+0000","last_became_peered":"2024-09-17T16:45:09.116800+0000","last_unstale":"2024-09-17T16:45:46.159121+0000","last_undegraded":"2024-09-17T16:45:46.159121+0000","last_fullsized":"2024-09-17T16:45:46.159121+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":4,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:09.531372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:09.531372+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:13.384803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"21'92","reported_seq":400,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.862789+0000","last_change":"2024-09-17T16:45:39.471528+0000","last_active":"2024-09-17T16:45:46.862789+0000","last_peered":"2024-09-17T16:45:46.862789+0000","last_clean":"2024-09-17T16:45:46.862789+0000","last_became_active":"2024-09-17T16:39:33.824237+0000","last_became_peered":"2024-09-17T16:39:33.824237+0000","last_unstale":"2024-09-17T16:45:46.862789+0000","last_undegraded":"2024-09-17T16:45:46.862789+0000","last_fullsized":"2024-09-17T16:45:46.862789+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:39.471476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:39.471476+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:44.801964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"21'92","reported_seq":55,"reported_epoch":1059,"state":"active+clean","last_fresh":"2024-09-17T16:45:49.773447+0000","last_change":"2024-09-17T16:45:05.833039+0000","last_active":"2024-09-17T16:45:49.773447+0000","last_peered":"2024-09-17T16:45:49.773447+0000","last_clean":"2024-09-17T16:45:49.773447+0000","last_became_active":"2024-09-17T16:45:05.085233+0000","last_became_peered":"2024-09-17T16:45:05.085233+0000","last_unstale":"2024-09-17T16:45:49.773447+0000","last_undegraded":"2024-09-17T16:45:49.773447+0000","last_fullsized":"2024-09-17T16:45:49.773447+0000","mapping_epoch":1043,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":1044,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:45:05.832960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:45:05.832960+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:06.109411+0000","scrub_duration":604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"21'92","reported_seq":389,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.159329+0000","last_change":"2024-09-17T16:44:28.102580+0000","last_active":"2024-09-17T16:45:46.159329+0000","last_peered":"2024-09-17T16:45:46.159329+0000","last_clean":"2024-09-17T16:45:46.159329+0000","last_became_active":"2024-09-17T16:39:33.800521+0000","last_became_peered":"2024-09-17T16:39:33.800521+0000","last_unstale":"2024-09-17T16:45:46.159329+0000","last_undegraded":"2024-09-17T16:45:46.159329+0000","last_fullsized":"2024-09-17T16:45:46.159329+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:44:28.102516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:44:28.102516+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:45:47.178083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"21'92","reported_seq":397,"reported_epoch":1058,"state":"active+clean","last_fresh":"2024-09-17T16:45:46.159329+0000","last_change":"2024-09-17T16:44:47.011248+0000","last_active":"2024-09-17T16:45:46.159329+0000","last_peered":"2024-09-17T16:45:46.159329+0000","last_clean":"2024-09-17T16:45:46.159329+0000","last_became_active":"2024-09-17T16:39:33.791607+0000","last_became_peered":"2024-09-17T16:39:33.791607+0000","last_unstale":"2024-09-17T16:45:46.159329+0000","last_undegraded":"2024-09-17T16:45:46.159329+0000","last_fullsized":"2024-09-17T16:45:46.159329+0000","mapping_epoch":884,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":885,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:44:47.011188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:44:47.011188+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:04.895257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"21'92","reported_seq":422,"reported_epoch":1057,"state":"active+clean","last_fresh":"2024-09-17T16:45:43.217214+0000","last_change":"2024-09-17T16:44:30.296885+0000","last_active":"2024-09-17T16:45:43.217214+0000","last_peered":"2024-09-17T16:45:43.217214+0000","last_clean":"2024-09-17T16:45:43.217214+0000","last_became_active":"2024-09-17T16:42:59.573860+0000","last_became_peered":"2024-09-17T16:42:59.573860+0000","last_unstale":"2024-09-17T16:45:43.217214+0000","last_undegraded":"2024-09-17T16:45:43.217214+0000","last_fullsized":"2024-09-17T16:45:43.217214+0000","mapping_epoch":968,"log_start":"0'0","ondisk_log_start":"0'0","created":882,"last_epoch_clean":969,"parent":"0.0","parent_split_bits":3,"last_scrub":"21'92","last_scrub_stamp":"2024-09-17T16:44:30.296833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:06:07.057987+0000","last_clean_scrub_stamp":"2024-09-17T16:44:30.296833+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:46:00.190211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":83,"seq":356482286006,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740292,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93004860,"statfs":{"total":95995035648,"available":95236976640,"internally_reserved":0,"allocated":1093632,"data_stored":776941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":756939204},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":791,"seq":3397319131248,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":362624,"kb_used_data":748,"kb_used_omap":15,"kb_used_meta":361840,"kb_avail":93382528,"statfs":{"total":95995035648,"available":95623708672,"internally_reserved":0,"allocated":765952,"data_stored":449261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15812,"internal_metadata":370524732},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948212,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92796940,"statfs":{"total":95995035648,"available":95024066560,"internally_reserved":0,"allocated":753664,"data_stored":441774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":970193344},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005509,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663936,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081216,"statfs":{"total":95995035648,"available":95315165184,"internally_reserved":0,"allocated":1093632,"data_stored":776941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":154,"seq":661424963983,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865860,"kb_used_data":1068,"kb_used_omap":41,"kb_used_meta":864726,"kb_avail":92879292,"statfs":{"total":95995035648,"available":95108395008,"internally_reserved":0,"allocated":1093632,"data_stored":776941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42701,"internal_metadata":885479731},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496081,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641400,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93103752,"statfs":{"total":95995035648,"available":95338242048,"internally_reserved":0,"allocated":753664,"data_stored":441774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":656013766},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":796,"seq":3418793967726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":286100,"kb_used_data":960,"kb_used_omap":14,"kb_used_meta":285105,"kb_avail":93459052,"statfs":{"total":95995035648,"available":95702069248,"internally_reserved":0,"allocated":983040,"data_stored":657996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14731,"internal_metadata":291948149},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986650,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":695564,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93049588,"statfs":{"total":95995035648,"available":95282778112,"internally_reserved":0,"allocated":778240,"data_stored":456748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150171,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":826240,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92918912,"statfs":{"total":95995035648,"available":95148965888,"internally_reserved":0,"allocated":958464,"data_stored":643022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":845085127},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663816,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081336,"statfs":{"total":95995035648,"available":95315288064,"internally_reserved":0,"allocated":970752,"data_stored":650509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575335,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726080,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93019072,"statfs":{"total":95995035648,"available":95251529728,"internally_reserved":0,"allocated":765952,"data_stored":449261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870456,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92874696,"statfs":{"total":95995035648,"available":95103688704,"internally_reserved":0,"allocated":753664,"data_stored":441774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":890567109},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:45:52.923 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-17T16:45:52.923 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell --fsid 19341aec-750e-11ef-bceb-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-17T16:45:53.184 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T16:45:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:53 smithi071 ceph-mon[30002]: from='client.19497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:45:53.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:53 smithi071 ceph-mon[30002]: pgmap v2310: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:53.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:53 smithi017 ceph-mon[27571]: from='client.19497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:45:53.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:53 smithi017 ceph-mon[27571]: pgmap v2310: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:53 smithi173 ceph-mon[30530]: from='client.19497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T16:45:53.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:53 smithi173 ceph-mon[30530]: pgmap v2310: 11 pgs: 11 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-17T16:45:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:54 smithi071 ceph-mon[30002]: purged_snaps scrub starts 2024-09-17T16:45:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:54 smithi071 ceph-mon[30002]: purged_snaps scrub ok 2024-09-17T16:45:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2959013262' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-17T16:45:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:54 smithi071 ceph-mon[30002]: from='client.? 172.21.15.17:0/2959013262' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-17T16:45:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:54 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:45:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:54 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:45:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:54 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:45:54.564 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:54 smithi071 ceph-mon[30002]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:54.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:54 smithi017 ceph-mon[27571]: purged_snaps scrub starts 2024-09-17T16:45:54.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:54 smithi017 ceph-mon[27571]: purged_snaps scrub ok 2024-09-17T16:45:54.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2959013262' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-17T16:45:54.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:54 smithi017 ceph-mon[27571]: from='client.? 172.21.15.17:0/2959013262' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-17T16:45:54.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:45:54.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:45:54.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:45:54.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:54 smithi017 ceph-mon[27571]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:54.596 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-17T16:45:54.609 INFO:tasks.cephadm:Teardown begin 2024-09-17T16:45:54.609 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T16:45:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:54 smithi173 ceph-mon[30530]: purged_snaps scrub starts 2024-09-17T16:45:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:54 smithi173 ceph-mon[30530]: purged_snaps scrub ok 2024-09-17T16:45:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2959013262' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-17T16:45:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:54 smithi173 ceph-mon[30530]: from='client.? 172.21.15.17:0/2959013262' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-17T16:45:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:54 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-17T16:45:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:54 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-17T16:45:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:54 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-17T16:45:54.645 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:54 smithi173 ceph-mon[30530]: from='mgr.14152 172.21.15.17:0/3499461088' entity='mgr.y' 2024-09-17T16:45:54.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T16:45:54.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T16:45:54.758 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-17T16:45:54.758 DEBUG:teuthology.orchestra.run.smithi017:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T16:45:54.795 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T16:45:54.819 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T16:45:54.835 INFO:tasks.cephadm:Stopping all daemons... 2024-09-17T16:45:54.835 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-17T16:45:54.835 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.a 2024-09-17T16:45:55.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:54 smithi017 systemd[1]: Stopping Ceph mon.a for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:45:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:55 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:45:55.169+0000 7f086f888640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:45:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:55 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a[27567]: 2024-09-17T16:45:55.169+0000 7f086f888640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-17T16:45:55.489 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:55 smithi017 podman[272206]: 2024-09-17 16:45:55.38561294 +0000 UTC m=+0.339731633 container died efaebd570b8a107a6fa3d4b5460d96b81d9c8016b5481270e67557b7c83f2702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-17T16:45:55.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:55 smithi017 podman[272206]: 2024-09-17 16:45:55.862565419 +0000 UTC m=+0.816684154 container cleanup efaebd570b8a107a6fa3d4b5460d96b81d9c8016b5481270e67557b7c83f2702 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T16:45:55.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Sep 17 16:45:55 smithi017 bash[272206]: ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-a 2024-09-17T16:45:56.298 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.a.service' 2024-09-17T16:45:56.353 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:45:56.354 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-17T16:45:56.354 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-17T16:45:56.354 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.b 2024-09-17T16:45:56.694 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:56 smithi071 systemd[1]: Stopping Ceph mon.b for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:45:57.009 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:56 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-b[29998]: 2024-09-17T16:45:56.690+0000 7f0f5662e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:45:57.009 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:56 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-b[29998]: 2024-09-17T16:45:56.690+0000 7f0f5662e640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-17T16:45:57.009 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:56 smithi071 podman[171170]: 2024-09-17 16:45:56.762508636 +0000 UTC m=+0.196064213 container died e490eba52e0029a9da5a238aa2dbfb5a29a925d47ca6aa5193017f561bf9aa8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:45:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:57 smithi071 podman[171170]: 2024-09-17 16:45:57.224855424 +0000 UTC m=+0.658411046 container cleanup e490eba52e0029a9da5a238aa2dbfb5a29a925d47ca6aa5193017f561bf9aa8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-b, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-17T16:45:57.314 INFO:journalctl@ceph.mon.b.smithi071.stdout:Sep 17 16:45:57 smithi071 bash[171170]: ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-b 2024-09-17T16:45:57.701 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.b.service' 2024-09-17T16:45:57.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:45:57.738 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-17T16:45:57.739 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-17T16:45:57.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.c 2024-09-17T16:45:58.060 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:57 smithi173 systemd[1]: Stopping Ceph mon.c for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:45:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-c[30526]: 2024-09-17T16:45:58.057+0000 7ff608f04640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:45:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-c[30526]: 2024-09-17T16:45:58.057+0000 7ff608f04640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-17T16:45:58.395 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:58 smithi173 podman[170556]: 2024-09-17 16:45:58.282682835 +0000 UTC m=+0.347040574 container died c696b47c7aa0fc882b65025e8146a26a44bcec75f719aa3fb34d44b2670a1f57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:45:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:58 smithi173 podman[170556]: 2024-09-17 16:45:58.691516133 +0000 UTC m=+0.755873867 container cleanup c696b47c7aa0fc882b65025e8146a26a44bcec75f719aa3fb34d44b2670a1f57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-c, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T16:45:58.895 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:58 smithi173 bash[170556]: ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-c 2024-09-17T16:45:59.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.c.service' 2024-09-17T16:45:59.214 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:58 smithi173 podman[170568]: 2024-09-17 16:45:58.952060397 +0000 UTC m=+0.669395130 container remove c696b47c7aa0fc882b65025e8146a26a44bcec75f719aa3fb34d44b2670a1f57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:45:59.214 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:59 smithi173 systemd[1]: ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-17T16:45:59.214 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:59 smithi173 systemd[1]: Stopped Ceph mon.c for 19341aec-750e-11ef-bceb-c7b262605968. 2024-09-17T16:45:59.214 INFO:journalctl@ceph.mon.c.smithi173.stdout:Sep 17 16:45:59 smithi173 systemd[1]: ceph-19341aec-750e-11ef-bceb-c7b262605968@mon.c.service: Consumed 29.171s CPU time. 2024-09-17T16:45:59.229 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:45:59.229 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-17T16:45:59.230 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-17T16:45:59.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@mgr.y 2024-09-17T16:45:59.527 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:45:59 smithi017 systemd[1]: Stopping Ceph mgr.y for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:45:59.793 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:45:59 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y[27799]: 2024-09-17T16:45:59.524+0000 7f38aa3e7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:45:59.793 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:45:59 smithi017 podman[272328]: 2024-09-17 16:45:59.556492096 +0000 UTC m=+0.132400937 container died aa2411db0b87de2a8648b0c540be0ee93cc39c58d800eba44bfe475830341620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T16:46:00.220 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:45:59 smithi017 podman[272328]: 2024-09-17 16:45:59.967453846 +0000 UTC m=+0.543362678 container cleanup aa2411db0b87de2a8648b0c540be0ee93cc39c58d800eba44bfe475830341620 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-17T16:46:00.221 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Sep 17 16:45:59 smithi017 bash[272328]: ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-y 2024-09-17T16:46:00.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-17T16:46:00.398 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:46:00.399 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-17T16:46:00.399 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-17T16:46:00.399 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@mgr.x 2024-09-17T16:46:00.734 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:46:00 smithi071 systemd[1]: Stopping Ceph mgr.x for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:46:01.006 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:46:00 smithi071 podman[171292]: 2024-09-17 16:46:00.733214481 +0000 UTC m=+0.141434259 container died dc4de72e6389fa10fde978cb3016af0c574850d3fcefbf06a5411e99a412817c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T16:46:01.313 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:46:01 smithi071 podman[171292]: 2024-09-17 16:46:01.10412276 +0000 UTC m=+0.512342535 container cleanup dc4de72e6389fa10fde978cb3016af0c574850d3fcefbf06a5411e99a412817c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-09-17T16:46:01.314 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Sep 17 16:46:01 smithi071 bash[171292]: ceph-19341aec-750e-11ef-bceb-c7b262605968-mgr-x 2024-09-17T16:46:01.570 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-17T16:46:01.604 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:46:01.604 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-17T16:46:01.604 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-17T16:46:01.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.0 2024-09-17T16:46:01.958 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:01 smithi017 systemd[1]: Stopping Ceph osd.0 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:46:02.239 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:01 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0[39749]: 2024-09-17T16:46:01.954+0000 7f8075b71640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:46:02.239 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:01 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0[39749]: 2024-09-17T16:46:01.954+0000 7f8075b71640 -1 osd.0 1060 *** Got signal Terminated *** 2024-09-17T16:46:02.239 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:01 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0[39749]: 2024-09-17T16:46:01.954+0000 7f8075b71640 -1 osd.0 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:46:07.825 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:07 smithi017 podman[272460]: 2024-09-17 16:46:07.533495893 +0000 UTC m=+5.730387666 container died 34e07b1056db2369fc879ee13fcbdf7969c6cc4d73e0f49f613b3c64a1e0a28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:46:08.238 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:08 smithi017 podman[272460]: 2024-09-17 16:46:08.006362844 +0000 UTC m=+6.203254692 container cleanup 34e07b1056db2369fc879ee13fcbdf7969c6cc4d73e0f49f613b3c64a1e0a28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-17T16:46:08.239 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:08 smithi017 bash[272460]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0 2024-09-17T16:46:08.643 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:08 smithi017 podman[272475]: 2024-09-17 16:46:08.324826348 +0000 UTC m=+0.785041939 container remove 34e07b1056db2369fc879ee13fcbdf7969c6cc4d73e0f49f613b3c64a1e0a28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:46:08.980 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:08 smithi017 podman[272564]: 2024-09-17 16:46:08.544681808 +0000 UTC m=+0.016581472 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:46:08.980 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:08 smithi017 podman[272564]: 2024-09-17 16:46:08.667349565 +0000 UTC m=+0.139249205 container create 8a858475dc0c4975e29f2b00a692746f53850181535d8ed95ab25bd8b0d522b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, ceph=True) 2024-09-17T16:46:08.980 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:08 smithi017 podman[272564]: 2024-09-17 16:46:08.869650419 +0000 UTC m=+0.341550077 container init 8a858475dc0c4975e29f2b00a692746f53850181535d8ed95ab25bd8b0d522b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T16:46:08.980 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:08 smithi017 podman[272564]: 2024-09-17 16:46:08.873022883 +0000 UTC m=+0.344922522 container start 8a858475dc0c4975e29f2b00a692746f53850181535d8ed95ab25bd8b0d522b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T16:46:09.239 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:08 smithi017 podman[272564]: 2024-09-17 16:46:08.978221529 +0000 UTC m=+0.450121167 container attach 8a858475dc0c4975e29f2b00a692746f53850181535d8ed95ab25bd8b0d522b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:46:09.239 INFO:journalctl@ceph.osd.0.smithi017.stdout:Sep 17 16:46:09 smithi017 podman[272564]: 2024-09-17 16:46:09.218789193 +0000 UTC m=+0.690688843 container died 8a858475dc0c4975e29f2b00a692746f53850181535d8ed95ab25bd8b0d522b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.license=GPLv2) 2024-09-17T16:46:09.935 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.0.service' 2024-09-17T16:46:09.978 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:46:09.978 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-17T16:46:09.979 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-17T16:46:09.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.1 2024-09-17T16:46:10.239 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:10 smithi017 systemd[1]: Stopping Ceph osd.1 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:46:10.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:10 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1[45329]: 2024-09-17T16:46:10.334+0000 7fb45431c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:46:10.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:10 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1[45329]: 2024-09-17T16:46:10.334+0000 7fb45431c640 -1 osd.1 1060 *** Got signal Terminated *** 2024-09-17T16:46:10.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:10 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1[45329]: 2024-09-17T16:46:10.334+0000 7fb45431c640 -1 osd.1 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:46:16.139 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:15 smithi017 podman[272670]: 2024-09-17 16:46:15.864473864 +0000 UTC m=+5.659492331 container died af36a2e0d772ee2d092d693d382995a990dbc7773dd800478133536f3dfdfcea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16) 2024-09-17T16:46:16.489 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:16 smithi017 podman[272670]: 2024-09-17 16:46:16.334965039 +0000 UTC m=+6.129983509 container cleanup af36a2e0d772ee2d092d693d382995a990dbc7773dd800478133536f3dfdfcea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:46:16.489 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:16 smithi017 bash[272670]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1 2024-09-17T16:46:16.989 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:16 smithi017 podman[272686]: 2024-09-17 16:46:16.670241278 +0000 UTC m=+0.806731283 container remove af36a2e0d772ee2d092d693d382995a990dbc7773dd800478133536f3dfdfcea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16) 2024-09-17T16:46:17.377 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:17 smithi017 podman[272774]: 2024-09-17 16:46:17.029260742 +0000 UTC m=+0.110598904 container create dc4d051d2d86f5906b761c90586d4e72c6040a82387c599019a4f58bb3115b72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True) 2024-09-17T16:46:17.377 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:17 smithi017 podman[272774]: 2024-09-17 16:46:16.935128094 +0000 UTC m=+0.016466257 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:46:17.378 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:17 smithi017 podman[272774]: 2024-09-17 16:46:17.273219558 +0000 UTC m=+0.354557728 container init dc4d051d2d86f5906b761c90586d4e72c6040a82387c599019a4f58bb3115b72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:46:17.378 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:17 smithi017 podman[272774]: 2024-09-17 16:46:17.276620497 +0000 UTC m=+0.357958659 container start dc4d051d2d86f5906b761c90586d4e72c6040a82387c599019a4f58bb3115b72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=) 2024-09-17T16:46:17.378 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:17 smithi017 podman[272774]: 2024-09-17 16:46:17.375789408 +0000 UTC m=+0.457127590 container attach dc4d051d2d86f5906b761c90586d4e72c6040a82387c599019a4f58bb3115b72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-17T16:46:17.739 INFO:journalctl@ceph.osd.1.smithi017.stdout:Sep 17 16:46:17 smithi017 podman[272774]: 2024-09-17 16:46:17.473647972 +0000 UTC m=+0.554986135 container died dc4d051d2d86f5906b761c90586d4e72c6040a82387c599019a4f58bb3115b72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-09-17T16:46:18.111 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.1.service' 2024-09-17T16:46:18.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:46:18.160 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-17T16:46:18.160 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-17T16:46:18.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.2 2024-09-17T16:46:18.398 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:18 smithi017 systemd[1]: Stopping Ceph osd.2 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:46:18.739 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:18 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2[50593]: 2024-09-17T16:46:18.546+0000 7f25ab8b9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:46:18.739 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:18 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2[50593]: 2024-09-17T16:46:18.546+0000 7f25ab8b9640 -1 osd.2 1060 *** Got signal Terminated *** 2024-09-17T16:46:18.739 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:18 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2[50593]: 2024-09-17T16:46:18.546+0000 7f25ab8b9640 -1 osd.2 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:46:24.383 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:24 smithi017 podman[272877]: 2024-09-17 16:46:24.111883119 +0000 UTC m=+5.732257877 container died fecf16cfa968098c0597e478fb855e80963b4f34ef0274f0460eaed4c413e138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:46:24.739 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:24 smithi017 podman[272877]: 2024-09-17 16:46:24.48034802 +0000 UTC m=+6.100722765 container cleanup fecf16cfa968098c0597e478fb855e80963b4f34ef0274f0460eaed4c413e138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-09-17T16:46:24.739 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:24 smithi017 bash[272877]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2 2024-09-17T16:46:25.109 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:24 smithi017 podman[272905]: 2024-09-17 16:46:24.756671679 +0000 UTC m=+0.638378963 container remove fecf16cfa968098c0597e478fb855e80963b4f34ef0274f0460eaed4c413e138 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-17T16:46:25.428 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:25 smithi017 podman[272992]: 2024-09-17 16:46:25.010442231 +0000 UTC m=+0.016393352 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:46:25.429 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:25 smithi017 podman[272992]: 2024-09-17 16:46:25.132543299 +0000 UTC m=+0.138494408 container create b33de0211cd86e982168c093214b4175cd802e05f13ccb33ffc2424889ee9316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:46:25.429 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:25 smithi017 podman[272992]: 2024-09-17 16:46:25.343156655 +0000 UTC m=+0.349107764 container init b33de0211cd86e982168c093214b4175cd802e05f13ccb33ffc2424889ee9316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:46:25.429 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:25 smithi017 podman[272992]: 2024-09-17 16:46:25.346549466 +0000 UTC m=+0.352500578 container start b33de0211cd86e982168c093214b4175cd802e05f13ccb33ffc2424889ee9316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-17T16:46:25.429 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:25 smithi017 podman[272992]: 2024-09-17 16:46:25.426710594 +0000 UTC m=+0.432661714 container attach b33de0211cd86e982168c093214b4175cd802e05f13ccb33ffc2424889ee9316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-17T16:46:25.739 INFO:journalctl@ceph.osd.2.smithi017.stdout:Sep 17 16:46:25 smithi017 podman[272992]: 2024-09-17 16:46:25.539110183 +0000 UTC m=+0.545061292 container died b33de0211cd86e982168c093214b4175cd802e05f13ccb33ffc2424889ee9316 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-17T16:46:26.106 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.2.service' 2024-09-17T16:46:26.150 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:46:26.150 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-17T16:46:26.150 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-17T16:46:26.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.3 2024-09-17T16:46:26.485 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:26 smithi017 systemd[1]: Stopping Ceph osd.3 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:46:26.739 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:26 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3[56373]: 2024-09-17T16:46:26.482+0000 7fdb990b2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:46:26.739 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:26 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3[56373]: 2024-09-17T16:46:26.482+0000 7fdb990b2640 -1 osd.3 1060 *** Got signal Terminated *** 2024-09-17T16:46:26.739 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:26 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3[56373]: 2024-09-17T16:46:26.482+0000 7fdb990b2640 -1 osd.3 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:46:29.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:28 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:28.713+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:29.814 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:29 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:29.498+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.904000+0000 front 2024-09-17T16:46:04.903905+0000 (oldest deadline 2024-09-17T16:46:29.003976+0000) 2024-09-17T16:46:30.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:29 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:29.672+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:30.294 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:29 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:29.995+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:30.489 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:30 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3[56373]: 2024-09-17T16:46:30.108+0000 7fdb956c7640 -1 osd.3 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.817252+0000 front 2024-09-17T16:46:04.817269+0000 (oldest deadline 2024-09-17T16:46:29.417930+0000) 2024-09-17T16:46:30.563 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:30 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:30.515+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.904000+0000 front 2024-09-17T16:46:04.903905+0000 (oldest deadline 2024-09-17T16:46:29.003976+0000) 2024-09-17T16:46:30.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:30 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:30.293+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:31.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:30.668+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:31.250 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:30 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:30.953+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:31.489 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:31 smithi017 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3[56373]: 2024-09-17T16:46:31.117+0000 7fdb956c7640 -1 osd.3 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.817252+0000 front 2024-09-17T16:46:04.817269+0000 (oldest deadline 2024-09-17T16:46:29.417930+0000) 2024-09-17T16:46:31.558 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:31 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:31.248+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:31.813 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:31 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:31.556+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.904000+0000 front 2024-09-17T16:46:04.903905+0000 (oldest deadline 2024-09-17T16:46:29.003976+0000) 2024-09-17T16:46:31.894 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:31 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:31.641+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:32.276 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:32 smithi017 podman[273095]: 2024-09-17 16:46:32.009322502 +0000 UTC m=+5.630778153 container died 3005472de41a558f2c22e68075d2a3bf4c5a5d75a5250787049a67f0729d3227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T16:46:32.286 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:31 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:31.966+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:32.395 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:32 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:32.303+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:32.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:32 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:32.107+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:32.542 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:32 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:32.284+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:32.739 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:32 smithi017 podman[273095]: 2024-09-17 16:46:32.483426083 +0000 UTC m=+6.104881720 container cleanup 3005472de41a558f2c22e68075d2a3bf4c5a5d75a5250787049a67f0729d3227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux ) 2024-09-17T16:46:32.739 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:32 smithi017 bash[273095]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3 2024-09-17T16:46:32.814 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:32 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:32.540+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.904000+0000 front 2024-09-17T16:46:04.903905+0000 (oldest deadline 2024-09-17T16:46:29.003976+0000) 2024-09-17T16:46:33.073 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:32 smithi017 podman[273114]: 2024-09-17 16:46:32.759093946 +0000 UTC m=+0.750838964 container remove 3005472de41a558f2c22e68075d2a3bf4c5a5d75a5250787049a67f0729d3227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-17T16:46:33.081 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:32 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:32.664+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:33.314 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:32 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:32.926+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:33.332 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:33.078+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:33.389 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:33 smithi017 podman[273204]: 2024-09-17 16:46:32.974187001 +0000 UTC m=+0.016209119 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:46:33.389 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:33 smithi017 podman[273204]: 2024-09-17 16:46:33.093299117 +0000 UTC m=+0.135321222 container create 5b92ccc0b8e6117c71c47944a04ab012a1d393feb6579914a5061332d50ff0b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-17T16:46:33.389 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:33 smithi017 podman[273204]: 2024-09-17 16:46:33.295581394 +0000 UTC m=+0.337603498 container init 5b92ccc0b8e6117c71c47944a04ab012a1d393feb6579914a5061332d50ff0b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-17T16:46:33.389 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:33 smithi017 podman[273204]: 2024-09-17 16:46:33.299014688 +0000 UTC m=+0.341036798 container start 5b92ccc0b8e6117c71c47944a04ab012a1d393feb6579914a5061332d50ff0b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3-deactivate, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731) 2024-09-17T16:46:33.389 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:33 smithi017 podman[273204]: 2024-09-17 16:46:33.387298332 +0000 UTC m=+0.429320450 container attach 5b92ccc0b8e6117c71c47944a04ab012a1d393feb6579914a5061332d50ff0b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-17T16:46:33.644 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:33.330+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:33.739 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:33 smithi017 podman[273204]: 2024-09-17 16:46:33.48944544 +0000 UTC m=+0.531467555 container died 5b92ccc0b8e6117c71c47944a04ab012a1d393feb6579914a5061332d50ff0b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-17T16:46:33.814 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:33 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:33.315+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:33.814 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:33 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:33.532+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.904000+0000 front 2024-09-17T16:46:04.903905+0000 (oldest deadline 2024-09-17T16:46:29.003976+0000) 2024-09-17T16:46:34.021 INFO:journalctl@ceph.osd.3.smithi017.stdout:Sep 17 16:46:33 smithi017 podman[273226]: 2024-09-17 16:46:33.998679725 +0000 UTC m=+0.500608341 container remove 5b92ccc0b8e6117c71c47944a04ab012a1d393feb6579914a5061332d50ff0b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-17T16:46:34.027 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.3.service' 2024-09-17T16:46:34.066 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:33.674+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:34.067 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:46:34.067 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-17T16:46:34.067 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-17T16:46:34.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.4 2024-09-17T16:46:34.281 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:34 smithi071 systemd[1]: Stopping Ceph osd.4 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:46:34.281 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:33 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:33.976+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:34.319 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:34 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:34.064+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:34.562 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:34 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:34.278+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:34.562 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:34 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:34.387+0000 7f94169d0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:46:34.562 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:34 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:34.387+0000 7f94169d0640 -1 osd.4 1060 *** Got signal Terminated *** 2024-09-17T16:46:34.562 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:34 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:34.387+0000 7f94169d0640 -1 osd.4 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:46:34.639 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:34 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:34.317+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:34.813 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:34 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:34.559+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.904000+0000 front 2024-09-17T16:46:04.903905+0000 (oldest deadline 2024-09-17T16:46:29.003976+0000) 2024-09-17T16:46:34.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:34 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:34.638+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:35.296 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:34 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:34.996+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:35.328 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:35 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:35.030+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:35.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:35 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:35.293+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:35.564 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:35 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:35.527+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.904000+0000 front 2024-09-17T16:46:04.903905+0000 (oldest deadline 2024-09-17T16:46:29.003976+0000) 2024-09-17T16:46:35.610 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:35 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:35.325+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:35.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:35 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:35.608+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:36.310 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:35 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:35.993+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:36.365 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:36 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:36.048+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:36.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:36 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:36.308+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:36.564 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:36 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:36.534+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.904000+0000 front 2024-09-17T16:46:04.903905+0000 (oldest deadline 2024-09-17T16:46:29.003976+0000) 2024-09-17T16:46:36.564 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:36 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:36.534+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:10.705059+0000 front 2024-09-17T16:46:10.705052+0000 (oldest deadline 2024-09-17T16:46:36.505610+0000) 2024-09-17T16:46:36.624 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:36 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:36.361+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:36.895 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:36 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:36.798+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:36.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:36 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:36.621+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:36.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:36 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:36.621+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:37.270 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:36 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:36.973+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:37.380 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:37 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:37.001+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:37.381 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:37 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:37.001+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:37.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:37 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:37.268+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:37.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:37 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:37.377+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:37.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:37 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:37.377+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:37.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:37 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:37.581+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:37.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:37 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:37.581+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:37.989 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:37 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:37.566+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.904000+0000 front 2024-09-17T16:46:04.903905+0000 (oldest deadline 2024-09-17T16:46:29.003976+0000) 2024-09-17T16:46:37.989 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:37 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:37.566+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:10.705059+0000 front 2024-09-17T16:46:10.705052+0000 (oldest deadline 2024-09-17T16:46:36.505610+0000) 2024-09-17T16:46:38.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:37 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:37.829+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:38.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:38 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:38.048+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:38.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:38 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:38.048+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:38.272 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:37 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:37.986+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:38.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:38 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:38.270+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:38.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:38 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:38.270+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:38.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:38 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:38.373+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:38.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:38 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:38.373+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:38.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:38 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:38.618+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:38.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:38 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:38.618+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:39.012 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:38 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:38.560+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:04.904000+0000 front 2024-09-17T16:46:04.903905+0000 (oldest deadline 2024-09-17T16:46:29.003976+0000) 2024-09-17T16:46:39.012 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:38 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4[33712]: 2024-09-17T16:46:38.560+0000 7f940efdd640 -1 osd.4 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:10.705059+0000 front 2024-09-17T16:46:10.705052+0000 (oldest deadline 2024-09-17T16:46:36.505610+0000) 2024-09-17T16:46:39.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:38 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:38.790+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:39.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:39 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:39.017+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:39.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:39 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:39.017+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:39.314 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:39 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:39.008+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:39.573 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:39 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:39.315+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:39.573 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:39 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:39.315+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:39.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:39 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:39.384+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:39.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:39 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:39.384+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:39.842 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:39 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:39.814+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.021836+0000 front 2024-09-17T16:46:15.021946+0000 (oldest deadline 2024-09-17T16:46:39.121848+0000) 2024-09-17T16:46:40.053 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:39 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:39.780+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:40.053 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:39 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:39.648+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:40.054 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:39 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:39.648+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:40.098 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:39 smithi071 podman[171429]: 2024-09-17 16:46:39.840289068 +0000 UTC m=+5.579480343 container died a407652e2efee21deac101801bafa2ba6631f8741367a7f51b6786ce7273cb0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:46:40.098 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:39 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:39.977+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:40.363 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:40 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:40.051+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:40.363 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:40 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:40.051+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:40.365 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:40 smithi071 podman[171429]: 2024-09-17 16:46:40.1871624 +0000 UTC m=+5.926353674 container cleanup a407652e2efee21deac101801bafa2ba6631f8741367a7f51b6786ce7273cb0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-17T16:46:40.365 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:40 smithi071 bash[171429]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4 2024-09-17T16:46:40.365 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:40 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:40.281+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:40.365 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:40 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:40.281+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:40.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:40 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:40.361+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:40.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:40 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:40.361+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:40.799 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:40 smithi071 podman[171453]: 2024-09-17 16:46:40.363496723 +0000 UTC m=+0.517782684 container remove a407652e2efee21deac101801bafa2ba6631f8741367a7f51b6786ce7273cb0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16) 2024-09-17T16:46:41.039 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:40 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:40.756+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:41.039 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:40 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:40.650+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:41.040 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:40 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:40.650+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:41.051 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:40 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:40.803+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.021836+0000 front 2024-09-17T16:46:15.021946+0000 (oldest deadline 2024-09-17T16:46:39.121848+0000) 2024-09-17T16:46:41.052 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:41.026+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:41.052 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:40 smithi071 podman[171549]: 2024-09-17 16:46:40.700698197 +0000 UTC m=+0.016048387 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:46:41.052 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:40 smithi071 podman[171549]: 2024-09-17 16:46:40.822982208 +0000 UTC m=+0.138332389 container create b10941586d18e5e5cd975905935f91af289179b22aa6f5f5fbb98eed1f777ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-17T16:46:41.052 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:41 smithi071 podman[171549]: 2024-09-17 16:46:41.050320835 +0000 UTC m=+0.365671018 container init b10941586d18e5e5cd975905935f91af289179b22aa6f5f5fbb98eed1f777ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux ) 2024-09-17T16:46:41.314 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:41 smithi071 podman[171549]: 2024-09-17 16:46:41.05410692 +0000 UTC m=+0.369457104 container start b10941586d18e5e5cd975905935f91af289179b22aa6f5f5fbb98eed1f777ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T16:46:41.314 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:41 smithi071 podman[171549]: 2024-09-17 16:46:41.142218258 +0000 UTC m=+0.457568531 container attach b10941586d18e5e5cd975905935f91af289179b22aa6f5f5fbb98eed1f777ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16) 2024-09-17T16:46:41.314 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:41.275+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:41.315 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:41.275+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:41.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:41 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:41.037+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:41.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:41 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:41.037+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:41.674 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:41 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:41.401+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:41.674 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:41 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:41.401+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:41.755 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:41 smithi071 podman[171549]: 2024-09-17 16:46:41.396379092 +0000 UTC m=+0.711729276 container died b10941586d18e5e5cd975905935f91af289179b22aa6f5f5fbb98eed1f777ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-17T16:46:42.006 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:41 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:41.824+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.021836+0000 front 2024-09-17T16:46:15.021946+0000 (oldest deadline 2024-09-17T16:46:39.121848+0000) 2024-09-17T16:46:42.006 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:41 smithi071 podman[171572]: 2024-09-17 16:46:41.987191367 +0000 UTC m=+0.581043654 container remove b10941586d18e5e5cd975905935f91af289179b22aa6f5f5fbb98eed1f777ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=) 2024-09-17T16:46:42.006 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:42 smithi071 systemd[1]: ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.4.service: Deactivated successfully. 2024-09-17T16:46:42.006 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:42 smithi071 systemd[1]: Stopped Ceph osd.4 for 19341aec-750e-11ef-bceb-c7b262605968. 2024-09-17T16:46:42.007 INFO:journalctl@ceph.osd.4.smithi071.stdout:Sep 17 16:46:42 smithi071 systemd[1]: ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.4.service: Consumed 20min 12.340s CPU time. 2024-09-17T16:46:42.007 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.4.service' 2024-09-17T16:46:42.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:46:42.047 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-17T16:46:42.047 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-17T16:46:42.047 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.5 2024-09-17T16:46:42.073 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:41 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:41.788+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:42.073 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:41 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:41.672+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:42.073 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:41 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:41.672+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:42.293 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:42 smithi071 systemd[1]: Stopping Ceph osd.5 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:46:42.293 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:42 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:42.026+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:42.371 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:42 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:42.071+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:42.371 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:42 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:42.071+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:42.563 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:42 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:42.405+0000 7f1f99cdd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:46:42.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:42 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:42.405+0000 7f1f99cdd640 -1 osd.5 1060 *** Got signal Terminated *** 2024-09-17T16:46:42.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:42 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:42.405+0000 7f1f99cdd640 -1 osd.5 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:46:42.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:42 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:42.290+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:42.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:42 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:42.290+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:42.625 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:42 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:42.368+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:42.625 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:42 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:42.368+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:42.895 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:42 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:42.818+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:42.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:42 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:42.623+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:42.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:42 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:42.623+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:43.309 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:42 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:42.840+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.021836+0000 front 2024-09-17T16:46:15.021946+0000 (oldest deadline 2024-09-17T16:46:39.121848+0000) 2024-09-17T16:46:43.310 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:43 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:43.039+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:43.367 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:43 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:43.049+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:43.367 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:43 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:43.049+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:43.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:43 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:43.306+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:43.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:43 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:43.306+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:43.648 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:43 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:43.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:43.648 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:43 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:43.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:43.648 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:43 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:43.365+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:43.648 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:43 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:43.365+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:44.078 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:43 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:43.793+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:44.262 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:44 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:44.043+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:44.262 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:43 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:43.830+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.021836+0000 front 2024-09-17T16:46:15.021946+0000 (oldest deadline 2024-09-17T16:46:39.121848+0000) 2024-09-17T16:46:44.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:44 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:44.076+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:44.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:44 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:44.076+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:44.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:44 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:44.315+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:44.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:44 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:44.315+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:44.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:44 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:44.259+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:44.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:44 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:44.259+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:44.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:44 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:44.620+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:44.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:44 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:44.620+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:44.895 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:44 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:44.833+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:44.895 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:44 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:44.833+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:45.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:45 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:45.004+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:45.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:45 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:45.004+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:45.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:44 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:44.793+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.021836+0000 front 2024-09-17T16:46:15.021946+0000 (oldest deadline 2024-09-17T16:46:39.121848+0000) 2024-09-17T16:46:45.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:44 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:44.793+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.523501+0000 front 2024-09-17T16:46:22.523333+0000 (oldest deadline 2024-09-17T16:46:44.223301+0000) 2024-09-17T16:46:45.366 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:45 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:45.099+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:45.366 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:45 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:45.099+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:45.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:45 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:45.308+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:45.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:45 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:45.308+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:45.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:45 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:45.308+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:45.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:45 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:45.594+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:45.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:45 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:45.594+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:45.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:45 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:45.594+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:45.646 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:45 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:45.363+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:45.646 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:45 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:45.363+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:45.646 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:45 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:45.363+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:46.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:45 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:45.968+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:46.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:45 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:45.968+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:46.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:45 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:45.790+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.021836+0000 front 2024-09-17T16:46:15.021946+0000 (oldest deadline 2024-09-17T16:46:39.121848+0000) 2024-09-17T16:46:46.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:45 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:45.790+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.523501+0000 front 2024-09-17T16:46:22.523333+0000 (oldest deadline 2024-09-17T16:46:44.223301+0000) 2024-09-17T16:46:46.112 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:45 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:45.793+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:46.112 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:45 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:45.793+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:46.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:46 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:46.109+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:46.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:46 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:46.109+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:46.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:46 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:46.266+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:46.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:46 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:46.266+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:46.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:46 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:46.266+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:46.826 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:46 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:46.398+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:46.826 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:46 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:46.398+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:46.826 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:46 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:46.398+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:46.827 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:46 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:46.550+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:46.827 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:46 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:46.550+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:46.827 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:46 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:46.550+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:47.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:46 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:46.994+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:47.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:46 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:46.994+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:47.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:46 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:46.746+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.021836+0000 front 2024-09-17T16:46:15.021946+0000 (oldest deadline 2024-09-17T16:46:39.121848+0000) 2024-09-17T16:46:47.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:46 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5[142795]: 2024-09-17T16:46:46.746+0000 7f1f922ea640 -1 osd.5 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.523501+0000 front 2024-09-17T16:46:22.523333+0000 (oldest deadline 2024-09-17T16:46:44.223301+0000) 2024-09-17T16:46:47.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:46 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:46.824+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:47.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:46 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:46.824+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:47.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:47.061+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:47.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:47.061+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:47.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:47 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:47.301+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:47.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:47 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:47.301+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:47.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:47 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:47.301+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:47.778 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:47.405+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:47.778 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:47.405+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:47.778 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:47.405+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:47.779 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:47.587+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:47.779 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:47.587+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:47.779 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:47.587+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:48.061 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:47.776+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:48.061 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:47 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:47.776+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:48.285 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:48 smithi071 podman[171651]: 2024-09-17 16:46:48.020437673 +0000 UTC m=+5.751900720 container died d46e44097d19934d74526cd6eab922679ad9de482202c482162f11b2a64d0478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-17T16:46:48.285 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:47 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:47.965+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:48.285 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:47 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:47.965+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:48.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:48.058+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:48.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:48.058+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:48.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:48 smithi071 podman[171651]: 2024-09-17 16:46:48.456711297 +0000 UTC m=+6.188174329 container cleanup d46e44097d19934d74526cd6eab922679ad9de482202c482162f11b2a64d0478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-17T16:46:48.564 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:48 smithi071 bash[171651]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5 2024-09-17T16:46:48.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:48 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:48.282+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:48.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:48 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:48.282+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:48.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:48 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:48.282+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:48.824 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:48.415+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:48.824 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:48.415+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:48.824 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:48.415+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:48.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:48.618+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:48.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:48.618+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:48.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:48.618+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:49.064 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:48 smithi071 podman[171666]: 2024-09-17 16:46:48.708235591 +0000 UTC m=+0.691921686 container remove d46e44097d19934d74526cd6eab922679ad9de482202c482162f11b2a64d0478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-09-17T16:46:49.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:48 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:48.924+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:49.065 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:48 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:48.924+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:49.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:48.821+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:49.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:48 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:48.821+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:49.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:49.069+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:49.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:49.069+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:49.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:49.069+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:49.380 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:49 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:49.243+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:49.380 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:49 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:49.243+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:49.380 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:49 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:49.243+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:49.380 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:49 smithi071 podman[171755]: 2024-09-17 16:46:49.012862477 +0000 UTC m=+0.015691398 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:46:49.380 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:49 smithi071 podman[171755]: 2024-09-17 16:46:49.117646374 +0000 UTC m=+0.120475360 container create e65dece795efbf89b0349cd3431ed6d8f5b0581414bc2c9fd0c49822eb357790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T16:46:49.380 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:49 smithi071 podman[171755]: 2024-09-17 16:46:49.303509486 +0000 UTC m=+0.306338425 container init e65dece795efbf89b0349cd3431ed6d8f5b0581414bc2c9fd0c49822eb357790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:46:49.380 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:49 smithi071 podman[171755]: 2024-09-17 16:46:49.306625469 +0000 UTC m=+0.309454383 container start e65dece795efbf89b0349cd3431ed6d8f5b0581414bc2c9fd0c49822eb357790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-17T16:46:49.380 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:49 smithi071 podman[171755]: 2024-09-17 16:46:49.378143424 +0000 UTC m=+0.380972351 container attach e65dece795efbf89b0349cd3431ed6d8f5b0581414bc2c9fd0c49822eb357790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-17T16:46:49.774 INFO:journalctl@ceph.osd.5.smithi071.stdout:Sep 17 16:46:49 smithi071 podman[171755]: 2024-09-17 16:46:49.496743211 +0000 UTC m=+0.499572139 container died e65dece795efbf89b0349cd3431ed6d8f5b0581414bc2c9fd0c49822eb357790 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:46:49.793 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:49.409+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:49.793 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:49.409+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:49.794 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:49.409+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:49.794 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:49.630+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:49.794 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:49.630+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:49.794 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:49.630+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:50.038 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:49 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:49.947+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:50.038 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:49 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:49.947+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:50.073 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.5.service' 2024-09-17T16:46:50.094 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:49.791+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:50.094 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:49 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:49.791+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:50.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:46:50.109 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-17T16:46:50.110 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-17T16:46:50.110 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.6 2024-09-17T16:46:50.313 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:50 smithi071 systemd[1]: Stopping Ceph osd.6 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:46:50.314 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:50.262+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:50.314 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:50.262+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:50.314 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:50.262+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:50.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:50.092+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:50.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:50.092+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:50.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:50.092+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:50.758 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:50.444+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:50.758 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:50.444+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:50.758 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:50.444+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:50.758 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:50.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:50.758 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:50.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:50.758 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:50.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:50.814 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:50.425+0000 7f895d795640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:46:50.814 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:50.425+0000 7f895d795640 -1 osd.6 1060 *** Got signal Terminated *** 2024-09-17T16:46:50.814 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:50.425+0000 7f895d795640 -1 osd.6 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:46:51.066 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:50.755+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:51.066 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:50 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:50.755+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:51.268 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:50.957+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:51.269 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:50 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:50.957+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:51.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:51.064+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:51.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:51.064+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:51.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:51.064+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:51.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:51 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:51.266+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:51.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:51 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:51.266+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:51.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:51 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:51.266+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:51.778 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:51.426+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:51.778 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:51.426+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:51.778 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:51.426+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:51.779 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:51.574+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:51.779 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:51.574+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:51.779 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:51.574+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:52.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:51.775+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:52.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:51 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:51.775+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:52.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:52.014+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:52.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:52.014+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:52.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:52.014+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:52.279 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:51 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:51.922+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:52.279 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:51 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:51.922+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:52.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:52 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:52.277+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:52.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:52 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:52.277+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:52.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:52 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:52.277+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:52.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:52 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:52.277+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.921838+0000 front 2024-09-17T16:46:29.921797+0000 (oldest deadline 2024-09-17T16:46:52.221804+0000) 2024-09-17T16:46:52.740 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:52.411+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:52.740 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:52.411+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:52.740 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:52.411+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:52.741 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:52.612+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:52.741 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:52.612+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:52.741 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:52.612+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:53.010 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:52.738+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:53.010 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:52 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:52.738+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:53.282 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:52 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:52.876+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:53.282 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:52 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:52.876+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:53.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:53.007+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:53.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:53.007+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:53.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:53.007+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:53.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:53 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:53.279+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:53.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:53 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:53.279+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:53.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:53 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:53.279+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:53.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:53 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:53.279+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.921838+0000 front 2024-09-17T16:46:29.921797+0000 (oldest deadline 2024-09-17T16:46:52.221804+0000) 2024-09-17T16:46:53.752 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:53.439+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:53.752 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:53.439+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:53.752 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:53.439+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:53.752 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:53.566+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:53.752 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:53.566+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:53.752 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:53.566+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:54.032 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:53.749+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:54.032 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:53 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:53.749+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:54.305 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:53 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:53.855+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:54.305 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:53 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:53.855+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:54.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:54.030+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:54.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:54.030+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:54.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:54.030+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:54.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:54 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:54.303+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:54.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:54 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:54.303+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:54.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:54 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:54.303+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:54.564 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:54 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:54.303+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.921838+0000 front 2024-09-17T16:46:29.921797+0000 (oldest deadline 2024-09-17T16:46:52.221804+0000) 2024-09-17T16:46:54.772 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:54.450+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:54.772 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:54.450+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:54.773 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:54.450+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:54.773 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:54.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:54.773 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:54.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:54.773 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:54.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:54.773 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:54.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:46:55.024 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:54.770+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:55.024 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:54 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:54.770+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:55.314 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:54 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:54.863+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:55.314 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:54 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:54.863+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:55.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:55.022+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:55.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:55.022+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:55.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:55.022+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:55.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:55.022+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:46:55.814 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:55 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:55.313+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.518786+0000 front 2024-09-17T16:46:06.518823+0000 (oldest deadline 2024-09-17T16:46:30.018977+0000) 2024-09-17T16:46:55.814 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:55 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:55.313+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.919786+0000 front 2024-09-17T16:46:12.919839+0000 (oldest deadline 2024-09-17T16:46:38.219809+0000) 2024-09-17T16:46:55.814 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:55 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:55.313+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:19.920918+0000 front 2024-09-17T16:46:19.920829+0000 (oldest deadline 2024-09-17T16:46:45.220716+0000) 2024-09-17T16:46:55.814 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:55 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6[44831]: 2024-09-17T16:46:55.314+0000 7f8959daa640 -1 osd.6 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.921838+0000 front 2024-09-17T16:46:29.921797+0000 (oldest deadline 2024-09-17T16:46:52.221804+0000) 2024-09-17T16:46:55.822 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:55.440+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:55.822 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:55.440+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:55.822 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:55.440+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:55.823 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:55.543+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:55.823 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:55.543+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:55.823 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:55.543+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:55.823 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:55.543+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:46:56.099 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:55 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:55.826+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:56.099 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:55 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:55.826+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:56.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:55.820+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:56.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:55.820+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:56.146 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:55 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:55.820+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:46:56.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:56.010+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:56.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:56.010+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:56.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:56.010+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:56.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:56.010+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:46:56.418 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:56 smithi071 podman[171856]: 2024-09-17 16:46:56.097553605 +0000 UTC m=+5.771640017 container died 5921e90110a6732d1db460769fce5d2d68a5546026fed46ba8eacc1ae0aeb54e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T16:46:56.418 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:56 smithi071 podman[171856]: 2024-09-17 16:46:56.416945621 +0000 UTC m=+6.091032025 container cleanup 5921e90110a6732d1db460769fce5d2d68a5546026fed46ba8eacc1ae0aeb54e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-17T16:46:56.783 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:56 smithi071 bash[171856]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6 2024-09-17T16:46:56.784 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:56 smithi071 podman[171871]: 2024-09-17 16:46:56.626808853 +0000 UTC m=+0.529255788 container remove 5921e90110a6732d1db460769fce5d2d68a5546026fed46ba8eacc1ae0aeb54e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-17T16:46:56.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:56.444+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:56.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:56.444+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:56.806 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:56.444+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:56.806 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:56.565+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:56.806 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:56.566+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:56.806 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:56.566+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:56.806 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:56.566+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:46:57.037 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:56 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:56.781+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:57.037 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:56 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:56.781+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:57.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:56.803+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:57.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:56.803+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:57.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:56 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:56.803+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:46:57.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:57.032+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:57.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:57.032+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:57.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:57.032+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:57.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:57.032+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:46:57.314 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:57 smithi071 podman[171959]: 2024-09-17 16:46:56.937606396 +0000 UTC m=+0.015847251 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:46:57.314 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:57 smithi071 podman[171959]: 2024-09-17 16:46:57.035672063 +0000 UTC m=+0.113912907 container create eb76bcf7319969b4a8299600bc8bda129bcba7dd07ce3181a5f759d7761036f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux ) 2024-09-17T16:46:57.314 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:57 smithi071 podman[171959]: 2024-09-17 16:46:57.240125251 +0000 UTC m=+0.318366108 container init eb76bcf7319969b4a8299600bc8bda129bcba7dd07ce3181a5f759d7761036f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-17T16:46:57.314 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:57 smithi071 podman[171959]: 2024-09-17 16:46:57.24414613 +0000 UTC m=+0.322386972 container start eb76bcf7319969b4a8299600bc8bda129bcba7dd07ce3181a5f759d7761036f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-17T16:46:57.797 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:57 smithi071 podman[171959]: 2024-09-17 16:46:57.314904772 +0000 UTC m=+0.393145622 container attach eb76bcf7319969b4a8299600bc8bda129bcba7dd07ce3181a5f759d7761036f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-17T16:46:57.797 INFO:journalctl@ceph.osd.6.smithi071.stdout:Sep 17 16:46:57 smithi071 podman[171959]: 2024-09-17 16:46:57.458849751 +0000 UTC m=+0.537090603 container died eb76bcf7319969b4a8299600bc8bda129bcba7dd07ce3181a5f759d7761036f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16) 2024-09-17T16:46:57.797 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:57.396+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:57.797 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:57.396+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:57.797 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:57.396+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:57.798 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:57.584+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:57.798 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:57.584+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:57.798 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:57.584+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:57.798 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:57.584+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:46:58.063 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:57 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:57.794+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:58.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:57 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:57.794+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:58.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:57 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:57.794+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:30.563149+0000 front 2024-09-17T16:46:30.563239+0000 (oldest deadline 2024-09-17T16:46:56.963742+0000) 2024-09-17T16:46:58.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.6.service' 2024-09-17T16:46:58.144 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:57.795+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:58.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:57.795+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:58.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:57 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:57.795+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:46:58.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:58.015+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:58.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:58.015+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:58.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:58.015+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:58.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:58.015+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:46:58.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:46:58.175 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-17T16:46:58.175 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-17T16:46:58.176 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.7 2024-09-17T16:46:58.499 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:58 smithi071 systemd[1]: Stopping Ceph osd.7 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:46:58.499 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:58 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:58.497+0000 7f29fca58640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:46:58.499 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:58 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:58.497+0000 7f29fca58640 -1 osd.7 1060 *** Got signal Terminated *** 2024-09-17T16:46:58.499 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:58 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:58.497+0000 7f29fca58640 -1 osd.7 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:46:58.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:58.354+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:58.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:58.354+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:58.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:58.354+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:58.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:58.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:58.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:58.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:58.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:58.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:58.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:58.590+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:46:59.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:58.787+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:46:59.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:58.787+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:46:59.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:58.787+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:46:59.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:58.971+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:46:59.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:58.971+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:46:59.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:58.971+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:46:59.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:58 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:58.971+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:46:59.313 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:58 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:58.827+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:46:59.314 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:58 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:58.827+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:46:59.314 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:58 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:58.827+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:30.563149+0000 front 2024-09-17T16:46:30.563239+0000 (oldest deadline 2024-09-17T16:46:56.963742+0000) 2024-09-17T16:46:59.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:59.351+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:46:59.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:59.351+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:46:59.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:46:59.351+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:46:59.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:59.566+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:46:59.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:59.566+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:46:59.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:59.566+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:46:59.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:46:59.566+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:00.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:59 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:59.777+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:47:00.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:59 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:59.777+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:47:00.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:46:59 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:46:59.777+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:30.563149+0000 front 2024-09-17T16:46:30.563239+0000 (oldest deadline 2024-09-17T16:46:56.963742+0000) 2024-09-17T16:47:00.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:59.760+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:00.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:59.760+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:00.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:46:59.760+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:00.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:59.961+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:00.145 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:59.961+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:00.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:59.961+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:00.146 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:46:59 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:46:59.961+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:00.612 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:00 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:00.338+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:00.612 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:00 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:00.338+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:00.612 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:00 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:00.338+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:00.895 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:00 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:00.773+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:00.895 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:00 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:00.773+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:00.895 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:00 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:00.773+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:00.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:00 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:00.610+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:00.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:00 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:00.610+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:00.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:00 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:00.610+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:00.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:00 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:00.610+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:01.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:00 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:00.739+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:47:01.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:00 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:00.739+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:47:01.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:00 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:00.739+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:30.563149+0000 front 2024-09-17T16:46:30.563239+0000 (oldest deadline 2024-09-17T16:46:56.963742+0000) 2024-09-17T16:47:01.297 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:01.005+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:01.297 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:01.005+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:01.298 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:01.005+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:01.298 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:01.005+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:01.298 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:01.005+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:01.585 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:01.295+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:01.585 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:01.295+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:01.585 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:01.295+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:01.895 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:01.737+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:01.895 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:01.737+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:01.896 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:01.737+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:01.896 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:01.583+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:01.896 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:01.583+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:01.896 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:01.583+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:01.896 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:01 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:01.583+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:02.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:01 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:01.764+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:47:02.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:01 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:01.764+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:47:02.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:01 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:01.764+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:30.563149+0000 front 2024-09-17T16:46:30.563239+0000 (oldest deadline 2024-09-17T16:46:56.963742+0000) 2024-09-17T16:47:02.065 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:01 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:01.764+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:36.964195+0000 front 2024-09-17T16:46:36.964213+0000 (oldest deadline 2024-09-17T16:47:01.664353+0000) 2024-09-17T16:47:02.339 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:02.029+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:02.339 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:02.029+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:02.339 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:02.029+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:02.339 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:02.029+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:02.339 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:02.029+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:02.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:02.337+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:02.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:02.337+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:02.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:02.337+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:02.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:02.549+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:02.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:02.549+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:02.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:02.549+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:02.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:02.549+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:03.039 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:02.777+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:03.039 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:02.777+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:03.039 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:02 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:02.777+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:03.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:02 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:02.740+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:03.957269+0000 front 2024-09-17T16:46:03.957346+0000 (oldest deadline 2024-09-17T16:46:29.257300+0000) 2024-09-17T16:47:03.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:02 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:02.740+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:22.560984+0000 front 2024-09-17T16:46:22.560997+0000 (oldest deadline 2024-09-17T16:46:44.161502+0000) 2024-09-17T16:47:03.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:02 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:02.740+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:30.563149+0000 front 2024-09-17T16:46:30.563239+0000 (oldest deadline 2024-09-17T16:46:56.963742+0000) 2024-09-17T16:47:03.064 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:02 smithi071 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7[62003]: 2024-09-17T16:47:02.740+0000 7f29f906d640 -1 osd.7 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:36.964195+0000 front 2024-09-17T16:46:36.964213+0000 (oldest deadline 2024-09-17T16:47:01.664353+0000) 2024-09-17T16:47:03.348 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:03.037+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:03.349 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:03.037+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:03.349 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:03.037+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:03.349 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:03.037+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:03.349 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:03.037+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:03.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:03.345+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:03.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:03.345+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:03.645 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:03.345+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:03.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:03.513+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:03.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:03.513+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:03.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:03.513+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:03.646 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:03.513+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:04.072 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:03.810+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:04.072 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:03.810+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:04.072 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:03.810+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:04.072 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:03 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:03.810+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:04.395 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:04.304+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:04.395 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:04.304+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:04.395 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:04.304+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:04.395 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:04.304+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.779098+0000 front 2024-09-17T16:46:38.779234+0000 (oldest deadline 2024-09-17T16:47:03.478971+0000) 2024-09-17T16:47:04.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:04.069+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:04.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:04.069+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:04.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:04.069+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:04.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:04.069+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:04.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:04.069+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:04.506 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:04 smithi071 podman[172061]: 2024-09-17 16:47:04.108192342 +0000 UTC m=+5.714459400 container died 012a121b3a81f37bedecd93d965ba29462235ac649273f031f204dcefc4ab090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-17T16:47:04.506 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:04 smithi071 podman[172061]: 2024-09-17 16:47:04.504884804 +0000 UTC m=+6.111151851 container cleanup 012a121b3a81f37bedecd93d965ba29462235ac649273f031f204dcefc4ab090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-17T16:47:04.786 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:04.518+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:04.787 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:04.518+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:04.787 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:04.518+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:04.787 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:04.518+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:04.813 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:04 smithi071 bash[172061]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7 2024-09-17T16:47:04.813 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:04 smithi071 podman[172078]: 2024-09-17 16:47:04.723132464 +0000 UTC m=+0.610649946 container remove 012a121b3a81f37bedecd93d965ba29462235ac649273f031f204dcefc4ab090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T16:47:05.085 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:04.784+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:05.085 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:04.784+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:05.085 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:04.784+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:05.085 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:04 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:04.784+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:05.311 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:05 smithi071 podman[172166]: 2024-09-17 16:47:05.023870061 +0000 UTC m=+0.110331371 container create 10bdfdec2a95fc9ca16ec41b858c7e22ab14c7eafebd703a8f3b3d7af51ab5fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-17T16:47:05.312 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:05 smithi071 podman[172166]: 2024-09-17 16:47:04.929063228 +0000 UTC m=+0.015524549 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:47:05.312 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:05 smithi071 podman[172166]: 2024-09-17 16:47:05.226306444 +0000 UTC m=+0.312767762 container init 10bdfdec2a95fc9ca16ec41b858c7e22ab14c7eafebd703a8f3b3d7af51ab5fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-17T16:47:05.312 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:05 smithi071 podman[172166]: 2024-09-17 16:47:05.229688348 +0000 UTC m=+0.316149651 container start 10bdfdec2a95fc9ca16ec41b858c7e22ab14c7eafebd703a8f3b3d7af51ab5fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-17T16:47:05.312 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:05 smithi071 podman[172166]: 2024-09-17 16:47:05.309689223 +0000 UTC m=+0.396150525 container attach 10bdfdec2a95fc9ca16ec41b858c7e22ab14c7eafebd703a8f3b3d7af51ab5fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-17T16:47:05.395 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:05.265+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:05.395 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:05.265+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:05.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:05.265+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:05.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:05.265+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.779098+0000 front 2024-09-17T16:46:38.779234+0000 (oldest deadline 2024-09-17T16:47:03.478971+0000) 2024-09-17T16:47:05.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:05.082+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:05.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:05.082+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:05.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:05.082+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:05.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:05.082+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:05.397 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:05.082+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:05.564 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:05 smithi071 podman[172166]: 2024-09-17 16:47:05.420209518 +0000 UTC m=+0.506670840 container died 10bdfdec2a95fc9ca16ec41b858c7e22ab14c7eafebd703a8f3b3d7af51ab5fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-17T16:47:05.836 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:05.504+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:05.837 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:05.504+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:05.837 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:05.504+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:05.837 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:05.504+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:05.974 INFO:journalctl@ceph.osd.7.smithi071.stdout:Sep 17 16:47:05 smithi071 podman[172189]: 2024-09-17 16:47:05.954809244 +0000 UTC m=+0.528983666 container remove 10bdfdec2a95fc9ca16ec41b858c7e22ab14c7eafebd703a8f3b3d7af51ab5fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-17T16:47:05.980 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.7.service' 2024-09-17T16:47:06.017 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:47:06.018 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-17T16:47:06.018 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-17T16:47:06.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.8 2024-09-17T16:47:06.132 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:06 smithi173 systemd[1]: Stopping Ceph osd.8 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:47:06.133 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:05.834+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:06.133 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:05.834+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:06.133 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:05.834+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:06.133 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:05 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:05.834+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:06.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:06.130+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:06.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:06.130+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:06.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:06.130+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:06.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:06.130+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:06.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:06.130+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:06.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:06.253+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:06.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:06.253+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:06.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:06.253+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:06.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:06.253+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.779098+0000 front 2024-09-17T16:46:38.779234+0000 (oldest deadline 2024-09-17T16:47:03.478971+0000) 2024-09-17T16:47:06.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:06.326+0000 7fa22e397640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:47:06.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:06.326+0000 7fa22e397640 -1 osd.8 1060 *** Got signal Terminated *** 2024-09-17T16:47:06.396 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:06.326+0000 7fa22e397640 -1 osd.8 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:47:06.829 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:06.527+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:06.830 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:06.527+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:06.830 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:06.527+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:06.830 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:06.527+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:07.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:06.827+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:07.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:06.827+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:07.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:06.827+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:07.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:06 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:06.827+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:07.559 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:07.240+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:07.560 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:07.240+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:07.560 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:07.240+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:07.560 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:07.240+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.779098+0000 front 2024-09-17T16:46:38.779234+0000 (oldest deadline 2024-09-17T16:47:03.478971+0000) 2024-09-17T16:47:07.560 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:07.146+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:07.560 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:07.146+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:07.560 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:07.146+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:07.560 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:07.146+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:07.560 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:07.146+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:07.851 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:07.557+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:07.851 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:07.557+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:07.851 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:07.557+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:07.851 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:07.557+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:08.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:07.849+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:08.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:07.849+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:08.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:07.849+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:08.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:07 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:07.849+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:08.591 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:08.207+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:08.591 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:08.207+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:08.591 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:08.207+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:08.591 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:08.207+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.779098+0000 front 2024-09-17T16:46:38.779234+0000 (oldest deadline 2024-09-17T16:47:03.478971+0000) 2024-09-17T16:47:08.591 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:08.207+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:43.479533+0000 front 2024-09-17T16:46:43.479550+0000 (oldest deadline 2024-09-17T16:47:08.179389+0000) 2024-09-17T16:47:08.592 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:08.187+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:08.592 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:08.187+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:08.592 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:08.187+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:08.592 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:08.187+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:08.592 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:08.187+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:08.853 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:08.588+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:08.853 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:08.588+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:08.853 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:08.588+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:08.853 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:08.588+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:08.853 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:08.588+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:09.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:08.851+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:09.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:08.851+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:09.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:08.851+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:09.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:08.851+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:09.145 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:08 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:08.851+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:09.558 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:09.182+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:09.558 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:09.182+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:09.558 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:09.182+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:09.558 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:09.182+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.779098+0000 front 2024-09-17T16:46:38.779234+0000 (oldest deadline 2024-09-17T16:47:03.478971+0000) 2024-09-17T16:47:09.558 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:09.182+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:43.479533+0000 front 2024-09-17T16:46:43.479550+0000 (oldest deadline 2024-09-17T16:47:08.179389+0000) 2024-09-17T16:47:09.558 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:09.171+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:09.558 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:09.171+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:09.558 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:09.171+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:09.558 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:09.171+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:09.559 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:09.171+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:09.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:09.555+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:09.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:09.555+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:09.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:09.555+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:09.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:09.555+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:09.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:09.555+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:10.158 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:09.900+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:10.158 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:09.900+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:10.158 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:09.900+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:10.159 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:09.900+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:10.159 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:09 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:09.900+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:10.159 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:10.129+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:10.159 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:10.129+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:10.159 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:10.129+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:10.159 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:10.129+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:10.159 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:10.129+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:10.525 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:10.156+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:10.525 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:10.156+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:10.525 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:10.156+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:10.525 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:10.156+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.779098+0000 front 2024-09-17T16:46:38.779234+0000 (oldest deadline 2024-09-17T16:47:03.478971+0000) 2024-09-17T16:47:10.526 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:10.156+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:43.479533+0000 front 2024-09-17T16:46:43.479550+0000 (oldest deadline 2024-09-17T16:47:08.179389+0000) 2024-09-17T16:47:10.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:10.522+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:10.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:10.522+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:10.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:10.522+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:10.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:10.522+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:10.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:10.522+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:11.179 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:11.133+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:11.179 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:11.133+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:11.179 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:11.133+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:11.180 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:11.133+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:11.180 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:11.133+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:11.180 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:10.925+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:11.180 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:10.925+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:11.180 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:10.925+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:11.180 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:10.925+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:11.180 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:10 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:10.925+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:11.488 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:11.177+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:06.174479+0000 front 2024-09-17T16:46:06.174656+0000 (oldest deadline 2024-09-17T16:46:32.074321+0000) 2024-09-17T16:47:11.488 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:11.177+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:12.075057+0000 front 2024-09-17T16:46:12.075010+0000 (oldest deadline 2024-09-17T16:46:36.774734+0000) 2024-09-17T16:47:11.488 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:11.177+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.275759+0000 front 2024-09-17T16:46:20.275769+0000 (oldest deadline 2024-09-17T16:46:44.375508+0000) 2024-09-17T16:47:11.488 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:11.177+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.779098+0000 front 2024-09-17T16:46:38.779234+0000 (oldest deadline 2024-09-17T16:47:03.478971+0000) 2024-09-17T16:47:11.488 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8[34453]: 2024-09-17T16:47:11.177+0000 7fa2269a4640 -1 osd.8 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:43.479533+0000 front 2024-09-17T16:46:43.479550+0000 (oldest deadline 2024-09-17T16:47:08.179389+0000) 2024-09-17T16:47:11.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:11.486+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:11.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:11.486+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:11.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:11.486+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:11.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:11.486+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:11.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:11.486+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:12.179 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:11 smithi173 podman[170694]: 2024-09-17 16:47:11.917033248 +0000 UTC m=+5.701622122 container died 7f4b9077b8cfa7b34b19bb02774b0cad29656d57087335a36f58d565b132c775 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T16:47:12.179 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:11.909+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:12.179 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:11.909+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:12.180 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:11.909+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:12.180 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:11.909+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:12.180 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:11 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:11.909+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:12.180 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:12.145+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:12.180 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:12.145+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:12.180 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:12.145+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:12.180 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:12.145+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:12.180 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:12.145+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:12.511 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:12 smithi173 podman[170694]: 2024-09-17 16:47:12.268846999 +0000 UTC m=+6.053435873 container cleanup 7f4b9077b8cfa7b34b19bb02774b0cad29656d57087335a36f58d565b132c775 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-17T16:47:12.511 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:12 smithi173 bash[170694]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8 2024-09-17T16:47:12.874 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:12 smithi173 podman[170711]: 2024-09-17 16:47:12.554039613 +0000 UTC m=+0.633807165 container remove 7f4b9077b8cfa7b34b19bb02774b0cad29656d57087335a36f58d565b132c775 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-17T16:47:12.874 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:12.509+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:12.874 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:12.509+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:12.874 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:12.509+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:12.874 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:12.509+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:12.874 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:12.509+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:13.130 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:12 smithi173 podman[170801]: 2024-09-17 16:47:12.77529345 +0000 UTC m=+0.015619343 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:47:13.130 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:12 smithi173 podman[170801]: 2024-09-17 16:47:12.888128429 +0000 UTC m=+0.128454307 container create 927fa544dfc2db2d2910b6433de34c701f833707e026118d8f694aeb33fb603d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS) 2024-09-17T16:47:13.131 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:13 smithi173 podman[170801]: 2024-09-17 16:47:13.090636207 +0000 UTC m=+0.330962080 container init 927fa544dfc2db2d2910b6433de34c701f833707e026118d8f694aeb33fb603d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-17T16:47:13.131 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:13 smithi173 podman[170801]: 2024-09-17 16:47:13.093981917 +0000 UTC m=+0.334307800 container start 927fa544dfc2db2d2910b6433de34c701f833707e026118d8f694aeb33fb603d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-17T16:47:13.131 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:12.918+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:13.131 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:12.918+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:13.131 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:12.918+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:13.131 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:12.918+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:13.131 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:12 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:12.918+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:13.395 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:13 smithi173 podman[170801]: 2024-09-17 16:47:13.207813419 +0000 UTC m=+0.448139302 container attach 927fa544dfc2db2d2910b6433de34c701f833707e026118d8f694aeb33fb603d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:47:13.395 INFO:journalctl@ceph.osd.8.smithi173.stdout:Sep 17 16:47:13 smithi173 podman[170801]: 2024-09-17 16:47:13.344724981 +0000 UTC m=+0.585050872 container died 927fa544dfc2db2d2910b6433de34c701f833707e026118d8f694aeb33fb603d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:47:13.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:13.127+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:13.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:13.127+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:13.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:13.127+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:13.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:13.127+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:13.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:13.127+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:13.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:13.540+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:13.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:13.540+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:13.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:13.540+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:13.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:13.540+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:13.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:13.540+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:14.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.8.service' 2024-09-17T16:47:14.049 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:47:14.049 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-17T16:47:14.049 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-17T16:47:14.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.9 2024-09-17T16:47:14.259 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:14.144+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:14.259 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:14.144+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:14.260 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:14.144+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:14.260 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:14.144+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:14.260 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:14.144+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:14.260 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:13.962+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:14.260 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:13.962+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:14.260 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:13.962+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:14.260 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:13.962+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:14.260 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:13 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:13.962+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:14.260 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:14 smithi173 systemd[1]: Stopping Ceph osd.9 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:47:14.541 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:14.398+0000 7f3630181640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:47:14.541 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:14.398+0000 7f3630181640 -1 osd.9 1060 *** Got signal Terminated *** 2024-09-17T16:47:14.541 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:14.398+0000 7f3630181640 -1 osd.9 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:47:14.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:14.538+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:14.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:14.538+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:14.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:14.538+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:14.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:14.538+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:14.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:14 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:14.538+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:15.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:15.004+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:15.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:15.004+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:15.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:15.004+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:15.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:15.004+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:15.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:15.004+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:15.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:15.186+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:15.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:15.186+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:15.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:15.186+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:15.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:15.186+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:15.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:15.186+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:15.894 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:15.554+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:15.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:15.554+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:15.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:15.554+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:15.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:15.554+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:15.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:15.554+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:16.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:15.994+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:16.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:15.994+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:16.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:15.994+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:16.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:15.994+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:16.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:15 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:15.994+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:16.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:16 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:16.211+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:16.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:16 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:16.211+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:16.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:16 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:16.211+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:16.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:16 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:16.211+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:16.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:16 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:16.211+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:16.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:16 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:16.526+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:16.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:16 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:16.526+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:16.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:16 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:16.526+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:16.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:16 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:16.526+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:16.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:16 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:16.526+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:17.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:17.024+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:17.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:17.024+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:17.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:17.024+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:17.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:17.024+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:17.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:17.024+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:17.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:17.216+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:17.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:17.216+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:17.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:17.216+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:17.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:17.216+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:17.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:17.216+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:17.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:17.216+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:17.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:17.523+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:17.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:17.523+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:17.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:17.523+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:17.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:17.523+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:17.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:17 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:17.523+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:18.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:18.041+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:18.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:18.041+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:18.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:18.041+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:18.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:18.041+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:18.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:18.041+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:18.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:18.184+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:18.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:18.184+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:18.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:18.184+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:18.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:18.184+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:18.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:18.184+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:18.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:18.184+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:18.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:18.561+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:18.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:18.561+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:18.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:18.561+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:18.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:18.561+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:18.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:18 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:18.561+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:19.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:19.022+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:11.115736+0000 front 2024-09-17T16:46:11.115868+0000 (oldest deadline 2024-09-17T16:46:36.415838+0000) 2024-09-17T16:47:19.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:19.022+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:20.517077+0000 front 2024-09-17T16:46:20.516997+0000 (oldest deadline 2024-09-17T16:46:44.616940+0000) 2024-09-17T16:47:19.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:19.022+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.518053+0000 front 2024-09-17T16:46:27.517935+0000 (oldest deadline 2024-09-17T16:46:55.118322+0000) 2024-09-17T16:47:19.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:19.022+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:38.019427+0000 front 2024-09-17T16:46:38.019574+0000 (oldest deadline 2024-09-17T16:47:03.319657+0000) 2024-09-17T16:47:19.395 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9[40029]: 2024-09-17T16:47:19.022+0000 7f362878e640 -1 osd.9 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:47.321373+0000 front 2024-09-17T16:46:47.321360+0000 (oldest deadline 2024-09-17T16:47:08.421149+0000) 2024-09-17T16:47:19.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:19.230+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:19.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:19.230+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:19.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:19.230+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:19.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:19.230+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:19.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:19.230+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:19.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:19.230+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:19.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:19.599+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:19.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:19.599+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:19.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:19.599+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:19.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:19.599+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:19.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:19 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:19.599+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:20.310 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:20 smithi173 podman[170905]: 2024-09-17 16:47:20.039587188 +0000 UTC m=+5.751095162 container died c6e7daad4ac3730b99b787a6083d46ff555564c3e535b8ecbab6b00291d09555 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True) 2024-09-17T16:47:20.310 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:20.232+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:20.310 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:20.232+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:20.310 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:20.232+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:20.310 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:20.232+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:20.310 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:20.232+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:20.310 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:20.232+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:20.612 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:20 smithi173 podman[170905]: 2024-09-17 16:47:20.499180653 +0000 UTC m=+6.210688626 container cleanup c6e7daad4ac3730b99b787a6083d46ff555564c3e535b8ecbab6b00291d09555 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-17T16:47:20.612 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:20 smithi173 bash[170905]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9 2024-09-17T16:47:20.895 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:20 smithi173 podman[170931]: 2024-09-17 16:47:20.792661915 +0000 UTC m=+0.751385544 container remove c6e7daad4ac3730b99b787a6083d46ff555564c3e535b8ecbab6b00291d09555 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2) 2024-09-17T16:47:20.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:20.609+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:20.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:20.609+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:20.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:20.609+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:20.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:20.609+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:20.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:20.609+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:20.896 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:20 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:20.609+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:53.871961+0000 front 2024-09-17T16:46:53.872049+0000 (oldest deadline 2024-09-17T16:47:20.272333+0000) 2024-09-17T16:47:21.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:21.220+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:21.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:21.220+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:21.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:21.220+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:21.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:21.220+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:21.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:21.220+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:21.396 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:21.220+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:21.396 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:21 smithi173 podman[171019]: 2024-09-17 16:47:21.018086011 +0000 UTC m=+0.015922029 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:47:21.396 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:21 smithi173 podman[171019]: 2024-09-17 16:47:21.127021595 +0000 UTC m=+0.124857600 container create 8457dfa1502200ec865588a17197c2f605b97c45b385ce9688f9ae2b790bcb70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T16:47:21.396 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:21 smithi173 podman[171019]: 2024-09-17 16:47:21.354581585 +0000 UTC m=+0.352417590 container init 8457dfa1502200ec865588a17197c2f605b97c45b385ce9688f9ae2b790bcb70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-17T16:47:21.396 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:21 smithi173 podman[171019]: 2024-09-17 16:47:21.358052981 +0000 UTC m=+0.355888986 container start 8457dfa1502200ec865588a17197c2f605b97c45b385ce9688f9ae2b790bcb70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T16:47:21.851 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:21 smithi173 podman[171019]: 2024-09-17 16:47:21.446138133 +0000 UTC m=+0.443974141 container attach 8457dfa1502200ec865588a17197c2f605b97c45b385ce9688f9ae2b790bcb70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-17T16:47:21.851 INFO:journalctl@ceph.osd.9.smithi173.stdout:Sep 17 16:47:21 smithi173 podman[171019]: 2024-09-17 16:47:21.546980247 +0000 UTC m=+0.544816262 container died 8457dfa1502200ec865588a17197c2f605b97c45b385ce9688f9ae2b790bcb70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-9-deactivate, ceph=True, GIT_CLEAN=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:47:21.851 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:21.636+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:21.851 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:21.636+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:21.852 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:21.636+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:21.852 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:21.636+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:21.852 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:21.636+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:21.852 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:21 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:21.636+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:53.871961+0000 front 2024-09-17T16:46:53.872049+0000 (oldest deadline 2024-09-17T16:47:20.272333+0000) 2024-09-17T16:47:22.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.9.service' 2024-09-17T16:47:22.266 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:47:22.266 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-17T16:47:22.266 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-17T16:47:22.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.10 2024-09-17T16:47:22.590 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:22 smithi173 systemd[1]: Stopping Ceph osd.10 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:47:22.590 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:22.587+0000 7ff0bb1ce640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:47:22.590 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:22.587+0000 7ff0bb1ce640 -1 osd.10 1060 *** Got signal Terminated *** 2024-09-17T16:47:22.591 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:22.240+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:22.591 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:22.240+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:22.591 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:22.240+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:22.591 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:22.240+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:22.591 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:22.240+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:22.591 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:22.240+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:22.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:22.587+0000 7ff0bb1ce640 -1 osd.10 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:47:22.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:22.663+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:22.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:22.663+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:22.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:22.663+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:22.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:22.663+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:22.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:22.663+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:22.895 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:22 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:22.663+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:53.871961+0000 front 2024-09-17T16:46:53.872049+0000 (oldest deadline 2024-09-17T16:47:20.272333+0000) 2024-09-17T16:47:23.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:23.205+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:23.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:23.205+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:23.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:23.205+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:23.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:23.205+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:23.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:23.205+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:23.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:23.205+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:24.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:23.708+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:24.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:23.708+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:24.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:23.708+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:24.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:23.708+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:24.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:23.708+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:24.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:23 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:23.708+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:53.871961+0000 front 2024-09-17T16:46:53.872049+0000 (oldest deadline 2024-09-17T16:47:20.272333+0000) 2024-09-17T16:47:24.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:24.254+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:24.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:24.254+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:24.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:24.254+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:24.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:24.254+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:24.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:24.254+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:24.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:24.254+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:25.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:24.706+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:25.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:24.706+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:25.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:24.706+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:25.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:24.706+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:25.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:24.706+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:25.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:24 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:24.706+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:53.871961+0000 front 2024-09-17T16:46:53.872049+0000 (oldest deadline 2024-09-17T16:47:20.272333+0000) 2024-09-17T16:47:25.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:25.295+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:25.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:25.295+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:25.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:25.295+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:25.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:25.295+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:25.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:25.295+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:25.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:25.295+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:26.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:25.717+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:26.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:25.717+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:26.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:25.717+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:26.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:25.717+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:26.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:25.717+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:26.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:25.717+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:53.871961+0000 front 2024-09-17T16:46:53.872049+0000 (oldest deadline 2024-09-17T16:47:20.272333+0000) 2024-09-17T16:47:26.145 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:25 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:25.717+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:00.272925+0000 front 2024-09-17T16:47:00.272939+0000 (oldest deadline 2024-09-17T16:47:25.572748+0000) 2024-09-17T16:47:26.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:26.315+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:26.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:26.315+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:26.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:26.315+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:26.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:26.315+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:26.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:26.315+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:26.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:26.315+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:27.146 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:26.670+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:05.064417+0000 front 2024-09-17T16:46:05.064484+0000 (oldest deadline 2024-09-17T16:46:28.564236+0000) 2024-09-17T16:47:27.146 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:26.670+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:14.365848+0000 front 2024-09-17T16:46:14.365965+0000 (oldest deadline 2024-09-17T16:46:36.065659+0000) 2024-09-17T16:47:27.146 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:26.670+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.966794+0000 front 2024-09-17T16:46:21.966644+0000 (oldest deadline 2024-09-17T16:46:44.866589+0000) 2024-09-17T16:47:27.146 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:26.670+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:27.767551+0000 front 2024-09-17T16:46:27.767568+0000 (oldest deadline 2024-09-17T16:46:53.567936+0000) 2024-09-17T16:47:27.146 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:26.670+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6812 osd.5 since back 2024-09-17T16:46:42.870030+0000 front 2024-09-17T16:46:42.870009+0000 (oldest deadline 2024-09-17T16:47:08.169983+0000) 2024-09-17T16:47:27.147 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:26.670+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:53.871961+0000 front 2024-09-17T16:46:53.872049+0000 (oldest deadline 2024-09-17T16:47:20.272333+0000) 2024-09-17T16:47:27.147 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:26 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10[141704]: 2024-09-17T16:47:26.670+0000 7ff0b37db640 -1 osd.10 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:00.272925+0000 front 2024-09-17T16:47:00.272939+0000 (oldest deadline 2024-09-17T16:47:25.572748+0000) 2024-09-17T16:47:27.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:27 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:27.300+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:27.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:27 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:27.301+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:27.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:27 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:27.301+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:27.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:27 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:27.301+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:27.646 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:27 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:27.301+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:27.646 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:27 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:27.301+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:27.646 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:27 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:27.301+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:01.122417+0000 front 2024-09-17T16:47:01.122407+0000 (oldest deadline 2024-09-17T16:47:27.022223+0000) 2024-09-17T16:47:28.341 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:28 smithi173 podman[171122]: 2024-09-17 16:47:28.066177251 +0000 UTC m=+5.580527604 container died e338f206d52c72e1fc2370b12ff67b2f97256f11cd12cb03d56a25d222b5a05d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-17T16:47:28.341 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:28 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:28.314+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:28.341 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:28 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:28.314+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:28.341 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:28 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:28.314+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:28.341 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:28 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:28.314+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:28.341 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:28 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:28.314+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:28.341 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:28 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:28.314+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:28.341 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:28 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:28.314+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:01.122417+0000 front 2024-09-17T16:47:01.122407+0000 (oldest deadline 2024-09-17T16:47:27.022223+0000) 2024-09-17T16:47:28.644 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:28 smithi173 podman[171122]: 2024-09-17 16:47:28.513312161 +0000 UTC m=+6.027662560 container cleanup e338f206d52c72e1fc2370b12ff67b2f97256f11cd12cb03d56a25d222b5a05d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-17T16:47:28.645 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:28 smithi173 bash[171122]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10 2024-09-17T16:47:29.118 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:28 smithi173 podman[171139]: 2024-09-17 16:47:28.773097653 +0000 UTC m=+0.709697103 container remove e338f206d52c72e1fc2370b12ff67b2f97256f11cd12cb03d56a25d222b5a05d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:47:29.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:29 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:29.324+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:29.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:29 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:29.324+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:29.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:29 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:29.324+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:29.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:29 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:29.324+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:29.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:29 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:29.324+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:29.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:29 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:29.324+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:29.395 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:29 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:29.324+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:01.122417+0000 front 2024-09-17T16:47:01.122407+0000 (oldest deadline 2024-09-17T16:47:27.022223+0000) 2024-09-17T16:47:29.395 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:29 smithi173 podman[171229]: 2024-09-17 16:47:29.019643115 +0000 UTC m=+0.015827632 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:47:29.396 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:29 smithi173 podman[171229]: 2024-09-17 16:47:29.123844025 +0000 UTC m=+0.120028578 container create ac30879fa0377168d6e0dcb89a6abb41809aebae7f41b76fe2749ed31f89e401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T16:47:29.396 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:29 smithi173 podman[171229]: 2024-09-17 16:47:29.326209911 +0000 UTC m=+0.322394431 container init ac30879fa0377168d6e0dcb89a6abb41809aebae7f41b76fe2749ed31f89e401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:47:29.396 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:29 smithi173 podman[171229]: 2024-09-17 16:47:29.329510786 +0000 UTC m=+0.325695297 container start ac30879fa0377168d6e0dcb89a6abb41809aebae7f41b76fe2749ed31f89e401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, maintainer=Guillaume Abrioux ) 2024-09-17T16:47:29.855 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:29 smithi173 podman[171229]: 2024-09-17 16:47:29.401070933 +0000 UTC m=+0.397255457 container attach ac30879fa0377168d6e0dcb89a6abb41809aebae7f41b76fe2749ed31f89e401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T16:47:29.855 INFO:journalctl@ceph.osd.10.smithi173.stdout:Sep 17 16:47:29 smithi173 podman[171229]: 2024-09-17 16:47:29.520538262 +0000 UTC m=+0.516722775 container died ac30879fa0377168d6e0dcb89a6abb41809aebae7f41b76fe2749ed31f89e401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-17T16:47:30.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.10.service' 2024-09-17T16:47:30.223 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:47:30.223 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-17T16:47:30.223 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-17T16:47:30.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.11 2024-09-17T16:47:30.551 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 systemd[1]: Stopping Ceph osd.11 for 19341aec-750e-11ef-bceb-c7b262605968... 2024-09-17T16:47:30.551 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:30.353+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:30.552 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:30.353+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:30.552 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:30.353+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:30.552 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:30.353+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:30.552 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:30.353+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:30.552 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:30.353+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:30.552 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:30.353+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:01.122417+0000 front 2024-09-17T16:47:01.122407+0000 (oldest deadline 2024-09-17T16:47:27.022223+0000) 2024-09-17T16:47:30.552 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:30.548+0000 7fa6f9a6e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T16:47:30.552 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:30.548+0000 7fa6f9a6e640 -1 osd.11 1060 *** Got signal Terminated *** 2024-09-17T16:47:30.552 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:30 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:30.548+0000 7fa6f9a6e640 -1 osd.11 1060 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T16:47:31.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:31 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:31.362+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:31.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:31 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:31.362+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:31.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:31 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:31.362+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:31.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:31 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:31.362+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:31.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:31 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:31.362+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:31.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:31 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:31.362+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:31.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:31 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:31.362+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:01.122417+0000 front 2024-09-17T16:47:01.122407+0000 (oldest deadline 2024-09-17T16:47:27.022223+0000) 2024-09-17T16:47:32.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:32 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:32.329+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:32.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:32 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:32.329+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:32.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:32 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:32.329+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:32.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:32 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:32.329+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:32.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:32 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:32.329+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:32.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:32 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:32.329+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:32.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:32 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:32.329+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:01.122417+0000 front 2024-09-17T16:47:01.122407+0000 (oldest deadline 2024-09-17T16:47:27.022223+0000) 2024-09-17T16:47:33.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:33.349+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:33.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:33.349+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:33.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:33.349+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:33.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:33.349+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:33.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:33.349+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:33.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:33.349+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:33.645 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:33 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:33.349+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:01.122417+0000 front 2024-09-17T16:47:01.122407+0000 (oldest deadline 2024-09-17T16:47:27.022223+0000) 2024-09-17T16:47:34.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:34 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:34.394+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:34.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:34 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:34.394+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:34.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:34 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:34.394+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:34.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:34 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:34.394+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:34.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:34 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:34.394+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:34.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:34 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:34.394+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:34.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:34 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:34.394+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:01.122417+0000 front 2024-09-17T16:47:01.122407+0000 (oldest deadline 2024-09-17T16:47:27.022223+0000) 2024-09-17T16:47:35.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:35 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:35.408+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-09-17T16:46:02.914292+0000 front 2024-09-17T16:46:02.914390+0000 (oldest deadline 2024-09-17T16:46:31.714450+0000) 2024-09-17T16:47:35.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:35 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:35.408+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-09-17T16:46:15.115749+0000 front 2024-09-17T16:46:15.115986+0000 (oldest deadline 2024-09-17T16:46:36.215966+0000) 2024-09-17T16:47:35.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:35 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:35.408+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-09-17T16:46:21.417122+0000 front 2024-09-17T16:46:21.417242+0000 (oldest deadline 2024-09-17T16:46:49.017703+0000) 2024-09-17T16:47:35.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:35 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:35.408+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-09-17T16:46:29.018253+0000 front 2024-09-17T16:46:29.018332+0000 (oldest deadline 2024-09-17T16:46:54.818769+0000) 2024-09-17T16:47:35.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:35 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:35.408+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-09-17T16:46:34.819471+0000 front 2024-09-17T16:46:34.819431+0000 (oldest deadline 2024-09-17T16:47:00.719333+0000) 2024-09-17T16:47:35.895 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:35 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:35.408+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6820 osd.6 since back 2024-09-17T16:46:55.321508+0000 front 2024-09-17T16:46:55.321484+0000 (oldest deadline 2024-09-17T16:47:17.021317+0000) 2024-09-17T16:47:35.896 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:35 smithi173 ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11[50745]: 2024-09-17T16:47:35.408+0000 7fa6f6083640 -1 osd.11 1060 heartbeat_check: no reply from 172.21.15.71:6828 osd.7 since back 2024-09-17T16:47:01.122417+0000 front 2024-09-17T16:47:01.122407+0000 (oldest deadline 2024-09-17T16:47:27.022223+0000) 2024-09-17T16:47:36.451 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:36 smithi173 podman[171333]: 2024-09-17 16:47:36.190061588 +0000 UTC m=+5.749061969 container died eca9d6cd5a3755393f8da4e6274e52cb41364fbc211f98dd754c991065dafd74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.build-date=20240731) 2024-09-17T16:47:36.794 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:36 smithi173 podman[171333]: 2024-09-17 16:47:36.624097897 +0000 UTC m=+6.183098271 container cleanup eca9d6cd5a3755393f8da4e6274e52cb41364fbc211f98dd754c991065dafd74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-17T16:47:36.794 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:36 smithi173 bash[171333]: ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11 2024-09-17T16:47:37.144 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:36 smithi173 podman[171348]: 2024-09-17 16:47:36.791807988 +0000 UTC m=+0.593581395 container remove eca9d6cd5a3755393f8da4e6274e52cb41364fbc211f98dd754c991065dafd74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-17T16:47:37.480 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:37 smithi173 podman[171436]: 2024-09-17 16:47:37.080597895 +0000 UTC m=+0.016291491 image pull a500243bbb0cc8c6726e1537e34504d4538602cfe0b0ac8f5e7dcf70d4cb6f8f quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0 2024-09-17T16:47:37.480 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:37 smithi173 podman[171436]: 2024-09-17 16:47:37.192765301 +0000 UTC m=+0.128458897 container create 3a90edcf2cf5fa22ec0b79ea6a024c69875c0e2e7a52d591213900cadcf92e85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-17T16:47:37.480 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:37 smithi173 podman[171436]: 2024-09-17 16:47:37.395001659 +0000 UTC m=+0.330695263 container init 3a90edcf2cf5fa22ec0b79ea6a024c69875c0e2e7a52d591213900cadcf92e85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T16:47:37.480 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:37 smithi173 podman[171436]: 2024-09-17 16:47:37.398221241 +0000 UTC m=+0.333914849 container start 3a90edcf2cf5fa22ec0b79ea6a024c69875c0e2e7a52d591213900cadcf92e85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11-deactivate, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-17T16:47:37.480 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:37 smithi173 podman[171436]: 2024-09-17 16:47:37.478269367 +0000 UTC m=+0.413962964 container attach 3a90edcf2cf5fa22ec0b79ea6a024c69875c0e2e7a52d591213900cadcf92e85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-17T16:47:37.882 INFO:journalctl@ceph.osd.11.smithi173.stdout:Sep 17 16:47:37 smithi173 podman[171436]: 2024-09-17 16:47:37.589116807 +0000 UTC m=+0.524810404 container died 3a90edcf2cf5fa22ec0b79ea6a024c69875c0e2e7a52d591213900cadcf92e85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7937eb648f1652aae087b08c64876b0aa6e55594e819c9a2dd402a47adfb5ee0, name=ceph-19341aec-750e-11ef-bceb-c7b262605968-osd-11-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-bharath10-testing-2024-09-17-1348-1889a16, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-17T16:47:38.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-19341aec-750e-11ef-bceb-c7b262605968@osd.11.service' 2024-09-17T16:47:38.208 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T16:47:38.208 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-17T16:47:38.208 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 19341aec-750e-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T16:47:38.420 INFO:teuthology.orchestra.run.smithi017.stdout:Deleting cluster with fsid: 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:47:40.272 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 19341aec-750e-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T16:47:40.472 INFO:teuthology.orchestra.run.smithi071.stdout:Deleting cluster with fsid: 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:47:42.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 19341aec-750e-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T16:47:42.523 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T16:47:44.071 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T16:47:44.105 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T16:47:44.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T16:47:44.165 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-17T16:47:44.168 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024/remote/smithi017/crash 2024-09-17T16:47:44.168 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/crash -- . 2024-09-17T16:47:44.256 INFO:teuthology.orchestra.run.smithi017.stderr:tar: /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T16:47:44.257 INFO:teuthology.orchestra.run.smithi017.stderr:tar: Error is not recoverable: exiting now 2024-09-17T16:47:44.259 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024/remote/smithi071/crash 2024-09-17T16:47:44.259 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/crash -- . 2024-09-17T16:47:44.380 INFO:teuthology.orchestra.run.smithi071.stderr:tar: /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T16:47:44.381 INFO:teuthology.orchestra.run.smithi071.stderr:tar: Error is not recoverable: exiting now 2024-09-17T16:47:44.382 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024/remote/smithi173/crash 2024-09-17T16:47:44.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/crash -- . 2024-09-17T16:47:44.428 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/19341aec-750e-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T16:47:44.428 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-09-17T16:47:44.429 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-17T16:47:44.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-17T16:47:44.528 INFO:teuthology.orchestra.run.smithi017.stdout:2024-09-17T16:20:00.000196+0000 mon.a (mon.0) 1905 : cluster [WRN] pg 4.b is stuck undersized for 76s, current state active+recovering+undersized+remapped, last acting [4,0] 2024-09-17T16:47:44.528 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-17T16:47:44.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[SEC\]' /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-17T16:47:44.571 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]' /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-17T16:47:44.651 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[WRN\]' /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-17T16:47:44.730 INFO:teuthology.orchestra.run.smithi017.stdout:2024-09-17T16:20:00.000196+0000 mon.a (mon.0) 1905 : cluster [WRN] pg 4.b is stuck undersized for 76s, current state active+recovering+undersized+remapped, last acting [4,0] 2024-09-17T16:47:44.730 INFO:tasks.cephadm:Compressing logs... 2024-09-17T16:47:44.730 DEBUG:teuthology.orchestra.run.smithi017:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T16:47:44.773 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T16:47:44.776 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T16:47:44.816 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.10.log 2024-09-17T16:47:44.816 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T16:47:44.816 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/objectstore_tool.2.log 2024-09-17T16:47:44.816 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.9.log 2024-09-17T16:47:44.816 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-17T16:47:44.816 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-17T16:47:44.816 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-17T16:47:44.816 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-17T16:47:44.816 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.11.log 2024-09-17T16:47:44.816 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log 2024-09-17T16:47:44.817 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.8.log 2024-09-17T16:47:44.817 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-17T16:47:44.821 INFO:teuthology.orchestra.run.smithi071.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-17T16:47:44.822 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/objectstore_tool.2.log 2024-09-17T16:47:44.822 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-17T16:47:44.825 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-17T16:47:44.825 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-17T16:47:44.825 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-17T16:47:44.825 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-17T16:47:44.825 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-17T16:47:44.826 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-17T16:47:44.826 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log 2024-09-17T16:47:44.826 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-17T16:47:44.827 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-17T16:47:44.827 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-17T16:47:44.838 INFO:teuthology.orchestra.run.smithi017.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-17T16:47:44.838 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-17T16:47:44.838 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-17T16:47:44.841 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-17T16:47:44.841 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.0.log: /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-17T16:47:44.841 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-17T16:47:44.841 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-17T16:47:44.841 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-17T16:47:44.841 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.cephadm.log: /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-17T16:47:44.841 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log 2024-09-17T16:47:44.841 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-17T16:47:44.841 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-17T16:47:44.850 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 85.8% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-17T16:47:44.857 INFO:teuthology.orchestra.run.smithi017.stderr: 82.7% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-17T16:47:44.876 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.8.log: /var/log/ceph/cephadm.log: 82.3% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-17T16:47:44.903 INFO:teuthology.orchestra.run.smithi017.stderr: 95.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-17T16:47:44.947 INFO:teuthology.orchestra.run.smithi173.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-17T16:47:44.952 INFO:teuthology.orchestra.run.smithi173.stderr: 89.7% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-17T16:47:44.957 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.2% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-17T16:47:44.959 INFO:teuthology.orchestra.run.smithi017.stderr: 91.1% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-17T16:47:44.964 INFO:teuthology.orchestra.run.smithi017.stderr: 89.7% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-17T16:47:45.009 INFO:teuthology.orchestra.run.smithi071.stderr: 94.7% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/objectstore_tool.2.log.gz 2024-09-17T16:47:45.027 INFO:teuthology.orchestra.run.smithi071.stderr: 82.4% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-17T16:47:45.062 INFO:teuthology.orchestra.run.smithi173.stderr: 91.2% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-17T16:47:45.067 INFO:teuthology.orchestra.run.smithi173.stderr: 94.6% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/objectstore_tool.2.log.gz 2024-09-17T16:47:45.101 INFO:teuthology.orchestra.run.smithi071.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-17T16:47:45.103 INFO:teuthology.orchestra.run.smithi071.stderr: 89.7% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-17T16:47:45.126 INFO:teuthology.orchestra.run.smithi071.stderr: 85.8% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-17T16:47:45.214 INFO:teuthology.orchestra.run.smithi071.stderr: 92.9% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-17T16:47:46.018 INFO:teuthology.orchestra.run.smithi173.stderr: 85.8% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-17T16:47:46.284 INFO:teuthology.orchestra.run.smithi017.stderr: 91.5% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-17T16:47:56.010 INFO:teuthology.orchestra.run.smithi173.stderr: 91.2% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-17T16:47:58.012 INFO:teuthology.orchestra.run.smithi071.stderr: 91.0% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-17T16:48:22.739 INFO:teuthology.orchestra.run.smithi017.stderr: 90.0% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-17T16:48:24.528 INFO:teuthology.orchestra.run.smithi173.stderr: 94.5% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.11.log.gz 2024-09-17T16:50:00.278 INFO:teuthology.orchestra.run.smithi173.stderr: 92.7% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.9.log.gz 2024-09-17T16:52:23.576 INFO:teuthology.orchestra.run.smithi173.stderr: 92.5% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.8.log.gz 2024-09-17T16:52:50.543 INFO:teuthology.orchestra.run.smithi173.stderr: 92.3% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.10.log.gz 2024-09-17T16:52:50.545 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-09-17T16:52:50.546 INFO:teuthology.orchestra.run.smithi173.stderr:real 5m5.755s 2024-09-17T16:52:50.546 INFO:teuthology.orchestra.run.smithi173.stderr:user 5m5.488s 2024-09-17T16:52:50.546 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m12.909s 2024-09-17T17:07:12.973 INFO:teuthology.orchestra.run.smithi071.stderr: 92.0% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-17T17:09:29.330 INFO:teuthology.orchestra.run.smithi071.stderr: 92.4% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-17T17:10:39.003 INFO:teuthology.orchestra.run.smithi071.stderr: 92.3% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-17T17:10:52.476 INFO:teuthology.orchestra.run.smithi071.stderr: 91.8% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-17T17:10:52.479 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-09-17T17:10:52.479 INFO:teuthology.orchestra.run.smithi071.stderr:real 23m7.688s 2024-09-17T17:10:52.479 INFO:teuthology.orchestra.run.smithi071.stderr:user 7m54.401s 2024-09-17T17:10:52.480 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m23.664s 2024-09-17T17:13:33.922 INFO:teuthology.orchestra.run.smithi017.stderr: 91.8% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-17T17:15:26.240 INFO:teuthology.orchestra.run.smithi017.stderr: 91.5% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-17T17:22:37.274 INFO:teuthology.orchestra.run.smithi017.stderr: 91.6% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-17T17:22:47.554 INFO:teuthology.orchestra.run.smithi017.stderr: 91.1% -- replaced with /var/log/ceph/19341aec-750e-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-17T17:22:47.556 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-17T17:22:47.556 INFO:teuthology.orchestra.run.smithi017.stderr:real 35m2.770s 2024-09-17T17:22:47.556 INFO:teuthology.orchestra.run.smithi017.stderr:user 12m57.258s 2024-09-17T17:22:47.556 INFO:teuthology.orchestra.run.smithi017.stderr:sys 0m35.156s 2024-09-17T17:22:47.558 INFO:tasks.cephadm:Archiving logs... 2024-09-17T17:22:47.559 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024/remote/smithi017/log 2024-09-17T17:22:47.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T17:24:55.368 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024/remote/smithi071/log 2024-09-17T17:24:55.370 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T17:26:14.275 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024/remote/smithi173/log 2024-09-17T17:26:14.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T17:27:04.319 INFO:tasks.cephadm:Removing cluster... 2024-09-17T17:27:04.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 19341aec-750e-11ef-bceb-c7b262605968 --force 2024-09-17T17:27:04.553 INFO:teuthology.orchestra.run.smithi017.stdout:Deleting cluster with fsid: 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T17:27:05.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 19341aec-750e-11ef-bceb-c7b262605968 --force 2024-09-17T17:27:05.808 INFO:teuthology.orchestra.run.smithi071.stdout:Deleting cluster with fsid: 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T17:27:06.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 19341aec-750e-11ef-bceb-c7b262605968 --force 2024-09-17T17:27:06.827 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: 19341aec-750e-11ef-bceb-c7b262605968 2024-09-17T17:27:07.465 INFO:tasks.cephadm:Removing cephadm ... 2024-09-17T17:27:07.466 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T17:27:07.485 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T17:27:07.501 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T17:27:07.517 INFO:tasks.cephadm:Teardown complete 2024-09-17T17:27:07.517 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T17:27:07.527 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T17:27:07.527 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T17:27:07.530 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T17:27:07.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T17:27:07.654 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T17:27:07.664 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T17:27:07.664 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T17:27:07.667 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T17:27:07.669 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T17:27:07.680 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-09-17T17:27:07.682 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-09-17T17:27:07.683 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-09-17T17:27:07.700 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T17:27:07.700 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-09-17T17:27:07.701 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv01.front.sepia.ceph.com 3 7 377 126 +242us[ +251us] +/- 41ms 2024-09-17T17:27:07.701 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 63 +191us[ +191us] +/- 75ms 2024-09-17T17:27:07.701 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -836us[ -836us] +/- 83ms 2024-09-17T17:27:07.701 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T17:27:07.729 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T17:27:07.729 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-09-17T17:27:07.729 INFO:teuthology.orchestra.run.smithi017.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 +244us[ +233us] +/- 42ms 2024-09-17T17:27:07.729 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 62 +235us[ +235us] +/- 75ms 2024-09-17T17:27:07.729 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -813us[ -824us] +/- 83ms 2024-09-17T17:27:07.730 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T17:27:07.738 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T17:27:07.738 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-09-17T17:27:07.739 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv01.front.sepia.ceph.com 3 6 377 1 +241us[ +230us] +/- 43ms 2024-09-17T17:27:07.739 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 58 +245us[ +235us] +/- 75ms 2024-09-17T17:27:07.739 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -800us[ -810us] +/- 83ms 2024-09-17T17:27:07.739 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T17:27:07.740 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T17:27:07.748 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T17:27:07.748 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T17:27:07.811 DEBUG:teuthology.orchestra.run.smithi017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T17:27:07.941 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T17:27:08.014 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T17:27:08.090 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-17T17:27:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:27:08.122 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-17T17:27:08.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:27:08.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-17T17:27:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:27:08.176 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T17:27:08.185 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T17:27:08.195 INFO:teuthology.task.internal:Duration was 5473.755274 seconds 2024-09-17T17:27:08.196 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T17:27:08.204 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T17:27:08.205 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T17:27:08.207 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T17:27:08.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T17:27:08.262 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T17:27:08.267 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T17:27:08.284 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T17:27:08.628 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T17:27:08.628 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2024-09-17T17:27:08.628 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T17:27:08.653 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-09-17T17:27:08.653 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T17:27:08.684 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-09-17T17:27:08.684 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T17:27:08.708 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T17:27:08.708 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T17:27:08.711 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T17:27:08.726 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T17:27:08.775 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T17:27:08.775 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T17:27:08.782 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T17:27:08.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T17:27:11.226 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T17:27:11.235 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T17:27:11.235 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T17:27:11.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T17:27:11.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T17:27:11.355 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T17:27:11.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:27:11.367 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:27:11.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:27:11.396 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2024-09-17T17:27:11.403 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-09-17T17:27:11.431 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-09-17T17:27:11.452 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:27:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:27:11.470 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:27:11.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:27:11.492 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:27:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:27:11.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T17:27:11.518 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T17:27:11.519 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024/remote/smithi017 2024-09-17T17:27:11.520 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T17:27:11.593 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024/remote/smithi071 2024-09-17T17:27:11.594 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T17:27:11.638 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909024/remote/smithi173 2024-09-17T17:27:11.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T17:27:11.689 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T17:27:11.689 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T17:27:11.692 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T17:27:11.695 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T17:27:11.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T17:27:11.753 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T17:27:11.753 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T17:27:11.763 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T17:27:11.763 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T17:27:11.766 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T17:27:11.768 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T17:27:11.779 INFO:teuthology.orchestra.run.smithi017.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 17:27 /home/ubuntu/cephtest 2024-09-17T17:27:11.782 INFO:teuthology.orchestra.run.smithi071.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 17:27 /home/ubuntu/cephtest 2024-09-17T17:27:11.793 INFO:teuthology.orchestra.run.smithi173.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 17:27 /home/ubuntu/cephtest 2024-09-17T17:27:11.795 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T17:27:11.804 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T17:27:11.827 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} duration: 5473.755273580551 failure_reason: '"2024-09-17T16:20:00.000196+0000 mon.a (mon.0) 1905 : cluster [WRN] pg 4.b is stuck undersized for 76s, current state active+recovering+undersized+remapped, last acting [4,0]" in cluster log' flavor: default owner: scheduled_skanta@teuthology success: false 2024-09-17T17:27:11.828 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T17:27:11.943 INFO:teuthology.run:FAIL